69
07/05/22 CPSC503 Winter 2008 1 CPSC 503 Computational Linguistics Lecture 8 Giuseppe Carenini
| Date post: | 08-Jan-2018 |
| Category: |
Documents |
| Upload: | kelley-adams |
| View: | 215 times |
| Download: | 0 times |
050323 CPSC503 Winter 2008 1
CPSC 503Computational Linguistics
Lecture 8Giuseppe Carenini
050323 CPSC503 Winter 2008 2
Knowledge-Formalisms Map
Logical formalisms (First-Order Logics)
Rule systems (and prob versions)(eg (Prob) Context-Free
Grammars)
State Machines (and prob versions)
(Finite State AutomataFinite State Transducers Markov Models)
Morphology
Syntax
PragmaticsDiscourse
and Dialogue
Semantics
AI planners
050323 CPSC503 Winter 2008 3
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 4
Problems with CFGsbull Agreement
bull Subcategorization
050323 CPSC503 Winter 2008 5
Agreementbull In English
ndash Determiners and nouns have to agree in number
ndash Subjects and verbs have to agree in person and number
bull Many languages have agreement systems that are far more complex than this (eg gender)
050323 CPSC503 Winter 2008 6
Agreementbull This dogbull Those dogs
bull This dog eatsbull You have itbull Those dogs eat
bull This dogsbull Those dog
bull This dog eatbull You has itbull Those dogs
eats
050323 CPSC503 Winter 2008 7
Possible CFG Solution
bull S -gt NP VPbull NP -gt Det Nombull VP -gt V NPbull hellip
bull SgS -gt SgNP SgVPbull PlS -gt PlNp PlVPbull SgNP -gt SgDet SgNombull PlNP -gt PlDet PlNombull PlVP -gt PlV NPbull SgVP -gtSgV NPbull hellip
Sg = singularPl = plural
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 8
CFG Solution for Agreementbull It works and stays within the
power of CFGsbull But it doesnrsquot scale all that
well (explosion in the number of rules)
050323 CPSC503 Winter 2008 9
Subcategorization
bull John sneezed the bookbull I prefer United has a flightbull Give with a flight
bull Def It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 2
Knowledge-Formalisms Map
Logical formalisms (First-Order Logics)
Rule systems (and prob versions)(eg (Prob) Context-Free
Grammars)
State Machines (and prob versions)
(Finite State AutomataFinite State Transducers Markov Models)
Morphology
Syntax
PragmaticsDiscourse
and Dialogue
Semantics
AI planners
050323 CPSC503 Winter 2008 3
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 4
Problems with CFGsbull Agreement
bull Subcategorization
050323 CPSC503 Winter 2008 5
Agreementbull In English
ndash Determiners and nouns have to agree in number
ndash Subjects and verbs have to agree in person and number
bull Many languages have agreement systems that are far more complex than this (eg gender)
050323 CPSC503 Winter 2008 6
Agreementbull This dogbull Those dogs
bull This dog eatsbull You have itbull Those dogs eat
bull This dogsbull Those dog
bull This dog eatbull You has itbull Those dogs
eats
050323 CPSC503 Winter 2008 7
Possible CFG Solution
bull S -gt NP VPbull NP -gt Det Nombull VP -gt V NPbull hellip
bull SgS -gt SgNP SgVPbull PlS -gt PlNp PlVPbull SgNP -gt SgDet SgNombull PlNP -gt PlDet PlNombull PlVP -gt PlV NPbull SgVP -gtSgV NPbull hellip
Sg = singularPl = plural
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 8
CFG Solution for Agreementbull It works and stays within the
power of CFGsbull But it doesnrsquot scale all that
well (explosion in the number of rules)
050323 CPSC503 Winter 2008 9
Subcategorization
bull John sneezed the bookbull I prefer United has a flightbull Give with a flight
bull Def It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 3
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 4
Problems with CFGsbull Agreement
bull Subcategorization
050323 CPSC503 Winter 2008 5
Agreementbull In English
ndash Determiners and nouns have to agree in number
ndash Subjects and verbs have to agree in person and number
bull Many languages have agreement systems that are far more complex than this (eg gender)
050323 CPSC503 Winter 2008 6
Agreementbull This dogbull Those dogs
bull This dog eatsbull You have itbull Those dogs eat
bull This dogsbull Those dog
bull This dog eatbull You has itbull Those dogs
eats
050323 CPSC503 Winter 2008 7
Possible CFG Solution
bull S -gt NP VPbull NP -gt Det Nombull VP -gt V NPbull hellip
bull SgS -gt SgNP SgVPbull PlS -gt PlNp PlVPbull SgNP -gt SgDet SgNombull PlNP -gt PlDet PlNombull PlVP -gt PlV NPbull SgVP -gtSgV NPbull hellip
Sg = singularPl = plural
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 8
CFG Solution for Agreementbull It works and stays within the
power of CFGsbull But it doesnrsquot scale all that
well (explosion in the number of rules)
050323 CPSC503 Winter 2008 9
Subcategorization
bull John sneezed the bookbull I prefer United has a flightbull Give with a flight
bull Def It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 4
Problems with CFGsbull Agreement
bull Subcategorization
050323 CPSC503 Winter 2008 5
Agreementbull In English
ndash Determiners and nouns have to agree in number
ndash Subjects and verbs have to agree in person and number
bull Many languages have agreement systems that are far more complex than this (eg gender)
050323 CPSC503 Winter 2008 6
Agreementbull This dogbull Those dogs
bull This dog eatsbull You have itbull Those dogs eat
bull This dogsbull Those dog
bull This dog eatbull You has itbull Those dogs
eats
050323 CPSC503 Winter 2008 7
Possible CFG Solution
bull S -gt NP VPbull NP -gt Det Nombull VP -gt V NPbull hellip
bull SgS -gt SgNP SgVPbull PlS -gt PlNp PlVPbull SgNP -gt SgDet SgNombull PlNP -gt PlDet PlNombull PlVP -gt PlV NPbull SgVP -gtSgV NPbull hellip
Sg = singularPl = plural
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 8
CFG Solution for Agreementbull It works and stays within the
power of CFGsbull But it doesnrsquot scale all that
well (explosion in the number of rules)
050323 CPSC503 Winter 2008 9
Subcategorization
bull John sneezed the bookbull I prefer United has a flightbull Give with a flight
bull Def It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 5
Agreementbull In English
ndash Determiners and nouns have to agree in number
ndash Subjects and verbs have to agree in person and number
bull Many languages have agreement systems that are far more complex than this (eg gender)
050323 CPSC503 Winter 2008 6
Agreementbull This dogbull Those dogs
bull This dog eatsbull You have itbull Those dogs eat
bull This dogsbull Those dog
bull This dog eatbull You has itbull Those dogs
eats
050323 CPSC503 Winter 2008 7
Possible CFG Solution
bull S -gt NP VPbull NP -gt Det Nombull VP -gt V NPbull hellip
bull SgS -gt SgNP SgVPbull PlS -gt PlNp PlVPbull SgNP -gt SgDet SgNombull PlNP -gt PlDet PlNombull PlVP -gt PlV NPbull SgVP -gtSgV NPbull hellip
Sg = singularPl = plural
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 8
CFG Solution for Agreementbull It works and stays within the
power of CFGsbull But it doesnrsquot scale all that
well (explosion in the number of rules)
050323 CPSC503 Winter 2008 9
Subcategorization
bull John sneezed the bookbull I prefer United has a flightbull Give with a flight
bull Def It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 6
Agreementbull This dogbull Those dogs
bull This dog eatsbull You have itbull Those dogs eat
bull This dogsbull Those dog
bull This dog eatbull You has itbull Those dogs
eats
050323 CPSC503 Winter 2008 7
Possible CFG Solution
bull S -gt NP VPbull NP -gt Det Nombull VP -gt V NPbull hellip
bull SgS -gt SgNP SgVPbull PlS -gt PlNp PlVPbull SgNP -gt SgDet SgNombull PlNP -gt PlDet PlNombull PlVP -gt PlV NPbull SgVP -gtSgV NPbull hellip
Sg = singularPl = plural
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 8
CFG Solution for Agreementbull It works and stays within the
power of CFGsbull But it doesnrsquot scale all that
well (explosion in the number of rules)
050323 CPSC503 Winter 2008 9
Subcategorization
bull John sneezed the bookbull I prefer United has a flightbull Give with a flight
bull Def It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 7
Possible CFG Solution
bull S -gt NP VPbull NP -gt Det Nombull VP -gt V NPbull hellip
bull SgS -gt SgNP SgVPbull PlS -gt PlNp PlVPbull SgNP -gt SgDet SgNombull PlNP -gt PlDet PlNombull PlVP -gt PlV NPbull SgVP -gtSgV NPbull hellip
Sg = singularPl = plural
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 8
CFG Solution for Agreementbull It works and stays within the
power of CFGsbull But it doesnrsquot scale all that
well (explosion in the number of rules)
050323 CPSC503 Winter 2008 9
Subcategorization
bull John sneezed the bookbull I prefer United has a flightbull Give with a flight
bull Def It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 8
CFG Solution for Agreementbull It works and stays within the
power of CFGsbull But it doesnrsquot scale all that
well (explosion in the number of rules)
050323 CPSC503 Winter 2008 9
Subcategorization
bull John sneezed the bookbull I prefer United has a flightbull Give with a flight
bull Def It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 9
Subcategorization
bull John sneezed the bookbull I prefer United has a flightbull Give with a flight
bull Def It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 10
Subcategorizationbull Sneeze John sneezedbull Find Please find [a flight to NY]NP
bull Give Give [me]NP[a cheaper fare]NP
bull Help Can you help [me]NP[with a flight]PP
bull Prefer I prefer [to leave earlier]TO-VP
bull Told I was told [United has a flight]S
bull hellip
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 11
Sobull So the various rules for VPs
overgeneratendash They allow strings containing verbs
and arguments that donrsquot go togetherndash For example
bullVP -gt V NP therefore Sneezed the bookbullVP -gt V S therefore go she will go there
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 12
Possible CFG Solution
bull VP -gt Vbull VP -gt V NPbull VP -gt V NP PPbull hellip
bull VP -gt IntransVbull VP -gt TransV NPbull VP -gt TransPPto NP
PPto
bull hellipbull TransPPto -gt
handgiveThis solution has the same problem as the one for agreement
OLD Grammar NEW Grammar
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 13
CFG for NLP summarybull CFGs cover most syntactic structure in
Englishbull But there are problems
(overgeneration)ndash That can be dealt with adequately
although not elegantly by staying within the CFG framework
bull There are simpler more elegant solutions that take us out of the CFG framework LFG XTAGShellipsee Chpt 15 ldquoFeatures and Unificationrdquo
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 14
Today 110bull Finish CFG for Syntax of NL
(problems)bull Parsingbull The Earley Algorithmbull Partial Parsing Chunckingbull Dependency Grammars Parsing
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 15
Parsing with CFGs
Assign valid trees covers all and only the elements of the input and has an S at
the top
ParserI prefer a morning flight
flight
Nominal
Nominal
CFG
Sequence of wordsValid parse trees
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 16
Parsing as Searchbull S -gt NP VPbull S -gt Aux NP VPbull NP -gt Det Nounbull VP -gt Verbbull Det -gt abull Noun -gt flightbull Verb -gt left
arrivebull Aux -gt do does
Search space of possible parse trees
CFG
defines
Parsing find all trees that cover all and only the words in the input
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 17
Constraints on Search
ParserI prefer a morning flight
flight
Nominal
NominalCFG
(search space)
Sequence of words Valid parse trees
Search Strategies bull Top-down or goal-directedbull Bottom-up or data-directed
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 18
Top-Down Parsingbull Since wersquore trying to find trees
rooted with an S (Sentences) start with the rules that give us an S
bull Then work your way down from there to the words flightInput
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 19
Next step Top Down Space
bull When POS categories are reached reject trees whose leaves fail to match all words in the input
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 20
Bottom-Up Parsingbull Of course we also want trees that
cover the input words So start with trees that link up with the words in the right way
bull Then work your way up from there
flight
flight
flight
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 21
Two more steps Bottom-Up Space
flightflightflight
flightflight
flightflight
helliphellip helliphellip helliphellip
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 22
Top-Down vs Bottom-Upbull Top-down
ndash Only searches for trees that can be answers
ndash But suggests trees that are not consistent with the words
bull Bottom-upndash Only forms trees consistent with the
wordsndash Suggest trees that make no sense
globally
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 23
So Combine Thembull Top-down control strategy to generate
treesbull Bottom-up to filter out inappropriate
parsesTop-down Control strategybull Depth vs Breadth firstbull Which node to try to expand nextbull Which grammar rule to use to expand a
node
(left-most)
(textual order)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 24
Top-Down Depth-First Left-to-Right Search
Sample sentence ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 25
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 26
flightflight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 27
flight flight
Example ldquoDoes this flight include a mealrdquo
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 28
Adding Bottom-up Filtering The following sequence was a waste of
time because an NP cannot generate a parse tree starting with an Aux
Aux Aux Aux Aux
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 29
Bottom-Up FilteringCategory Left Corners
S Det Proper-Noun Aux VerbNP Det Proper-Noun
Nominal NounVP Verb
Aux Aux Aux
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 30
Problems with TD-BU-filteringbull Left recursionbull Ambiguitybull Repeated Parsing
bull SOLUTION Earley Algorithm (once again dynamic programming)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 31
(1) Left-RecursionThese rules appears in most English
grammarsS -gt S and SVP -gt VP PPNP -gt NP PP
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 32
(2) Structural Ambiguity
ldquoI shot an elephant in my pajamasrdquo
of PP
of NP parses
hellip hellip6 4697 1430hellip hellip
Three basic kindsAttachmentCoordinationNP-
bracketing
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 33
(3) Repeated Workbull Parsing is hard and slow Itrsquos
wasteful to redo stuff over and over and over
bull Consider an attempt to top-down parse the following as an NP
ldquoA flight from Indi to Houston on TWArdquo
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 34
flight
NP -gt Det NomNP-gt NP PP Nom -gt Nounhelliphellip
fails and backtracks
starts fromhellip
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 35
flight
NP -gt Det NomNP-gt NP PP Nom -gt Noun
fails and backtracks
restarts fromhellip
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 36flight
restarts fromhellip
fails and backtracks
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 37
restarts fromhellip
Success
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 38
4
3
2
1
Buthellip
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 39
Dynamic ProgrammingFills tables with solution to
subproblemsParsing sub-trees consistent
with the input once discovered are stored and can be reused1 Does not fall prey to left-recursion
2 Stores ambiguous parse compactly
3 Does not do (avoidable) repeated work
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 40
Earley Parsing O(N3)bull Fills a table in a single sweep over the
input wordsndash Table is length N +1 N is number of
wordsndash Table entries represent
bullPredicted constituentsbullIn-progress constituentsbullCompleted constituents and their locations
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 41
For Next Timebull Read 127bull Read in Chapter 13 (Parsing) 1342 135bull Optional Read Chapter 16 (Features and Unification) ndash skip algorithms and
implementation
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
Final Project Decisionbull Two ways Select and NLP task
problem or a technique used in NLP that truly interests you
bull Tasks summarization of helliphellip computing similarity between two termssentences (skim through the textbook)
bull Techniques extensions variations combinations of what we saw in class ndash Max Entropy Classifiers or MM Dirichlet Multinomial Distributions050323 CPSC503 Winter 2008 42
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
Final Project goals (and hopefully contributions )
bull Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques
bull Proposing a novel (minimally is OK) different solution
bull Apply a technique which has been used for nlp taskA to a different nlp taskB
bull Apply a technique to a different dataset or to a different language
bull Proposing a different evaluation measure050323 CPSC503 Winter 2008 43
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
Final Project Examples Ideasbull Look on the course WebPage
050323 CPSC503 Winter 2008 44
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 45
Today 110bull Finish CFG for Syntax of NLbull Parsingbull The Earley Algorithmbull Partial Parsing Chuncking
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 46
StatesThe table-entries are called states and expressbull what is predicted from that pointbull what has been recognized up to that pointRepresentation dotted-rules + location
S -gt VP [00] A VP is predicted at the start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending at 3
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 47
Graphically
S -gt VP [00]NP -gt Det Nominal [12]VP -gt V NP [03]
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 48
Earley answerbull Answer found by looking in the table in
the right placebull The following state should be in the final
column
bull ie an S state that spans from 0 to n and is complete
S ndashgt [0n]
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 49
Earley Parsing Procedurebull So sweep through the table from 0 to n in
order applying one of three operators to each statendash predictor add top-down predictions to the
chartndash scanner read input and add corresponding
state to chartndash completer move dot to right when new
constituent foundbull Results (new states) added to current or
next set of states in chartbull No backtracking and no states removed
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 50
Predictorbull Intuition new states represent top-
down expectationsbull Applied when non-part-of-speech non-
terminals are to the right of a dotS --gt bull VP [00]
bull Adds new states to end of current chartndash One new state for each expansion of the
non-terminal in the grammarVP --gt bull V [00]VP --gt bull V NP [00]
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 51
Scanner (part of speech)bull New states for predicted part of
speechbull Applicable when part of speech is
to the right of a dotVP --gt bull Verb NP [00] ( 0 ldquoBookhelliprdquo 1 )
bull Looks at current word in inputbull If match adds state(s) to next chart
Verb --gt book bull NP [01]
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 52
Completerbull Intuition wersquove found a constituent so
tell everyone waiting for thisbull Applied when dot has reached right end
of ruleNP --gt Det Nom bull [13]
bull Find all states wdot at 1 and expecting an NPVP --gt V bull NP [01]
bull Adds new (completed) state(s) to current chartVP --gt V NP bull [03]
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 53
Example ldquoBook that flightrdquobull We should findhellip an S from 0 to 3
that is a completed statehellip
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 54
Example ldquoBook that flightrdquo
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 55
So far only a recognizerhellip
bull Then simply read off all the backpointers from every complete S in the last column of the table
To generate all parsesbull When old states waiting for the just
completed constituent are updated =gt add a pointer from each ldquoupdatedrdquo to ldquocompletedrdquoChart [0]
hellipS5 S-gtVP [00] []S6 VP -gt Verb [00] []S7 VP -gt Verb NP[00] []hellip
Chart [1]S8 Verb -gt book [01] []S9 VP -gt Verb [01] [S8]S10 S-gtVP [01] [S9]S11 VP-gtVerb NP [01] []hellip
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 56
Error Handlingbull What happens when we look at the
contents of the last table column and dont find a S --gt statendash Is it a total loss Nondash Chart contains every constituent and
combination of constituents possible for the input given the grammar
bull Also useful for partial parsing or shallow parsing used in information extraction
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 57
Earley and Left Recursionbull So Earley solves the left-recursion
problem without having to alter the grammar or artificially limiting the searchndash Never place a state into the chart
thatrsquos already therendash Copy states before advancing
them
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 58
Earley and Left Recursion 1bull S -gt NP VPbull NP -gt NP PP
bull The first rule predictsndash S -gt NP VP [00] that addsndash NP -gt NP PP [00]ndash stops there since adding any
subsequent prediction would be fruitless
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 59
Earley and Left Recursion 2bull When a state gets advanced make a
copy and leave the original alonehellip
bull Say we have NP -gt NP PP [00]bull We find an NP from 0 to 2 so we
createNP -gt NP PP [02]
bull But we leave the original state as is
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 60
Dynamic Programming Approaches
bull Earleyndash Top-down no filtering no
restriction on grammar formbull CKY
ndash Bottom-up no filtering grammars restricted to Chomsky-Normal Form (CNF) (ie -free and each production either A-gt BC or A-gt a)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 61
Today 410bull The Earley Algorithmbull Partial Parsing Chunking
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 62
Chunkingbull Classify only basic non-recursive
phrases (NP VP AP PP)ndash Find non-overlapping chunksndash Assign labels to chunks
bull Chunk typically includes headword and pre-head material
[NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 63
Approaches to Chunking (1) Finite-State Rule-Based
bull Set of hand-crafted rules (no recursion) eg NP -gt (Det) Noun Noun
bull Implemented as FSTs (unionizeddeteminizedminimized)
bull F-measure 85-92bull To build tree-like structures several
FSTs can be combined [Abney rsquo96]
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 64
Approaches to Chunking (1) Finite-State Rule-Based
bull hellip several FSTs can be combined
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 65
Approaches to Chunking (2) Machine Learning
bull A case of sequential classificationbull IOB tagging (I) internal (O) outside
(B) beginningbull Internal and Beginning for each
chunk type =gt size of tagset (2n + 1) where n is the num of chunk types
bull Find an annotated corpus
bull Select feature setbull Select and train a
classifier
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 66
Context window approachbull Typical features
ndash Current previous following wordsndash Current previous following POSndash Previous chunks
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 67
Context window approachbull Specific choice of machine learning
approach does not seem to matterbull F-measure 92-94 rangebull Common causes of errors
ndash POS tagger inaccuracies ndash Inconsistencies in training corpusndash Ambiguities involving conjunctions (eg
ldquolate arrivals and cancellationsdeparture are common in winterrdquo )
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 68
For Next Timebull Read Chapter 14 (Probabilistic
CFG and Parsing)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
050323 CPSC503 Winter 2008 69
Dependency Grammarsbull Syntactic structure binary relations
between wordsbull Links grammatical function or very
general semantic relation
bull Abstract away from word-order variations (simpler grammars)
bull Useful features in many NLP applications (for classification summarization and NLG)
