Word Sense Disambiguation

2014.05.10

Minho Kim(karma@pusan.ac.kr)

Foundation of Statistical Natural Language Processing

Motivation

• Computationally determining which sense of a word is activated by its use in a particular context.

– E.g. I am going to withdraw money from the bank.

• One of the central challenges in NLP.• Needed in:

– Machine Translation: For correct lexical choice.

– Information Retrieval: Resolving ambiguity in queries.

– Information Extraction: For accurate analysis of text.

Senses and ambiguity

• Many words have different meanings (senses) in different contexts– E.g. Bank river bank; financial institution

• The problem in general is more complicated by the fact that the “senses” of a particular word are just subtly different.

Homonym and Polysemy

POS Tagging

• Some words are used in different parts of speech– “They're waiting in line at the ticket office.” Noun– “You should line a coat with fur” Verb

• The techniques used for tagging and senses disambiguation are bit different. – For tagging the local context is heavily used – looking

at the use of determiners and predicates and the like. – For word sense disambiguation the techniques look at

a broader context of the word. Tagging is explored in Chapter 10.

Methodological Preliminaries

• Corpus Based Approaches– Rely on corpus evidence.– Supervised and unsupervised learning– Train a model using tagged or untagged corpus.– Probabilistic/Statistical models.

• Knowledge Based Approaches– Rely on knowledge resources like WordNet,

Thesaurus etc.– May use grammar rules for disambiguation.– May use hand coded rules for disambiguation.

• Hybrid Approaches– Use corpus evidence as well as semantic relations form

WordNet.

Corpus Based Approaches

• Supervised and unsupervised learning– In supervised learning, we know the actual

“sense” of a word which is labeled – Supervised learning tends to be a classification

task– Unsupervised tends to be a clustering task

• Providing labeled corpora is expensive• Knowledge sources to help with task– Dictionaries, thesaurus, aligned bilingual texts

Pseudowords

• When one has difficulty coming up with sufficient training and test data, one techniques is to create “pseudowords” from an existing corpora.

• For e.g. replace banana and door with the pseudoword “banana-door”. – The ambiguous set is the text with

pseudowords. – The disambiguated set is the original text.

Upper and lower bounds

• Upper and lower bounds on performance– Upper bound is usually defined as human

performance– Lower bound is given by the simplest possible

algorithm• Most Frequent Class• Naïve Bayes

• Evaluation measure– Precision, Recall, F-measure

Supervised Disambiguation

Classification and ClusteringClass A

Class B

Class C

ModelA

B

C

Sense Tagged Corpus<p>BSAA0011-00018403 서양에만 서양 /NNG + 에 /JKB + 만 /JXBSAA0011-00018404 젤리가 젤리 /NNG + 가 /JKSBSAA0011-00018405 있는 있 /VV + 는 /ETMBSAA0011-00018406 것이 것 /NNB + 이 /JKCBSAA0011-00018407 아니라 아니 /VCN + 라 /ECBSAA0011-00018408 우리 우리 /NPBSAA0011-00018409 나라에서도 나라 /NNG + 에서 /JKB + 도 /JXBSAA0011-00018410 앵두 앵두 /NNGBSAA0011-00018411 사과 사과 __05/NNGBSAA0011-00018412 모과 모과 __02/NNGBSAA0011-00018413 살구 살구 /NNGBSAA0011-00018414 같은 같 /VA + 은 /ETMBSAA0011-00018415 과일로 과일 __01/NNG + 로 /JKBBSAA0011-00018416 '과편 '을 '/SS + 과편 /NNG + '/SS + 을 /JKOBSAA0011-00018417 만들어 만들 /VV + 어 /ECBSAA0011-00018418 먹었지만 먹 __02/VV + 었 /EP + 지만 /ECBSAA0011-00018419 수박은 수박 __01/NNG + 은 /JXBSAA0011-00018420 물기가 물기 /NNG + 가 /JKSBSAA0011-00018421 너무 너무 /MAGBSAA0011-00018422 많고 많 /VA + 고 /ECBSAA0011-00018423 펙틴질이 펙틴질 /NNG + 이 /JKSBSAA0011-00018424 없어 없 /VA + 어 /ECBSAA0011-00018425 가공해 가공 __01/NNG + 하 /XSV + 아 /ECBSAA0011-00018426 먹지 먹 __02/VV + 지 /ECBSAA0011-00018427 못했다 . 못하 /VX + 았 /EP + 다 /EF + ./SF</p>

Notational Conventions

Supervised task

• The idea here is that there is a training set of exemplars which tags each word that needs to be disambiguated with the correct “sense” of the word.

• The task is to correctly classify the word sense in the testing set using the statistical properties gleaned from the training set for the occurrence of the word in a particular context

• This chapter explores two approaches to this problem– Bayesian approach and information theoretic

approach

Bayesian Classification

16

Prior Probability

• Prior probability: the probability before we consider any additional knowledge

)(AP

17

Conditional probability

• Sometimes we have partial knowledge about the outcome of an experiment

• Conditional (or Posterior) Probability

• Suppose we know that event B is true

• The probability that A is true given the knowledge about B is expressed by

)|( BAP

P(B)

P(A,B)P(A|B)

18

http://ai.stanford.edu/~paskin/gm-short-course/lec1.pdf

19

Conditional probability (cont)

)()|(

)()|(),(

APABP

BPBAPBAP

• Note: P(A,B) = P(A ∩ B) • Chain Rule• P(A, B) = P(A|B) P(B) = The probability that A and B both happen is the

probability that B happens times the probability that A happens, given B has occurred.

• P(A, B) = P(B|A) P(A) = The probability that A and B both happen is the probability that A happens times the probability that B happens, given A has occurred.

• Multi-dimensional table with a value in every cell giving the probability of that specific state occurring

20

Chain Rule

P(A,B) = P(A|B)P(B)

= P(B|A)P(A)

P(A,B,C,D…) = P(A)P(B|A)P(C|A,B)P(D|A,B,C..)

21

Chain Rule Bayes' rule

P(A,B) = P(A|B)P(B)

= P(B|A)P(A)

P(B)

P(B|A)P(A)P(A|B)

Bayes' rule

Useful when one quantity is more easy to calculate; trivial consequence of the definitions we saw but it’ s extremely useful

22

Bayes' rule

P(B)

P(A|B)P(A)P(A|B)

Bayes' rule translates causal knowledge into diagnostic knowledge.

For example, if A is the event that a patient has a disease, and B is the event that she displays a symptom, then P(B | A) describes a causal relationship, and P(A | B) describes a diagnostic one (that is usually hard to assess).

If P(B | A), P(A) and P(B) can be assessed easily, then we get P(A | B) for free.

23

Example

• S:stiff neck, M: meningitis

• P(S|M) =0.5, P(M) = 1/50,000 P(S)=1/20

• I have stiff neck, should I worry?

0002.020/1

000,50/15.0

)(

)()|()|(

SP

MPMSPSMP

24

(Conditional) independence

• Two events A e B are independent of each other if

P(A) = P(A|B)

• Two events A and B are conditionally independent of each other given C if

P(A|C) = P(A|B,C)

25

Back to language

• Statistical NLP aims to do statistical inference for the field of NLP– Topic classification

• P( topic | document ) – Language models

• P (word | previous word(s) )– WSD

• P( sense | word)

• Two main problems– Estimation: P in unknown: estimate P – Inference: We estimated P; now we want to find

(infer) the topic of a document, o the sense of a word

26

Language Models (Estimation)

• In general, for language events, P is unknown

• We need to estimate P, (or model M of the language)

• We’ll do this by looking at evidence about what P must be based on a sample of data

27

Estimation of P

• Frequentist statistics– Parametric– Non-parametric (distribution free)

• Bayesian statistics– Bayesian statistics measures degrees of belief– Degrees are calculated by starting with prior beliefs

and updating them in face of the evidence, using Bayes theorem

• 2 different approaches, 2 different philosophies

28

Inference

• The central problem of computational Probability Theory is the inference problem:

• Given a set of random variables X1, … , Xk and their joint density P(X1, … , Xk), compute one or more conditional densities given observations.– Compute

• P(X1 | X2 … , Xk)• P(X3 | X1 )• P(X1 , X2 | X3, X4,) • Etc …

• Many problems can be formulated in these terms.

29

Bayes decision rule

• w: ambiguous word• S = {s1, s2, …, sn } senses for w • C = {c1, c2, …, cn } context of w in a corpus• V = {v1, v2, …, vj } words used as contextual features for

disambiguation

• Bayes decision rule– Decide sj if P(sj | c) > P(sk | c) for sj ≠ sk

• We want to assign w to the sense s’ where

s’ = argmaxsk P(sk | c)

30

Bayes classification for WSD

• We want to assign w to the sense s’ where

s’ = argmaxsk P(sk | c)

• We usually do not know P(sk | c) but we can compute it using Bayes rule

)P(sP(c)

)P(c|s

P(c)

)P(c,s|c)P(s k

kk

k

))P(sP(c|s)P(sP(c)

)P(c|s|c)P(ss kkk

k

k sksksk maxargmaxargmaxarg'

31

Naïve Bayes classifier

))P(sP(c|ss kk

ksmaxarg'

• Naïve Bayes classifier widely used in machine learning

• Estimate P(c | sk) and P(sk)

32

Naïve Bayes classifier))P(sP(c|s

ss kk

kmaxarg'

cvj

jjj )sP(v)sc|vvP()P(c|s kkk

in

||}in {

• Estimate P(c | sk) and P(sk)• w: ambiguous word

• S = {s1, s2, …, sn } senses for w

• C = {c1, c2, …, cn } context of w in a corpus

• V = {v1, v2, …, vj } words used as contextual features for disambiguation

• Naïve Bayes assumption:

33

Naïve Bayes classifier

cvj

jjj )sP(v)sc|vvP()P(c|s kkk

in

||}in {

• Naïve Bayes assumption:

– Two consequences

– All the structure and linear ordering of words within the context is ignored bags of words model

– The presence of one word in the model is independent of the others• Not true but model “easier” and very “efficient”• “easier” “efficient” mean something specific in the probabilistic framework

– We’ll see later (but easier to estimate parameters and more efficient inference)

– Naïve Bayes assumption is inappropriate if there are strong dependencies, but often it does very well (partly because the decision may be optimal even if the assumption is not correct)

34

Naïve Bayes for WSD

)P(s)sP(vs kk

k cvj

js

in

|maxarg'

))P(sP(c|ss kk

ksmaxarg' Bayes decision rule

cvj

jjj )sP(v)sc|vvP()P(c|s kkk

in

||}in { Naïve Bayes assumption

)sC

)svC)sP(v

k

k

k

jj

(

(|

, Count of vj when sk

w)C

)sC)P(s

k

k

(

( Prior probability of sk

Estimation

35

Naïve Bayes Algorithm for WSD

• TRAINING (aka Estimation)• For all of senses sk of w do

– For all words vj in the vocabulary calculate

– end• end

• For all of senses sk of w do

• end

)sC

)svC)sP(v

k

k

k

jj

(

(|

,

w)C

)sC)P(s

k

k

(

(

36

Naïve Bayes Algorithm for WSD

• TESTING (aka Inference or Disambiguation)• For all of senses sk of w do

– For all words vj in the context window c calculate

– end• end

• Choose s= sk of w do

))P(sP(c|s|c)P(s)sscore kkkk (

)sscores k

sk(maxarg'

An information-theoretic approach

Information theoretic approach

• Look for key words (informant) that disambiguates the sense of the word.

Flip-Flop Algorithm

t translations for the ambiguous wordx possible values for indicators

The algorithm works by searching for a partition of senses that maximizes the mutual information. The algorithm stops when the increase becomes insignificant.

Stepping through flip-flop algorithm for the French word: prendre

Disambiguation process

• Once the partition set for P& Q (indicator words determined), then disambiguation is simple:

1. For every occurrence of the ambiguous word, determine the value of xi – the indicator word.

2. If xi is in Q1, assign it to sense 1; if not assign it to sense 2.

Decision Lists

Decision Lists and Trees

• Very widely used in Machine Learning. • Decision trees used very early for WSD research (e.g.,

Kelly and Stone, 1975; Black, 1988). • Represent disambiguation problem as a series of

questions (presence of feature) that reveal the sense of a word.– List decides between two senses after one positive answer– Tree allows for decision among multiple senses after a

series of answers• Uses a smaller, more refined set of features than “bag

of words” and Naïve Bayes.– More descriptive and easier to interpret.

Decision List for WSD (Yarowsky, 1994)

• Identify collocational features from sense tagged data. • Word immediately to the left or right of target :

– I have my bank/1 statement.– The river bank/2 is muddy.

• Pair of words to immediate left or right of target :– The world’s richest bank/1 is here in New York.– The river bank/2 is muddy.

• Words found within k positions to left or right of target, where k is often 10-50 :– My credit is just horrible because my bank/1 has made

several mistakes with my account and the balance is very low.

Building the Decision List

•Sort order of collocation tests using log of conditional probabilities. •Words most indicative of one sense (and not the other) will be ranked highly.

))|2(

)|1((log

inCollocatioiFSpinCollocatioiFSp

Abs

Computing DL score– Given 2,000 instances of “bank”, 1,500 for bank/1 (financial

sense) and 500 for bank/2 (river sense)• P(S=1) = 1,500/2,000 = .75• P(S=2) = 500/2,000 = .25

– Given “credit” occurs 200 times with bank/1 and 4 times with bank/2.

• P(F1=“credit”) = 204/2,000 = .102• P(F1=“credit”|S=1) = 200/1,500 = .133• P(F1=“credit”|S=2) = 4/500 = .008

– From Bayes Rule… • P(S=1|F1=“credit”) = .133*.75/.102 = .978• P(S=2|F1=“credit”) = .008*.25/.102 = .020

– DL Score = abs (log (.978/.020)) = 3.89

Using the Decision List

• Sort DL-score, go through test instance looking for matching feature. First match reveals sense…

DL-score Feature Sense

3.89 credit within bank Bank/1 financial

2.20 bank is muddy Bank/2 river

1.09 pole within bank Bank/2 river

0.00 of the bank N/A

Using the Decision List

CREDIT?

BANK/1 FINANCIAL IS MUDDY?

POLE? BANK/2 RIVER

BANK/2 RIVER

Support Vector Machine(SVM)

50

Linear classifiers: Which Hyperplane?

• Lots of possible solutions for a, b, c.• Some methods find a separating

hyperplane, but not the optimal one [according to some criterion of expected goodness]

– E.g., perceptron

• Support Vector Machine (SVM) finds an optimal* solution.– Maximizes the distance between the

hyperplane and the “difficult points” close to decision boundary

– One intuition: if there are no points near the decision surface, then there are no very uncertain classification decisions

This line represents the

decision boundary:

ax + by − c = 0

Ch. 15

51

Another intuition

• If you have to place a fat separator between classes, you have less choices, and so the capacity of the model has been decreased

Sec. 15.1

52

Support Vector Machine (SVM)Support vectors

Maximizesmargin

• SVMs maximize the margin around the separating hyperplane.

• A.k.a. large margin classifiers

• The decision function is fully specified by a subset of training samples, the support vectors.

• Solving SVMs is a quadratic programming problem

• Seen by many as the most successful current text classification method* *but other discriminative

methods often perform very similarly

Sec. 15.1

Narrowermargin

From Text to Feature Vectors

• My/pronoun grandfather/noun used/verb to/prep fish/verb along/adv the/det banks/SHORE of/prep the/det Mississippi/noun River/noun. (S1)

• The/det bank/FINANCE issued/verb a/det check/noun for/prep the/det amount/noun of/prep interest/noun. (S2)

P-2 P-1 P+1 P+2

fish check

river

interest

SENSE TAG

S1 adv det prep

det Y N Y N SHORE

S2 det verb

det N Y N Y FINANCE

K-NN

+

+

+

+

+

++ +

o

o

o o

o

o

oo

o

o

o

oo

o

o

o

o

o?

Supervised Approaches – Comparisons

55

Approach Average Precision

Average Recall Corpus Average Baseline Accuracy

Naïve Bayes 64.13% Not reported Senseval3 – All Words Task

60.90%

Decision Lists 96% Not applicable Tested on a set of 12 highly polysemous English words

63.9%

Exemplar Based disambiguation (k-NN)

68.6% Not reported WSJ6 containing 191 content words

63.7%

SVM 72.4% 72.4% Senseval 3 – Lexical sample task (Used for disambiguation of 57 words)

55.2%

Perceptron trained HMM

67.60 73.74% Senseval3 – All Words Task

60.90%

Dictionary-Based Discrimination

Overview

• In this section use of Dictionaries and Thesauri for the purposes of word sense disambiguation is explored.

• Lesk (1986) explores use of dictionary• Yarowsky (1992) explores use of Roget’s thesaurus.• Dagan & Itai(1994) explore the use of a bilingual

dictionary.• Also, a careful examination of the distribution properties

of words may provide additional cues. Commonly ambiguous word may not appear with more than one meaning in any given text.

Disambiguation based on sense definitions

Thesaurus-based disambiguation

• Algorithm proposed by Walker (1987)1. Comment: Given: context c

2. for all senses sk of w do

3. score(sk) = 4. end

5. choose s’ s.t. s’ = argmaxsk score(sk)

Where t(sk) is the subject code of sense sk, and

= 1; iff t(sk) is one of the subject codes of vj and 0 otherwise.

cvj jk vst )),((

)),(( jk vst

Yarowski’s adaptation

• Context is simply a 100-word window centered around the word to be disambiguated

• Algorithm adds words to the thesaurus category if it happens more often than chance in the context of that category. For instance “Navratilova” occurs more often than not only in a “sports” context if you are analyzing news articles.

• One can look at this as key markers in the context to guide the disambiguation process.

Thesaurus-based disambiguation

Disambiguation based on translations in a second-language corpus

• The insight to this methodology is that words that may have multiple senses in English tend to manifest themselves as different words in other languages. And if you have a body of translations available that you can draw upon, you can use it for disambiguation purposes.

Using Second language corpus

One sense per discourse, one sense per collocation

• One sense per discourse – the sense of a target word is highly consistent within any given document

• One sense per collocation. Nearby words provide strong and consistent clues to the sense of the word.

Example of one sense per discourse

Yarowski’s Algorithm

Unsupervised disambiguation

Sense Tagging and Sense Discrimination

• Sense tagging – ability to tag occurrences of a word in one sense or other

• Sense discrimination – ability to recognize that the sense of a word is different without worrying about the actual sense

• Example– K-means– EM-algorithm

70

K-means Demo1. User set up the number of

clusters they’d like. (e.g. k=5)

71

K-means Demo 1. User set up the number of

clusters they’d like. (e.g. K=5)

2. Randomly guess K cluster Center locations

72

K-means Demo 1. User set up the number of

clusters they’d like. (e.g. K=5)

2. Randomly guess K cluster Center locations

3. Each data point finds out which Center it’s closest to. (Thus each Center “owns” a set of data points)

73

K-means Demo 1. User set up the number of

clusters they’d like. (e.g. K=5)

2. Randomly guess K cluster centre locations

3. Each data point finds out which centre it’s closest to. (Thus each Center “owns” a set of data points)

4. Each centre finds the centroid of the points it owns

74

K-means Demo 1. User set up the number of

clusters they’d like. (e.g. K=5)

2. Randomly guess K cluster centre locations

3. Each data point finds out which centre it’s closest to. (Thus each centre “owns” a set of data points)

4. Each centre finds the centroid of the points it owns

5. …and jumps there

75

K-means Demo 1. User set up the number of

clusters they’d like. (e.g. K=5)

2. Randomly guess K cluster centre locations

3. Each data point finds out which centre it’s closest to. (Thus each centre “owns” a set of data points)

4. Each centre finds the centroid of the points it owns

5. …and jumps there

6. …Repeat until terminated!

Disambiguation using clustering

cv

kjks

jk

svPsPs )]|(log)([logmaxarg'

Decide s’ where,

Unsupervised Clustering