Information Retrieval and Web SearchIR models: Vector Space Model

Instructor: Rada Mihalcea

[Note: Some slides in this set were adapted from an IR course taught by Ray Mooney at UT Austin (who in turn adapted them from Joydeep Ghosh), and from an IR course taught by Chris Manning at Stanford)

Ranked Retrieval• Thus far, our queries have all been Boolean

– Documents either match or don’t• Good for expert users with precise understanding

of their needs and the collection– Also good for applications: Applications can easily

consume 1000s of results• Not good for the majority of users

– Most users incapable of writing Boolean queries (or they are, but they think it’s too much work)

– Most users don’t want to wade through 1000s of results• This is particularly true of Web search

Ch. 6

Problem with Boolean Search• Boolean queries often result in either too few

(=0) or too many (1000s) results.• Query 1: “standard user dlink 650” → 200,000

hits• Query 2: “standard user dlink 650 no card

found”: 0 hits• It takes a lot of skill to come up with a query that

produces a manageable number of hits.– AND gives too few; OR gives too many

Ch. 6

Ranked Retrieval Models• Rather than a set of documents satisfying a query

expression, in ranked retrieval, the system returns an ordering over the (top) documents in the collection for a query

• Free text queries: Rather than a query language of operators and expressions, the user’s query is just one or more words in a human language

• In principle, there are two separate choices here, but in practice, ranked retrieval has normally been associated with free text queries and vice versa

4

Not a Problem in Ranked Retrieval• When a system produces a ranked result set,

large result sets are not an issue– Indeed, the size of the result set is not an issue– We just show the top k ( ≈ 10) results– We don’t overwhelm the user

– Premise: the ranking algorithm works

Ch. 6

Scoring as the Basis of Ranked Retrieval• We wish to return in order the documents most

likely to be useful to the searcher• How can we rank-order the documents in the

collection with respect to a query?• Assign a score – say in [0, 1] – to each document• This score measures how well document and

query “match”.

Ch. 6

Query-document Matching Scores• We need a way of assigning a score to a

query/document pair• Let’s start with a one-term query• If the query term does not occur in the document:

score should be 0• The more frequent the query term in the

document, the higher the score (should be)

Take 1: Jaccard coefficient• Jaccard: A commonly used measure of overlap of

two sets A and B• Jaccard(A,B) = |A ∩ B| / |A ∪ B|• Jaccard(A,A) = 1• Jaccard(A,B) = 0 if A ∩ B = 0• A and B don’t have to be the same size.• Always assigns a number between 0 and 1.

Ch. 6

Exercise: Jaccard Coefficient• What is the query-document match score that the

Jaccard coefficient computes for each of the two documents below?

• Query: march of dimes• Document 1: caesar died in march• Document 2: the long march

Ch. 6

Issues with Jaccard for Scoring• It does not consider term frequency (how many

times a term occurs in a document)• Rare terms in a collection are more informative

than frequent terms. Jaccard does not consider this information

• We need a more sophisticated way of normalizing for length

Ch. 6

Recall (from Boolean Retrieval): Binary Term-Document Incidence Matrix

Each document is represented by a binary vector ∈ {0,1}|V|

Sec. 6.2

Term-document Count Matrices• Consider the number of occurrences of a term in

a document: – Each document is a count vector: a column below

Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth

Antony 157 73 0 0 0 0Brutus 4 157 0 1 0 0Caesar 232 227 0 2 1 1

Calpurnia 0 10 0 0 0 0Cleopatra 57 0 0 0 0 0

mercy 2 0 3 5 5 1

worser 2 0 1 1 1 0

Sec. 6.2

Vector-Space Model• t distinct terms remain after preprocessing

– Unique terms that form the VOCABULARY• These “orthogonal” terms form a vector space.

Dimension = t = |vocabulary| – 2 terms bi-dimensional; …; t-terms t-dimensional

• Each term, i, in a document or query j, is given a real-valued weight, wij.

• Both documents and queries are expressed as t-dimensional vectors: dj = (w1j, w2j, …, wtj)

Vector-Space Model

Query as vector:• We regard query as short document• We return the documents ranked by the

closeness of their vectors to the query, also represented as a vector.

• Vector-space model was developed in the SMART system (Salton, c. 1970) and standardly used by TREC participants and web IR systems

Graphic Representation

Example:D1 = 2T1 + 3T2 + 5T3

D2 = 3T1 + 7T2 + T3

Q = 0T1 + 0T2 + 2T3

T3

T1

T2

D1 = 2T1+ 3T2 + 5T3

D2 = 3T1 + 7T2 + T3

Q = 0T1 + 0T2 + 2T3

7

32

5

• Is D1 or D2 more similar to Q?

• How to measure the degree of similarity? Distance? Angle? Projection?

Document Collection Representation• A collection of n documents can be represented in the vector space model by a term-document matrix.

• An entry in the matrix corresponds to the “weight” of a term in the document; zero means the term has no significance in the document or it simply doesn’t exist in the document. T1 T2 ….

Tt

D1 w11 w21 … wt1

D2 w12 w22 … wt2

: : : : : : : :Dn w1n w2n … wtn

Term Frequency tf• The term frequency tfij of term i in document j is

defined as the number of times that term i occurs in document j.

• More frequent terms in a document are more important, i.e. more indicative of the topic.

• May want to normalize term frequency (tf) : tfij = freqij / max{fij}

• We want to use tf when computing query-document match scores. But how?

Term Frequency tf

• Raw term frequency is not what we want:– A document with 10 occurrences of the term is more

relevant than a document with 1 occurrence of the term.

– But not 10 times more relevant.• Relevance does not increase proportionally with

term frequency.

Document Frequency

• Rare terms are more informative than frequent terms– Recall stop words

• Consider a term in the query that is rare in the collection (e.g., arachnocentric)

• A document containing this term is very likely to be relevant to the query arachnocentric

• → We want a high weight for rare terms like arachnocentric.

Sec. 6.2.1

Document Frequency (continued)• Frequent terms are less informative than rare

terms• Consider a query term that is frequent in the

collection (e.g., high, increase, line)• A document containing such a term is more likely

to be relevant than a document that doesn’t• But it’s not a sure indicator of relevance.• → For frequent terms, we want high positive

weights for words like high, increase, and line• But lower weights than for rare terms.• We will use document frequency (df) to capture

this.

Sec. 6.2.1

Idf Weight• dfi is the document frequency of term i: the

number of documents that contain term i– dfi is an inverse measure of the informativeness of term i– dfi n, where n is the number of documents in the

collection• We define the idf (inverse document frequency)

of term i by

– We use log (N/dfi) instead of N/dfi to “dampen” the effect of a high df (as compared to tf)

Sec. 6.2.1

idf example, suppose N = 1 million

term dft idftcalpurnia 1

animal 100

sunday 1,000

fly 10,000

under 100,000

the 1,000,000

There is one idf value for each term i in a collection.

Sec. 6.2.1

Collection vs. Document Frequency

• The collection frequency of i is the number of occurrences of i in the collection, counting multiple occurrences.

• Example:

• Which word is a better search term (and should get a higher weight)?

Word Collection frequency Document frequency

insurance 10440 3997

try 10422 8760

Sec. 6.2.1

tf-idf Weighting

• The tf-idf weight of a term is the product of its tf weight and its idf weight.

• Best known weighting scheme in information retrieval– Theoretically proven to work well (Papineni, NAACL 2001)– Note: the “-” in tf-idf is a hyphen, not a minus sign!– Alternative names: tf.idf, tf x idf

• Increases with the number of occurrences within a document

• Increases with the rarity of the term in the collection

Sec. 6.2.2

Computing tf-idf: An ExampleGiven a document containing terms with given frequencies: A(3), B(2), C(1)Assume collection contains 10,000 documents and document frequencies of these terms are: A(50), B(1300), C(250)Then:A: tf = 3/3; idf = log(10000/50) = 5.3; tf-idf = 5.3B: tf = 2/3; idf = log(10000/1300) = 2.0; tf-idf = 1.3C: tf = 1/3; idf = log(10000/250) = 3.7; tf-idf = 1.2

Binary → Count → Weight matrix

Each document is now represented by a real-valued vector of tf-idf weights ∈ R|V|

Sec. 6.3

Documents as Vectors• So we have a |V|-dimensional vector space• Terms are axes of the space• Documents are points or vectors in this space• Very high-dimensional: tens of millions of

dimensions when you apply this to a web search engine

• These are very sparse vectors - most entries are zero.

Sec. 6.3

Query as Vector• Query vector is typically treated as a document

and also tf-idf weighted.• Alternative is for the user to supply weights for

the given query terms.

Similarity Measure• We now have vectors for all documents in the

collection, a vector for the query, how to compute similarity?

• A similarity measure is a function that computes the degree of similarity between two vectors.

• Using a similarity measure between the query and each document:– It is possible to rank the retrieved documents in the

order of presumed relevance.– It is possible to enforce a certain threshold so that the

size of the retrieved set can be controlled.

First Cut: Euclidean Distance

• Distance between vectors d1 and d2 is the length of the vector |d1 – d2|.– Euclidean distance

• Exercise: Determine the Euclidean distance between the vectors (0, 3, 2, 1, 10) and (2, 7, 1, 0, 0)

• Why is this not a great idea?• We still haven’t dealt with the issue of length

normalization– Long documents would be more similar to each other by

virtue of length, not topic

Second Cut: Manhattan Distance

• Or “city block” measure– Based on the idea that generally in American cities you

cannot follow a direct line between two points.

• Uses the formula:

• Exercise: Determine the Manhattan distance between the vectors (0, 3, 2, 1, 10) and (2, 7, 1, 0, 0)

x

y

n

iii yxYXManhDist

1

||),(

Third Cut: Inner Product• Similarity between vectors for the documents

d1 and document d2 can be computed as the vector inner product:

where wij is the weight of term i in document j• For binary vectors, the inner product is the

number of matched query terms in the document (size of intersection).

• For weighted term vectors, it is the sum of the products of the weights of the matched terms.

Properties of Inner Product• Favors long documents with a large number of

unique terms.– Again, the issue of normalization

• Measures how many terms matched but not how many terms are not matched.

Exercise

Cosine Similarity• Distance between vectors d1 and d2 captured by

the cosine of the angle x between them.• Note – this is similarity, not distance

t 1

d2

d1

t 3

t 2

θ

Cosine Similarity

• wij – weight of term i in document j• Cosine of angle between two vectors• The denominator involves the lengths of the

vectors• So the cosine measure is also known as the

normalized inner product

Exercise• Documents: Sense and Sensibility, Pride and

Prejudice; Wuthering Heights • Vocabulary of four terms

• Measure cos(SaS,PaP), cos(SaS,WH)

term SaS PaP WHaffection 115 58 20jealous 10 7 11gossip 2 0 6wuthering 0 0 38

term SaS PaP WHaffection 0.789 0.832 0.524

jealous 0.515 0.555 0.465

gossip 0.335 0 0.405

wuthering 0 0 0.588

Exercise• Rank the following by decreasing cosine

similarity:– Two documents that have only frequent words (the, a,

an, of) in common.– Two documents that have no words in common.– Two documents that have many rare words in common

(wingspan, tailfin).

Cosine Similarity vs. Inner Product• Cosine similarity measures the cosine of

the angle between two vectors.• Inner product normalized by the vector

lengths.

D1 = 2T1 + 3T2 + 5T3 CosSim(D1 , Q) = 10 / (4+9+25)(0+0+4) = 0.81D2 = 3T1 + 7T2 + 1T3 CosSim(D2 , Q) = 2 / (9+49+1)(0+0+4) = 0.13

Q = 0T1 + 0T2 + 2T3

2

t3

t1

t2

D1

D2

Q

1

D1 is 6 times better than D2 using cosine similarity but only 5 times better using inner product.

t

i

t

i

t

i

ww

ww

qdqd

iqij

iqij

j

j

1 1

22

1

)(

CosSim(dj, q) =

qdj

InnerProduct(dj, q) =

Comments on Vector Space Models

• Simple, mathematically based approach. • Considers both local (tf) and global (idf) word

occurrence frequencies.• Provides partial matching and ranked results.• Tends to work quite well in practice despite

obvious weaknesses.• Allows efficient implementation for large

document collections.

Problems with Vector Space Model

• Missing semantic information (e.g. word sense).• Missing syntactic information (e.g. phrase

structure, word order, proximity information).• Assumption of term independence• Lacks the control of a Boolean model (e.g.,

requiring a term to appear in a document).– Given a two-term query “A B”, may prefer a document

containing A frequently but not B, over a document that contains both A and B, but both less frequently.