Information Retrieval Models Part I

IR ModelsPart I — Foundations

Thomas Roelleke, Queen Mary University of LondonIngo Frommholz, University of Bedfordshire

Autumn School for Information Retrieval and ForagingSchloss Dagstuhl, September 2014

ASIRF Sponsors:

IR Models

Acknowledgements

The knowledge presented in this tutorial and the Morgan & Claypool book isthe result of many many discussions with colleagues.

People involved in the production and reviewing: Gianna Amati and DjoerdHiemstra (the experts), Diane Cerra and Gary Marchionini (Morgan &Claypool), Ricardo Baeza-Yates, Norbert Fuhr, and Mounia Lalmas.

Thomas’ PhD students (who had no choice): Jun Wang, Hengzhi Wu, FredForst, Hany Azzam, Sirvan Yahyaei, Marco Bonzanini, Miguel Martinez-Alvarez.

Many more IR experts including Fabio Crestani, Keith van Rijsbergen, StephenRobertson, Fabrizio Sebastiani, Arjen deVries, Tassos Tombros, Hugo Zaragoza,ChengXiang Zhai.

And non-IR experts Fabrizio Smeraldi, Andreas Kaltenbrunner and Norman

Fenton.

2 / 133

IR Models

Table of Contents

1 Introduction

2 Foundations of IR Models

3 / 133

Introduction

Warming UpBackground: Time-Line of IR ModelsNotation

Introduction

Warming Up

IR Models

Introduction

Warming Up

Information Retrieval Conceptual Model

DD

rel.

judg.

α

αQ βQ

Dβ

ρIR

DQ

DD

Q

D

Q

R

[Fuhr, 1992]

6 / 133

IR Models

Introduction

Warming Up

Vector Space Model, Term Space

Still one of the prominent IR frameworks is the Vector SpaceModel (VSM)

A term space is a vector space where each dimensionrepresents one term in our vocabulary

If we have n terms in our collection, we get an n-dimensionalterm or vector space

Each document and each query is represented by a vector inthe term space

7 / 133

IR Models

Introduction

Warming Up

Formal Description

Set of terms in our vocabulary: T = {t1, . . . , tn}T spans an n-dimensional vector space

Document dj is represented by a vector of document termweights

Query q is represented by a vector of query term weights

8 / 133

IR Models

Introduction

Warming Up

Document Vector

Document dj is represented by a vector of document termweights dji ∈ R:

Document term weights can be computed, e.g., using tf andidf (see below)

9 / 133

IR Models

Introduction

Warming Up

Document Vector

Document dj is represented by a vector of document termweights dji ∈ R:

Weight of term

in document

Document term weights can be computed, e.g., using tf andidf (see below)

10 / 133

IR Models

Introduction

Warming Up

Query Vector

Like documents, a query q is represented by a vector of queryterm weights qi ∈ R:

~q =

q1

q2

. . .qn

qi denotes the query term weight of term tiqi is 0 if the term does not appear in the query.

qi may be set to 1 if the term does appear in the query.Further query term weights are possible, for example

2 if the term is important1 if the term is just “nice to have”

11 / 133

IR Models

Introduction

Warming Up

Retrieval Function

The retrieval function computes a retrieval status value (RSV)using a vector similarity measure, e.g. the scalar product:

RSV (dj , q) = ~dj × ~q =n∑

i=1

dji × qi

t

t

1

2

q

d

d

1

2

Ranking of documents according to decreasing RSV

12 / 133

IR Models

Introduction

Warming Up

Example Query

Query: “ side effects of drugs on memory and cognitive abilities”

ti Query ~q ~d1~d2

~d3~d4

side effect 2 1 0.5 1 1drug 2 1 1 1 1memory 1 1 0 1 0cognitive ability 1 0 1 1 0.5

RSV 5 4 6 4.5

Produces the ranking d3 > d1 > d4 > d2

13 / 133

IR Models

Introduction

Warming Up

Term weights: Example Text

In his address to the CBI, Mr Cameron is expected to say:“Scotland does twice as much trade with the rest of the UK thanwith the rest of the world put together – trade that helps to supportone million Scottish jobs.” Meanwhile, Mr Salmond has set out sixjob-creating powers for Scotland that he said were guaranteed witha “Yes” vote in the referendum. During their televised BBC debateon Monday, Mr Salmond had challenged Better Together headAlistair Darling to name three job-creating powers that were beingoffered to the Scottish Parliament by the pro-UK parties in theevent of a “No” vote.

Source: http://www.bbc.co.uk/news/uk-scotland-scotland-politics-28952197

What are good descriptors for the text? Which are more, which areless important? Which are “informative”? Which are gooddiscriminators?How can a machine answer these questions?

14 / 133

IR Models

Introduction

Warming Up

Frequencies

The answer is countingDifferent assumptions:

The more frequent a term appears in a document, the moresuitable it is to describe its content

Location-based count. Think of term positions or locations.In how many locations of a text do we observe the term?The term ‘scotland’ appears in 2 out of 138 locations in theexample text

The less documents a term occurs in, the more discriminativeor informative it is

Document-based count. In how many documents do weobserve the term?Think of stop-words like ‘the’, ‘a’ etc.

Location- and document-based frequencies are the buildingblocks of all (probabilistic) models to come

15 / 133

Introduction

Background: Time-Line of IR Models

IR Models

Introduction


Timeline of IR Models: 50s, 60s and 70s

Zipf and Luhn: distribution of document frequencies;[Croft and Harper, 1979]: BIR without relevance;[Robertson and Sparck-Jones, 1976]: BIR;[Salton, 1971, Salton et al., 1975]: VSM, TF-IDF;[Rocchio, 1971]: Relevance feedback; [Maron and Kuhns, 1960]:On Relevance, Probabilistic Indexing, and IR

17 / 133

IR Models

Introduction


Timeline of IR Models: 80s

[Cooper, 1988, Cooper, 1991, Cooper, 1994]: Beyond Boole,Probability Theory in IR: An Encumbrance;[Dumais et al., 1988, Deerwester et al., 1990]: Latent semanticindexing; [van Rijsbergen, 1986, van Rijsbergen, 1989]: P(d → q);[Bookstein, 1980, Salton et al., 1983]: Fuzzy, extended Boolean

18 / 133

IR Models

Introduction



[Ponte and Croft, 1998]: LM;[Brin and Page, 1998, Kleinberg, 1999]: Pagerank and Hits;[Robertson et al., 1994, Singhal et al., 1996]: Pivoted DocumentLength Normalisation; [Wong and Yao, 1995]: P(d → q);[Robertson and Walker, 1994, Robertson et al., 1995]: 2-Poisson,BM25; [Margulis, 1992, Church and Gale, 1995]: Poisson;[Fuhr, 1992]: Probabilistic Models in IR;[Turtle and Croft, 1990, Turtle and Croft, 1991]: PIN’s;[Fuhr, 1989]: Models for Probabilistic Indexing

19 / 133

IR Models

Introduction



ICTIR 2009 and ICTIR 2011; [Roelleke and Wang, 2008]: TF-IDFUncovered; [Luk, 2008, Robertson, 2005]: Event Spaces;[Roelleke and Wang, 2006]: Parallel Derivation of Models;[Fang and Zhai, 2005]: Axiomatic approach; [He and Ounis, 2005]:TF in BM25 and DFR; [Metzler and Croft, 2004]: LM andPIN’s;[Robertson, 2004]: Understanding IDF;[Sparck-Jones et al., 2003]: LM and Relevance;[Croft and Lafferty, 2003, Lafferty and Zhai, 2003]: LM book;[Zaragoza et al., 2003]: Bayesian extension to LM;[Bruza and Song, 2003]: probabilistic dependencies in LM;[Amati and van Rijsbergen, 2002]: DFR;[Lavrenko and Croft, 2001]: Relevance-based LM;[Hiemstra, 2000]: TF-IDF and LM; [Sparck-Jones et al., 2000]:probabilistic model: status

20 / 133

IR Models

Introduction


Timeline of IR Models: 2010 and Beyond

Models for interactive and dynamic IR (e.g.iPRP [Fuhr, 2008])

Quantum models[van Rijsbergen, 2004, Piwowarski et al., 2010]

21 / 133

Introduction

Notation

IR Models

Introduction

Notation

Notation

A tedious start ... but a must-have.

Sets

Locations

Documents

Terms

Probabilities

23 / 133

IR Models

Introduction

Notation

Notation: Sets

Notation description of events, sets, and frequencies

t, d , q, c, r term t, document d , query q, collection c, rele-vant r

Dc , Dr Dc = {d1, . . .}: set of Documents in collection c;Dr : relevant documents

Tc , Tr Tc = {t1, . . .}: set of Terms in collection c; Tr :terms that occur in relevant documents

Lc , Lr Lc = {l1, . . .}; set of Locations in collection c; Lr :locations in relevant documents

24 / 133

IR Models

Introduction

Notation

Notation: Locations

Notation description of events, sets, andfrequencies

Traditional notation

nL(t, d) number of Locations at whichterm t occurs in document d

tf, tfd

NL(d) number of Locations in docu-ment d (document length)

dl

nL(t, q) number of Locations at whichterm t occurs in query q

qtf, tfq

NL(q) number of Locations in query q(query length)

ql

25 / 133

IR Models

Introduction

Notation

Notation: Locations



nL(t, c) number of Locations at whichterm t occurs in collection c

TF, cf(t)

NL(c) number of Locations in collec-tion c

nL(t, r) number of Locations at whichterm t occurs in the set Lr

NL(r) number of Locations in the setLr

26 / 133

IR Models

Introduction

Notation

Notation: Documents



nD(t, c) number of Documents inwhich term t occurs in the setDc of collection c

nt , df(t)

ND(c) number of Documents in theset Dc of collection c

N

nD(t, r) number of Documents inwhich term t occurs in the setDr of relevant documents

rt

ND(r) number of Documents in theset Dr of relevant documents

R

27 / 133

IR Models

Introduction

Notation

Notation: Terms



nT (d , c) number of Terms in docu-ment d in collection c

NT (c) number of Terms in collec-tion c

28 / 133

IR Models

Introduction

Notation

Notation: Average and Pivoted Length

Let u denote a collection associated with a set of documents. Forexample: u = c, or u = r , or u = r .

Notation description of events, sets, and frequen-cies


avgdl(u) average document length: avgdl(u) =NL(u)/ND(u) (avgdl if collection im-plicit)

avgdl

pivdl(d , u) pivoted document length: pivdl(d , u) =NL(d)/avgdl(u) = dl/avgdl(u)(pivdl(d) if collection implicit)

pivdl

λ(t, u) average term frequency over all docu-ments in Du: nL(t, u)/ND(u)

avgtf(t, u) average term frequency over elite docu-ments in Du: nL(t, u)/nD(t, u)

29 / 133

IR Models

Introduction

Notation

Notation: Location-based Probabilities

Notation Description of Probabili-ties


PL(t|d) := nL(t,d)NL(d)

Location-based within-document term probabil-ity

P(t|d) = tfd|d| , |d | = dl = NL(d)

PL(t|q) := nL(t,q)NL(q)

Location-based within-query term probability

P(t|q) =tfq|q| , |q| = ql = NL(q)

PL(t|c) := nL(t,c)NL(c)

Location-based within-collection term probabil-ity

P(t|c) = tfc|c| , |c| = NL(c)

PL(t|r) := nL(t,r)NL(r)

Location-based within-relevance term probabil-ity

Event space PL: Locations (LM, TF)

30 / 133

IR Models

Introduction

Notation

Notation: Document-based Probabilities

Notation Description of Probabilities Traditional notation

PD(t|c) := nD (t,c)ND (c)

Document-based within-collection term probability

P(t) = ntN

, N = ND(c)

PD(t|r) := nD (t,r)ND (r)

Document-based within-relevance term probability

P(t|r) = rtR

, R = ND(r)

PT (d |c) := nT (d,c)NT (c)

Term-based document proba-bility

Pavg(t|c) := avgtf(t,c)avgdl(c)

probability that t occursin document with averagelength; avgtf(t, c) ≤ avgdl(c)

Event space PD: Documents (BIR, IDF)

31 / 133

IR Models

Introduction

Notation

Toy Example

Notation ValueNL(c) 20ND(c) 10avgdl(c) 20/10=2

Notation Valuedoc1 doc2 doc3

NL(d) 2 3 3pivdl(d , c) 2/2 3/2 3/2

Notation Valuesailing boats

nL(t, c) 8 6nD(t, c) 6 5PL(t|c) 8/20 6/20PD(t|c) 6/10 5/10λ(t, c) 8/10 6/10avgtf(t, c) 8/6 6/5

32 / 133

Foundations of IR Models

TF-IDFPRF: The Probability of Relevance FrameworkBIR: Binary Independence RetrievalPoisson and 2-PoissonBM25LM: Language ModellingPIN’s: Probabilistic Inference NetworksRelevance-based ModelsFoundations: Summary


TF-IDF

IR Models


TF-IDF

TF-IDF

Still a very popular model

Best known outside IR research, ery intuitive

“TF-IDF is not a model; it is just a weighting scheme in thevector space model”

“TF-IDF is purely heuristic; it has no probabilistic roots.”

But:

TF-IDF and LM are dual models that can be shown to bederived from the same root.Simplified version of BM25

35 / 133

IR Models


TF-IDF

TF Variants: TF(t, d)

TFtotal(t, d) := lftotal(t, d) := nL(t, d) (= tfd)

TFsum(t, d) := lfsum(t, d) :=nL(t, d)

NL(d)= PL(t|d)

(=

tfddl

)TFmax(t, d) := lfmax(t, d) :=

nL(t, d)

nL(tmax, d)

TFlog(t, d) := lflog(t, d) := log(1 + nL(t, d)) (= log(1 + tfd))

TFfrac,K (t, d) := lffrac,K (t, d) :=nL(t, d)

nL(t, d) + Kd

(=

tfdtfd + Kd

)TFBM25,k1,b(t, d) := ... :=

nL(t, d)

nL(t, d) + k1 · (b · pivdl(d , c) + (1− b))

36 / 133

IR Models


TF-IDF

TF Variants: Collection-wide TF(t, c)

Analogously to TF(t, d), the next definition defines the variants ofTF(t, c), the collection-wide term frequency.Not considered any further here.

37 / 133

IR Models


TF-IDF

TFtotal and TFlog

0 50 100 150 200

050

100

150

200

nL(t,d)

tf tot

al

0 50 100 150 200

01

23

45

nL(t,d)

tf log

Bias towards documents with many terms(e.g. books vs. Twitter tweets)

TFtotal:

too steepassumes all occurrences are independent(same impact)

TFlog:

less impact to subsequent occurrencesthe base of the logarithm is rankinginvariant, since it is a constant:

TFlog,base(t, d) :=ln(1 + tfd)

ln(base)

38 / 133

IR Models


TF-IDF

Logarithmic TF: Dependence Assumption

The logarithmic TF assigns less impact to subsequent occurrencesthan the total TF does. This aspect becomes clear whenreconsidering that the logarithm is an approximation of theharmonic sum:

TFlog(t, d) = ln(1 + tfd) ≈ 1 +1

2+ . . .+

1

tfdtfd > 0

Note: ln(n + 1) =∫ n+1

11x dx .

Whereas: TFtotal(t, d) = 1 + 1 + ...+ 1

The first occurrence of a term counts in full, the secondcounts 1/2, the third counts 1/3, and so forth.

This gives a particular insight into the type of dependence thatis reflected by “bending” the total TF into a saturating curve.

39 / 133

IR Models


TF-IDF

TFsum, TFmax and TFfrac: Graphical Illustration

0 50 100 150 200

0.0

0.2

0.4

0.6

0.8

1.0

nL(t,d)

tf

tfsum: NL(d)=200

tfsum: NL(d)=2000

tfmax: NL(tmax d)=500

tfmax: NL(tmax d)=400

0 50 100 150 200

0.0

0.2

0.4

0.6

0.8

1.0

nL(t,d)

tf fra

c

tffrac K=1tffrac K=5

tffrac K=10

tffrac K=20

tffrac K=100

tffrac K=200

tffrac K=1000

40 / 133

IR Models


TF-IDF

TFsum, TFmax and TFfrac: Analysis

Document length normalisation (TFfrac: K may depend ondocument length)

Usually TFmax yields higher TF-values than TFsum

Linear TF variants are not really important anymore, sinceTFfrac (TFBM25) delivers better and more stable quality

TFfrac yields relatively high TF-values already for smallfrequencies, and the curve saturates for large frequencies

The good and stable performance of BM25 indicates that thisnon-linear nature is key for achieving good retrieval quality.

41 / 133

IR Models


TF-IDF

Fractional TF: Dependence Assumption

What we refer to as “fractional TF”, is a ratio:

ratio(x , y) =x

x + y=

tfdtfd + Kd

= TFfrac,K (t, d)

The ratio is related to the harmonic sum of squares.

n

n + 1≈ 1 +

1

22+ . . .+

1

n2n > 0

This approximation is based on the following integral:∫ n+1

1

1

z2dz =

[−1

z

]n+1

1

= 1− 1

n + 1=

n

n + 1

TFfrac assumes more dependence than TFlog. k-th occurrence of aterm has an impact of 1/k2.

42 / 133

IR Models


TF-IDF

TFBM25

TFBM25 is a special TFfrac used in the BM25 modelK is proportional to the pivoted document length(pivdl(d , c) = dl/avgdl(c)) and involves adjustmentparameters (k1, b).The common definition is:

KBM25,k1,b(d , c) := k1 · (b · pivdl(d , c) + (1− b))

For b = 1, K is equal to k1 for average documents, less thank1 for short documents and greater than k1 for longdocuments.Large b and k1 lead to a strong variation of K with an highimpact on the retrieval scoreDocuments shorter than the average have an advantage overdocuments longer than the average 43 / 133

IR Models


TF-IDF

Inverse Document Frequency IDF

The IDF (inverse document frequency) is the negativelogarithm of the DF (document frequency).

Idea: the less documents a term appears in, the morediscrimiative or ’informative’ it is

44 / 133

IR Models


TF-IDF

DF Variants

DF(t, c) is a quantification of the document frequency, df(t, c).The main variants are:

df(t, c) := dftotal(t, c) := nD(t, c)

dfsum(t, c) :=nD(t, c)

ND(c)= PD(t|c)

(=

df(t, c)

ND(c)

)dfsum,smooth(t, c) :=

nD(t, c) + 0.5

ND(c) + 1

dfBIR(t, c) :=nD(t, c)

ND(c)− nD(t, c)

dfBIR,smooth(t, c) :=nD(t, c) + 0.5

ND(c)− nD(t, c) + 0.5

45 / 133

IR Models


TF-IDF

IDF Variants

IDF(t, c) is the negative logarithm of a DF quantification. Themain variants are:

idftotal(t, c) := − log dftotal(t, c)

idf(t, c) := idfsum(t, c) := − log dfsum(t, c) = − log PD(t|c)

idfsum,smooth(t, c) := − log dfsum,smooth(t, c)

idfBIR(t, c) := − log dfBIR(t, c)

idfBIR,smooth(t, c) := − log dfBIR,smooth(t, c)

IDF is high for rare terms and low for frequent terms.

46 / 133

IR Models


TF-IDF

Burstiness

A term is bursty is it occurs often in the documents in whichit occurs

Burstiness is measured by the average term frequency in theelite set:

avgtf(t, c) =nL(t, c)

nD(t, c)

Intuition (relevant vs. non-relevant documents):

A good term is rare (not frequent, high IDF) and solitude (notbursty, low avgtf) in all documents (all non-relevantdocuments)Among relevant documents, a good term is frequent (low IDF,appears in many relevant documents) and bursty (high avgtf)

47 / 133

IR Models


TF-IDF

IDF and Burstiness

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

0 0.2 0.4 0.6 0.8 1

PD(t|c)

idf(t,c)

λ(t,c) = 1

λ(t,c) = 2

λ(t,c) = 4

10

8

6

4

2

BURSTY

RARE

avgtf(t,c)

P (t|c)0.1 0.2 0.3 0.4 0.5

RARE

SOLITUDE

FREQUENT

SOLITUDE

FREQUENT

BURSTY

BIR

Example: nD(t1, c) = nD(t2, c) = 1, 000. Same IDF.nL(t1, d) = 1 for 1,000 documents, whereasnL(t2, d) = 1 for 999 documents, and nL(t2) = 1, 001 for one doc.nL(t1, c) = 1, 000 and nL(t2, c) = 2, 000.avgtf(t1, c) = 1 and avgtf(t2, c) = 2.λ(t, c) = avgtf(t, c) · PD(t|c)

48 / 133

IR Models


TF-IDF

TF-IDF Term Weight

Definition (TF-IDF term weight wTF-IDF)

wTF-IDF(t, d , q, c) := TF(t, d) · TF(t, q) · IDF(t, c)

49 / 133

IR Models


TF-IDF

TF-IDF RSV

Definition (TF-IDF retrieval status value RSVTF-IDF)

RSVTF-IDF(d , q, c) :=∑t

wTF-IDF(t, d , q, c)

RSVTF-IDF(d , q, c) =∑t

TF(t, d) · TF(t, q) · IDF(t, c)

50 / 133

IR Models


TF-IDF

Probabilistic IDF: Probability of Being Informative

IDF so far is not probabilistic

In probabilistic scenarios, a normalised IDF value such as

0 ≤ idf(t, c)

maxidf(c)≤ 1

can be useful

maxidf(c) := − log 1ND(c) = log(ND(c)) is

the maximal value of idf(t, c) (when a termoccurs only in 1 document)

0 50000 100000 150000 200000

89

1011

12

Nd(c)

max

idf

The normalisation does not affect the ranking

51 / 133

IR Models


TF-IDF

Probability that term t is informative

A probabilistic semantics of a max-normalised IDF can be achievedby introducing an informativeness-based probability,[Roelleke, 2003], as opposed to the normal notion ofoccurrence-based probability, and we denote the probability asP(t informs|c), to contrast it from the usual

P(t|c) := P(t occurs|c) = nD(t,c)ND(c) .

52 / 133


PRF: The Probability of RelevanceFramework

IR Models


PRF: The Probability of Relevance Framework


Relevance is at the core of any information retrieval model

With r denoting relevance, d a document and q a query, theprobability that a document is relevant is

P(r |d , q)

54 / 133

IR Models



Probability Ranking Principle

[Robertson, 1977], “The Probability Ranking Principle (PRP) inIR”, describes the PRP as a framework to discuss formally “what isa good ranking”? [Robertson, 1977] quotes Cooper’s formalstatement of the PRP:

“If a reference retrieval system’s response to each requestis a ranking of the documents in the collections in orderof decreasing probability of usefulness to the user whosubmitted the request, ..., then the overall effectivenessof the system ... will be the best that is obtainable onthe basis of that data.”

Formally, we can capture the principle as follows. Let A and B berankings. Then, a ranking A is better than a ranking B if at everyrank, the probability of satisfaction in A is higher than for B, i.e.:

∀rank : P(satisfactory|rank,A) > P(satisfactory|rank,B) 55 / 133

IR Models



PRF: Illustration

Example (Probability of relevance)Let three users u1, u2, u3 have judged document-query pairs.

User Doc Query Judgement R

u1 d1 q1 ru1 d2 q1 ru1 d3 q1 ru2 d1 q1 ru2 d2 q1 ru2 d3 q1 ru3 d1 q1 ru3 d2 q1 ru4 d1 q1 ru4 d2 q1 r

PU(r |d1, q1) = 3/4, and PU(r |d2, q1) = 1/4, and PU(r |d3, q1) = 0/2.Subscript in PU : “event space”, a set of users.Total probability:

PU(r |d , q) =∑u∈U

P(r |d , q, u) · P(u)

56 / 133

IR Models



Bayes Theorem

Relevance judgements can be incomplete

In any case, for a new query we often do not have judgements

The probability of relevance is estimated via Bayes’Theorem:

P(r |d , q) =P(d , q|r) · P(r)

P(d , q)=

P(d , q, r)

P(d , q)

Decision whether or not to retrieve a document is based onthe so-called “Bayesian decision rule”:retrieve document d , if the probability of relevance is greaterthan the probability of non-relevance:

retrieve d for q if P(r |d , q) > P(r |d , q)

57 / 133

IR Models



Probabilistic Odds, Rank Equivalence

We can express the probabilistic odds of relevance

O(r |d , q) =P(r |d , q)

P(r |d , q)=

P(d , q, r)

P(d , q, r)=

P(d , q|r)

P(d , q|r)· P(r)

P(r)

Since P(r)/P(r) is a constant, the following rank equivalenceholds:

O(r |d , q)rank=

P(d , q|r)

P(d , q|r)

Often we don’t need the exact probability, but aneasier-to-compute value that is rank equivalent (i.e. it preserves theranking)

58 / 133

IR Models



Probabilistic Odds

The document-pair probabilities can be decomposed in two ways:

P(d , q|r)

P(d , q|r)=

P(d |q, r) · P(q|r)

P(d |q, r) · P(q|r)(⇒ BIR/Poisson/BM25)

=P(q|d , r) · P(d |r)

P(q|d , r) · P(d |r)(⇒ LM?)

The equation where d depends on q, is the basis of BIR,Poisson and BM25...document likelihood P(d |q, r)

The equation where q depends on d has been related to LM,P(q|d), [Lafferty and Zhai, 2003], “Probabilistic RelevanceModels Based on Document and Query Generation”

This relationship and the assumptions required to establish it,are controversial [Luk, 2008]

59 / 133

IR Models



Documents as Feature Vectors

The next step represents document d as avector ~d = (f1, . . . , fn) in a space of features ~fi :

P(d |q, r) = P(~d |q, r)

A feature could be, for example, the frequency of a word(term), the document length, document creation time, time oflast update, document owner, number of in-links, or numberof out-links.

See also the Vector Space Model in the Introduction – herewe used term weights as featuresAssumptions based on features:

Feature independence assumptionNon-query term assumptionTerm frequency split

60 / 133

IR Models



Feature Independence Assumption

The features (terms) are independent events:

P(~d |q, r) ≈∏i

P(fi |q, r)

Weaker assumption for the fraction of feature probabilities(linked dependence):

P(d |q, r)

P(d |q, r)≈∏i

P(fi |q, r)

P(fi |q, r)

Here it is not required to distinguish between these twoassumptions

P(fi |q, r) and P(fi |q, r) may be estimated for instance bymeans of relevance judgements

61 / 133

IR Models



Non-Query Term Assumption

Non-query terms can be ignored for retrieval (ranking)

This reduces the number of features/terms/dimensions toconsider when computing probabilities

For non-query terms, the feature probability is the same in relevantdocuments and non-relevant documents.

for all non-query terms:P(fi |q, r)

P(fi |q, r)= 1

Then

62 / 133

IR Models



Term Frequency Split

The product over query terms is split into two parts

First part captures the fi > 0 features, i.e. the document terms

Second part captures the fi = 0 features, i.e. thenon-document terms

63 / 133


BIR: Binary Independence Retrieval

IR Models




The BIR instantiation of the PRF assumes the vectorcomponents to be binary term features,i.e. ~d = (x1, x2, . . . , xn), where xi ∈ {0, 1}Term occurrences are represented in a binary feature vector ~din the term space

The event xt = 1 is expressed as t, and xt = 0 as t

65 / 133

IR Models



BIR Term Weight

We save the derivation ... after few steps from O(r |d , q) ...

Event Space: d and q are binary vectors.

Definition (BIR term weight wBIR)

wBIR(t, r , r) := log

(PD(t|r)

PD(t|r)· PD(t|r)

PD(t|r)

)

Simplified form (referred to as F1) considering term presence only:

wBIR,F1(t, r , c) := logPD(t|r)

PD(t|c)

66 / 133

IR Models



BIR RSV

Definition (BIR retrieval status value RSVBIR)

RSVBIR(d , q, r , r) :=∑

t∈d∩qwBIR(t, r , r)

RSVBIR(d , q, r , r) =∑

t∈d∩qlog

P(t|r) · P(t|r)

P(t|r) · P(t|r)

67 / 133

IR Models



BIR Estimations

Relevance judgements for 20 documents (examplefrom [Fuhr, 1992]):

di 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20x1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0x2 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0R r r r r r r r r r r r r r r r r r r r r

ND(r) = 12; ND(r) = 8; P(t1|r) = P(x1 = 1|r) = 8/12 = 2/3P(t1|r) = 2/8 = 1/4; P(t1|r) = 4/12 = 1/3; P(t1|r) = 6/8 = 3/4Dito for t2.

68 / 133

IR Models



Missing Relevance Information I

Estimation of non-relevant documents:

Take collection-wide term probability as approximation (r ≡ c)

P(t|r) ≈ P(t|c)

Use the set of all documents minus the set of relevantdocuments (r = c\r)

P(t|r) ≈ nD(t, c)− nD(t, r)

ND(c)− ND(r)

69 / 133

IR Models



Missing Relevance Information II

“Empty” set problem: For ND(r) = ∅ (no relevant documents),P(t|r) is not defined. Smoothing deals with this situation.

Add the query to the set of relevant documents and thecollection:

P(t|r) =nd(t, r) + 1

ND(r) + 1

Other forms of smoothing, e.g.

P(t|r) =nd(t, r) + 0.5

ND(r) + 1

This variant may be justified by Laplace’s law of succession.

70 / 133

IR Models



RSJ Term Weight

Recall wBIR,F1(t, r , c) := log PD(t|r)PD(t|c)

Definition (RSJ term weight wRSJ)

The RSJ term weight is a smooth BIR term weight.The probability estimation is as follows:P(t|r) := (nD(t, r) + 0.5)/(ND(r) + 1);P(t|r) := ((nD(t, c) + 1)− (nD(t, r) + 0.5))/((ND(c) + 2)− (ND(r) + 1)).

wRSJ,F4(t, r , r , c) :=

log

((nd (t, r) + 0.5)/(ND(r)− nd (t, r) + 0.5)

(nD(t, c)−nD(t, r)+0.5)/((ND(c)−ND(r))− (nD(t, c)−nD(t, r)) + 0.5)

)

71 / 133


Poisson and 2-Poisson

IR Models



Poisson Model I

Less known model; however, there are good reasons to look at thismodel

“Demystify” the Poisson probability – some research studentsresign when hearing “Poisson” ;-)

Next to the BIR model the natural instantiation of a PRFmodel; the BIR model is a special case of the Poisson model

The 2-Poisson probability is arguably the foundation of theBM25-TF quantification [Robertson and Walker, 1994],“SomeSimple Effective Approximations to the 2-Poisson Model”.

73 / 133

IR Models



Poisson Model II

The Poisson probability is a model of randomness. Divergencefrom randomness (DFR), which is based on the probabilityP(t ∈ d |collection) = P(tfd > 0|collection). The probabilityP(tfd |collection) can be estimated by a Poisson probability.

The Poisson parameter λ(t, c) = nL(t, c)/ND(c), i.e. theaverage number of term occurrences, relates Document-basedand Location-based probabilities.

avgtf(t, c) · PD(t|c) = λ(t, c) = avgdl(c) · PL(t|c)

We refer to this relationship as Poisson Bridge since theaverage term frequency is the parameter of the Poissonprobability.

74 / 133

IR Models



Poisson Model III

The Poisson model yields a foundation of TF-IDF (see Part II)

The Poisson bridge helps to relate TF-IDF and LM (seePart II)

75 / 133

IR Models



Poisson Distribution

Let t be an event that occurs in average λt times. The Poissonprobability is:

PPoisson,λt (k) :=λktk!· e−λt

76 / 133

IR Models



Poisson Example

The probability that k = 4 sunny daysoccur in a week, given the averageλ = p · n = 180/360 · 7 = 3.5 sunnydays per week, is:

0.00

0.05

0.10

0.15

0.20

k

P3.

5(k)

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

PPoisson,λ=3.5(k = 4) =(3.5)4

4!· e−3.5 ≈ 0.1888

77 / 133

IR Models



Poisson PRF: Basic Idea

The Poisson model could be used to estimate the documentprobability P(d |q, r) as the product of the probabilities of thewithin-document term frequencies kt :

P(d |q, r) = P(~d |q, r) =∏t

P(kt |q, r)

kt := tfd := nL(t, d)

~d is a vector of term frequencies (cf. BIR: binary vector ofoccurrence/non-occurrence)

78 / 133

IR Models



Poisson PRF: Odds

Rank-equivalence, non-query term assumption and probabilisticodds lead us to:

O(r |d , q)rank=∏t∈q

P(kt |r)

P(kt |r), kt := tfd := nL(t, d)

Splitting the product into document and non-document termsyields:

O(r |d , q)rank=

∏t∈d∩q

P(kt |r)

P(kt |r)·∏

t∈q\d

P(0|r)

P(0|r)

79 / 133

IR Models



Poisson PRF: Meaning of Poisson Probability

For the set r (r) and a document d of length dl, we look at:

How many times would we expect to see a term t in d?

How many times do we actually observe t in d (= kt)?

P(kt |r) is highest if our expectation is met by our observation.

λ(t, d , r) = dl · PL(t|r) is a number of expected occurrences in adocument with length dl (how many times will we draw t if we haddl trials?)

P(kt |r) = PPoisson,λ(t,d ,r)(kt): probability to observe kt occurrencesof term t in dl trials

80 / 133

IR Models



Poisson PRF: Example

dl = 5,PL(t|r) = 5/10 = 1/2λ(t, d , r) = 5 · 1/2 = 2.5P(2|r) = PPoisson,2.5(2) = 0.2565156

0.00

0.05

0.10

0.15

0.20

0.25

k

P2.

5(k)

0 1 2 3 4 5 6 7 8 9 10

81 / 133

IR Models



Poisson Term Weight

Event Space: d and q are frequency vectors.

Definition (Poisson term weight wPoisson)

wPoisson(t, d , q, r , r) := TF(t, d) · logλ(t, d , r)

λ(t, d , r)

wPoisson(t, d , q, r , r) = TF(t, d) · logPL(t|r)

PL(t|r)

λ(t, d , r) = dl · PL(t|r)

TF(t, d) = kt (smarter TF quantification possible)

82 / 133

IR Models



Poisson RSV

Definition (Poisson retrieval status value RSVPoisson)

RSVPoisson(d , q, r , r) :=

∑t∈d∩q

wPoisson(t, d , q, r , r)

+

len normPoisson

RSVPoisson(d , q, r , r) =

∑t∈d∩q

TF(t, d) · logPL(t|r)

PL(t|r)

+

dl ·∑t

(PL(t|r)− PL(t|r))

83 / 133

IR Models



2-Poisson

Viewed as a motivation for the BM25-TF quantification,[Robertson and Walker, 1994], “Simple Approximations to the2-Poisson Model”. In an exchange with Stephen Robertson, heexplained:

“The investigation into the 2-Poisson probabilitymotivated the BM25-TF quantification.Regarding thecombination of TF and RSJ weight in BM25, TF can beviewed as a factor to reflect the uncertainty aboutwhether the RSJ weight wRSJ is correct; for terms with arelatively high within-document TF, the weight is correct;for terms with a relatively low within-document TF, thereis uncertainty about the correctness. In other words, theTF factor can be viewed as a weight to adjust the impactof the RSJ weight.”

84 / 133

IR Models



2-Poisson Example: How many cars to expect? I

How many cars are expected on a given commuter car park?

Approach 1: In average, there are 700 cars per week. Thedaily average is: λ = 700/7 = 100 cars/day.

Then, Pλ=100(k) is the probability that there are k carswanting to park on a given day.

Estimation is less accurate than an estimation based on a2-dimensional model – Mo-Fr are the busy days, and onweek-ends, the car park is nearly empty.

This means that a distribution such as (130, 130, 130, 130,130, 25, 25) is more likely than 100 each day.

85 / 133

IR Models



2-Poisson Example: How many cars to expect? II

Approach 2: In a more detailed analysis, we observe 650 carsMon-Fri (work days) and 50 cars Sat-Sun (week-end days).

The averages are: λ1 = 650/5 = 130 cars/work-day,λ2 = 50/2 = 25 cars/we-day.

Then, Pµ1=5/7,λ1=130,µ2=2/7,λ2=25(k) is the 2-dimensionalPoisson probability that there are k cars looking for a carpark.

86 / 133

IR Models



2-Poisson Example: How many cars to expect? III

Main idea of the 2-Poisson probability: combine (interpolate,mix) two Poisson probabilities

P2-Poisson,λ1,λ2,µ(kt) := µ ·λkt1

kt !· e−λ1 + (1− µ) ·

λkt2

kt !· e−λ2

Such a mixture model is also used with LM (later)

λ1 could be over all documents whereas λ2 could be over anelite set (e.g. documents that contain at least one query term)

87 / 133


BM25

IR Models


BM25

BM25

One of the most prominent IR modelsThe ingredients have already been prepared:

TFBM25,K (t, d)wRSJ,F4(t, r , r , c)

Wikipedia 2014 formulation: Given a query Q containingkeywords q1, . . . , qn:

score(D,Q) =n∑

i=1

IDF(qi ) ·f (qi ,D) · (k1 + 1)

f (qi ,D) + k1 · (1− b + b · |D|avgdl )

with f (qi ,D) = nL(qi ,D) (qi ’s term frequency)Here:

Ignore (k1 + 1) (ranking invariant!)Use wRSJ,F4(t, r , r , c). If relevance information is missing:wRSJ,F4(t, r , r , c) ≈ IDF (t)

89 / 133

IR Models


BM25

BM25 Term Weight

Definition (BM25 term weight wBM25)

wBM25,k1,b,k3(t, d , q, r , r) :=∑t∈d∩q

TFBM25,k1,b(t, d) · TFBM25,k3(t, q) · wRSJ(t, r , r)

TFBM25,k1,b(t, d) :=tfd

tfd + k1 · (b · pivdl(d) + (1− b))

90 / 133

IR Models


BM25

BM25 RSV

Definition (BM25 retrieval status value RSVBM25)

RSVBM25,k1,b,k2,k3(d , q, r , r , c) := ∑t∈d∩q

wBM25,k1,b,k3(t, d , q, r , r)

+ len normBM25,k2

Additional length normalisation

len normBM25,k2(d , q, c) := k2 · ql · avgdl(c)− dl

avgdl(c) + dl

(ql is query length). Suppresses long documents (negative lengthnorm).

91 / 133

IR Models


BM25

BM25: Summary

We have discussed the foundations of BM25, an instance of theprobability relevance framework (PRF).

TF quantification TFBM25 using TFfrac and the pivoteddocument length

TFBM25 can be related to probability theory throughsemi-subsumed event occurrences (out of scope of this talk)

RSJ term weight wRSJ as smooth variant of the BIR termweight (“0.5-smoothing” can be explained through Laplace’slaw of succession)

92 / 133

IR Models


BM25

Document- and Query-Likelihood Models

We are now leaving the world of document-likelihood models andmove towards LM, the query-likelihood model.

P(d , q|r)

P(d , q|r)=

P(d |q, r) · P(q|r)

P(d |q, r) · P(q|r)(⇒ BIR/Poisson/BM25)

=P(q|d , r) · P(d |r)

P(q|d , r) · P(d |r)(⇒ LM?)

93 / 133


LM: Language Modelling

IR Models



Language Modelling (LM)

Popular retrieval model since the late 90s[Ponte and Croft, 1998, Hiemstra, 2000,Croft and Lafferty, 2003]

Compute probability P(q|d) that a document generates aquery

q is a conjunction of term events: P(q|d) ∝∏

P(t|d)

Zero-probability problem: terms t that don’t appear in d leadto P(t|d) = 0 and hence P(q|d) = 0

Mix the within-document term probability P(t|d) and thecollection-wide term probability P(t|c)

95 / 133

IR Models



Probability Mixture

Let three events “x , y , z”, and two conditional probabilities P(z |x)and P(z |y) be given.Then, P(z |x , y) can be estimated as a linear combination/mixtureof P(z |x) and P(z |y).

P(z |x , y) ≈ δx · P(z |x) + (1− δx) · P(z |y)

Here, 0 < δx < 1 is the mixture parameter.The mixture parameters can be constant (Jelinek-Mercer mixture),or can be set proportional to the total probabilities.

96 / 133

IR Models



Probability Mixture Example

Let P(sunny,warm, rainy, dry,windy|glasgow) describe theprobability that a day in Glasgow is sunny, the next day iswarm, the next rainy, and so forth

If for one event (e.g. sunny), the probability were zero, thenthe probability of the conjunction (product) is zero. A mixturesolves the problem.

For example, mix P(x |glasgow) with P(x |uk) whereP(x |uk) > 0 for each event x .

Then, in a week in winter, when P(sunny|glasgow) = 0, andfor the whole of the UK, the weather office reports 2 of 7 daysas sunny, the mixed probability is:

P(sunny|glasgow, uk) = δ · 0

7+ (1− δ) · 2

7

97 / 133

IR Models



LM1 Term Weight

Event Space: d and q are sequences of terms.

Definition (LM1 term weight wLM1)

P(t|d , c) := δd · P(t|d) + (1− δd) · P(t|c)

wLM1,δd (t, d , q, c) := TF(t, q) · log

=P(t|d ,c)︷︸︸︷(δd · P(t|d) + (1− δd) · P(t|c))

P(t|d): foreground probability

P(t|c): background probability

98 / 133

IR Models



Language Modelling Independence Assumption

We assume terms are independent, meaning the product over theterm probabilities P(t|d) is equal to P(q|d):

Probability that mixture of background and foregroundprobabilities generates query as sequence of terms:

t IN q: sequence of terms (e.g.q = (sailing, boat, sailing))

t ∈ q: set of terms; TF(sailing, q) = 2

Note: log(P(t|d , c)TF(t,q)

)= TF(t, q) · log (P(t|d , c))

99 / 133

IR Models



LM1 RSV

Definition (LM1 retrieval status value RSVLM1)

RSVLM1,δd (d , q, c) :=∑t∈q

wLM1,δd (t, d , q, c)

100 / 133

IR Models



JM-LM Term Weight

For constant δ, the score can be divided by∏

t IN q(1− δ). Thisleads to the following equation, [Hiemstra, 2000]:

P(q|d , c)

P(q|c) ·∏

t IN q(1− δ)=∏

t∈d∩q

(1 +

δ

1− δ· P(t|d)

P(t|c)

)TF(t,q)

Definition (JM-LM (Jelinek-Mercer) term weight wJM-LM)

wJM-LM,δ(t, d , q, c) := TF(t, q) · log

(1 +

δ

1− δ· P(t|d)

P(t|c)

)

101 / 133

IR Models



JM-LM RSV

Definition (JM-LM retrieval status value RSVJM-LM)

RSVJM-LM,δ(d , q, c) :=∑

t∈d∩qwJM-LM,δ(t, d , q, c)


t∈d∩qTF(t, q) · log

(1 +

δ

1− δ· P(t|d)

P(t|c)

)We only need to look at terms that appear in both document andquery.

102 / 133

IR Models



Dirichlet-LM Term Weight

Document-dependent mixture parameter: δD = dldl+µ

Definition (Dirich-LM term weight wDirich-LM)

wDirich-LM,µ(t, d , q, c) := TF(t, q) · log

(µ

µ+|d |+|d ||d |+µ

· P(t|d)

P(t|c)

)

103 / 133

IR Models



Dirich-LM RSV

Definition (Dirich-LM retrieval status value RSVDirich-LM)

RSVDirich-LM,µ(d , q, c) :=∑t∈q

wDirich-LM,µ(t, d , q, c)

RSVDirich-LM(d , q, c, µ) =∑t∈q

TF(t, q) · log

(µ

µ+ |d |+

|d ||d |+ µ

· P(t|d)

P(t|c)

)

104 / 133


PIN’s: Probabilistic InferenceNetworks

IR Models


PIN’s: Probabilistic Inference Networks


Random variables andconditional dependencies asdirected acyclic graph(DAG)

Minterm: Conjunction ofterm events (e.g. t1 ∧ t2)

Decomposition of (disjoint)minterms X leads tocomputation

P(q|d) =∑x∈X

P(q|x)·P(x |d)

106 / 133

IR Models



Link Matrix

Probability flow in a PIN can be described via a link ortransition matrix

Link matrix L contains the transition probabilitiesP(target|x , source)

Usually, P(target|x , source) = P(target|x) is assumed, andthis assumption is referred to as the linked independenceassumption.

107 / 133

IR Models



PIN Link Maxtrix

L :=

[P(q|x1) . . . P(q|xn)P(q|x1) . . . P(q|xn)

]

(P(q|d)P(q|d)

)= L ·

P(x1|d)...

P(xn|d)

108 / 133

IR Models



Link Matrix for 3 Terms

For illustrating the link matrix, a matrix for three terms is shownnext.

LTransposed =

P(q|t1, t2, t3) P(q|t1, t2, t3)P(q|t1, t2, t3) P(q|t1, t2, t3)P(q|t1, t2, t3) P(q|t1, t2, t3)P(q|t1, t2, t3) P(q|t1, t2, t3)P(q|t1, t2, t3) P(q|t1, t2, t3)P(q|t1, t2, t3) P(q|t1, t2, t3)P(q|t1, t2, t3) P(q|t1, t2, t3)P(q|t1, t2, t3) P(q|t1, t2, t3)

109 / 133

IR Models



Special Link Matrices

The matrices Lor and Land reflect the boolean combination of thelinkage between a source and a target.

Lor =

[1 1 1 1 1 1 1 00 0 0 0 0 0 0 1

]

Land =

[1 0 0 0 0 0 0 00 1 1 1 1 1 1 1

]

110 / 133

IR Models



y = A x

(P(q|d)P(q|d)

)= L ·

P(t1, t2, t3|d)P(t1, t2, t3|d)P(t1, t2, t3|d)P(t1, t2, t3|d)P(t1, t2, t3|d)P(t1, t2, t3|d)P(t1, t2, t3|d)P(t1, t2, t3|d)

111 / 133

IR Models



Turtle/Croft Link Matrix

Let wt := P(q|t) be the query term probabilities. Then, estimatethe link matrix elements P(q|x), where x is a boolean combinationof terms, as follows.

LTurtle/Croft =

[1 w1+w2

w0

w1+w3w0

w1w0

w2+w3w0

w2w0

w3w0

0

0 w3w0

w2w0

w2+w3w0

w1w0

w1+w3w0

w1+w2w0

1

]

[Turtle and Croft, 1992, Croft and Turtle, 1992]

112 / 133

IR Models



PIN Term Weight

Definition (PIN-based term weight wPIN)

wPIN(t, d , q, c) :=1∑

t′ P(q|t ′, c)· P(q|t, c) · P(t|d , c)

113 / 133

IR Models



PIN RSV

Definition (PIN-based retrieval status value RSVPIN)

RSVPIN(d , q, c) :=∑t

wPIN(t, d , q, c)

RSVPIN(d , q, c) =1∑

t P(q|t, c)·∑

t∈d∩qP(q|t, c) · P(t|d , c)

P(q|t, c) proportional pidf(t, c)P(t|d , c) proportional TF(t, d)

114 / 133

IR Models



PIN RSV Computation Example

Let the following term probabilities be given:

ti P(ti |d) P(q|ti )sailing 2/3 1/10, 000boats 1/2 1/1, 000

Terms are independent events. P(t|d) is proportional to thewithin-document term frequency, and P(q|t) is proportional to theIDF. The RSV is:

RSVPIN(d , q, c) =1

1/10, 000 + 1/1, 000· (1/10, 000 · 2/3 + 1/1, 000 · 1/2) =

=1

11/10, 000· (2/30, 000 + 1/2, 000) =

10, 000

11· (2 + 15)/30, 000 =

=1

11· (2 + 15)/3 = 17/33 ≈ 0.51

115 / 133


Relevance-based Models

IR Models




VSM (Rocchio’s formulae)

PRF

[Lavrenko and Croft, 2001], “Relevance-based LanguageModels”: LM-based approach to estimate P(t|r). “Massivequery expansion”.

117 / 133

IR Models



Relevance Feedback in the VSM

[Rocchio, 1971], “Relevance Feedback in Information Retrieval”, isthe must-have reference and background for what a relevancefeedback model aims at. There are two formulations thataggregate term weights:

weight(t, q) = weight(t, q) +1

|R|·∑d∈R

~d − 1

|R|·∑d∈R

~d

weight(t, q) = α · weight(t, q) + β ·∑d∈R

~d − γ ·∑d∈R

~d

118 / 133


Foundations: Summary

IR Models




1 TF-IDF: Semantics of TF quantification.

2 PRF: basis of BM25. LM?

3 BIR

4 Poisson

5 BM25: The P(d |q)/P(d) side of IR models.

6 LM: The P(q|d)/P(q) side of IR models.

7 PIN’s: Special link matrix .

8 Relevance-based Models

120 / 133

IR Models



Model Overview: TF-IDF, BIR, Poisson

RSVTF-IDF(d , q, c) :=∑t

wTF-IDF(t, d , q, c)

wTF-IDF(t, d , q, c) := TF(t, d) · TF(t, q) · IDF(t, c)

RSVBIR(d , q, r , r) :=∑

t∈d∩q

wBIR(t, r , r)

wBIR(t, r , r) := log(

PD (t|r)PD (t|r)

· PD (t|r)PD (t|r)

)RSVPoisson(d , q, r , r) :=

∑t∈d∩q

wPoisson(t, d , q, r , r)

+ len normPoisson

wPoisson(t, d , q, r , r) := TF(t, d) · log λ(t,d,r)λ(t,d,r)

(= TF(t, d) · log PL(t|r)

PL(t|r)

)

121 / 133

IR Models



Model Overview: BM25, LM, DFR

RSVBM25,k1,b,k2,k3 (d , q, r , r , c) :=

∑t∈d∩q

wBM25,k1,b,k3 (t, d , q, r , r)

+ len normBM25,k2

wBM25,k1,b,k3 (t, d , q, r , r) :=∑

t∈d∩q TFBM25,k1,b(t, d) · TFBM25,k3 (t, q) · wRSJ(t, r , r)


t∈d∩q

wJM-LM,δ(t, d , q, c)

wJM-LM,δ(t, d , q, c) := TF(t, q) · log(

1 + δ1−δ ·

P(t|d)P(t|c)

)RSVDirich-LM,µ(d , q, c) :=

∑t∈q

wDirich-LM,µ(t, d , q, c)

wDirich-LM,µ(t, d , q, c) := TF(t, q) · log(

µµ+|d| + |d|

|d|+µ ·P(t|d)P(t|c)

)RSVDFR,M(d , q, c) :=

∑t∈d∩q

wDFR,M(t, d , c)

wDFR-1,M(t, d , c) := −log PM(t ∈ d |c) wDFR-2,M(t, d , c) := −log PM(tfd |c)

122 / 133

IR Models



The End (of Part I)

Material from book: “IR Models: Foundations and Relationships”.Morgan & Claypool, 2013.References and textual explanations of formulae in book.

123 / 133

IR Models



Bib I

Amati, G. and van Rijsbergen, C. J. (2002).

Probabilistic models of information retrieval based on measuring the divergence from randomness.ACM Transaction on Information Systems (TOIS), 20(4):357–389.

Bookstein, A. (1980).

Fuzzy requests: An approach to weighted Boolean searches.Journal of the American Society for Information Science, 31:240–247.

Brin, S. and Page, L. (1998).

The anatomy of a large-scale hypertextual web search engine.Computer Networks, 30(1-7):107–117.

Bruza, P. and Song, D. (2003).

A comparison of various approaches for using probabilistic dependencies in language modeling.In SIGIR, pages 419–420. ACM.

Church, K. and Gale, W. (1995).

Inverse document frequency (idf): A measure of deviation from Poisson.In Proceedings of the Third Workshop on Very Large Corpora, pages 121–130.

124 / 133

IR Models



Bib II

Cooper, W. (1991).

Some inconsistencies and misnomers in probabilistic IR.In Bookstein, A., Chiaramella, Y., Salton, G., and Raghavan, V., editors, Proceedings of the FourteenthAnnual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages57–61, New York.

Cooper, W. S. (1988).

Getting beyond Boole.Information Processing and Management, 24(3):243–248.

Cooper, W. S. (1994).

Triennial ACM SIGIR award presentation and paper: The formalism of probability theory in IR: A foundationfor an encumbrance.In [Croft and van Rijsbergen, 1994], pages 242–248.

Croft, B. and Lafferty, J., editors (2003).

Language Modeling for Information Retrieval.Kluwer.

Croft, W. and Harper, D. (1979).

Using probabilistic models of document retrieval without relevance information.Journal of Documentation, 35:285–295.

125 / 133

IR Models



Bib III

Croft, W. and Turtle, H. (1992).

Retrieval of complex objects.In Pirotte, A., Delobel, C., and Gottlob, G., editors, Advances in Database Technology — EDBT’92, pages217–229, Berlin et al. Springer.

Croft, W. B. and van Rijsbergen, C. J., editors (1994).

Proceedings of the Seventeenth Annual International ACM SIGIR Conference on Research and Developmentin Information Retrieval, London, et al. Springer-Verlag.

Deerwester, S., Dumais, S., Furnas, G., Landauer, T., and Harshman, R. (1990).

Indexing by latent semantic analysis.Journal of the American Society for Information Science, 41(6):391–407.

Dumais, S. T., Furnas, G. W.and Landauer, T. K., and Deerwester, S. (1988).

Using latent semantic analysis to improve information retrieval.pages 281–285.

Fang, H. and Zhai, C. (2005).

An exploration of axiomatic approaches to information retrieval.In SIGIR ’05: Proceedings of the 28th annual international ACM SIGIR conference on Research anddevelopment in information retrieval, pages 480–487, New York, NY, USA. ACM.

126 / 133

IR Models



Bib IV

Fuhr, N. (1989).

Models for retrieval with probabilistic indexing.Information Processing and Management, 25(1):55–72.

Fuhr, N. (1992).

Probabilistic models in information retrieval.The Computer Journal, 35(3):243–255.

Fuhr, N. (2008).

A probability ranking principle for interactive information retrieval.Information Retrieval, 11:251–265.

He, B. and Ounis, I. (2005).

Term frequency normalisation tuning for BM25 and DFR models.In ECIR, pages 200–214.

Hiemstra, D. (2000).

A probabilistic justification for using tf.idf term weighting in information retrieval.International Journal on Digital Libraries, 3(2):131–139.

Kleinberg, J. (1999).

Authoritative sources in a hyperlinked environment.Journal of ACM, 46.

127 / 133

IR Models



Bib V

Lafferty, J. and Zhai, C. (2003).

Probabilistic Relevance Models Based on Document and Query Generation, chapter 1.In [Croft and Lafferty, 2003].

Lavrenko, V. and Croft, W. B. (2001).

Relevance-based language models.In SIGIR, pages 120–127. ACM.

Luk, R. W. P. (2008).

On event space and rank equivalence between probabilistic retrieval models.Inf. Retr., 11(6):539–561.

Margulis, E. (1992).

N-Poisson document modelling.In Belkin, N., Ingwersen, P., and Pejtersen, M., editors, Proceedings of the Fifteenth Annual InternationalACM SIGIR Conference on Research and Development in Information Retrieval, pages 177–189, New York.

Maron, M. and Kuhns, J. (1960).

On relevance, probabilistic indexing, and information retrieval.Journal of the ACM, 7:216–244.

Metzler, D. and Croft, W. B. (2004).

Combining the language model and inference network approaches to retrieval.Information Processing & Management, 40(5):735–750.

128 / 133

IR Models



Bib VI

Piwowarski, B., Frommholz, I., Lalmas, M., and Van Rijsbergen, K. (2010).

What can Quantum Theory Bring to Information Retrieval?In Proc. 19th International Conference on Information and Knowledge Management, pages 59–68.

Ponte, J. and Croft, W. (1998).

A language modeling approach to information retrieval.In Croft, W. B., Moffat, A., van Rijsbergen, C. J., Wilkinson, R., and Zobel, J., editors, Proceedings of the21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,pages 275–281, New York. ACM.

Robertson, S. (1977).

The probability ranking principle in IR.Journal of Documentation, 33:294–304.


Understanding inverse document frequency: On theoretical arguments for idf.Journal of Documentation, 60:503–520.


On event spaces and probabilistic models in information retrieval.Information Retrieval Journal, 8(2):319–329.

129 / 133

IR Models



Bib VII

Robertson, S., S. Walker, S. J., Hancock-Beaulieu, M., and Gatford, M. (1994).

Okapi at TREC-3.In Text REtrieval Conference.

Robertson, S. and Sparck-Jones, K. (1976).

Relevance weighting of search terms.Journal of the American Society for Information Science, 27:129–146.

Robertson, S. E. and Walker, S. (1994).

Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval.In [Croft and van Rijsbergen, 1994], pages 232–241.

Robertson, S. E., Walker, S., and Hancock-Beaulieu, M. (1995).

Large test collection experiments on an operational interactive system: Okapi at TREC.Information Processing and Management, 31:345–360.

Rocchio, J. (1971).

Relevance feedback in information retrieval.In [Salton, 1971].

Roelleke, T. (2003).

A frequency-based and a Poisson-based probability of being informative.In ACM SIGIR, pages 227–234, Toronto, Canada.

130 / 133

IR Models



Bib VIII

Roelleke, T. and Wang, J. (2006).

A parallel derivation of probabilistic information retrieval models.In ACM SIGIR, pages 107–114, Seattle, USA.

Roelleke, T. and Wang, J. (2008).

TF-IDF uncovered: A study of theories and probabilities.In ACM SIGIR, pages 435–442, Singapore.

Salton, G., editor (1971).

The SMART Retrieval System - Experiments in Automatic Document Processing.Prentice Hall, Englewood, Cliffs, New Jersey.

Salton, G., Fox, E., and Wu, H. (1983).

Extended Boolean information retrieval.Communications of the ACM, 26:1022–1036.

Salton, G., Wong, A., and Yang, C. (1975).

A vector space model for automatic indexing.Communications of the ACM, 18:613–620.

131 / 133

IR Models



Bib IX

Singhal, A., Buckley, C., and Mitra, M. (1996).

Pivoted document length normalisation.In Frei, H., Harmann, D., Schauble, P., and Wilkinson, R., editors, Proceedings of the 19th AnnualInternational ACM SIGIR Conference on Research and Development in Information Retrieval, pages 21–39,New York. ACM.

Sparck-Jones, K., Robertson, S., Hiemstra, D., and Zaragoza, H. (2003).

Language modelling and relevance.Language Modelling for Information Retrieval, pages 57–70.

Sparck-Jones, K., Walker, S., and Robertson, S. E. (2000).

A probabilistic model of information retrieval: development and comparative experiments: Part 1.Information Processing and Management, 26:779–808.

Turtle, H. and Croft, W. (1991).

Efficient probabilistic inference for text retrieval.In Proceedings RIAO 91, pages 644–661, Paris, France.

Turtle, H. and Croft, W. (1992).

A comparison of text retrieval models.The Computer Journal, 35.

132 / 133

IR Models



Bib X

Turtle, H. and Croft, W. B. (1990).

Inference networks for document retrieval.In Vidick, J.-L., editor, Proceedings of the 13th International Conference on Research and Development inInformation Retrieval, pages 1–24, New York. ACM.

van Rijsbergen, C. J. (1986).

A non-classical logic for information retrieval.The Computer Journal, 29(6):481–485.


Towards an information logic.In Belkin, N. and van Rijsbergen, C. J., editors, Proceedings of the Twelfth Annual International ACMSIGIR Conference on Research and Development in Information Retrieval, pages 77–86, New York.


The Geometry of Information Retrieval.Cambridge University Press, New York, NY, USA.

Wong, S. and Yao, Y. (1995).

On modeling information retrieval with probabilistic inference.ACM Transactions on Information Systems, 13(1):38–68.

Zaragoza, H., Hiemstra, D., Tipping, M. E., and Robertson, S. E. (2003).

Bayesian extension to the language model for ad hoc information retrieval.In ACM SIGIR, pages 4–9, Toronto, Canada.

133 / 133

Date post:	28-Nov-2014
Category:	Data & Analytics
Upload:	ingo-frommholz
View:	780 times
Download:	5 times

Information Retrieval Models Part I

Data & Analytics