1

Terms and Query Operations

Information Retrieval: Data Structures and Algorithms

by W.B. Frakes and R. Baeza-Yates (Eds.) Englewood Cliffs, NJ: Prentice Hall, 1992.

Chapter 7 - 9

2

Lexical Analysis and Stoplists

Chapter 7

3

Lexical Analysis for Automatic Indexing

Lexical AnalysisConvert an input stream of characters into stream words or token.

What is a word or a token? Tokens consist of letters.

» Digits: Most numbers are not good index terms.counterexamples: case numbers in a legal database, “B6” and “B12” in vitamin database.

» Hyphens– break hyphenated words: state-of-the-art, state of the art

– keep hyphenated words as a token: “Jean-Claude”, “F-16”

» Other punctuation: often used as parts of terms, e.g., OS/2

» Case: usually not significant in index terms

4

Lexical Analysis for Automatic Indexing(Continued)

Issues: recall and precision» breaking up hyphenated terms

increase recall but decrease precision» preserving case distinctions

enhance precision but decrease recall» commercial information systems

usually take a conservative (recall enhancing) approach

5

Lexical Analysis for Query Processing

Tasks» depend on the design strategies of the lexical analyzer for

automatic indexing (search terms must match index terms)

» distinguish operators like Boolean operators» distinguish grouping indicators like parentheses and

brackets» flag illegal characters as unrecognized tokens

6

STOPLISTS (negative dictionary)

Avoid retrieving almost every item in a database regardless of its relevance.

Example (derived from Brown corpus): 425 wordsa, about, above, across, after, again, against, all, almost, alone, along, already, also, although, always, among, an, and, another, any, anybody, anyone, anything, anywhere, are, area, areas, around, as, ask, asked, asking, asks, at, away, b, back, backed, backing, backs, be, because, became, …

Commercial Information systems tend to take a conservative approach, with few stopwords

7

Implementing Stoplists

Approaches» examine lexical analyzer output and remove any stopwords» remove stopwords as part of lexical analysis

8

Stemming Algorithms

Chapter 8

9

Stemmers

Programs that relate morphologically similar indexing and search terms

Stem at indexing time» advantage: efficiency and index file compression

» disadvantage: information about the full terms is lost Example (CATALOG system), stem at search time

Look for: system usersSearch Term: users Term Occurrences

1. user 152. users 13. used 34. using 2

10

Conflation Methods

Manual Automatic (stemmers)

» table lookup

» successor variety

» n-gram

» affix removallongest match vs. simple removal

Evaluation» correctness

» retrieval effectiveness

» compression performance

11

Successor Variety

Definition (successor variety of a string)the number of different characters that follow it in words in some body of text

Examplea body of text: able, axle, accident, ape, aboutsuccessor variety of apple1st: 4 (b, x, c, p)2nd: (e)

12

Successor Variety (Continued)

IdeaThe successor variety of substrings of a term will decrease as more characters are added until a segment boundary is reached, i.e., the successor variety will sharply increase.

ExampleTest word: READABLECorpus:ABLE, BEATABLE, FIXABLE, READ,

READABLE, READING, RED, ROPE, RIPEPrefix Successor Variety LettersR 3 E, O, IRE 2 A, DREA 1 DREAD 3 A, I, SREADA 1 BREADAB 1 LREADABL 1 EREADABLE 1 blank

13

The successor variety stemming process

Determine the successor variety for a word. Use this information to segment the word.

» cutoff methoda boundary is identified whenever the cutoff value is reached

» peak and plateau methoda character whose successor variety exceeds that of the character immediately preceding it and the character immediately following it

» complete word methoda segment is a complete word

» entropy method

Select one of the segments as the stem.

14

n-gram stemmers

Diagrama pair of consecutive letters

Shared diagram method (Adamson and Boreham, 1974)association measures are calculated between pairs of terms

where A: the number of unique diagrams in the first word, B: the number of unique diagrams in the second, C: the number of unique diagrams shared by A and B

SC

A B

2

15

n-gram stemmers (Continued)

Examplestatistics => st ta at ti is st ti ic cs

unique diagrams => at cs ic is st ta tistatistical => st ta at ti is st ti ic ca al

unique diagrams => al at ca ic is st ta ti

SC

A B

2 2 6

7 80 80

*.

16

n-gram stemmers (Continued)

similarity matrixdetermine the semantic measures for all pairs of terms in the database

word1 word2 word3 ... wordn-1

word1

word2 S21

word3 S31 S32

.

.Wordn Sn1 Sn2 Sn3 … Sn(n-1)

terms are clustered using a single link clustering method

» most pairwise similarity measures were 0

» using a cutoff similarity value of .6

17

Affix Removal Stemmers

ProcedureRemove suffixes and/or prefixes from terms leaving a stem, and transform the resultant stem.

Example: plural formsIf a word ends in “ies” but not “eies” or “aies”

then “ies” --> “y”If a word ends in “es” but not “aes”, “ees”, or “oes”

then “es” --> “e”If a word ends in “s”, but not “us” or “ss”

then “s” --> NULL

Ambiguity

18

Affix Removal Stemmers (Continued)

Iterative longest match stemmerremove the longest possible string of characters from a word according to a set of rules» recoding: AxC--> AyC, e.g., ki --> ky» partial matching: only n initial characters of stems are used

in comparing Different versions

Lovins, Slaton, Dawson, Porter, …Students can refer to the rules listed in the text book.

19

Thesaurus Constructions

Chapter 9

20

Thesaurus Construction

IR thesaurusa list of terms (words or phrases) along with relationships among them physics, EE, electronics, computer and control

INSPEC thesaurus (1979)

cesium (銫， Cs) USE caesium (USE: the preferred form)

computer-aided instructionsee also education (cross-referenced terms) UF teaching machines (UF: a set of alternatives)BT educational computing (BT: broader terms, cf. NT) TT computer applications (TT: root node/top term)RT education (RT: related terms) teachingCC C7810C (CC: subject area)FC C7810Cf (subject area)

21

Usage

IndexingSelect the most appropriate thesaurus entries for representing the document.

SearchingDesign the most appropriate search strategy.

» If the search does not retrieve enough documents, the thesaurus can be used to expand the query.

» If the search retrieves too many items, the thesaurus can suggest more specific search vocabulary.

22

Features of Thesauri (1/5)

Coordination Level» the construction of phrases from individual terms

» pre-coordination: contains phrases– phrases are available for indexing and retrieval– advantage: reducing ambiguity in indexing and searching– disadvantage: searcher has to be know the phrase formulation rules– lower recall

» post-coordination: does not allow phrases– phrases are constructed while searching– advantage: users do not worry about the exact word ordering – disadvantage: the search precision may fall, e.g.,

library school vs. school library– lower precision

23

Features of Thesauri (2/5)

» intermediate level: allows both phrases and single words– the higher the level of coordination, the greater the precision of the

vocabulary but the larger the vocabulary size– it also implies an increase in the number of relationships to be encoded

Precoordination is more common in manually constructed thesauri.

Automatic phrase construction is still quite difficult and therefore automatic thesaurus construction usually implies post-coordination

24

Features of Thesauri (3/5)

Term Relationships» Aitchison and Gilchrist (1972)

– equivalence relationships: synonymy or quasi-synonymy

– hierarchical relationships, e.g., genus (屬 )-species(種 )

– nonhierarchical relationships, e.g., thing-part, bus and seat e.g., thing-attribute, rose and fragrance

» Wang, Vandendorpe, and Evens (1985)– parts-wholes, e.g., set-element, count-mass

– collocation relations: words that frequently co-occur in the same phrase or sentence

– paradigmatic relations (詞形變化 ): e.g., “moon” and “lunar”

– taxonomy and synonymy

– antonymy relations

25

Features of Thesauri (4/5)

Number of entries for each term» homographs: words with multiple meanings

» each homograph entry is associated with its own set of relations

» problem: how to select between alternative meanings

» typically the user has to select between alternative meanings Specificity of vocabulary

» is a function of the precision associated with the component terms

» disadvantage: the size of the vocabulary grows since a large number of terms are required to cover the concepts in the domain

» high specificity implies a high coordination level

» a highly specific vocabulary promotes precision in retrieval

26

Features of Thesauri (5/5)

Control on term frequency of class members» for statistical thesaurus construction methods

» terms included in the same thesaurus class have roughly equal frequencies

» the total frequency in each class should also be roughly similar Normalization of vocabulary

» Normalization of vocabulary terms is given considerable emphasis in manual thesauri

» terms should be in noun form

» noun phrases should avoid prepositions unless they are commonly known

» a limited number of adjectives should be used

» ...

27

Thesaurus Construction

Manual thesaurus construction» define the boundaries of the subject area» collect the terms for each subarea

sources: indexes, encyclopedias, handbooks, textbooks, journal titles and abstracts, catalogues, ...

» organize the terms and their relationship into structures» review (and refine) the entire thesaurus for consistency

Automatic thesaurus construction» from a collection document items» by merging existing thesaurus

28

1. Construction of vocabulary normalization and selection of terms phrase construction depending on the coordination level desired

2. Similarity computations between terms identify the significant statistical associations between terms

3. Organization of vocabulary organize the selected vocabulary into a hierarchy on the basis of the associations computed in step 2.

Thesaurus Construction from Texts

29

Construction of Vocabulary

Objectiveidentify the most informative terms (words and phrases)

Procedure(1) Identify an appropriate document collection. The document collection should be sizable and representative of the subject area.(2) Determine the required specificity for the thesaurus.(3) Normalize the vocabulary terms. (a) Eliminate very trivial words such as prepositions and conjunctions. (b) Stem the vocabulary. (4) Select the most interesting stems, and create interesting phrases for a higher coordination level.

30

Stem evaluation and selection

Selection by frequency of occurrence» each term may belong to category of high, medium or low

frequency» terms in the mid-frequency range are the best for indexing

and searching

31

Stem evaluation and selection (Continued)

selection by discrimination value (DV)» the more discriminating a term, the higher its value as an

index term» procedure

– compute the average inter-document similarity in the collection

– Remove the term K from the indexing vocabulary, and recompute the average similarity

– DV(K)=(average similarity without K)-(average similarity with k)

– The DV for good discriminators is positive.

32

Phrase Construction

Salton and McGill procedure1. Compute pairwise co-occurrence for high-frequency words.2. If this co-occurrence is lower than a threshold, then do not consider the pair any further.3. For pairs that qualify, compute the cohesion value. COHESION(ti, tj)=

co-occurrence-frequency/(sqrt(frequency(t i)*frequency(tj)))

COHESION(ti, tj)=size-factor* co-occurrence-frequency/(frequency(ti)*frequency(tj))

where size-factor is the size of thesaurus vocabulary 4. If cohesion is above a second threshold, retain the phrase

33

Phrase Construction (Continued)

Choueka Procedure1. Select the range of length allowed for each collocational expression. E.g., 2-6 wsords2. Build a list of all potential expressions from the collection with the prescribed length that have a minimum frequency.3. Delete sequences that begin or end with a trivial word (e.g., prepositions, pronouns, articles, conjunctions, etc.)

4. Delete expressions that contain high-frequency nontrivial words.5. Given an expression, evaluate any potential sub-expressions for relevance. Discard any that are not sufficiently relevant.6. Try to merge smaller expressions into larger and more meaningful ones.

34

Term-Phrase Formation

Term Phrasea sequence of related text words carry a more specific meaning than the single termse.g., “computer science” vs. computer;

DocumentFrequency

N

Thesaurustransformation

Phrasetransformation

35

Similarity Computation

Cosinecompute the number of documents associated with both terms divided by the square root of the product of the number of documents associated with the first term and the number of documents associated with the second term.

Dicecompute the number of documents associated with both terms divided by the sum of the number of documents associated with one term and the number associated with the other.

36

Vocabulary Organization

Clustering Forsyth and Rada (1986)

» Assumptions: » (1) high-frequency words have broad meaning, while low-

frequency words have narrow meaning. » (2) if the density functions of two terms have the same

shape, then the two words have similar meaning.

1. Identify a set of frequency ranges.2. Group the vocabulary terms into different classes based on their frequencies and the ranges selected in step 1.3. The highest frequency class is assigned level 0, the next, level 1, and so on.

37

Forsyth and Rada (cont.)

4. Parent-child links are determined between adjacent levels as follows. For each term t in level i, compute similarity between t and every term in level i-1. Term t becomes the child of the most similar term in level i-1. If more than one term in level i-1qualifies for this, then each becomes a parent of t. In other words, a term is allowed to have multiple parents.

5. After all terms in level i have been linked to level i-1 terms,

check level i-1terms and identify those that have no children.

Propagate such terms to level i by creating an identical

“dummy” term as its child.

6. Perform steps 4 and 5 for each level starting with level.

38

Merging Existing Thesauri

simple mergelink hierarchies wherever they have terms in common

complex merge» link terms from different hierarchies if they are similar

enough.» similarity is a function of the number of parent and child

terms in common