Information Retrieval

Liam Quin,

Barefoot Computing,

Toronto

Agenda

• Overview of Information Retrieval• What people want, and how to give

it to them• Things people don’t know they

want, and how to do them

Chapter One: The Problem

gooseberry

Gooseberry Picking Hurts

Gooseberries have thorns.

Gooseberry pickers in Botswana might not wear shirts (or shoes).

When you pick one gooseberry, others fall to the ground.

The harvest would be improved if we could retrieve the fallen fruit safely.

There are texts on this on the Internet.

Searching for an Answer

• Search for “information on texts about gooseberry retrieval” on the web.

The Result

• Pages on text retrieval. . .• . . . on information retrieval. . .• . . .and, at the top of the list. . .

Cycling in Cape Gooseberry, Labrador

Lessons

• Indexes on words alone aren’t enough

• Word order can be important• Relevance Ranking is often bogus• Sometimes you have to wear shirt

and shoes.

How can we Improve?

• Better textual analysis• Word order• Context• Metadata• External categorisation (RDF, Topic

Maps)• Grow thornless gooseberries

Better Textual Analysis• Part of speech information during

indexing• Stemming (boy/boys, foot/feet,

run/running, me/mine)• Record more in index (caps, separation)• Co-location analysis (mine next to gold)• Ask the User what she means in the

query (mine as in of me, or as in quarry?)• Thesaurus Expansion of queries

Word Order

• Give added weight to word order:• information retrieval vs. retrieval

of information• Times Square vs. square times• include all words (What If Inc., The

Times)

Context

• Co-location of words helps disambiguate

• The xml containing element• Feedback from nearby documents

(e.g. on the same website, or in the same chapter or publication)

• Domain-specific information at index-time

Metadata

• Add information to documents• Dublin Core (e.g. Warwick

Framework)• The html meta element• The html rel/rev attributes in links

External Categorisation

• Use xml schemas to add context information

• Document or site-wide information• Resource Description Framework• Topic Maps (iso 13250)• Categorise the result set [see

picture]

Grow thornless gooseberries• Sometimes it’s easier to change

the problem than to solve it as stated.

• Sometimes people don’t describe the problem that they need solved.

• Sometimes it’s easier to solve a more general problem (thornless fruit? Or padded shirts)

What Most People Want

• Find this string or phrase in this element.

• That’s all most people ask for.• It’s all they want.• But it’s hardly ever all they need.

The real needs

• Needs of other staff• Executives who understand the

problem• Indirect needs

– internal use by software– other departments– private uses by sneaky employees– enabling technologies change

perspectives

I didn’t know I could...

• Quality control– check for known errors– find unusual words or phrases– phrases not marked up

• Analysis– look for unusual markup– co-location (phrase summary)

Oooh, can you really?

• Automatic linking– Glossary– Glossary Index page– Dictionary Samples

• Add markup automatically– based on phrases in context

Summery Summary

• You may need more than you thought…

• …but it might do more than you expected…

• …but...

There is no gooseberry pie for lunch.

Liam QuinBarefoot ComputingToronto

http://www.valinor.sorcery.net/~liam/

liam@holoweb.net