•Basic programming skills (preferably JAVA)•Familiarity with the web (HTML, browsing, etc.)•Fundamentals of Web Programming (20-753).
Grading30% homeworks (2 programming assignments)30% miniproject (student teams will propose)15% midterm (5 pages notes, calculator OK, no laptops)25% final (10 pages notes, calculator OK, no laptops)
Bulletin BoardSchedule/syllabusLecture notes (in powerpoint)HomeworkAnnouncements & discussions
Textbook and Reference Materials (1)
Required: Class notes (slides on web site)and handouts (to be provided)
Required: "Understanding Search Engines:Mathematical Modeling and Text Retrieval"by Michael W. Berry, Murray BrowneAvailable at http://www.siam.org (tel: 1-800-447-7426)
Optional: Background reading material provided
Textbook and Reference Materials (2)Optional: "Advances in Information Retrieval" Edited
by Croft, Kluwer Academic Pub., 2000 [more detailed state-of-the-art IR book]
Optional: "Machine Learning" by Tom M. Mitchell,WCB McGraw-Hill [Tools for textcategorization and data mining.]
Information Retrieval: The Challenge (1)
Text DB includes:(1) Rainfall measurements in the Sahara continue to show a steadydecline starting from the first measurements in 1961. In 1996 only12mm of rain were recorded in upper Sudan, and 1mm in SouthernAlgiers...
(2) Dan Marino states that professional football risks loosing the numberone position in heart of fans across this land. Declines in TV audienceratings are cited...
(3) Alarming reductions in precipitation in desert regions are blamed fordesert encroachment of previously fertile farmland in Northern Africa.Scientists measured both yearly precipitation and groundwater levels...
Information Retrieval: The Challenge (2)
User query states:"Decline in rainfall and impact on farms near Sahara"
Challenges•How to retrieve (1) and (3) and not (2)?•How to rank (3) as best?•How to cope with no shared words?
Information Retrieval in eCommerce (1)
Bringing in CustomersHow do Web-search engines work?
How to maximize hits on my eCommerce pages?
How to maximize preselection of customers who willtransact?
Information Retrieval in eCommerce (2)
Analyzing the Competition•How do we find the competition?•How will customers find the competition?•Can we do preemptive information strikes?
Text Mining•How to learn what customers want most?•How to find out what they missed, but wanted?•How to discover customer search/browsingpatterns?
Information Retrieval Assumption (1)
Basic IR task•There exists a document collection {Dj }
•Users enters at hoc query Q
•Q correctly states user’s interest
•User wants {Di } < {Dj } most relevant to Q
"Shared Bag of Words" assumptionEvery query = {wi }Every document = {wk }...where wi & wk in same Σ
All syntax is irrelevant (e.g. word order)All document structure is irrelevantAll meta-information is irrelevant(e.g. author, source, genre)=> Words suffice for relevance assessment
Information Retrieval Assumption (2)
Information Retrieval Assumption (3)
Retrieval by shared words
If Q and Dj share some wi , then Relevant(Q, Dj )
If Q and Dj share all wi , then Relevant(Q, Dj )
If Q and Dj share over K% of wi , then Relevant(Q, Dj)
Boolean Queries (1)Industrial use of SilverQ: silverR: "The Count’s silver anniversary..."
"Even the crash of ’87 had a silver lining...""The Lone Ranger lived on in syndication...""Sliver dropped to a new low in London..."...
Q: silver AND photographyR: "Posters of Tonto and the Lone Ranger..."
"The Queen’s Silver Anniversary photos..."...
Boolean Queries (2)
Q: (silver AND (NOT anniversary)AND (NOT lining)AND emulsion)
OR (AgI AND crystalAND photography))
R: "Silver Iodide Crystals in Photography...""The emulsion was worth its weight in
silver..."...
Boolean Queries (3)
Boolean queries are:
a) easy to implement
b) confusing to compose
c) seldom used (except by librarians)
d) prone to low recall
e) all of the above
Beyond the Boolean Boondoggle (1)
Desiderata (1)
•Query must be natural for all users
•Sentence, phrase, or word(s)
•No AND’s, OR’s, NOT’s, ...
•No parentheses (no structure)
•System focus on important words
•Q: I want laser printers now
Beyond the Boolean Boondoggle (2)Desiderata (2)
• Find what I mean, not just what I say Q: cheap car insurance(pAND (pOR
• Searching Document Archives– Libraries (title, subject, full-text)– Data bases of patents and applications– DBs of legal cases (e.g. Lexis, Westlaw)
• Searching the Web– Pure search engines (Google, Inktomi, …)– Browsing + Search (Yahoo, Terra-Lycos, …)– Meta-search (Metacrawler, Vivisimo, …)
• Corporate or Government Intranets• Non-traditional (e.g. Software DBs, News)
INFORMATION RETRIEVAL (IR) EVOLUTION
• IR in the 1980s:– Single collection with < 106 documents (archive)– Boolean queries with unordered-set answer
• IR circa 2000:– Single collection with > 109 documents (web)– Free-form queries with ranked-list answer
• IR circa 2010:– Multiple collections > 1012 docs (invisible web)– “Find what I mean” queries with clustering,
summarization and customization.
Content for Rest of the Course (1)[See the course BB for the latest updates to the
course schedule.]
Under the Hood•The vector space model for retrieval•Building an inverted index•Term weighting and selection•Web spidering•Automated text categorization
Content for Rest of the Course (2)IR Uses in eCommerce
•How to make search engine work for you•How to build optimal search-attractive web sites•The business(es) of web-based information
Beyond Web Search Engines•Speech processing primer•Information extraction from web pages•Data mining primer•Multi-media applications•Business models
Optional Quick Review of Linear AlgebraIf you know n-dimensional vectors, matrices, computing
inner products, etc.., Then you do not need this review. You may take a break.
If you learned this material, but do not remember it, please stay and listen to refresh your knowledge.
If you never learned linear algebra, stay, listen and (optionally) read either:
• G. Hadley. Linear Algebra. Addison-Wesley, 1961. Ch 3.
• Or, Stephen W. Goode. An Introduction to Differential Equations and Linear Algebra. Prentice Hall, 1991. Ch.3).
![Under The Hood [Part II] Web-Based Information Architectures MSEC 20-760 Mini II Jaime Carbonell.](https://static.documents.pub/doc/80x56/56649d5d5503460f94a3c9f9/under-the-hood-part-ii-web-based-information-architectures-msec-20-760-mini.jpg)
