Building a scalabledistributed

WWW search engine …

NOT in Perl! Presented by Alex Chudnovsky (http://www.majectic12.co.uk)

at Birmingham Perl Mongers User Group (http://birmingham.pm.org)

V1.0 27/07/05

Contents

1. History

2. Goals

3. Architecture

4. Implementation

5. Why not Perl?

6. Conclusions

7. Credits

8. Recommended reading

History(of my work in area of information retrieval)

1. First primitive pathetic stone-age search engine: 1000 documents in the “index” (1997, Perl)

2. Second engine using proper inverted indexing for Jungle.com: 500,000 products indexed (Perl + Java, 2002)

3. Current: 50,000,000 pages indexed with a lot more to go (to be revealed, 2005)

Goals

1. Build a distributed WWW search engine capable of dealing with at least 1 bln web pages based on principles of SETI@Home and D.NET

2. See to it that the chosen language for implementation (more on this later) fits purpose or more likely learn how to make it work

3. Eventually make some money out of it

Architecture

1. Data collection (crawling)

2. Indexing: turning text into numbers

3. Merging: turning indexed barrels into single searchable index

4. Searching: locating documents for given keywords

Data collection (crawling)

Base

Issues URLs to crawl and receives compressed pages

Distributed crawlers – receive lists of URLs to crawl, crawl them and send back compressed data.

In the future will do distributed indexing

Note: this stage is optional if you already have data to index, ie list of products with their descriptions

Crawler screenshot 1

Crawler screenshot 2

Crawler screenshot 3

Crawler screenshot 4

Crawler screenshot 5

Current Stats

Source: http://www.majestic12.co.uk/projects/dsearch/stats.php as of 27/07/05

Indexing

Indexing is a process of turning words into numbers and creating inverted index.

Data barrel

Doc #0: Birmingham Perl Mongers

Doc #1: Birmingham City

Doc #2: Perl City

Lexicon(maps words to their numeric WordIDs)

Birmingham – 0Perl – 1Mongers – 2City – 3

Inverted Index(Each of the WordID has list of

(ideally sorted) DocIDs)

0 -> 0, 11 -> 0, 22 -> 0,3 -> 1, 2

Note: if you use database then it make sense to have clustered index on WordID

Merging

Individual indexed barrels

Single searchable

index

Note: this stage is not necessary if just one barrel is used as there will be no need to remap all Ids from local to their global equivalents.

Searching

Searching is a process of finding documents that contain words from search query

Doc #0: Birmingham Perl Mongers

Doc #1: Birmingham City

Doc #2: Perl City

Lexicon(maps words to their numeric WordIDs)

Birmingham – 0Perl – 1Mongers – 2City – 3

Inverted Index(lists DocIDs for each of the WordID)

0 -> 0, 11 -> 0, 22 -> 0,3 -> 1, 2

Note: if you use database then it make sense to cluster on WordID

Search query: “Birmingham Perl”

WordIDs: 0, 1

Intersection of DocIDs present in both lists (implementation of boolean AND logic):

0 (Brum) 1 (Perl) Result

0 0 Matched!

1 n/a Not matched!

n/a 2 Not matched!

Search engine screenshot 1

Search engine screenshot 2

Implementation

1. Microsoft .NET C# ported to Linux using Mono (http://www.mono-project.com)

2. ~90k lines of code (minimal copy/paste) written from scratch

3. Low level of dependencies (SharpZipLib/SQLite/NPlot)

Why not Perl?(using C# instead)

1. Not strong in GUI department

2. Hard to deal with Multi-Threading and Asyncronous sockets

3. OOP is more of a hack

4. Lax compile-time checks due to not being strictly typed

5. Fear of performance bottlenecks forcing to use C++

6. Hard to profile for performance analysis

7. Managed memory lacks support for pointers (?)

8. Poor exceptions handling

9. I wanted something new :)

Conclusions

Still work in progress, but some conclusions can be made already:

1. Inverted indexing approach helps to achieve fast searches

2. Its tough to build one – don’t try if you ain’t going to see it through!

3. Crawler is one tough piece of code – 6 months vs 2 months on searching

4. .NET C# is a decent language suitable for heavy duty tasks like this

Credits

1. R&D: Alex Chudnovsky <alexc@majestic12.co.uk>

2. Pioneers*: FiddleAbout, dazza12, lazytom, Mordac, linuxbren, Cyber911, www.vanginkel.info, Vari, ASB, SEOBy.org, arni, japonicus, webstek.info | Pimpel, DimPrawn, Zyron, partys-bei-uns.de, jake, bull at webmasterworld, nada, dodgy4, sri-heinz

* Volunteers running crawler and who crawled at least 1 mln URLs as of 27/07/05

Recommended reading

1. “The Anatomy of a Large-Scale Hypertextual Web Search Engine” Sergey Brin and Lawrence Page of Google (http://www-db.stanford.edu/~backrub/google.html)

2. “Managing Gigabytes” Ian h. Witten et al ISBN 1-55860-570-3

Join!

Join the project (unmetered broadband required!):

majestic12.co.uk

Your name could be here!