Searching very large bodies of data using a transparent peer-to-peer proxy Mike Taylor and Marc...

transcript

Searching very large bodiesof data using a transparent

peer-to-peer proxyMike Taylor and Marc Cromme, Index Data

mike@indexdata.commarc@indexdata.dk

Albertosaurus sacophagus skull modified from Carr 1999.(Not relevant to the talk, but pretty.)

Overview

Where we're headed in the next half-hour:

The problemStandardised semantically rich search-and-retrieve protocols

ANSI/NISO Z39.50SRU

Transparent protocol proxiesFan-out proxies, singly and in combinationPeer-to-peer proxiesOperation of the peer-to-peer proxy networkUsing the peer-to-peer proxy in AlvisConclusions

Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>

The Problem

The key advantage of the Internet is distribution– That's why there is so much information out there.The key problem of the Internet is aggregation– That's why it's so darned hard to find anything!

How can we get at all that tasty data?

Monolithic systems can only get us so far.Even Google – with its huge index – is limited by its inability toprobe into the “deep web”. It is limited to dumb screen-scraping.

We propose a solution made up of many autonomous nodes.We will approach this in several steps.

Step 1: standardised search-and-retrieve protocols

Z39.50 client

Z39.50

Library of CongressZ39.50 server

British LibraryZ39.50 server

Z39.50 client

Z39.50

Z39.50 client

Z39.50

Local catalogueZ39.50 server

MetasearchingZ39.50 client

Z39.50

Z39.50 Z39.50

This is possible becauseof the semantic alignmentof the servers.

So can Z39.50 save the world?

Then the serpent saith unto Adam, “Lo, why doth thineinformation service not use XML?” And Adam saith, “Verily,Z39.50 worketh just fine.” But the serpent, who was subtleof tongue, saith unto him, “But XML is more fashionable.”And, behold, Adam was deceived, and did fall.

– The Book of Standards, ch. 3, v. 4-6.

Welcome to the 21st Century

Z39.50

Z39.50 Z39.50

Everything

must be XML

Welcome to the 21st Century

Z39.50

Z39.50 Z39.50Resistance

Resistance

is useless!

XML-based search-and-retrieve protocols

The binary Z39.50 protocol is superseded by SRU.(Search/Retrieve by Url). This is a NISO-registeredstandard for expressing queries using rich URLs, to obtainXML responses that contain records matching the query.

http://sru.miketaylor.org.uk/sru.pl?version=1.1&operation=searchRetrieve&query=dinosaur&startRecord=1&maximumRecords=1&recordSchema=dc

An SRU response (single DC record)

<?xml version="1.0"?><zs:searchRetrieveResponse xmlns:zs='http://www.loc.gov/zing/srw/'> <zs:version>1.1</zs:version> <zs:numberOfRecords>29</zs:numberOfRecords> <zs:records> <zs:record> <zs:recordSchema>info:srw/schema/1/dc-v1.1</zs:recordSchema> <zs:recordPacking>xml</zs:recordPacking> <zs:recordPosition>1</zs:recordPosition> <zs:recordData> <srw_dc:dc xmlns:srw_dc="info:srw/schema/1/dc-schema" xmlns="http://purl.org/dc/elements/1.1/"> <title>Fossils</title> <creator>Lappi, Megan.</creator> <type>text</type> <publisher>New York, NY: Weigl Publishers</publisher> <date>2005</date> <language>en</language> <description>Studying fossils -- Fossil facts -- Gone forever -- A fossil is born -- From bone to stone -- Insects in amber -- Dinosaur footprints</description> <identifier>http://www.loc.gov/catdir/toc/ecip0415/2004004136.html</identifier> <identifier>URN:ISBN:1590362136</identifier> </srw_dc:dc> </zs:recordData> </zs:record> </zs:records></zs:searchRetrieveResponse>

So we can go back to doing what we did before

Z39.50

Z39.50 Z39.50

Z39.50

Z39.50 Z39.50

Library of CongressSRU server

MetasearchingSRU client

British LibrarySRU server

Local catalogueSRU server

SRU SRU

SRU gives us the samesemantic alignmentas Z39.50.

SRU's query language: CQL

CQL (Common Query Language) is used by SRU.It may also be used in other contexts (including Z39.50).

Its syntax is easy to learn, but very expressive.

dinosaurtitle=dinosaurtitle=(dinosaur or pterosaur) and author=martilldc.title=*saur and dc.author=martilltitle exact "the complete dinosaur" and date < 2000name=/phonetic "smith"fish prox/distance<3/unit=sentence frog

2. Transparent protocol proxies

Just a Squid acts as a proxy for the dumb HTTP protocol, so we canhave proxies for semantically rich search-and-retrieve protocols.

YAZ Proxy is one such – http://indexdata.com/yazproxy

Because the protocol is rich, the proxy can do more than Squid:Performance improvements:

Cache and re-use intialised sessionsCache and re-use search resultsCache and re-use fetched records

Server protection:Query sanitisation (for broken servers ... you know who you are)Client throttling, based on request frequency or bandwidth

Protocol-level and application-level logging.

2. Transparent protocol proxies

SRU proxy

SRU client

3. Fan-out proxies – free metasearching for simple clients

MetasearchingSRU proxy

British LibrarySRU server

Local catalogueSRU server

SRU SRU

SRU client

The client knows nothingabout what is happeningto its innocent requests.

All the metasearchingintelligence goes here.

3. Cascading fan-out proxies

Server 1

Proxy 1

Proxy 2 Proxy 3

Client

Server 5 Server 6Server 3Server 2 Server 4

Hey, go nuts

Server

Proxy Proxy

Client

Server ProxyServerProxy Server

ServerServer Proxy

ServerProxy Proxy

ProxyServer Server

ServerServer Server

ServerServer Server Serveretc., etc., etc. ...

Why this doesn't actually work

Scaling problems!

Every proxy must be administrated:Information about searched resources kept up to dateProxies must be kept running– a single failure knocks out a whole subtree

Load on serversEvery server is visited by every queryWhat happens when a proxy calls another proxy higher up the tree?

Loop detection is difficult in protocols such as Z39.50 and SRU.

4. The peer-to-peer proxy

Another appropriateSRU server

Some appropriateSRU server

Really coolSRU server

SRU SRU

SRU client

Big cloud of peers, acting as a proxy

What's going on here?

Life is simple at the edges of the cloud:

SRU clients connect to peers that act as SRU serversSRU servers respond to requests from peers that act as SRU clients

This means that off-the-shelf SRU clients and servers can be used.Web-based SRU clients can be redeployedServers such as the Library of Congress catalogue are availableYou can use our free Z39.50/SRU-enabled XML database, Zebrahttp://indexdata.com/zebra

Although the cloud has its own structure, it is opaque to the clients andservers at the edge.

What's going on here?

Life is a little more complex within the cloud:

Peers must communicate between themselves using a P2P protocolPeers associated with a server must also make SRU requestsPeers associated with a client must also handle SRU requests

Each peer may act for a client, a server, both, or neither:“Client” peers are the entry-points into the P2P cloud“Server” peers actually get the job of searching done“Servent” peers behave as both clients and serversSome peers may participate in the network for routing purposes only.

Clearly the “servent” peer is the general case that all the other specialise.This only needs to be written once.

Why this rocks

The separation between the edges and the cloud is very important.

SRU clients and servers are easy to writeThere will be lots of them out there:

Clients providing many different user interfacesServers providing access to many different collections

“Servent” peers are difficult to writeBut that's OK, because only one such peer need ever be builtMany instances make up a single peer-to-peer proxy cloudThe cloud can be used by many clients, and can use many servers

Client and server writers don't have to think about the hard stuff.

5. Operation of the peer-to-peer proxy network

What goes on inside the mysterious cloud?

There is a dedicated peer-to-peer protocol used to:Introduce a new peer to the networkWelcome a new peer with an initial list of neighboursPass queries in a chain between peersReturn search results back along the chain

We won't cover the protocol in detail here, but:Many P2P protocols have “Ping” and “Pong” messages for introductionsIn ours, we have different messages:

New peers cry “Cathy!” when they nuzzle up the networkExisting peers respond with a cry of “Heathcliff!”

Everybody needs good neighbours

Key principle: the network contains NO global information.(So there is no single point of failure.)

Each peer knows only about a few “nearby” peers – its “neighbours”Old neighbours are dropped from the pool if they don't prove usefulNew neighbours are discovered in search responses:

Peer A forwards a query to its neighbour Peer BPeer B can't answer it, so it forwards it to its neighbour, Peer XPeer X responds with useful informationPeer A accepts this response, and remembers Peer X for next time.

The “usefulness” of peers may be judged relative to specific subject areasrather than with a single absolute score.

SRU client

Peer A

SRU client

Peer BPeer ACan you helpme with this?

SRU client

Peer B Peer XPeer ACan you helpme with this?

No, but I'llask my friend.

Dinosaur DataSRU server

SRU client

Here you go, pal.

SRU client

Here you go, pal.

(I'll remember that.)

Why this rocks, part II

The peer-to-peer network has the following desirable properties:

joinable – easy for new peers to join the networkadaptive – the system evolves to improve through timeautonomous – each peer can have its own strategies and policiesrobust – can cope seamlessly with holes appearing in the networkefficient – generates minimal network traffictunable – has parameters that we can play with“ecologically diverse” – different peers may be tuned differently

Most of these properties are related to the key issue:NO GLOBAL KNOWLEDGE!

Life history of a query

Queries can't be allowed to wander around the network forever.Each query begins with a certain lifespan.Each peer that accepts a query decrements its lifespan by one.The remaining lifespan travels with the query to other peers.

If a query is passed to multiple peers, the lifespan is divided between them:It might be divided equally between two relevant neighboursIt might be split between many neighboursIt might be allocated to a single promising neighbour

When a query's lifespan is expired, the peer may not propagate it further.

Ecological diversity

Different peers will have different strategies for propagating queries.Some will tend to fan out in a broad but shallow patternSome will produce tend to pass almost all lifespan to a single neighbour.Some will behave differently depending on the query

We hope that diversity of peer strategies will help make the network robust.

A query may carry with it a hint about how it likes its lifetime to be spent.“Tracer bullet” queries have a long, thin trajectory, then fan out.This is a useful way to periodically probe remote parts of the network,

in order to discover new and relevant neighbours.

6. Using the peer-to-peer proxy in Alvis

Alvis is an ongoing European collaborative project to build what the proposaldocument calls a “Superpeer semantic search engine”.

Named after the dwarf Alvis (“all-wise”) from Norse mythology, who answeredThor's questions all night ... (And then turned to stone when the sun rose.)

The Alvis “superpeers” are what we just call “peers” in this presentation.

This is because Alvis also has another whole layer of peers. These implementa distributed hash table (DHT) of individual keys to find a suitable entry-pointto the superpeer network.

Testing will show us how much this optimisation buys us.

7. Conclusions

Standardised search-and-retrieve protocols facilitate interoperability.A well-defined protocol can be proxied.Proxies may transparently perform many different services.They may perform metasearching (“fan-out proxy”).Metasearching proxies may be cascaded.In practice, such cascades are hard to maintain, and scale poorly.Instead, an entire cloud of peers may function as a proxy.Queries are routed through the cloud to reach the most appropriate servers.Neither clients nor servers need know anything at all about the proxy.Tunable parameters allow us to tweak performance.The European project Alvis is built on such a peer-to-peer proxy.We want to see this kind of network running “in the wild” with many nodes.

Thanks for listening!

Mike Taylor and Marc Cromme, Index Datamike@indexdata.commarc@indexdata.dk

Albertosaurus sacophagus skull modified from Carr 1999.(We should all take the time to look at more dinosaurs.)

Searching very large bodies of data using a transparent peer-to-peer proxy Mike Taylor and Marc...

Documents