Intelligent peer networks for collaborative Web search

Filippo Menczer and Le-Shin Wu and Ruj AkavipatSchool of Informatics

Indiana University, Bloomington, IN 47408{fil,lewu,rakavipa}@indiana.edu

Abstract

Collaborative query routing is a new paradigm forWeb search that treats both established search en-gines and other publicly available indices as intelli-gent peer agents in a search network. The approachmakes it transparent for anyone to build their own(micro) search engine, by integrating establishedWeb search services, desktop search, and topicalcrawling techniques. The challenge in this model isthat each of these agents must learn about its envi-ronment — the existence, knowledge, diversity, re-liability, and trustworthiness of other agents — byanalyzing the queries received from and results ex-changed with these other agents. We present the 6Speer network, which uses machine learning tech-niques to learn about the changing query environ-ment. We show that simple reinforcement learningalgorithms are sufficient to detect and exploit se-mantic locality in the network, resulting in efficientrouting and high-quality search results. A proto-type of 6S is available for public use and is intendedto assist in the evaluation of different AI techniquesemployed by the networked agents.

IntroductionCentralized search engines cannot cover the en-tire Web (Lawrence & Giles 1999) because it istoo large, fast-growing and fast-changing (Brew-ington & Cybenko 2000; Fetterly et al. 2003;Ntoulas, Cho, & Olston 2004). As a result, cur-rent centralized search engine focus on “important”portions of the Web. However, the notion of impor-tance is highly subjective: the biases that are intro-duced to address the needs of the “average” usercan result in diminished effectiveness in satisfyingmany atypical search needs. Therefore, the “oneengine fits all” model cannot handle the increas-ing size, rate of change, and heterogeneity of theWeb and its users. In addition, as search becomesmore prevalent at the desktop level, users will in-creasingly want to make subsets of the files indexed

Copyright c© 2008, Association for the Advancementof Artificial Intelligence (www.aaai.org). All rights re-served.

in their computers available to others via the Inter-net. Peer networks provide us with an architecturefor extending Web search technology to capture thecontextual needs of a diverse population of users,while leveraging their resources.

There are several models of peer network topolo-gies and query protocols, including structured, un-structured, flooding, distributed hash tables, andhierarchical (Androutsellis-Theotokis & Spinellis2004). Our design of a collaborative Web searchnetwork is guided by the principle of semantic lo-cality: peers with shared interests are likely to com-municate with each other more frequently than un-related agents, so they should be able to reach eachother in a few virtual hops. However, a dense net-work would generate too much traffic. A goodtopology favors both effectiveness and efficiency,by making it possible for a query to reach a relevanttarget peer in few steps, without imposing a largetraffic load on the entire network. Small-world net-works (Watts & Strogatz 1998) provide both clus-tered communities and enough randomness to keepthe network distance small between any two peers.Effective search requires that the clusters be associ-ated with a high semantic similarity between neigh-bors (Watts, Dodds, & Newman 2002). Becausethere is no global knowledge of the network (whatpeers are currently present, what information theyhold, and what information they seek), and the net-work is very dynamic (peers may join and leave thenetwork at any time), we cannot impose semanticlocality into the network by design; instead, we ex-plore AI techniques through which semantic local-ity will emerge as the result of local interactionsand learning by individual peer agents.

Our research group is currently developing 6S,an intelligent multi-agent application for peer-based Web search (Wu, Akavipat, & Menczer2005; Akavipat et al. 2006). The name is a contrac-tion of “six degrees of separation” and “search,” toreflect the social network of peer agents at the baseof the collaborative search process. Each 6S peeragent is both a (limited) directory hub and a con-tent provider; it has its own topical crawler (based

Figure 1: 6S search mechanism and peer discov-ery. 6S is designed not to have peers aggressivelyflooding the network for searching or discoveringnew peers. Therefore, the 6S peer only forwardsits query to a small number of selected neighbors.A time-to-live mechanism ensures that a forwardedquery will not survive in the network too long.Here Alice’s agent A receives good results fromagent C for Query 1. These results are forwardedthrough B. Later, A can send Query 2 directly tothe newly discovered neighbor C.

on local context), which supports a local searchengine—typically but not necessarily a small one.As shown in Figure 1, queries are first matchedagainst the local engine, and then routed to neigh-bors to obtain more results. While receiving re-sponses, an agent may discover new peers throughits current neighbors. The new neighbor peers canlater contact each other directly.

Figure 2 compares the collaborative search net-work framework with existing search models. Twomajor features we want to merge are contextuallearning (as in intelligent Web agents) and socialcollaboration (as in file-sharing peer networks). In-telligent Web agents leverage local context fromboth the user and the information environmentwhile learning to perform their tasks. Similarly, 6Sagents use the local context captured from the userand from interactions with other peers, as they learnto route queries to the most appropriate neighbors.The local user context of a 6S agent is a documentcollection created by the user.

With respect to social collaboration, 6S agentsuse a network to share information via queries andresponses, as do nodes in a P2P network. With-out relying on a centralized resource collection,our search model emulates the information find-ing and spreading mechanisms in social networks.However, powerful central search engines such asGoogle and Yahoo can very well contribute to andprofit from the social collaborative framework; in-

Figure 2: Two dimensions of search systems: thedegree of social collaboration—as, for example,among networked agents—and the degree of usercontextual learning as in intelligent Web agentssuch as InfoSpiders (Menczer & Belew 2000), fo-cused crawlers (Chakrabarti, van den Berg, & Dom1999), and other topical crawlers.

deed we expect that they would quickly turn intopopular hubs thanks to their large collections andpopularity-driven ranking algorithms. Therefore,our model of collaborative Web search allows usersto integrate both centralized and social search en-gines transparently. As in file-sharing networks,the incentive for people to collaborate is selfish —they can profit by participating in the network asthey gain access to additional sources tailored totheir needs.

Implementation and Deployment of 6S6S is designed to make it easy and transparent forusers to index and share a collection of Web pages,i.e., to build a “micro search engine.” A 6S ser-vent (server+client) application integrates a topicalcrawler, a document indexing system, a retrievalengine, a P2P network communication system, anda contextual learning system. In the current im-plementation, 6S relies on two open-source plat-forms: Nutch (nutch.org) for its search engineand JXTA (Waterhouse 2001) for the P2P networkcommunication framework.

From the user’s perspective, the main featuresof 6S are peer search, a personal Web index man-agement system, and a browser extension. Thepeer search functionality is an extension of localsearch. Local search is performed using the built-in search engine to provide users with relevant re-sults from their local collections. Next, the ap-plication automatically selects neighbors that are

best suited to answer the user’s query based on thepeer’s prior query-response experience, and sendsthe query to those peers. (Using the same mech-anism, those neighbors forward the query to otherpeers, and so on; cf. Figure 1.) Finally, the resultsobtained locally and from other peers’ responsesfor the same query are combined to remove du-plicates, re-ranked based on a simple voting algo-rithm, and then presented to the user. Results areupdated dynamically as they arrive.

Behind the scenes, the application analyzes theresults received from other peers, comparing themto the local search results, to learn a representationof the other peers. This representation is then usedto improve the peer selection algorithm, which isat the heart of the query-routing process. The de-tail of the machine learning algorithm used for thispurpose are discussed below. The more a user em-ploys the peer search network, the more she trainsthe system to better locate relevant information inthe future.

The personal Web index management systemhelps a user automatically create a Web index.In fully automated (one-click) mode, the appli-cation selects pages from a local bookmark fileand supplements them with results from a topicalWeb crawler. Consider for example a user Alice.The application analyzes the queries in her Websearch history to construct a topic description, thenlaunches the crawler. This process takes place thefirst time Alice sets up her peer, if she so chooses.Subsequently, 6S periodically updates the indexwith new additions from the Alice’s bookmarks orwith a new topical Web crawl based on her recentsearch history and current Web index.

As shown in Figure 3, the index managementfeature also allows users to manually create oradd to the personal index, or to launch crawlerswith starting seeds and topics of choice. The cur-rent implementation employs a best-N-first topi-cal crawler, which has been proven both efficientand effective for supporting a dynamic search en-gine among a number of crawling algorithms (Pant,Bradshaw, & Menczer 2003; Menczer, Pant, &Srinivasan 2004). Briefly, the crawler is given aset of topic keywords that is either entered by theuser or extracted from the user’s Web search his-tory, and a number of seed pages that are obtainedfrom the user’s personal bookmarks and/or the lo-cal document collection. The URLs to be vis-ited are prioritized by the similarity between thetopic and the page in which a URL is encountered.Some additional mechanisms guarantee that thecrawler is sufficiently exploratory. This crawler ispublicly available (informatics.indiana.edu/fil/IS/JavaCrawlers). Once the in-dex is built, the user can manage (tag, modify,delete, or recover) any indexed documents. For ex-ample Alice may index the documents in a review

folder, and provide a topic “data mining” to guidethe crawler. She can modify and/or tag the indexeddocuments as well.

An extension for the Firefox Web browser en-ables convenient access to 6S while the applicationis active as a background process. As shown in Fig-ure 4, users can submit search queries to the localpeer and the 6S network, see the results returnedthough the network, and instruct the application toindex new pages — all from the browser. Searchresults are shown along with the usual informationlike in any traditional search engine (title, snippet,etc), as well as information about the peers that pro-vided each result. To export pages to the local peerand share them with the 6S community, users canuse the Bookmarks drop-down menu and select op-tions to index all the bookmarks, or just the cur-rent page. The latter option is also available in acontextual (right-click) menu. For example, uponreceiving a relevant page from another 6S peer inresponse to a query about “social networks,” Alicemay choose to bookmark this page in her local 6Sindex, thus making it possible to share this pagewith other peers with related queries in the future.

Inside a 6S AgentEach 6S agent uses a reinforcement learning al-gorithm to track the profiles of other peers basedon their past interactions. A neighbor profile isthe information that a particular agent maintains toestimate the neighbor’s likelihood to provide rel-evant results for various keywords. For example,if a neighbor has previously provided good resultsfor Alice’s query “open source software,” her agentshould internalize this information so as to predictthat this might be a good peer to forward a futurequery on “free software.” By learning profile in-formation, agents try to increase the probability ofchoosing appropriate neighbors for their queries.

Interactions with peers reveal information ofvarying reliability. We want to capture all avail-able information in profiles, but must discriminatecues on the basis of their reliability. To achieve thisgoal we let each peer maintain two profiles for fo-cused and expanded information, respectively. Thefocused profile concerns only query terms, whilethe expanded profile includes keywords that co-occur frequently with query terms within hit pages.Each profile has the same structure and is repre-sented as a matrix W , where each element wp,k isan estimate of how knowledgeable and reliable ispeer p with respect to keyword k. When p returnsresults for a query containing k, wp,k is updatedto reflect the quality of these results. The resultsfrom p are compared to local ones to obtain a re-inforcement signal: good results induce a reward,by which wp,k is increased, while poor results in-duce a penalty and wp,k is decreased. The updateoccurs through a running average to slowly forget

Figure 3: Setup of a 6S peer. To create a personal Web index, the user may provide a crawling topic, anumber of seed pages extracted from the user’s bookmarks, and/or a local document collection. These cuesare used to guide a topical crawler. The crawling results are then indexed for keyword searching. For eachindexed document the user can assign or modify tags, which are searchable by the local engine. Users canalso delete/undelete any document entries or remove/update the entire index.

Figure 4: With 6S running as a background process, a user can access 6S without leaving the Web browserthrough the 6S extension for Firefox. It allows the user to search through the 6S community, export book-marks to 6S, or index a single Web page. All these operations can be done with only a few clicks.

Figure 5: Peer profile update. A peer’s response to a query can indicate a peer’s knowledge with respectto that query. This knowledge is captured by the focused profile,W f . In addition, keywords that co-occurwith query terms within hit pages may reflects (less reliable) information about the peer’s knowledge. This iscaptured by the expanded profile, W e.

Figure 6: Peer selection for query routing. For forwarding a query, known peers are ranked by similarity σbetween the query and the peer profiles. The reliability parameter α regulates the contributions of focusedand expanded profiles. Typically 0.5 < α < 1 to reflect higher confidence in focused profile weights as theycome from direct responses to queries.

Figure 7: Semantic locality in emergent 6S communities. (The networks shown are conceptual mock-up.)Agents initialize and maintain peer profiles by first asking a neighbor for its description, defined as a listof most frequent keywords in the neighbor’s index; then updating these profiles through query/responseinteractions. Such interactions cause the peers to route queries in such a way that peers with similar interestscluster together to find quality results quickly (high clustering coefficient), while it is still possible to reachany peer in a small number of steps (small diameter).

past performance while tracking new information.One of the main motivations behind this approachis that the learning context is likely to be extremelynon-stationary, with highly dynamic peers interestsand collections. Details are illustrated in Figure 5.Suppose for example that Alice submits the query“Lama,” and that Peer 10 returns a set of hits withan average score of S10 = 0.8. Further supposethat the results from Alice’s local index yield anaverage score of S1 = 0.2. If the previous valueof the weight associated with the term “Lama” inAlice’s profile of Peer 10 was zero, the new valuewould be wf

10,Lama = 0.5γ, where γ is a learningrate (0 < γ < 1). For multi-word queries, the sameupdate rule is applied to each term in the query.

In principle, a peer could track an arbitrarilylarge number of other agents. Every time that anew agent is discovered, its profile can be addedto W . In practice, the size of W may be limitedby storage availability. An agent can drop profilesfor the least promising peers when space shortagerequires it. Queries can only be routed to knownagents, i.e., those whose profiles are inW . To routea new query, known peers are ranked by the similar-ity between their profiles and the query, as shownin Figure 6.

Each 6S agent uses the above peer learning andquery-routing algorithms to refine a model of theother peers. The collaborative network in 6S isformed by the dynamic communication among thepeers: queries and responses being sent and for-warded. The instantaneous topology of such a col-laborative network reflects several dynamic pro-cesses: the changing Web collections indexed bythe peers, the evolving information needs of theusers, and the knowledge that agents learn about

Figure 8: Activity on the 6S network in 12 weekssince the prototype release (January–April 2007).The number of active users (those who submit andforward queries) has increased slowly from about20 to almost 40. Note that participants can joinor leave the network arbitrarily. The query trafficthrough the network is rather variable, with burstsfollowing releases of software updates.

others. Initially, when peers know nothing of eachother, queries are routed randomly, and we ob-serve a random network topology. As 6S agentsrefine their internal models of others based on ob-served queries and responses, query routing be-comes more content-driven. Semantic localitymeans that queries should be routed efficiently to-ward knowledgeable peers, and peers with similarinterests should end up closer in the collaborationnetwork. We postulate that such a locality shouldlead to the emergence of semantic clusters, as illus-trated in Figure 7, and thus prevent congestion.

The 6S Collaboration NetworkBefore the 6S prototype was developed, we exper-imented with a number of peer representations andmachine learning algorithms for query routing byrunning simulations with realistic synthetic usersand queries. The details of these simulations andour findings have been reported elsewhere (Wu,Akavipat, & Menczer 2005; Akavipat et al. 2006).Here we summarize a number of promising prop-erties about the 6S network, highlighted by theseexperiments:

• The agents rapidly form clusters (spontaneousgroups that communicate more within the groupthan outside), displaying a query topology thatconverges to a small-world network after eachpeer has routed as few as five or six queries, andthis change in topology leads to an increase inthe quality of the results.

• The clusters, which are formed by agents’ querytraffic, identify communities of peers with simi-lar interests, indicating that the network exhibitssemantic locality.

• The collective search performance of the net-work improves when more sophisticated learn-ing algorithms are employed by the agents toroute queries, and as more network resources be-come available. Performance degrades grace-fully as bandwidth and CPU cycles becomescarcer.

• The 6S peers achieve a search quality (in termsof precision and recall) that is comparable to thatof Google, and significantly outperform a cen-tralized search engine with the same resources(crawl size) as the combined 6S peer collective.

• The 6S algorithms scale well up to 500 peers, themaximum number of users we were able to sim-ulate in a closely controlled testing environment.

Since the release of the 6S prototype, we havebeen tracking a small community of early adoptersto see if these results hold “in the wild.” This userstudy is designed to observe how people use 6S andhow the collaborative search network evolves withusers’ activities. To this end, data is recorded andtransmitted from participants’ computers to a col-lection server through a secure channel once a day.The data collected includes query routing informa-tion, queries, results, size of personal Web index,and most common indexed terms. Figure 8 plotsthe activity of the network in its first 12 weeks oflife. The data and feedback we are collecting arehelping to improve the software by making it moretransparent, persistent, robust, and interactive. Forexample, in the prototype used to collect this data,the application does not run in the background,so that users quitting the application automaticallyleave the network. This behavior will be changed in

future releases, so that a peer can remain active anduseful even when the user is not interacting with it.

Figure 9 visualizes the collaborative search net-work. We can distinguish the query network, whichshows the propagation of queries among peers,from the response network, which shows who pro-vides results to whom. There is evident heterogene-ity in the number of queries received and resultssent. One of the nodes in the network is a specialpeer that submits queries to the Yahoo search en-gine via its API, and returns the results obtainedfrom Yahoo. This node is effectively “Yahoo indisguise” — but the other peers know nothing ofits identity. We wanted to determine whether thenetwork would learn to rely on this peer, whichis clearly very good, given its universal expertise.Indeed, the Yahoo peer does become very central,with the highest number of incoming queries andalso the highest number of incoming edges (peersthat forward queries to it) for most of the exper-iment duration. It also provides many results toother peers.

Figure 10 plots the small-world statistics of the6S collaborative query network within our userstudy period. The diameter is defined as the av-erage shortest path across all pairs of nodes (withadjustments to deal with disconnected networks).The network’s clustering coefficient is the averageof nodes’ clustering coefficients, across all nodes.An individual node i’s clustering coefficient ci isthe fraction of triangles in which i participates, outof the possible ones. That is, ci is the number ofpairs of neighbors of i that are also neighbors ofeach other, divided by the total number of pairs ofneighbors of i. It is interesting to compare thesemeasures with what one would observe in a randomnetwork, which is known to have a very short diam-eter and a very small clustering coefficient. There-fore, for each week, we construct an ensemble ofrandom networks with the same numbers of nodesand edges as the 6S networks. Then we measure byhow much the diameter and clustering coefficientin 6S exceed the average ones from the random net-works. As Figure 10 shows, the diameter remainssmall but the clustering coefficient grows consider-ably. These conditions indicate the emergence of asmall-world topology in our peer network (Watts &Strogatz 1998).

Related WorkA P2P computer network relies on the computingpower and bandwidth of the participants in the net-work, rather than concentrating it in a relatively fewservers. The most popular use of a P2P networkis for file sharing. Applications such as Gnutella,BitTorrent and KaZaa (Androutsellis-Theotokis &Spinellis 2004) allow peers to share content filesamong peers without having to set up dedicatedservers and acquiring large bandwidth to support

Figure 9: Weekly snapshots of two 6S collaborative search networks. To visualize the query network (left),we aggregate the queries routed during week 12 of our user study. This was the week with the largest numberof active peers. Edge width is proportional to the number of queries exchanged between two peers. The areaof each node is proportional to the number of queries received by the peer, which is an indirect measure ofcentrality, authority, and/or reliability of the peer as learned by the other agents. To visualize the responsenetwork (right), we aggregate results sent during week 6, which was the one with largest number of queriesand responses. This network is visualized from end to end, i.e., an edge directly connects the provider andthe receiver of a result, irrespective of the chain of peers through which the results were actually routed.Edge width is proportional to number of results exchanged, and node size is proportional to number of resultsprovided. Thus, larger nodes are more helpful. In both networks, inactive nodes (those with no incomingqueries or outgoing results) are not shown. The node marked with a white rectangle is the Yahoo searchengine in disguise (see text); by design, this peer does not generate or forward queries, yet it is the mostpopular target of queries and the second most productive provider of results.

Figure 10: Relative difference between the di-ameter and clustering coefficient of the collabora-tive query network and those in random networks.To measure both diameter and clustering coeffi-cient, we disregard edge directionality. Trend linesshow that the diameter remains equal to the ran-dom graph diameter, while the clustering coeffi-cient increases considerably, compared to the ran-dom graphs.

the whole community. P2P file-sharing applica-tions are by no means replacing dedicated serversin content distribution. They simply provide analternative for content distribution by trading thespeed and reliability of dedicated servers for theease of sharing, lower cost, fault tolerance, andlower bandwidth requirement of a file sharer.

Just as P2P file-sharing applications are used tofacilitate content distribution, P2P applications canbe developed to facilitate Web search. There isa wide variety of peer-based search applications.For example, a model proposed by the YouSe-arch project is based on maintaining a centralizedsearch registry for query routing (such as Napster),while providing the peers with the capability tocrawl and index local portions of the Web (Bawaet al. 2003). NeuroGrid employs a learning mech-anism to adjust metadata describing the contentsof nodes (Joseph 2002). A similar idea has beenproposed to distribute and personalize Web searchusing a query-based model and collaborative filter-ing (Pujol, Sanguesa, & Bermudez 2003).

An intermediate approach between the com-pletely decentralized flood network (as in Gnutella)and the centralized registry is to store index lists indistributed, shared hash tables (Suel et al. 2003).In pSearch (Tang, Xu, & Dwarkadas 2003), latentsemantic analysis (Deerwester et al. 1990) is per-formed over such distributed hash tables to provide

peers with keyword search capability. Another al-ternative is that of hybrid peer networks, in whichmultiple special directory nodes (hubs) constructand use content models of neighboring nodes to de-termine how to route query messages through thenetwork (Lu & Callan 2003).

Similar ideas are receiving increasing attentionin the multi-agent literature. For example, a modelproposed by Bulka et al. (2006) includes a learningalgorithm by which each agent uses local informa-tion and previous experience to refine a classifier.The agent then uses the classifier to decide whichagent groups to join or whether to form a new groupto complete a task. Pearce & Tambe (2007) studyoptimal collaborative strategies based on local in-teractions for teams of agents to solve distributedconstraint optimization problems.

Status and Future Work6S is freely available at Sixearch.org. Wehope to attract a community of users, which willallow us to test its scalability and robustness, whileimproving its usability and effectiveness. Be-cause collaborative peer search represents a newparadigm for Web search, the interface betweenthe 6S network and its users is critical. It is im-portant that we understand how users interact with6S and how to best keep their experience positive.The user study, still under way, should provide uswith information that will help improve 6S. If userscontinue to find 6S useful, they will maintain theirpresence in the peer network.

We plan to explore additional learning algo-rithms to improve the performance of 6S’s adap-tive query routing. For example, we want to minethe streams of queries and responses that are for-warded though a peer. In the Gnutella v0.6 file-sharing network, peers tend to issue queries that arevery similar to the content of files they have avail-able for sharing (Asvanund et al. 2003). This sug-gests that a profile of a peer’s knowledge should beupdated based on the queries the peer issues in ad-dition to the query responses that it produces. An-other technique we would like to examine, queryrelaxation, was proposed in a semantic Web set-ting (Tempich, Staab, & Wranik 2004). A peerthat queries for RDF data assumes that a neigh-bor may have knowledge about a topic/query if ithas knowledge about a more specific version ofthe topic/query. While our application is arguablymore difficult due to the unstructured nature ofgeneric Web pages, we hope that the promisingscalability results obtained for semantic Web datawill generalize to Web IR.

A number of other IR techniques are also underconsideration. For example, profiles in the currentprototype are based on simple vector space repre-sentations. Similarity between queries and docu-ments is based on simple vector cosine measures.

While these techniques are well established, theyhave limitations when one considers keyword spar-sity, ambiguity, synonymity, and so on. Richer rep-resentation, for example based on co-occurrencestatistics (e.g., LSI (Deerwester et al. 1990)) orsemantic ontologies (e.g., WordNet) could addresssome of these issues.

A peer selection algorithm should be able notonly to determine which peers are best suited for agiven query, but also to predict which combinationsof peers provide the least redundant results. Ex-isting peer selection algorithms take into accountonly the predicted query-specific precision qualityof known peers for peer ranking. In a purely un-structured network such as 6S, however, each peercrawls the Web independently based on its own in-terests, without any central control mechanism. Asa result, it is likely that peers with similar interestswill have a high degree of overlap between theirdocument collections. Consider the extreme caseof two peers with identical collections. In a naivepeer selection approach, if one peer is selected asa good neighbor, the other peer will definitely beselected as well. However, forwarding a query toboth peers will generate no more relevant resultsthan submitting to one peer alone, due to their col-lection overlap. We are investigating extensions tothe peer selection algorithm in which a peer wouldpay attention to the overlap between two neighborsin order to maximize recall as well as precision.

Finally, in developing a collaborative peer basedsearch network, one has to think about protectingthe system from abuse. For example, by exploitingknowledge of how peers learn from query interac-tions, attackers can craft their responses to maketargeted peers favor the attackers for future query-ing while directing users to spam content. Collud-ers can also set up peers that provide some high-quality responses, but mixed with pointers to spam-ming peers. In addition, the victims may inadver-tently help the attackers by forwarding other peers’queries to them, thus exposing those peers to thesame response attacks. To prevent such exploita-tion, a collaborative search network such as 6Sneeds a security component. We are working ona reputation system that can help distinguish spam-mers from honest peers.

Acknowledgements

This paper owes considerable improvements to thecomments of two anonymous reviewers. We aregrateful to the Apache Lucene project for the Nutchopen source search engine code and to Sun Mi-crosystems for the JXTA open source P2P code.This work was supported by Dr. Menczer’s NSFCareer Grant IIS-0348940.

ReferencesAkavipat, R.; Wu, L.-S.; Menczer, F.; and Magui-tman, A. 2006. Emerging semantic communitiesin peer web search. In P2PIR ’06: Proceedingsof the international workshop on Information re-trieval in peer-to-peer networks, 1–8. New York,NY, USA: ACM Press.Androutsellis-Theotokis, S., and Spinellis, D.2004. A survey of peer-to-peer content distribu-tion technologies. ACM Comput. Surv. 36(4):335–371.Asvanund, A.; Bagala, S.; Kapadia, M.; Krish-nan, R.; Smith, M.; and Telang, R. 2003. Intelli-gent club management in P2P networks. In Proc.Workshop on P2P Systems.Bawa, M.; Bayardo Jr, R.; Rajagoplan, S.; andShekita, E. 2003. Make it fresh, make it quick —searching a network of personal webservers. InProc. 12th International World Wide Web Confer-ence.Brewington, B. E., and Cybenko, G. 2000. Howdynamic is the Web? In Proc. 9th InternationalWorld-Wide Web Conference.Bulka, B.; Gaston, M.; and desJardins, M. 2006.Local strategy learning in networked multi-agentteam formation. Journal of Autonomous Agentsand Multi-Agent Systems (JAAMAS).Chakrabarti, S.; van den Berg, M.; and Dom, B.1999. Focused crawling: a new approach to topic-specific Web resource discovery. Computer Net-works (Amsterdam, Netherlands: 1999) 31(11–16):1623–1640.Deerwester, S.; Dumais, S.; GW, F.; Landauer, T.;and Harshman, R. 1990. Indexing by Latent Se-mantic Analysis. Journal of the American Societyfor Information Science 41:391–407.Fetterly, D.; Manasse, M.; Najork, M.; andWiener, J. 2003. A large-scale study of the evo-lution of Web pages. In Proc. 12th InternationalWorld Wide Web Conference.Joseph, S. 2002. Neurogrid: Semantically routingqueries in Peer-to-Peer networks. In Proc. Intl.Workshop on Peer-to-Peer Computing.Lawrence, S., and Giles, C. 1999. Accessibilityof information on the Web. Nature 400:107–109.Lu, J., and Callan, J. 2003. Content-based re-trieval in hybrid peer-to-peer networks. In Proc.12th Intl. Conf. on Information and KnowledgeManagement (CIKM’03).Menczer, F., and Belew, R. K. 2000. Adap-tive retrieval agents: Internalizing local contextand scaling up to the web. Machine Learning39(2/3):203–242.Menczer, F.; Pant, G.; and Srinivasan, P. 2004.Topical web crawlers: Evaluating adaptive algo-

rithms. ACM Transactions on Internet Technology4(4):378–419.Ntoulas, A.; Cho, J.; and Olston, C. 2004. What’snew on the Web?: The evolution of the Web froma search engine perspective. In Proceedings of the13th International Conference on the World WideWeb, 1–12. ACM Press.Pant, G.; Bradshaw, S.; and Menczer, F. 2003.Search engine-crawler symbiosis. In Koch, T.,and Solvberg, I., eds., Proc. 7th European Confer-ence on Research and Advanced Technology forDigital Libraries (ECDL), Lecture Notes in Com-puter Science, Vol. 2769. Berlin: Springer Verlag.Pearce, J. P., and Tambe, M. 2007. Quality guar-antees on k-optimal solutions for distributed con-straint optimization problems. In Proc. 20th In-ternational Joint Conference on Artificial Intelli-gence (IJCAI).Pujol, J.; Sanguesa, R.; and Bermudez, J. 2003.Porqpine: A distributed and collaborative searchengine. In Proc. 12th Intl. World Wide Web Con-ference.Suel, T amd Mathur, C.; Wu, J.-W.; Zhang, J.;Delis, A.; Kharrazi, M.; Long, X.; and Shanmu-gasundaram, K. 2003. ODISSEA: A Peer-to-Peerarchitecture for scalable Web search and informa-tion retrieval. In International Workshop on theWeb and Databases (WebDB).Tang, C.; Xu, Z.; and Dwarkadas, S. 2003. Peer-to-peer information retrieval using self-organizingsemantic overlay networks. In Proc. ACM SIG-COMM ’03.Tempich, C.; Staab, S.; and Wranik, A. 2004. RE-MINDIN’: Semantic query routing in peer-to-peernetworks based on social metaphors. In Proc. 13thconference on World Wide Web, 640–649. ACMPress.Waterhouse, S. 2001. JXTA Search: Distributedsearch for distributed networks. Technical report,Sun Microsystems Inc.Watts, D., and Strogatz, S. 1998. Collectivedynamics of “small-world” networks. Nature393:440–442.Watts, D.; Dodds, P.; and Newman, M. 2002.Identity and search in social networks. Science296:1302–1305.Wu, L.-S.; Akavipat, R.; and Menczer, F. 2005.6S: Distributing crawling and searching acrossWeb peers. In Proceedings of the IASTED Inter-national Conference on Web Technologies, Appli-cations, and Services.