ShortCuts: Using Soft State

To Improve DHT Routing

Kiran Tati and Geoffrey M. Voelker

Department of Computer Science and Engineering

University of California, San Diego

La Jolla, CA 92093-0114

{ktati, voelker}@cs.ucsd.edu

Abstract

Distributed hash tables are increasingly being pro-posed as the core substrate for content delivery ap-plications in the Internet, such as cooperative Webcaches, Web index and search, and content deliverysystems. The performance of these applications builton DHTs fundamentally depends on the effectivenessof request routing within the DHT. In this paper, weshow how to use soft state to achieve routing perfor-mance that approaches the aggressive performanceof one-hop schemes, but with an order of magnitudeless overhead on average. We use three kinds of hintcaches to improve routing latency: local hint caches,path hint caches, and global hint caches. Local hintcaches use large successor lists to short cut final hops.Path hint caches store a moderate number of effec-tive route entries gathered while performing lookupsfor other nodes. And global hint caches store directroutes to peers distributed across the ID space. Basedupon our simulation results, we find that the combi-nation of the hint caches significantly improves Chordrouting performance: in a network of 4,096 peers, thehint caches enable Chord to route requests with aver-age latencies only 6% more than algorithms that usecomplete routing tables with significantly less over-head.

1 Introduction

Peer-to-peer overlay networks provide a distributed,fault-tolerant, scalable architecture on which wide-area distributed systems and applications can bebuilt. An increasing trend has been to propose con-tent delivery services on peer-to-peer networks, in-cluding cooperative Web caches [8], Web indexingand searching [13,15], content delivery systems [2,11],and Usenet news [5]. Popular designs of these overlaynetworks implement a distributed hash table (DHT)interface to higher level software. DHTs map keys ina large, virtual ID space to associated values storedand managed by individual nodes in the overlay net-work. DHTs use a distributed routing protocol toimplement this mapping. Each node in the overlaynetwork maintains a routing table. When a nodereceives a request for a particular key, it forwardsthe request to another node in its routing table thatbrings the request closer to its destination.

A natural trade off in the design of these routingprotocols is the size of the routing table and the la-tency of routing requests. Larger routing tables canreduce routing latency in terms of the number ofhops to reach a destination, but at the cost of ad-ditional route maintenance overhead. Because theperformance and overhead of DHT overlay networksfundamentally depend upon the distributed routingprotocol, significant work has focused on the problemof balancing the degree of routing state and mainte-nance with route performance.

1

Initial systems like Chord [27], Pastry [25],Tapestry [30], and CAN [23] use routing tables ofdegree O(log n) to route requests in O(log n)) hops,where n is the number of hosts in the network.Newer algorithms improve the theoretical boundson routing state and hops. Randomized algorithmslike Viceroy [16] and Symphony [17] achieve small,constant-degree routing tables to route requests onaverage in O(log n) and O(log log n) hops, respec-tively. Koorde [9] is a tunable protocol that can routerequests with a latency ranging from O(log n) toO(log n/ log log n) hops for routing tables of constantsize to O(log n)) size, respectively. Other approaches,such as Kelips [7], Structured Superpeers [21], Bee-hive [22], and CUP [24] focus on achieving constant-time O(1) hops to route requests at the expense ofhigh degree routing tables, hierarchical routing, tai-loring to traffic distributions, or aggressive updateprotocols to maintain consistency among the largerouting tables in each peer.

In this paper, we argue that the appropriate use ofcached routing state within the routing protocol canprovide competitive improvements in performancewhile using a simple baseline routing algorithm. Wedescribe and evaluate the use of three kinds of hintcaches containing route hints to improve the routingperformance of distributed hash tables (DHTs): localhint caches store direct routes to successors in the IDspace; path hint caches store direct routes to peersaccumulated during the natural processing of lookuprequests; and global hint caches store direct routes toa set of peers roughly uniformly distributed acrossthe ID space.

These hint caches require state similar to previousapproaches that route requests in constant-time hops,but they do not require the complexity and communi-cation overhead of a distributed update mechanismto maintain consistency among cached routes. In-stead, the hint caches do not explicitly maintain con-sistency in response to peer arrivals and departuresother than as straightforward extensions of the stan-dard operations of the overlay network. We showthat hint cache inconsistency does not degrade theirperformance benefits.

We evaluate the use of these hint caches by sim-ulating the latest version of the Chord DHT [4] and

extending it to use the three hint caches. We evalu-ate the effectiveness of the hint caches under a varietyof conditions, including highly volatile peer turnoverrates and relatively large network sizes. Based uponour simulation results, we find that the combinationof the hint caches significantly improves Chord rout-ing performance. In networks of 4,096 peers, the hintcaches enable Chord to route requests with averagelatencies only 6% more than algorithms like “One-Hop” that use complete routing tables, while requir-ing an order of magnitude less bandwidth to main-tain the caches and without the complexity of a dis-tributed update mechanism to maintain consistency.

The remainder of the paper is organized as follows.In Section 2, we discuss related work on improvingrouting performance in peer-to-peer overlays. Sec-tion 3 describes how we extend Chord to use the lo-cal, path, and global hint caches. Section 5 describesour simulation methodology for evaluating the hintcaches, and presents the results of our evaluations.Finally, Section 6 summarizes our results and con-cludes.

2 Related Work

Initial distributed routing protocols for DHT peer-to-peer overlay networks were designed to balance rout-ing state overhead and route maintenance overheadwhile providing provably attractive routing perfor-mance. Chord [27], Pastry [25], Tapestry [30], andKademlia [18] use routing tables with degree O(log n)to route requests in O(log n) hops through the net-works.

Since the performance and overhead of these net-works fundamentally depend upon the distributedrouting protocol, significant research has focused onthe problems of improving route maintenance andperformance. As a result, newer algorithms have im-proved the theoretical bounds on routing degree androuting hops. The most efficient of these algorithmsachieve either constant degree routing state or con-stant hop routing latency, often, however, at the priceof additional system complexity.

A number of algorithms seek to minimize routemaintenance overhead by using only constant-size

2

O(1) routing tables per node, such as Viceroy [16],Symphony [17], and [9]. Several efforts have alsobeen made to achieve constant-time O(1) hops toroute requests at the cost of high-degree routing ta-bles. These approaches use gossip protocols to prop-agate route updates [7], hierarchical routing throughstructured superpeers [21], complete routing tablesconsistently maintained using a hierarchical updateprotocol [6], and reliance on traffic distributions [22].

Chord/DHash++ [4] exploits the fact that lookupsfor replicated values only need to reach a peer nearthe owner of the key associated with the value (sincethe peer will have a replica). Although this is ap-propriate for locating any one of a number of repli-cas, many applications require exact lookup. The“OneHop” approach uses a very aggressive updatemechanism to route requests in only a single hop [6].However, this approach requires a hierarchical updatedistribution tree overlayed on the DHT, and requiressignificant communication overhead to distribute up-dates to all nodes in the system. Beehive exploits thepower-law property of lookup traffic distributions [22]to achieve constant-time lookups. However, a largeclass of applications induce different types of lookuptraffic.

Perhaps the most closely related work is the Con-trolled Update Protocol (CUP) for managing pathcaches on peers [24]. CUP uses a query and updateprotocol to keep path caches consistent with peer ar-rivals and departures. CUP was implemented in thecontext of the CAN overlay network, and evaluatedrelative to straightforward path caches with expira-tion. Although the CUP path caches are analogousto the path hint caches in our work, our work dif-fers in a number of ways with the CUP approach.Rather than providing an update mechanism to keepcaches consistent, we instead combine the use of lo-cal hint caches with path and global hint caches toimprove performance and tolerate inconsistency. Wealso evaluate hint caches with a baseline DHT routingalgorithm that routes in O(log n) hops (rather thana range of coordinate dimensions).

3 Design

Distributed hash tables (DHT) increasingly serve asthe foundation for a wide range of content deliverysystems and applications. The DHT lookup oper-ation is the fundamental operation on which appli-cations base their communication. As a result, theperformance of these applications directly depends onthe performance of the lookup operation, and improv-ing lookup performance improves performance for allapplications layered on DHTs.

The primary goal of our work is to reduce lookupperformance as close to direct routing with much lessoverhead than previous approaches and without rely-ing upon specific traffic patterns. We also integratethe cache update mechanism to refresh cached routeentries into the routing protocol to minimize the up-date complexity as well as overhead. To achieve thisgoal, each peer employs three hint caches. Localhint caches store direct routes to neighbors in theID space. Path hint caches store direct routes topeers accumulated during the natural processing oflookup requests. Finally, global hint caches store di-rect routes to a set of peers roughly uniformly dis-tributed across the ID space. We call them hintcaches since the cached routes are hints that maypotentially be stale or inconsistent. We also considerthem soft-state hints since they can be reconstructedquickly at any time and they are not necessary forthe correctness of the routing algorithm.

The following sections describe the behavior ofeach of the three hint caches. Although these cachesare applicable to DHTs in general, we describe themin the context of integrating them into the ChordDHT as a concrete example. So we start with a briefoverview of the Chord lookup operation and routingalgorithm as background.

3.1 The Chord DHT

In Chord, all peers in the overlay form a circularlinked list. Each peer has one successor and onepredecessor. Each peer also maintains O(log n) suc-cessors and O(log n) additional peers called fingers.The owner of a key is defined as a peer for whichthe key is in between the peer’s predecessor’s ID and

3

its ID. The lookup operation for a given key returnsthe owner peer by successively traversing the over-lay. Peers construct their finger tables such that thelookup operation traverses progressively closer to theowner in each step. In recursive lookup, the initia-tor peer uses its routing table to contact the closestpeer to the key. This closest peer then recursivelyforwards the lookup request using its routing table.Included in the request is the IP address of the initi-ating peer. When the request reaches the peer thatowns the key, that peer responds directly to the ini-tiator. This lookup operation contacts O(log n) ap-plication level intermediate peers to reach the ownerfor a given key.

We augment Chord with the three hint caches.Chord uses these hint caches as simple extensions toits original routing table. When determining the nextbest hop to forward a request, Chord considers theentries in its original finger table as well as all entriesin the hint caches.

3.2 Local Hint Caches

Local hints are direct routes to neighbors in the IDspace. They are extensions of successor lists in Chordand leaf nodes in Pastry, except that their purposeis to improve routing performance. With a cache oflocal hints, a peer can directly reach a small fractionof peers directly and peers can short cut the finalhops of request routing.

Local hints are straightforward to implement in asystem like Chord using its successor lists. Normally,each peer maintains a small list of its successors tosupport fault-tolerance within Chord and upper layerapplications. Peers request successor lists when theyjoin the network. As part of a process called stabi-lization in Chord, each peer also periodically pings itssuccessor to determine liveness and to receive updatesof new successors in its list. This stabilization pro-cess is fundamental for maintaining lookup routingcorrectness, and most DHT designs perform similarprocesses to maintain successor liveness.

We propose enlarging these lists significantly — onthe order of a thousand entries — to become localhint caches. Growing the successor lists does not in-troduce any additional updates, but it does consume

additional bandwidth. The additional bandwidth re-quired is S

Hentries per second where S is the num-

ber of entries in local hint cache, and H is the halflife time of peers in the system. Each peer change,either joining or leaving, requires two entries to up-date. Similar to [14], we define the half life as thetime in seconds for half of the peers in the systemto either leave or join the DHT. For perspective, astudy of the Overnet peer-to-peer file sharing systemmeasured a half life of four hours [1].

The overhead of maintaining the local hint cache isquite small. For example, when S is 1000 entries andH is four hours, then each peer will receive 0.07 extraentries per second during stabilization. Since entriesare relatively small (e.g., 64 bytes), this correspondsto only a couple of bytes/sec of overhead.

Local hint caches can be inconsistent due to peerarrivals and departures. When a peer fails or a newpeer joins, for example, its immediate predecessorwill detect the failure or join event during stabiliza-tion. It will then update its successor list, and startpropagating this update backwards along the ringduring subsequent rounds of stabilization. Conse-quently, the further one peer is from one its succes-sors, the longer it takes that peer to learn that thesuccessor has failed or joined.

The average amount of stale data in the local hint

cache is R∗S∗(S+1)4∗H

, where R is the stabilization pe-riod in seconds (typically one second). On average apeer accumulates 1

2∗Hpeers per second of stale data.

Since a peer updates its x’th successor every x∗R sec-onds, it accumulates x∗R

2∗Hstale entries from its x’th

successor. If a peer has S successors, then on aver-age the total amount of stale data is

∑S

i=1i∗R2∗H

. Ifthe system half life time H is four hours and the lo-cal hint cache size is 1000 peers, then each peer onlyhas 1.7% stale entries. Of course, a peer can fur-ther reduce the stale data by using additional updatemechanisms, introducing additional bandwidth andcomplexity. Given the small impact on routing, weargue that such additions are unnecessary.

3.3 Path Hint Caches

The distributed nature of routing lookup requests re-quires each peer to process the lookup requests of

4

other peers. These lookup requests are generatedboth by the application layered on top of the DHT aswell as the DHT itself to maintain the overlay struc-ture. In the process of handling a lookup request,a given peer can get information about other peersthat contact it as well as the peer that initiated thelookup.

With Path Caching with Expiration (PCX) [24],peers cache path entries when handling lookup re-quests, expiring them after a time threshold. PCXcaches entries to the initiator of the request as wellas the result of the lookup, and the initiator cachesthe results of the lookup. In PCX, a peer storesroutes to other peers without considering the latencybetween itself and these new peers. In many cases,these extra peers are far away in terms latency. Usingthese cached routes to peers can significantly add tothe overall latency of lookups (Figure 6(a)). HencePCX, although it reduces hops (Figure 6(b)), can alsocounter-intuitively increase lookup latency.

Instead, peers should be selective in terms ofcaching information about routes to other peerslearned while handling lookups. We propose a se-lection criteria based on the latency to select a peerto cache it. A peer x caches a peer y if the latencyto y from x is less than the latency from x to peer z,where (1) z is in x’s finger table already and (2) itsID comes immediately before y’s ID if x orders theIDs of its finger table peers. For example, assume yfalls between a and b in x’s finger table and then peerx contacts a to perform the lookup request for an IDbetween (a, b]. If we insert y, then x would contact yfor the ID between (y, b]. Since the latency to y fromx is less than the latency a from x, the lookup latencymay reduce for IDs between (y, b]. As a result, x willcache the hop to y. We call the cache that collectssuch hints the path hint cache.

We would like to maintain the path hint cachewithout the cost of keeping entries consistent. Thefollowing cache eviction mechanism tries to achievethis goal. Since a small amount of stale data will notaffect lookup performance significantly (Figure 4), apeer tries to choose a time period to evict entries inthe path hint cache such that amount of stale data inits path cache is small, around 1%. The average timeto accumulate d percentage of stale data in the path

hint cache is 2 ∗ d ∗ h seconds, where h is the halvingtime [14]. Hence a peer can use this time period asthe eviction time period.

Although the improvement provided by path hintcaches is somewhat marginal (1–2%), we still use thisinformation since it takes advantage of existing com-munication and comes free of cost.

3.4 Global Hint Caches

The goal of the global hint cache is to approximatetwo-hop route coverage of the entire overlay networkusing a set of direct routes to low-latency, or nearby,peers. Ideally, entries in the global hint cache provideroutes to roughly equally distributed points in the IDspace; for Chord, these nearby routes are to peersroughly equally distributed around the ring.

These nearby peers work particularly well in com-bination with the local hint caches at peers. Whenrouting a request, a peer can forward a lookup to oneof its global cache entries whose local hint cache hasa direct route to the destination. With a local hintcache with 1000 entries, a global hint cache with afew thousand nodes will approximately cover an en-tire system of few million peers in two hops.

A peer populates its global hint cache by collectingroute entries to low-latency nodes by walking the IDspace. A peer x contacts a peer y from its routingtable to request a peer z from y’s local hint cache.The peer x can repeat this process from z until itreaches one of its local hint cache peers. We call thisprocess space walking.

While choosing peer z, we have three requirements:minimizing the latency from x, minimizing x’s globalhint cache size, and preventing gaps in coverage dueto new peer arrivals. Hence, we would like to have alarge set of peers to choose from to find the closestpeer, to choose the farthest peer in the y’s local hintcache to minimize the global hint cache size, and tochoose the closer peer in y’s local hint cache to pre-vent gaps. To balance these three requirements, whendoing a space walk to fill the global hint cache we usethe second half of the successor peers in the local hintcache.

Each peer uses the following algorithm to main-tain the global hint cache. Each peer maintains an

5

index pointer into the global hint cache called therefresh pointer. Initially, the refresh pointer pointsto the first entry in the global hint cache. The peerthen periodically walks through the cache and ex-amines cache entries for staleness. The peer only re-freshes a cache entry if the entry has not been used inthe previous half life time period. The rate at whichthe peer examines entries in the global hint cache is

g

2∗d∗h, where d is targeted percentage of stale data

in the global hint cache, g is the global hint cachesize, and h is the halving time. This formula is basedon the formula for stale data in the path hint cache(Section 3.3).

d is a system configuration parameter, and peerscan estimate h based upon peer leave events in thelocal hint cache. For example, if the halving time his four hours, the global hint cache size g is 1000, andthe maximum staleness d is 0.125%, then the refreshtime period is 3.6 seconds. Note that if a peer usesan entry in the global hint cache to perform a lookup,it implicitly refreshes it as well and consequently re-duces the overhead of maintaining the hint cache.

Scaling the system to a very large number of nodes,such as two million peers, the global hint cache wouldhave around 4000 entries and peers would require oneping message per second to maintain 0.5% stale datain very high churn situations like one-hour halvingtimes. Such overheads are small, even in large net-works.

Peers continually maintain the local and path hintcaches after they join the DHT. In contrast, a peerwill only start space walking to populate its globalhint cache if it receives a threshold explicit lookup re-quests directly from the application layer (as opposedto routing requests from other peers). The global hintcache is only useful for the lookups made by the peeritself. Hence, it is unnecessary to maintain this cachefor a peer that is not making any lookup requests.Since a peer can build this cache very quickly (Fig-ure 9), it benefits from this cache soon after it startsmaking application level lookups. A peer maintainsthe global hint cache using the above algorithm aslong as it receives lookups from applications on thepeer.

3.5 Discussion

Our goal is to achieve near-minimal request rout-ing performance with significantly less overhead thanprevious approaches. Local hint caches require S

Hen-

tries/sec additional stabilization bandwidth, where Sis the number of entries in the local hint cache and His the half life of the system. Path hint caches requireno extra bandwidth since they incorporate informa-tion from requests sent to the peer. And, in the worstcase, global hint caches require one ping message per2∗d∗h

gseconds to refresh stale entries.

For comparison, in the “OneHop” approach [6]each peer periodically communicates N

2∗Hentries to

update its routing table, an order of magnitude moreoverhead. With one million peers at four hour half lifetime, for example, peers in “OneHop” would need tocommunicate at least 35 entries per second to main-tain the state consistently, whereas the local hintcache requires 0.07 entries per second and one pingper 28 seconds to maintain the global hint cache.

4 Shun Pikes

The direct route provided by the Internet betweentwo peers may not be the best route [26]. Accord-ing to the latest study, 17% pairs of peers reduce 25milliseconds in latency if we use alternate paths [12]with some intermediate hops. Our goal is to reducethe DHT lookup latency by exploiting the alternatepaths that are better than the direct paths. Eachpeer tries to forward the lookup request to a peerthat is closest to the lookup key from its routing ta-ble until the lookup request reaches the owner. Wetry to use these alternate paths to reach these inter-mediate hops if alternative paths reduce latency.

The simplest approach to find alternate shortestpaths is to run the single source shortest paths algo-rithm (either Dijkstra or Bellman-Ford) at each peerusing the local hint cache and global hint cache as theintermediate hops. The shortest paths from a singlenode to all other nodes in a complete graph can beapproximated closely by the shortest paths that areconstructed from a few random subset of nodes inthe graph similar to Internet [?]. In other words, the

6

shortest paths constructed for single node with somerandom nodes and edges to these random nodes arealmost as good as the shortest paths constructed fromthe same node to all other nodes in the graph withcomplete graph. In our case, the local hint cache hassmall set of random subset of peers in the DHT andthe global hint cache has the closest peers for a givenpeer. Hence, the shortest path constructed from apeer x to any other peer y in the DHT using just x’slocal and global hint cache peers as intermediate hopsis within constant factor from shortest path from x toy if x uses all other peers in the DHT as intermediatehops to construct the shortest path.

Each peer constructs the shortest paths to all otherpeers in its caches and tries to uses this informationin forwarding the lookup operation. As previously,a peer x tries to forward the lookup request to theclosest peer y in the identifier space. However if thereis a better path from x to y through peer z then peerx forwards the lookup to peer z, instead of directlyforwarding to y, to forward the lookup request to they.

Overall this new lookup algorithm combining withthe local and global hint cache performs better thanthe “OneHop” approach with significantly less over-head. However, “OneHop” could be modified to takeadvantages of detours to improve the lookup opera-tion and can achieve similar lookup performance.

5 Methodology and Results

In this section we describe our DHT simulator andour simulation methodology. We also define our per-formance metric, average space walk time, to evalu-ate the benefits of our hint caches on DHT routingperformance.

5.1 Chord Simulator

Although the caching techniques are applicable toDHTs in general, we chose to implement and eval-uate them in the context of Chord [27] due to its rel-ative simplicity. Although the Chord group at MITmakes its simulator available for external use [10],we chose to implement our own Chord simulator to-

gether with our hint caching extensions. We im-plemented a Chord simulator according to the recentdesign in [4] that optimizes the lookup latency bychoosing nearest fingers. It is an event-driven sim-ulator that models network latencies, but assumesinfinite bandwidth and no queuing in the network.Since our experiments had small bandwidth require-ments, these assumptions have a negligible effect onthe simulation results.

We separated the successor list and finger tablesto simplify the implementation of the hint caches.During stabilization, each peer periodically pings itssuccessor and predecessor. If it does not receive anacknowledgment to its ping, then it simply removesthat peer from it tables. Each peer also periodicallyrequests a successor list update from its immediatesuccessor, and issues lookup requests to keep its fin-ger table consistent. When the lookup reaches thekey’s owner, the initiating peer chooses as a fingerthe peer with the lowest latency among the peers inthe owner’s successor list.

For our experiments, we used a period of one sec-ond to ping the successor and predecessor and a 15minute time period to refresh the fingers. A fingeris refreshed immediately if a peer detects that thefinger has left the DHT while performing the lookupoperation. These time periods are same as ones usedin the Chord implementation [10].

To compare different approaches, we want to eval-uate the potential performance of a peer’s routingtable for a given approach. We do this by defininga new metric called space walk latency. The spacewalk latency for a peer is the average time it takesto perform a lookup to any other peer on the DHTat a given point of time. We define a similar metric,space walk hops, in terms of hops rather than latency.The space walk time is a more complete measurementthan a few thousands of random sample lookups be-cause space walk time represent lookup times to allpeers in the network.

We simulate experiments in three phases: an ini-tial phase, a stabilization phase, and an experimentphase. The initial phase builds the Chord ring ofa specified number of nodes, where nodes join thering at the rate of 50 nodes per second. The stabi-lization phase settles the Chord ring over 15 minutes

7

and establishes a stable baseline for the Chord rout-ing data structures. The experimental phase simu-lates the peer request workload and peer arrival anddeparture patterns for a specified duration. The sim-ulator collects results to evaluate the hint cachingtechniques only during the experimental phase.

Because membership churn is an important aspectof overlay networks, we study the performance ofthe hint caches using three different churn scenar-ios: twenty-four-hour, four-hour, and one-hour halflife times. The twenty-four-hour half life time repre-sent the churn in a distributed file system with manystable corporate/university peers [29]. The four-hourhalf life time represent the churn in a file sharing peer-to-peer network with many home users [19]. And theone-hour half life time represent extremely aggressiveworst-case churn scenarios [6].

For the simulations in this paper, we use an over-lay network of 8,192 peers with latency characteris-tics derived from real measurements. We start withthe latencies measured as part of the Vivaldi [3] eval-uation using the King [12] measurement technique.This data set has approximately 1,700 DNS servers,but only has complete all-pair latency informationfor 468 of the DNS servers. To simulate a larger net-work, for each one of these 468 DNS servers we createroughly 16 additional peers to represent peers in thesame stub networks as the DNS servers. We cre-ate these additional peers to form a network of 8,192peers. We model the latency among hosts within thegroup as zero to correspond to the minimal latencyamong hosts in the same network. We model the la-tency among hosts between groups according to themeasured latencies from the Vivaldi data set and werefer this data set as a “King”. The minimum, av-erage, and maximum latencies among groups are 2,165, and 795 milliseconds, respectively. As a timeoutvalue for detecting failed peers, we use a single roundtrip time to that peer (according to the optimizationsin [4]).

Using measurements to create the network modeladds realism to the evaluation. At the same time,though, the evaluation only scales to the limits ofthe measurements. To study the hint caches on sys-tems of much larger scale, we also performed exper-iments using another network model. We have two

different data sets in this model and both of themare significantly larger than above data set. First,we created a matrix of network latency among 8,192groups by randomly assigning a latency between twogroups from the range of 10 to 500 milliseconds anda latency within a group from the range of 1 to 5milliseconds. We then created an overlay network of262,144 peers by randomly assigning each peer to onegroup, keeping the groups balanced. We also createdanother data set with same latencies among groupsand with in a group for 65536 peers that are ran-domly distributed into 2,048 groups. We refer thesedata sets as a “Random”.

The goal of “Random” data set is to show the per-formance of various caches at large scale networksthat we can simulate on the resources that are avail-able to us. We used two different clusters [20, 28] toscale the computational resources. However, in someof the experiments we couldn’t parallelize our simu-lation easily hence we used smaller “Random” dataset with 65,536 peers.

5.2 Local Hint Caches

In this experiment we evaluate the performance of thelocal hint cache compared with two baseline routingalgorithms, “Standard” and “OneHop.” “Standard”is the default routing algorithm in Chord++ [4] thatoptimized for lookup latency by choosing nearest fin-gers. “OneHop” maintains complete routing tableson all peers [6].

Figures 1(a) and 1(b) show the cumulative distri-butions for the space walk latencies and hops, re-spectively, across all peers in the system. Since thereis no churn in this experiment, we calculate the la-tencies and hops after the network stabilizes whenreaching the experimental phase; we address churnin the next experiment. Figure 1(a) shows resultsfor “Standard” and “OneHop” and local hint cachesizes ranging from 64–1024 successors; Figure 1(b)omits “OneHop” since it only requires one hop for alllookups with stable routing tables.

Figure 1(a) shows that the local hint caches im-prove routing performance over the Chord baseline,and that doubling the cache size roughly improvesspace walk latency by a linear amount. The median

8

0

10

20

30

40

50

60

70

80

90

100

200 250 300 350 400 450 500 550 600 650 700 750 800 850Space Walk Latency (in M illiseconds)

Perc

enta

ge o

f Pee

rs

OneHopStandard64-Successors256-Successors1024-Successors

(a) Latency distributions

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5Spacw Walk Hops

Perc

enta

ge o

f Pee

rs

Standard64-Successors256-Successors1024-Successors

(b) Hop count distributions

Figure 1: Sensitivity of local hint cache size on lookupperformance for “King” data set.

space walk latency drops from 432 ms in Chord to355 ms with 1024 successors in the local hint cache(a decrease of 18%). Although an improvement, thelocal hint cache alone is still substantially slower than“OneHop”, which has a median space walk latency of273 ms (a decrease of 37% is needed).

Figure 1(b) shows similar behavior for local hintcaches in terms of hops. A local hint cache with 1024successors decreases median space walk latency by2.5 hops, although using such a cache still requiresone more hop than “OneHop”.

The performance local hint cache for “Random”data set in Figure 2(a) and Figure 2(b) is similar tothe performance of local hint cache for “King” dataset in Figure 1(a) and Figure 1(b) respectively. Weused 65,536 peers for the “Random” data set to be

0

10

20

30

40

50

60

70

80

90

100

450 500 550 600 650 700 750Space Walk Latency (in Milliseconds)

Perc

enta

ge o

f Pee

rs

OneHopStandard64-Successors256-Successors1024-Successors

(a) Latency distributions

0

10

20

30

40

50

60

70

80

90

100

2.4 2.8 3.2 3.6 4 4.4 4.8 5.2 5.6Space Walk Hops

Perc

enta

ge o

f Pee

rs

Standard64-Successors256-Successors1024-Successors

(b) Hop count distributions

Figure 2: Sensitivity of local hint cache size on lookupperformance for “Random” data set.

consistent with other Figures in this section.

From the graph, we see that doubling the local hintcache size improves the number of hops by at most0.5. Doubling the local hint cache size reduces hopcount by one for half of the peers, and the remaininghalf does not benefit from the increase. For exam-ple, consider a network of 100 peers where each peermaintains 50 other peers in its local hint cache. Foreach peer, 50 peers are one hop away and the other50 peers are two hops away. As a result, the spacewalk hop distance is 1.5 hops. If we increase the localhint cache to 100 peers, then each peer reduces thehop distance for only the 50 peers that were two hopsaway in the original scenario. In this case, the spacewalk hop distance is 1.

When there is no churn in the system, the lookup

9

performance when measured in terms of hops eitherremains the same or improves when we double thelocal hint cache size. The results are more compli-cated when we measure lookup performance in termsof latency. Most peers improves their lookup laten-cies to other peers and, on average, increasing localhint cache improves the space walk latency. However,lookup latency to individual peers can increase whenwe double the local hint cache size in some cases.This is because Internet routing does not necessar-ily follow the triangular inequality: routing throughmultiple hops may have lower latency than a directroute between two peers. Since we derive our net-work latency model from Internet measurements, ourlatency results reflect this characteristic of Internetrouting.

5.3 Staleness in Local Hint Caches

The previous experiment measured the benefit of us-ing the local hint cache in a stable network, and wenow measure the staleness in terms of stale entriesin the local hint cache and the effect of staleness onlookup performance.

In this experiment, we use a local hint cache sizeof 1024 successors. To calculate stale data in localhint caches, we ran the simulator with King data setand Random data set with 65,536 peers for an experi-mental phase of 30 minutes. During the experimentalphase, the network experiences churn in terms of peerjoins and leaves. We vary peer churn in the networkby varying the half life of peers in the system fromone hour to one day; we select nodes to join or leavefrom a uniform distribution.

Figure 3 shows the fraction of stale entries in lo-cal hint caches for various system half life times asa cumulative distribution across all peers. We calcu-lated the fraction of stale entries by sampling eachpeer’s local hint cache every second and determiningthe number of stale entries. We then averaged thesamples across the entire simulation run. Each point(x, y) on a curve indicates that y percentage of peershave at most x% stale data in their local hint caches.As expected, the amount of stale data increases asthe churn increases. Note that the amount of staledata is always less than the amount calculated from

0

10

20

30

40

50

60

70

80

90

100

0 6 12 18 24 30 36 42 48 54Stale Data (Number of Stale Entries)

Perc

enta

ge o

f Pee

rs

OneHourFourHoursOneDay

(a) King Data set

0

10

20

30

40

50

60

70

80

90

100

0 6 12 18 24 30 36 42 48 54 60 66 72Stale Data (Number of Stale Entries)

Perc

enta

ge o

f Pee

rsOneHourFourHoursOneDay

(b) Random Data set

Figure 3: Stale data distribution under various churnsituations for King and Random data sets.

our analysis in Section 3.2 since the analysis conser-vatively assumes worst case update synchronization.

Figure 4 shows the effect of stale local hint cacheentries on lookup performance across all peers forKing data set and Random data set with 32768 peers.It shows results for the same system half life times asFigure 3 and adds results for an “Infinite” half lifetime. An “Infinite” half life means that there is nochurn, no stale entries in the hint cache, and thereforerepresents the best-case latency. At the end of the ex-periment phase in the simulation, we used the stateof each peer’s routing table to calculate the distri-bution of space walk latencies across all peers. Eachpoint (x, y) in the figure indicates that y percentageof peers have at most x space walk latency. We cut offthe y-axis at 75% of peers to highlight the difference

10

0

10

20

30

40

50

60

70

80

200 250 300 350 400Space Walk Latency (in Milliseconds)

Perc

enta

ge o

f Pee

rs

InfiniteOneDayFourHoursOneHour

(a) King data set

0

10

20

30

40

50

60

70

80

90

100

520 540 560 580 600 620Space Walk Latency (in Milliseconds)

Perc

enta

ge o

f Pee

rs

InfiniteOneDayFourHoursOneHour

(b) Random data set

Figure 4: Space walk latency distributions under var-ious churn situations.

between the various curves.The space walk latencies for a four hour half life

time are similar to the latencies from the ideal casewith no churn (medians differ by only 1.35%). Fromthese results we conclude that the small amountof stale data (1–2%) does not significantly degradelookup performance, and that the local hint cache up-date mechanism maintains fresh entries well. As thechurn rate increases, stale data increases and lookupperformance also suffers. At an one hour half life,lookup performance increases moderately.

Note that the “Infinite” half life time curves in Fig-ure 4(a) and Figure 4(b) performs better than the1024 successors curve in Figure 1(a) and Figure 2(a)even though one would expect them to be the same.The reason they differ is that the finger table entriesin these two cases are different. When we evaluated

0

10

20

30

40

50

60

70

80

90

100

0 0.1 0.2 0.3 0.4 0.5Update Size (Num ber of Peers per Second)

Perc

enta

ge o

f Pee

rs

OneHourFourHoursOneDay

(a) King data set

0

10

20

30

40

50

60

70

80

90

100

0 0.1 0.2 0.3 0.4 0.5Update Size (Number of Peers per Second)

Perc

enta

ge o

f Pee

rs

OneHourFourHoursOneDay

(b) Random data set

Figure 5: Update traffic distribution under variouschurn situations.

the local hint cache, we used a routing table with13 successors and added the remaining successors tocreate the local hint cache without changing the fin-ger table. When we evaluated the stale data effectsin the local hint cache we have 1024 successors fromwhich to choose “nearest” fingers. As a result, theperformance is better.

5.4 Update Traffic

In the previous section we evaluated the effect of staledata on lookup performance under various churn sce-narios. In this section we evaluate the update trafficload under various churn scenarios to evaluate theupdate traffic bandwidth required by large local hint

11

caches.We performed a similar experiment as in Sec-

tion 5.3 for King data set and Random data set with65536 peers. However, instead of measuring staledata entries we measured the update traffic size. Wecalculated the average of all update samples per sec-ond for each peer over its lifetime in terms of thenumber of entries communicated. Figure 5 presentsthis average for all peers as cumulative distributions.Each curve in Figure 5 corresponds to a differentchurn scenario. A point (x, y) on each curve rep-resent the y percentage of peers that have at most xentries of average update traffic. The average updatetraffic closely matches the estimate from our analysisin Section3.2. The average update traffic (0.4 en-tries/second) is extremely low even under worst caseconditions. Hence, this traffic does not impose a bur-den on the system.

5.5 Path Hint Caches

Next we evaluate the performance of the path hintcache (PHC) described in Section 3.3 compared topath caching with expiration (PCX) as well as Chord.PCX is the technique of caching path entries de-scribed in [24]. When handling lookup requests onbehalf of other nodes, PCX caches route entries tothe initiator of the request as well as the result of thelookup.

In this experiment, we simulate a network of“King” data set with a 30-minute experimentalphase. We study the lower-bound effect of the pathhint caches in that we do not perform any applicationlevel lookups. Instead, the path hint caches are onlypopulated by traffic resulting from network stabiliza-tion. We did not simulate application level lookuptraffic to separate its effects on cache performance;with applications performing lookups, the path hintcaches may provide more benefit, although it willlikely be application-dependent. Since there is nochurn, cache entries never expire. To focus on theeffect of path caches only, we used a local hint cachesize of 13, the size of the standard Chord successorlist, and no global hint cache. We collected the rout-ing tables for all peers at the end of the simulationsand calculated the space walk latencies and hops.

0

10

20

30

40

50

60

300 350 400 450 500Space Walk Latency (in M illiseconds)

Perc

enta

ge o

f Pee

rs

StandardPHCPCX

(a) Latency Distribution

0

10

20

30

40

50

60

70

80

90

100

3 3.5 4 4.5Space Walk Hops

Perc

enta

ge o

f Pee

rsStandardPHCPCX

(b) Hop count distributions

Figure 6: Lookup performance of path caching withexpiration (PCX), path hint cache (PHC), and stan-dard Chord for “King” data set.

Figure 6(a) shows the cumulative distribution ofspace walk latencies across all peers at the end of thesimulation for the various path caches and standardChord. Each point (x, y) in this figure indicates thaty percent peers have at most x space walk latency.From these results we see that, as expected, the pathhint cache improves latency only marginally. How-ever, the path hint cache is essentially free, requiringno communication overhead and a small amount ofmemory to maintain.

We also see that PCX performs worse even thanChord. The reason for this is that PCX optimizesfor hops and caches routing information independentof the latency between the caching peer and the peerbeing cached. The latest version of Chord and our

12

0

10

20

30

40

50

60

70

80

90

100

600 650 700 750 800 850 900 950Space Walk Latency (in Milliseconds)

Perc

enta

ge o

f Pee

rs

StandardPHCPCX

(a) Latency Distribution

0

10

20

30

40

50

60

70

80

90

100

3.5 4 4.5 5 5.5Space Walk Hops

Perc

enta

ge o

f Pee

rs

StandardPHCPCX

(b) Hop count distributions

Figure 7: Lookup performance of path caching withexpiration (PCX), path hint cache (PHC), and stan-dard Chord for “Random” data set.

path hint caches use latency to determine what en-tries to place and use in the caches and in the routingtables. For peers with high latency, it is often bet-ter to use additional hops through low-latency peersthan fewer hops through high-latency peers.

Figure 6(b) shows this effect as well by presentingthe cumulative distribution of space walk hops acrossall peers for the various algorithms. Each point (x,y) in this figure indicates that y percent peers haveat most x space walk hops. Using the metric of hops,PCX performs better than both Chord and PHC.Similar to results in previous work incorporating la-tency into the analysis, these results again demon-strate that improving hop count does not necessarilyimprove latency. Choosing routing table and cache

0

10

20

30

40

50

60

70

80

90

100

200 250 300 350 400 450 500 550 600 650 700 750 800 850Space Walk Latency (in Milliseconds)

Perc

enta

ge o

f Pee

rs

StandardLocalCacheGlobalCache-32OneHopGlobalCache-256

(a) King data set

0

10

20

30

40

50

60

70

80

90

100

490 520 550 580 610 640 670 700 730 760 790 820Space Walk Latency (in ms)

Perc

enta

ge o

f Pee

rs

StandardLocalCacheGlobalCache-128OneHop

(b) Random data set

Figure 8: Global hint cache performance in differentnetwork models.

entries in terms of latency is important for improv-ing performance.

We did the same experiment replacing the ”King”data set with the “Random” data set with 65,536peers and Figures 7 shows the results of this exper-iment. These results are qualitatively similar to theresults of above “King” data set.

The path hint cache are small and each peer ag-gressively evicts the cache entries to minimize thestale data. Hence the effects of stale data on lookuprequest performance is marginal.

5.6 Global Hint Caches

Finally, we evaluate the performance of using theglobal hint cache together with the local and pathhint caches. We compare its performance with “Stan-

13

dard” Chord and “OneHop”. In this experiment, wesimulated 8,192 peers with both 32 and 256 entries intheir local hint caches. We have two different config-urations of local hint caches to compare the extent towhich the global hint cache benefits from having morecandidate peers from which to select nearby peers toplace in the global hint cache. (Note that the globalhint cache uses only the second half of the nodes inthe local hint cache to select nearest nodes; hence,the global hint cache uses only 16 and 128 entries tochoose nearest peers in the above two cases.) Eachpeer constructs its global cache when it joined thenetwork as described in Section 3.4. We collectedthe peer’s routing tables once the network reacheda stable state during the experimentation phase, andcalculated the space walk latencies for each peer fromthe tables.

Figure 8(a) shows the cumulative distributions ofspace walk latencies across all peers for the vari-ous algorithms. The “Standard”, “OneHop”, “Local-Cache”, “GlobalCache-32”, and “GlobalCache-256”curves represent Chord, the “OneHop” approach, a1024-entry local hint cache, a 32-entry global hintcache with a 256-entry local hint cache, and a 256-entry global hint cache with a 32-entry local hintcache. Comparing the size of local hint caches used topopulate the global hint cache, we find that the me-dian space walk latency of “GlobalCache-256”is 287ms and “GlobalCache-32” is 305 ms; the performanceof the global hint cache improved only 6% when ithas more peers in the local hint cache to choose thenearest peer.

We also did same experiment with Random dataset for 262,144 peers. We set the local hint cachesize to 128 instead of 1024 to grow the global hintcaches to few thousand peers which is the expectedglobal hint cache size when the network has few mil-lion peers. This is the maximum network size wecould able to simulate on our cluster machines. Atthis point, the main memory becomes the bottle-neckfor our simulation. The results are presented in Fig-ure 8(b) and the global hint cache performed closeto the “OneHop” approach (the median space walklatency of global hint cache is 2% more than the me-dian space walk latency of “OneHop” approach).

Comparing algorithms, we find that the median la-

tency of the global hint cache comes within 6% of the“OneHop” approach when the global hint cache uses128 of 256 entries in the local hint cache to choosenearby peers. Although these results are from a sta-ble system without churn, the amount of stale datain the global hint cache under churn is similar to thelocal hint cache under churn because both of themuse a similar update mechanism. Hence, the globalhint cache performance under churn is same or a littlebetter than the local hint cache performance underchurn because global hint cache is filled with the near-est peers as opposed to the random peers in local hintcache. As a result, the effect of stale data in the globalhint cache is negligible for a one-day system half lifetime and four-hour system half life time. Overall, oursoft-state approach approaches the lookup latency ofalgorithms like “OneHop” that use significantly morecommunication overhead to maintain complete rout-ing tables.

5.6.1 Global Hint Cache Build Time

Since we contact closer peers while constructing theglobal hint cache, one can build this cache within afew minutes. To demonstrate this, we calculated thetime to build the global hint cache for King data setwith 8,192 peers and Random data set with 262,144peers. Figure 9 presents the results of this experimentas a distribution of cache build times. Each point(x, y) on a curve indicates that y percentage of peersneeds at most x seconds to build the cache. Each peerhas around 500 peers in its the global hint caches and16 peers in its local hint cache for King data set andin the Random data set each peer has around 1500peers in the global hint cache and 128 peers in thelocal hint cache.

In the King data set, on the average it took 45seconds to build the global hint cache. A peer canspeed up this process by initiating walks from mul-tiple peers from its routing table in parallel. Thecurves labeled “Two”, “Four”, and “Eight” representthe cache build times with two, four, and eight paral-lel walks, respectively. As expected, cache build timereduces as we increase the number of parallel walks.The median reduces from 32 seconds for single walkto 12 seconds for four parallel walks. We see only

14

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150Time to Build Global Hint Cache (in seconds)

Perc

enta

ge o

f Pee

rs

OneTwoFourEight

(a) King data set

0

10

20

30

40

50

60

70

80

90

100

0 4 8 12 16 20 24 28 32 36 40Time to Build Global Hint Cache (in seconds)

Perc

enta

ge o

f Pee

rs

OneTwoFourEight

(b) Random data set

Figure 9: Global hint cache build time

a small benefit of increasing the parallel walks afterfour parallel walks.

The global hint cache build times for Random dataset has similar trends as the global hint cache buildtimes for King data set. Even though global hintcache sizes are bigger in the Random data set thebuild time is less comparing with the King data set.The Random data set has more peers in the local hintcache which improve the chances of finding a nearestpeer for a given peer which in turn improves the buildtime.

5.6.2 Network Coordinates

So far we have assumed that, when populating theglobal hint caches, peers are aware of the latenciesamong all other peers in the system. As a result, theresults represent upper bounds. In practice, peers

0

10

20

30

40

50

60

70

80

90

100

200 250 300 350 400 450 500 550 600 650 700Space Walk Latency (in Milliseconds)

Perc

enta

ge o

f Pee

rs

GlobalCache-32Vivaldy-32GlobalCache-256Vivaldy-256

Figure 10: Effects of Network Coordinates

will likely only track the latencies of other peers theycommunicate with, and not have detailed knowledgeof latencies among arbitrary peers. One way to solvethis problem is to use a distributed network coordi-nate system such as Vivaldi [3]. Of course, networkcoordinate systems introduce some error in the la-tency prediction. To study the effect of coordinatesystems for populating global hint caches, we nextuse Vivaldi to estimate peer latencies.

In our simulation we selected the nearest node ac-cording to network latency estimated according tothe Vivaldi network coordinate system, but calcu-lated the space walk time using actual network la-tency. We did this for the “GlobalCache-32” and“GlobalCache-256” curves in Figure 8. Figure 10shows these curves and the results using Vivaldi coor-dinates as “Vivaldi-32” and “Vivaldi-256”. The per-formance using the coordinate system decreases 6%on average in both cases, showing that the coordinatesystem performs well in practice.

5.7 Shun Pikes

In section 4 we described ways to exploit the detoursto improve the lookup performance even further thanthe “OneHop” approach. In this section we evaluatedthe shun pikes for the King data set. The latencydistributions between all pairs are presented in Fig-ure 11(a) and Figure 11(b) shows the performanceresults of our new lookup algorithm.

First we calculated the shortest path between all

15

0

10

20

30

40

50

60

70

80

90

100

0 50 100 150 200 250 300 350 400 450Latency (in Milliseconds)

Perc

enta

ge o

f Pee

rs

OneHopApprxOptimal

(a) Latency distributions

0

10

20

30

40

50

60

70

80

90

100

150 200 250 300 350 400 450 500 550 600 650Space Walk Latency (in Milliseconds)

Perc

enta

ge o

f Pee

rs

OneHopApprxOptimalGHC

(b) Space walk latency distributions

Figure 11: Effects of shun pikes

pair of peers in the data set assuming that each peerhas full knowledge of other peers. We just simply ranthe shortest path algorithm on network latency ma-trix. The result of this is presented in Figure 11(a) asthe “Optimal” labeled curve. We then calculated theshortest paths from each peer to all other peers as-suming each peer has 64 peers in its local hint cacheand approximately 64 peers in the global hint cache.The resultant latency distributed is represented as“Apprx” curve in Figure 11(a). The curve “OneHop”shows the the network latency distribution. Overallfor 40% of pairs shorest path latency improved overthe one hop latency. The “Apprx” and “Optimal”performed similarly for smaller latencies and differa bit at higher latencies. Though theoratically the“Apprx” should be constant factor away from the op-timal case in general Internet like graphs. However,

in this case we got similar performace as the optimalcase when each node has partial infomrationaboutthe enitre graph.

The Figure 11(b) shows the space walk latencyfor one hop, shortest path latencies, our new lookupalgorithm and our global and local hint caches .The “OneHop” and “Optimal” curves represent theone hop and shortest path latency approaches. The“GHC” curve represents the space walk latency ofour global hint cache along with the local hint cachethat is same as the “GlobalCache-256” curve in Fig-ure 8(a). The “Apprx” curve represents the spacewalk latency of our approach where each peer hasonly partial information about the entire networkand each peer tries to optimize the lookup as de-scribed in section 4. The median space latency forour caches, one hop, our new lookup algorithm andshortest paths is 287, 271, 262 and 228 milliseconds.Our new lookup algorithm improved approximately10% over our global and local hint caches and itimproved around 3% over one hop approach. Asexpected our lookup algorithm performed approxi-mately 15% slower than the optimal because we needone extra hop to complete the lookup. Overall ournew lookup algorithm performed a little better thanthe one hop approach.

6 Conclusions

In this paper, we describe and evaluate the use ofthree kinds of hint caches containing route hints toimprove the routing performance of distributed hashtables (DHTs): local hint caches store direct routesto neighbors in the ID space; path hint caches storedirect routes to peers accumulated during the naturalprocessing of lookup requests; and global hint cachesstore direct routes to a set of peers roughly uniformlydistributed across the ID space.

We simulate the effectiveness of these hint cachesas extensions to the Chord DHT. Based upon oursimulation results, we find that the combination ofhint caches significantly improves Chord routing per-formance with little overhead. For example, in net-works of 4,096 peers, the hint caches enable Chord toroute requests with average latencies only 6% more

16

than algorithms like “OneHop” that use completerouting tables, while requiring an order of magnitudeless bandwidth to maintain the caches and withoutthe complexity of a distributed update mechanism tomaintain consistency.

7 Acknowledgments

We would like thank the anonymous reviewers of ourWCW submission for their valuable feedback for im-proving this paper. We would also like to express ourgratitude to Marvin McNett for system support forperforming our simulation experiments, and to FrankDabek for providing us with the network latency in-formation used in all simulations. Support for thiswork was provided in part by AFOSR MURI Con-tract F49620-02-1-0233 and DARPA FTN ContractN66001-01-1-8933.

References

[1] R. Bhagwan, S. Savage, and G. M. Voelker. Un-derstanding availability. In 2nd InternationalWorkshop on Peer-to-Peer Systems, Feb. 2003.

[2] W. J. Bolosky, J. R. Douceur, D. Ely, andM. Theimer. Feasibility of a serverless dis-tributed file system deployed on an existing setof desktop pcs. In Proceedings of SIGMETRICS,June 2000.

[3] M. Castro, P. Druschel, A.-M. Kermarrec,A. Nandi, A. Rowstron, and A. Singh. Split-stream: High-bandwidth multicast in coopera-tive environments. In 19th ACM Symposium onOperating Systems Principles, Oct. 2003.

[4] R. Cox, F. Dabek, F. Kaashoek, J. Li, andR. Morris. Practical, distributed network coor-dinates. In proceedings of Second Workshop onHot Topics in Networks, Nov. 2003.

[5] F. Dabek, J. Li, E. Sit, J. Robertson, M. F.Kaashoek, and R. Morris. Designing a dhtfor low latency and high throughput. In

ACM/USENIX Symposium on Networked Sys-tems Design and Implementation, Mar. 2004.

[6] F. D. Emil Sit and J. Robertson. Usenetdht: Alow overhead usenet server. In 3rd InternationalWorkshop on Peer-to-Peer Systems, Feb. 2004.

[7] A. Gupta, B. Liskov, and R. Rodrigues. Ef-ficient routing for peer-to-peer overlays. InACM/USENIX Symposium on Networked Sys-tems Design and Implementation, Mar. 2004.

[8] I. Gupta, K. Birman, P. Linga, A. Demers, andR. van Renesse. Kelips: Building an efficientand stable p2p dht through increased memoryand background overhead. In 2nd InternationalWorkshop on Peer-to-Peer Systems, Feb. 2003.

[9] S. Iyer, A. Rowstron, and P. Druschel. Squir-rel: A decentralized and peer-to-peer web cache.In 21st ACM Symposium on Principles of Dis-tributed Computing, July 2002.

[10] F. Kaashoek and D. R. Karger. Koorde: A sim-ple degree-optimal hash table. In 2nd Interna-tional Workshop on Peer-to-Peer Systems, Feb.2003.

[11] M. F. Kaashoek and R. Morris.http://www.pdos.lcs.mit.edu/chord/.

[12] D. Kostic, A. Rodriguez, J. Albrecht, andA. Vahdat. Bullet: High bandwidth data dis-semination using an overlay mesh. In 19thACM Symposium on Operating Systems Princi-ples, Oct. 2003.

[13] S. S. Krishna P. Gummadi and S. D. Gribble.King: Estimating latency between arbitrary in-ternet end hosts. In 2nd Internet MeasurementWorkshop, Nov. 2002.

[14] J. Li, B. T. Loo, J. M. Hellerstein, M. F.Kaashoek, D. R. Karger, and R. Morris. Onthe feasibility of peer-to-peer web indexing andsearch. In 2nd International Workshop on Peer-to-Peer Systems, Feb. 2003.

17

[15] D. Liben-Nowell, H. Balakrishnan, andD. Karger. Observations on the dynamicevolution of peer-to-peer networks. In First In-ternational Workshop on Peer-to-Peer Systems,Mar. 2002.

[16] B. T. Loo, S. Krishnamurthy, and O. Cooper.Distributed web crawling over dhts. TechnicalReport UCB/CSD-4-1305, UC Berkeley, 2004.

[17] D. Malkhi, M. Naor, and D. Ratajczak. Viceroy:A scalable and dynamic emulation of the but-terfly. In 21st ACM Symposium on Principles ofDistributed Computing, July 2002.

[18] G. S. Manku, M. Bawa, and P. Raghavan. Sym-phony: Distributed hashing in a small world. In4th USENIX Symposium on Internet Technolo-gies and Systems, Mar. 2003.

[19] P. Maymounkov and D. Mazires. Kademlia: Apeer-to-peer information system based on thexor metric. In 1st International Workshop onPeer-to-Peer Systems, Mar. 2002.

[20] J. McCaleb. http://www.overnet.com/.

[21] M. McNett. https://ramp.ucsd.edu/group.

[22] A. Mizrak, Y. Cheng, V. Kumar, and S. Savage.Structured superpeers: Leveraging heterogene-ity to provide constant-time lookup. In IEEEWorkshop on Internet Applications, June 2003.

[23] V. Ramasubramanian and E. G. Sirer. Bee-hive: O(1) lookup performance for power-lawquery distributions in peer-to-peer overlays. InACM/USENIX Symposium on Networked Sys-tems Design and Implementation, Mar. 2004.

[24] S. Ratnasamy, P. Francis, M. Handley, R. Karp,and S. Shenker. A scalable content-addressablenetwork. In proceedings of ACM SIGCOMM,Aug. 2001.

[25] M. Roussopoulos and M. Baker. Cup: Con-trolled update propagation in peer-to-peer net-works. In USENIX Annual Technical Confer-ence, June 2003.

[26] A. Rowstron and P. Druschel. Pastry: Scalable,decentralized object location, and routing forlarge-scale peer-to-peer systems. In IFIP/ACMInternational Conference on Distributed SystemsPlatforms (Middleware), Nov. 2001.

[27] S. Savage, T. Anderson, A. Aggarwal, D. Becker,N. Cardwell, A. Collins, E. Hoffman, J. Snell,A. Vahdat, G. Voelker, and J. Zahorjan. De-tour: A case for informed internet routing andtransport. IEEE Micro, 19, 1999.

[28] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek,and H. Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications.In proceedings of ACM SIGCOMM, Aug. 2001.

[29] M. Thorup and U. Zwick. Compact routingschemes. In Proceedings of the thirteenth annualACM symposium on Parallel algorithms and ar-chitectures, 2001.

[30] C. UCSD. http://activeweb.ucsd.edu/.

[31] B. Y. Zhao, J. Kubiatowicz, and A. Joseph.Tapestry: An infrastructure for fault-tolerantwide-area location and routing. Technical Re-port UCB/CSD-01-1141, UC Berkeley, Apr.2001.

18