Dealing with Network Partitions and Mergers inStructured Overlay Networks
TALLAT MAHMOOD SHAFAAT
A Dissertation submitted tothe Royal Institute of Technology (KTH)
in partial fulfillment of the requirements forthe degree of Licentiate of Philosophy
May 2009
The Royal Institute of Technology (KTH)School of Information and Communication Technology
Department of Electronic, Computer, and Software SystemsStockholm, Sweden
TRITA-ICT/ECS AVH 09:01 KTH School of Information andISSN 1653-6363 Communication TechnologyISRN KTH/ICT/ECS/AVH-09/01-SE SE-100 44 StockholmISBN 978-91-7415-290-6 SWEDEN
Akademisk avhandling som med tillstånd av Kungl Tekniska högskolan framläggestill offentlig granskning för avläggande av teknologie licentiatesexamen i datalogifridagen den 8 May 2009 klockan 14.00 i sal D i Forum IT-Universitetet, KunglTekniska Högskolan, Isajordsgatan 39, Kista.
© Tallat Mahmood Shafaat, May 2009
Tryck: Universitetsservice US AB
i
Abstract
Structured overlay networks form a major class of peer-to-peer systems,which are touted for their abilities to scale, tolerate failures, and self-manage.Any long lived Internet-scale distributed system is destined to face networkpartitions. Although the problem of network partitions and mergers is highlyrelated to fault-tolerance and self-management in large-scale systems, it hashardly been studied in the context of structured peer-to-peer systems. Thesesystems have mainly been studied under churn (frequent joins/failures), whichas a side effect solves the problem of network partitions, as it is similar tomassive node failures. Yet, the crucial aspect of network mergers has beenignored. In fact, it has been claimed that ring-based structured overlay net-works, which constitute the majority of the structured overlays, are intrinsi-cally ill-suited for merging rings.
In this thesis, we present a number of research papers representing ourwork on handling network partitions and mergers in structured overlay net-works. The contribution of this thesis is threefold. First, we provide a solu-tion for merging ring-based structured overlays. Our solution is tuneable, bya fanout parameter, to achieve a trade-off between message and time complex-ity. Second, we provide a network size estimation algorithm for ring-basedstructured overlays. We believe that an estimate of the current network sizecan be used for tuning overlay parameters that change according to the net-work size, for instance the fanout parameter in our merger solution. Third, weextend our work from fixing routing anomalies to achieving data consistency.We argue that decreasing lookup inconsistencies on the routing level aids inachieving data consistency in applications built on top of overlays. We studythe frequency of occurence of lookup inconsistencies and discuss solutions todecrease the affect of lookup inconsistencies.
Keywords: Structured Overlay Networks, Distributed Hash Tables, DHTs,Network Partitions, Network Mergers, Network Size Estimation, Lookup In-consistencies, Gossiping
iii
Acknowledgements
I am extremely grateful to Ali Ghodsi for providing me enormous help and en-couragement during this thesis. His intellect, enthusiasm and approach to solvingproblems has been, and will always be, a source of inspiration for me. He notonly taught me how to do research properly, but also how to think logically aboutmany issues in life, thus being an excellent mentor. In research, he has providedme prodigious guidance from making me realize the importance of identifying aresearch problem, to improving my writing skills. I will definitely feel privileged tocontinue working with him.
I am also highly indebted to Prof. Seif Haridi for giving me the opportunityto work under his supervision. His expanse of knowledge and methodology ofsupervision is remarkable. Not only did I learn a lot from him, I also tremendouslyenjoyed my time as a student. I would like to thank Prof. Vladmimir Vlassovas well for providing valuable feedback in general and this thesis in particular.Furthermore, I would like to thank Sverker Janson for offering me the chance to bea member of CSL at SICS.
This research work was partially funded by the European project SELFMAN.I would like to express my gratitute to all of my colleagues involved in the project.
I would also like to thank my colleagues, both at KTH and SICS, for a lotof fruitful and conducive discussions; Ahmad Al-Shishtawy, Cosmin Arad, AmirPayberah, Fatemeh Rahimian, Joel Höglund and Jim Dowling. Special thanks tomy friends in Sweden; Kashif, Tahir, Magnus and Nilsson, for making my stay inSweden cherishable for the rest of my life.
Finally, I would like to dedicate this work to my parents, my sisters and brother.Their continuous support and belief in me has been a tremendous source of inspi-ration.
To my parents
Contents
I Thesis Overview 1
1 Introduction 31.1 Peer-to-peer Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Problem Definition 9
3 Structured Overlay Model 133.1 Motivation for the Unidirectional Ring Geometry . . . . . . . . . . . 133.2 A Model of a Ring-based Overlay . . . . . . . . . . . . . . . . . . . . 133.3 Dealing with Partitions and Failures in Chord . . . . . . . . . . . . . 14
4 Thesis Contribution 154.1 List of publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.2 Handling Network Partitions and Mergers . . . . . . . . . . . . . . . 164.3 Key-based Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . 174.4 Network Size Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 18
5 Conclusion 195.1 Handling Network Partitions and Mergers . . . . . . . . . . . . . . . 195.2 Key-based Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . 205.3 Network Size Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 205.4 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Bibliography 23
II Research Papers 29
6 Paper A : Handling Network Partitions and Mergers 31
7 Paper B : Key-based Consistency 55
8 Paper C : Network Size Estimation 71vii
Part I
Thesis Overview
1
Chapter 1
Introduction
With the advent of the Internet, many applications were developed to provide ITservices over the network. In essence, these applications build a distributed sys-tem where one or more computers, also known as nodes, provide some service toother computers over the network. This paradigm presented great challenges to re-searchers. The initial challenge was to make these systems scalable as the number ofnodes using the service increases. Furthermore, as Internet is spread geographicallyand uses various network components being managed by different administrators,failure of nodes and network links is a common norm in such systems. Thus, achiev-ing fault-tolerance in these systems is another vital research target.
The first approach to built such systems was a client-server paradigm. While theclient-server paradigm is still popular and effective, it has its drawbacks. The maindisadvantage of server-based systems is dependence on one (or a few) server(s)to provide service to a large number of clients. Using a single server leads to asingle point of failure and is obviously not scalable. Similarly, the machines used asservers have to be tremendously powerful in terms of network connectivity, storageand processing power to handle growing number of clients. Thus, using multipleservers is an expensive solution. This lead to finding other solutions, one of thembeing peer-to-peer systems.
In this chapter, we give a brief introduction of the peer-to-peer approach forbuilding large-scale distributed systems to provide services over the Internet. Al-though this approach can be used on any network infrastructure over which enti-ties/nodes can communicate, such as a wireless system, we use the Internet as areference in our discussions.
1.1 Peer-to-peer Systems
With the advancement of technology, network connectivity, storage and processingpower became cheaper. As a result, computers at the edge of the network, e.g.personal computers, became more powerful. This lead to the grand vision: usage
3
4 CHAPTER 1. INTRODUCTION
of resources available at the edge of the network. The realization of the grandvision resulted in systems that are now known as peer-to-peer systems. The basicidea of peer-to-peer (P2P) systems is that a node may act as both, a server and aclient. Thus, a node can use services provided by other nodes, while it also providesservices to other nodes.
Since many of the edge machines are less reliable compared to dedicated servers,achieving fault-tolerance becomes an even more non-trivial challenge in peer-to-peersystems. Furthermore, since there is no single point of control, edge machines canjoin and leave the system as they please. Thus, another crucial challenge is thatthe system should provide easy management, with machines coming up and goingdown all the time. This lead to an attractive feature of P2P systems, namelyself-management, where the system requires minimum manual configuration andmanagement.
One of the first peer-to-peer systems appeared in 1999, called Napster [36].Napster was mainly used for sharing music files. While Napster removed the burdenof hosting the shared files on the servers, it still used dedicated servers for theindexing service. The next challenge was to make a decentralized, scalable and fault-tolerant indexing service. This would enable a node to publish information aboutan item in a decentralized fashion. Similarly, a node would be able to find/lookupinformation about an item published earlier in the system. To achieve this, nodesthat are part of the system are connected to each other over the Internet insteadof connecting to the server. Thus, nodes have network connection informationabout some other nodes, called neighbours of the node, participating in the system.The information about neighbours of a node are stored locally in a data-structurecalled the routing table of the node. The routing table includes names and networkconnection information about neighbours, thus enabling a node to route messagesto other nodes. In essence, the routing tables of all nodes create a routing system ontop of the existing network infrastructure, e.g. the Internet. The overall networkrouting view created by routing tables of all the nodes is known as an overlaynetwork. Since an overlay uses the Internet to route messages, the Internet iscalled the underlay network. This is shown in Figure 1.1.
It is often convenient to view a peer-to-peer system as a graph. In the graph, thenodes in the overlay are represented as vertices. Similarly, the neighbours of a nodeare represented as edges. The shape of the graph depends on how the neighbours ofa node are selected in the overlay. Based on the shape of the graph, P2P systems areclassified into two broad categories: unstructured overlay networks and structuredoverlay networks. Figure 1.2 depicts this classification and is explained in thefollowing sections.
Unstructured Overlay NetworksAs the name suggests, in unstructured overlay networks, there is no particularstructure of the overlay, i.e. the graph induced by the nodes is unstructured.Gnutella [14] and Kazaa [24] are two popular examples of unstructured overlay
1.1. PEER-TO-PEER SYSTEMS 5
Underlay – e.g.
The Internet
Overlay –
The P2P Systema d
cb
1
7
5
4
32
6
Figure 1.1: An overlay network built on top of an underlay network. The over-lay consists of nodes a, b, c and d, while the underlay consists of components(nodes/routers/switches etc.) 1, 2, 3, 4, 5, 6 and 7. In the overlay, a can send amessage to a neighbour b. Such a message travels through the underlay components1, 3, and 4.
networks currently being used on the Internet. In Gnutella, a node has randomneighbours in the network that are changing all the time. To search for a dataitem, a node floods the network with the query by sending it to all its neighbours.Each node receiving the query forwards it to all its neighbours. Once the queryreaches a node that has the requested target data item, the data item is transferedto the querying node. Normally, a query contains a time-to-live entry so that theflooding process terminates at some point.
The main disadvantages of this approach are hampered scalability and guaran-tees on finding the data item. The scalability is hampered because flooding thenetwork with messages is costly, especially when there are millions of nodes. Simi-larly, when using a time-to-live, it might happen that the query terminates beforereaching the node that had the particular data item.
Structured Overlay NetworksIn structured overlay networks, a structure is induced by the neighbour pointers ofnodes. The structure is called the geometry of the structured overlay network. Astructured overlay network utilizes an identifier space. Nodes are assigned identifiersfrom this space, and each node is responsible for certain identifiers. The basicoperation that structured overlay networks offer is a lookup for an identifier. Theresult of a lookup for an identifier is the node responsible for the identifier.
6 CHAPTER 1. INTRODUCTION
(a) An Unstructured Overlay Network (b) A Structured Overlay Network
Figure 1.2: An unstructured overlay network, and a structured overlay networkwith a ring geometry.
Structured overlay networks have the attractive property that starting from anynode in the network, any other node is reachable in few steps (usually O(log(N)),where N is the number of nodes in the system). Structured overlay networks havethe additional desirable features of scalability and higher guarantees on finding apublished data item compared to unstructured overlay networks, while requiringminimum number of neighbours per node.
Distributed Hash Tables (DHTs) are a popular data-structure built on top ofstructured overlay networks. As the name suggests, DHTs provide an abstraction tostore data items under a key in the network. The data item can later be retrievedthrough the key that was used to store it. To achieve this, DHTs provide twooperations; put(k, v), to store a data item with value v under key k, and get(k), toretrieve a data item stored with key k. Both put and get use the overlays lookupoperation to reach the node responsible for the key k. The data item is then storedon/retrieved from the responsible node.
Chord [55] Pastry [45] CAN [43] One-hop DHT [25]Routing Table Size O(logN) O(logN) 2d N
Lookup Steps O(logN) O(logN) O(N 1d ) O(1)
Table 1.1: Comparison of properties of selected structured overlays. Here, N is thenetwork size.
Since structured overlay networks can have different geometries, different ap-proaches emerged to build a structured overlay. These approaches mainly differedin their geometry, while using different sizes of routing tables and number of stepsneeded for a lookup. Some of the popular geometries employed to build struc-
1.1. PEER-TO-PEER SYSTEMS 7
tured overlay networks are listed below, while Table 1.1 compares some popularstructured overlays.
1. Ring: Chord [55], DKS [13], Accordion [27]
2. Tree: Plaxton [41], P-Grid [2]
3. Torus: CAN [43]
4. XOR: Kademlia [32]
5. Hybrid: Pastry [45] (ring and tree)
A more detailed description on the working of a structured overlay is given inChapter 3.
Gossip/Epidemic AlgorithmsGossiping is an important technique used in large-scale distributed systems to solvemany problems. It has gained tremendous popularity in P2P systems because it isscalable, yet simple to use and robust to failures. In gossiping, information is spreadin the network similar to the way a rumor is spread, where continuous exchange ofa rumor between pairs of people results in its global spread. Gossip algorithms arealso referred to as epidemic algorithms since in gossiping, information is spread inthe system in a manner similar to the spread of a viral infection in a community.
Gossip algorithms are periodic, where in a period, each node chooses a randomnode in the system to gossip with. This gossip can be sending information only(push-only), receiving information only (pull-only) or an exchange of information(push-pull). It has been shown that given each node has access to random nodes inthe system, gossiping can be used to spread an information to all nodes in O(logN)steps, where N is the size of the network [40].
Gossip algorithms were first used by Demers et. al. [9] in 1987. They employedgossiping with a technique called anti-entropy to maintain a replicated database.In their solution, whenever a replica receives any changes, it starts to gossip thechanges with other replicas. This gossip spreads like an epidemic in the network.On receiving such a gossip, a replica can use anti-entropy to update its local statebased on the changes mentioned in the gossip, and resolve any inconsistencies.Thus, the replicated database remains updated and consistent.
Gossip algorithms have since been used for solving various problems. Some ofits usage in P2P systems are:
1. Disseminating information to all nodes in the system, such as broadcasting amessage [5].
2. Managing membership in an overlay to provide a node with continuous accessto random nodes in the system [56, 20].
8 CHAPTER 1. INTRODUCTION
3. Computing aggregates of values locally stored at all nodes, such as average,summation, maximum, and minimum [21].
4. Maintaining routing tables in structured overlay networks [25, 16].
5. Clustering nodes with similar properties or preferences [57].
General discussion on P2P systemsNowadays, one of the main usage of peer-to-peer systems on the Internet is con-
tent distribution, ranging from file sharing to watching live videos. BitTorrent [7]and eMule [10] are two popular file sharing systems. Their popularity is evidentfrom the fact that in January 2005, the total number of users of eMule reached 4million [11]. Similarly, PPLive [42] and Sopcast [54] are popular live video stream-ing P2P systems, with a reported 3.5 million daily active users of PPLive [18].
One would naturally wonder when to host a service using a P2P system [44].The main factors to consider before making such a decision are as follows:
• Budget: P2P systems are a cheap solution, thus, they should be preferredin case of a low budget. Otherwise, buying high-end powerful servers withqualified administrators to manage them should be the choice.
• Relevance: Since nodes in a P2P system act as both client and server, theyhave to contribute their resources to the system. Thus, the service providedshould be relevant for the users so that they have an incentive to contributetheir resources.
• Mutual trust: A node in the P2P system can be contributed by anyone.This leads to potential exploit room for nodes that are malicious. In such acase, a node can provide false service or no service at all. Thus, the decisionof using a P2P system should be based on the dependability of trust in thesystem, or the P2P system used should be able to tolerate such behaviour.
• Criticality: If the service to be provided is critical, using a P2P system isnot recommended.
Thesis organizationThis chapter provides an introduction to the thesis. Chapter 2 discusses the
problems addressed in this work. We present a model of a structured overlaynetwork that is used in this thesis in Chapter 3. Our contributions are summarizedin Chapter 4. Section 4.1 contains a list of publications made while working on thisthesis work and shorty discusses our work on addressing various issues mentioned inChapter 2. Next, Chapter 5 concludes, and presents some open challenges that areinteresting future directions to extend our work. Finally, we provide three selectedpapers from the list of publications in Chapter 6, 7 and 8.
Chapter 2
Problem Definition
Structured overlay networks and DHTs are touted for their ability to provide scala-bility, fault-tolerance, and self-management, making them well-suited for Internet-scale distributed applications. As these applications are long lived, they will alwayscome across network partitions. Since overlays1 are known to be fault-tolerant andself-manage, they have to be resilient to network partitions.
Although network partitions are not very common, they do occur. Internetfailures, resulting in partitioned networks can occur due to large area link fail-ures, router failures, physical damage to links/routers, routers misconfigurationand buggy software updates. Overloaded routers, network wide congestion due todenial of service (DoS) attacks and routing loops [39] can also have the same effectas a network partition. Similarly, natural disasters can result in Internet failures.This was observed when an earthquake in Taiwan in December 2006 exposed theissue that global traffic passes through a small number of seismically active “chokepoints” [37]. Several countries in the region connect to the outside world throughthese choke points. A number of similar events leading to Internet failures haveoccurred [6]. On a smaller scale, the aforementioned causes can disconnect an entireorganization from the Internet [38], thus partitioning the organization. Apart fromphysical failures, DoS (denial of service) might control enough nodes to effectivelypartition the overlay.
In fact, while deploying an overlay-based application on Internet, the first majorproblem reported and strongly suggested to be solved by Mislove et. al. [34] was:
A reliable decentralized system must tolerate network partitions.
While deploying ePOST [34] on PlanetLab, Mislove et. al. recorded the numberof partitions experienced over a period of 85 days. Figure 2.1 shows their results,which clearly suggests that partitions occur all the time.
1for the rest of this thesis, we use the word overlay for structured overlay network unlessspecified otherwise.
9
10 CHAPTER 2. PROBLEM DEFINITION
Figure 2.1: Number of connected components observed over an 85 day period onPlanetLab. Figure taken from ePOST [34].
Apart from network partitions, the problem of merging multiple overlays isinteresting and useful in itself. Decentralized bootstrapping of overlays [8] canbe done by building overlays separately and independently, regardless of existingoverlays. Later, these overlays can be merged into one. Also, it might be thatoverlays build independently later have to be merged due to overlapping interests.
Consequently, a crucial requirement for practical overlays is that they shouldbe able to deal with network partitions and mergers.
Some overlays cope with network partitions, but none can deal with networkmergers. This is because a network partition, as seen from the perspective of asingle node, is identical to massive node failures. Since overlays have been designedto cope with churn (node joins and failures), they can self-manage in the presence ofsuch partitions. However, most overlays cannot cope with network mergers, whichis the focus of this thesis work.
The merging of overlays gives rise to problems on two different levels: routinglevel and data level. The routing level is concerned with healing of the routinginformation after a partition merger. The data level is concerned with the consis-tency of the data items stored in the overlay. In this thesis, we focus mainly on theproblem at the routing level. We deal with the following issues:
• We address the problem of network partition and merger on the routing level,effectively fixing the routing pointers of the nodes in the overlay. We alsoaim to address scenarios to merge overlays that may arise other than networkpartitions. For instance, decentralized boot-strapping of overlays, and merger
11
of overlays that were created independently, yet later, they have to mergeddue to overlapping interests.
• We address the problem of estimation of the current network size in structuredoverlay networks. The network merging algorithm we provide as a solutionto the first problem has a fanout parameter to control the trade-off betweenmessage and time complexity (explained in Chapter 6). In a self-managingsystem, this fanout parameter can be selected according to the size of the net-work. Apart from its potential use in the network merger algorithm, networksize estimation in overlays is an interesting problem in itself with many us-age scenarios in structured overlays such as load-balancing, adjusting routingtable sizes, and tuning the rate of routing table maintenance.
• We address the problem of data inconsistencies that can arise due to lookupinconsistencies. As our solution for merging overlays focuses on the routinglevel, we believe that an interesting continuation of our work would be tofocus on the data level for consistency. As an initial step, we are interestedin knowing the frequency of occurrence of lookup inconsistencies, and issuesgiving rise to lookup inconsistencies. Furthermore, we want to investigatetechniques to reduce lookup inconsistencies. Since this thesis work mainlyfocuses on the routing level, the techniques we investigate to reduce lookupinconsistencies are also on the routing level. We aim to show how solutionson the routing level can reduce data inconsistencies in overlays.
Chapter 3
Structured Overlay Model
This thesis work focuses on ring-based structured overlay networks. Next, we mo-tivate this choice. Thereafter, we briefly discuss a model for a ring-based overlayused in this thesis work. As an example, we discuss how Chord [55], a ring-basedoverlay, overcomes partitions and failures in the overlay.
3.1 Motivation for the Unidirectional Ring Geometry
We confine ourselves to unidirectional ring-based overlays, such as Chord [55], Skip-Net [17], DKS [13], Koorde [23], Mercury [4], Symphony [31], EpiChord [26], andAccordion [28]. But we believe that our algorithms can be adapted easily to otherring-based overlays, such as Pastry [45]. For a more detailed account on direc-tionality and structure in overlays, please refer to Onana et al. [3] and Aberer etal. [1].
The reason for confining ourselves to ring-based overlays is twofold. First, ring-based overlays constitute a majority of the existing overlays. Second, Gummadi etal. [15] diligently compared the geometries of different overlays, and showed thatthe ring geometry is the one most resilient to failures, while it is just as good asthe other geometries when it comes to proximity.
To simplify the discussion, and presentation of our algorithms, we use notationthat indicates the use of the Chord [55] overlay. But the ideas are directly applicableto all unidirectional ring-based overlays.
3.2 A Model of a Ring-based Overlay
A ring-based overlay makes use of an identifier space, which for our purposes isdefined as a set of integers {0, 1, · · · ,N − 1}, where N is some apriori fixed, large,and globally known integer. This identifier space is perceived as a ring that wrapsaround at N − 1.
13
14 CHAPTER 3. STRUCTURED OVERLAY MODEL
Every node in the system has a unique identifier from the identifier space. Nodeidentifiers are typically assumed to be uniformly distributed on the identifier space.Each node keeps a pointer, succ, to its successor on the ring. The successor of anode with identifier p is the first node found going in clockwise direction on the ringstarting at p. Similarly, every node also has a pointer, pred, to its predecessor onthe ring. The predecessor of a node with identifier q is the first node met going inanti-clockwise direction on the ring starting at q. A successor-list is also maintainedat every node r, which consists of r’s c immediate successors, where c is typicallyset to log2(n), where n is the network size.
Ring-based overlays also maintain additional routing pointers on top of thering to enhance routing. To be concrete, assume that these are placed as in Chord.Hence, each node p keeps a pointer to the successor of the identifier p+2i ( mod N )for 0 ≤ i < log2(N ). Our algorithms can easily be adapted to other schemes forplacing these additional pointers.
3.3 Dealing with Partitions and Failures in Chord
Chord handles joins and leaves using a protocol called periodic stabilization. Leavesare handled by having each node periodically check whether pred is alive, andsetting pred := nil if it is found dead. Moreover, each node periodically checksto see if succ is alive. If it is found to be dead, it is replaced by the closest alivesuccessor in the successor-list.
Joins are also handled periodically. A joining node makes a lookup to find itssuccessor s on the ring, and sets succ := s. Each node periodically asks for itssuccessor’s pred pointer, and updates succ if it finds a closer successor. Thereafter,the node notifies its current succ about its own existence, such that the successorcan update its pred pointer if it finds that the notifying node is a closer predecessorthan pred. Hence, any joining node is eventually properly incorporated into thering.
As mentioned previously, a single node cannot distinguish massive simultaneousnode failures from a network partition. As periodic stabilization can handle massivefailures [29], it also recovers from network partitions, making each component ofthe partition eventually form its own ring. We have simulated such scenarios andconfirmed these results. The problem that remains unsolved, which is the focus ofour work, is how several independent rings can be efficiently merged.
Chapter 4
Thesis Contribution
In this chapter, we summarize the contributions of this thesis work. First, we listthe publications that were produced during this work. Afterwards, we provide asummary of our contribution for each problem area that we worked on.
4.1 List of publications
1. Tallat M. Shafaat, Ali Ghodsi, Seif Haridi. Dealing with Network Partitionsin Structured Overlay Networks. Journal of Peer-to-Peer Networking andApplications (PPNA), 2009 (To appear). DOI: 10.1007/s12083-009-0037-7
2. Tallat M. Shafaat, Ali Ghodsi, Seif Haridi. Managing Network Partitions inStructured P2P Networks. Book Chapter in X. Shen, H. Yu, J. Buford, andM. Akon, editors, Handbook of Peer-to-Peer Networking. Springer-Verlag,July 2009.
3. Tallat M. Shafaat, Ali Ghodsi, Seif Haridi. A Practical Approach to Net-work Size Estimation for Structured Overlays. In Karin Anna Hummel andJames P. G. Sterbenz, editors, Proceedings of the Third International Work-shop on Self- Organizing Systems (IWSOS’08), volume 5343 of Lecture Notesin Computer Science, pages 71–83. Springer-Verlag, 2008.
4. Tallat M. Shafaat, Monika Moser, Thorsten Schütt, Alexander Reinefeld, AliGhodsi, Seif Haridi. Key-Based Consistency and Availability in StructuredOverlay Networks. In Proceedings of the 3rd International ICST Conferenceon Scalable Information Systems (Infoscale’08). ACM, June 2008.- Two page poster version published in Proceedings of the 17th IEEE Sym-posium on High Performance Distributed Computing (HPDC’08), pages 235–236. ACM, 2008.
5. Tallat M. Shafaat, Monika Moser, Ali Ghodsi, Thorsten Schütt, Seif Haridi,Alexander Reinefeld. On Consistency of Data in Structured Overlay Net-
15
16 CHAPTER 4. THESIS CONTRIBUTION
works. In Proceedings of the 3rd CoreGRID Integration Workshop, April2008.
6. Tallat M. Shafaat, Ali Ghodsi, Seif Haridi. Handling Network Partitions andMergers in Structured Overlay Networks. In Proceedings of the 7th Inter-national Conference on Peer-to-Peer Computing (P2P’07), pages 132–139.IEEE Computer Society, September 2007.
4.2 Handling Network Partitions and Mergers
The work done for handling network partitions and mergers has been published ina conference paper [46], a journal paper [48] and a book chapter [49]. The journalpaper appears as Chapter 6 in this thesis.
In our work, we have motivated that handling underlying network partitions andmergers is a core requirement for structured overlays. We argue that since fault-tolerance, scalability and self-management are the basic properties of overlays, theyshould tolerate network partitions and mergers.
Our contribution is two-fold. First, we propose a mechanism for detecting ascenario where a partition occurred and later, the underlying network merged.Second, we propose two algorithms for merging overlays, simple ring unificationand gossip-based ring unification. Simple ring unification is a low-cost solutionwith respect to the number of messages sent (message complexity), yet it suffersfrom two problems: (1) slow convergence time (O(N) time for a network size ofN), and (2) less robustness to churn.
Gossip-based ring unification addresses both short-comings of simple ring unifi-cation, i.e. it has a high convergence rate (O(logN) time for a network size of N),and is robust to churn, yet it is a high-cost solution in terms of message complexity.In our solution, we provide a fanout parameter that can be used to control thetrade-off between message and time complexity in gossip-based ring unification. Acomparison of the two algorithms is given in Table 4.2.
We evaluate both algorithms extensively. Furthermore, we compare our solutionto a self-stabilizing algorithm presented by Shaker et. al. [53], as, a self-stabilizingalgorithm can be used to merge multiple overlays. The comparison is presentedin our journal paper [48], which also appears as Chapter 6 in this thesis. Thecomparison shows that our solution consumes lesser time and messages comparedto the self-stabilizing algorithm.
Lookups made after the merge is complete perform normally. An interestingissue is the behaviour of lookups made during the merger of the overlays. Suchlookups may not always succeed in finding the related data item. For instance, saythere are two overlays to be merged, O1 and O2, and a data item is stored withkey k in O1. Consider a node n initially part of O1, and a node m initially partof O2. During the merger of O1 and O2, a lookup made from n may end up onm. Though m is responsible for key k, it may not have yet received the data itemfor k as the merger has not completed at m. Thus, the data item for k will appear
4.3. KEY-BASED CONSISTENCY 17
unavailable to node n. After the merge is complete, n will be able to access k.A trivial solution to this problem is that when n learns that the key is currentlyunavailable, it retries the lookup after a while.
For some applications, availability may be critical. Thus, using retries after awhile might not be a feasible solution. To provide higher availability when themerge is taking place, a node should store both its old routing pointers (that itoriginally had in its overlay) and new routing pointers (assigned by the mergeprocess). Whenever a node receives a lookup, it routes the lookup through both, theold and new pointers. Thus, lookups will succeed even during the merge process.This approach, originally proposed by Datta [8], has some drawbacks. First, itincreases the message complexity for lookups. Second, it may be difficult to keeptrack of which overlay a node belongs to, especially when more than one overlaysare merged. Finally, since knowledge of the completion of the merge process isnot locally available to nodes, it is not known when to drop the old pointers.Nevertheless, we believe this to be an interesting problem, and approach, and leavea full analysis and solution as future work.
Related work on techniques of merging other overlays such as Pastry and P-Gridis discussed in our book chapter [48].
Simple Ring Unification Gossip-based Ring UnificationTime Complexity High LowMessage Complexity Low HighResilience to Churn Low High
Table 4.1: Comparison of Simple Ring Unification and Gossip-based Ring Unifica-tion.
4.3 Key-based Consistency
Our work on key-based consistency has been published as a conference paper [52], aworkshop paper [50] and a poster [51]. The conference paper appears as Chapter 7in this thesis.
In our work, we argue that it is nontrivial to provide consistent data serviceson top of structured overlays since key/identifier lookups can return inconsistentresults. We study the frequency of occurrence of lookup inconsistencies. We furtherpropose a solution to reduce lookup inconsistencies by assigning responsibilities ofkey intervals to nodes. As a side effect, our solution may lead to unavailability ofkeys. Thus, we present our results as a trade-off between consistency and availabil-ity.
Since many distributed algorithms employ quorum techniques, we extend ourwork by analyzing the probability that majority-based quorum techniques will func-tion correctly in overlays in spite of lookup inconsistencies.
18 CHAPTER 4. THESIS CONTRIBUTION
4.4 Network Size Estimation
The work done on network size estimation in ring-based structured overlays hasbeen published in a workshop [47] and appears as Chapter 8 in this thesis work.
Gossip-based aggregation [22] is known to be a highly accurate method of es-timating the current network size of an overlay [33]. In our work, we discuss theshort-comings of gossip-based aggregation for size estimation. We argue that themain disadvantage of gossip-based aggregation is that a failure of a single node earlyin an epoch can severely affect the estimate. Similarly, gossip-based aggregationrequires predefining the convergence time which may lead to inaccurate results.
Our contribution is an aggregation based solution that provides an estimate ofthe current network size for ring-based overlays. While deriving our solution, wealso address the disadvantages of aggregation by Jelasity et. al. [22]. We evaluateour solution extensively and show its effectiveness under churn and for variousnetwork sizes.
Chapter 5
Conclusion
In this chapter, we present our conclusions for the work done during this thesiswork. We present conclusions separately for each problem area that we addressed.Finally, we present some interesting future work to extend our research efforts.
5.1 Handling Network Partitions and Mergers
In our work on handling network partitions and mergers, we have argued that theproblem of partitions and mergers in structured peer-to-peer systems, when theunderlying network partitions and recovers, is of crucial importance. We presenta simple and a gossip-based algorithm for merging similar ring-based structuredoverlay networks after the underlying network merges.
Though we believe that the problem of dealing with network mergers is crucial,we think that such events happen more rarely. Hence, it might be justifiable incertain application scenarios that a slow paced algorithm runs in the background,consuming little resources, while ensuring that any potential problems with parti-tions will eventually be rectified. In such scenarios, our simple ring unification ismore suitable. If on the other hand, one would prefer to speed up the unificationprocess by consuming more messages, our gossip-based ring unification is more suit-able. We show how the algorithm can be tuned to achieve a tradeoff between thenumber of messages consumed and the time before the overlay converges. We eval-uate our solution in realistic dynamic conditions, and show that with high fanoutvalues, the algorithm can converge quickly under churn. We also show that our solu-tion generates few messages even if a node falsely starts the algorithm in an alreadyconverged overlay. Finally, we show that our algorithm recovers from pathologicalscenarios, such as loopy rings, which might result from network partitions.
19
20 CHAPTER 5. CONCLUSION
5.2 Key-based Consistency
In this work, we studied the frequency of lookup inconsistencies and found thatone of its main causes is inaccurate failure detectors. Hence, the choice of a fail-ure detection algorithm is of crucial importance in DHTs. While effects of lookupinconsistencies can be reduced by using local responsibilities, we show that usingresponsibility of keys may affect availability of keys. This is a trade-off betweenavailability and consistency. Many data dependent applications may prefer unavail-ability to inconsistency.
We also conclude that using quorum-based techniques amongst replicas of dataitems further reduce lookup inconsistencies. Since majority-based quorum tech-niques require a majority of the replicas to make progress, these algorithms maystill make progress even with unavailability of some keys/nodes. Thus, using acombination of local responsibilities and quorum techniques is attractive in scal-able applications.
Due to the dynamics and decentralization of overlays, it is difficult to buildabstractions with stronger consistency guarantees on top of overlays. Encouragedby our results, we propose to use techniques on the routing level to decrease datainconsistencies. These techniques can be used with techniques at the data level tofurther improve the results.
5.3 Network Size Estimation
Knowledge of the current network size of a structured peer-to-peer system is aprime requirement for many systems, which prompted us to finding solutions forsize estimation. Previous studies have shown that gossip-based aggregation algo-rithms [22], though being expensive, produce accurate estimates of the networksize [33]. We demonstrate the shortcomings in existing aggregation approaches tonetwork size estimation and present a solution that overcomes the deficiencies. Inour work, we argue for an adaptive approach to convergence in gossip-based aggre-gation algorithms. Our solution is resilient to massive node failures and is aimedto work on non-random topologies such as structured overlay networks.
5.4 Future Work
We believe that dealing with partitions and mergers is a small part of a bigger,and more important, goal: making structured overlays that can recover from anyconfiguration. We believe that it is desirable to make a self-stabilizing ring algo-rithm, which can be proved to recover from all possible states, including loopy andpartitioned while consuming minimum time and messages.
We believe that it is interesting to investigate whether gossip-based topologygenerators, such as T-man [19] and T-chord [35], can be used to handle networkmergers. These services, however, make use of an underlying membership service,
5.4. FUTURE WORK 21
such as Cyclon [56], Scamp [12], or Newscast [20]. Hence, one has to first investigatehow well such membership services recover from network partitions (we believe thisto be interesting in itself). Thereafter, one can explore how such topology generatorscan be incorporated into an overlay.
Another future direction of our work would be to perform a theoretical anal-ysis of the ring unification algorithms. It will be interesting to prove the conver-gence of the algorithms into one merged ring, starting from any number of rings.Furthermore, an analysis of the time complexity can provide bounds on time tillconvergence.
Investigation of the affect of partitions and mergers on the data level is an-other possible extension of our work. In this regard, the goal is to achieve someguarantees on the consistency and availability of the data. As our initial work onkey-based consistency indicates, we would like to achieve a tunable trade-off be-tween consistency and availability. Finally, we believe that studying the behaviourof lookups during the merge process is important for achieving higher availabilityof keys. Thus, solutions to improve availability of keys during the merge process isanother future direction.
As part of our work on network size estimation, we plan to investigate a concretecorrelation between the fanout parameter used in gossip-based ring unification andthe estimated network size. Furthermore, our work can be extended to be used fornon-ring based overlays, such as overlays using the XOR-metric [32] and butterflynetworks [30].
Bibliography
[1] K. Aberer, L. O. Alima, A. Ghodsi, S. Girdzijauskas, S. Haridi, andM. Hauswirth. The Essence of P2P: A Reference Architecture for OverlayNetworks. In Proceedings of the 5th International Conference on Peer-To-PeerComputing (P2P’05), pages 11–20. IEEE Computer Society, August 2005.
[2] K. Aberer, P. Cudré-Mauroux, A. Datta, Z. Despotovic, M. Hauswirth,M. Punceva, and R. Schmidt. P-Grid: a self-organizing structured P2P system.SIGMOD Record, 32(3):29–33, 2003.
[3] L. O. Alima, A. Ghodsi, and S. Haridi. A Framework for Structured Peer-to-Peer Overlay Networks. In Post-proceedings of Global Computing, volume3267 of Lecture Notes in Computer Science (LNCS), pages 223–250. SpringerVerlag, 2004.
[4] A. R. Bharambe, M. Agrawal, and S. Seshan. Mercury: Supporting ScalableMulti-Attribute Range Queries. In Proceedings of the ACM SIGCOMM 2004Symposium on Communication, Architecture, and Protocols, pages 353–366,Portland, OR, USA, March 2004. ACM Press.
[5] K. P. Birman, M. Hayden, O. Ozkasap, Z. Xiao, M. Budiu, and Y. Min-sky. Bimodal Multicast. ACM Transactions on Computer Systems (TOCS),17(2):41–88, 1999.
[6] F. Jahanian C. Labovitz, A. Ahuja. Experimental Study of Internet Stabilityand Wide-Area Backbone Failures. Technical Report CSE-TR-382-98, Univer-sity of Michigan, November 1998.
[7] B. Cohen. Incentives Build Robustness in BitTorrent. In First Workshop onEconomics of Peer-to-Peer Systems, Berkeley, CA, USA, June 2003.
[8] A. Datta. Merging Intra-Planetary Index Structures: Decentralized Boot-strapping of Overlays. In Proceedings of the First International Conferenceon Self-Adaptive and Self-Organizing Systems (SASO 2007), pages 109–118,Boston, MA, USA, July 2007. IEEE Computer Society.
23
24 BIBLIOGRAPHY
[9] A. Demers, D. Greene, C. Hauser, W. Irish, J. Larson, S. Shenker, H. Sturgis,D. Swinehart, and D. Terry. Epidemic Algorithms for Replicated DatabaseMaintenance. In Proceedings of the 7th Annual ACM Symposium on Principlesof Distributed Computing (PODC’87), pages 1–12, New York, NY, USA, 1987.ACM Press.
[10] eMule. http://www.emule-project.net/, March 2009.
[11] eMule Statistics. http://www.infoanarchy.org/en/EMule, March 2009.
[12] A. J. Ganesh, A.-M. Kermarrec, and L Massoulié. SCAMP: Peer-to-PeerLightweight Membership Service for Large-Scale Group Communication. InProceedings of the 3rd International Workshop on Networked Group Communi-cation (NGC’01), volume 2233 of Lecture Notes in Computer Science (LNCS),pages 44–55, London, UK, 2001. Springer-Verlag.
[13] A. Ghodsi. Distributed k-ary System: Algorithms for Distributed Hash Tables.PhD dissertation, KTH—Royal Institute of Technology, Stockholm, Sweden,December 2006.
[14] Gnutella. http://www.gnutella.com, 2006.
[15] K. Gummadi, R. Gummadi, S. Gribble, S. Ratnasamy, S. Shenker, and I. Sto-ica. The impact of DHT routing geometry on resilience and proximity. InProceedings of the ACM SIGCOMM 2003 Symposium on Communication, Ar-chitecture, and Protocols, pages 381–394, New York, NY, USA, 2003. ACMPress.
[16] I. Gupta, K. Birman, P. Linga, A. Demers, and R. van Renesse. Kelips:Building an Efficient and Stable P2P DHT Through Increased Memory andBackground Overhead. In Proceedings of the 2nd International Workshop onPeer-to-Peer Systems (IPTPS’03), volume 2735 of Lecture Notes in ComputerScience (LNCS), pages 160–169, Berkeley, CA, USA, 2003. Springer-Verlag.
[17] N. Harvey, M. B. Jones, S. Saroiu, M. Theimer, and A. Wolman. Skipnet: Ascalable overlay network with practical locality properties. In Proceedings of the4th USENIX Symposium on Internet Technologies and Systems (USITS’03),Seattle, WA, USA, March 2003. USENIX.
[18] G. Huang. Experiences with PPLive. Key note talk at Peer-to-Peer Streamingand IP-TV Workshop Invited held with ACM Sigcomm, Kyoto, Japan, August2007.
[19] M. Jelasity and Ö. Babaoglu. T-man: Gossip-based overlay topology manage-ment. In Proceedings of 3rd Workshop on Engineering Self-Organising Systems(EOSA’05), volume 3910 of Lecture Notes in Computer Science (LNCS), pages1–15. Springer-Verlag, 2005.
25
[20] M. Jelasity, W. Kowalczyk, and M. van Steen. Newscast Computing. TechnicalReport IR–CS–006, Vrije Universiteit, November 2003.
[21] M. Jelasity and A. Montresor. Epidemic-Style Proactive Aggregation in LargeOverlay Networks. In Proceedings of the 24th International Conference onDistributed Computing Systems (ICDCS’04), pages 102–109, Tokyo, Japan,March 2004. IEEE Computer Society.
[22] M. Jelasity, A. Montresor, and Ö. Babaoglu. Gossip-based Aggregation inLarge Dynamic Networks. ACM Transactions on Computer Systems (TOCS),23(3), August 2005.
[23] M. F. Kaashoek and D. R. Karger. Koorde: A Simple Degree-optimal Dis-tributed Hash Table. In Proceedings of the 2nd Interational Workshop onPeer-to-Peer Systems (IPTPS’03), volume 2735 of Lecture Notes in ComputerScience (LNCS), pages 98–107, Berkeley, CA, USA, 2003. Springer-Verlag.
[24] Kazaa. http://www.kazaa.com/, March 2009.
[25] B. Leong and J. Li. Achieving One-Hop DHT Lookup and Strong Stabilizationby Passing Tokens. In 12th International Conference on Networks (ICON’04),Singapore, November 2004. IEEE Computer Society.
[26] B. Leong, B. Liskov, and E. Demaine. EpiChord: Parallelizing the ChordLookup Algorithm with Reactive Routing State Management. In 12th In-ternational Conference on Networks (ICON’04), Singapore, November 2004.IEEE Computer Society.
[27] J. Li. Routing Tradeoffs in Dynamic Peertopeer Networks. PhD dissertation,MIT—Massachusetts Institute of Technology, Massachusetts, USA, November2005.
[28] J. Li, J. Stribling, R. Morris, and M. F. Kaashoek. Bandwidth-efficient man-agement of DHT routing tables. In Proceedings of the 2nd USENIX Symposiumon Networked Systems Design and Implementation (NSDI’05), Boston, MA,USA, May 2005. USENIX.
[29] D. Liben-Nowell, H. Balakrishnan, and D. R. Karger. Observations on theDynamic Evolution of Peer-to-Peer Networks. In Proceedings of the First In-ternational Workshop on Peer-to-Peer Systems (IPTPS’02), volume 2429 ofLecture Notes in Computer Science (LNCS). Springer-Verlag, 2002.
[30] D. Malkhi, M. Naor, and D. Ratajczak. Viceroy: A scalable and dynamicemulation of the butterfly. In Proceedings of the 21st Annual ACM Symposiumon Principles of Distributed Computing (PODC’02), New York, NY, USA,2002. ACM Press.
26 BIBLIOGRAPHY
[31] G. S. Manku, M. Bawa, and P. Raghavan. Symphony: Distributed Hashingin a Small World. In Proceedings of the 4th USENIX Symposium on Inter-net Technologies and Systems (USITS’03), Seattle, WA, USA, March 2003.USENIX.
[32] P. Maymounkov and D. Mazieres. Kademlia: A Peer-to-Peer InformationSystem Based on the XOR metric. In Proceedings of the First InterationalWorkshop on Peer-to-Peer Systems (IPTPS’02), Lecture Notes in ComputerScience (LNCS), pages 53–65, London, UK, 2002. Springer-Verlag.
[33] E. Le Merrer, A.-M Kermarrec, and L. Massoulie. Peer to peer size estimationin large and dynamic networks: A comparative study. In Proc. of the 15thIEEE Symposium on High Performance Distributed Computing, pages 7–17.IEEE, 2006.
[34] A. Mislove, A. Post, A. Haeberlen, and P. Druschel. Experiences in build-ing and operating ePOST, a reliable peer-to-peer application. In WillyZwaenepoel, editor, Proceedings of the 1st ACM SIGOPS/EuroSys EuropeanConference on Computer Systems. ACM European Chapter, April 2006.
[35] A. Montresor, M. Jelasity, and Ö. Babaoglu. Chord on Demand. In Proceedingsof the 5th International Conference on Peer-To-Peer Computing (P2P’05).IEEE Computer Society, August 2005.
[36] Napster. http://www.napster.com, 2006.
[37] Taiwan Earthquake on December 2006. http://www.pinr.com/report.php?ac=view_report\&report_id=602, January 2008.
[38] D. Oppenheimer, A. Ganapathi, and D. A. Patterson. Why do internet ser-vices fail, and what can be done about it? In USITS’03: Proceedings of the4th conference on USENIX Symposium on Internet Technologies and Systems,pages 1–1, Berkeley, CA, USA, 2003. USENIX Association.
[39] V. Paxson. End-to-end routing behavior in the Internet. IEEE/ACM Trans-actions on Networking (TON), 5(5):601–615, 1997.
[40] B. Pittel. On spreading a rumor. SIAM Journal on Applied Mathematics,47(1):213–223, Feb 1987.
[41] C. G. Plaxton, R. Rajaraman, and A. W. Richa. Accessing nearby copiesof replicated objects in a distributed environment. In Proceedings of the9th Annual ACM Symposium on Parallelism in Algorithms and Architectures(SPAA’97), pages 311–320, New York, NY, USA, 1997. ACM Press.
[42] PPLive. http://www.pplive.com/, March 2009.
27
[43] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. A ScalableContent-Addressable Network. In Proceedings of the ACM SIGCOMM 2001Symposium on Communication, Architecture, and Protocols, pages 161–172,San Diego, CA, U.S.A., August 2001. ACM Press.
[44] M. Roussopoulos, M. Baker, D. S. H. Rosenthal, T. J. Giuli, P. Maniatis, andJ. C. Mogul. 2 P2P or Not 2 P2P? In Proceedings of the Third InterationalWorkshop on Peer-to-Peer Systems (IPTPS’04), Lecture Notes in ComputerScience (LNCS), pages 33–43, La Jolla, CA, USA, 2004. Springer-Verlag.
[45] A. Rowstron and P. Druschel. Pastry: Scalable, distributed object locationand routing for large-scale peer-to-peer systems. In Proceedings of the 2ndACM/IFIP International Conference on Middleware (MIDDLEWARE’01),volume 2218 of Lecture Notes in Computer Science (LNCS), pages 329–350,Heidelberg, Germany, November 2001. Springer-Verlag.
[46] T. M. Shafaat, A. Ghodsi, and S. Haridi. Handling Network Partitions andMergers in Structured Overlay Networks. In Proceedings of the 7th Interna-tional Conference on Peer-to-Peer Computing (P2P’07), pages 132–139. IEEEComputer Society, September 2007.
[47] T. M. Shafaat, A. Ghodsi, and S. Haridi. A practical approach to network sizeestimation for structured overlays. In Karin Anna Hummel and James P. G.Sterbenz, editors, Proceedings of the Third International Workshop on Self-Organizing Systems (IWSOS’08), volume 5343 of Lecture Notes in ComputerScience, pages 71–83. Springer-Verlag, 2008.
[48] T. M. Shafaat, A. Ghodsi, and S. Haridi. Dealing with network partitionsin structured overlay networks. Peer-to-Peer Networking and Applications(PPNA), 2009.
[49] T. M. Shafaat, A. Ghodsi, and S. Haridi. Managing Network Partitions inStructured P2P Networks. In X. Shen, H. Yu, J. Buford, and M. Akon, editors,Handbook of Peer-to-Peer Networking. Springer-Verlag, July 2009.
[50] T. M. Shafaat, M. Moser, A. Ghodsi, T. Schütt, S. Haridi, and A. Reinefeld.On consistency of data in structured overlay networks. In Proceedings of the3rd CoreGRID Integration Workshop, April 2008.
[51] T. M. Shafaat, M. Moser, A. Ghodsi, T. Schütt, S. Haridi, and A. Reinefeld.Poster: key-based consistency and availability in structured overlay networks.In Proc. of the 17th IEEE Symposium on High Performance Distributed Com-puting, pages 235–236. ACM, 2008.
[52] T. M. Shafaat, M. Moser, T. Schütt, A. Reinefeld, A. Ghodsi, and S. Haridi.Key-Based Consistency and Availability in Structured Overlay Networks. InProceedings of the 3rd International ICST Conference on Scalable InformationSystems (Infoscale’08). ACM, June 2008.
28 BIBLIOGRAPHY
[53] A. Shaker and D. S. Reeves. Self-Stabilizing Structured Ring Topology P2PSystems. In Proceedings of the 5th International Conference on Peer-To-PeerComputing (P2P’05), pages 39–46. IEEE Computer Society, August 2005.
[54] SopCast. http://www.sopcast.com/, March 2009.
[55] I. Stoica, R. Morris, D. Liben-Nowell, D. R. Karger, M. F. Kaashoek, F. Dabek,and H. Balakrishnan. Chord: a scalable peer-to-peer lookup protocol for inter-net applications. IEEE/ACM Transactions on Networking (TON), 11(1):17–32, 2003.
[56] S. Voulgaris, D. Gavidia, and M. van Steen. Cyclon: Inexpensive membershipmanagement for unstructured p2p overlays. Journal of Network and SystemsManagement, 13(2), 2005.
[57] S. Voulgaris and M. van Steen. Epidemic-Style Management of Semantic Over-lays for Content-Based Searching. In Proceedings of the 11th European Con-ference on Parallel Computing (EUROPAR’05). Springer-Verlag, 2005.
