+ All Categories
Home > Documents > Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in...

Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in...

Date post: 24-Jul-2020
Category:
Upload: others
View: 0 times
Download: 0 times
Share this document with a friend
23
Safari: A self-organizing, hierarchical architecture for scalable ad hoc networking Shu Du, Ahamed Khan, Santashil PalChaudhuri, Ansley Post, Amit Kumar Saha, Peter Druschel, David B. Johnson * , Rudolf Riedi Rice University, Department of Computer Science, 6100 Main Street, MS 132, Houston, TX 77005-1892, USA Received 14 October 2006; received in revised form 8 April 2007; accepted 9 April 2007 Available online 24 May 2007 Abstract As wireless devices become more pervasive, mobile ad hoc networks are gaining importance, motivating the develop- ment of highly scalable ad hoc networking techniques. In this paper, we give an overview of the Safari architecture for highly scalable ad hoc network routing, and we present the design and evaluation of a specific realization of the Safari architecture, which we call Masai. We focus in this work on the scalability of learning and maintaining the routing state necessary for a large ad hoc network. The Safari architecture provides scalable ad hoc network routing, the seamless inte- gration of infrastructure networks when and where they are available, and the support of self-organizing, decentralized network applications. Safari’s architecture is based on (1) a self-organizing network hierarchy that recursively groups par- ticipating nodes into an adaptive, locality-based hierarchy of cells; (2) a routing protocol that uses a hybrid of proactive and reactive routing information in the cells and scales to much larger numbers of nodes than previous ad hoc network routing protocols; and (3) a distributed hash table grounded in the network hierarchy, which supports decentralized net- work services on top of Safari. We evaluate the Masai realization of the Safari architecture through analysis and simula- tions, under varying network sizes, fraction of mobile nodes, and offered traffic loads. Compared to both the DSR and the L+ routing protocols, our results show that the Masai realization of the Safari architecture is significantly more scalable, with much higher packet delivery ratio and lower overhead. Ó 2007 Elsevier B.V. All rights reserved. Keywords: Ad hoc network routing; Peer-to-peer networking; Landmark routing; Hierarchical routing; Reactive routing; Proactive routing; Hybrid routing; Safari; Masai 1. Introduction In an ad hoc network, individual, potentially mobile nodes cooperate to form a network without the aid of existing infrastructure such as wireless base stations or access points. Instead, each mobile node acts not only as a host but also as a router, for- warding packets for other mobile nodes, to allow 1570-8705/$ - see front matter Ó 2007 Elsevier B.V. All rights reserved. doi:10.1016/j.adhoc.2007.04.007 * Corresponding author. Tel.: +1 713 348 3063; fax: +1 713 348 5930. E-mail addresses: [email protected] (S. Du), ahamed_khan@ yahoo.com (A. Khan), [email protected] (S. PalChaudhuri), [email protected] (A. Post), [email protected] (A.K. Saha), [email protected] (P. Druschel), [email protected] (D.B. Johnson), [email protected] (R. Riedi). Available online at www.sciencedirect.com Ad Hoc Networks 6 (2008) 485–507 www.elsevier.com/locate/adhoc
Transcript
Page 1: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

Available online at www.sciencedirect.com

Ad Hoc Networks 6 (2008) 485–507

www.elsevier.com/locate/adhoc

Safari: A self-organizing, hierarchical architecture for scalablead hoc networking

Shu Du, Ahamed Khan, Santashil PalChaudhuri, Ansley Post, Amit Kumar Saha,Peter Druschel, David B. Johnson *, Rudolf Riedi

Rice University, Department of Computer Science, 6100 Main Street, MS 132, Houston, TX 77005-1892, USA

Received 14 October 2006; received in revised form 8 April 2007; accepted 9 April 2007Available online 24 May 2007

Abstract

As wireless devices become more pervasive, mobile ad hoc networks are gaining importance, motivating the develop-ment of highly scalable ad hoc networking techniques. In this paper, we give an overview of the Safari architecture forhighly scalable ad hoc network routing, and we present the design and evaluation of a specific realization of the Safariarchitecture, which we call Masai. We focus in this work on the scalability of learning and maintaining the routing statenecessary for a large ad hoc network. The Safari architecture provides scalable ad hoc network routing, the seamless inte-gration of infrastructure networks when and where they are available, and the support of self-organizing, decentralizednetwork applications. Safari’s architecture is based on (1) a self-organizing network hierarchy that recursively groups par-ticipating nodes into an adaptive, locality-based hierarchy of cells; (2) a routing protocol that uses a hybrid of proactiveand reactive routing information in the cells and scales to much larger numbers of nodes than previous ad hoc networkrouting protocols; and (3) a distributed hash table grounded in the network hierarchy, which supports decentralized net-work services on top of Safari. We evaluate the Masai realization of the Safari architecture through analysis and simula-tions, under varying network sizes, fraction of mobile nodes, and offered traffic loads. Compared to both the DSR and theL+ routing protocols, our results show that the Masai realization of the Safari architecture is significantly more scalable,with much higher packet delivery ratio and lower overhead.� 2007 Elsevier B.V. All rights reserved.

Keywords: Ad hoc network routing; Peer-to-peer networking; Landmark routing; Hierarchical routing; Reactive routing; Proactiverouting; Hybrid routing; Safari; Masai

1570-8705/$ - see front matter � 2007 Elsevier B.V. All rights reserved

doi:10.1016/j.adhoc.2007.04.007

* Corresponding author. Tel.: +1 713 348 3063; fax: +1 713 3485930.

E-mail addresses: [email protected] (S. Du), [email protected] (A. Khan), [email protected] (S. PalChaudhuri),[email protected] (A. Post), [email protected] (A.K.Saha), [email protected] (P. Druschel), [email protected](D.B. Johnson), [email protected] (R. Riedi).

1. Introduction

In an ad hoc network, individual, potentiallymobile nodes cooperate to form a network withoutthe aid of existing infrastructure such as wirelessbase stations or access points. Instead, each mobilenode acts not only as a host but also as a router, for-warding packets for other mobile nodes, to allow

.

Page 2: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

486 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

nodes to communicate even if they are not directlywithin radio transmission range of each other. Thisinfrastructure independence makes ad hoc networksvery useful in many scenarios such as disaster reliefefforts, battlefields communications, and networkconnectivity in economically disadvantaged areasof the world.

With the rapid proliferation of wireless devices,the use of ad hoc networking is expected to grow,and with it, the size of ad hoc networks that maybe created. At the same time, the field of decentral-ized, self-organizing distributed systems has seen sig-nificant advances in recent years and has opened newalternatives in providing ad hoc network services.Work on these two areas has in the past proceededlargely independently. Our Safari architecture bringstogether these two areas, aiming to create a frame-work for protocols and algorithms that provideslarge-scale ad hoc network connectivity, seamlesslyintegrated with infrastructure networks when andwhere they are available, supporting mobile and sta-tionary nodes, together with decentralized networkservices. Safari exploits synergies among ad hocnetworking and decentralized distributed systemsresearch.

In this paper, we give an overview of our Safari

architecture, and we present the design and evalua-tion of a specific realization of the Safari architec-ture, which we call Masai. We focus in this workon the scalability of learning and maintaining therouting state necessary for a large ad hoc network.Masai consists of a scalable routing protocol andan automatic self-organizing hierarchy formationprotocol on which the routing is built. Routing ofpackets in the network is guided by this hierarchy,and is capable of scaling to large numbers of mobilenodes. We assume that nodes in the ad hoc networkare willing to cooperate with each other. Manynodes may be power constrained, but for example,stationary nodes may not be; we strive, however,to make the protocol efficient in its network usage,as doing so conserves network bandwidth as wellas power.

The Masai realization of our Safari architectureis based in general on the concept of landmark rout-

ing [36,35,37] and has similarities to existing proto-cols that apply landmark routing to ad hocnetworks, such as LANMAR [27] and L+ [9]. How-ever, unlike these previous systems, Masai is ahybrid protocol, carefully combining proactive andreactive routing mechanisms to substantiallyincrease the network’s scalability and the protocol’s

ability to successfully deliver data packets with verylow overhead in spite of high node mobility. Weprovide a detailed discussion of this and other differ-ences between the Masai realization of Safari andprevious systems in Section 6.

Our evaluation in this paper is based on bothanalysis and simulation, under different networksizes, percentage of mobile nodes, and workloads.Our simulation results demonstrate that the pro-tocol is significantly more scalable than existingprotocols.

In Section 2, of this paper, we describe our Safariarchitecture. The design of the Masai realization ofthe Safari architecture is presented in Section 3,including the protocols for hierarchy self-organiza-tion and routing. In Section 4, we present modelingand analysis results of the Masai realization ofSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In Sec-tion 6, we discuss related work in the area ofscalable ad hoc networking, and we conclude in Sec-tion 7.

2. Overview of the safari architecture

In this section, we provide an overview of theSafari architecture. Safari provides a self-organizingnetwork hierarchy, a scalable routing protocol, andseamless integration of infrastructure network com-ponents where and when available. Safari alsoincludes an integrated distributed hash table(DHT) service, which supports decentralized net-work services and applications. This hierarchicalstructure allows routing in the Safari architectureto be based on a hybrid of proactive and reactiverouting information, greatly increasing the net-work’s scalability. Any node in the Safari architec-ture may be mobile or stationary.

The Safari hierarchy recursively groups nodesinto cells, cells into supercells, and so on, basedon an automatic self-selection of a subset of thenodes to operate as ‘‘landmarks’’ [36], calleddrums in the Safari architecture. Each drum nodetransmits periodic beacon packets, which are for-warded by all nodes within a well-defined, limitedscope in the network. The drums are not other-wise actively involved in routing data packetsfrom any source node to its destination; instead,hearing the beacon packets from drums givesnodes forwarding data packets ‘‘a sense of direc-tion’’ within the network topology of the hierar-chical Safari architecture.

Page 3: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

Table 1Summary of architectural components described in this paper

Component Description Section

Beaconing Provides proximity and routestoward existing drums

Section III-A

Drum selection Provides automatic selectionof drum nodes

Section III-B

Cell membership Provides association betweennodes and drums to form cells

Section III-C

Routing Provides routing betweennodes in the network

Section III-D

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 487

In general, for k P 0, level-k cells in Safari aregrouped into level-(k + 1) cells, and so on, withinthe recursively defined hierarchy; for simplicity ofterminology, we refer to individual nodes as level-0 cells. The lowest level at which drums exist is atlevel-1; individual nodes at level-0 do not operateas drums. We refer to level-1 cells also as fundamen-

tal cells, as at this level, the cell is composed only ofindividual nodes.

The drums are likewise organized hierarchically,with a subset of the individual nodes self-selectingto become level-1 drums, and recursively, a subsetof level-k drums self-selecting to become level-(k + 1) drums. Each level-k drum is at the same timealso a level-i drum for all levels i < k. Each drum hasa unique identifier, and each drum at level-i identi-fies a cell at level-i. The drum selection is based ona distributed algorithm with no centralized coordi-nation. Nodes of the same level are roughly equallyspaced (in terms of hop counts) throughout theentire ad hoc network. As nodes, including possiblydrum nodes, move within the network, the hierar-chy is maintained through the beacon packets andthe automatic self-selection of nodes to operate asdrums; over time, the set of nodes that are currentlydrums is not fixed.

Overall, the periodic beacon packets from eachdrum node aid the hierarchy formation, give nodesan indication of their topological location within thehierarchy, and provide routing information toward

the drum’s cell. As noted above, the drums do not

have any special role in data packet forwarding,and they are thus no more loaded with handlingdata packets than normal nodes. The drum identifi-ers form a topological location-dependent hierarchi-cal address for each node, which the node storesunder its own unique identifier in the network usinga distributed hash table (DHT); carefully choosingmultiple storage nodes improves robustness and effi-ciency of lookup.

To route a data packet, the packet is forwardedaccording to the hierarchical address of the packet’sdestination, routing recursively at each level towards

the drum for the destination node’s cell. We assumethe existence of bidirectional wireless links. To routetowards a drum at a given level, packets in Safariare routed following the reverse path of the mostrecent beacon received from that drum. Once thepacket reaches the fundamental cell of the destina-tion, any effective traditional ad hoc network routingprotocol can be used, since the size of a fundamentalcell is limited.

The Safari architecture owes its scalability to thefollowing design features:

• Self-organization: Beacons maintain the hierar-chy, allow each node to determine its hierarchicaladdress, and provide next-hop routing informa-tion. The overhead for disseminating beacons islogarithmic in the size of the network and inde-pendent of the traffic load or the level of mobility.

• Scalable routing: Each node maintains informa-tion about only the beacons it overhears, provid-ing it with next-hop routing information. Theamount of state a node maintains is logarithmicin the size of the network.

• Decentralized operation: The Safari architectureis fully self-organizing. Participating nodes playsymmetric roles. All nodes equally share the loadof disseminating beacons.

• Local view: Each participating node maintains alocal view of its surrounding network, withdetailed information about its immediate neigh-borhood and progressively more coarse-grainedinformation about distant parts of the network.

The following section details a particular realiza-tion of the Safari architecture, which we call Masai.The design of Masai consists of specific protocolsthat provide the self organization and scalable rout-

ing aspects of the Safari architecture. Specific proto-cols for providing distributed address resolution[10,22], a distributed hash table, and seamless inte-gration of any available infrastructure-based net-work components are beyond the scope of thispaper. The components described in this paper aresummarized in Table 1.

3. The masai realization of the safari architecture

Masai is a specific realization of the Safari archi-tecture; this section describes the four basic mecha-nisms that make up this realization: the beaconing

Page 4: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

488 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

protocol, the drum level selection algorithm, themembership algorithm, and the scalable routing pro-

tocol. The first three mechanisms allow the networkto self-organize, achieving and maintaining thedesired Safari hierarchical structure of the network,even under node mobility, node failures, and parti-tion and merging of networks. The fourth mecha-nism, the Masai routing protocol, is composed ofmechanisms for inter-cell routing, route repair ininter-cell routes, and intra-cell routing.

3.1. Beaconing protocol

Each drum periodically locally broadcasts to itsneighbors a beacon packet advertising the drum’sexistence and providing location information. Abeacon originating from a drum of level-n is calleda level-n beacon. Each beacon contains a beacon

sequence number, a beacon level, a hierarchical

address that equal those of the originating drum;and a hop count, that is set to zero at the originatingdrum and incremented by each forwarding node. Asmentioned in Section 2, a level-n drum is also alevel-i drum for all levels i < n, and a drum thusoriginates beacons for all levels for which it is adrum. A drum maintains a single beacon sequencenumber and increments it for each new beacon thatit originates at any level.

A drum at some level-n transmits a level-n bea-con every Tn seconds, which is forwarded by allnodes within Dn number of hops from that drum.This forwarding rule allows beacons to reach allnodes that could potentially associate with the orig-inating drum according to the membership algo-rithm described in Section 3.3. Higher levelbeacons are emitted at a lower frequency than lowerlevel beacons, since mobile nodes cross-over the lar-ger regions covered by higher level cells less fre-quently than they cross-over the regions coveredby lower level cells. For scalability, Tn and Dn aregiven by the geometric progressions:

Dn ¼ c� Dn�1 ¼ cn�1 � D1

T n ¼ b� T n�1 ¼ bn�1 � T 1

ð1Þ

where c > 1 and b > 1 are system parameters. The va-lue of D1 is based in general on the largest hop countthat the on-demand routing protocol used withinfundamental cells can generally support efficiently.

Although a drum originates beacons for all levelsfor which it is a drum, if some level-n drum is sched-

uled to originate a level-j beacon and a level-k bea-con at (approximately) the same time, for j < k 6 n,the drum omits originating this level-j beacon, sincethe range over which the level-k beacon will be for-warded covers the range of the level-j beacon. Whena node receives a beacon of some level-k, it treats italso as a beacon of the same sequence number forall levels i < k.

In addition, to increase routing scalability (Sec-tion 3.4), a level-n beacon is also forwarded by allnodes already associated in the level-(n + 1) cell ofthe originating drum. The exact mechanism bywhich a cell structure is formed is discussed in Sec-tion 3.3. Beacons are forwarded according to theunion of the two forwarding rules described above.

Each node stores information from the beacons itreceives in a local cache of beacons, called the DrumAd Hoc Routing Table (DART) for that node. Thiscache is used for self-organization and for routing.In addition to information from the beacon, a nodealso stores in its DART the time of reception of the

beacon and the neighbor node identifier from which it

received the beacon. The latter allows data packetforwarding along the reverse path of the beacon,while the former is used to keep the cache up todate. Upon receipt of a beacon, the node creates anew DART entry and starts a timer for that entry.Whenever a new entry is created in the DART orthe timer for a DART entry expires, the drum levelselection (Section 3.2) and the membership algo-rithms (Section 3.3) are invoked. Given that the dis-tance between drums at level-n, Dn, is a geometricprogression, there should be only O(log x), wherex is the number of nodes, DART entries. It is notnecessary to keep DART entries for nodes thatare reachable via intra-cell routing. Due to thesetwo facts, the memory and processing overhead ofthe DART should be small, even as the size of thenetwork increases.

As mentioned earlier, a drum has no active role inrouting or in maintenance of the hierarchy. The onlyspecial function of a drum is to originate beacons,while other nodes forward those beacons. In partic-ular, data packets are routed only towards but notnecessarily through drums, as will be described inSection 3.4. Thus, all nodes share the forwardingworkload (for data and beacon packets) equally.

3.2. Drum level selection algorithm

In order to be efficient under dynamic changes inthe network such as node mobility or failure, new

Page 5: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 489

drums can arise and existing drums can retire. Thedrum level selection algorithm is run at a node aftereach change in the DART at that node. The algo-rithm ensures that eventually, the DART at a nodeof some level-n satisfies the following conditions:

(i) The DART contains at least one non-expiredbeacon of a level-(n + 1) drum at most Dn+1

hops away.(ii) There is no non-expired beacon of a level-n

drum less than h · Dn hops away (0 < h < 1is a hysteresis factor).

The desired state of the DART is achieved by thenode changing its level so as to meet the invariants.

If condition (i) above is violated, the level of thenode is changed to n + 1 and the node waits a ran-dom back-off time before it announces its new levelwith its level-(n + 1) beacon.

If condition (ii) is violated, two or more drums ofthe same level are too close to each other. The drumwith the highest node identifier remains at the samelevel, and the rest of these drums reduce their levelby 1. The factor h (0 < h < 1) creates a ‘‘hysteresis’’that prevents oscillations in this drum retirementprocess. A level-n drum retiring could cause condi-tion (i) to be violated for other nearby nodes, eachof which will then increase its level. However, thesenodes will be at least Dn hops away from any level-ndrum. Since a conflict between drums requires thisdistance to reduce to h · Dn < Dn, there is no oscil-lation. Values of h close to 1 causes extra drumsto retire immediately, possibly resulting in oscilla-tions as new drums arise to replace the retired drum.Values of h close to 0 allows extra drums to survivein closer proximity, at the cost of more beacon over-head. Exploring appropriate values of h under dif-ferent levels of mobility is a subject of future workand is not discussed in this paper. Throughout therest of this paper we choose h = 0.5.

3.3. Membership algorithm

The presence of drums induces a natural cluster-ing of nodes. Each node associates with a drum of alevel-1 greater than its own level, and selects thisdrum according to the contents of its own DART.Typically, a node associates with the one higherlevel drum that is the least number of hops away.

A node’s association is made unilaterally and isnot communicated back to the drum. A nodeinvokes the membership algorithm after it has run

the drum level selection algorithm. In its basic ver-sion, for a node of some level-n, this algorithmchooses the closest (in terms of hop count) of alldrums of level n + 1 for which this node has aDART entry that has not expired. However, thisrule might lead nodes near the cell border to oscil-late between drums. In order to prevent such oscil-lations, this rule is enhanced by assigning eachDART entry a weight calculated from the fre-quency, the distance, and the number of beaconsreceived. The node associates with the drum corre-sponding to the DART entry with the highestweight. In our design presented in this paper, weenforce that the new drum is at least 2 hops closerthan the current one, and that at least 3 beaconshave been received from the new drum. Themembership algorithm cannot ensure that the nodeassociates with a drum that is at most Dn+1 hopsaway. This is the duty of the drum level selectionalgorithm.

The membership algorithm gives a unique ances-try for each node. Using this membership informa-tion, each node is assigned a hierarchical addressbased on the drum structure. This hierarchicaladdress plays a vital role in routing.

The hierarchical membership structure can beviewed as a tree, and every node in the network isassigned a hierarchical address. The hierarchicaladdress of a drum at some level-i is the concatena-tion of the hierarchical address of the level-(i + 1)drum with which it is associated and a randomlygenerated unique number. Thus, if ADDRESS(Xi)denotes the hierarchical address of some level-idrum, Xi, and PARENT(Xi) denotes the level-(i + 1) drum with which Xi has associated, then

ADDRESSðX iÞ¼ ADDRESSðPARENTðX iÞÞ � RANDðbÞ

where RAND(b) is a uniform random number of b

bits, and ‘‘Æ’’ means concatenation. With a large va-lue of b, the probability that two drums at the samelevel will chose the same random number can bemade negligible. The hierarchical address of any leafnode, L, is given by

ADDRESSðLÞ ¼ ADDRESSðPARENTðLÞÞ

When a node powers on, it associates with a drumat level-1 and sets its hierarchical address to thehierarchical address of this drum. This implies thatall nodes in the same fundamental cell have thesame hierarchical address.

Page 6: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

490 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

At start up of a node, the node has an emptyhierarchical address and will forward every beacon.The node randomly chooses a timeout from a fixedwindow, and if no beacon is received before thistimer expires, the node will increase its own levelto become a drum and emit a beacon. If many nodesare simultaneously powered on, the spacing due tothe random timeout should prevent all nodes frombecoming drums. If more drums are chosen thannecessary, extra drums will reduce their level asdescribed earlier in Section 3.2. The beacons of thefirst few drums of a new ad hoc network will reachthroughout the network. Once at least two of thenodes have increased their levels to become level-2drums, the level-1 beacon scope will be confinedby the cell structure.

When two ad hoc networks merge, i.e., whennodes of two networks with tree depth (hierarchyheight) n and m 6 n overhear each other’s beacons,then these beacons are not stopped immediatelyand penetrate the other network. The level-k bea-cons with k P m, in particular, are forwardedthroughout both networks as described in Section3.1, for the reason that either the beacon or theforwarding node has a hierarchical address oflength only m and thus cannot differ in the k + 1element of the hierarchical address. The smallernetwork quickly learns of the high level drum inthe other network, and associates with it, updatingits hierarchical address in the process. This corre-sponds to merging a smaller tree at the appropriatelevel into the larger tree. If the depths happen to beequal, the root of one of the trees will increase itsdrum level to become the new root, as discussedpreviously.

3.4. Scalable routing

Routing in the Masai realization of the Safarihierarchical architecture can be divided into twophases. Given the hierarchical address of the desti-nation node, the first phase, called inter-cell routing,delivers the packet to the fundamental cell of thedestination node. Once the packet reaches this cell,the second phase, called intra-cell routing, deliversthe packet to the destination node within that fun-damental cell.

Inter-cell routing is based on the destinationnode’s hierarchical address and on the beaconrecords stored in the DART of each intermediateforwarding node. As mentioned previously, weassume the existence of bidirectional wireless links.

Inter-cell routing operates by following the reversepaths of the beacons emitted by the drums of thecell at each level in which the destination node islocated. Conversely, intra-cell routing can bebased on any state-of-the-art on-demand routingprotocol, since the size of the fundamental cell iskept small. We choose DSR [17,18] for the intra-cell routing protocol for its demonstrated stabilitywith high performance in small ad hoc networks[7].

When a source node S with hierarchical addressSnÆSn�1, . . . , S1 has a packet to send to node D,node S retrieves the hierarchical addressDnÆDn�1, . . . , D1 of D using a lookup service, suchas the one used in L+ [9]. The efficiency of thelookup service is not considered in this paper, butthe achievable network performance is limited bythe quality of the information that the service pro-vides, especially when nodes are quickly changinghierarchical addresses due to mobility. If the nodesS and D belong to the same fundamental cell(Si = Di for all 1 6 i 6 n), the lookup service is notused, and intra-cell routing is invoked immediately.Otherwise, inter-cell routing is invoked.

With inter-cell routing, the source S adds thehierarchical address of D to the packet headerbefore sending the packet; the destination hierarchi-cal address is thus available to the following inter-mediate forwarding nodes without a separatehierarchical address lookup. To forward a packetat an intermediate node, the node uses the samelogic as the source node to determine whether tocontinue with inter-cell routing or to invoke intra-cell routing.

Fig. 1 shows an example of Masai routing inwhich a packet is originated from a source node Sto a destination node D. Node S uses inter-cell rout-ing to forward the packet to the first node along thepath labeled by ‘‘a’’, which is the reverse path of thebeacon originated by the level-2 drum X with whichD is associated. Each node along this path likewiseforwards the packet until it reaches a node that hasa DART entry corresponding the level-1 drum Y

with which D is associated; the packet then is routedalong the path labeled by ‘‘b’’, which is the reversepath of the beacon originated by this level-1 drumY. Once the packet reaches a node that is a memberof the destination fundamental cell, intra-cell rout-ing is used to deliver the packet to the destinationnode D along the route labeled by ‘‘c’’; this pathis dynamically discovered by the intra-cell on-demand routing protocol.

Page 7: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

0 500 1000 1500 2000 2500 30000

500

1000

1500

2000

2500

3000

a aa a a

a

a a bb b

cc

S

D

Y

X

b

bb

Level-1 Cell Boundary Level-1 Drum

Level-2 Cell Boundary Level-2 Drum

Other NodeRouting Path

Fig. 1. Masai routing example: a packet from source node S isrouted to destination node D.

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 491

3.4.1. Proactive inter-cell routing

The basic strategy of inter-cell routing is to fol-low the reverse path of the beacons that this nodehas received. When a node receives a beacon, itstores in its DART the node identifier of the trans-mitting neighbor node from which it received thebeacon (along with the information contained inthe beacon). When the node has a data packet tosend or forward to a specific cell, it uses the DARTto determine the next hop toward that cell.

As described in Section 3.1, each drum originatesbeacons that are forwarded to all nodes in its owncell and to nodes in the cells of its siblings (the samelevel cells that also share the same higher leveldrum). This mechanism ensures that any node thatis in the same supercell will have the next-hop rout-ing information to all fundamental cells in thissupercell.

Unlike some clustered routing protocols in adhoc networks that assume the existence of clusterheads with special routing functions, our drumsare not necessarily part of the route taken for datatransmission. As illustrated in Fig. 1, once thepacket enters the level-2 cell of drum X, any nodein that cell will have the routing information tothe fundamental cell of drum Y, and the packet neednot be routed through the drum X itself.

For inter-cell routing, each packet contains thefollowing information in its header in addition tothe hierarchical address of the destination; a for-warding node updates these fields with its DART

entry used for forwarding this packet, until thepacket reaches the fundamental cell of the destina-tion node:

• Prefix match length: The prefix match length

between the destination node’s hierarchicaladdress and the entry in the DART entry usedfor forwarding this packet. The prefix matchlength between two hierarchical addressesBn Æ Bn�1, . . . , B1 and Cn Æ Cn�1 . . . , C1 is the larg-est integer k such that Bi = Ci for alln � k < i 6 n.

• Sequence number: The sequence number of thebeacon from the DART entry used for forward-ing this packet.

• Hop count: The distance in hop count of the cur-rent forwarding node to the drum of the beaconfrom the DART entry used for forwarding thispacket.

During the inter-cell routing process, each for-warding node should use the best DART entry ithas to deliver the packet to the next hop; this for-warding node must use the same DART entry toupdate the above three fields in the packet header.When a forwarding node searches its DART for acandidate DART entry to use in routing the packet,the node must guarantee that the candidate providesbetter routing information than the three fields cur-rently listed in the packet’s header: specifically, theprefix match length must be larger, or, if it is thesame, the sequence number must be greater; if bothprefix match length and sequence number are thesame, the hop count in the candidate entry mustbe less than the hop count field in the packet header.

With the above requirements, inter-cell routing isguaranteed loop free. At any given time, a node canuse only one unique DART entry to forward a givenpacket (its best entry, by the above selection algo-rithm). Moreover, DART entries reflect the reversepaths of the beacons, which are loop-free since anynode forwards a given beacon only once. Thereverse paths of beacons thus form tree branchesoriginating at the respective drum. The paths tra-versed by data packets are composed of branch seg-ments from different trees. When a packet isforwarded toward a drum, the packet always travelsupward toward the root along the branches of thecorresponding tree, which must be loop free.Finally, when the prefix match length increases dur-ing the packet delivery, the packet has jumped toanother tree since now the packet is forwarded

Page 8: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

492 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

toward a different, lower level drum. Because theinter-cell forwarding algorithm forwards a packetfrom a node with some prefix match length onlyto a node with greater than or equal prefix matchlength, the packet cannot jump back to a tree thatwas previously traversed. Therefore, the entire tra-versed path of a packet is loop-free.

3.4.2. Route repair in inter-cell routing

In the inter-cell routing algorithm as describedabove in Section 3.4.1, a node forwarding a packetfollows the reverse path of the drum beacons. How-ever, the corresponding DART entry at that nodemay have already expired, or due to partitions inthe network or the unreliable wireless medium,some beacons might not have reached their intendedscope; in these cases, the node attempting to for-ward the packet might not have any useful entryin its DART. Furthermore, even if the forwardingnode does have a relevant DART entry, transmis-sion of the packet to the indicated next-hop nodemight fail, for example due to node mobility or fail-ure; we assume that a failure in transmitting apacket to the next-hop node can be detected aftera limited number of retransmission attempts, forexample through link-layer feedback as providedin the IEEE 802.11 MAC protocol [14].

When a forwarding node has no relevant DARTentry for some packet, or when transmission of thepacket to the next-hop node fails (after a limitednumber of retransmission attempts), the forwardingnode invokes on-demand (reactive) local route

repair to find an alternate route to continue for-warding the packet. After buffering any packets thatcould not be sent due to the failure, the node locallybroadcasts a hop-limited Masai LOCAL ROUTE

REQUEST packet containing the following informa-tion derived from the undelivered packet: (1) thecurrent value of the prefix match length, (2) thesequence number for the beacon being followed,(3) the hop count of the beacon being followed,and (4) the hierarchical address of the unreachablefinal destination node. In some other protocols,such as AODV [29], intermediate forwarding nodesmay initiate route discovery for route repair. How-ever, our mechanism for on-demand local routerepair has unique requirements, as our DART datastructure was originally obtained from the proactivebeacon packets, not from a reactive route discoveryprocess.

A node receiving a LOCAL ROUTE REQUEST

searches its DART for the hierarchical address of

the destination node. If the node finds a longer pre-fix match for the destination hierarchical address orif the node finds a prefix match of the same lengthbut with a greater sequence number, or if the prefixmatch length and the sequence number are both thesame but the hop count in the DART entry is lessthan that in the REQUEST, the node returns a MasaiLOCAL ROUTE REPLY containing the informationfrom the matching DART entry, back to the origi-nator of the REQUEST. Once the REPLY is receivedby the requester, the previously buffered packetsare routed using the reverse path followed by theREPLY just received. A node receiving multipleLOCAL ROUTE REPLYs chooses the REPLY with thelongest prefix match. If two REPLYs have the samelength prefix match, the node chooses the REPLY

with the greater sequence number or lower hopcount.

If a node receiving a LOCAL ROUTE REQUEST can-not reply and if the REQUEST is not a duplicate ofone received earlier, the node forwards the REQUEST

by locally rebroadcasting it. The REQUEST for-warder also must make sure that the REQUEST isstill within the transmission hop limit and thatthe REQUEST generally travels ‘‘downhill’’ towardthe destination, given the following definition ofnode ‘‘altitude.’’ The ‘‘altitude’’ of a node withrespect to a destination is defined by a combinationof the prefix match length, the sequence number ofthe beacon, and the hop count to the relevant drumnode, in that order. The longer the prefix matchlength, or the higher the sequence number, or thesmaller the hop count, the ‘‘lower’’ is the ‘‘alti-tude.’’ Similar to Gradient Routing [30], the statein each node’s DART generally forms a downwardgradient toward the respective drum node. TheLOCAL ROUTE REQUEST, therefore, can be for-warded with limited ‘‘uphill’’ hops other than thegeneral transmission hop limit. For example, wecan allow the transmission hop limit to be 4 andthe ‘‘uphill’’ limit to be 2. In this way, the requestpacket can be forwarded ‘‘downhill’’ or ‘‘level’’ upto 4 hops to find better routing information in anode’s DART; it can be forwarded no more than2 hops ‘‘uphill.’’ The local exploration of theREQUEST is more efficient as the transmissions arepredominantly ‘‘downhill’’ as expected.

When forwarding the LOCAL ROUTE REPLY, nodesupdate their DART state as if forwarding a beaconpacket, such that subsequent data packets can beforwarded normally using their updated DARTentry.

Page 9: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 493

3.4.3. Reactive intra-cell routing

When an intermediate node receives a datapacket for forwarding, the node checks if it is inthe same fundamental cell as the destination, i.e.,the node’s hierarchical address matches the hierar-chical address of the destination. If so, the packethas reached the fundamental cell of the destinationand intra-cell routing is used to further forward thepacket to the destination. Although any ad hoc net-work routing protocol can be used as the basis forintra-cell routing, we choose to use DSR [17,18],as it is a purely reactive protocol that has beenshown to perform well [7]. DSR is a source routingprotocol, with each packet containing a sourceroute (although the explicit source route can beremoved from most data packets [13]). The DSRprotocol consists of two mechanisms: Route Discov-

ery and Route Maintenance. To perform a RouteDiscovery for a destination node D, a source nodeS broadcasts a DSR ROUTE REQUEST packet thatis flooded through the network in a controlled man-ner. This REQUEST is answered by a DSR ROUTE

REPLY either from node D or from some other nodethat knows a route to D in its Route Cache. Toreduce the frequency and propagation of ROUTE

REQUESTs, each node aggressively caches sourceroutes that it learns or overhears.

In traditional DSR, the flooding of a ROUTE

REQUEST might be carried throughout the network,thus making Route Discovery increasingly expen-sive with increasing network size. Since in our caseof intra-cell routing, it is already known that thedestination exists in this fundamental cell, Masaiintra-cell Route Discovery is limited to within thatfundamental cell. Since different fundamental cellsmay have different sizes, the Route Discovery rangeis based on the dynamic membership of the nodesinstead of on a predefined hop count limit. Specifi-cally, whenever a node receives a Masai intra-cellROUTE REQUEST, it compares it’s own hierarchicaladdress with the ROUTE REQUEST initiator’s hierar-chical address and forwards the packet only if thetwo hierarchical addresses match. This techniqueis scalable, as the fundamental cell size does notincrease with the network size.

An originator node A may have the wrong hier-archical address of the destination node B, as B

might have changed it’s cell membership recentlyand this change is not yet known to node A. To takeadvantage of the high probability of the destinationstill remaining in the vicinity of its previous funda-mental cell, a hop count threshold is introduced in

the intra-cell Route Discovery that allows theintra-cell ROUTE REQUEST to additionally be for-warded one or two hops beyond the fundamentalcell. Each node forwarding the ROUTE REQUEST thuschecks if its own hierarchical address differs fromthat of the originator of the ROUTE REQUEST and ifso, increments the hop count field in the packet. Ifthe hop count is less than a threshold, the REQUEST

is forwarded, and otherwise it is dropped. This hopcount threshold creates a fuzzy boundary for for-warding the ROUTE REQUEST beyond the cell, allow-ing routing with less overhead.

4. Models and analysis

In this section, we use analysis, based on modelsfrom statistical physics, to assess the performance ofthe drum level selection protocol and the overheadresulting from periodic beacons in the Masai reali-zation of the Safari architecture. We validate thepredictions made by these models throughsimulations.

4.1. RSA: a model for drum formation

The random sequential adsorption (RSA) model[34] describes molecular adsorption processes,which exhibit strong similarities to the drum forma-tion in the Masai realization of the Safari. It wasfirst studied by Renyi [31] and has gained great pop-ularity known as the car parking problem, in whichcars of unit length arrive sequentially at randomlocations along a street; each car attempts to ‘‘park’’at its current location, or leaves if it would overlapwith the space already occupied by another parkedcar. To apply this model to the drum formation pro-cess in Masai, we make two simplifying assumptionsin this analysis.

4.1.1. The instantaneous propagation assumption

We use an instantaneous propagation assump-tion in this analysis to ensures with high probabilitythe sequential character of drum formation, similarto the car parking problem (RSA). This assumptionstates that

T max � T min � T

where T is the time for a packet to traverse D1 hops.We assume that each node, after it powers on, waitsfor a random time, uniformly distributed betweenTmin and Tmax, to receive a beacon from a drum, be-fore becoming a level-1 drum itself.

Page 10: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

494 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

Consider two nodes A and B, within D1 hops ofeach other, which each just powered on and arewaiting to receive a beacon from any level-1 drum.Under the instantaneous propagation assumption,with high probability, the first beacon of a node thatjust became a drum reaches the other node before itstimer expires.

Therefore, with high probability, level-1 drumsform or ‘‘arrive’’ sequentially, rather than simulta-neously. In particular, they are separated by atleast D1 hops without need to resolve conflicts. Theargument extends easily to all levels of the drumhierarchy.

4.1.2. The poisson arrivals assumption

Since we consider in this section the performanceof the drum level selection from a ‘‘cold start’’ of theentire ad hoc network, it is natural to assume thatall nodes arrive (power on) at the same time, withlocations according to a (homogeneous) Poissonpoint process in the plane. Extensions to three-dimensional space are immediate. As a consequenceof this Poisson arrivals assumption, the number ofnodes arriving in disjoint regions of the plane arestatistically independent of each other. The overallnumber of nodes in the network under this modelis a Poisson random variable. More importantly,given the number of nodes in a region, their loca-tions are statistically independent and uniformlydistributed. This justifies the use of the RSA modelin our analysis, given the number of nodes.

As we will study mainly the mean behavior of theprotocol, it is useful to introduce the node density, q,as the expected overall number N of nodes in thenetwork divided by the overall area A of thenetwork.

4.1.3. Drum formation conforms to the RSA model

Under the above two assumptions on propaga-tion and location in our analysis, the drum levelselection can be thought of as nodes attempting to‘‘park a disc-shaped car’’ in the following sense. Anode arrives at its actual (physical) location at thetime when its waiting timer expires after poweringon. Under the Poisson arrivals assumption andgiven the number of participating nodes, this loca-tion is uniformly distributed and independent ofother node locations. If the node has not receiveda beacon from any level-1 drum, the node willincrease its level from 0 to 1, becoming a drum.Under the instantaneous propagation approxima-tion, this new drum is at least D1 hops away from

any existing drums, which can be interpreted asparking a disc of radius D1/2 hops. Otherwise, ifthe node did receive a beacon from a drum beforeexpiration of its timer, it leaves its level set to 0(the node is not a drum), which can be interpretedas a failed car parking: the node ‘‘leaves’’ the drumcompetition.

The drum formation process will proceed aslong as a disc of radius D1/2 hops can be parkedwithout collision (i.e., without overlapping withthe space already occupied by another parkedcar). Indeed, since we measure distance here interms of number of hops, the presence of a discof radius D1 hops from every existing level-1 drumimplies the presence of a node that will become adrum itself upon expiration of its timer. The forma-tion stops at the so-called ‘‘jamming limit’’ whenno further disc can be parked without collidingwith existing discs.

Although equating in this analysis geometricaldistance with hop distance introduces a distortion,this distortion is homogeneous over space, with highprobability, if the density q is sufficiently large, andamounts to a change of units. For clarity, we denoteby q0 the node density in the hop metric sense. Itsvalue depends on the transmission range of eachradio and the spatial density of nodes. From simu-lations, we estimate a value of q0 ’ 1.4 for nodedensity q of 10 nodes per radio range and for a radiorange of 250 m.

The estimate of q0 is obtained by simulating apacket broadcast initiated by a node that is situatednear the center of the simulation area in a stationarynetwork and recording the hop count at which thispacket is received at all the other nodes in the net-work. The number of nodes that receive the packetwithin H hops is q0pH2. Hence q0 is obtained bydividing this number by pH2.

To state the analytical result from applying theRSA model, we denote by N0 = N the total numberof nodes in the network, and denote by N1 theaverage number of level-1 drums that form. Then,using the fact that the parking density at the jam-ming limit [34,31] is 54.7%, the ratio of level-1drums to the total number of nodes can be esti-mated as

N 1

N 0

’ 0:547p4

D21q0

: ð2Þ

The graph in Fig. 2 was obtained by simulating thedrum protocol in ns-2 for 100 s with 500 mobile

Page 11: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

4 5 6 7 8 9 10 110

0.005

0.01

0.015

0.02

0.025

0.03

Num

ber

of le

vel-1

dru

ms

per

node

TheoreticalSimulation

D1

Fig. 2. Number of level-1 drums per node (ns-2 simulation).

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 495

nodes. Node speeds in these simulations are uni-formly distributed between 5 and 15 m/s. Each sim-ulation was run 10 times, and the error bars indicatethe standard deviation from the mean. The mobilitymodel here is a modified billiard ball model withrandom reflection angle upon hitting simulationarea edges. The advantage of this model here is thatit results in a uniform distribution of the nodes evenduring mobility, providing a closer match to theRSA model which assumes uniform arrival loca-tions of cars. It is well known that in the RandomWaypoint model [7], the nodes tend to show a largerconcentration at the center; however, this bias to-wards the center is less pronounced at the beginningwhen starting the nodes in a uniform distribution.

If there is spatial non-uniformity in the node dis-tribution, the value of q0 will also vary spatially. Itsvalue will be low at low node density regions andhigh at high node density regions. This variationmay lead to deviation from what is predicted byEq. (2) above. However the inverse square relationon D1 in Eq. (2) will still hold.

The graph shows the number of level-1 drums pernode at time 100 s, at which time the drum forma-tion process is complete. The timer expires uni-formly in [0,150 s]. Hence, the instantaneouspropagation approximation holds. The graph alsoshows the number of level-1 drums based on theRSA model analysis. These results demonstrate thatthe RSA model successfully predicts the number ofdrums that have formed at the convergence of thedrum level selection protocol.

For higher level drums, the car parking problembecomes a constrained parking problem, as the

parking ‘‘substrate’’ consists now of lower leveldrums. Being less dense and quite discrete, this setof drum nodes provides a less ideal approximationof a continuum, and the appropriate model is theRSA-random sites (RSA-RS) model [16]. Themodel effectively reduces the packing density some-what and results in the following generalization ofEq. (2):

Ni

Niþ1

¼ aiDiþ1

Di

� �2

: ð3Þ

Here, Ni is the average number of level-i drums thatform, and ai is the ratio of the average packing den-sities of the RSA-RS process associated with level-idrums and the level-(i + 1) drums. As Di+1/Di is in-creased, the substrate becomes a closer approxima-tion of a continuum, and ai tends to unity. TheRSA-RS model can also be used to deal with net-works that are less dense than assumed in Eq. (2).

4.2. Collision model for drum retirement

When two level-n drums move to within h · Dn

hops of each other, one of them will retire. Becausethe radii of the discs are the half of the distancebetween the two drums, this is analogous to a colli-sion of two fictitious discs of radii h · Dn/2 hops.Again, results from statistical physics of molecularcollisions allow to compute the frequency of drumretirements [32] under the modified billiard ballmobility model above, corresponding to the motionof gas molecules. Adapting the theory for two-dimensional discs, we obtain the drum retirementrate k, the number of level-n drum retirements pers in the network, to be

k ’ ADnvaverageq2nffiffiffi

2p ð4Þ

where vaverage is the average velocity of the nodes, A

is the area covered by the network, and qn is the spa-tial density of level-n drums in terms of hop dis-tance. To obtain a simpler functional form, wesubstitute qn = Nn/A. However, from the RSA mod-el described previously, Nn is proportional toN 0=D2

n. In summary, we find that Eq. (4) takes thefunctional form k ’ ðN 2=AD3

nÞ. Thus, the stabilityof the drums increases with the level of the drums.

The drum formation rate at equilibrium must beequal to the retirement rate and is therefore alsogiven by Eq. (4), if we assume a fairly stable popu-lation of drums.

Page 12: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

SimulationLinear Fit

# of

add

ress

cha

nges

due

to c

ell c

ross

ing

1/D1

0 0.05 0.1 0.15 0.2 0.250

200

400

600

800

1000

1200

1400

Fig. 3. Level-1 cell crossing frequency follows a 1D1

law (ns-2simulation).

496 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

4.3. Overhead characterization

This section characterizes some of the overheadcaused by the beacons, leveraging the models andanalysis described above.

4.3.1. Drum flooding overhead

From Eq. (3), for every level-(i + 1) drum, thereare ai(Di+1/Di)

2 level-i drums, where ai is a propor-tionality factor that is close to unity. Consider a net-work with l levels of drum hierarchy (l is O(log N0)).Let the level-i drum emit beacon packets once everyTi s. A level-i drum beacon reaches all nodes in thenext higher level (level-(i + 1)) cell. So a node willreceive drum packets from all level-i drums thatare within the same level-(i + 1) cell that containsthis node (ai(Di+1/Di)

2 of them), for i running from1 to l � 1 and from the highest level drum.

Therefore, overhead due to drum beacon floods,Xdrum, defined by the number of beacon packets for-warded per s per node, is

X drum � a1D2

2

D21

1

T 1

þ a2D2

3

D22

1

T 2

þ � � � þ al�1D2

l

D2l�1

� 1

T l�1

þ 1

T l: ð5Þ

The actual overhead due to drum beacon floods is alittle greater than that shown in Eq. (5), becausebeacon floods propagate a minimum number ofhops (Di). Therefore, a level-i drum’s beacon pack-ets reach some nodes of a neighboring level-(i + 1)cell if the drum is at the border of the level-(i + 1)cell. However, the error in Eq. (5) is negligible, sincesuch overhearing nodes are at the border of the celland hence are far fewer than interior nodes of a cell.Moreover such a node will receive such a ‘‘leaked’’beacon packet only once for every, order of,(Di+1/Di)

2 broadcasts from its own level-(i + 1) cell.The Di’s and Ti’s are geometrically increasing, asshown by Eq. (1). Hence, Eq. (5) implies thatXdrum = H(1).

4.3.2. Frequency of cell changes for a node

A node’s hierarchical address could change if thenode moves into a new cell. Since the average inter-spacing (in hops) between two level-(i + 1) drums isproportional to Di+1, if the average movementspeed of the level-i drum node is v, it will cross theboundary of its level-(i + 1) cell in a time propor-tional to Di+1/v. In particular, the frequency withwhich a node moves to an adjacent fundamental cellis proportional to v/D1.

Fig. 3 shows the result of an ns-2 simulationunder the modified billiard ball mobility modeldescribed above. For each point, 10 simulationsare run, and the variance is shown. Each simulationis run for 100 simulated seconds, and the data dur-ing the first 25 s was discarded to remove initialtransient behavior. The x-axis is the inverse of D1,and the y-axis is the total number over all 500 nodesof changes in the hierarchical address for a node dueto level-1 cell crossings. The results show that thenumber of hierarchical address changes due tolevel-1 cell crossings is proportional to 1/D1.

5. Simulation evaluation

In addition to the analysis presented in Section 4,we have also evaluated the performance and scala-bility of the routing protocol in the Masai realiza-tion of the Safari architecture, using detailed ns-2network simulations. We compare the performanceof the Masai routing protocol with another land-mark-based protocol, L+ [9], and with DSR[17,18], a well-known purely reactive routing proto-col for ad hoc networks.

We used version 2.1b8a of ns-2, with the Mon-arch Project wireless and mobile extensions for ns-2 [7]. These extensions provide detailed modelingof an IEEE 802.11-based network with a wirelessphysical rate of 11 Mbps and a nominal wirelesstransmission range of 250 m. We implemented ourMasai design by extending the existing DSR codedistributed with ns-2, since we use DSR for routingwithin a fundamental Masai cell.

We set the beacon broadcast limit to 3 wirelesshops (D1 = 3), thus allowing the fundamental cellto be up to 6 wireless hops in diameter; DSR has

Page 13: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 497

been shown to perform well, without significantoverhead, in networks of this size [7]. Level-1 drumsoriginate beacon packets every 1 s (T1 = 1). We set cin Eq. (1) (Section 3.1) to be 2 and set b in Eq. (1)also to be 2. For the local route repair mechanismin Masai (as in Section 3.4.2), we used 4 hops asthe transmission limit and 2 hops as the ‘‘uphill’’limit.

We simulated a large number of network topolo-gies, with sizes ranging from 50 nodes to 1500 nodesand locations randomly distributed in a two-dimen-sional area. However, the x- and y-dimensions ofthe network area were modified so that the averagenetwork density was kept constant at the equivalentof 50 nodes in a 670 m · 670 m area; the averagenumber of nodes per nominal wireless transmissionarea is thus approximately 20. This node density isthe same as that used by Broch et al. [7] and hasbeen used in many other ad hoc network simula-tions. We have found that this average density tendsto avoid temporary network partitions when nodesrandomly move around. Due to the very large mem-ory consumption of ns-2 with large numbers ofnodes, we had to limit our simulations to a maxi-mum of 1500 network nodes. Nevertheless, theseresults, together with our analysis in Section 4, dem-onstrate the scaling potential and efficiency of rout-ing in the Masai realization of Safari.

Nodes in these simulations move according to theRandom Waypoint mobility model [7], with a max-imum speed of 10 m/s (average of 5 m/s) and aPause Time of 0. However, since as Yoon et al.[38] point out, the original Random Waypointmodel suffers from a decaying average node speedover the life of the simulation, as suggested by them,we added a minimum speed limit of 1 m/s in oursimulations to avoid this problem. Each simulationruns for 900 simulated seconds.

The network workload in these simulations con-sists of constant bit rate (CBR) flows, with each flow

Table 2Header field sizes for each Masai packet type

Packet type Header field and size (bytes)

Control Hierarchicaladdress

BEACON 1 4 · levelLOCAL ROUTE REQUEST 1 4 · levelLOCAL ROUTE REPLY 1 4 · levelDATA in inter-cell phase 1 4 · levelDATA in intra-cell phase 4 N/A

consisting of a randomly chosen source and destina-tion node. Each flow lasts 90 s and generates 64-bytepackets at constant rate of 4 packets/second. Thefirst 350 s in each simulation run are used to observethe performance of the cell organization and drumselection; at time 350 s, data flows then begin arriv-ing according to a Poisson distribution. This trafficpattern is more challenging to the routing protocolthan the typical continuous long lifetime CBRflows, since it requires using new routes to manymore unique destinations.

Each data point in our graphs for this perfor-mance evaluation represents the average of 16 indi-vidual simulations, created from the combination of8 different randomly generated mobility patternsand 2 different randomly generated data traffic pat-terns. The error bars in these graphs are calculatedas the sample standard deviation of the 16 runsfor each data point; to more clearly show the errorbars, some data points in some graphs have beenshifted slightly along the x-axis for different curves.

Table 2 shows the header fields and sizes used ineach type of packet used in the Masai realization ofthe Safari architecture. Each row of the table repre-sents one type of packet and shows the size of theheader fields used by that packet type, and each col-umn shows a possible header field and the corre-sponding size (in bytes) of that field for thatpacket type (header fields not used in a given packettype are indicated as ‘‘N/A’’). The total size of theMasai header for each packet type is thus the sumof the values in that corresponding row. Other partsof each packet, such as the standard MAC and IPheaders, and the payload for DATA packets, arenot shown in the table but are counted in the totalpacket size in our simulations. Since we base ourintra-cell routing protocol on DSR, we use the samepacket formats as used in the standard ns-2 imple-mentation of DSR for all packets in the intra-cellphase in Masai.

Beaconsequence

Hopcount

Matchedprefix length

Sourceroute

4 1 N/A N/A4 1 1 4 · length4 1 1 4 · length4 1 1 N/AN/A N/A N/A 4 · length

Page 14: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

0 200 400 600 800 1000 1200 1400 16000.75

0.8

0.85

0.9

0.95

1

Number of Nodes

Pac

ket D

eliv

ery

Rat

io

MasaiL+DSR

Fig. 4. Packet delivery ratio vs. network size, with all nodesmobile.

498 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

5.1. Routing scalability

An important goal of Safari is to provide routingin large ad hoc networks. We show that the designmeets this goal by evaluating the packet deliveryratio (PDR), routing overhead, packet deliverylatency, and routing path lengths used across differ-ent network sizes, up to 1500 nodes, in the Masairealization of the Safari architecture.

PDR is defined as the fraction of application datapackets originated that are successfully received bythe application layer at the respective destinationnode. Routing overhead per node is defined as theaverage network bandwidth consumed per nodeover all non-data packet transmissions. Overheadpackets include beacon packets, LOCAL ROUTE

REQUEST and LOCAL ROUTE REPLY packets for routerepair, and all DSR routing packets. Every routingoverhead packet contributes to this overhead eachtime it is transmitted (originated or forwarded).Furthermore, as data traffic does not start for thefirst 350 s, overhead bandwidth per node is calcu-lated as an average over the period between time350 and time 900 s (the end of the simulation).For the results shown in this section, all nodes inthe network are mobile, with a speed between1 m/s and 10 m/s, and the traffic in the networkfor each simulation is 100 CBR flows.

We compared Masai’s performance against L+and against DSR. For DSR, we used the versiondistributed with ns-2; we made only minor changesto it to make it support up to 32 hops in its sourcerouting packet header, and likewise expanded itsRoute Discovery hop limit. For L+, we used thens-2 code provided by the authors of L+, and weused its published default parameters [9]. Since wedo not consider Masai’s address lookup service inthis paper, we similarly disabled the addressupdate/query services of L+ and made the currenthierarchical address of a packet’s destination nodeavailable to the source node at no cost, in order toensure a fair comparison between Masai and L+.Due to limitations in our experimental platform,we were unable to simulate DSR in networks of1500 nodes and limited our DSR simulations to1000 nodes.

Fig. 4 shows the comparative PDR performanceof Masai, L+, and DSR. Masai delivers close to100% of all data packets at all network sizes, despitethe continuous mobility of all nodes in the network.At a network size of 1500 nodes, Masai achieves anaverage PDR of about 99.6%, with a very small

standard deviation, whereas DSR and L+ achievedsignificantly lower PDR performance in very largenetworks. DSR slightly outperforms L+ in smallnetworks (50 nodes), since it is entirely reactivewhereas L+ is entirely proactive. Masai, with bothproactive and reactive mechanisms, outperformsboth L+ and DSR in these small networks. Withincreasing network size, the PDR of DSR drops sig-nificantly, due to the increasing length of routes thatmust be maintained.

The PDR of L+ shows fluctuations as the net-work size increases. We are not sure of the causeof this fluctuation, but we conjecture that it is dueto the interaction between the protocol’s purely pro-active routing and, as the network size increases, thediscontinuous increases in the number of landmarksat each level and the total number of hierarchy lev-els. With increasing network size, the average size ofeach cell in L+ becomes larger and larger until anew landmark is created and if necessary a new levelof the hierarchy forms. The landmarks become fur-ther and further apart in hop count (path length).As a result, due to the mobility of nodes in the net-work, packets being forwarded may more oftenencounter broken links on the path to a remotelandmark; the likelihood of such a broken linkincreases as the path length increases. As the net-work size passes the point at which a new landmarkor level of the hierarchy is created, the average sizeof each cell is reduced, as landmarks are now onaverage closer together in hop count; the path tothe remote landmarks become shorter, and thechances that a packet being forward encounters abroken link on the way to the next landmark isreduced. Typical runs of L+ create a hierarchy of

Page 15: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 499

2 levels for 50 and 100 nodes; 3 levels for 200, 400,and 600 nodes; and 4 levels for 800, 1000, and 1500nodes. For 800- and 1000-node networks with L+,there is typically only one level-4 landmark, but800-node networks have 4 level-3 landmarks while1000-node networks have 7 level-3 landmarks. Thisdifference in number of level-3 landmarks gives1000-node networks a much smaller average sizeof the level-3 cells, accounting for the rise in PDRfor L+ at a network size of 1000 nodes. A similareffect accounts for the rise at 200 nodes.

Masai experiences similar changes in its hierar-chy with changing network size. For example, theaverage size of level-2 cells in Masai ranged fromabout 150 to 300 nodes, but the reactive local repairmechanism in Masai allows it to overcome any bro-ken links (e.g., due to mobility) in following thereverse path of beacons from remote drums. ThePDR of Masai thus remains almost constant asthe network size increases.

To better understand the reasons behind each ofthe few dropped data packets experienced withMasai, we examined the ns-2 trace files to determinewhat caused each data packet loss. Fig. 5 shows thepercentage of packet losses from each possible causein the 1500-node networks using Masai. Of the totalof 2434 lost data packets across the 16 simulationruns, more than 86% are due to failure in inter-cellrouting, around 5% are due to failure in intra-cellrouting, and around 7% of the lost packets occurredsimply because the packets were still in transit whenthe simulation time ended. The remaining 35dropped data packets (around 1.4%) were due tovarious reasons such as overflowed network inter-face queues.

Inter-cellno route 86.2%

Intra-cellno route 5.3%

End ofsimulation

7.1%

Other1.4%

Fig. 5. Percentage of packet loss for each cause for Masai in1500-node networks, with all nodes mobile.

Fig. 6 shows the routing overhead per node forMasai, L+, and DSR as the network size increases,with all nodes mobile as described above. BothMasai and L+ have a convex shape to their over-head bandwidth curves, becoming flatter as the net-work size increases, implying that their overheadscales logarithmically. To fully confirm our theoret-ical prediction for Masai’s overhead from Section 4,simulations of larger networks must be done. How-ever, this is currently limited by the inability of ns-2to scale to such very large networks. L+ shows verysmall error bars for its routing overhead, which isexpected for a purely proactive routing protocol.DSR shows very low overhead for small networksizes, but its overhead increases sharply as the net-work size increases, reaching approximately thesame level as for Masai and L+ at 1000 nodes.Masai’s overhead is higher than DSR’s for networksless than 1000 nodes, but this overhead should beconsidered relative to Masai’s significantly higherPDR (Fig. 4). Masai shows significantly lower over-head than does L+, due to factors such as the factthat individual nodes (at level-0) do not send bea-cons in Masai but do periodically advertise them-selves in L+. Overall, since Masai delivers a muchhigher fraction of the data packets than does L+or DSR, its advantage in efficiency is much greaterthan what is shown in this figure.

Fig. 7 shows the average packet delivery latencyfor the three protocols as the network size increases.In each of the protocols, average latency increasesas the network size grows larger. The average deliv-ery latency of Masai is higher than that of the othertwo protocols, though, due to a small number ofdelivered packets that required significantly longer

0 200 400 600 800 1000 1200 1400 16000

1

2

3

4

5

6

7

8

9

10

Number of Nodes

Ove

rhea

d B

andw

idth

(kb

ps/n

ode)

MasaiL+DSR

Fig. 6. Overhead bandwidth per node vs. network size, with allnodes mobile.

Page 16: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

0 200 400 600 800 1000 1200 1400 16000

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Number of Nodes

Ave

rage

Pac

ket D

eliv

ery

Late

ncy

(s) Masai

L+DSRMasai 99%Masai 95%

Fig. 7. Average packet delivery latency vs. network size, with allnodes mobile.

0 200 400 600 800 1000 1200 1400 16000

2

4

6

8

10

12

Number of Nodes

Del

iver

ed P

acke

t Pat

h Le

ngth

MasaiL+DSR

Fig. 8. Number of hops vs. network size, with all nodes mobile.

500 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

to be delivered. The longer delivery latency for thesepackets is caused by a combination of the timetaken by the local route repair mechanism in Masai(Section 3.4.2) and the increased overall routelength that may be created by this repair until thenext BEACON packet passes through that part ofthe network. In contrast, L+ and DSR have nolocal route repair mechanism such as this and thusinstead must drop packets that could have benefitedfrom local repair; this difference is demonstrated bythe substantially higher PDR for Masai than for L+or DSR, as shown in Fig. 4. To further illustrate thispoint, we also show in Fig. 7 the average deliverylatency of the 99% and 95% of delivered packetsin Masai with the lowest delivery latency. When just1% of the delivered packets are excluded, the aver-age latency improves to roughly equal to that ofL+ or DSR, and when 5% are excluded, Masai’saverage latency is clearly below the other two proto-cols, even though the difference in Masai’s PDRover L+ or DSR (Fig. 4) is greater than 5%.

Another observation from Fig. 7 is that Masaiand DSR have larger variance in their average deliv-ery latency than does L+. This increased variance isdue to the fact that L+ uses a purely proactive rout-ing mechanism, whereas DSR is entirely reactiveand Masai is a hybrid of reactive and proactivemechanisms (Masai intra-cell routing is entirelyreactive, as is its local route repair in inter-cell rout-ing). By its nature, any reactive routing mechanismmay sometimes cause a packet to be delayed whilethe protocol searches the network to discover anew route, adding to delivery latency variation sinceonly some packets require a new route discovery.Proactive routing, on the other hand, does not expe-rience such delays but instead pays the cost of ongo-

ing background overhead for exchanging routinginformation in order to attempt to always keep allroutes up-to-date.

Fig. 8 shows the average path length (number ofhops) used by delivered packets, with increasing net-work size, for Masai, L+, and DSR. Masai shows asmall disadvantage compared to L+ and DSR; forexample, in networks of 1000 nodes, Masai’s aver-age path length is about 1 hop longer than for L+and one half hop longer than for DSR. This differ-ence is primarily caused by the fact that Masai deliv-ers more of the data packets than does L+ or DSR,as shown in Fig. 4. Whereas L+ or DSR must dis-card a data packet if it encounters a broken linkalong the packet’s route (DSR’s ‘‘packet salvaging’’mechanism is able to avoid discarding some of thesepackets [17,18]), Masai uses its local route repairmechanism to find a new route in such circum-stances and is thus able to successfully deliver thepacket. However, the resulting total route used bythe packet may be longer than optimal, as the bro-ken link may be replaced by more than a single newhop; this slightly longer route persists until the nextperiodic beacon packet from the relevant drum rees-tablishes an optimal path back toward toward thatdrum.

5.2. Effect of mobility

The objective of Masai is to provide scalablerouting for a large-scale ad hoc network environ-ment. Whereas all results presented above are forall nodes being mobile, it is unlikely that all nodesin a real network will be mobile all the time. Wethus study here scenarios using the Masai realiza-tion of Safari, with different fractions of the nodes

Page 17: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

0 20 40 60 80 1000

1

2

3

4

5

6

7

8

9

10

Percentage of Mobile Nodes

Ove

rhea

d B

andw

idth

(kb

ps/n

ode)

Total OverheadProactive OverheadReactive Overhead

Fig. 10. Overhead bandwidth per node vs. percentage of mobilenodes in 1000-node networks.

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 501

being mobile, ranging from all mobile to all station-ary. Whether a node is mobile or stationary is notknown to Masai in our simulations; if, for example,the stationary nodes were known, choosing them asdrums in Masai would provide better performancedue to fewer hierarchical address changes, but wedo not explore such optimizations here.

In these simulations, we fixed the number ofnodes in the network at 1000 nodes and the numberof CBR flows at 100. We varied the percentage ofmobile nodes from 0% nodes being mobile (allnodes are stationary) to 25%, 50%, 75%, and100% nodes being mobile (all nodes are mobile).As above, mobile nodes move according to the Ran-dom Waypoint mobility model [7] with a speedbetween 1 m/s and 10 m/s.

Fig. 9 shows the change in PDR with the numberof mobile nodes varying from 0% to 100%.Although there is a slight decrease in PDR withincreasing mobility (the y-axis of the graph is mag-nified, ranging from 0.99 to 1.0 PDR), the PDR isaround 99.7% even for 100% mobile nodes, showingthat Masai reacts very well to mobility.

Fig. 10 shows the changes in routing overheadwith this increase in percentage of mobile nodes,broken down by the overhead caused by Masai’sproactive mechanism (beaconing) and reactivemechanisms (local route repair and intra-cell rout-ing). The proactive overhead in Masai increasesslowly as the mobility degree increases, due tochanges to which nodes are drums. In particular,when an existing drum node moves away from theother nodes in its cell, another node there becomesa drum, while the first drum node may continue also

0 20 40 60 80 1000.99

0.992

0.994

0.996

0.998

1

Percentage of Mobile Nodes

Pac

ket D

eliv

ery

Rat

io

Fig. 9. Packet delivery ratio vs. percentage of mobile nodes in1000-node networks (the y-axis ranges only from 0.99 to 1.0PDR).

as a drum in its new location or may take some timebefore deciding that it no longer needs to be a drum.As expected, reactive overhead grows as the per-centage of mobile nodes increases.

Fig. 11 shows the changes in average packetdelivery latency with the increase in percentage ofmobile nodes; as in Fig. 7, we also show here theaverage delivery latency for the 99% and 95% ofpackets with the lowest latency. As the number ofmobile nodes increases, the delay increases slightly,and the gap between the total average latency andthe fastest 99% average latency grows larger. Thiswidening gap suggests that although the latencyevery packet experiences grows, the largest impactis on the packets that require the most effort to deli-ver, experiencing perhaps more than one local routerepair for a packet before delivery.

0 20 40 60 80 1000

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Percentage of Mobile Nodes

Ave

rage

Pac

ket D

eliv

ery

Late

ncy

(s) Total Average

99% Average95% Average

Fig. 11. Average packet delivery latency vs. percentage of mobilenodes in 1000-node networks.

Page 18: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

0 50 100 150 200 250 300 3500

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Number of Flows

Ave

rage

Pac

ket D

eliv

ery

Late

ncy

(s) Total Average

99% Average95% Average

Fig. 14. Average packet delivery latency vs. traffic load in 1000-node networks, with all nodes mobile.

502 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

5.3. Effect of traffic load

We now show how routing performance in theMasai realization of Safari varies with different lev-els of traffic load. The number of nodes here is con-stant at 1000, and all nodes are mobile with a speedbetween 1 m/s and 10 m/s. We vary the traffic loadwith 10 flows, 100 flows, 200 flows, and 300 flows.

Figs. 12 and 13, respectively, show the PDR androuting overhead at different levels of traffic load.The PDR decreases very slightly as the networkbecomes congested (the y-axis in Fig. 12 ranges onlyfrom 0.99 to 1.0 PDR). The total overhead alsoincreases with the increase of traffic load from 100to 300 CBR flows. This increase in overhead isbecause, as the number of destinations grow, more

0 50 100 150 200 250 300 3500.99

0.992

0.994

0.996

0.998

1

Number of Flows

Pac

ket D

eliv

ery

Rat

io

Fig. 12. Packet delivery ratio vs. traffic load in 1000-nodenetworks, with all nodes mobile (the y-axis ranges only from 0.99to 1.0 PDR).

0 50 100 150 200 250 300 3500

1

2

3

4

5

6

7

8

9

10

Number of Flows

Ove

rhea

d B

andw

idth

(kb

ps/n

ode)

Total OverheadProactive OverheadReactive Overhead

Fig. 13. Overhead bandwidth per node vs. traffic load in 1000-node networks, with all nodes mobile.

uses of DSR route discovery are needed to findintra-cell routes the final destinations. On the otherhand, the proactive (beacon) overhead remainsnearly constant.

Fig. 14 shows the average packet delivery latencyat different levels of traffic load. As the traffic loadincreases, the delay also increases slightly. The gapbetween the total average latency and the 99% aver-age latency also increases as more flows are addedto the network, due to an increase in number ofpackets affected by the need for local route repair.

5.4. Network bootstrapping

Finally, we evaluated the behavior of the Masairealization of Safari during network bootstrapping,when all nodes in the network power up and initial-ize at the same time. This study demonstrates the‘‘worst case’’ behavior of node initialization usingthe Masai beaconing protocol (Section 3.1), drumlevel selection algorithm (Section 3.2), and member-ship algorithm (Section 3.3), as in most real net-works, the individual nodes typically do not allpower on simultaneously.

As described previously, when a node powers up,it initially waits for a period of time during which itforwards beacons from other nodes (drums) andattempts to choose some existing level-1 drum asits parent; the node waits until the expiration of thisperiod before deciding, if necessary, to increase itsown level and become a drum itself. Each node ran-domly selects this waiting period between 1 to 100 sin our simulations. This randomized waiting periodavoids all of the nodes increasing their own leveland becoming a drum at the same time.

Page 19: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 503

Figs. 15 and 16 characterize the performance ofMasai during network bootstrapping in one of our1500-node simulations with 100 data flows; allnodes are mobile with a speed between 1 m/s and10 m/s. Fig. 15 shows the changes in number ofdrums at different levels of the hierarchy over theduration of the simulation run. At time 100 s, thenetwork has stabilized with a single level-4 drum,which remains the case throughout the remainderof the simulation. As all nodes are mobile, the num-ber of drums at lower levels fluctuates somewhat,and the lower the drum level, the more fluctuationthere is. Fig. 16 shows the changes in network over-head over the duration of the simulation. For thefirst data point (20 s), the overhead is initially high,primarily because, as described in Section 3.3, thenodes initially forward all beacons, until the cell

0 200 400 600 800 10000

5

10

15

20

25

30

35

40

45

50

Simulation Time

Num

ber

of D

rum

s

Level-1 drumsLevel-2 drumsLevel-3 drumsLevel-4 drums

Fig. 15. Number of drums vs. simulation time in a 1500-nodenetwork, with all nodes mobile.

0 200 400 600 800 10000

2

4

6

8

10

12

Simulation Time

Ove

rhea

d B

andw

idth

(kb

ps/n

ode)

Total OverheadProactive OverheadReactive Overhead

Fig. 16. Overhead bandwidth per node vs. simulation time in a1500-node network, with all nodes mobile.

structure of the network begins to form and thereare at least two level-2 drums; after this point, thelevel-1 beacon scope will be confined by the cellstructure. After the beacon overhead stabilizes,there is still some fluctuation in overhead, as differ-ent nodes become drums or cease being drums dueto node mobility. After time 350 s, when the CBRflows begin, reactive overhead begins to contributeto the total overhead but remains low throughout.

6. Related work

In this section, we discuss related work in scal-able routing for ad hoc networks and how the Safariarchitecture and the Masai realization of Safari dif-fer from existing approaches.

Ad hoc network routing protocols can generallybe classified as either proactive (periodic) or reactive

(on-demand). Proactive protocols (e.g., [28,15,25])attempt to maintain up-to-date routes to all possibledestinations at all times, whereas reactive protocols(e.g., [17,29]) attempt to discover or maintain routesonly when needed to destinations for current com-munication. Reactive routing protocols have beenshown to have generally lower overhead than proac-tive protocols, and they can react much morequickly as routes in the network change. However,for very large networks or very high rates of mobil-ity, the overhead of current reactive protocols cangrow quickly.

A number of approaches to scalable ad hoc net-work routing have been proposed. Geographicalrouting techniques (e.g., [6,19,20,22,3]) allow rout-ing with state proportional only to the number ofneighbors at each node, but they require GPS orother location techniques. Moreover, a source nodemust know the location of the destination beforesending packets, thus requiring a location distribu-tion and maintenance service. DREAM [3] andLAR [20], respectively, employ proactive or reactiveflooding of the network and hence do not scalewith the size of the network. GLS [21] is a scalablelocation service for geographical routing in ad hocnetworks. However, even if the location service forgeographical routing can be made scalable, GPSdevices can be expensive and consume power, anddo not function indoors, limiting the application ofgeographical routing techniques. Although Safarimay be less scalable than geographical routing,Safari provides a practical, self-organizing hierarchyand is not dependent on GPS or other specializeddevices. Also, geographic routing protocols generally

Page 20: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

504 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

drop in performance in the presence of voids, as theprotocol must backtrack to find a suitable next-hopnode closer to the destination [19]; since Safari rout-ing, instead, is based on following the reverse pathof beacon packets, as long as the wireless links arebi-directional, such voids are not a problem forSafari.

Clustering techniques (e.g., [1,12,24,23,8,33,2,39]) can increase scalability, but existing activeclustering mechanisms require periodic refreshingof neighborhood information and introduce signifi-cant maintenance overhead due to global queryflooding. Another class of protocols use the ideaof routing via dedicated fixed anchors (e.g., [4,5]),but such techniques depend on deployment of fixedanchor nodes. Gao et al. [11] propose a randomizedkinetic clustering algorithm to create a set of clus-ters in a set of moving nodes; they also present adetailed evaluation of the properties of such cluster-ing. Although this work can be used to create a col-laborative and hierarchical ad hoc network, it is notclear how this algorithm would be implemented in areal network and how it would perform in such areal network. For example, the algorithm presenteddoes not consider the inherent unreliability of wire-less networks.

Techniques based on landmark routing, first pro-posed by Tsuchia [36,35,37], have also been pro-posed for scalable routing in ad hoc networks.Similar to our drum hierarchy in Safari, landmarknodes self-organize themselves into a hierarchy,such that landmarks at a given level in the hierar-chy are an approximately equal number of networkhops apart. The address of a node consists of thesequence of identifiers of the nearest landmarks,from highest to lowest level. During routing, anode extracts from the destination address thehighest level landmark identifier that differs fromits own node address, and forwards the packettowards the landmark with that identifier. Themapping from node identifiers to their currentaddress is maintained in a distributed fashion.Landmark routing achieves scalability by dramati-cally reducing the size of per-node routing tablesat the expense of somewhat longer routes. The ori-ginal landmark scheme, designed for large wirednetworks such as the internet, had only routers aslandmarks. End nodes did not participate in thehierarchy. LANMAR [27] attempts to scale mobilead hoc networks by combining ideas from land-mark routing and from Fisheye State Routing[26]. It specifically targets, however, ad hoc net-

works consisting of groups of nodes related infunctionality and mobility.

The previous routing protocol that is most simi-lar to Masai is L+ [9]. L+ modifies the lookup ser-vice of landmark routing to make it more scalableand modifies the routing to better handle mobilenodes. Although L+ has a number of similaritiesto our Masai realization of the Safari architecture,the fundamental differences between the two lie inhow they perform routing. L+ uses a purely proac-tive approach to routing and is based on DSDV.Masai, on the other hand, employs a hybrid of pro-active and reactive routing approaches. L+ keeps alist of routes to any destination and switches to thenext route when the current best route breaks. Italso needs to trigger a distance vector update when-ever there is any change in connectivity due tomobility or channel loss. As the proactive part ofrouting, Masai uses reverse beacon paths, avoidingall per-destination overhead. When such a reverseroute breaks, we perform on-demand local routerepair, thereby switching to reactive routing to finda new route and repair the routing state. In addi-tion, unlike L+, the hierarchy in Masai does notextend down to the lowest level but stops at funda-mental cells. Within each fundamental cell, Masaiuses a purely reactive protocol, thus reducing over-head and improving scalability. In summary, L+inherits many of the problems associated with pro-active ad hoc network routing protocols, whichMasai avoids. With increasing mobility, the fre-quency of the proactive updates in L+ (or any pro-active protocol) must increase proportionally,significantly increasing its overhead, a problemavoided in Masai; between these proactive updates,Masai can repair routes using reactive local routerepair.

7. Conclusions

In this paper, we have given an overview of theSafari architecture for scalable routing in ad hocnetworks. We have also presented the design andevaluation of a specific realization of the Safariarchitecture, which we call Masai. We focus in thiswork on the scalability of learning and maintainingthe routing state necessary for a large ad hoc net-work. Masai includes a probabilistic, self-organizingnetwork hierarchy formation protocol, comple-mented with a new hybrid routing protocol thatuses this hierarchy. This hybrid routing protocolconsists of both proactive and reactive components,

Page 21: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 505

helping the routing to scale to a much larger num-bers of nodes than previous ad hoc network routingprotocols. Nodes in the Safari architecture are givenhierarchical addresses, and each node’s unique nodeidentifier is mapped to its address using a distrib-uted hash table (DHT) that leverages the hierarchi-cal network structure. We have evaluated the Masairealization of the Safari architecture through analy-sis and simulations, under increasing network size,increasing fraction of mobile nodes, and increasingoffered traffic load. Our simulation results demon-strate that the protocol is significantly more scalablethan existing protocols. Compared to both the DSRand the L+ routing protocols, in particular, Safarihas much higher packet delivery ratio (PDR) andlower overhead, successfully supporting routing inmuch larger ad hoc networks.

Acknowledgements

This work was supported in part by NSF underGrants CNS-0520280, CNS-0435425, CNS-0338856, and CNS-0325971; and by a gift from Sch-lumberger. The views and conclusions containedhere are those of the author and should not be inter-preted as necessarily representing the official policiesor endorsements, either express or implied, of NSF,Schlumberger, Rice University, or the US Govern-ment or any of its agencies.

References

[1] Suman Banerjee, Samir Khuller, A clustering scheme forhierarchical control in multi-hop wireless networks, in:Proceedings of INFOCOM 2001, April 2001.

[2] Stefano Basagni, Imrich Chlamtac, Andras Farago, Ageneralized clustering algorithm for peer-to-peer networks,in: Proceedings of the Workshop on Algorithmic Aspects ofCommunication, held in conjunction with ICALP 1997, July1997.

[3] Stefano Basagni, Imrich Chlamtac, Violet R. Syrotiuk, BarryA. Woodward, A, Distance routing effect algorithm formobility (DREAM), in: Proceedings of the Fourth ACM/IEEE Annual International Conference on Mobile Comput-ing and Networking (MobiCom’98), October 1998, pp. 76–84.

[4] Ljubica Blazevic, Silvia Giordano, Jean-Yves Le Boudec,Anchored path discovery in terminode routing, in: Proceed-ings of the Second International IFIP-TC6 NetworkingConference on Networking Technologies, Services, andProtocols (Networking 2002), May 2002, pp. 141–153.

[5] Ljubica Blazevic, Silvia Giordano, Jean-Yves Le Boudec,Self-organized terminode routing, Cluster Computing 5 (2)(2002) 205–218. April.

[6] Prosenjit Bose, Pat Morin, Ivan Stojmenovic, Jorge Urrutia,Routing with guaranteed delivery in ad hoc wireless network,Wireless Networks 7 (6) (2001) 609–616, November.

[7] Josh Broch, David A. Maltz, David B. Johnson, Yih-ChunHu, Jorjeta G. Jetcheva, A performance comparison ofmulti-hop wireless ad hoc network routing protocols, in:Proceedings of the Fourth Annual ACM/IEEE InternationalConference on Mobile Computing and Networking (Mobi-Com’98), October 1998, pp. 85–97.

[8] Mainak Chatterjee, Sajal K. Das, Damla Turgut, WCA: aweighted clustering algorithm for mobile ad hoc networks,Cluster Computing 5 (2) (2002) 193–204. April.

[9] Benjie Chen, Robert Morris, L+: scalable landmark routingand address lookup for multi-hop wireless network. Tech-nical Report MIT-LCS-TR-837, Laboratory for ComputerScience, Massachusetts Institute for Technology, Cam-bridge, Massachusetts, 2002.

[10] Saumitra M. Das, Himabindu Pucha, Y. Charlie Hu,Performance comparison of scalable location services forgeographic ad hoc routing, in: Proceedings of INFOCOM2005, March 2005.

[11] Jie Gao, Leonidas J. Guibas, John Hershberger, Li Zhang,An Zhu, Discrete mobile centers, Discrete and Computa-tional Geometry 30 (1) (2003) 45–65, May.

[12] Zygmunt J. Haas, Marc R. Pearlman, The performance ofquery control schemes for the zone routing protocol, in:Proceedings of the ACM SIGCOMM’98 Conference onApplications, Technologies, Architectures, and Protocols forComputer Communication, August 1998, pp. 167–177.

[13] Yih-Chun Hu, David B. Johnson, Implicit source routes foron-demand ad hoc network routing, in: Proceedings of the2001 ACM International Symposium on Mobile Ad HocNetworking & Computing (MobiHoc 2001), October 2001,pp. 1–10.

[14] IEEE Computer Society LAN MAN Standards Committee,Wireless LAN Medium Access Control (MAC) and PhysicalLayer (PHY) Specifications, IEEE Std 802.11-1997, TheInstitute of Electrical and Electronics Engineers, New York,NY, 1997.

[15] P. Jacquet, P. Muhlethaler, T. Clausen, A. Laouiti, A.Qayyumand, L. Viennot, Optimized link state routingprotocol for ad hoc networks, in: Proceedings of the FifthIEEE International Multi Topic Conference (INMIC 2001),December 2001.

[16] Xuezhi Jin, N.-H. Linda Wang, Gilles Tarjus, Julian Talbot,Irreversible adsorption on non-uniform surfaces: the randomsite model, Journal of Physical Chemistry 97 (17) (1993)4256–4258.

[17] David B. Johnson, David A. Maltz, Dynamic source routingin ad hoc wireless networks, in: Tomasz Imielinski, HankKorth (Eds.), Mobile Computing, Kluwer Academic Pub-lishers, 1996, pp. 153–181 (Chapter 5).

[18] David B. Johnson, David A. Maltz, Yih-Chun Hu, Thedynamic source routing protocol (DSR) for mobile ad hocnetworks for IPv4. Internet Request for Comments RFC4728, February 2007. <http://www.ietf.org/rfc/rfc4728.txt>.

[19] Brad Karp, H.T. Kung, GPSR: greedy perimeter statelessrouting for wireless networks, in: Proceedings of the SixthAnnual International Conference on Mobile Computing andNetworking (MobiCom 2000), August 2000, pp. 243–254.

[20] Young-Bae Ko, Nitin H. Vaidya, Location-aided routing(LAR) in mobile ad hoc networks, in: Proceedings of theFourth ACM/IEEE Annual International Conference onMobile Computing and Networking (MobiCom’98), Octo-ber 1998, pp. 66–75.

Page 22: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

506 S. Du et al. / Ad Hoc Networks 6 (2008) 485–507

[21] Jinyang Li, John Jannotti, Douglas S.J. De Couto, David R.Karger, Robert Morris, A scalable location service forgeographic ad hoc routing, in: Proceedings of the SixthAnnual International Conference on Mobile Computing andNetworking (MobiCom 2000), August 2000, pp. 120–130.

[22] Wen-Hwa Liao, Yu-Chee Tseng, Jang-Ping Sheu, GRID: afully location-aware protocol for mobile ad hoc networks,Telecommunication Systems 18 (1–3) (2001) 37–60.

[23] Chunhung Richard Lin, Mario Gerla, Adaptive clusteringfor mobile wireless networks, IEEE Journal on SelectedAreas in Communications 15 (7) (1997) 1265–1275.

[24] A. Bruce McDonald, Taieb Znati, A mobility-based frame-work for adaptive clustering in wireless ad hoc networks,IEEE Journal on Selected Areas in Communications 17 (8)(1999). August.

[25] Richard Ogier, topology dissemination based on reverse-path forwarding (TBRPF): correctness and simulation eval-uation. Technical report, SRI International, Menlo Park,California, October 2003.

[26] Guangyu Pei, Mario Gerla, Tsu-Wei Chen, Fisheye staterouting in mobile ad hoc networks, in: Proceedings of theICDCS Workshop on Wireless Networks and MobileComputing, April 2000, pp. D71–D78.

[27] Guangyu Pei, Mario Gerla, Xiaoyan Hong, LANMAR:Landmark routing for large scale wireless ad hoc networkswith group mobility, in: Proceedings of the First AnnualWorkshop on Mobile Ad Hoc Networking and Computing(MobiHoc 2000), August 2000, pp. 11–18.

[28] Charles E. Perkins, Pravin Bhagwat, Highly dynamic desti-nation-sequenced distance-vector routing (DSDV) formobile computers, in: Proceedings of the SIGCOMM’94Conference on Communications Architectures, Protocolsand Applications, August 1994, pp. 234–244.

[29] Charles E. Perkins, Elizabeth M. Royer, Ad-Hoc on-demanddistance vector routing, in: Proceedings of the Second IEEEWorkshop on Mobile Computing Systems and Applications(WMCSA’99), February 1999, pp. 90–100.

[30] Robert D. Poor, Gradient Routing in Ad Hoc Networks,Media Laboratory, Massachusetts Institute of Technology,2000. <http://www.media.mit.edu/pia/Research/ESP/texts/poorieeepaper.pdf>.

[31] A. Renyi, On a one-dimensional problem concerningrandom space-filling, Publication of Mathematical Instituteof Hungarian Academy of Science 3 (1958) 109–127.

[32] W.R. Salzman, Kinetic Molecular Theory: Knudsen Flow,Molecular Collisions, Mean Free Path, Department ofChemistry, University of Arizona, 2004. <http://www.chem.arizona.edu/ salzmanr/480a/480ants/knumolco/knumolco.html>.

[33] Jacob Sharony, An architecture for mobile radio networkswith dynamically changing topology using virtual subnets,Mobile Networks and Applications 1 (1) (1996) 75–86.

[34] Robert H. Swendsen, Dynamics of random sequentialadsorption, Physics Review A 24 (1981) 504–508, July.

[35] Paul F. Tsuchiya, The landmark hierarchy: description andanalysis. Technical Report MTR-87W00152, MITRE Cor-poration, Bedford, Massachusetts, June 1987.

[36] Paul F. Tsuchiya, The landmark hierarchy: a new hierarchyfor routing in very large networks, in: Proceedings of theSIGCOMM’88 Symposium: Communications Architecturesand Protocols, August 1988, pp. 35–42.

[37] Paul F. Tsuchiya, Landmark routing: architecture, algo-rithms, and issues. Technical Report MTR-87W00174,MITRE Corporation, Bedford, Massachusetts, May1988.

[38] Jungkeun Yoon, Mingyan Liu, Brian Noble, Randomwaypoint considered harmful, in: Proceedings of INFOCOM2003, March 2003, pp. 1312–1321.

[39] Hongwei Zhang, Anish Arora, GS3: scalable self-configura-tion and self-healing in wireless networks, in: Proceedings ofthe Twenty-First Annual Symposium on Principles ofDistributed Computing (PODC 2002), July 2002, pp. 58–67.

Shu Du is currently a graduate studentin the Department of Computer Scienceat Rice University and is expecting hisPh.D. in Computer Science from Ricein 2007. His current research is inmobile and wireless systems, especiallyin routing and MAC protocols ofmultihop wireless systems. He receivedthe M.S. degree in Computer Sciencefrom Rice in 2004. Before joining Rice,he received Bachelor and Master of

Technology degrees in Computer Science and Engineering fromTsinghua University, Beijing, China, in 1999 and 2001,

respectively.

Ahamed Khan received the B.Tech.degree in Electrical Engineering from theIndian Institute of Technology (IIT)Madras, India in 2001 and the M.S.degree in Electrical Engineering fromRice University in 2005. From 2005 to2006 he worked at Qualcomm, SanDiego in the mobile phone softwaredivision. Currently, he is working atQualcomm, Hyderabad in the DSPgroup. His projects are in vocoding and

echo cancellation domains. His current interests are in wirelesscommunications, signal processing, and real time embedded

systems.

Santashil PalChaudhuri received theB.Tech. degree from Indian Institute ofTechnology (IIT) Kharagpur in 2000,and M.S. and Ph.D. degrees from RiceUniversity in 2002 and 2006, respec-tively, all in Computer Science. Since2005, he has been associated withwireless networking startups, MeruNetworks and Aruba Networks as aMember of Technical Staff. His Ph.D.thesis focused on a cross-layered

architecture for wireless sensor networks, where he designedadaptive protocols for routing, scheduling and synchronization.

His research interests lie in wireless networking protocols,systems, and architectures, particularly in mesh and sensornetworks.
Page 23: Safari: A self-organizing, hierarchical architecture for ...dbj/pubs/adhoc-safari.pdfSafari, and in Section 5, we give detailed simula-tion-based performance results for Masai. In

S. Du et al. / Ad Hoc Networks 6 (2008) 485–507 507

Ansley Post is pursuing his Ph.D. inComputer Science at Rice University. Heis interested in self-organizing systems ingeneral, and wireless systems specifically.He received the B.S. degree in ComputerScience from Georgia Tech in 2002, andthe M.S. degree in Computer Sciencefrom Rice University in 2005.

Amit Kumar Saha received the B.Tech.

degree in Computer Science and Engi-neering from Indian Institute of Tech-nology (IIT) Kharagpur, India, in 1999,and the M.S. and Ph.D. degrees inComputer Science from Rice Universityin 2003 and 2007, respectively. He iscurrently with Tropos Networks, aleading company delivering metro-scaleWi-Fi mesh network systems. Hisresearch interests are in the area of

mobile and wireless systems with a special emphasis on meshnetworking.

Peter Druschel is the founding director ofthe Max Planck Institute for SoftwareSystems. Previously, he was a Professorof Computer Science at Rice University,where he had been a member of thefaculty since 1994. He does research indistributed systems, operating systems,and networks. He received the Dipl.-Ing.(FH) degree from FachhochschuleMunich, Germany, in 1986, and thePh.D. degree from the University of

Arizona in 1994. He received an NSF CAREER Award in 1995and an Alfred P. Sloan Fellowship in 2000.

David B. Johnson is an Associate Pro-fessor of Computer Science and Electri-cal and Computer Engineering at RiceUniversity. Prior to joining the faculty atRice in 2000, he was an Associate Pro-fessor of Computer Science at CarnegieMellon University, where he had been amember of the faculty for 8 years. Hereceived the Ph.D. degree in ComputerScience in 1990 from Rice University.Professor Johnson is leading the Mon-

arch Project (Mobile Networking Architectures) research group

at Rice and has also been very active since 1993 in the InternetEngineering Task Force (IETF), the principal protocol standardsdevelopment body for the Internet. He was one of the maindesigners of the IETF Mobile IP protocol for IPv4 and is theprimary designer of Mobile IP for IPv6, and his group’s DynamicSource Routing protocol (DSR) for ad hoc networks has beenpublished by the IETF as an Experimental protocol for theInternet. Professor Johnson is currently the Chair of ACMSIGMOBILE, the Association for Computing Machinery’s Spe-cial Interest Group on Mobility of Systems, Users, Data, andComputing. He received an NSF CAREER Award in 1995.

Rudolf H. Riedi received the M.Sc.degree in 1986 and the Ph.D. degree in1993, both from ETH Zurich, Switzer-land, in Mathematics. From 1993 to1995, he was with B. Mandelbrot at theMathematics Department of Yale Uni-versity. He spent 1995–1997 with INRIAin Paris, France. In 1997, he joined theElectrical and Computer EngineeringDepartment from where he moved to theDepartment of Statistics in 2003, both at

Rice University. He currently holds a position as an AssociateProfessor. His research interests lie in the theory and practice of

multifractals, the statistics of scaling processes, and multiscaleanalysis and synthesis, with applications in computer networkingand computational finance. He won the ETHZ Polya prize in1986. He has consulted with AT&T Labs and held long-termvisits with the departments of ECE, Melbourne, Physics of ENSLyon, and Computer Science of UFMG, Brazil.

Recommended