Experimental evaluation of wireless simulation assumptions

David Kotz, Calvin Newport, Robert S. Gray, Jason Liu, Yougu Yuan, and Chip Elliott

Dartmouth Computer Science Technical Report TR2004-507June 2004

Abstract

All analytical and simulation research on ad hocwireless networks must necessarily model radiopropagation using simplifying assumptions. Al-though it is tempting to assume that all radios havecircular range, have perfect coverage in that range,and travel on a two-dimensional plane, most re-searchers are increasingly aware of the need to rep-resent more realistic features, including hills, obsta-cles, link asymmetries, and unpredictable fading. Al-though many have noted the complexity of real ra-dio propagation, and some have quantified the effectof overly simple assumptions on the simulation ofad hoc network protocols, we provide a comprehen-sive review of six assumptions that are still part ofmany ad hoc network simulation studies. In particu-lar, we use an extensive set of measurements from alarge outdoor routing experiment to demonstrate theweakness of these assumptions, and show how theseassumptions cause simulation results to differ signif-icantly from experimental results. We close with aseries of recommendations for researchers, whetherthey develop protocols, analytic models, or simula-tors for ad hoc wireless networks.

1 MotivationMobile ad hoc networking (MANET) has become alively field within the past few years. Since it is diffi-cult to conduct experiments with real mobile com-puters and wireless networks, nearly all publishedMANET articles are buttressed with simulation re-sults, and the simulations are based on common sim-plifying assumptions. Many such assumptions may

Note to readers who may have read the 2003 version of thispaper as a TR [KNE03]: this revised version of the paper has anentirely new data set collected from a live ad hoc network ex-periment, a simulation study to demonstrate the impact of theseaxioms on three ad hoc routing protocols, and a new list of rec-ommendations for routing protocol designers.

be too simple; a recent article inIEEE Commu-nicationswarns that “An opinion is spreading thatone cannot rely on the majority of the published re-sults on performance evaluation studies of telecom-munication networks based on stochastic simulation,since they lack credibility” [PJL02]. It then pro-ceeded to survey 2200 published network simulationresults to point out systemic flaws.

We recognize that the MANET research commu-nity is increasingly aware of the limitations of thecommon simplifying assumptions. Our goal in thispaper is to make a constructive contribution to theMANET community by a) quantitatively demon-strating the weakness of these assumptions, b) com-paring simulation results to experimental results toidentify how simplistic radio models can lead to mis-leading results in ad hoc network research, c) con-tributing a real dataset that should be easy to incor-porate into simulations, and d) listing recommenda-tions for the designers of protocols, models, and sim-ulators.

2 Radios in Theory and Practice

The top example in Figure1 provides a simple modelof radio propagation, one that is used in many simu-lations of ad hoc networks; contrast it to the bottomexample of a real signal-propagation map, drawnat random from the web. Measurements of Berke-ley Motes demonstrate a similar non-uniform non-circular behavior [GKW+02, ZHKS04]. The sim-ple model is based on Cartesian distance in an X-Yplane. More realistic models take into account an-tenna height and orientation, terrain and obstacles,surface reflection and absorption, and so forth.

Of course, not every simulation study needs to usethe most detailed radio model available, nor exploreevery variation in the wide parameter space affordedby a complex model. The level of detail necessaryfor a given analytic or simulation study depends on

1

the characteristics of the study. The majority of re-sults published to date use the simple models, how-ever, with no examination of the sensitivity of resultsto the (often implicit) assumptions embedded in themodel.

There are real risks to protocol designs based onoverly simple models of radio propagation. First,“typical” network connectivity graphs look quite dif-ferent in reality than they do on a Cartesian grid. Anantenna placed top of a hill has direct connectivitywith all other nearby radios, for example, an effectthat cannot be observed in simulations that representonly flat plains. Second, it is often difficult in real-ity to estimate whether or not one has a functioningradio link between nodes, because signals fluctuategreatly due to mobility and fading as well as inter-ference. Broadcasts are particularly hard-hit by thisphenomenon as they are not acknowledged in typicalradio systems. Protocols that rely on broadcasts (e.g.,beacons) or “snooping” may therefore work signifi-cantly worse in reality than they do in simulation.

Figure2 depicts one immediate drawback to theover-simplified model of radio propagation. Thethree different models in the figure, the Cartesian“Flat Earth” model, a three-dimensional model thatincludes a single hill, and a model that includes(absorptive) obstacles, all produce entirely differentconnectivity graphs, even though the nodes are in thesame two-dimensional positions. As all the nodesmove, the ways in which the connectivity graphchanges over time will be different in each scenario.

Figure3 presents a further level of detail. At thetop, we see a node’s trajectory past the theoretical (T)and practical (P) radio range of another node. Be-neath we sketch the kind of change in link qualitywe might expect under these two models. The the-oretical model (T) gives a simple step function inconnectivity: either one is connected or one is not.Given a long enough straight segment in a trajectory,this leads to a low rate of change in link connectivity.As such, this model makes it easy to determine whentwo nodes are, or are not, “neighbors” in the ad hocnetwork sense.

In the more realistic model (P), the quality of thelink is likely to vary rapidly and unpredictably, evenwhen two radios are nominally “in range.” In thesemore realistic cases, it is by no means easy to de-termine when two nodes have become neighbors, or

Typical theoretical model

Source: Comgate Engineeringhttp://www.comgate.com/ntdsign/wireless.html

Figure 1: Real radios, such as the one at the bot-tom, are more complex than the common theoreticalmodel at the top. Here different colors, or shades ofgray, represent different signal qualities.

when a link between two nodes is no longer usableand should be torn down. In the figure, suppose that alink quality of 50% or better is sufficient to considerthe nodes to be neighbors. In the diagram, the prac-tical model would lead to the nodes being neighborsbriefly, then dropping the link, then being neighborsagain, then dropping the link.

In addition to spatial variations in signal quality,a radio’s signal quality varies over time, even for astationary radio and receiver. Obstacles come andgo: people and vehicles move about, leaves flutter,doors shut. Both short-term and long-term changesare common in reality, but not considered by mostpractical models. Some, but not all, of this variationcan be masked by the physical or data-link layer ofthe network interface. Link connectivity can comeand go; one packet may reach a neighbor success-fully, and the next packet may fail.

2

Flat Earth

3-D

Obstacles

Figure 2: The Flat Earth model is overly simplistic.

Although the theoretical model may be easy to usewhen simulating ad hoc networks, it leads to an in-correct sense of the way the network evolves overtime. For example, in Figure3, the link quality (andlink connectivity) varies much more rapidly in prac-tice than in theory. Many algorithms and protocolsmay perform much more poorly under such dynamicconditions. In some, particularly if network connec-tivity changes rapidly with respect to the distributedprogress of network-layer or application-layer proto-cols, the algorithm may fail due to race conditionsor a failure to converge. Simple radio models failto explore these critical realities that can dramati-cally affect performance and correctness. For exam-ple, Ganesan et al. measured a dense ad hoc networkof sensor nodes and found that small differences inthe radios, the propagation distances, and the tim-ing of collisions can significantly alter the behaviorof even the simplest flood-oriented network proto-cols [GKW+02]. Others [GC04, ZHKS04] have re-cently used two-node experiments to quantify spe-cific characteristics of radio propagation, and used

Time

0%

100%

T P

TP

Link

Qua

lity

Node Trajectory Past Another Node

Figure 3: Difference between theory (T) and practice(P).

simulation to evaluate the impact of those character-istics on ad hoc routing protocols.

In summary,“good enough” radio models are quiteimportant in simulation of ad hoc networks. TheFlat Earth model, however, is by no means goodenough. In the following sections we make this ar-gument more precise.

3 Models used in researchWe surveyed a set of MobiCom and MobiHoc pro-ceedings from 1995 through 2002. We inspectedthe simulation sections of every article in which RFmodeling issues seemed relevant, and categorizedthe approach into one of three bins:Flat Earth, Sim-ple, andGood. This categorization required a fairamount of value judgment on our part, and we omit-ted cases in which we could not determine these ba-sic facts about the simulation runs.

Figure 4 presents the results. Note that evenin the best years, the Simple and Flat-Earth pa-pers significantly outnumber the Good papers. Afew [TMB01, JLW+96] deserve commendation forthoughtful channel models.

Flat Earth models are based on Cartesian X–Yproximity, that is, nodesA and B communicate if

3

95 96 97 98 99 00 01 02 03

0123456789

101112131415161718

GoodSimpleFlat Earth

Figure 4: The number of papers in each year of Mo-bicom and MobiHoc that fall into each category.

and only if nodeA is within some distance of nodeB.

Simple models are, almost without excep-tion, ns-2 models using the CMU 802.11 radiomodel [FV02].1 This model provides what has some-times been termed a “realistic” radio propagationmodel. Indeed it is significantly more realistic thanthe “Flat Earth” model, e.g., it models packet delayand loss caused by interference rather than assum-ing that all transmissions in range are received per-fectly. We still call it a “simple” model, however,because it embodies many of the questionable ax-ioms we detail below. In particular, the standard re-lease ofns-2 provides a simple free-space model(1/r2), which has often been termed a “Friis-free-space” model in the literature, and atwo-ray ground-reflection model. Both are described in thens-2document package [FV02].

The free-space model is similar to the “Flat Earth”model described above, as it does not include ef-fects of terrain, obstacles, or fading. It does, how-ever, model signal strength with somewhat finer de-tail than just “present” or “absent.”

The two-ray ground-reflection model, which con-siders both the direct and ground-reflected propaga-tion path between transmitter and receiver, is better,but not particularly well suited to most MANET sim-

1Other network simulators sometimes have better radio mod-els. OpNet is one commercial example; see opnet.com. Most ofthe research literature, however, uses ns-2.

ulations. It has been reasonably accurate for predict-ing large-scale signal strength over distances of sev-eral kilometers for cellular telephony systems usingtall towers (heights above 50m), and also for line-of-sight micro-cell channels in urban environments.Neither is characteristic of typical MANET scenar-ios. In addition, while this propagation model doestake into account antenna heights of the two nodes, itassumes that the earth is flat (and there are otherwiseno obstructions) between the nodes. This may be aplausible simplification when modeling cell towers,but not when modeling vehicular or handheld nodesbecause these are often surrounded by obstructions.Thus it too is a “Flat Earth” model, even more so ifthe modeler does not explicitly choose differing an-tenna heights as a node moves.2

More recently, ns-2 added a third channelmodel—the “shadowing” model described earlier byLee [Lee82]—to account for indoor obstructions andoutdoor shadowing via a probabilistic model [FV02].The problem withns-2 ’s shadowing model is thatthe model does not consider correlations: a real shad-owing effect has strong correlations between two lo-cations that are close to each other. More precisely,the shadow fading should be modeled as a two-dimensional log-normal random process with expo-nentially decaying spatial correlations (see [Gud91]for details). To our knowledge, only a few simulationstudies include a valid shadowing model. For exam-ple, WiPPET considers using the correlated shadow-ing model to compute a gain matrix to describe radiopropagation scenarios [KLM +00]. WiPPET, how-ever, only simulates cellular systems. The simula-tion model we later use for this study considers theshadowing effect as a random process that is tempo-rally correlated; between each pair of nodes we usethe same sample from the log-normal distribution ifthe two packets are transmitted within a pre-specifiedtime period.3

Zhou et al. recently explored how signal strengthvaried with the angle between sender and receiver,between different (supposedly identical) senders,and with battery level. They developed a modifi-

2See also Lundberg [Lun02], Sections 4.3.4–4.3.5, for addi-tional remarks on the two-ray model’s lack of realism.

3A recent study by Yuen et al. proposes a novel approach tomodeling the correlation as a Gauss-Markov process [YLA02].We are currently investigating this approach.

4

cation to path loss models that adds some randomvariation across angles and across senders, and thenshow how these better models lead to different sim-ulation results than the original models. Differentrouting algorithms react differently to the more re-alistic radio model, leading a better understandingof each algorithm’s strengths and weaknesses. Al-though they motivate their work with 2-node experi-ments, they do not have the ability to compare large-scale experiments with their simulation results as wedo.

Good modelshave fairly plausible RF propaga-tion treatment. In general, these models are used inpapers coming from the cellular telephone commu-nity, and concentrate on the exact mechanics of RFpropagation. To give a flavor of these “good” mod-els, witness this quote from one such paper [ER00]:

In our simulations, we use a model forthe path loss in the channel developed byErceg et al. This model was developedbased on extensive experimental data col-lected in a large number of existing macro-cells in several suburban areas in NewJersey and around Seattle, Chicago, At-lanta, and Dallas. . . . [Equation followswith parameters for antenna location in 3-D, wavelength, and six experimentally de-termined parameters based on terrain andfoliage types.] . . . In the results presentedin this section, . . . the terrain was assumedto be either hilly with light tree densityor flat with moderate-to-heavy tree density.[Detailed parameter values follow.]

Of course, the details of RF propagation are notalways essential in good network simulations; mostcritical is the overall realism of connectivity andchanges in connectivity (Are there hills? Are therewalls?). Along these lines, we particularly liked thesimulations of well-known routing algorithms pre-sented by Johansson et al. [JLH+99], which used rel-atively detailed, realistic scenarios for a conferenceroom, event coverage, and disaster area. Althoughthis paper employed thens-2 802.11 radio model,it was rounded out with realistic network obstaclesand node mobility.

4 Common MANET axiomsFor the sake of clarity, let us be explicit about somebasic “axioms” upon which most MANET researchexplicitly or implicitly relies. These axioms, not allof which are orthogonal, deeply shape how networkprotocols behave. We note that all of these axiomsare contradicted by the actual measurements reportedin the next section.0: The world is flat.1: A radio’s transmission area is circular.2: All radios have equal range.3: If I can hear you, you can hear me (symmetry).4: If I can hear you at all, I can hear you perfectly.5: Signal strength is a simple function of distance.

There are many combinations of these axiomsseen in the literature. In extreme cases, the combi-nation of these axioms leads to a simple model likethat in the top diagram in Figure1. Some papers as-sume Axioms 0–4 and yet use a simple signal prop-agation model that expresses some fading with dis-tance; a threshold on signal strength determines re-ception. Some papers assume Axioms 0–3 and add areception probability to avoid Axiom 4.

In this paper we address the research communityinterested in ad hoc routing protocols and other dis-tributed protocols at the network layer. The net-work layer rests on the physical and medium-access(MAC) layers, and its behavior is strongly influencedby their behavior. Indeed many MANET researchprojects consider the physical and medium-accesslayer as a single abstraction, and use the above ax-ioms to model their combined behavior. We take thisnetwork-layer point of view through the remainder ofthe paper. Although we mention some of the individ-ual physical- and MAC-layer effects that influencethe behavior seen at the network layer, we do not at-tempt to identify precisely which effects cause whichbehaviors; such an exercise is beyond the scope ofthis paper. In the next two sections we show that1) the above axioms do not adequately describe thenetwork-layer’s view of the world, and that 2) theuse of these axioms leads simulations to results thatdiffer radically from reality.

5 The RealityUnfortunately, real wireless network devices are notnearly as simple as those considered by the axioms in

5

the preceding section. Although Gaertner and Cahillexplicitly explore the relationship between link qual-ity and radio characteristics or environmental con-ditions, they do so with only two nodes and withno evaluation of the impact on simulation or imple-mented routing protocols [GC04]. Similarly, Zhouet al. use two-node experiments to motivate theirstudy of the impact of radio irregularity on simula-tion results [ZHKS04], but explore only that issueand do not validate their simulation study with ex-perimental data.

In this section, we use data collected from a largeMANET experiment in which forty laptops withWiFi and GPS capability roamed a field for over anhour while exchanging broadcast beacons. Althoughour experiment represents just one environment, it isnot unlike that used in many simulation-based stud-ies today (a flat square field with no obstacles andrandomly moving nodes). For the purposes of thispaper, it serves to demonstrate that the axioms areuntrue even in a simple environment, and that fairlysophisticated simulation models were necessary forreasonable accuracy.

At different times during the field test, the lap-tops also tested the costs and capabilities of differentrouting algorithms. A companion paper [GKN+04]explores that experiment and compares four routingprotocols, in what is to our knowledge the largestoutdoor experiment with a mobile ad hoc wirelessnetwork.4

We begin with a description of the experimentalconditions and the data collected.

5.1 Experimental data

The outdoor routing experiment took place on a rect-angular athletic field measuring approximately 225(north-south) by 365 (east-west) meters. This fieldcan be roughly divided into four flat, equal-sized sec-tions, three of which are at the same altitude, and oneof which is approximately four to six meters lower.There was a short, steep slope between the upper andlower sections.

4Lundgren et al. [LLN+02] briefly describes a slightly largerexperiment, but indoors, with a limited mobility pattern, andwith only a brief comparison of two routing algorithms.

Each Linux laptop5 had a wireless card6 operat-ing in peer-to-peer mode at 2 Mb/s. This fixed ratemade it much easier to conduct the experiment, sincewe did not need to track (and later model) automaticchanges to each card’s transmission rate. Most cur-rent wireless cards are multi-rate, however, whichcould lead toAxiom 6: Each packet is transmit-ted at the same bit rate.We leave the effects of thisaxiom as an area for future work.

To reduce interference from our campus wirelessnetwork, we chose a field physically distant fromcampus, and we configured the cards to use wirelesschannel 9, for maximum possible separation from thestandard channels (1, 6 and 11). In addition, we con-figured each laptop to collect signal-strength statis-tics for each received packet.7 Finally, each laptophad a Garmin eTrex GPS unit attached via the serialport. These GPS units did not have differential GPScapabilities, but were accurate to within thirty feetduring the experiment.

Each laptop recorded its current position (latitude,longitude and altitude) once per second, synchro-nizing its clock with the GPS clock to provide sub-second, albeit not millisecond, time synchronization.Every three seconds, thebeacon service programoneach laptopbroadcasta beacon containing the cur-rent laptop position (as well as the last known po-sitions of the other laptops). Each laptop that re-ceived such a beacon updated its internal position ta-ble, and sent aunicast acknowledgmentto the beaconsender via UDP. Each laptop recorded all incomingand outgoing beacons and acknowledgments in an-other log file. The beacons allowed us to maintain acontinuous picture of network connectivity, and, for-

5A Gateway Solo 9300 running Linux kernel version 2.2.19with PCMCIA Card Manager version 3.2.4

6We used a Lucent (Orinoco) Wavelan Turbo Gold 802.11b.Although these cards can transmit at different bit rates and canauto-adjust this bit rate depending on the observed signal-to-noise ratio, we used an ad hoc mode in which the transmissionrate was fixed at 2 Mb/s. Specifically we used firmware version4.32 and the proprietary ad hoc “demo” mode originally devel-oped by Lucent. Although the demo mode has been deprecatedin favor of the IEEE 802.11b defined IBSS, we used it to en-sure consistency with a series of ad hoc routing experiments ofwhich this outdoor experiment was the culminating event. Ourgeneral results, which revolve around signal-strength measure-ments and beacon-reception probabilities, do not depend on aparticular ad hoc mode.

7We used thewvlan cs , rather thanorinoco cs , driver.

6

tunately, also represent network traffic that would beexchanged in many real MANET applications, suchas our earlier work [Gra00] where soldiers must seethe current locations of their fellows. Finally, ev-ery second each laptop queried the wireless driver toobtain the signal strength of the most recent packetreceivedfrom every other laptop, and recorded thissignal strength information in a third log.8 Queryingevery second for all signal strengths was much moreefficient than querying for individual signal strengthsafter each received packet.

These three logs provide all the data that weneed to examine the axioms. Much more was go-ing on in the experiment, however, since the over-all goal was to compare the performance of fourrouting algorithms, APRL [KK98], AODV [PR99],ODMRP [LSG02], and STARA [GK97]. The lap-tops automatically ran each routing algorithm for 15minutes, generating random UDP data traffic for thir-teen out of the fifteen minutes, and pausing for twominutes between each algorithm to handle cleanupand setup chores. The traffic-generation parame-ters were set to produce the traffic volumes ob-served in our prototype situational-awareness appli-cations [Gra00], approximately 423 outgoing bytes(including UDP, IP and Ethernet headers) per lap-top per second, a relatively modest traffic volume.We do not describe the algorithms further here, sincethe routing and data traffic serves only as anothersource of collisions from the standpoint of the ax-ioms. Note, however, that each transmitted packetwas destined for only a single recipient, reducingODMRP to the unicast case.

Finally, the laptops moved continuously. At thestart of the experiment, the participants were dividedinto equal-sized groups of ten each, each participantgiven a laptop, and each group instructed to ran-domly disburse in one of the four sections of thefield (three upper and one lower). The participantsthen walked continuously, always picking a sectiondifferent than the one in which they were currentlylocated, picking a random position within that sec-tion, walking to that position in a straight line, andthen repeating. This approach was chosen since it

8For readers familiar with Linux wireless services, note thatwe increased the IWSPY limit from 8 to 64 nodes, so that wecould capture signal-strength information for the full set of lap-tops.

was simple, but still provided continuous movementto which the routing algorithms could react, as wellas similar spatial distributions across each algorithm.

During the experiment, seven laptops generated nonetwork traffic due to hardware and configuration is-sues, and an eighth laptop generated the position bea-cons only for the first half of the experiment. Weuse the data from the remaining thirty-two laptopsto test the axioms, although later we simulate thirty-three laptops since only seven laptops generated nonetwork traffic at all. In addition, STARA gener-ated an overwhelming amount of control traffic, andwe excluded the STARA portion of the experimentfrom our axiom tests. The final axiom dataset con-tains fifty-three contiguous minutes of beacons andacknowledgments for thirty-two laptops.

5.2 Axiom 0

The world is flat.

Common stochastic radio propagation models as-sume a flat earth, and yet clearly the Earth is notflat. Even at the short distances considered by mostMANET research, hills and buildings present obsta-cles that dramatically affect wireless signal propa-gation. Furthermore, the wireless nodes themselvesare not always at ground level; indeed, Gaertner andCahill noted a significant change in link quality be-tween ground-level and waist-level nodes [GC04].

Even where the ground is nearly flat, note thatwireless nodes are often used in multi-story build-ings. Indeed two nodes may be found at exactly thesamex, y location, but on different floors. (This con-dition is common among the WiFi access points de-ployed on our campus.) Any Flat Earth model wouldassume that they are in the same location, and yetthey are not. In some tall buildings, we found it wasimpossible for a node on the fourth floor to hear anode in the basement, at the samex, y location.

We need no data to “disprove” this axiom. Ulti-mately, it is the burden of all MANET researchers toeither a) use a detailed and realistic terrain model, ac-counting for the effects of terrain, or b) clearly con-dition their conclusions as being valid only on flat,obstacle-free terrain.

7

5.3 Axioms 1 and 2

A radio’s transmission area is circular.

All radios have equal range.

The real-world radio map of Figure1 makes itclear that the signal coverage area of a radio is farfrom simple. Not only is it neither circular nor con-vex, it often is non-contiguous.

We combine the above two intuitive axioms into amore precise, testable axiom that corresponds to theway the axiom often appears (implicitly) in MANETresearch.

Testable Axiom 1. The success of a transmissionfrom one radio to another depends only on the

distance between radios.

Although it is true that successful communica-tion usually becomes less likely with increasing dis-tance, there are many other factors: (1) All radios arenot identical. Although in our experiment we used“identical” WiFi cards, there are reasonable applica-tions where the radios or antennas vary from nodeto node. (2) Antennas are not perfectly omnidirec-tional. Thus, the angle of the sender’s antenna, theangle of the receiver’s antenna, and their relative lo-cations all matter. (3) Background noise varies withtime and location. Finally, (4) there are hills and ob-stacles, including people, that block or reflect wire-less signals (that is, Axiom 0 is false).

From the point of view of the network layer, thesephysical-layer effects are compounded by MAC-layer effects, notably, that collisions due to trans-missions from other nodes in the ad hoc network (orfrom third parties outside the set of nodes formingthe network) reduce the transmission success in waysthat are unrelated to distance. In this section, we useour experimental data to examine the effect of an-tenna angle, sender location, and sender identity onthe probability distribution of beacon reception overdistance.

We first demonstrate that the probability of a bea-con packet being received by nearby nodes dependsstrongly on the angle between sender and receiverantennas. In our experiments, we had each studentcarry their “node,” a closed laptop, under their armwith the wireless interface (an 802.11b device in PC-card format) sticking out in front of them. By exam-ining successive location observations for the node,

we compute the orientation of the antenna (wirelesscard) at the time it sent or received a beacon. Then,we compute two angles for each beacon: the an-gle between the sender’s antenna and the receiver’slocation, and the angle between the receiver’s an-tenna and the sender’s location. Figure5 illustratesthe first of these two angles, while the second is thesame figure except with the labels Source and Desti-nation transposed. Figure6 shows how the beacon-reception probability varied with both angles.

To compute Figure6, we consider all possible val-ues of each of the two angles, each varying from[−180, 180). We divide each range into buckets of45 degrees, such that bucket 0 represents angles in[0, 45), bucket 45 represents angles in[45, 90), andso forth. Since we bucket both angles, we obtain thetwo-dimensional set of buckets shown in the figure.We use two counters for each bucket, one account-ing for actual receptions, and the other for potentialreceptions (which includes actual receptions). Eachtime a node sends a beacon, every other laptop is apotential recipient. For every other laptop, therefore,we add one to the potential-reception count for thebucket representing the angles between the senderand the potential recipient. If we can find a receivedbeacon in the potential recipient’s beacon log thatmatches the transmitted beacon, we also add one tothe actual-reception count for the appropriate count.The beacon reception ratio for a bucket is thus thenumber of actual receptions divided by the numberof potential receptions. Each beacon-reception prob-ability is calculated without regard to distance, andthus represents the reception probability across alldistances. In addition, for all of our axiom analyses,we considered only the western half of the field, andincremented the counts only when both the senderand the (potential) recipient were in the western half.By considering only the western half, which is per-fectly flat and does not include the lower-altitude sec-tion, we eliminate the most obvious terrain effectsfrom our results. Overall, there were 40,894 beaconstransmitted in the western half of the field, and aftermatching and filtering, we had 275,176 laptop pairs,in 121,250 of which the beacon was received, and in153,926 of which the beacon was not received.

Figure6 shows that the orientation of both anten-nas was a significant factor in beacon reception. Ofcourse, there is a direct relationship between the an-

8

Direction of radio

(and node

movement)

Destination

Source

The angle

between the

source radio and

the destination

Figure 5: The angle between the sending laptop’s an-tenna (wireless card) and the destination laptop. Weexpress the angles on the scale of -180 to 180, ratherthan 0 to 360, to better capture the inherent symme-try. -180 and 180 both refer to the case where thesending antenna is pointing directly away from theintended destination.

tenna angles and whether the sender or receiver (hu-man or laptop) is between the two antennas. With asender angle of 180, for example, the receiver is di-rectly behind the sender, and both the sender’s bodyand laptop serves as an obstruction to the signal. Adifferent kind of antenna, extending above the levelof the participants’ heads, would be needed to sepa-rate the angle effects into two categories, effects dueto human or laptop obstruction, and effects due to theirregularity of the radio coverage area.

Although the western half of our test field wasflat, we observed that the beacon-reception prob-ability distribution varied in different areas. Wesubdivided the western half into four equal-sizedquadrants (northwest, northeast, southeast, south-west), and computed a separate reception probabil-ity distribution for beacons sent from each quad-rant. Figure7 shows that the distribution of beacon-reception probability was different for each quadrant,by about 10–15 percent for each distance. We buck-eted the laptop pairs according to the distance be-tween the sender and the (intended) destination—the leftmost bar in the graph, for example, is thereception probability for laptop pairs whose sepa-ration was in the range[0, 25). Although there are

−180−135

−90 −45

0 45

90 135

180−180 −135 −90 −45 0 45 90 135 180

0.3

0.35

0.4

0.45

0.5

0.55

Sender to Destination

Destination to Sender

Probability of Beacon Reception by Angle

Pro

babi

lity

of B

eaco

n R

ecep

tion

Figure 6: The probability of beacon reception (overall distances) as a function of the two angles, theangle between the sender’s antenna orientation andthe receiver’s location, and the angle between thereceiver’s antenna orientation and the sender’s loca-tion. In this plot, we divide the angles into bucketsof 45 degrees each, and include only data from thewestern half of the field.

many possible explanations for this quadrant-basedvariation, whether physical terrain, external noise, ortime-varying conditions, the difference between dis-tributions is enough to make it clear that the locationof the sender is not to be ignored.

The beacon-reception probability in the westernhalf of the field also varied according to the identityof the sender. Although all equipment used in everynode was an identical model purchased in the samelot and configured identically, the distribution wasdifferent for each sender. Figure8 shows the meanand standard deviation of beacon-reception proba-bility computed across all sending nodes, for eachbucket between 0 and 300 meters. The buckets be-tween 250 and 300 meters were nearly empty. Al-though the mean across nodes, depicted by the boxes,is steadily decreasing, there also is substantial varia-tion across nodes, depicted by the standard-deviationbars on each bucket. This variation cannot be ex-plained entirely by manufacturing variations withinthe antennas, and likely includes terrain, noise andother factors, even on our space of flat, open ground.It also is important to note, however, that there are

9

0

0.2

0.4

0.6

0.8

1

0 25 50 75 100 125 150 175 200 225 250 275 300

Pro

babi

lity

of B

eaco

n R

ecep

tion

Distance In Meters

SWSE

NWNE

Figure 7: The probability of beacon reception variedfrom quadrant to quadrant within the western half ofthe field.

0

0.2

0.4

0.6

0.8

1

0 25 50 75 100 125 150 175 200 225 250 275 300

Bea

con

Rec

eptio

n P

roba

bilit

y

Distance in Meters

Figure 8: The average and standard deviation of re-ception probability across all nodes, again for thewestern half of the field.

only 500-1000 data points for each (laptop, destina-tion bucket) pair. With this number of data points,statistical-significance issues come into play. In par-ticular, if a laptop is moving away from most otherlaptops, we might cover only a small portion of thepossible angles, leading to markedly different resultsthan for other laptops. Overall, the effect of identityon transmission behavior bears further study with ex-periments specifically designed to test it.

In other work, Ganesan et al. used a networkof Berkeley “motes” to measure signal strengthof a mote’s radio throughout a mesh of motenodes [GKW+02].9 The resulting contour map isnot circular, nor convex, nor even monotonically de-creasing with distance. Indeed, since the coverage

9The Berkeley mote is currently the most common researchplatform for real experiments with ad hoc sensor networks.

area of a radio is not circular, it is difficult to evendefine the “range” of a radio.

Zhou et al. [ZHKS04] also note that signalstrength varies with the angle between sender and re-ceiver, angle between receiver and sender, and senderidentity, using two-node experiments.

5.4 Axiom 3

If I can hear you, you can hear me (symmetry).

More precisely,

Testable Axiom 3: If an unacknowledged messagefromA to B succeeds, an immediate reply fromB

to A succeeds.

This wording adds a sense of time, since it isclearly impossible (in most MANET technologies)for A andB to transmit at the same time and resultin a successful message, and sinceA andB may bemoving, it is important to consider symmetry over abrief time period so thatA andB have not movedapart.

There are many factors affecting symmetry, fromthe point of view of the network layer, including thephysical effects mentioned above (terrain, obstacles,relative antenna angles) as well as MAC-layer colli-sions. It is worth noting that the 802.11 MAC layerincludes an internal acknowledgment feature, and alimited amount of re-transmission attempts until suc-cessful acknowledgment. Thus, the network layerdoes not perceive a frame as successfully deliveredunless symmetric reception was possible. Thus, forthe purposes of this axiom, we chose to examinethe broadcast beacons from our experimental dataset,since the 802.11 MAC has no internal acknowledg-ment for broadcast frames. Since all of our nodessent a beacon every three seconds, we were able toidentify symmetry as follows: whenever a nodeBreceived a beacon from nodeA, we checked to seewhetherB’s next beacon was also received by nodeA.

Figure9 shows the conditional probability of sym-metric beacon reception. If the physical and MAClayer behavior was truly symmetric, this probabilitywould be 1.0 across all distances. In reality, the prob-ability was never much more than 0.8, most likelydue to MAC-layer collisions between beacons. Since

10

0

0.2

0.4

0.6

0.8

1

0 25 50 75 100 125 150 175 200 225 250 275 300

Pro

babi

lity

of B

eaco

n R

ecep

tion

Distance in Meters

Figure 9: The conditional probability of symmetricbeacon reception as it varied with the distance be-tween two nodes, again for the western half of thefield.

0

0.2

0.4

0.6

0.8

1

0 5 10 15 20 25 30

Pro

babi

lity

Node

Figure 10: The conditional probability of symmetricbeacon reception as it varied across individual nodes,again for the western half of the field.

this graph depends on the joint probability of a bea-con arriving fromA to B and then another fromB toA, the lower reception probability of higher distancesleads to a lower joint probability and a lower condi-tional probability. Figure10 shows how the condi-tional probability varied across all the nodes in theexperiment. The probability was consistently closeto its mean 0.76, but did vary from node to node witha standard deviation of 0.029 (or 3.9%). Similarly,when calculated for each of the four quadrants (notshown), the probability also was consistently closeto its mean 0.76, but did have a standard deviation of0.033 (or 4.3%).

In other work, Ganesan et al. [GKW+02] notedthat about 5–15% of the links in their ad hoc sensornetwork were asymmetric. In that paper, an asym-

metric link had a “good” link in one direction (withhigh probability of message reception) and a “bad”link in the other direction (with a low probability ofmessage reception). [They do not have a name for alink with a “mediocre” link in either direction.]

Zhou et al. also found through simulation that theuse of angular variations in signal strength naturallyled to asymmetric links in simulation, and that someprotocols were unable to adapt gracefully to asym-metry [ZHKS04].

Overall, it is clear that reception is far from sym-metric. Nonetheless, many researchers assume thisaxiom is true, and that all network links are bidirec-tional. Some do acknowledge that real links may beunidirectional, and usually discard those links so thatthe resulting network has only bidirectional links. Ina network with mobile nodes or in a dynamic envi-ronment, however, link quality can vary frequentlyand rapidly, so a bidirectional link may become uni-directional at any time. It is best to develop protocolsthat do not assume symmetry.

5.5 Axiom 4

If I can hear you at all, I can hear you perfectly.

Testable Axiom 4: The reception probabilitydistribution over distance exhibits a sharp cliff; thatis, under some threshold distance (the “range”) thereception probability is 1 and beyond that threshold

the reception probability is 0.

Looking back at Figure8, we see that the beacon-reception probability does indeed fade with the dis-tance between the sender and the receiver, rather thanremaining near 1 out to some clearly defined “range”and then dropping to zero. There is no visible “cliff.”The commonns-2 model, however, assumes thatframe transmission is perfect, within the range of aradio, and as long as there are no collisions. Al-thoughns-2 provides hooks to add a bit-error-rate(BER) model, these hooks are unused. More sophis-ticated models do exist, particularly those developedby Qualnet and the GloMoSim project10 that are be-ing used to explore how sophisticated channel mod-els affect simulation outcomes.

Takai examines the effect of channel models onsimulation outcomes [TBTG01], and also concluded

10http://www.scalable-networks.com/pdf/mobihocpreso.pdf

11

that different physical layer models can have dra-matically different effect on the simulated perfor-mance of protocols [TMB01], but lack of data pre-vented them from further validating simulation re-sults against real-world experiment results, whichthey left as future work. Zhou et al. also did not val-idate their simulation results against real-world ex-periment results. We compared the simulation re-sults with data collected from a real-world experi-ment, and recommend below that simple models ofradio propagation should be avoided whenever com-paring or verifying protocols, unless that model isknown to specifically reflect the target environment.

5.6 Axiom 5

Signal strength is a simple function of distance.

Rappaport [Rap96] notes that the average signalstrength should fade with distance according to apower-law model. While this is true, one shouldnot underestimate the variations in a real environ-ment caused by obstruction, reflection, refraction,and scattering. In this section, we show that thereis significant variation for individual transmissions.

Testable Axiom 5: We can find a good fit between asimple function and a set of (distance, signal

strength) observations.

To examine this axiom, we consider only receivedbeacons, and use the recipient’s signal log to ob-tain the signal strength associated with that beacon.More specifically, the signal log actually containsper-second entries, where each entry contains thesingle strength of the most recent packet receivedfrom each laptop. If a data or routing packet arrivesimmediately after a beacon, the signal-log entry ac-tually will contain the signal strength of that secondpacket. We do not check for this situation, since thesignal information for the second packet is just asvalid as the signal information for the beacon. Itis best, however, to view our signal values as thoseobserved within one second of beacon transmission,rather than the values associated with the beaconsthemselves.

As a starting point, Figure11shows themeanbea-con signal strength observed during the experimentas a function of distance, as well as best-fit linear and

169

171

173

175

177

179

181

183

185

12.5 37.5 62.5 87.5 112.5 137.5 162.5 187.5 212.5 237.5

Distance (meters)

Sig

nal

Str

eng

th (

dB

m +

255

)

Observed

Power

Linear

Figure 11: Linear and power-curve fits for the meansignal strength observed in the western half of thefield. Note that we show the signal strength as re-ported by our wireless cards (which is dBm scaledto a positive range by adding 255), and we plot themean value for each distance bucket at the midpointof that bucket.

power curves. The power curve is a good fit and val-idates Rappaport’s observation. When we turn ourattention to the signal strength of individual beacons,however, as shown in Figures12and13, there clearlyis no simple (non-probabilistic) function that will ad-equately predict the signal strength of an individualbeacon based on distance alone.

The reason for this difficulty is clear: our envi-ronment, although simple, is full of obstacles andother terrain features that attenuate or reflect the sig-nal, and the cards themselves do not necessarily ra-diate with equal power in all directions. In our case,the most common obstacles were the people and lap-tops themselves, and in fact, we initially expected todiscover that the signal strength was better behavedacross a specific angle range (per Figure6) thanacross all angles. Even for the seemingly good caseof both source and destination angles between 0 and45 degrees (i.e., the sender and receiver roughly fac-ing each other), we obtain a distribution (not shown)remarkably similar to Figure12. Other angle rangesalso show the same distribution as Figure12.

Overall, noise-free, reflection-free, obstruction-free, uniformly-radiating environments are simplynot real, and signal strength of individual transmis-

12

150

160

170

180

190

200

210

220

230

0 25 50 75 100 125 150 175 200 225 250 275 300

Sig

nal S

tren

gth

(dB

m +

255

)

Distance in Meters

Figure 12: A scatter plot demonstrating the poor cor-relation between signal strength and distance. Werestrict the plot to beacons both sent and received onthe western half of the field, and show the mean sig-nal strength as a heavy dotted line.

sions will never be a simple function of distance. Re-searchers must be careful to consider how sensitivetheir simulation results are to signal variations, sincetheir algorithms will encounter significant variationonce deployed.

6 ImpactWe demonstrate above that the axioms are untrue, buta key question remains: what is the effect of theseaxioms on the quality of simulation results? In thissection, we begin by comparing the results of ouroutdoor experiment with the results of a best-effortsimulation model, and then progressively weaken themodel by assuming some of the axioms. The purposeof this study is not to claim that our simulator can ac-curately model the real network environment, but in-stead to show quantitatively the impact of the axiomson the simulated behavior of routing protocols.

Clearly, analytical or simulation research in wire-less networking must work with an abstraction of re-ality, modeling the behavior of the wireless networkbelow the layer of interest. Unfortunately, overlysimplistic assumptions can lead to misleading or in-correct conclusions. Our results provide a counter-example to the notion that these axioms are sufficientfor research on ad hoc routing algorithms. We donot claim to validate, or invalidate, the results of anyother published study. Indeed, our point is that theburden is on the authors of past and future studies toa) clearly lay out their assumptions, b) demonstrate

160.5170.5

180.5190.5

200.5

2.5 37

.5 72.5

107.

5

142.

5

177.

5

212.

5

0

50

100

150

200

250

300

350

Number of Data Points

Signal Strength (dBm + 255) Distance

Figure 13: Same as Figure12except that it shows thenumberof observed data points as a function of dis-tance and signal strength. There is significant weightrelatively far away from the mean value.

whether those assumptions are reasonable within thecontext of their study, and c) clearly identify any lim-itations in the conclusions they draw.

While others have used simulation to explorethe impact of different radio propagation mod-els [TMB01, ZHKS04], we use the identical imple-mentation of the routing protocol in both the simula-tor and the experiment [LYN+04], use a large num-ber of nodes in an outdoor experiment [GKN+04],and are able to compare our simulation results withthe actual experiment.

6.1 Our simulator

Our SWAN simulator for wireless ad hoc networksprovides an integrated, configurable, and flexible en-vironment for evaluating ad hoc routing protocols,especially for large-scale network scenarios. SWANcontains a detailed model of the IEEE 802.11 wire-less LAN protocol and a stochastic radio channelmodel, both of which were used in this study.

We used SWAN’s direct-execution simulationtechniques to execute within the simulator thesamerouting code that was used in the experiments fromthe previous section [LYN+04]. We modified thereal routing code only slightly to allow multipleinstances of a routing protocol implementation torun simultaneously in the simulator’s single addressspace. We extended the simulator to read the nodemobility and application-level data logs generated by

13

the real experiment. In this way, we were able to re-produce the same network scenario in simulation asin the real experiment. Moreover, by directly run-ning the routing protocols and the beacon serviceprogram, the simulator generated the same types oflogs as in the real experiment.

In the next few sections, we describe three simu-lation models with progressively unrealistic assump-tions, and then present results to show the impact.

6.2 Our best model

We begin by comparing the results of the outdoor ex-periment with the simulation results obtained withour best signal propagation model and a detailed802.11 protocol model. The best signal propagationmodel is a stochastic model that captures radio signalattenuation as a combination of two effects: small-scale fading and large-scale fading. Small-scale fad-ing describes the rapid fluctuation in the envelopeof a transmitted radio signal over a short period oftime or a small distance, and primarily is causedby multipath effects. Although small-scale fadingis in general hard to predict, wireless researchersover the years have proposed several successful sta-tistical models for small-scale fading, such as theRayleigh and Ricean distributions. Large-scale fad-ing describes the slowly varying signal-power levelover a long time interval or a large distance, and hastwo major contributing factors: distance path-lossand shadow fading. The distance path-loss modelsthe average signal power loss as a function of dis-tance: the receiving signal strength is proportionalto the distance between the transmitter and the re-ceiver raised to a given exponent. Both the free-spacemodel and the two-ray ground reflection model men-tioned earlier can be classified as distance path-lossmodels. The shadow fading describes the variationsin the receiving signal power due to scattering; it canbe modeled as a zero-mean log-normal distribution.Rappaport [Rap96] provides a detailed discussion ofthese and other models.

For our simulation, given the light traffic used inthe real experiment, we used a simple SNR thresh-old approach instead of a more computational in-tensive BER approach. Under heavier traffic, thischoice might have substantial impact [TMB01]. Forthe propagation model, we chose 2.8 as the distancepath-loss exponent and 6 dB as the shadow fading

Experiment Simulation ErrorAODV 42.3% 46.8% 10.5%APRL 17.5% 17.7% 1.1%

ODMRP 62.6% 56.9% -9.2%

Table 1: Comparing packet delivery ratios betweenreal experiment and simulation.

log normal standard. These values, which must bedifferent for different types of terrain, produce sig-nal propagation distances consistent with our obser-vations from the real network. Finally, for the 802.11model, we chose parameters that match the settingsof our real wireless cards. We then conducted thesimulation of the wireless network with 40 nodes,of which 7 did not generate any network traffic, butwere available for selection as potential packet des-tinations. This duplicated the 7 crashed nodes fromthe real experiment, and allowed us to reproduce thesame traffic pattern.

Table1 shows the difference in the overall packetdelivery ratio (PDR)—which is the total number ofpackets received by the application layer divided bythe total number of packets sent—between the realexperiment and the simulation. The simple propa-gation model produced relatively good results: therelative errors in predicted PDR were within 10%for all three routing protocols tested. We caution,however, that one cannot expect consistent resultswhen generalizing the simple stochastic radio propa-gation model to deal with all network scenarios. Af-ter all, this model assumes some of the axioms wehave identified, including flat earth, omni-directionalradio propagation length, and symmetry. Thus thismodel, our best, nonetheless assumes some of thesame axioms we discount in the preceding section!This ironic situation is testimony to the difficulty ofdetailed radio and environment modeling; in situa-tions where such assumptions are clearly invalid—for example, in an urban area—we should expect themodel to deviate further from reality. On the otherhand, this approximation is sufficient for the pur-poses of this paper, because we can still demonstratehow the other axioms may affect performance.

On the other hand, since the model produced goodresults amenable to our particular outdoor experi-ment scenario, we use it in this study as the base-

14

line to quantify the effect of the axioms on simulationstudies. As we show, these assumptions can signif-icantly undermine the validity of the simulation re-sults.

6.3 Simpler models

Next we weakened our simulator by introducing asimpler signal propagation model. We used the dis-tance path-loss component from the previous model,but disabled the variations in the signal receivingpower introduced by the stochastic processes. Notethat these variations are a result of two distinct ran-dom distributions: one for small-scale fading andthe other for shadow fading. The free-space model,the two-ray ground reflection model, and the genericdistance path-loss model with a given exponent—allused commonly by wireless network researchers—differ primarily in the maximum distance that a sig-nal can travel. For example, if we assume that thesignal transmission power is 15 dBm and the re-ceiving threshold is -81 dBm, the free-space modelhas a maximum range of 604 meters, the two-rayground reflection model a range of 251 meters, andthe generic path-loss model (with an exponent of 2.8)a range of only 97 meters. Indeed, the SWAN authorsalso noted that the receiving range plays an impor-tant role in ad hoc routing: longer distance shortensthe data path and can drastically change the routingmaintenance cost [LYN+04].

In this study, we chose to use the two-ray groundreflection model since its signal travel distancematches observations from the real experiment.11

This weaker model assumes Axiom 4: “If I can hearyou at all, I can hear you perfectly,” and specificallythe testable axiom “The reception probability distri-bution over distance exhibits a sharp cliff.” With-out variations in the radio channel, all signals travelthe same distance, and successful reception is sub-ject only to the state of interference at the receiver.In other words, the signals can be received success-fully with probability 1 as long as no collision occursduring reception.

Finally, we consider a third model that further

11When we consider the full experiment field, which providespossible reception ranges of over 500 meters, we see almost noreceptions beyond 250 meters. The 251-meter range of the 2-raymodel is computed from a well-known formula, using a fixedtransmit power (15 dBm) and antenna height (1.0 meter).

0%

20%

40%

60%

80%

100%

0 50 100 150 200 250 300

Bea

con

rece

ptio

n ra

tio

Distance (in meters)

real experimentbest modelno variationsperfect channel

Figure 14: The beacon reception ratio at differentdistances between the sender and the receiver. Theprobability for each distance bucket is plotted as apoint at the midpoint of its bucket; this format is eas-ier to read than the boxes used in earlier plots.

weakens the simulator by assuming that the radiopropagation channel isperfect. That is, if the dis-tance between the sender and the recipient is belowa certain threshold, the signal is received success-fully with probability 1; otherwise the signal is al-ways lost. The perfect-channel model represents anextreme case where the wireless network model in-troduces no packet loss from interference or colli-sion, and the reception decision is based solely ondistance. To simulate this effect, we bypassed theIEEE 802.11 protocol layer within each node and re-placed it with a simple protocol layer that calculatessignal reception based only on the transmission dis-tance.

6.4 The Results

First, we look at the reception ratio of the beaconmessages, which were periodically sent via broad-casts by the beacon service program on each node.We calculate the reception ratio by inspecting the en-tries in the beacon logs, just as we did for the real ex-periment. Figure14plots the beacon reception ratiosduring the execution of the AODV routing protocol.The choice of routing protocol is unimportant in thisstudy since we are comparing the results between thereal experiment and simulations. We understand thatthe control messages used by the routing protocolmay slightly skew the beacon reception ratio due tothe competition at the wireless channel.

Compared with the two simple models, our best

15

0%

10%

20%

30%

40%

50%

60%

70%

80%

30 10 3 1 0.3

Pac

ket d

eliv

ery

ratio

Average packet inter-arrival time at each node (sec)

AODV

real experimentbest modelsno variationsperfect channel

Figure 15: Packet delivery ratios for AODV.

model is a better fit for the real experiment results.It does, however, slightly inflate the reception ratiosat shorter distances and underestimate them at longerdistances. More important for this study is the dra-matic difference we saw when signal power varia-tions were not included in the propagation model.The figure shows a sharp cliff in the beacon recep-tion ratio curve: the quality of the radio channelchanged abruptly from relatively good reception tozero reception as soon as the distance threshold wascrossed. The phenomenon is more prominent for theperfect channel model. Since the model had no inter-ference and collision effects, the reception ratio was100% within the propagation range.

Next, we examine the effect of different simula-tion models on the overall performance of the rout-ing protocols. Figures15–17 show the packet deliv-ery ratios, for the three ad hoc routing algorithms,as we varied the application traffic intensity by ad-justing the average packet inter-arrival time at eachnode. Note the logarithmic scale for thex-axes inthe plots. The real experiment’s result is representedby a single point in each plot.

Figures15–17 show that the performance of rout-ing algorithms predicted by different simulationmodels varied dramatically. For AODV and APRL,both simple models exaggerated the packet deliveryratio significantly. In those models, the simulatedwireless channel was much more resilient to errorsthan the real network, since there were no spatial ortemporal fluctuations in signal power. Without vari-ations, the signals had a much higher chance to besuccessfully received, and in turn, there were fewerroute invalidations, and more packets were able to

0%

10%

20%

30%

40%

50%

60%

70%

80%

30 10 3 1 0.3

Pac

ket d

eliv

ery

ratio

Average packet inter-arrival time at each node (sec)

APRL

real experimentbest modelsno variationsperfect channel

Figure 16: Packet delivery ratios for APRL.

0%

10%

20%

30%

40%

50%

60%

70%

80%

30 10 3 1 0.3

Pac

ket d

eliv

ery

ratio

Average packet inter-arrival time at each node (sec)

ODMRP

real experimentbest modelsno variationsperfect channel

Figure 17: Packet delivery ratios for ODMRP.

find routers to their intended destinations. The per-formance of the perfect-channel model remained in-sensitive to the traffic load since the model did notinclude collision and interference calculations at thereceiver, explaining the divergence of the two simplemodels as the traffic load increases. For ODMRP,we cannot make a clear distinction between the per-formance of the best model and of the no-variationmodel. One possible cause is that ODMRP is amulticast algorithm and has a more stringent band-width demand than the strictly unicast protocols. Aroute invalidation in ODMRP triggers an aggressiveroute rediscovery process, and could cause signifi-cant packet loss under any of the models.

In summary, the assumptions embedded inside thewireless network model have a great effect on thesimulation results. On the one hand, our best wire-less network model assumes some of the axioms, yetthe results do not differ significantly from the realexperiment results. On the other hand, one must beextremely careful when assuming some of the ax-

16

ioms. If we had held our experiment in an envi-ronment with more hills or obstacles, the simulationresults would not have matched as well. Even inthis relatively flat environment, our study shows thatproper modeling of the lossy characteristics of theradio channel has a significant impact on the rout-ing protocol behaviors. For example, using our bestmodel, one can conclude from Figure15 and Fig-ure 17 that ODMRP performed better than AODVwith light traffic load (consistent with real experi-ment), but that their performance was comparablewhen the traffic was heavy. If we use the model with-out variations, however, one might arrive at the oppo-site conclusion, that AODV performed consistentlyno worse than ODMRP. The ODMRP results areinteresting by themselves, since the packet-deliverydegradation as the traffic load increases is more thanmight be expected for an algorithm designed to findredundant paths (through the formation of appropri-ate forwarding groups). Bae has shown, however,that significant degradation can occur as intermedi-ate nodes move, paths to targets are lost, and routerediscovery competes with other traffic [BLG00]. Inaddition, the node density was high enough that eachforwarding group could have included a significantfraction of the nodes, leading to many transmittedcopies of each data packet. An exploration of thisissue is left for future work.

7 Conclusions, recommendations

In recent years, dozens of Mobicom and Mobihocpapers have presented simulation results for mobilead hoc networks. The great majority of these papersrely on overly simplistic assumptions of how radioswork. Both widely used radio models, “flat earth”andns-2 “802.11” models, embody the followingset of axioms: the world is two dimensional; a radio’stransmission area is roughly circular; all radios haveequal range; if I can hear you, you can hear me; ifI can hear you at all, I can hear you perfectly; andsignal strength is a simple function of distance.

Others have noted that real radios and ad hocnetworks are much more complex than the sim-ple models used by most researchers [PJL02], andthat these complexities have a significant impacton the behavior of MANET protocols and algo-rithms [GKW+02]. In this paper, we enumerated

the set of common assumptions used in MANET re-search, and presented a real-world experiment thatstrongly contradicts these “axioms.” The results castdoubt on published simulation results that implic-itly rely on these assumptions, e.g., by assuminghow well broadcasts are received, or whether “hello”propagation is symmetric.

We conclude with a series of recommendations,...for the MANET research community:

1. Choose your target environment carefully, clearlylist your assumptions about that environment,choose simulation models and conditions thatmatch those assumptions, and report the results ofthe simulation in the context of those assumptionsand conditions.

2. Use a realistic stochastic model when verifying aprotocol, or comparing a protocol to existing pro-tocols. Furthermore, any simulation should ex-plore a range of model parameters since the effectof these parameters is not uniform across differ-ent protocols. Simple models are still useful forthe initial exploration of a broad range of designoptions, due to their efficiency.

3. Consider three-dimensional terrain, with moder-ate hills and valleys, and corresponding radiopropagation effects. It would be helpful if thecommunity agreed on a few standard terrains forcomparison purposes.

4. Include some fraction of asymmetric links (e.g.,whereA can hearB but not vice versa) and sometime-varying fluctuations in whetherA’s packetscan be received byB or not. Here thens-2“shadowing” model may prove a good startingpoint.

5. Use real data as input to simulators, where possi-ble. For example, using our data as a static “snap-shot” of a realistic ad hoc wireless network withsignificant link asymmetries, packet loss, elevatednodes with high fan-in, and so forth, researchersshould verify whether their protocols form net-works as expected, even in the absence of mobil-ity. The dataset also may be helpful in the devel-opment of new, more realistic radio models.

17

...for simulation and model designers:

1. Allow protocol designers to run the same codein the simulator as they do in a real sys-tem [LYN+04], making it easier to compare ex-perimental and simulation results.

2. Develop a simulation infrastructure that encour-ages the exploration of a range of model parame-ters.

3. Develop a range of propagation models that suitdifferent environments, and clearly define the as-sumptions underlying each model. Models en-compassing both physical and data-link layerneed to be especially careful.

4. Support the development of standard terrain andmobility models, and formats for importing realterrain data or mobility traces into the simulation.

...for protocol designers:

1. Consider carefully your assumptions of lower lay-ers. In our experimental results, we found thatthe success of a transmission between radios de-pends on many factors (ground cover, antennaangles, human and physical obstructions, back-ground noise, and competition from other nodes),most of which cannot be accurately modeled, pre-dicted or detected at the speed necessary to makeper-packet routing decisions. A routing proto-col that relies on an acknowledgement quicklymaking it from target or source over the reversepath, that assumes that beacons or other broad-cast traffic can be reliably received by most or alltransmission-range neighbors, or that uses an in-stantaneous measure of link quality to make sig-nificant future decisions, is likely to function sig-nificantly differently outdoors than under simula-tion or indoor tests.

2. Develop protocols that adapt to environmentalconditions. In our simulation results, we foundthat the relative performance of two algorithms(such as AODV and ODMRP) can change sig-nificantly, and even reverse, as simulation as-sumptions or model parameters change. Althoughsome assumptions may not significantly affect theagreement between the experimental and simu-lation results, others may introduce radical dis-agreement. For similar reasons, a routing proto-

col tested indoors may work very differently out-doors. Designers should consider developing pro-tocols that make few assumptions about their en-vironment, or are able to adapt automatically todifferent environmental conditions.

3. Explore the costs and benefits of control traffic.Both our experimental and simulation results hintthat there is a tension between the control trafficneeded to identify and use redundant paths andthe interference that this extra traffic introduceswhen the ad hoc routing algorithm is trying to re-act to a change in node topology. The importanceof reducing interference versus identifying redun-dant paths (or reacting quickly to a path loss)might appear significantly different in real exper-iments than under simple simulations, and proto-col designers must consider carefully whether ex-tra control traffic is worth the interference price.

Availability. We will make our simulator and ourdataset available to the research community uponcompletion of the camera-ready version of this pa-per. The dataset, including the actual position andconnectivity measurements, would be valuable as in-put to future simulation experiments. The simulatorcontains several radio-propagation models.

Acknowledgements

We are extremely grateful to the many people thathelped make this project possible.

Jim Baker supplied a floorplan for every build-ing with AP locations marked. Gurcharan Khanna,James Pike, and the FO&M department supplied thecampus base map. Erik Curtis, a Dartmouth under-graduate, painstakingly mapped each floorplan to thecampus base map.

Qun Li, Jason Liu, Ron Peterson, and Felipe Per-rone all provided invaluable feedback on early ver-sions of this paper.

Dr. Jason Redi loaned us his collection of Mobi-com proceedings.

This project was supported in part by a grant fromthe Cisco Systems University Research Program, theDartmouth Center for Mobile Computing, DARPA(contract N66001-96-C-8530), the Department ofJustice (contract 2000-CX-K001), the Department

18

of Defense (MURI AFOSR contract F49620-97-1-03821), and the Office for Domestic Preparedness,U.S. Department of Homeland Security (Award No.2000-DT-CX-K001). Additional funding providedby the Dartmouth College Dean of Faculty office inthe form of a Presidential Scholar Undergraduate Re-search Grant, and a Richter Senior Honors ThesisResearch Grant. Points of view in this document arethose of the author(s) and do not necessarily repre-sent the official position of the U.S. Department ofHomeland Security, the Department of Justice, theDepartment of Defense, or any other branch of theU.S. Government.

References

[BLG00] Sang Ho Bae, Sung-Ju Lee, and MarioGerla. Unicast performance analysis ofthe ODMRP in a mobile ad-hoc net-work testbed. InICCCN 2000, October2000.

[ER00] Moncef Elaoud and Parameswaran Ra-manathan. Adaptive allocation ofCDMA resources for network-levelQoS assurances. In Proceedings of theSixth Annual International Conferenceon Mobile Computing and Networking,pages 191–199. ACM Press, 2000.

[FV02] Kevin Fall and Kannan Varad-han. The ns Manual, April 142002. www.isi.edu/nsnam/ns/ns-documentation.html.

[GC04] Gregor Gaertner and Vinny Cahill.Un-derstanding link quality in 802.11 mo-bile ad hoc networks. IEEE InternetComputing, Jan/Feb 2004.

[GK97] P. Gupta and P. R. Kumar. A systemand traffic dependent adaptive routingalgorithm for ad hoc networks. InProc.of the 36th IEEE Conference on De-cision and Control, pages 2375–2380,December 1997.

[GKN+04] Robert S. Gray, David Kotz, CalvinNewport, Nikita Dubrovsky, Aaron

Fiske, Jason Liu, Christopher Masone,Susan McGrath, and Yougu Yuan.Out-door experimental comparison of fourad hoc routing algorithms. Tech-nical Report TR2004-511, Dept. ofComputer Science, Dartmouth College,2004.

[GKW+02] Deepak Ganesan, Bhaskar Krishna-machari, Alec Woo, David Culler, Deb-orah Estrin, and Stephen Wicker.Com-plex behavior at scale: An experimentalstudy of low-power wireless sensor net-works. Technical Report UCLA/CSD-TR 02-0013, UCLA Computer Science,2002.

[Gra00] Robert S. Gray. Soldiers, agents andwireless networks: a report on a mili-tary application. InProceedings of the5th International Conference and Ex-hibition on the Practical Applicationof Intelligent Agents and Multi-Agents,April 2000.

[Gud91] M. Gudmundson. Correlation modelfor shadow fading in mobile radio sys-tems. Electronics Letters, 27(3):2145–2146, November 1991.

[JLH+99] Per Johansson, Tony Larsson, Nick-las Hedman, Bartosz Mielczarek, andMikael Degermark. Scenario-basedperformance analysis of routing proto-cols for mobile ad-hoc networks. InProceedings of the Fifth Annual Inter-national Conference on Mobile Com-puting and Networking, pages 195–205.ACM Press, 1999.

[JLW+96] Jan Jannink, Derek Lam, JenniferWidom, Donald C. Cox, and NarayananShnivakumar. Efficient and flexi-ble location management techniquesfor wireless communication systems.In Proceedings of the Second AnnualInternational Conference on MobileComputing and Networking, pages 38–49. ACM Press, 1996.

19

[KK98] B. Karp and H. T. Kung. Dynamicneighbor discovery and loopfree, multi-hop routing for wireless, mobile net-works. Harvard University, May 1998.

[KLM +00] O. E. Kelly, J. Lai, N. B. Mandayam,A. T. Ogielski, J. Panchal, and R. D.Yates. Scalable parallel simulations ofwireless networks with WiPPET: Mod-eling of radio propagation, mobilityand protocols.ACM/Kluwer MONET,5(3):199–208, September 2000.

[KNE03] David Kotz, Calvin Newport, and ChipElliott. The mistaken axioms ofwireless-network research. TechnicalReport TR2003-467, Dept. of Com-puter Science, Dartmouth College, July2003.

[Lee82] W. .C. Y. Lee.Mobile communicationsengineering. McGraw-Hill, 1982.

[LLN +02] Henrik Lundgren, David Lundberg,Johan Nielsen, Erik Nordstrom, andChristian Tschudin. A large-scaletestbed for reproducible ad hoc protocolevaluations. In Proceedings of the ThirdAnnual IEEE Wireless Communicationsand Networking Conference (WCNC),pages 412–418. IEEE Computer Soci-ety Press, March 2002.

[LSG02] S.-J. Lee, W. Su, and M. Gerla. On-demand multicast routing protocol inmultihop wireless mobile networks.ACM/Kluwer MONET, 7(6):441–453,December 2002.

[Lun02] David Lundberg.Ad hoc protocol eval-uation and experiences of real world adhoc networking. Master’s thesis, De-partment of Information Technology,Uppsala University, Sweden, 2002.

[LYN +04] Jason Liu, Yougu Yuan, David M.Nicol, Robert S. Gray, Calvin C. New-port, David Kotz, and Luiz Felipe Per-rone. Simulation validation using di-rect execution of wireless ad-hoc rout-ing protocols. InProceedings of the

Workshop on Parallel and DistributedSimulation (PADS), May 2004. Nom-inated for Best Paper award.

[PJL02] K. Pawlikowski, H.-D.J Jeong, and J.-S.R. Lee. On credibility of simulationstudies of telecommunication networks.IEEE Communications, 40(1):132–139,January 2002.

[PR99] C. E. Perkins and E. M. Royer. Adhoc on-demand distance vector routing.In WMCSA, pages 90–100, February1999.

[Rap96] T. S. Rappaport.Wireless Communica-tions, Principles and Practice. PrenticeHall, Upper Saddle River, New Jersey,1996.

[TBTG01] M. Takai, R. Bagrodia, K. Tang, andM. Gerla. Efficient wireless networksimulations with detailed propagationmodels. Wireless Networks, 7(3):297–305, May 2001.

[TMB01] Mineo Takai, Jay Martin, and RajiveBagrodia. Effects of wireless physi-cal layer modeling in mobile ad hocnetworks. In Proceedings of the 2001ACM International Symposium on Mo-bile Ad-Hoc Networking and Comput-ing (MobiHoc 2001), pages 87–94.ACM Press, 2001.

[YLA02] W. H. Yuen, H. Lee, and T. Andersen.A simple and effective cross layer net-working system for mobile ad hoc net-works. In 13th IEEE Int’l Symposiumon Personal, Indoor and Mobile RadioCommunications, September 2002.

[ZHKS04] Gang Zhou, Tian He, Sudha Krishna-murthy, and John A. Stankovic.Im-pact of radio irregularity on wire-less sensor networks. In Proceed-ings of the 2004 International Confer-ence on Mobile Systems, Applications,and Services (MobiSys), pages 125–138, Boston, MA, June 2004. ACMPress.

20