A Design Space Analysis of Internet Connectivityfor Mobile Ad hoc Networks

Abstract

We present a design space analysis of the problem of providingInternet connectivity for mobile ad hoc networks (MANETs).Currently there are a plethora of proposals to solve this problemin the research community. However, we argue that many ofthe existing designs suffer from complexity and design solutionsthat have not been properly evaluated. One reason for this isthe lack of implementations. We believe that a design spaceanalysis will help to clear the field and lead to better designs. Weillustrate this approach by presenting a new system design forMANET Internet connectivity. We have evaluated our system insimulation and have found that not only is it more simple thancommon existing approaches, but also more efficient and lesslikely to fail in the face of routing updates. In addition, we haveimplemented our system in a real testbed to illustrate its abilityto provide a complete mobility solution together with MobileIP.

1 Introduction

In recent years, routing protocol implementations for mo-bile ad hoc networks (MANETs) have become increas-ingly abundant, but practical experiences from real worldscenarios are still limited. One explanation is that stan-dard scenarios tend to be too limited in scope or that peo-ple are not convinced about their applicability to reality.Internet connectivity is a potential service that could in-crease the benefit of MANETs and also make the appli-cation scenarios more relevant. However, one obstacleon the way to reach that goal is the lack of consensus inthe research community on what it concretely means tohave Internet connectivity in a MANET. It is clear thatit means that ad hoc nodes should be able to establishcommunication with hosts in the Internet, but does it alsomean that those nodes should, for example, be able tomove within or between networks, change between mul-tiple gateways or use several at once, or that nodes shouldbe globally reachable or not? Therefore, the problem weconsider in this paper is the design of Internet connectivityfor MANETs that can handle node mobility, both withinand inbetween networks, having continuous and uninter-rupted Internet connections whenever there is at least onepotential route to one or more gateways. Without this

clear view of the problem of Internet connectivity it isdifficult or impossible to design a solution that is robust,efficient and that in the end solves the problem becausethe problem is not clearly specified.

Herein lies also the problem with the state of MANETInternet connectivity in general as we see it. Whenwe have tried to implement Internet connectivity forMANETs we found that many of the proposals out thereare confusing and difficult to understand and evaluate.That is often because they are not clear about what prob-lem they try to solve, other than the loosely defined prob-lem of Internet connectivity. Another reason is complex-ity. Proposals, typically in the form of Internet drafts, tryto cover all angles by stating they support multiple so-lutions to the same problem, e.g., forwarding strategy.The interactions between different system elements andrespective choice of solution are therefore difficult to pre-dict. The drafts have generally not been implemented orevaluated, other than sometimes in simulation. Hence thesoundness of their approach is not proven and the assump-tions and design choices lead to fragile designs or designsthat are not applicable in reality. One concrete example ofthis is the commonly suggested usage of a default routein a MANET that under some circumstances, as we showlater, makes nodes experience stalled TCP connections inthe face of multiple gateways to the Internet.

The question then is, how can we avoid designs thatsuffer from problems like this and in the end constructmore robust Internet connectivity designs for MANETs?In this paper we present one approach that we believe an-swers this question to satisfaction. This approach con-sists of, through a problem diagnosis, defining a designspace that aims to provide reference points for the analy-sis and evaluation of existing design proposals, as well asto improve the quality of new designs. Our hope is thatthis design space will give researchers in the field a morecoherent view of the problems to solve, the trade-offs indesign choices and in the end lead to less effort spent ondivergent proposals.

The primary contribution of this paper is our diagnosisand presentation of the design space for Internet connec-tivity. A second contribution is the description, imple-mentation and evaluation of a complete system for Inter-net connectivity based on this diagnosis. Our design isrobust in that it works in very challenging scenarios and

hence less stringent ones too, whilst achieving an accept-able trade-off between performance and overhead. In ad-dition to robustness, our design is simple and flexible bysupporting, e.g, multi-homing and interfacing to MobileIP while still requiring minimal modifications to existingrouting protocols. The design uses tunneling to achieveindirection (non shortest path routing) and has been in-tegrated with an implementation of the AODV [14] rout-ing protocol. The design is evaluated in simulation bycomparing it to another common proposal using a stan-dard default route also implemented by us. Our resultsshow that our design is more robust, achieves better per-formance and is more flexible. Since our implementationalso works in the real systems, we provide results fromreal world experiments to illustrate the interfacing withMobile IP.

The rest of the paper is structured as follows. In section2 we introduce and diagnose the general principles andproblems of providing Internet connectivity for MANETsthat provides the input for defining our design space. Thefollowing section 3 reviews related work in the context ofthis diagnosis. In section 4 we define the design spacefor MANET Internet connectivity. The following sec-tion 5 describes our system for Internet connectivity inMANETs that we have designed based on our designspace analysis. Section 6 reports on results from evalu-ating our system in simulation through a comparison toanother competing design. We also provide results fromreal world experiments showing our design’s applicabilityto reality. Section 7 concludes the paper with a discussion.

2 Problem Diagnosis

In this section we diagnose the problem of Internet con-nectivity in MANETs to provide the necessary input inorder to define our design space. Our diagnosis is basedon an as challenging scenario as possible, without beingunrealistic. That is because the design space is defined byall possible design solutions and a more challenging sce-nario means more solutions. We start by decomposing theproblem of Internet connectivity in ad hoc networks intothree sub problems:

i) Determining a node’s location, ii) Discover-ing gateways and iii) Establishing and main-taining consistent forwarding states to gate-ways.

The natures of these problems are different dependingon the assumptions for the specific scenario. Unless thescenario is very specific or there is an administrative en-tity in the network, it is hard to make any assumptions onwhat the network looks like. An ad hoc network is, bydefinition, to some degree unmanaged. Under those cir-cumstances it is not possible to assume that there is, for

example, only one gateway, that nodes move in a certainway or that nodes use a specific prefix for their config-ured IP address. Hence, we argue that a general Internetconnectivity solution must be robust enough to handle themost most challenging scenarios. We define such a sce-nario with the following assumptions:

1. There might be multiple gateways to the Internet

2. Nodes are mobile, at both micro and macro scales

3. The routing protocol is reactive and hop-by-hop, i.e.,each node has a limited horizon in the view of thenetwork and only knows the next hop towards a des-tination.

4. Nodes do not share a common IP-prefix

To list a few example applications where we be-lieve this type of scenario makes sense we point toRoboCupRescue [3], a competition to build autonomousrelief robots where some teams use ad hoc networkingto communicate between the robots. The robots couldrelay telemetry information onto the Internet or anotherfixed network where it can be processed by rescue crews.Here robots are deployed quickly in a environment that isunknown and hostile, hence it is not possible to assumeanything less than the worst case. Other potential applica-tions are remote surveillance [4] or planetary explorationwhere a set of autonomous mobile robots collaborativelyexplore the surface of a distant planet, relaying teleme-try information to one or more orbiting satellites part ofthe inter-planetary Internet [10]. Here it is necessary toassume the worst case, because there is no way to easilychange the system after deployment. Another motivationfor a challenging scenario is that a design for such a sce-nario also works in less stringent ones, but the same mightnot be true for a system designed for less stringent sce-narios in the first place and hence would be limited in itsapplicability. We now motivate each of the assumptions1-4 and describe their implications on the sub problemsi), ii) and iii) above.

Multiple Gateways. Since every node is a potentialrouter and there is no sole administrator, a node might alsobe a gateway. Any node with an Internet connection couldpotentially offer that service to other nodes in the ad hocnetwork if it so wishes. Multiple gateways have implica-tions for problem ii) in that discovering several gatewaysgives the option to either select one gateway at a timeor use several at once. For iii) in that a TCP connectionmight break if the forwarding state is suddenly re-pointedto another gateway somewhere along a path without theexplicit knowledge of the source of the connection.

2

Mobility. For the second point we argue that nodesmight be (micro) mobile within a MANET, but theyshould also be able to seamlessly move between differentMANETs and be (macro) mobile between a MANET andthe Internet. The latter assumption might require, e.g.,Mobile IP [13] and hence integration with the Internetconnectivity system. The mobility assumption also hasimplications on i) and iii). Agent registration must matchthat of the currently used gateway and if a route switchesto another gateway, the source nodes using that route mustbe notified so that they can re-register with the new agentthere.

Routing. The mobility assumption implies a routingprotocol that reacts swiftly to topology changes. The im-plications of reactiveness for problem i) are that the proto-col only maintains a partial network state (routes to activedestinations only). Therefore, in combination with prefix-less addressing, there is no way to easily determine nodelocations, i.e., whether a node is located in the MANETor in the Internet. For ii) it is important that the Internetconnectivity design supports reactive gateway discovery.The partial network view of the routing protocol in com-bination with hop-by-hop forwarding is a problem for iii).Each hop on the forwarding path runs the risk of repeat-ing the problem of determining node locations for everypacket.

Addressing. Prefix-less or flat addresses is a commonassumption in the ad hoc network research communityand is a requirement for macro mobility. A node should,in line with the Mobile IP specification, be able to bringits preconfigured home address into the ad hoc networkand use it for routing. Hence, there is no common prefixamong nodes and the ad hoc network is flat in both a rout-ing and addressing sense. As mentioned above, this hasimplications for problem i) in combination with reactiverouting. Using a proactive protocol or prefixes/subnetssolve the problem since node locations can be determinedeither by checking the routing table or by examining theIP address prefix of the destination address.

In addition to the functionality for operation in theworst case scenario, an Internet connectivity design couldoffer optional functionality for flexibility, for example,exploiting multiple gateways for the purpose of multi-homing or load balancing. Before defining our designspace for Internet connectivity in MANETs we review therelated work in the context of the problem diagnosis.

3 Related Work

We classify related work in two main categories: 1) Inter-net drafts that describe a system framework or protocol.Some drafts that we have reviewed are outdated and are

no longer easily found. They are therefore not addressedhere. 2) Publications presenting an evaluation of a sys-tem, either a new or one from category 1. These almostexclusively focus on evaluating the overhead of differentMobile IP agent or gateway discovery approaches in sim-ulation. Since this is not the focus of this paper, we leavemost of them out.

Both categories have in common that the designs gen-erally have not been implemented, except sometimes insimulation. Code is almost never available. Thereforethey are hard to evaluate and the details of each systemhard to grasp. We now review the categories in order.

3.1 Internet drafts

Belding-Royer et al. propose Globalv4 [5], which inte-grates Mobile IP with the AODV routing protocol. Glob-alv4 assumes flat addressing and hence a destination’s lo-cation is determined by a local flood. If no route reply isreceived from the local search for a destination it is as-sumed that the destination is on the Internet. Delay is anobvious concern for this approach. In addition to floodingthe network for determining node locations, gateway dis-covery is performed either by flooding the network withan agent solicitation, or by having the gateway periodi-cally flood the network with agent advertisements. Wefind the latter approach odd considering the reactive rout-ing. Since Globalv4 is targeted towards integration withMobile IP it implies that there might be multiple MIPagents in the network acting as gateways. Therefore, wefind it surprising that the routing approach taken is hop-by-hop and hence there is no way to enforce that datapackets are routed through the same gateway as a nodeis currently registered at. That is because the routing pro-tocol only cares about shortest paths and if another gate-way suddenly is closer, the routing might update to reflectthat. This update might go unnoticed by the source nodeif it happens on an intermediate node on the path to thegateway. We expect that this mismatch might occur occa-sionally until the views of AODV and Mobile IP are thesame again. We have found no available implementationsof Globalv4.

One of the most established Internet connectivity pro-posals are Globalv6 by Wakikawa et al. [18]. It is anInternet draft targeted towards IPv6 networks for bothreactive and proactive routing protocols. In Globalv6,nodes use one link local (MANET) address and one glob-ally routable address for communication with the Inter-net. Intuitively this would double the overhead in theMANET because of the nodes’ dual identities. Gatewaysare discovered through solicitation or gateway advertise-ment floods. In addition to this nodes must also flood thenetwork once more to determine node locations in caseof reactive routing. Forwarding to a gateway is done us-ing the extended default route concept that we later de-

3

scribe in section 4.3.2. This concept gives raise to a num-ber of problems, such as cascading route requests [12],mismatching route state in nodes and more. Only few ofthese problems are addressed in the draft. The details ofthese problems are explained and discussed in section 4.3.Despite being one of the most established proposals forInternet connectivity, we have only been able to find ref-erences to one available implementation of Globalv6 [9].It is for the ns-2 simulator and implements parts of anearly draft.

Jelger et al. [9] propose a system that ensures pre-fix continuity for MANETs that connect to the Internetthrough one or more gateways. All gateways announcetheir prefixes into the ad hoc network. Nodes carefullyselect addresses within the prefix of, e.g., the closest gate-way, creating disjoint stub networks that share the sameprefix. These stub networks contain a routing tree suchthat nodes can restrict themselves to storing a single de-fault route. Such a system works best with proactive rout-ing protocols and low network mobility. It uses IPv6 andtargets specific scenarios where prefix continuity is im-portant (e.g., Hot-spot operators). The focus of Jelger’swork is a complement to other Internet connectivity solu-tions and lies outside the scope of the work presented inthis paper.

3.2 Evaluation of Systems

Sun et al. describe in [16] a system that looks similar toGlobalv4 (note the author overlap). They examine the ef-fect of varying the Mobile IP agent beaconing interval fordifferent network sizes. They also study the performancein terms of average packet latency and AODV overhead.Similar solutions for integrating MIP with ad hoc net-works can be found in [17, 19].

Jonsson et al. studies in [11] the integration of Mo-bile IP in MANETs. They describe a system calledMIPMANET where Mobile IP is adapted to work withMANETs running the AODV routing protocol. Tunnel-ing from ad hoc nodes to the foreign agent is proposed asa way to achieve default route like behavior. However,the main result presented is the effect of using unicastor broadcast transmissions for periodic agent advertise-ments. We believe that periodic agent advertisements arenot suitable for ad hoc networks using reactive routing.Ratanchandani et al. suggest in [15] a similar solutionto MIPMANET. However, they also study the efficiencyof agent discovery and suggest a hybrid approach wherethe TTL of agent announcements is used to limit propa-gation to a n-hop neighborhood. Nodes further away needto send agent solicitation messages to discover agents. Insimulation they experimentally derive an optimal TTL forthis approach.

Gateway discovery in on-demand MANETs is studiedin [7], where Engelstad et al. examines problems with

gateway proxy route replies in the presence of NetworkAddress Translation (NAT). They find that tunneling togateways is one way to avoid race conditions from proxyroute replies when there are multiple gateways. This is inline with our findings as well.

Engelstad, Tønnesen, Hafslund and Egeland [8] studyInternet connectivity in multi-homed proactive ad hoc net-works. They also suggest tunneling to gateways for proac-tive routing and in particular to achieve multi-homing.The routing protocol’s global view of the ad hoc networkmakes it easier to support Internet connectivity.

4 Design Space for MANET Inter-net Connectivity

In this section we explore the design space of MANETInternet connectivity to get a better understanding of thechoices and trade-offs that are available for system de-signers. Table 3.2 shows the system elements and respec-tive design choices that make up the design space. Whatfollows is a discussion of each of the elements, their de-sign choices and their applicability for the scenario de-scribed in section 2.

4.1 Determining a Node’s Location

The problem of determining a node’s location can be han-dled in different ways. If the MANET nodes share a com-mon prefix or if the nodes have a global network view(e.g., as in proactive routing) they can determine node lo-cation by looking at the prefix of the destination or byinvestigating the routing table. However, for our sce-nario there must be an efficient way to handle node lo-cation. One approach suggested by Jonsson in [11] andWakikawa in [18], is to flood the MANET with a routerequest. The lack of replies can be used by the sourcenode as an indication that the destination resides in theInternet. This approach has obvious efficiency issues andincreases the delay of route establishment.

A more efficient approach than timing out on the routerequest is suggested by Broch et al. [6]. A gateway could,in response to a route request, send a proxy route reply tosignal that it can route to the requested destination. Thisis analogous to the functionality of proxy ARP, but overmultiple hops. To use the proxy-signaling, the gatewaymust determine that the destination is not in the ad hocnetwork. This can be done in different ways, includingflooding the network with a new route request, by keepinga list of currently known active nodes (visitor list) or bypinging the destination on the gateway’s network interfaceattached to the Internet.

4

System Element Design Choice

Node location Prefix, flood, gateway, global viewGateway discovery Advertisement, solicitation, hybrid, integrated

Forwarding

Direct︷ ︸︸ ︷

host route, default route,

Indirect︷ ︸︸ ︷

tunneling, source routing

Table 1: Design Space for MANET Internet connectivity.

4.2 Gateway Discovery

Gateways provide a service to other nodes in a MANETin that they offer Internet connectivity. Therefore, gate-way discovery share similarities with service discoveryin general. Two approaches dominate; service solicita-tion and service advertisement, but there are also hybridones that are a combination of both. A service solicita-tion corresponds to the reactive routing approach wherenodes request information on-demand when needed. Theservice advertisement approach in turn corresponds to theproactive approach, where the provider of the service, inthis case the gateway, regularly advertise its services intothe network. In the context of MANETs there are ob-vious trade-offs with these two approaches. It does notmake sense for a reactive MANET to integrate the adver-tisement approach and in the same way it does not makesense for a proactive MANET to use the solicitation ap-proach.

Since a gateway in a MANET is just another ad hocnode, the most efficient design would be to integrate thegateway resolution process with that of path resolution inthe routing protocol. The proxy gateway approach de-scribed in the previous section is an example of such adesign.

Another design consideration is gateway election whenthere are several to chose from. Election can take placeeither at the gateways or at end-nodes. In the proxy ap-proach, gateways can selectively reply to route requestsdepending on specific policies. A network operator mightnot want to announce its gateway services to some nodes,or an overloaded gateway might stop replying when theload reaches a certain threshold. Using end-node election,it is possible for nodes to use heuristics to chose a gate-way, for example, depending on spatial proximity, load, oreven to chose several gateways for multi-homing or loadbalancing. However, the MANET must also support for-warding to multiple gateways.

4.3 Forwarding

The forwarding approach employed in a MANET plays acrucial role for the flexibility of an Internet connectivitydesign.

4.3.1 Direct Forwarding Strategies

The direct forwarding strategies always do “shortest path”forwarding and do not diverge from the default path. Theimplications of Internet connectivity in this context isthat forwarding is performed towards a destination out-side the domain of the MANET. Direct forwarding is typ-ically done hop-by-hop over a transient forwarding stateinstalled in nodes between a source and destination. Inthe design space of MANET Internet connectivity we findtwo main approaches to do forwarding to gateways:

Host Routes A host route is a distributed state installedin all nodes comprising a path and consists of a numberof consecutive mappings between a destination and a nexthop. There is one set of mappings for each destinationand hence there is no aggregation. Since the state is dis-tributed it is possible to do local changes to the state andhence the path without all nodes on the path being awareof it. This also leads to a problem in the context of mul-tiple gateways. It is not possible for end-nodes to enforcewhich gateway a route goes through. We described howthis affects Globalv4 in section 3.

Default Routes The common notion of a default routeis that from a Local Area Network (LAN) where it is arouting table entry pointing to the first router in the In-ternet. It represents the default next hop to send packetsto that do not match any other explicit entry in a host’srouting table. In contrast to a host route entry, whichmatches only one destination, a default route provides ag-gregation as it can map a number of destinations. In aMANET, where there may be several gateways locatedmultiple hops away, the default route concept is not as ap-plicable as in a LAN. There are two main issues: 1) Thedefault route can only express the next hop, hence it is notpossible to associate a specific gateway with the defaultroute (Figure 1). 2) With reactive routing the aggregationprovided by the default route is lost because the lack of arouting table entry for the destination cannot be taken asa sign that a packet should be sent on the default route.This can lead to cascading route requests as mentioned insection 3.

5

4.3.2 Extending the Default Route Concept

There are some suggestions how to extend the defaultroute concept for use in MANETs. Figure 1 shows an ex-ample routing table for a node three hops from a gatewaywith address 192.168.1.1.

Destination Next Hop Hop Cnt

default 366.35.250.151

192.168.1.1 63.3.5.2363.3.5.23 63.3.5.23

default192.168.1.1

_

31

Destination Next Hop Hop Cnt63.3.5.2363.3.5.23

66.35.250.151default

default63.3.5.23

1_

3

(b)(a)

Figure 1: Two examples of routing tables using a defaultroute.

In (a) the default route is used without modification andmaps to the next hop (63.3.5.23). Note that there is noexplicit entry for the gateway and hence there is no wayfor this source node to know which gateway the defaultroute leads to. In section 2 we explained that this can leadto interrupted connections. In (b), suggested in the Glob-alv6 proposal [18], the default route maps to the gatewayaddress which in turn is used to find out the correspond-ing next hop. Here it is possible to know which gatewaythe default route leads to. However, this mapping statemust also be consistent at each hop to the gateway. Both(a) and (b) need an extra host route entry for the destina-tion to avoid subsequent route discoveries with reactiverouting protocols. This state is installed when the nodereceives a route reply.

Although frequently suggested in related work, we findfrom our analysis that a default route is a concept thatmaps badly to MANET routing. The main reasons forthis are: 1) With reactive routing the network must stillbe flooded to determine whether a destination should beassociated to the default route. 2) The extra host routestate that is associated with the default route removes thebenefit of aggregation and combined with 1) the wholepoint of a default route is lost. 3) The host to default routemapping state needs to be replicated on all nodes betweensource and gateway. If the default route changes to in-clude new intermediate nodes, those nodes must also beupdated with all the host route state associated with thatdefault route. We refer to this problem as the state repli-cation problem and show in section 6 that this can lead toerroneous routing. 4) The extra mappings in the routingtable adds extra overhead to the lookup process.

4.3.3 Indirect Forwarding Strategies

An indirect forwarding strategy is one that allows non-shortest path forwarding of packets. This is useful if asource node wants packets to traverse a specific point onthe way to the destination. This point can be, e.g., a proxy,Mobile IP home agent or in the case of MANETs a spe-cific gateway. The reason to specify a gateway to traverse

is that the gateway might have state, e.g, in the case of be-ing a NAT or Mobile IP agent, that the source node is de-pendent upon. Hence, if the route to the Internet switchesto another gateway without the knowledge of the sourcenode, the connections that node maintains to the Internetwill break. The main way to achieve indirection is by us-ing source routing or flow based routing (as in MPLS).Another way is to use encapsulation, i.e., tunneling. Inthis section we will focus on tunneling since it has thebenefit of enabling hop-by-hop forwarding to achieve in-direction by encapsulating the packet in an extra IP headerthat specifies the indirection point. Figure 2 shows tunnel-ing between a source node and a gateway in a MANET.The packets for the Internet destination is decapsulated atthe gateway.

InternetMANET

IP:GW DataIP:Dest

DataIP:Dest

DataIP:Src

DataIP:Src

GatewayCBA

Destination

Source

Figure 2: With tunneling the gateway becomes an indi-rection point. Packets for the Internet are encapsulatedwith a gateway’s address and can be “locally” routed inthe MANET between the source and gateway. Tunnelingin the reverse direction is not necessary since the destina-tion will then be a node in the MANET.

The obvious downside of a tunneling approach is theoverhead and potential complexity of the encapsulation.However, tunneling also provides a number of benefitsthat we describe below:

• Protocol transparency. The tunneling concept istransparent with existing routing protocols. The min-imum required modifications are extra routing tablestates in the source and gateway nodes which do notaffect the protocol. There is no need for new state orrouting modifications at intermediate nodes.

• No cascading route requests. Cascading effects arenot a problem with tunneling because only the sourcenode and the gateway need to know about the desti-nation in the fixed Internet. Inside the ad hoc net-work Internet packets are explicitly addressed for agateway.

• Route aggregation. Tunneling achieves route ag-gregation at intermediate nodes since all Internetdestinations are encapsulated by gateway addressesinstead of one entry for each destination which is thecase for host routes and default routes.

6

• Stability. Once a source node has configured a tun-nel to a gateway, that tunnel will not be diverted toanother gateway unless connectivity with the gate-way is completely lost. In that case the source nodewill be notified and can take proper actions. For ex-ample, to re-register at a new gateway in case thesource node is using Mobile IP.

• Multiple gateways. Source nodes can maintainroutes to multiple gateways for fault tolerance andload balancing. Tunneling allow Internet trafficbound for different gateways over a common inter-mediate hop, which is not the case for default routeforwarding (see Figure 3). Redundant tunnels can be

(a) (b) (c)

Figure 3: Multiple gateway support: (a) A default routepoints to only one gateway at once. (b) With tunnelingtwo nodes can share an intermediate hop while still main-taining tunnels to different gateways, (c) or one node canhave tunnels to two gateways at once.

used as as backup routes if the connectivity to onegateway is lost. This principle will avoid route re-quest floods for all its current Internet destinations atconnectivity loss. Furthermore, tunnels to multiplegateways are useful when a node wants to do a softhand-over between gateways.

• Efficient forwarding. In terms of routing table look-ups, tunneling is more efficient than the extended de-fault route counterpart. A source node needs to per-form two look-ups in the routing table. On interme-diate nodes, only one regular look-up is needed.

There are other potential optimizations that can beachieved with tunneling, usually with the trade-off that itrequires more tight integration with the routing protocol.For example, Intermediate nodes could be made gatewayaware. An extra flag can be used to mark gateway routesin the routing tables. This can potentially avoid route re-quests if the source node directly can determine that itsown packets should be tunneled to the Internet, e.g., byusing a local prefix address.

4.4 Discussion

The analysis of different approaches in the design spaceof Internet connectivity provides a good starting point forsystem designers to better understand the trade-offs andproblems. We believe this is important to construct more

robust, efficient and clear designs that are easy to imple-ment and in the end applicable to reality. There are alsoother things to consider in the design space that provideoptimizations or flexibility in addition to robustness. Oneexample is the ability to support multi-homing. Today itis not uncommon that mobile devices have more than onenetwork interface to connect to the Internet through differ-ent access networks. For example, a laptop may have botha GPRS and a WiFi interface. These multi-homed devicescould route packets over both interfaces at the same timeto achieve smooth hand-over or load balancing. Or, inmulti-homed sites there might be only one network in-terface, but multiple gateways in the same network. Forsimilar reasons, connections to multiple gateways mightbe beneficial.

5 A Design for Internet Connectiv-ity in MANETs

In this section we describe our design of Internet con-nectivity that builds on the design space analysis. Weuse AODV as the reference routing protocol to base ourdesign on, because it matches the targeted scenario pre-sented in section 2. We chose tunneling for our designbecause of the benefits described in section 4.3. For thepurpose of the analysis of our design we also integrateddefault route forwarding as a reference in the comparisonin section 6.

5.1 Implementation Details

We chose to implement the same type of gateway discov-ery and route setup mechanisms for both default routesand tunneling, so that a comparison would be as fair aspossible. We adopted a proxy RREP solution (as sug-gested by Broch et al [6]) where the route discovery pro-cedure of AODV is slightly extended to unify gateway dis-covery and route setup. This modification integrates wellwith AODV’s reactiveness and is backwards compatible.A node initiates a route discovery by flooding the networkwith RREQs as it would normally do when it does nothave a route to a destination. A gateway that receives thisRREQ determines address locality (i.e., whether the desti-nation is an Internet host or an ad hoc node) and will senda RREP to the ad hoc source node if the destination is anInternet host. The address locality check at the gateway isimplemented through a prefix check or using a visitor list.The gateway’s proxy RREP carries an extra AODV ex-tension with the IP address of the requested Internet hostfor which an ad hoc network node issued a RREQ. TheRREP itself looks like a response to a RREQ for the gate-way. This extended RREP is used to configure the defaultroute or tunnel state at the same time as the gateway routeis configured.

7

In the source node’s routing table, gateways are markedwith a G flag. Although not used for any purpose at thispoint, this flag could be used to indicate backup tunnelsfor faster hand-off. The RREP extension received fromthe gateway is used to configure an Internet host entry onthe source node only. This entry points to the gateway,is marked with an I flag and has a limited life time. Itmaps fixed Internet addresses to the appropriate gatewayaddresses and represent tunnel entries. Our tunneling ap-proach has been integrated into the AODV-UU [1] imple-mentation. This code runs in both in simulation (ns-2)and on Linux systems. For the Linux version we use Min-imal IP encapsulation (RFC 2004), which translates to anoverhead of 8 bytes for each data packet sent through agateway. We chose to implement default routes accord-ing to the Globalv6 draft [18]. Routing tables look likethe one in Figure 1 (b). Host route entries on intermedi-ate nodes are necessary to avoid cascading route look-ups.As an option we have also implemented a feature to some-times drop conflicting RREPs that want to update a routeto a new gateway. We show in the evaluation that thisimproves the performance of default routes, but does notsolve all the problems.

Some optimizations still remain. At this point thetunneling implementation does not support intermediatenode reply for Internet destinations. Another optimiza-tion would be to enable the use of backup tunnels. AnInternet destination marked with an I flag, could easilybe re-pointed to the next active tunnel that is marked witha G flag, without the need for a new RREQ flood.

6 Evaluation

In this section we compare two Internet connectivity de-signs with a focus on forwarding strategies. We presentsimulation results that compare the performance of de-fault route and tunnel forwarding using constant bit rate(CBR) UDP traffic and (FTP) TCP traffic. We show thattunnel forwarding constantly performs better than the de-fault route route counterpart. The simulations provide themain result of this paper. We then continue to presentresults from our experimental testbed. The purpose is toprovide a proof-of-concept of how the tunneling approachcan function together with AODV and Mobile IP in a realsystem.

6.1 Simulations

We use ns-2 version 2.26 and the ns-2 AODV-UU im-plementation of AODV. We have gateway forwarding forboth default routes and tunneling. We chose to evalu-ate gateway forwarding in network scenarios where wescale the number of nodes from 10 to 20 mobile nodes,incrementing by two at a time. Two gateways are used inthe simulations. We keep node density fixed at 2× 10−5

nodes per m2. Thus, the area size (with an x:y ratio of1:2) grows with increased number of nodes. We foundthis density to be a good balance between network sizeand number of nodes. We have one mobile node per223 m square (the two gateway nodes excluded) and withthe default ns-2 transmission radius of 250m (using the“TwoRayGround” model), we have reasonable coverageas indicated by the performance figures. With a fixed den-sity, routes on average are likely to be longer as the num-ber of nodes and area size increase. This allows us to eval-uate forwarding behavior with increasing path length. Thead hoc nodes move in the simulated area according to therandom waypoint model with a max speed of 20 m/s. Werandomly generated 50 movement pattern files for eachnetwork size (i.e., number of nodes and area size). Onemovement run lasts for 200 seconds. These 50 patternswere used for all experiments with default routes and tun-neling to ensure that the results were comparable. Per-formance averages were taken over all 50 runs for eachnetwork size and forwarding strategy.

6.1.1 CBR Performance

In our first experiment we examine CBR performance be-cause the traffic is predictable and there is no feedbackloop, i.e., no acknowledgments like in TCP. This alsomeans that it does not matter to which gateway the trafficis forwarded, since there is no return traffic. We wouldtherefore not expect dramatic differences between strate-gies. Two CBR sources, sending 512 byte packets at a rateof 10 per second, is randomly selected among the ad hocnodes. They will start to communicate with a host on thefixed network by randomly choosing one out of two possi-ble. The CBR sources start 5 seconds into the simulationand continue until the end at 200 seconds.

In Figure 4 we see a comparison of aggregated deliv-ery ratios (data packets received divided by data packetstransmitted) for tunnel forwarding and default route for-warding. Although the variance for each point in the di-agram is quite high, the differences between curves aresignificant. The variance is caused by the randomnessin movement patterns. As can be seen from the left fig-ure the delivery ratio decreases with the number of nodes.This is expected since the number of hops to the gatewaywill also increase with the size. Hence will the probabil-ity of connectivity loss also increase with hop length andconsequently the control traffic will increase to handle thelosses. This overall pattern occur in all our measurements.

We note that tunnel forwarding consistently achievesbetter delivery ratio than the default route approach. Ourhypotheses is that the default route solution occasion-ally suffers from the state replication problem (incorrectlyreplicated or missing state on the nodes along a defaultroute). If there is missing state, the AODV protocol willsend a route error message to upstream nodes to invalidate

8

70

75

80

85

90

95

100

8 10 12 14 16 18 20

Del

iver

y ra

tio (

%)

Number of mobile nodes

TunnelingDefault routes

Default routes mod. 0

2

4

6

8

10

12

8 10 12 14 16 18 20

Con

trol

traf

fic /

data

(%

)

Number of mobile nodes

TunnelingDefault routes

Default routes mod.

Figure 4: CBR Delivery ratio and normalized control traffic overhead using 2 CBR sources and increasing path lengths.

their default routes and force them to be rediscovered.This would explain the larger amount of control messageoverhead of default route forwarding compared to tunnel-ing in the right graph of Figure 4. It is likely that missingstate is more frequent when we have more nodes and alarger simulated area, thus on the average longer routes.The delivery ratio in Figure 4 supports this view, since thegap between tunnel forwarding and default route forward-ing increases with more nodes. To verify our hypotheseswe changed the AODV implementation so that it alwaysforwards packets on a configured default route. This willwork because we only have traffic for the two fixed hostsand it provides a reference for how default routes shouldwork without state replication problems. However, this“hack” is not useable in real life. To mitigate the prob-lem of not being able to enforce the usage of a particu-lar gateway, the modification also drops route replies thattry to reconfigure an existing default route to point to an-other gateway. The simulations with these modificationsare called “default route mod.” in the figures and showthat the modifications bring the CBR delivery ratio muchcloser to tunnel forwarding as expected.

6.1.2 TCP Performance

For TCP, it is important that the return traffic (i.e., the ac-knowledgments) from the fixed network is sent throughthe same gateway as the forward traffic. Otherwise TCPacknowledgments might get lost in case they are sent to agateway which does not have connectivity with the sourcenode. To support this we modified ns-2’s Mobile IP towork with the AODV implementation. Mobile IP’s agentdiscovery was removed and replaced with AODV’s RREQmechanism just to simplify the set-up. For other partsMobile IP works as specified. Whenever a mobile adhoc node selects or switches gateway it will register withthe agent at that gateway so that return traffic is deliveredthere.

For this experiment, our scenario’s two gateways areassigned as Home Agent (HA) and Foreign Agent (FA),respectively. The scenario configuration otherwise re-mains the same. We created two FTP sources. The aggre-gated throughput of these two is limited by the gatewaycapacity.

In Figure 5, we see that the expected drop in per-formance with the number of nodes, caused by the in-creased probability of losing connections. When com-paring strategies we see that tunnel forwarding constantlyachieves a higher TCP throughput than default route for-warding. The lower throughput for default routes is likelycaused by a change in the default route → gateway map-ping somewhere on the path to a gateway without thesource node being notified. Consequently, the sourcenode never registers at the new gateway and the acknowl-edgments might be lost, resulting in a TCP timeout.

The TCP goodput (ratio of TCP packets successfullydelivered to the total number of TCP packets transmit-ted) in the right figure and the control traffic overheadin Figure 6 supports this view. Surprisingly, the good-put of tunnel forwarding is slightly lower than that of de-fault route forwarding. Although the difference is smallenough to fall within the error margin, one explanationis that with less timeouts for tunneling it will send morepackets than default route and thus also retransmit morepackets. This would reduce the goodput of tunnelingwhile it still has a higher throughput than default route. Atthe same time, default route forwarding is not retransmit-ting that much, indicating that the decreased throughputis caused by timeouts. Control traffic is also likely to in-crease, since with tunnel forwarding, AODV spends moretime delivering packets than idle in timeouts. In combi-nation this will give less goodput for tunneling. Interest-ing is that the modified default route forwarding does notshow a similarly strong improvement in this experiment asin the CBR case. This is in line with our assumptions thatdefault route forwarding does not work well with TCP in

9

0

0.1

0.2

0.3

0.4

8 10 12 14 16 18 20

TC

P T

hrou

ghpu

t (M

bps)

Number of mobile nodes

TunnelingDefault routes

Default routes mod. 0.95

0.96

0.97

0.98

0.99

1

8 10 12 14 16 18 20

TC

P G

oodp

ut

Number of mobile nodes

TunnelingDefault routes

Default routes mod.

Figure 5: TCP throughput and goodput using 2 TCP sources and increasing path lengths.

multiple gateway scenarios. We will explore this issuefurther in the next section.

6.1.3 Maintaining Consistent Gateway Connectivity

In section 4.3 we described the inability to “track” gate-ways with both host routes and default routes. With Mo-bile IP, return traffic should be sent to the gateway atwhich the ad hoc source node is currently registered. Ifthis is a different gateway from the one that forward traf-fic is sent over, the TCP acknowledgments might be stuckthere if there is no alternative path. This could explainwhy TCP with default route forwarding seems to spendmore time idle in timeouts.

0

1

2

3

4

5

6

7

8

9

10

8 10 12 14 16 18 20

Con

trol

traf

fic /

data

(%

)

Number of mobile nodes

TunnelingDefault routes

Default routes mod.

Figure 6: Normalized control traffic overhead in TCP sce-nario.

We wanted to verify with an experiment that the lowthroughput is caused by timeouts and whether droppingconflicting route replies solves the problem. We con-structed a scenario where the routing protocol is subjectedto frequent gateway changes. A mobile node (MN) com-municating with a host on the fixed network moves back

and forth, forcing change in connectivity between twogateways (a HA and a FA). Connectivity with the gate-ways are always possible through intermediate nodes sothat the hop count to the closest gateway is always three.This scenario was created to mimic the experimental setupwe have used for our real world experiments presented insection 6.2. Figure 8 illustrates this scenario.

MN will initiate an FTP file transfer to the fixed host atthe start of the simulation and lasts 200 seconds. Duringthis time MN will move back and forth twice, with equalconnectivity to both gateways at times 25, 75, 125 and175 seconds. Thus a gateway change should be triggeredaround those times. The other nodes remain stationaryand will forward traffic to and from the gateways.

Figure 7 shows a TCP sequence number trace of a sim-ulation run. In this scenario we expected to see time gapsin between sequence numbers corresponding to hand-overpoints. Tunnel forwarding reaches the highest sequencenumber at 200 seconds. There are expected gaps for tun-neling during gateway changes, but they are not so vis-ible due to random effects on TCP. The unmodified de-fault route forwarding on the other hand has two long pe-riods where there are no packets sent at all and TCP time-outs. The first timeout corresponds well to the time of thefirst gateway change and the other with the third gatewaychange. It seems as if traffic is only forwarded over oneof the gateways (the home agent).

We wanted to find out the exact cause of this behav-ior and therefore examined our log files. We found thefollowing explanation: With unmodified default routes,route replies from both gateways at hand-over installsconflicting state, updating the gateway mapping on in-termediate nodes while not propagating correctly to MN.MN believes the Internet host can be reached through theHA, when in reality the packets are forwarded through theFA. Since the forwarding to the fixed host still works, MNwill keep and continue refreshing its default route point-ing to the HA. MN incorrectly concludes that it does not

10

have to register at the FA, causing the acknowledgmentsto be lost at HA. This will continue until MN loses theconnectivity to the FA and regains connectivity with theHA. Thus, TCP will go into a timeout.

0

1000

2000

3000

4000

5000

6000

7000

0 20 40 60 80 100 120 140 160 180 200

Seq

uenc

e N

umbe

r

Time (s)

Sequence Number Trace

Tunneling

Default routes

D. route drop RREP

Figure 7: Sequence number trace showing how defaultroutes have problems with multiple gateways.

Modified default routes drop route replies conflictingwith a configured default route. In the resulting sequencenumber trace we see that this solves the problem as ex-pected. In this isolated case the route reply is the culprit.However, dropping these conflicting route replies seemsto have little effect in the general case as we experiencedfrom the CBR and TCP simulations. Thus we concludethat the state replication problem has a bigger impact onthe performance than the gateway tracking problem.

6.2 Experimental Results

In this section we illustrate the functionality of ourreal world implementation of Internet connectivity usingAODV and Mobile IP. We have already investigated ourdesigns ability to perform well and correctly in simula-tion. The purpose here is therefore to show that our sys-tem works in the real world as well and can interface withMobile IP to provide uninterrupted TCP connections tothe Internet while changing gateway.

We have implemented an experimental testbed usingour design with tunnels toghether with Mobile IP. Ourmobile host (MH) is a IBM T30 2.0 GHz laptop runningLinux while the rest of the ad hoc network nodes (CH)are IBM T31 laptops also running Linux. The foreignagents (FAs) are LinkSys WRT54G routers running theOpenWRT Linux distribution with kernel 2.4.30. All adhoc nodes run the AODV-UU implementation with our In-ternet connectivity. The Home agent (HA) is a 2.4 GHzPentium 4 desktop Linux machine. To provide continu-ous reachability for the MH, Mobile IP is used to redirectthe MHs return traffic whenever it changes its location,i.e., registers with a new FA. The HUT Dynamics Mobile

IP implementation [2] was chosen for this purpose. Wehad to make slight modifications to the Mobile IP imple-mentation to interoperate better with the AODV ad hocrouting.

HA

MH

FA FA

CHHA

MH

FA FA

CH

Figure 8: Example scenario in our experimental testbedsetup using LinkSys routers. The mobile host (MH) hasgateway connectivity over the ad hoc network. Commu-nication with the correspondent host (CH) is establishedthrough a bidirectional tunnel between a foreign agent(FA) and the home agent (HA). When a new gateway isdiscovered, the MH re-registers with the FA at that gate-way and the bidirectional tunnel is re-pointed to the newFA.

The integration of AODV and Mobile IP was imple-mented as follows. We disabled all proactive agent ad-vertisements in the FAs. Whenever the MH floods thenetwork with a AODV route request, the FAs will an-swer with an extended route reply (described in the pre-vious section) indicating that this Internet destination canbe reached through the FA. Immediately after this reply,the AODV daemon on the FA triggers the MIP daemonto send an agent advertisement to the MH on the newlyestablished route. The AODV daemon on the MH willforce its Mobile IP daemon to select the FA that matchesthe gateway selected by AODV. When the MH receivesthe agent advertisement it will send a registration messageto the selected FA that will configure a bidirectional tun-nel between the FA and the HA. The experimental testbedconfiguration is illustrated in figure 8. We ran a number ofexperiments using both bidirectional Ping traffic and TCPfile transfers with varying number of hops to the gateway.Due to space limitations we present only one test usingTCP. The experiment lasts for 60 seconds and during thattime a mobile node moves from one gateway acting as aFA to another FA gateway while transferring data usingTCP to a correspondent host on the Internet, via its homeagent.

Figure 9 shows the sequence number trace for an exam-ple run. The time of mobility is identified by the periodsof jaggedness in the trace. The gateway change occursaround time 35 s, as can be seen by the slight interruptionin TCP’s progress.

11

0

500000

1e+06

1.5e+06

2e+06

2.5e+06

3e+06

0 10 20 30 40 50 60

Nor

mal

ized

Seq

uenc

e nu

mbe

r

Time (s)

Figure 9: Sequence number trace from experimental testshowing the TCP progress while switching gateways.

7 Conclusions

We have diagnosed the problems of providing Internetconnectivity in MANETs and from that we have defined adesign space. We argue that this is important because In-ternet connectivity is a very compelling service that hasthe potential to increase the benefits of MANETs andhence their deployment in real life. A design space willhelp other potential designers to better understand thetrade-offs and problems of the different design choices. Italso provides a framework for evaluating existing Internetconnectivity designs.

One important and distinguishing point in our diagno-sis is that it builds on a scenario that is as challengingas possible. The lack of targeted scenarios for many de-signs make them hard to understand and evaluate. A clearand challenging scenario enforces robustness in the de-sign choices and resulting solution also works for lesschallenging scenarios. In that sense we have a based ourreasoning around a “worst case”. We have surveyed theexisting solutions to Internet connectivity and have foundthat most of them are not robust enough, not even for lessstringent scenarios. Furthermore, many of the solutionsare only proposals and have not been implemented andevaluated, in particular side-by-side, neither in simulationnor in reality.

We have compared two solutions in the design space,one using default routes and one using tunneling. Fromour analysis we found that the commonly proposed de-fault route solutions to Internet connectivity lack the abil-ity to, among other things, express indirection. On theother hand, an indirect forwarding approach using tunnel-ing, does not suffer from the default route’s inherent prob-lems and is also more flexible in terms of multi-homingsupport and is also more efficient. Our conclusion is thattunneling or other approaches to express indirection arerequired to build efficient and robust Internet connectivitydesigns for MANETs. From our analysis and the conclu-sions from the evaluation we believe that our complete de-sign for Internet connectivity is simpler and more efficient

than other approaches that have not been implemented orproperly evaluated. We have illustrated through experi-ments how our system can operate robustly together withMobile IP in a multiple gateway environment and howTCP sessions can be maintained while switching betweendifferent gateways.

References[1] The AODV-UU implementation. http://www.docs.uu.se/

scanet/aodv.

[2] Dynamics Mobile IP. http://dynamics.sourceforge.net.

[3] RoboCupRescue. http://www.rescuesystem.org/robocuprescue/.

[4] Rotondus: Durable mobile robots for outdoor surveillance. http://www.rotundus.se.

[5] E. Belding-Royer, Y. Sun, and C. Perkins. Global connectivityfor IPv4 mobile ad hoc networks, November 2001. IETF InternetDraft, draft-royer-manet-globalv4-00.txt, (work in progress).

[6] J. Broch, D. A. Maltz, and D. B. Johnson. Supporting hierarchyand heterogeneous interfaces in multi-hop wireless ad hoc net-works. In Proceedings of the Workshop on Mobile Computing.IEEE, 1999.

[7] P. Engelstad and G. Egeland. NAT-based Internet connectivityfor on-demand ad hoc networks. In WONS 2004, pages 344–358,2004.

[8] P. Engelstad, A. Tønnesen, A. Hafslund, and G. Egeland. Internetconnectivity for multi-homed proactive ad hoc networks. In TheIEEE International Conference on Communications (ICC) 2004,June 2004.

[9] A. Hamidian. A study of internet connectivity for mobile ad hocnetworks in ns 2. Master’s thesis, Lund Institute of Technology,January 2003.

[10] I. S. I. S. interest group. nterplanetary internet project. http://www.ipnsig.org.

[11] U. J onsson, F. Alriksson, T. Larsson, P. Johansson, and G. Q.Maguire Jr. MIPMANET - Mobile IP for Mobile Ad hoc Net-works. In 1st ACM international symposium on Mobile ad hocnetworking and computing (Mobihoc’00), 2000.

[12] A. Nilsson, C. E. Perkins, A. J. Touminen, R. Wakikawa, and J. T.Malinen. AODV and IPv6 Internet access for ad hoc networks.Mobile Computer and Communications Review, 6(3):102–103.

[13] C. Perkins. IP mobility support for IPv4, January 2002. IETFInternet RFC 3220.

[14] C. Perkins, E. Belding-Royer, and S. Das. Ad hoc on-demanddistance vector (AODV) routing, July 2003. IETF Internet RFC3561.

[15] P. Ratanchandani and R. Kravets. A hybrid approach to internetconnectivity for mobile ad hoc networks. In IEEE WCNC, 2003.

[16] Y. Sun, E. M. Belding-Royer, and C. E. Perkins. Internet con-nectivity for ad hoc mobile networks. International Journal ofWireless Information Networks, 9(2):75–88, April 2002.

[17] Y.-C. Tseng, C.-C. Shen, and W.-T. Chen. Integrating mobile IPwith ad hoc networks. Computer, (5):48–55, 2003.

[18] R. Wakikawa, J. Malinen, C. Perkins, A. Nilsson, and A. Tuomi-nen. Global connectivity for IPv6 mobile ad hoc networks, (workin progress), July 2005. IETF Internet Draft, draft-wakikawa-manet-globalv6-04.txt.

[19] C. Ahlund and A. Zaslavsky. Software solutions to Internet con-nectivity in mobile ad hoc networks. In 4th International Con-ference on Product Focused Software Process Improvement (PRO-FES), 2002.

12