Research and Engineering Centerfor Unmanned Vehicles

Controlled Mobility of Helper Nodes in Delay-Tolerant

Wireless Networks

Summary of Dissertation Project

Daniel Henkel

Outline

The Network EnvironmentResearch Question/MethodologyResearch ContributionsSingle and Parallel Relay ModesEvaluation of Direct, Relay, Ferrying ModesPath Planning with Reinforcement LearningPrototype Ferrying ImplementationPublications

Network Environment - DTNs

We are dealing with Challenged Networks– Long distances (up to 100’s of kilometers)– High network latency (in sec/min/hours)– Varying and asymmetric data rates– No end-to-end paths may exist at any point in time,

i.e., no contemporaneous connectionsOur focus: Delay-Tolerant Networks (DTNs)– Special protocols enable communication (dtnrg.org)

Examples: (high path loss, long distances)– Ad hoc networks, sensor networks (smart dust),

underwater/ underground networks, space and ship-based networks

Node Mobility can be controlled: small airplanes

Three Link Modes in DTNs

GS1

UAV1UAV3

UAV2

Central Idea:Use controlled node mobility to establish best link using most appropriate mode.

Direct

Relay

FerryingA

B

How can A and B communicate in a DTN with mobility-controlled helper nodes?

Modes Defined

Direct: a direct wireless link between sender and receiver characterized by communication distance, the environment, and coding scheme.Relay: data packets traverse a series of wireless direct links from sender via one or multiple relays to the receiver. Relays can be (re-)positioned arbitrarily.Ferrying: data packets are sent wirelessly from sender to a mobile node which then stores, physically carries, and finally forwards them to the destination. Mobile nodes capable of buffering and carrying data are called “ferries”.

Research Question

Given controlled mobility of wireless network nodes, to what extent can mobile helper nodes improve network performance beyond one-hop direct communications, and which relaying and data ferrying strategies would maximize these benefits? How could ferrying be implemented in a delay-tolerant network?

Methodology

Analytical evaluation of throughput and packet delay bounds in direct, relay, and ferry communication. Detailed analysis of single and parallel transmitrelaying and variable data rate ferrying.Computer simulation of relaying and ferrying with long distances and extreme path-loss exponents.Ferry path planning using reinforcement learning to minimize average packet delay.Prototype design and implementation of reliable, delay-tolerant ferrying communication.

Research Contributions

Contributions:– Understanding of trade-offs between Direct, Relay, Ferry modes– Phase-plots show best communication mode for given application

scenario– Understanding of impact of variable communication rate on

ferrying mode– First application of Reinforcement Learning method for ferry path

planning (minimizing avg. packet delay)– OPNET Simulation of ferrying available to research community– Combined Application: Ferry forms communication hub; Relaying

and Direct maximize R, minimize τ– Prototype of reliable, delay-tolerant ferrying infrastructure can be

used by other researchers for data collection with mobile nodes (robots, planes, humans)

Direct Communication

)1(log2 SNIRWR +=

εαd

dPS =)(

αη /0WTkB=

)11(log2 ηεdWRD +=

Shannon capacity law

Signal strength

Thermal noise (normalized)

Data rate

[4]

“Single Transmit” Relay Model

Only one node transmits at any given time, a.k.a., the noise-limited case.

⎟⎠⎞

⎜⎝⎛

++=

111

kdR

kR DRS

⎟⎠⎞

⎜⎝⎛

+

+=

1

)1(

kdR

LkD

RSτ

S Rdk

t=0

Optimizing “Single Transmit”

Where is the trade-off?dk vs. # of transmissions

Optimal number of relays as function of distance, radio parameters and path loss:

),( εαCdkopt ⋅=

1−= −εε

αdckopt ε

εε ε

ccc

+⋅=+

1)1ln(with

Throughput vs. # of relays

[4]

“Parallel Transmit” Relay Model

Multiple nodes can transmit at any given time, a.k.a., the interference limited case

> Optimal distance between parallel transmissions?

{ } ⎭⎬⎫

⎩⎨⎧

+= ),,(

,1min1max ρ

ρρkdR

kR IRP

)1(log2NI

SI PP

PWR+

+=∑ ⎟⎟⎠

⎞⎜⎜⎝

⎛−

++

+=i

I iidkP εε

ε

ρρβ

)1(1

)1(1)1(

S Rρ

t=0 t=0 t=0

Optimizing “Parallel Tx”

Where is the trade-off?– interference vs. # parallel transports

One constellation of k and ρ maximizes throughputSolution using Matlab simulation

k

ρ

R

link reuse factor

100

101

102

103

1

2

3

4

5x 10

6

# of relays

Thr

ough

put [

bps]

100

101

102

103

1

2

3

4

5x 10

6

# of relays

Thr

ough

put [

bps]

100

101

102

103

1

2

3

4

5x 10

6

# of relays

Thr

ough

put [

bps]

100

101

102

103

1

2

3

4

5x 10

6

# of relays

Thr

ough

put [

bps]

Parallel TxSingle TxDirect Tx

d=2km d=4km d=8km d=16km

2km 4km 8km 16km

ε = 5 PN/α = 10-15 W

Analysis: Direct and Relay Modes

Depending on distance and environment there is one “best mode” which maximizes throughput.

[4]

Rate-Distance Phase Plot

Plot showing themode which satisfies a given throughout criterion with lowest average packet delay.

[4]

Optimal Number of Relays Parallel TX

kopt

kopt = ∞

Depending on distanceand path loss exponentan infinite number of relays might be neededto approach themaximum throughput.

Parallel Tx: Optimal Reuse Factor ρ

d=10km PN/α = 4.14·10-15 (based on 802.11)

k+1ρopt

5555565422

5555555423

5555544324

5554332225

5543222226

∞256128643216842ε

ρopt ≈ min{k+1, 5}

Optimal re-use factoras functionof path lossand numberof relays.

Starting ata certain kL,best reuseis always 5!!

[2]

Delay—Rate Phase Plot

Relay

Direct

100

101

102

103

104

105

100

101

102

103

104

data rate [bps]

dela

y[s

ec]

Unachievable

Ferry

Direct

Relay

Plot for specificdistance only.Mode resultschange withdistance.

Plot shows whichmode can achievea desired (R,τ)combination.

λAλB

AB

HF

DC λD

λC

The Ferry Path Planning Problem

What sequence of node visits minimizes average packet delay?

Example: sensor nodes A, B, C, D are generating packets with rates λi destined for collection station H.Ferry F can transport packets between nodes.

One cycle visits every nodeInsight: far-away nodes with little data to send

Visit them less often

Current algorithms based on Traveling Salesman SolutionA Bhub

fA fB

UAV

dA dB

New idea: cycle defined by visit frequencies pi

pA pB

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

hub separation from A as fraction of total distance A−B [%]

optim

al d

elay

as

frac

tion

of n

aive

del

ay [%

]

fA=1 fBfA=3 fBfA=10 fBfA=30 fBfA=100 fB

B

B

The Drawback of TSP

Simple example:

Corresponding inefficiency of TSP:

Idea: express delay in terms of pi, then minimize over set {pi}pi as probability distribution

Expected service time of any packet

Inter-service time: exponential distribution with mean T/pi

Weighted delay:

A

Chub

fA

fC

UAV

dA dB

pA pB

BfB

D

fD

pC

pDdD

dC

∑=i

ii pT τ

GoalMinimize average delay

∑ =i

ip 10≥ip

∑∑=i j i

ijj

Fpfp τ

τ

Stochastic Modeling

[1]

Probability of choosing node i for next visit:

∑=

jjj

iii f

fp

ττ/

/

Implementation: deterministic algorithm1. Set ci = 02. ci = ci + pi while max{ci} < 13. k = argmax {ci}4. Visit node k; ck = ck-15. Go to 2.

Pretty simplistic view of the world !Random selection ignores many parameters -> AI method

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

hub separation from A as fraction of total distance A−B [%]

optim

al d

elay

as

frac

tion

of n

aive

del

ay [%

]

fA=1 fBfA=3 fBfA=10 fBfA=30 fBfA=100 fB

Improvement over TSP!

Stochastic Solution and Algorithm

[1]

Agent– Performs Actions

Environment– Gives Rewards– Puts Agent in

situations called States

Goal:– Learn what to do in

a given state (Policy)

The Beauty:Learns model of environment and retains it.

Agent’s objective: maximize reward over time

New Approach: Reinforcement Learning

Learning optimal strategy from interacting with environment.

Reward Criterium:

∫ −+=1

0

)(),( 0

t

t

t dteNtasr βλ

Reward Criterion

State Space:

Action Space:

Reward:

Solution Method:

Round trips through subset of nodes, e.g., A, B, C, D, AB, AC,…DCBA

Temporal Difference Learning implemented in C++ RL framework

Tuples of accumulated traffic per node )},,,{( DCBA bbbbS =

[1]

Ferry Path Planning Results

TSP = Traveling Salesman solutionRL = Reinforcement Learning

RR = Round Robin (naive)STO = Stochastic Modeling

[1]

Sensor

Sensor

Sensor

Ferrying Implementation

TerminusGateway

SMS

GatewaySMS

SMS

Network 1 Network 2 Network 3

6.0.0.0 10.0.0.0 192.168.0.0

MANET InternetEthernet

• Custody transfer, modified UDP protocol make E2E trans-mission reliable in DTNs.• Implemented using ClickModular Router framework.• Single board computers,Atheros WiFi chipsets.

[6,7]

Publications I

[1] D. Henkel and T. X. Brown, “Towards autonomous data ferry route design through reinforcement learning,” in Proc. of IEEE WoWMoM/Autonomic and Opportunistic Communications (AOC’08), Newport Beach, CA, June 23–27, 2008.

[2] D. Henkel and T. X. Brown, “Delay-tolerant communication using mobile robotic helper nodes,” in Proc. of ICST WiOpt/Workshop on Wireless MultihopCommunications in Networked Robotics (WMCNR’08), Berlin, March 31–April 4, 2008.

[3] D. Henkel and T. X. Brown, “Route design for UAV-based data ferries in delay tolerant wireless networks,” in Proc. of AIAA Infotech@Aerospace 2007 Conference and Exhibit, Rohnert Park, CA, May 7–10, 2007.

[4] D. Henkel and T. X. Brown, “Optimizing the use of relays for link establishment in wireless networks,” in Proc. of IEEE Wireless Communications and Networking Conference (WCNC’07), Hong Kong, March 11–15, 2007.

[5] D. Henkel and T. X. Brown, “On controlled node mobility in delay-tolerant networks of un-manned aerial vehicles,” in Proc. of International Symposium on Advanced Radio Technologies, ISART ’06, Boulder, CO, March 2006, pp. 7–9.

[6] D. Henkel, C. Dixon, J. Elston, and T. X. Brown, “A reliable sensor data collection network using unmanned aircraft,” in Proc. Second International Workshop on Multi-hop Ad Hoc Networks: from theory to reality (REALMAN), Florence, Italy, May 26, 2006.

Publications II

[7] D. Henkel and T. X. Brown, “Sensor data collection through a delay-tolerant MANET of small unmanned aircraft,” in Proc. of 6th Scandinavian Workshop on Wireless Ad-hoc Networks (Adhoc’06), Johannesberg Estate, Stockholm, May 3-4, 2006.

[8] A. Jenkins, D. Henkel, and T. Brown, “Poster/demo: Reliable data collection in challenged networks using unmanned aircraft,” in Proc. of ACM MobiCom Workshop on Challenged Networks (CHANTS’07), Montreal, CAN, September, 9–14 2007.

[9] A. Jenkins, D. Henkel, and T. Brown, “Sensor data collection through unmanned aircraft gateways,” in Proc. of AIAA Infotech@Aerospace 2007 Conference and Exhibit, Rohnert Park, CA, May 7–10, 2007.

[10]A. Jenkins, D. Henkel, and T. Brown, “Sensor data collection through gateways in a highly mobile mesh network,” in Proc. of IEEE Wireless Communications and Networking Conference (WCNC’07), Hong Kong, March 11–15, 2007, pp. 2784–2789.

[11] C. Dixon, D. Henkel, E. Frew, and T. X. Brown, “Phase transitions for controlled mobility in wireless ad hoc networks,” in AIAA Guidance, Navigation, and Control Conference, Keystone, CO, August 2006.

[12] S. Jadhav, T. X. Brown, S. Doshi, D. Henkel, and R.-G. Thekkekunnel, “Lessons learned constructing a wireless ad hoc network test bed,” in Proc. of WiOpt/Workshop in Wireless Network Measurements WiNMee, Trentino, Italy, April 3–7, 2005.

Publications III

[13] E. W. Frew, T. X. Brown, C. Dixon, and D. Henkel, “Establishment and maintenance of a delay tolerant network through decentralized mobility control,” in Proc. IEEE International Conference On Networking, Sensing and Control, Ft Lauderdale, FL, Apr. 2006, pp. 23–25.

[14] T. X. Brown, S. Doshi, S. Jadhav, D. Henkel, and R.-G. Thekkekunnel, “A full scale wireless ad hoc network test bed,” in Proceedings of ISART’05, ser. NTIA Special Publications SP-05-418, March 2005, pp. 51–60.

[15] T. X. Brown, B. Argrow, C. Dixon, S. Doshi, R.-G. Thekkekunnel, and D. Henkel, “Ad hoc UAV-ground network (AUGNet),” in AIAA 3rd Unmanned Unlimited Technical Conference, Chicago, IL, September 2004.

Book Chapter[16] T. Brown, B. Argrow, E. Frew, C. Dixon, D. Henkel, J. Elston, and H. Gates, Emerging

Technologies in Wireless LAN. Cambridge University Press, 2008, ch. Experiments Using Small Unmanned Aircraft to Augment a Mobile Ad Hoc Network, pp. 695–717.