““Data over Time”Data over Time”Delay Tolerant NetworksDelay Tolerant Networks
Joy GhoshJoy GhoshLANDERLANDER
cse@buffalocse@buffalo
OverviewOverview
Challenged Internets (CI)Challenged Internets (CI) Shortfalls of TCP/IP in CIShortfalls of TCP/IP in CI Concept of DTNConcept of DTN
Message-oriented Message-oriented Overlay architectureOverlay architecture Classes of Service (CoS)Classes of Service (CoS)
Routing IssuesRouting Issues ConclusionConclusion
Challenged Internets (CI)Challenged Internets (CI)
Some causes:Some causes: Host / Router mobility extremesHost / Router mobility extremes Power managementPower management InterferenceInterference
ExamplesExamples Terrestrial mobile nets: partitionsTerrestrial mobile nets: partitions Sensor / Actuator: power savingSensor / Actuator: power saving Military ad-hoc: jammingMilitary ad-hoc: jamming
Paths & LinksPaths & Links
High end-to-end latencyHigh end-to-end latency Transmission timeTransmission time Propagation delayPropagation delay Processing timeProcessing time Queuing timeQueuing time
Low asymmetric data ratesLow asymmetric data rates Return channel may be even absentReturn channel may be even absent Avoid chatty conversationAvoid chatty conversation Data carousels / de-fragmentationData carousels / de-fragmentation
Non-faulty DisconnectionNon-faulty Disconnection Due to motionDue to motion
Predictable (satellites, busses) Predictable (satellites, busses) Opportunistic (node coming in range)Opportunistic (node coming in range) End user or router motionEnd user or router motion Third entity motion causing blockThird entity motion causing block
Due to low-duty-cycleDue to low-duty-cycle PredictablePredictable Low capability devices save powerLow capability devices save power Usually triggered by events - sensorsUsually triggered by events - sensors
Routing Routing lack of reach is normal – not a faultlack of reach is normal – not a fault Pre-schedule data based on outage infoPre-schedule data based on outage info
Paths & LinksPaths & Links
Long Queuing TimesLong Queuing Times Next hop unavailableNext hop unavailable Alternative next hops Alternative next hops Persistent storagePersistent storage
Paths & LinksPaths & Links
Interoperability considerationsInteroperability considerations Most protocols only use LL & MACMost protocols only use LL & MAC Simple & local in scopeSimple & local in scope May use application specific framing May use application specific framing
formatsformats May fail to implement reliability, May fail to implement reliability,
congestion control, and securitycongestion control, and security Not well matched for Internet ProtocolsNot well matched for Internet Protocols Minimal assumptions of underlying Minimal assumptions of underlying
protocol stack capabilities for protocol stack capabilities for interoperationinteroperation
Network ArchitectureNetwork Architecture
SecuritySecurity Exotic media Exotic media forwarding costly forwarding costly Protection of forwarding serviceProtection of forwarding service Access control matrix for CoSAccess control matrix for CoS Authentication at critical pointsAuthentication at critical points End-to-end security not suitableEnd-to-end security not suitable
Too much delay for key exchangeToo much delay for key exchange Drop unwanted traffic ASAPDrop unwanted traffic ASAP
Network ArchitectureNetwork Architecture
Limited longevityLimited longevity Hostile environmentsHostile environments Power exhasutionPower exhasution Delegation of reliable deliveryDelegation of reliable delivery
Low duty cycleLow duty cycle Save power / event triggeredSave power / event triggered
Limited resourcesLimited resources Limited memory & processingLimited memory & processing
End System CharacteristicsEnd System Characteristics
OverviewOverview
Challenged Internets (CI)Challenged Internets (CI) Shortfalls of TCP/IP in CIShortfalls of TCP/IP in CI Concept of DTNConcept of DTN
Message-oriented Message-oriented Overlay architectureOverlay architecture Classes of Service (CoS)Classes of Service (CoS)
Routing IssuesRouting Issues ConclusionConclusion
TCP/IP basicsTCP/IP basics
Expectations of TCP/IPExpectations of TCP/IP Map: IP packet format, domain names Map: IP packet format, domain names
physical network physical network Fundamental serviceFundamental service
Fully-connected, best-effort, unicast Fully-connected, best-effort, unicast datagram transportdatagram transport
Suffers when:Suffers when: Links change radically (wireless, ATM)Links change radically (wireless, ATM) Service enhanced significantly (IP Service enhanced significantly (IP
multicast, QoS)multicast, QoS)
TCP/IP applied directly to CITCP/IP applied directly to CI
Transport Layer (TCP, SCTP, Transport Layer (TCP, SCTP, UDP)UDP) RTT too high for slow startRTT too high for slow start Waste of bandwidthWaste of bandwidth False congestion indicationFalse congestion indication MSL in RFC793 is 2 mins MSL in RFC793 is 2 mins Low Low
TCP/IP applied directly to CITCP/IP applied directly to CI
Network Layer (IP)Network Layer (IP) Lossy network Lossy network smaller frames smaller frames Frequent fragmentationFrequent fragmentation No fragment retransmissionNo fragment retransmission Maximum TTL is 225 Maximum TTL is 225 Low Low
TCP/IP applied directly to CITCP/IP applied directly to CI
Application Layer (Routing)Application Layer (Routing) Delayed soft-state refreshesDelayed soft-state refreshes Label links as non-operating (e.g., Label links as non-operating (e.g.,
RIP, BGP)RIP, BGP) Lack of response to HELLO Lack of response to HELLO
messages (e.g., OSPF)messages (e.g., OSPF) Application Layer (Others)Application Layer (Others)
Delay cause application timeoutDelay cause application timeout Delay may outlive application lifeDelay may outlive application life
Application Development PracticeApplication Development Practice
Network based applicationsNetwork based applications Application level timeoutsApplication level timeouts Lack of failover – no auto routeLack of failover – no auto route Synchronous programming Synchronous programming Chatty applications protocols Chatty applications protocols
(FTP requires 6 RTT)(FTP requires 6 RTT) New network API requiredNew network API required
Proxies & Protocol BoostersProxies & Protocol Boosters
Performance Enhancing Proxies (PEPs), Performance Enhancing Proxies (PEPs), Protocol Boosters - Link repair approachesProtocol Boosters - Link repair approaches
““fool” TCP/IP based end stations to operate fool” TCP/IP based end stations to operate more efficiently over paths with poor linksmore efficiently over paths with poor links
Different levels of transparencyDifferent levels of transparency Local connection terminationLocal connection termination Modification of TCP ACK streamModification of TCP ACK stream Retransmit lost TCP packets w/o end user inputRetransmit lost TCP packets w/o end user input
Use of PEPs discouraged due to fragilityUse of PEPs discouraged due to fragility Confound end-to-end diagnosticsConfound end-to-end diagnostics Hurt security below transport (IPSEC)Hurt security below transport (IPSEC) Alternative: application layer proxiesAlternative: application layer proxies
Internet to special network name mapping and Internet to special network name mapping and protocol translationprotocol translation
Use of Electronic MailUse of Electronic Mail
ProsPros Asynchronous message deliveryAsynchronous message delivery Delay toleranceDelay tolerance Flexible name/address semanticsFlexible name/address semantics
ConsCons Lack of dynamic routingLack of dynamic routing SMTP is a chatty protocolSMTP is a chatty protocol Mostly reliable with likely failure Mostly reliable with likely failure
notificationnotification
OverviewOverview
Challenged Internets (CI)Challenged Internets (CI) Shortfalls of TCP/IP in CIShortfalls of TCP/IP in CI Concept of DTNConcept of DTN
Message-oriented Message-oriented Overlay architectureOverlay architecture Classes of Service (CoS)Classes of Service (CoS)
Routing IssuesRouting Issues ConclusionConclusion
Delay Tolerant NetworkDelay Tolerant Networkmessage-oriented overlay architecturemessage-oriented overlay architecture
Interoperability among CI & InternetInteroperability among CI & Internet Overlaid above different protocol stacks in Overlaid above different protocol stacks in
various networksvarious networks Store & forward message delivery for non-Store & forward message delivery for non-
interactive trafficinteractive traffic a-priori knowledge of communication a-priori knowledge of communication
capacity needs (intermediate storage, capacity needs (intermediate storage, retransmission bandwidth)retransmission bandwidth)
Virtual circuits unsuitable due to pre-Virtual circuits unsuitable due to pre-establishment of networkestablishment of network
Favors asynchronous I/O programmingFavors asynchronous I/O programming
Network of Regional NetworksNetwork of Regional Networks
Similar protocol stacks in one regionSimilar protocol stacks in one region DTN Gateways – Metanet WaypointDTN Gateways – Metanet Waypoint
Provides entry to a region (protocol translation)Provides entry to a region (protocol translation) Name mapping from global tuple to locally resolved nameName mapping from global tuple to locally resolved name Enforces policy and controlEnforces policy and control Persistent storage for reliable deliveryPersistent storage for reliable delivery
Delay Isolation via Delay Isolation via Transport Layer TerminationTransport Layer Termination
Persistent storage
Bundle EncapsulationBundle Encapsulation
Bundles consist of 3 things:Bundles consist of 3 things: Source application’s user dataSource application’s user data Control information from source application to Control information from source application to
destination application (how to handle data)destination application (how to handle data) Bundle headerBundle header
DTN Region NamingDTN Region Naming
Name TuplesName Tuples
(Late Binding)
InterPlaNetary (IPN) Internet InterPlaNetary (IPN) Internet
Class of Service (CoS)Class of Service (CoS)
Challenged Internet Challenged Internet limited limited resourcesresources
Priority based resource Priority based resource allocationallocation
Subset of types of service by Subset of types of service by USPSUSPS
Class of Service (CoS)Class of Service (CoS)
Challenged Internet Challenged Internet limited resources limited resources Priority based resource allocationPriority based resource allocation USPS has simple but elaborate servicesUSPS has simple but elaborate services Subset of types of service by USPSSubset of types of service by USPS DTN CoSDTN CoS
Custody TransferCustody Transfer Return ReceiptReturn Receipt Custody-Transfer NotificationCustody-Transfer Notification Bundle-Forwarding NotificationBundle-Forwarding Notification Priority of Delivery: Bulk, Normal, ExpeditedPriority of Delivery: Bulk, Normal, Expedited AuthenticationAuthentication
DTN Class of Service (CoS)DTN Class of Service (CoS)
Custody TransferCustody Transfer
Path Selection & SchedulingPath Selection & Scheduling No end-to-end pathNo end-to-end path Routes are a cascade of time-dependent Routes are a cascade of time-dependent
contactscontacts Contact parametersContact parameters
Start & end timesStart & end times CapacityCapacity LatencyLatency End pointsEnd points DirectionDirection
Predictable vs. Opportunistic contactsPredictable vs. Opportunistic contacts Multi-commodity flow optimization problemMulti-commodity flow optimization problem ““flows over time” optimization problem – flows over time” optimization problem –
NP-hardNP-hard More under “Routing Issues”More under “Routing Issues”
Protocol Translation &Protocol Translation &Convergence LayersConvergence Layers
Bundle forwarder may augment Bundle forwarder may augment underlying protocol stackunderlying protocol stack Reliable deliveryReliable delivery Connection setup with notificationConnection setup with notification Flow controlFlow control Congestion controlCongestion control Message boundariesMessage boundaries
May fall back to retransmission May fall back to retransmission timers in case of failed connectionstimers in case of failed connections
Structure of Bundle ForwarderStructure of Bundle Forwarder
Time SynchronizationTime Synchronization
Required for scheduling, path Required for scheduling, path selection, and congestion selection, and congestion managementmanagement
Devices in challenged environments Devices in challenged environments often collect data / position w.r.t. timeoften collect data / position w.r.t. time
Pre-programmed control instructions to Pre-programmed control instructions to be executed in future timebe executed in future time
Reliability of Reliability of post-facto post-facto data analysisdata analysis Removal of expired pending messagesRemoval of expired pending messages Protocols like NTP are used Protocols like NTP are used
SecuritySecurity
Per-hop security to avoid delayPer-hop security to avoid delay Drop unwanted packets ASAPDrop unwanted packets ASAP Only edge routers need per user certificateOnly edge routers need per user certificate Not secure against router compromiseNot secure against router compromise
Usercertificate
Congestion & Flow ControlCongestion & Flow Control Flow Flow
DTN forwarder uses available underlying region-DTN forwarder uses available underlying region-local transport protocollocal transport protocol
If not available, its constructed at the DTN If not available, its constructed at the DTN convergence layerconvergence layer
CongestionCongestion Long absence of contacts leads to massive Long absence of contacts leads to massive
storage of datastorage of data Data for which custody has been accepted cannot Data for which custody has been accepted cannot
be discarded be discarded Priority queue for custody storagePriority queue for custody storage
Large messages are denied custodyLarge messages are denied custody Messages are spooled FCFS based on priorityMessages are spooled FCFS based on priority
Potential problemsPotential problems Priority invasion (lower custodially recvd blocks high)Priority invasion (lower custodially recvd blocks high) Head-of-line blocking (custodially recvd w/o contact)Head-of-line blocking (custodially recvd w/o contact)
Proactive : admission controlProactive : admission control Reactive: reservation, rejection, etc.Reactive: reservation, rejection, etc.
OverviewOverview
Challenged Internets (CI)Challenged Internets (CI) Shortfalls of TCP/IP in CIShortfalls of TCP/IP in CI Concept of DTNConcept of DTN
Message-oriented Message-oriented Overlay architectureOverlay architecture Classes of Service (CoS)Classes of Service (CoS)
Routing IssuesRouting Issues ConclusionConclusion
Network model for RoutingNetwork model for Routing
Routing IssuesRouting Issues
No end-to-end pathNo end-to-end path Both proactive and reactive failBoth proactive and reactive fail
Capacity is time dependentCapacity is time dependent Long buffering of messagesLong buffering of messages
Multiple contactsMultiple contacts Might be more optimal to wait for Might be more optimal to wait for
future contact than send on future contact than send on available oneavailable one
Routing ObjectiveRouting Objective Maximize probability of message deliveryMaximize probability of message delivery Prevent buffer overflowPrevent buffer overflow Speed up the release of resourcesSpeed up the release of resources Minimize the end-to-end delayMinimize the end-to-end delay
Use knowledge of future topology dynamics in Use knowledge of future topology dynamics in path selectionpath selection
Per-hop routing preferred over Source Per-hop routing preferred over Source routingrouting Flexibility to change next hopFlexibility to change next hop
Message splitting is allowedMessage splitting is allowed Fragmentation of large messagesFragmentation of large messages Each fragment routed independently Each fragment routed independently
Knowledge OraclesKnowledge Oracles
Contacts Summary OracleContacts Summary Oracle Time-invariant / aggregate Time-invariant / aggregate
characteristics of contactscharacteristics of contacts Contacts OracleContacts Oracle
Time-varying DTN multi-graphTime-varying DTN multi-graph Queuing OracleQueuing Oracle
Buffer occupancy at any node at Buffer occupancy at any node at any timeany time
Traffic Demand OracleTraffic Demand Oracle Future traffic demandFuture traffic demand
Routing Algorithm ClassesRouting Algorithm Classes
Zero KnowledgeZero Knowledge No oracles / minimal extremeNo oracles / minimal extreme
Complete KnowledgeComplete Knowledge All oracles / LP formulation All oracles / LP formulation
Partial KnowledgePartial Knowledge One or more oraclesOne or more oracles Practical algorithmsPractical algorithms
Knowledge vs. Performance Knowledge vs. Performance
Zero Knowledge RoutingZero Knowledge Routing
First Contact (FC)First Contact (FC) Forward to first available contactForward to first available contact No sense of direction / may oscillateNo sense of direction / may oscillate
Requires only local knowledgeRequires only local knowledge Trivial implementationTrivial implementation ImprovementsImprovements
Incorporate sense of trajectoryIncorporate sense of trajectory Use path vector to prevent loopsUse path vector to prevent loops
Partial Knowledge RoutingPartial Knowledge Routing
Assigns cost to edgesAssigns cost to edges Costs reflect estimated delay on edgeCosts reflect estimated delay on edge
Queuing time: time till contact availableQueuing time: time till contact available Transmission delay: time to inject into edgeTransmission delay: time to inject into edge Propagation delay: time to travel on edgePropagation delay: time to travel on edge
Computes minimum cost pathComputes minimum cost path Conveniently uses different oraclesConveniently uses different oracles Computationally efficient distributed Computationally efficient distributed
algorithms already exist for shortest-algorithms already exist for shortest-path based routing problemspath based routing problems
Finds however single path from source Finds however single path from source to destination – no optimal splittingto destination – no optimal splitting
Partial Knowledge RoutingPartial Knowledge Routing Cost function: Cost function: ωω (e, t) (e, t)
If message arrives at node ‘u’ at time ‘t’, and if edge ‘e’ If message arrives at node ‘u’ at time ‘t’, and if edge ‘e’ (between ‘u’ & ‘v’) is chosen, message will reach node ‘v’ (between ‘u’ & ‘v’) is chosen, message will reach node ‘v’ at time ‘t + at time ‘t + ωω (e, t)’ (e, t)’
FIFO: if tFIFO: if t11 < t < t22, t, t11 + + ωω (e, t (e, t11) <= t) <= t22 + + ωω (e, t (e, t22)) Algorithm with Time-invariant CostsAlgorithm with Time-invariant Costs
Use modified Dijkstra’s algorithmUse modified Dijkstra’s algorithm If L[v] > (L[u] + If L[v] > (L[u] + ωω (e, L[u] + T)) then (e, L[u] + T)) then
L[v] L[v] (L[u] + (L[u] + ωω (e, L[u] + T)) (T (e, L[u] + T)) (T start time) start time)
Minimum Expected Delay (MED)Minimum Expected Delay (MED) Oracles: Contacts SummaryOracles: Contacts Summary Edge cost = avg. wait time + prop delay + trx delayEdge cost = avg. wait time + prop delay + trx delay Proactive routing is used for time-invariant costProactive routing is used for time-invariant cost Fixed routes for all messagesFixed routes for all messages Minimizes avg. waiting time but doesn’t optimize pathMinimizes avg. waiting time but doesn’t optimize path ImprovementImprovement
Dynamically make use of superior contacts per-hopDynamically make use of superior contacts per-hop Multiple disjoint paths to balance loadMultiple disjoint paths to balance load
Partial Knowledge RoutingPartial Knowledge Routing Cost function: Cost function: ωω’ (e, t, m, s)’ (e, t, m, s)
Edge ‘e’, time ‘t’, message size ‘m’, node assigning cost ‘s’Edge ‘e’, time ‘t’, message size ‘m’, node assigning cost ‘s’ ωω’ (e, t, m, s) = t’ (e, t, m, s) – t + d (e, t’)’ (e, t, m, s) = t’ (e, t, m, s) – t + d (e, t’)where,where,
c (e, t) c (e, t) capacity of edge ‘e’ at time ‘t’ capacity of edge ‘e’ at time ‘t’ Q (e, t, s) Q (e, t, s) queue size at source of edge ‘e’, at time ‘t’ as queue size at source of edge ‘e’, at time ‘t’ as
predicted by node ‘s’predicted by node ‘s’ t’ t’ earliest time queues data at ‘e’ and message can be earliest time queues data at ‘e’ and message can be
injected into the edgeinjected into the edge Integral Integral volume of data through ‘e’ in interval [t, t’’] volume of data through ‘e’ in interval [t, t’’] d (e, t’) d (e, t’) propagation delay seen by message propagation delay seen by message
Partial Knowledge RoutingPartial Knowledge Routing
Algorithms with Time-varying CostsAlgorithms with Time-varying Costs Earliest Delivery (ED)Earliest Delivery (ED)
Contacts Oracle Contacts Oracle Q (e, t, s) = 0 Q (e, t, s) = 0 Source routedSource routed Buffer overflow Buffer overflow cascaded delay cascaded delay
Earliest Delivery with Local Queuing (EDLQ)Earliest Delivery with Local Queuing (EDLQ) Contacts OracleContacts Oracle Q (e, t, s) = data queued for ‘e’ at ‘t’ if e = (s , *)Q (e, t, s) = data queued for ‘e’ at ‘t’ if e = (s , *)
= 0 otherwise= 0 otherwise Per-hop routed Per-hop routed path vector to avoid loops path vector to avoid loops
Earliest Delivery with All Queues (EDAQ)Earliest Delivery with All Queues (EDAQ) Contacts + Queuing OraclesContacts + Queuing Oracles Q (e, t, s) = data queued for ‘e’ at ‘t’ at node sQ (e, t, s) = data queued for ‘e’ at ‘t’ at node s Source routedSource routed Reservation of edge capacity along computed pathReservation of edge capacity along computed path
Complete Knowledge RoutingComplete Knowledge Routing
Complete Knowledge RoutingComplete Knowledge Routing
Complete Knowledge RoutingComplete Knowledge Routing
Simulation Scene 1: Simulation Scene 1: Routing to Remote VillageRouting to Remote Village Village: Kwazulu-NatalVillage: Kwazulu-Natal City: CapetownCity: Capetown Country: South AfricaCountry: South Africa Dialup Dialup 4 Kb/s, (11pm to 6am contact) 4 Kb/s, (11pm to 6am contact) 3 PACSAT satellites3 PACSAT satellites
OSCAR-11, PACSAT, PCSATOSCAR-11, PACSAT, PCSAT Each making 4 trips over city & village Each making 4 trips over city & village
(connecting them)(connecting them) 3 motorbikes 3 motorbikes 1Mbps (5 min contacts) 1Mbps (5 min contacts)
Each making several trips in dayEach making several trips in day MessagesMessages
From Village From Village 1 KB, From City 1 KB, From City 10 KB 10 KB TimeTime
48 Hrs, starting 01/11/2004 11:59pm48 Hrs, starting 01/11/2004 11:59pm
Simulation Scene 1: Simulation Scene 1: Routing to Remote VillageRouting to Remote Village
Simulation Scene 2: Simulation Scene 2: Network of City BusesNetwork of City Buses 20 city buses in San Francisco20 city buses in San Francisco 3 scheduled bus trips (A, B, C)3 scheduled bus trips (A, B, C) ““stop” generated at each turnstop” generated at each turn Constant speed between stopsConstant speed between stops Pause of 0 to 5 min at each stopPause of 0 to 5 min at each stop Speed between 10 and 20 m/sSpeed between 10 and 20 m/s Radio range 100 m by defaultRadio range 100 m by default Traffic: 12 hrs, 1 hr intervals, 20 random src / dst Traffic: 12 hrs, 1 hr intervals, 20 random src / dst
bus pairs, 200 messages per intervalbus pairs, 200 messages per interval
Simulation Scene 2: Simulation Scene 2: Network of City BusesNetwork of City Buses
OverviewOverview
Challenged Internets (CI)Challenged Internets (CI) Shortfalls of TCP/IP in CIShortfalls of TCP/IP in CI Concept of DTNConcept of DTN
Message-oriented Message-oriented Overlay architectureOverlay architecture Classes of Service (CoS)Classes of Service (CoS)
Routing IssuesRouting Issues ConclusionConclusion
Delay Tolerant Network Delay Tolerant Network Overlay ArchitectureOverlay Architecture Need to explore challenged InternetsNeed to explore challenged Internets Interoperability amongst different Interoperability amongst different
regions of networksregions of networks Current Internet Protocol stack not Current Internet Protocol stack not
suitablesuitable Delay Tolerant Network is an optionDelay Tolerant Network is an option Overlaid architecture above Overlaid architecture above
Transport protocolsTransport protocols Routing protocols for DTN need to Routing protocols for DTN need to
cope with “Data over Time”cope with “Data over Time”
ReferencesReferences
Kevin Fall, Kevin Fall, A Delay-Tolerant Network A Delay-Tolerant Network Architecture for Challenged Internets, Architecture for Challenged Internets, Intel Intel Research Berkeley, Technical Report, IRB-Research Berkeley, Technical Report, IRB-TR-03-003TR-03-003
S. Jain, K. Fall, R. Patra, S. Jain, K. Fall, R. Patra, Routing in a Routing in a Delay Tolerant Network, Delay Tolerant Network, ACM ACM SIGCOMM’04, Sep, 2004SIGCOMM’04, Sep, 2004
Forrest Warthman, (Warthman Forrest Warthman, (Warthman Associates), Associates), Delay-Tolerant Networks Delay-Tolerant Networks (DTNs), (DTNs), A Tutorial, Version 1.1, 3/5/03A Tutorial, Version 1.1, 3/5/03