End-to-End Issues

Route Diversity

Load balancingo Per packet splittingo Per flow splitting

Spill over Route change

o Failureo policy

Route flapping

2

Diversity Effects

Packet reordero TCP performanceo Larger playback buffer

Jitter

However, jitter can occur due cross traffic.What contribute more to jitter?

3

What contribute more to jitter?

Understand the origins of e2e delay variationso Result from existence of multiple routes• designed load balancing or transient failures

o Result of variations within each route• intra-route issues (congestion)

4

SS DD

SS DD

Related Work [Wang et al., Pucha et al.] studied the impact that

specific routing events have on the overall delayo Routing changes result in significant RTT delay increaseo However, variability is small

[Augustin et al.] examined the delay between different parallel routes in short time epocho Only 12% have a delay difference which is larger than 1ms

[Pathak et al.] studied the delay asymmetry o There is a strong correlation between one-way delay

changes and route changes

5

Key Differences

We study the RTT delay along longer time periods

Examine the difference of the delay distribution between parallel routes

Focus on the origin of delay variabilityoWithin each route (e.g. congestion)o Due to multiple routes (e.g. load-balance)

6

How do we measure?

• Use DIMES for conducting two experiments– 2006 and 2009– Over 100 agents measures to each other– Broad set of ASes and geo locations– Active traceroute (ICMP and UDP)– Each agent probes all target IPs every 1-2 hours– 4 days of probing– Collect the route IPs and e2e delay

7

Vantage Point Statistics (1)

2006o 102 VPsoMillion tracerouteso 6861 e2e pairso VPs in North

America (70), Western Europe (14), Australia (10), Russia (6), Israel (2)

2009o 105 VPsoMillion tracerouteso 10950 e2e pairso VPs in Western

Europe (41), North America (38), Russia (14), Australia (4), South America (2), Israel (2), Asia (4)

8

Vantage Point Statistics (2)

2006 o 18% tier-1o 78% tier-2o 3% small companieso 1% educational

2009o 14% tier-1o 58% tier-2

o 28% educational

Only 7 agents participated in both

9

Identifying Routes and Pairs

Using community-based infrastructureo Routes can start and end in non-routable IPso Users can measure from different locations

Only the routable section of each path is consideredo The source (S) is the first routable IPo The destination (D) is the last routable IP

10

SS DD TargetVP

Some Accounting

• The e2e pair Pi=(S,D) contains all the measured paths between S and D

• For a pair Pi , the set Eij contains the measured

paths that follow route j• For a pair Pi , the dominant route Ei

r is the route that was seen the most times– There can be several dominant routes with equal

prevalence– For brevity we assume there is one at index r

11

Delay Stability (1)

Each route Eij has a set of RTT delays,

corresponding to each measured path Each delay is a sample, we consider the

segment length resulting from the 95% confidence interval surrounding the mean value – CI(Ei

j) High variance samples result in long CI

12

Delay Stability (2)

RTT stability of two routes is the intersection (overlap) between their CI’s

13

Key Concept

Overlapping CI’s (left)- intra-route delay variance Non-overlapping (right) - inter-route delay variance

14

Route Stability (1)

Prevalence is the overall appearance ratio of a route j of pair Pi

As a stability measure, we use the prevalence of the dominant route r

15

Route Stability (2)

Use Edit Distance (ED) as a measure for the difference between two routeso Counting insert, delete and substitute operations

Normalize ED by the maximal route length of the two compared routeso Can compare between ED of routes with different

lengthso marks normalized ED for pair i between

routes j and r

16

Route Stability (3)

The stability is the weighted average of ED of all non-dominant routes to the dominant route of nearest length:

17

Things to Note

We measure RTT valueso Capture forward and reverse path delayo Route stability is measured on the forward path• However, 90% of our routes have very similar forward

and reverse paths• Indicating that stability of one-way is a good estimation

Using UDP and ICMP o Capture all possible routes, not flowso Upper bound for instability

18

Results – Route Statistics

• Both have roughly the same route length and median delay• Shorter routes than Paxson’s (15-16hops)• Median delay is around 100msec

19

Results – Route Stability

• Overall stable e2e routing (25% of pairs with single route)• Academy pairs have higher stability (55% single route)• USA pairs have slightly higher stability (35% single route)• RouteISM < 0.2 for over 90% of the pairs

Load balancersNot visible in academy pairs

20

Results – Origin of Delay Instability

• 70% of the cases, changes in route delay is within routes (high overlap) and not due to multiple path routing

• In 15% of the cases (20% in the 2006 data sets) the change in delay is mainly due to route changes

21

Results – Route and Delay Instability

• When the difference between routes is high, higher chances of different delay distribution

• Prevalence does not significantly indicate level of overlap!22

High percentage of zero overlap

Results – Additional Findings

Over 95% of the academic pairs have an overlap of over 0.7o Academic networks have little usage of load-

balancing and “spill-over” backup routes Only 5% of the USA pairs have overlap of 0

23

Conclusions

Delay variations are mostly within the routeso 70% of the pairso 95% of the academic pairs

Internet e2e routes are mostly stable The academic Internet is significantly more

stable than commercial networks

24