1
Correlating Internet Performance & Route Changes to Assist in Trouble-
shooting from an End-user PerspectiveLes Cottrell, Connie Logg, Jiri Navratil SLAC
Passive and Active Monitoring WorkshopAntibes, Juan-les-Pins, France
April 19-20, 2004 www.slac.stanford.edu/grp/scs/net/talk03/pam04.ppt
Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also
supported by IUPAP
2
OutlineSet of integrated measurement tools to aid in
troubleshooting for end “user”
• Traceroute measurements/analysis
• Topology visualization
• Lightweight bandwidth estimation
• Overall visualization
• Level change anomaly automated detection
• Correlation of performance & route changes
3
Traceroute measurement
• Every 10 minutes for each host – Run standard traceroute 2 sec timeout, 1
query/hop, <= 30hops• For some hosts use ICMP traceroute
– End host responds (7/40)– Intermediate host responds (1/40)
• Two cases UDP probes better than ICMP• One case neither ICMP or UDP probes help
– Both forward & reverse (use ssh for reverse route)
• Need ssh access to remote host for rev trace– Else no reverse route (not a disaster)
4
Significant changes• Compare current and previous traceroutes:
– If traceroute reports “unknown host” => unknown (!) – Else for each hop/node
• If both current & previous hops have valid IP addresses (i.e. router does not respond & traceroute reports “*”)
– If different i.e. some kind of Route Change has occurred
» If IPs same for 1st 3 octets then => same subnet/colo (:)
» Else if IPs in same AS then => same AS (a)» Else significant change => assign unique route number
If only one hop different => color route # orange ( ) Else color route => color route # red ( )
– Elseif 30 hops => no route change but last hop unreachable (|)» If last hop not pingable => color red (|)
– Else => no route change (●)• Elseif one or both IPs are “*” => route change unclear (*)
• If “Icmp checksum is wrong” color character orange• If significant bandwidth change color cell
5
Route table• Compact so can see many routes at once
History navigation
Multiple route changes (due to GEANT), later restored to original route
Available bandwidth
Raw traceroute logs for debugging
Textual summary of traceroutes for email to ISPDescription of route numbers with date last seen
User readable (web table) routes for this host for this day
Route # at start of day, gives idea of root stability
Mouseover for hops & RTT
6
Another example
TCP probe type
Host not pingable
Intermediate router does not
respondICMP checksum
error
Level change
Get AS information for routes
7
Topology• Choose times and hosts and submit request
DLCLRC
CLRC
IN2P3
CESnet
ESnet
JAnetGE
AN
TNodes colored by ISPMouseover shows node namesClick on node to see subroutesClick on end node to see its path backAlso can get raw traceroutes with AS’
Alternate rt
SLAC
Alternate routeHour of day
8
Available bandwidth• Uses ABwE/Abing (packet pair dispersion)
– Needs server at remote end or ssh to launch server– Fast (< 1 sec)– Lightweight
• < 40 packets for both forward & reverse estimates (5800 Bytes)
– Uses min delay for capacity– Inter packet dispersion for cross-traffic– Available BW = Capacity (min RTT) – Cross-traffic (var)
• Good agreement with other methods• Even if poor absolute agreement (25% cases) can spot changes
– Also provides RTT
• Make measurements to about 60 hosts at 5 minute intervals (deployed in IEPM, MonALISA, PlanetLab)
9
Available Bandwidth• From SLAC to Caltech Mar 19, 2004
Dynamic bandwidth capacity (DBC)
Available bandwidth =DBC – X-traffic
Cross-traffic
Iperf
10
Achievable throughput & file transfer
• IEPM-BW– High impact (iperf, bbftp, GridFTP …) measurements 90+-15 min intervals
Select focal area
Fwd route change
Rev route change
Min RTT
Iperf
bbftpiperf1
abing
Min RTT
11
Put it all together• Two examples
– Agreement of iperf & abing– Route changes and available bandwidth
12
AbWE
Iperf
28 days bandwidth history. During this time we can see several different situations caused by
different routing from SLAC to CALTECH
Drop to 100 Mbits/s by Routing (BGP) errors
Drop to 622 Mbits/s path
back to new CENIC path
New CENIC path 1000 Mbits/s
Reverse Routing changes
Forward Routing changes
Scatter plot graphs of Iperf versus ABw on different paths (range 20–800 Mbits/s) showing agreement of two methods
(28 days history)
RTT
BbftpIperf 1 stream
13
Changes in network topology (BGP) can result in dramatic changes in performance
Snapshot of traceroute summary table
Samples of traceroute trees generated from the table
ABwE measurement one/minute for 24 hours Thurs Oct 9 9:00am to Fri Oct 10 9:01am
Drop in performance(From original path: SLAC-CENIC-Caltech to SLAC-Esnet-LosNettos (100Mbps) -Caltech )
Back to original path
Changes detected by IEPM-Iperf and AbWE
Esnet-LosNettos segment in the path(100 Mbits/s)
Hour
Rem
ote
host
Dynamic BW capacity (DBC)
Cross-traffic (XT)
Available BW = (DBC-XT)
Mbit
s/s
Notes:1. Caltech misrouted via Los-Nettos 100Mbps commercial net 14:00-17:002. ESnet/GEANT working on routes from 2:00 to 14:003. A previous occurrence went un-noticed for 2 months4. Next step is to auto detect and notify
Los-Nettos (100Mbps)
14
Automatic Step change Detection
• Too many graphs to review each morning!• Motivated by drop in bandwidth between SLAC &Caltech
– Started late August 2003– Reduced achievable throughput by factor of 5– Not noticed until October 2003– Caused by faulty routing over commercial network – After notifying ISP, it was fixed in 4 hours!– See http://www.slac.stanford.edu/grp/scs/net/case/caltech/ for details
SLAC Caltech achievable throughput April – November 2003 Started
15
Automatic available bandwidth step change detection
• Still developing, evolving from earlier work:– Arithmetic weighted moving averages– NLANR work, see http://
byerley.cs.waikato.ac.nz/~tonym/papers/event.pdf
• Roughly speaking:– Has a history buffer to describe past behavior
• History buffer duration currently 600 mins
– Plus a trigger buffer of data suggesting a change• Trigger buffer duration (evaluating typically 10-60 mins) indicates
how long the change has to occur for
– History mean () and std. dev. () use by trigger selector• If new_value outside +- sensitivity add to trigger buffer• If new_value outside +- 2*sensitivity then also an outlier (don’t
add to stats)• Else goes in history buffer
16
Algorithm
• If this is a trigger value compare with and save direction of change
• If this is a trigger and the direction has changed, reset trigger buffer– Move trigger data to history buffer, recalculate stats,
clear trigger buffer
• If trigger buffer full calculate trigger mean t and t
– If ( - t)/ threshold then a & reset trigger buffer
– Else remove oldest value from trigger buffer
17
ExamplesSLAC to Caltech available bandwidth April 6-8, 2004Alerts
History duration: 600 mins, trigger duration: 30 mins, threshold: 40%, sensitivity: 2With trigger duration: 60 only see one alert, with trigger duration: 10 catch alerts
Rou
te c
hang
e
SLAC to NIKHEF (Amsterdam)Mbit/sAvailBW
Route changesSLAC - NIKHEF
Unreachable
18
BW vs Route changes• Route & throughput changes from 11/28/03 thru 2/2/04
– Most (80%) route changes do not result in throughput change– About half throughput changes are due to route changes
Location (# nodes)
# route chgs
# with thru inc.
# with thru decr.
# thru chgs
# thru with rte
# thru chg w/o rte
Europe (8) 370 2 4 10 6 4
Canada & US (21)
1206 24 25 71 49 221
Japan (13) 142 2 2 9 4 5
19
More Information• ABwE:
– http://moat.nlanr.net/PAM2003/PAM2003papers/3781.pdf
• IEPM– http://www-iepm.slac.stanford.edu/– http://moat.nlanr.net/PAM2003/PAM2003papers/3768.pdf
• Traceroute examples:– www.slac.stanford.edu/comp/net/iepmlite/tracesummaries/to
day.html
• Step change analysis– http://byerley.cs.waikato.ac.nz/~tonym/papers/event.pdf