1
How is the Internet Performing?
Les Cottrell – SLACEcole SIG at nouvelles Technologies en Democratic Republic Congo, 12-
17 Septembre, Organisee par l’Universite de Kinshasa
www.slac.stanford.edu/grp/scs/net/talk11/perform.ppt
2
Overview• Internet characteristics
– Users, capacities, satellites, packet sizes, protocols, routing, flows
• How is it used apps etc.• How the Internet worldwide is performing as
seen by various measurements and metrics • Application requirements
USERS
3
Internet Usage growth ‘95-’10
495 00 05 09
Millions of Users
Penetration %
0 200 400 600
AsiaEuropeN AmerL AmerAfricaM EastAustrlasia
N AmericaAustraliaEuropeL AmericaM EastAsiaAfricaWorld
0 20% 50% 80%
Millions of users
Year
1500
1000
500
2.09B Mar 2011
Example: China• China not connected to the Internet until May 1994
– 1st permanent IHEP/Beijing used satellite via SLAC– www.computerworld.com.au/article/128099/china_cel
ebrates_10_years_being_connected_internet
5
Where are they
6
• Internet city connections
Internet Users 2002
2.8% growth/year~¼ world pop uses InternetDeveloped world saturatingDeveloping catching up73% penetration US43% users from Asia
Capacities
7
What have they got?
8Capacity
From TelegeographyCapacity
Who is still on Satellite
9
Terrestrial
GEOS
Min
RTT
(ms)
GEOS (Geostationary Earth Orbit Satellite)
good coverage, but expensive in $/Mbps
broadband costs 50 times that in US, >800% of monthly salary c.f. 20% in US
AND long delays min RTT > 450ms, usually much larger due to congestion
Easy to spotClear signature
Packet sizes & types
10
11
Packet size• primarily 3 sizes: WHY?
Packet size (bytes)Cu,
mul
a tiv
e pr
o ba b
i lity
%
Packets
Bytes
Mean ~ 420Bytes, median ~ 80Bytes
Measured Feb 2000 at Ames Internet eXchange
~ 84M packets, < 0.05% fragmented
close to minimum=telnet and ACKs, 1500 (max Ethernet payload, e.g. FTP, HTTP); ~ 560Bytes for TCP implementations not using max transmission unit discovery
12
Internet protocol use• There are 3 main
protocols in use on the Internet:– UDP (connectionless
datagrams, best effort delivery),
– TCP (Connection oriented, “guaranteed” delivery in order)
– ICMP (Control Message protocol)
Time Feb-May 2001
Flow
s/1 0
min
InO
ut
TCP dominates today
SLAC protocol flowsTCP
UDP
ICMP
Routing
13
14
Hops• Hop counts seen from 4 Skitter sites (Japan, S. Cal,
N. Cal, E. Canada, i.e. 10-15 hops on average
Hop Count
Weak RTT dependenceon hop count
95%
50%5%
RTT
Hops
15
Richness of connectivity• Angle = longitude of AS HQ in whois records• Radius=1-log(outdegree(AS)+1)/(maxoutdegree + 1)
– Outdegree = number of next Hops As’ accepting traffic• Deeper blue & red more connections
• All except 1 of top 15 AS’ are in US, exception in Canada
• Few links between ISPs in Europe and Asia
Today’s routing less via USwww.nytimes.com/2008/08/30/business/30pipes.html
16
• Invented in US• 1st 30 yrs most traffic
thru US• 70%=>20% in 10yrs• No central control• Patriot act=>store info
outside US• China, India, Japan
making larger investments
• More level playing field• Harder for CIA!
17
Routes are not symmetric• Min, 50% & 90%
RTT measured by Surveyor
• Notice big differences in RTTs
• May be due to different paths in the 2 directions or to different loading
Advanced to U. Chicago
RTT
ms
RTT
ms U. Chicago to Advanced
Flows
18
19
Flow sizes
Heavy tailed, in ~ out, UDP flows shorter than TCP, packet~bytes75% TCP-in < 5kBytes, 75% TCP-out < 1.5kBytes (<10pkts)UDP 80% < 600Bytes (75% < 3 pkts), ~10 * more TCP than UDPTop UDP = AFS (>55%), Real(~25%), SNMP(~1.4%)Can roughly characterize as power law with slope & intercept
SNMP
RealA/V
AFS fileserver
20
Flow lengths
• 60% of TCP flows less than 1 second• Would expect TCP streams longer lived
– But 60% of UDP flows over 10 seconds, maybe due to heavy use of AFS at SLAC
– Another (CAIDA) study indicates UDP flows are shorter than TCP flows
TCP outbound flows
Active time in secs
Measured by Netflowflows tied off at 30 mins
Applications
21
Web
22
By mid 2012: IE < 50%, Chrome overtakes Firefox
Usage• P2p hit by RIAA law suits• Moving to video, social networking
– Video on demand double/2 years ’08-’13• iPhones (only peripherally a phone)
– Mobile traffic doubles each year
23
Yahoo
GoogleFacebook
YouTube
Growth of Video• P2P traffic, still the largest share of Internet traffic
today, will decrease as a percentage of overall Internet traffic.
24
• Internet video streaming and downloads are beginning to take a larger share of bandwidth, and will grow to nearly 60 percent of all consumer Internet traffic in 2014.
How Internet is used & when
25
Enterprise& tier 1
asert.arbornetworks.com/2009/08/the-internet-after-dark
26
Web use characteristics• Size of web objects varies from site to site, server to
server and by time of day.– Typical medians in 2000 varied from 1500 to 4000 bytes
• Also varies by object type, e.g. medians for– movies few 100KB to MBs, postscript & audio few
100KB, text, html, applets and images few thousand KB
Bytes
• Size of average web page tripled in 5 years 2003-2008
• www.websiteoptimization.com/speed/tweak/average-web-page/
Why increasing• New users (easier for user, more coverage)• New apps: You-Tube, climate modeling …• New tools: manual(hand tuned) Automatic
generation– Web 2: Ajax, Javascript, CSS
• Broadband more elaborate/attractive designs possible
27
• desktop to web apps• e.g. mail, calendars,
photo albums, games...
28Log Plot of ESnet Monthly Accepted Traffic, January 1990 – December 2008
0.0
0.1
1.0
10.0
100.0
1000.0
10000.0
100000.0
Jan,
90
Jan,
91
Jan,
92
Jan,
93
Jan,
94
Jan,
95
Jan,
96
Jan,
97
Jan,
98
Jan,
99
Jan,
00
Jan,
01
Jan,
02
Jan,
03
Jan,
04
Jan,
05
Jan,
06
Jan,
07
Jan,
08
Jan,
09
Jan,
10
observation, 1990-2008.1, 1, 10, 100, 1000Exponential fit and projection 2 years forward
Impact on backbones: e.g. Current and Historical ESnet Traffic PatternsTe
raby
tes /
mon
th Oct 19931 TBy/mo.
Aug 1990100 MBy/mo.
Jul 199810 TBy/mo.
38 months
57 months
40 months
Nov 2001100 TBy/mo.
Apr 20061 PBy/mo.
53 months
ESnet Traffic Increases by10X Every 47 Months, on
Average
July 201010 PBy/mo.
Performance by Metric
29
30
What does performance depend on?• End-to end internet performance seen by
applications depends on:– round trip times– packet loss– jitter– reachability– bottleneck bandwidth– implementation/configurations– application requirements
• Data transmitted in packets
mse
c.
ITU G.114 300 ms RTT limit for voice
31
RTT from SLAC to the WorldRTT ~ distance/(0.6*c) + hops * router delayRouter delay = queuing + clocking in & out + processing
2/3 countries of world Ok for voice, rest mainly in Africa
What is the problem with > 300ms?
32
RTT from California to world
Longitude (degrees)
300ms
300ms
RTT (ms.)
Freq
uenc
y
RTT
(ms)
Source = Palo Alto CA, W. Coast
E. C
oas t
US
W. C
o ast
US
Euro
pe &
S. A
mer
ica
Europe
0.3*0.6c
Bra
zil
E. C
oast
Data from CAIDA Skitter project
WHY these distributions?
33
Longitude
RTT(
ms)
Seen from Japan
RTT from Japan to world
Jitter• Variability of RTT, many ways to measure• “Jitter” = IQR(ipdv); ipdv(i) =RTT(i) – RTT(i-1)• Usually at edges, so ~distance independent• Impacts smooth flows e.g. VoIP, video, real-time• Haptics (surgery) < 1ms; H.323 <40ms with buffer
34Internet Jitter seen from SLAC to World Sep’08
Can improvevoice with de-jitter buffer, e.g. 70msto smooth theflow
But….
Losses• On good lines
usually congestion• Wireless dB loss,
net devices• Usually last mile
35
• Distance independent• Big effect
• Realtime, games, Voice, typing echo
• 1% loss VoIP annoying
Derived Throughput
36 36
Behind Europe5 Yrs: Russia, Latin America, Mid East 6 Yrs: SE Asia9 Yrs: South Asia12 Yrs: Cent. Asia16 Yrs: AfricaCentral Asia, and
Africa are in Danger of Falling
Even Farther behind
In 10 years at the current rate
Africa will be 1000 times worse
than Europe
Derived throughput ~ 8 * 1460 /(RTT * sqrt(loss))Mathis et. al
1993
Where is best Throughput?
37
Voice over IP• Affected by:
– Loss, RTT, Jitter, • Quality measured
by Mean Opinion Score (MOS)
38
– Can convert from RTT, loss & jitter to MOS– MOS values: 1=bad; 2=poor; 3=fair; 4=good; 5=excellent. – Typical reasonable range for Voice over IP (VoIP) is 3.5 to 4.2.– Russia and L.America improved dramatically in 2000-2002 as
moved from GEOS to terrestrial.– US, Europe, E. Asia, Russia and the M East (all above MOS = 3.5)
good. S.E. Asia marginal, S. Asia, C Asia need a lot of patience– Africa beginning to get there.
39
Application requirements• Based on ITU Y1541 & Stanford (Haptics)
• The VoIP loss of 10^-3 used to be 0.25 but that assumed random flat loss– actual loss is often bursty
• Tail drop in routers• Sync loss in circuits, bridge spanning tree reconfiguration,
route changes
Application
Real time
VoIP WAN connectivity
Web free services
Stream video
Haptics (remote surgery)
1 way delay 150ms 150ms 1000ms undefined 400ms 160ms
‘jitter” 50ms 1000ms undefined 17ms 1msLoss 10-3 10-3 10-3 undefined 10-5 0.1
What’s next• Mobile devices• 40G (transAtlantic, US) & 100Gb backbones• On demand dynamic dedicated services (layers 1 &
2)– Reserve a path at some bandwidth for some time– Use QoS to deliver– HEP, Radio Astronomy, climate research
• IPv6
40
Questions & more study
• www.internetworld.stats.com• www-iepm.slac.stanford.edu/pinger• www.slac.stanford.edu/comp/net/wan-mon/
tutorial.html • www.slac.stanford.edu/xorg/icfa/icfa-net-paper-jan0
9/report-jan09.doc
• http://www.cablemap.info/
41
Compare with Development IndicesAbv. Name Organization Countries Date of
Data
GDP Gross Domestic Product per capita CIA 229 2001-2006
HDI Human Development Index UNDP 175 2004
DAI Digital Access Index ITU 180 1995-2003
NRI Network Readiness Index World Economic Forum 120 2007
TAI Technology Achievement Index UNDP 72 1995-2000
DOI Digital Index ITU 180 2004-2005OI Index ITU 139 1996-2003
CPI Corruption Perception Index Transparency Organization 180 2007
42
Choose most: up-to-date, countries, important factorsHDI & DOI
