Benchmarking Broadband Internet with Bismark
Srikanth Sundaresan, Nick FeamsterGeorgia Tech, Atlanta, USA
Walter de Donato, Antonio PescapéUniversity of Napoli Federico II, Napoli, Italy
Renata TeixeiraUniversity Pierre et Marie Curie, Paris, France
Broadband Connectivity is exploding
● 200 million Americans have broadband
● National Broadband plan aims to increase that
● But, what is current experience with broadband?
Talk Outline
● Questions about broadband performance● Bismark: our measurement platform● Some initial results● Future work
Are users really getting what they pay for?
Study in France confirms this view
● Over 20,000 users across major ISPs, geographical regions
● Latency and throughput study from end hosts
6
Significant performance variation
Cum
ulative
fraction o
f use
rs
Fewer than half of the users achieve 80% of
advertised SLA. Why?
S. Sundaresan, L. Di Cioccio, N. Feamster, R. Teixeira. “Which Factors Affect Home Network Performance?”
● ISP, service plan, geographical location have effect.
● Gap between plan rate and achieved rates
How do we study access link performance?
● Study from outside– Dischinger et al (IMC 2008), Netalyzr (IMC
2010)
– Not continuous, not many per user, no view into home
● Study from inside– Grenouille project
– Hard to account for device diversity
– Hard to account for home network
We don't really understand home networks
● How much of user performance is affected by factors outside ISP's control?
● Home networks have a big impact– Network setup – multiple diverse devices
– WiFi effect
– Competing traffic
– User behavior
Talk Outline
● Questions about broadband performance
● Bismark: our measurement platform
● Some initial results● Future work
Bismark
● Deploy programmable gateways in homes● Noxbox deployment: about 35 around Atlanta● SamKnows deployment: about 11000 around
the US
NoxBox Netgear
The design
Internet
MeasurementServer
MeasurementServer
ManagementServerDB
Architecture Overview
Router
Router
Vantage point between the home and the ISP
● Observes all traffic passing through network● Isolate individual factors affecting network
performance– Effect of home network
– Wireless
– Access link
Measurements - Active
● Latency, Jitter and Packet Loss– Last mile and end-to-end latency (ping)
– Upstream and downstream jitter, packet loss (DITG)
– Important for gaming, VoIP
● Throughput– Upstream and downstream (curl, shaperprobe)
– Capacity, available bandwidth of access link
● DNS lookup time and faults– Affects user performance
Measurements – Passive
● ARP and DHCP logs– Nature of home network
● Aggregate flow statistics– Incoming and outgoing (TIE)
– Understand usage patterns, available bandwidth
● Wireless stations– Access point configurations (airodump-ng)
– Understand home wireless networks
Current Deployment
● 10 boxes deployed● 7 in ATT, 2 in Comcast, 1 ClearWire● Most of the deployments within Atlanta● All measurements done to server in GaTech
Talk Outline
● Questions about broadband performance● Bismark: our measurement platform
● Some initial results● Future work
Overview of results
● Initial look at latency from gateway to:– First IP hop in ISP network (last mile)
– Server in GaTech
– High variation across homes, ISPs
● Initial throughput analysis– Single threaded HTTP throughput using curl
– Path capacity using shaperprobe
Latencies vary within same ISPR
TT
(ms )
RT
T(m
s )
Baselines Different for 2 ATT customers
Access link technology has impact
● High variation in WiMax and Cable● ADSL latencies are more tightly bound
RT
T(m
s )
RT
T(m
s )
Comcast Clear
Last mile latencies dominate● All but 2ms comes from last mile● ADSL last mile – 8 to 25ms, WiMAX – ~ 75ms!
Very high correlation (0.95) to end-to-end latency
End-to-end latency
Last mile latency
Plan rate
Kbps
CD
F
Single ATT customer
Single threaded TCP downloads consistently see about 80% of promised capacity
Dowstream throughput doesn't quite reach maximum limit
Why not?
● Single threaded throughput error?● Multiple parallel TCP sessions see higher
throughput (> 95% for ATT)– However, highly disruptive to user
● Capacity measurements also close to limit● We are interested in realistic usage scenarios
– What does a normal user with normal usage patterns see?
Throughput Consistently greater for ATT
Plan rate
Single ATT customerKbps
CD
F
Upstream throughput exceeds advertised limits
Key findings
● Latency is important– Baseline varies
– Highly dependent on access link technology
– Last mile latency dominates, highly correlated to end-to-end latency
● Downloads are consistently lower than promised rate
● Uploads are consistently higher
Talk Outline
● Questions about broadband performance● Bismark: our measurement platform● Some initial results
● Future work
Future Deployment and Measurements
● Larger deployment for critical amount of data● Variety in ISP, service plans, geographical
locations– Working with SamKnows
● User study to understand– What users expect from their Internet
– How home networks are set up
● How can we improve user experience?
More questions about performance
● Does application affect performance?● How do different factors affect performance?
– Latency, ISP, service plan
● Better statistical tools to analyze current data● What would you like to know/measure?
Outsourcing Home Network Configuration and Management
● What prevents untrusted tools from:– Sniffing my traffic?
– Consuming all my bandwidth?
– Using my printer?
● Solution: Hold tools accountable to a management policy
● PlanetLab-like testbed for home networks?