Internet Market Failures:Technological Causes and SolutionsBob BriscoeChief Researcher, BT
Jan 2012
This work was partly funded by Trilogy, a research project supported by the European Community www.trilogy-project.org
© British Telecommunications plc
Traffic Growth• to meet conservative traffic growth forecasts with conservative cost improvements
• traffic growth: fixed 35% pa, mobile 74% pa• cost efficiency improvements: 15% pa
• EU fixed & mobile operators would need €30B CapEx 2011–2014* • in addition to €73B CapEx expected using 2006–2010 trend [ATKearney]
• expected revenues only maintain the CapEx trend, not the extra• most traffic growth only commodity (over-the-top) services
*of course, the problem continues to get worse beyond 2014
alternative outcomes?• return on capital employed falls for all
operators (12% 9%)• capital leaves telecoms sector
• traffic management somehow identifies 50% least valuable videos• vendors admit deep packet
inspection cannot do this
• surely the market will sort this out...?
Cumulative €9.8Bn
Estimated Capex required to fund incremental capacityfor European fixed Internet networks
© British Telecommunications plc
Surely the market will sort this out?
• won’t the price per byte rise to suppress demand? • no, faith in the market is misplaced for the Internet
• the problem is buried under two levels of confusion1. the wrong metric
2. the right metric is only selectively visible
© British Telecommunications plc
#1. the wrong metric
• volume in bytes doesn’t represent cost of usage• raising the price per byte will suppress bytes, not cost
the right metric• marginal* cost of usage = contribution to congestion
• bytes are only half the story• need bytes weighted by instantaneous congestion level
• represents contribution to the cost others experience (degraded QoE)• equivalently, cost of capacity upgrade to alleviate the congestion
* plus subscription element to cover fixed costs
© British Telecommunications plc
measuring contribution to congestion
• user’s contribution to congestion= bytes weighted by congestion level= bytes dropped (or marked)= ‘congestion-volume’
• as simple to measure as volume
bit-rate
time
congestion
time
10GB
0.01% congestion
1MB
1% congestion
1MB
300MB100MB
3MB
1%
0.01%
© British Telecommunications plc
#2. right metric is only selectively visible
• in the Internet architecture...
• end hosts detect congestion as gaps in sequence spaces• not necessarily visible by network operator
• sequence numbers in transport layer
• potentially encrypted payload of network layer
• gaps may be filled by packets on other paths
• network nodes know what they drop locally
• but a traffic management node • cannot see what other nodes have dropped,
especially in other networks
• cannot collect up each customer’s contribution to congestion
• network operator cannot control what it cannot see
© British Telecommunications plc
menu
• problems with cost metrics for a shared network
• relation to economic theory of market failure
• technical solution
• multiple types of market failure
• summary
© British Telecommunications plc
8
Sharing: Lest we Forget
• IP, Ethernet, MPLS are all designed to share a network
• sharing is central to all developments in consumer access*• because largest element of costs is geographical dispersion
• for dedicated consumer access, utilisation as speed • average utilisation only 0.5% for a 40M dedicated access• was 1.25% for 4M access; still higher in dial-up days
• cost efficiency is driving sharing closer to the end-user• that’s why
ADSL VDSL, passive optical networks, cable, WiFi, cellular
Inte
rne
t to
po
log
y vi
sua
liza
tion
pro
du
ced
by
Wa
lrus
(C
ourt
esy
of
Yo
ung
Hyu
n,
CA
IDA
)
* and obviously core, campus & enterprise networks too
© British Telecommunications plc
what economic theory tells us
• shared Internet bandwidth: ‘use-up-able’ and non-excludable• a common good
• free-riding typically reduces the incentive to supply
• common goods tend to be under-supplied and over-consumed
• common goods suffer from what is called a market failure• when revenue does not rise to fund extra ~€30B of capacity in Europe
• the market will not “sort this out”
prevent non-contributors benefiting?
excludable non-excludable
use by A prevents simultaneous use
by B?
rivalrous(‘use-up-able’)
private goodsfood, clothing, toys, furniture, cars
common goodsfish, hunting game, water
non-rivalrous(irreducible)
club goodssatellite television
public goodsnational defense, free-to-air TV, air, published info
free-riding problems
tragedy of the commons
problems
© British Telecommunications plc
Market failure
“when a free marketdoes not allocate goods and services efficiently”
• non-excludability is one of the causes of market failure
price supply
demand
shared capacity
© British Telecommunications plc
the wrong metricmarket failure as a consequence
• operators suppress bytes, not cost• using volume limits or charges
• punishes cost sensitive protocols, eg:• BitTorrent uTP (micro transport protocol)
IETF LEDBAT (Low Extra Delay Background Transport)*• EQS Equitable Quality Streaming [Crabtree09]
– e.g. EQS can halve capacity needed for video streaming– but ‘fair’ queuing stops it working – ironic as EQS is fairer
• operators cannot limit congestion when necessary
• capacity is under-supplied and over-consumed
price supply
demand
shared capacity
* uTP & LEDBAT are actually designed to only yield to self-congestion, because operators do not reward yielding to others
© British Telecommunications plc
Potential Consequences of Market Failure
• congestion and poor quality• “under-supplied and over-consumed”
• random slaughter• operator can’t target costly usage, so it picks on traffic it likes
least until cost reduced enough• infrastructure no longer an engine for innovation
• vertical integration• Internet service subsidised by value-added services• However, can never compete with over-the-top services while
also subsidising their bandwidth• resort to government subsidy and control
• or someone averts the market failure...
price supply
demand
shared capacity
© British Telecommunications plc
(Just) Change the Internet Protocol
• the challenge
• shift IP to the private goods model
• keep traditional freedom to share everything
excludable non-excludable
rivalrous(‘use-up-able’)
private goods common goods IP
non-rivalrous(irreducible)
club goods public goods
© British Telecommunications plc
14
IETF Congestion Exposure (ConEx)basic signals and functional units
transportsender
transportreceiver
congestednetwork element
DATA
ACKS
congestion feedback
ConEx markingsin the IP layerIETF working-group in progress
congestionloss/marks
new with ConEx
existing
© British Telecommunications plc
15
Congestion Exposure (ConEx)functions built on basic signals
transportsender
transportreceiver
congestednetwork element
policy auditDATA
ACKS
congestion feedback
ConEx markingsin the IP layer
congestionloss/marks
new with ConEx
existing
© British Telecommunications plc
policy • only throttles traffic when your contribution to congestion anywhere in the Internet exceeds your allowance
• incentive to avoid congestion
• only throttles traffic when your contribution to congestion anywhere in the Internet exceeds your allowance
• incentive to avoid congestion
ConEx-based policy function exampleflat-fee congestion-policing
bulkcongestion
policer
Internet
0.3%congestion
0%
0.1%
2 Mb/s0.3Mb/s6 Mb/s
Acceptable Use Policy
contribution to congestion allowance: 1GB/month
@ €15/month
Allows ~70GB per day of data in typical conditions
ConEx makes many policy actions possible • downgrade class of service• re-route• report to management system, e.g. for charging• or just ignore
© British Telecommunications plc
Market Failure – not one, but many
1. non-excludability2. information asymmetry
A. consumer-providerB. provider-provider
3. externalities A. (-) congestionB. (+) network effects
4. transaction costs5. market power6. switching costs
the Internet suffers from them all…
© British Telecommunications plc
information asymmetry
• Classic example: info about quality of used cars [Akerlof71]
• seller knows which cars are ‘lemons’ (poor quality)• buyer doesn’t, but knows chance of getting a lemon• buyer suppresses average willingness to pay in case • demand falls, supply falls, market failure• Nobel Prize in Economics, 2001
a) between network consumer & provider• TCP/IP exhibits a weird reversal of the same problem• cost/quality more visible to end-host than network• same outcome: market failure [Briscoe08]
b) between networks• sending net takes out sub-contract with receiving net to deliver packets• but quality of delivery is invisible to the principle contractor [Constan01]
© British Telecommunications plc
information symmetry solutiona) between consumer & provider
• light usage can go much faster• hardly affects completion time of
heavy usage
• if operator can see congestionit can reward those who avoid it• equitable quality streaming (EQS) and
protocols like uTP, LEDBAT & BITSbit-rate
time
bit-rate
time
1. TCP
4. DPI + FQ
ConEx +weightedsharing
bit-rate
time
2. FQ fair queuing
bit-rate
time
3. volume cap
bit-rate
time
p2p
© British Telecommunications plc
information asymmetry problemsb) between networks
• NetFlix vs Comcast
• gov.fr vs google.com
© British Telecommunications plc
information symmetry solutionb) between networks
transportsender
transportreceiver
congestednetworkelement
policycontrolbased onmetering
blackminus
red
congestionmarking
ConExmarking
© British Telecommunications plc
1. non-excludability
2. information asymmetryA. consumer-providerB. provider-provider
3. externalities A. (-) congestionB. (+) network effects
4. transaction costs
5. market power
6. switching costs
Market Failure – not one, but many
© British Telecommunications plc
market power
Net Neutrality
• a multi-faceted issue• primarily an issue with the lower layer (access network) market• that has surfaced at the Internet layer in the US• (see spare slide)
focusing on Internet Layer issues…• there are concerns around who controls quality of service
• ConEx has beeen used in the US debate • to show that QoS per se must not be outlawed• because it can be user-controlled
* so far
© British Telecommunications plc
1. non-excludability
2. information asymmetryA. consumer-providerB. provider-provider
3. externalities A. (-) congestionB. (+) network effects
4. transaction costs
5. market power
6. switching costs
Market Failure – not one, but many
multipath TCP (MPTCP)…another subject
© British Telecommunications plc
Take-away Messages
• 7 types of market failure• Internet technology suffers from them all
• causes and solutions are technological – at the Internet layer
• On-track to add ConEx (and MPTCP) to IP• addressed 6 / 7 types of market failure• without losing feel of classic Internet• flat fee & freedom to share, anything anywhere
• Simple – imagine an industry that just works by simple economics• without all the strife and government intervention we are so used to• without all the complex capacity allocation & QoS mechanisms
© British Telecommunications plc
more info
• Bob Briscoe, “Practical Microeconomics and Internet Resource Sharing Protocols”Trilogy Summer School lecture: Videos available from www.trilogy-project.org
• IETF ConEx: tools.ietf.org/wg/conex/
• Resources related to ConeX and re-feedback: bobbriscoe.net/projects/refb/
• [ATKearney] "A Viable Future Model for the Internet," Technical Report, A.T.Kearney (2010)
• [Akerlof71] Akerlof, G., "The Market for `Lemons': Quality, Uncertainty and Market Mechanisms," Quarterly Journal of Economics 84:488--500 (August 1970)
• [Constan01] Constantiou, I.D. & Courcoubetis, C.A., "Information Asymmetry Models in the Internet Connectivity Market," In: Proc. 4th Internet Economics Workshop (May 2001)
• [Briscoe08] Briscoe, B., Moncaster, T. & Burness, L., "Problem Statement: Transport Protocols Don't Have To Do Fairness," Internet Engineering Task Force Internet Draft draft-briscoe-tsvwg-relax-fairness-01 (July 2008) (Work in progress)
• [Crabtree09] B. Crabtree, M. Nilsson, P. Mulroy and S. Appleby “Equitable quality video streaming” Computer Communications and Networking Conference, Las Vegas, (January 2009)
Internet Market Failures:Technological Causes and Solutions
Q&A
© British Telecommunications plc
28
Congestion Exposure (ConEx)basic signals and functional units
transportsender
transportreceiver
congestednetwork element
ECN
loss
SACK
ECE
Re-Echo-ECN
Re-Echo-Loss
DATA
ACKS
congestion feedback
ConEx markingsin the IP layer
congestionloss/marks
© British Telecommunications plc
market power
Net Neutrality
• Well-known that access networks are a natural monopoly
• US regulatory failure to remedy at lowest layer• resulted in issue resurfacing at the IP layer• monopoly/duopoly access provider favouring its own services
• EU: government interventions at lower layer • has averted need to intervene and distort the Internet layer*
• net neutrality is a solution at the Internet layer for a problem at the lower layer (access link)
* so far
© British Telecommunications plc
example integrated end-to-end business model
• focused on revealing variable costs
• capacity fees can still reflect value flows
• kick-starts new e2e value flows(another talk)• otherwise advertising
is limit to growth
• note: an example• you choose how to use a cost metric• not me
NA NBND
R1S1
£$¥
€
£ $
downstreamcongestion
usagecharges
flat charges
€
¥
¥