Royalty Cost Based Royalty Cost Based Optimization for Video Optimization for Video
Compression Compression
Emrah Akyol, Onur G. Guleryuz, and M. Reha CivanlarEmrah Akyol, Onur G. Guleryuz, and M. Reha Civanlar
DoCoMo USA Labs,DoCoMo USA Labs,
Palo Alto, CAPalo Alto, CA
USAUSA
22
OutlineOutline
Setup and motivationSetup and motivation Problem definitionProblem definition Our solution with some interesting simulation Our solution with some interesting simulation
results.results. ConclusionConclusion
33
Setup-1: Diverse set of terminals in Setup-1: Diverse set of terminals in media deliverymedia delivery
Cell phones
PDAs
…
HDTVs
Media
Encoding 1
Encoding 2
Encoding K
…
Required quality/ effective bandwidth
…
Increasing media quality/effective bandwidth
Example terminals decoding the media
content licensing cost 1
content licensing cost 2
content licensing cost K
44
Setup-2: Diverse set of toolsSetup-2: Diverse set of tools
Tool 1
Tool 2Tool 3
Tool 4
…
Tool T
Media data
Decoded Media
Tool j = video motion compensation
Integer-pixel accuracy motion compensation
1/2-pixel accuracy motion compensation
1/4-pixel accuracy motion compensation
1/8-pixel accuracy motion compensation
Increasing quality
Compression tools, error correction tools, transport
tools, …
Tools have different
royalty/licensing costs.
(Media Consumer)
55
Thought ExperimentThought Experiment
• Many media delivery technologies available.
• One can transport media through a variety of networks, using a vast range of tools
that correspond to vast ranges in efficiency in end to end delivery.
• Rather than restricting to rigid toolsets, standard profiles, etc., can one be flexible
and allow all tools to contribute?
• Standardization process mostly allows a coarse set of options. It mostly caters to the
average good. Many good tools and technologies get cut out because they are not
general enough. Selection almost always involves compromises.• Standardization royalties may force simple tools and sophisticated tools equal share
of the revenue. Impetus for alternative avenues for tool deployment.• There is movement in this direction (software decoders, MPEG RVC, …). • MPEG RVC (reconfigurable video coding):
• put many video compression tools (potentially overlapping functionality) in one big library,
• language syntax to specify which tools are needed in decoding a given video stream.
Why?Why?
66
Media Quality
Effective Bandwidth
Royalty Cost
• Royalty cost of delivering media at a particular quality and bandwidth.• Example surface defined by achievable [quality, bandwidth, royalty cost] triplets. • All triplets below the surface are achievable.
Media delivery cost surfaceMedia delivery cost surface
(content licensing + tool licensing)
77
Cuts from the surfaceCuts from the surface
Media Quality
Effective Bandwidth
Q
Media Quality
Effective Bandwidth
C
Royalty Cost
Royalty Cost
Media Quality
Effective Bandwidth
B
Royalty Cost
“MPEG-1”
“h.264/AVC”
88
Complicated Royalty CostsComplicated Royalty Costs
• My content costs $5 for cell phone terminals, $15 for HDTV terminals.
• My compression tool is free of charge.
• My compression tool costs $0.001 per use.
• My tool costs $0.10 per movie, $0.15 for sports, ....
• My tool costs $0.10 except when combined with all free tools, in which case it too
becomes free.
• My tool is free for not-for-profit use.
• …
99
System Level: Media ServerSystem Level: Media Server
Video segment
Flexible Encoder Bitstream
Tool set selection
Encoding parameters
Registry
I will mostly talk about
compression related tools
1010
Media Server Registry
User
Certificate can be used to verify with information from the user site (for each media segment or periodically) to ensure the legitimacy of the media.
Time instant : certificate
Time instant : Encoded media and associated certificate
System Level: Media DeliverySystem Level: Media Delivery : quality : bandwidth : tool list…
Time instant :
Time instant :
1111
Rate - Distortion - Royalty Cost Rate - Distortion - Royalty Cost Optimization SetupOptimization Setup
Problem Definition:For each of the M segments, find the set of tools to use such that distortion is minimized under total rate and total royalty cost constraints, i.e.,
s.t. ,
This is a simplification.
Optimization can get elaborate.
No reason to pay for the latest/greatest tools • if plenty of bandwidth• if simple/easy content• if cheaper tools are available• …
1212
Optimization Example Using Optimization Example Using Compression Tools Compression Tools
Table II: Rate and distortion changes with different tools. The utilized tools are subpixel accurate ME, loop filter, advanced entropy coding, and multiple reference frames respectively. The rate reduction is shown as percentage with respect to the baseline along with PSNR gain at QP=25.
Clip-1 foreman
Clip-2 akiyo
Clip-3 coast.
Clip-4 mobile
Clip-5 bus
Tool-1Rate
56.51% 55.93% 14.93% 51.56% 50.05%
ΔPSNR 0.40 0.48 0.72 0.64 0.80
Tool-2Rate
1.05% 0.37% 0.76% 0.02% 0.45%
ΔPSNR 0.05 0.25 0.01 0.06 0.06
Tool-3Rate
7.21% 4.69% 10.85% 7.81% 4.85%
ΔPSNR 0.06 0.02 0.06 0.06 0.05
Tool-4Rate
15.28% 2.45% 0.65% 19.30% 5.87%
ΔPSNR 0.18 0.06 0.02 0.25 0.12Table I: Assigned cost of using each tool for two different cases
Case-1 Case-2
Tool-1 (Subpixel MV)
10 10
Tool-2 (Loop filter)
10 0.13
Tool-3(CABAC)
10 1.55
Tool-4(Multiple Ref.)
10 1.82
• Case 1: “Flat Rate”, similar to today’s licensing.• Case 2: “Fair Rate”, each tool gets paid based on its contribution
1313
““Flat Rate”Flat Rate”
400 600 800 1000 120033
34
35
36
37
38
39
Rate (kbps)
PS
NR
(dB
)
Adaptive vs Constant Tool Selection, Case-1
C=%100Adaptive toolset, C=75%Constant toolset, C=75%Adaptive toolset, C=50%Constant toolset, C=50%Adaptive toolset, C=25%Constant toolset, C=25%
• Constant toolset: Optimized tools for the entire duration of the content.• Adaptive toolset: Optimized tools for each GOP (toolset can change every GOP = media segment)
• concatenation of 10 clips• Significant reduction in royalty costs with small loss in efficiency • Inefficient tools get cut out (lobbying for tools is useless).• Adaptive ~ constant
Quality
1414
““Fair Rate”Fair Rate”
0 500 1000 1500 200032
33
34
35
36
37
38
39
40
Rate (kbps)
PS
NR
(dB
)
Adaptive vs Constant Tool Selection, Case-2
C=%100Adaptive toolset, C=75%Constant toolset, C=75%Adaptive toolset, C=25%Constant toolset, C=25%Adaptive toolset, C=12%Constant toolset, C=12%
• Adaptive better than constant.• Significant reductions are difficult.
Quality
1515
Tool UsageTool Usage
• Adaptive allows more tools to contribute (tools that are good in niche situations get used).
Tool-1 Tool-2 Tool-3 Tool-4
C=75%Adaptive
97% 48% 98% 57%
C=75% Constant
100% 0% 100% 0%
C=25%Adaptive
69% 0% 20% 10%
C=25% Constant
100% 0% 0% 0%
C=75%Adaptive
76% 72% 98% 67%
C=75%Constant
100% 100% 0% 0%
C=25%Adaptive
29% 41% 2% 18%
C=25%Constant
0% 0% 100% 100%
“Flat Rate”
“Fair Rate”
1616
ConclusionConclusion• A system that allows practical deployment of royalty cost optimized media delivery.A system that allows practical deployment of royalty cost optimized media delivery.
• Very interesting optimization problem with sophisticated royalty costs. Very interesting optimization problem with sophisticated royalty costs.
• Content adaptive.Content adaptive.
• Increases efficiency. Allows non-standard tools to contribute. Increases efficiency. Allows non-standard tools to contribute.
• Much reduced royalties when plenty of bandwidth/resources.Much reduced royalties when plenty of bandwidth/resources.
• Flat rate: significant reductions in royalty cost possible. Flat rate: significant reductions in royalty cost possible.
• Fair rate: significant reductions difficult.Fair rate: significant reductions difficult.
• Adaptive optimization allows each tool to contribute when its niche comes.Adaptive optimization allows each tool to contribute when its niche comes.
• Can allow other resources relevant to media delivery into optimization (power consumption, Can allow other resources relevant to media delivery into optimization (power consumption,
memory usage, etc.)memory usage, etc.)
• Going forwardGoing forward::
• Optimization issues.Optimization issues.
• More tools.More tools.
• Dependencies among tools.Dependencies among tools.
• Addition of communication/networking related tools.Addition of communication/networking related tools.
• Fair rate• Allow segment based licensing
1717
• Combines game theory and economic concepts with rate-distortion Combines game theory and economic concepts with rate-distortion
• How much are customers willing to pay to get quality Q at bandwidth B? (Utilization curves).How much are customers willing to pay to get quality Q at bandwidth B? (Utilization curves).
• Optimal costs based on utilization curves and game theory.Optimal costs based on utilization curves and game theory.
• What does today’s licensing look like?What does today’s licensing look like?
• Games Games
Going forward: How should tools Going forward: How should tools be priced?be priced?
Distortion
Rate
T1 + T2: cost3(R)
T2: cost2(R)T1: cost1(R)Free
R
free regionD,R not
achievable