75
Lecture 3 Mei-Chen Yeh 2010/03/16
Lecture 3
Mei-Chen Yeh2010/03/16
Announcements (1)
• Assignment formats:– A word or a pdf file– Subject ( 主旨 ):
Multimedia System Design-Assignment #X-your student id-your nameMultimedia System Design-Assgnment #2-697470731- 游宗毅
– File name ( 檔名 )Assignment #X-your student id-your nameAssignment #2-697470731- 游宗毅 .doc
Announcements (2)
• For the assignment#2…• If you did not use either a pdf or a doc file, please re-send
your report to TA using the format.• Due 03/16 (today)
…and based on TA’s clock
Announcements (3)
• The reading list is finally released!• Sources:
– Proceedings of ACM MM 2008– Proceedings of ACM MM 2009– The best paper in MM 2006 and MM 2007
• Interesting papers not on the list? Let me know!
Announcements (4)
• So you need to…– Browse the papers– Discuss with your partners about the paper choice– …and do it as soon as possible! I love that paper!
That paper sucks…
How to access the papers?
• The ACM digital library– http://portal.acm.org/– Should be able to download the papers if you connect to the site on
campus
• MM08 paper on the web– http://mpac.ee.ntu.edu.tw/multimediaconf/acmmm2008.html
• Google search
Next week in class
• Bidding papers! ( 論文競標 )• Each team will get a ticket, where you put
your points.
Ticket # 7Team name: 夜遊隊Team members: 葉梅珍 游宗毅-------------------------------------------------------------------paper1 paper2 paper3 paper4 … paper 25
50 10 15 20 5
Total : 100 points
Bidding rules
• The team with the most points gets the paper.• Every team gets one paper.• When a tie happens…
…and the winner takes the paper.
More about the bidding process
• Just, fair, and open!
• I will assign a paper to teams in which no one show up for the bid.
公平 公正 公開
Questions
Multimedia Compression (1)
Outline
• Introduction• Information theory• Entropy ( 熵 ) coding
– Huffman coding– Arithmetic coding
Why data compression?
• Transmission and storage
– For uncompressed video• CD-ROM (650MB) could store 650MB x 8 / 221Mbps ≈ 23.5
seconds• DVD-5 (4.7GB) could store about 3 minutes
Approximate Bit Rates for Uncompressed Sources
What is data compression?
• To represent information in a compact form (as few bits as possible)
• Technique
Original
Reconstructed data
Compressed data
Compression Reconstruction
Codec = encoder + decoder
Technique (cont.)
• Lossless– The reconstruction is identical to the
original.
• Lossy– Involves loss of information Lossy!
Lossless? Not necessarily true!
Do not send money!Do now send money!
Encoder Decoder
source
Example: image codec
Performance MeasuresHow do we say a method is good or bad?
• The amount of compression• How closely the reconstruction is• How fast the algorithm performs• The memory required to implement
the algorithm• The complexity of the algorithm• …
Two phases: modeling and coding
• Modeling– Discover the structure in the data– Extract information about any redundancy
• Coding– Describe the model and the residual (how the
data differ from the model)
Original
Compressed data
Fewer bits!
Encoder
Example (1)
• 5 bits * 12 samples = 60 bits• Representation using fewer bits?
Example: Modeling
,...2,18ˆ nnxn
n
Example: CodingnxOriginal data
8ˆ nxnModel 10 11 12 13 14 15 16 17 18 19 20
Residual nnn xxe ˆ 1 0 -1 1 -1 0 1 -1 -1 1 10
• {-1, 0, 1} • 2 bits * 12 samples = 24 bits (compared with
60 bits before compression)We use the model to predict the value, then encode the residual!
Another Example
• Morse Code (1838)
Shorter codes are assigned to letters that occur more frequently!
A Brief Introduction to Information Theory
Information Theory (1)
• A quantitative ( 量化的 ) measure of information– You will win the lottery tomorrow.– The sun will rise in the east tomorrow.
• Self-information [Shannon 1948]P(A): the probability that the event A will happen)(log
)(
1log)( AP
APAi bb
b determines the unit of information
The amount of surprise or uncertainty in the message
Information Theory (2)
• Example: flipping a coin– If the coin is fairP(H) = P(T) = ½i(H) = i(T) = -log2(½) = 1 bit
– If the coin is not fairP(H) = 1/8, P(T)=7/8i(H) = 3 bits, i(T) = 0.193 bitsThe occurrence of a HEAD conveys more information!
Information Theory (3)
• For a set of independent events Ai
– Entropy (the average self-information)
– The coin example• Fair coin (1/2, 1/2): H=P(H)i(H) + P(T)i(T) = 1• Unfair coin (1/8, 7/8): H=0.544
SAi
)(log)()()()( ibiii APAPAiAPSH
Information Theory (4)
• Entropy– The best a lossless compression scheme can do– Not possible to know for a physical source– Estimate (guess)!
• Depends on our assumptions about the structure of the data
Estimation of Entropy (1)
• 1 2 3 2 3 4 5 4 5 6 7 8 9 8 9 10– Assume the sequence is i.i.d.
• P(1)=P(6)=P(7)=P(10)=1/16P(2) =P(3)=P(4)=P(5)=P(8)=P(9)=2/16
• H = 3.25 bits
– Assume sample-to-sample correlation exists• Model: xn = xn-1 + rn
• Residuals: 1 1 1 -1 1 1 1 -1 1 1 1 1 1 -1 1 1• P(1)=13/16, P(-1)=3/16• H = 0.7 bits
Estimation of the Entropy (2)
• 1 2 1 2 3 3 3 3 1 2 3 3 3 3 1 2 3 3 1 2– One symbol at a time
• P(1) = P(2) = ¼, P(3) = ½• H = 1.5 bits/symbol• 30 (1.5*20) bits are required in total
– In blocks of two• P(1 2) = ½, P(3 3)=½• H = 1 bit/block• 10 (1*10) bits are required in total
Coding
Coding (1)• The assignment of binary sequences to elements of an
alphabet
• Rate of the code: average number of bits per symbol• Fixed-length code and variable-length code
letter_
alphabet
codeword
code
Ambiguous Not uniquely decodable
Prefix codeinstantaneous
DecodableWith one-
symbol delay
Coding (3)
• Example of not uniquely decodable code
Letters Codea1 0a2 1a3 00a4 11
100a2 a3 =>a2 a1 a1 =>
back
100
Coding (4)
• Not instantaneous, but uniquely decodable code
Oops!
a2
a2 a3 a3 a3 a3 a3 a3 a3 a3
Prefix Codes
• No codeword is a prefix to another codeword• Uniquely decodable
Huffman Coding
• Basic algorithm• Extended form• Adaptive coding
Huffman Coding
• Observations of prefix codes
– Frequent symbols have shorter codewords– The two symbols that occur least frequently have
the same lengthHuffman procedure:Two least frequent symbols differ only in the last bit. Ex: m0, m1
Algorithm
• A = {a1, a2, a3, a4, a5}
• P(a1) = 0.2, P(a2) = 0.4, P(a3) = 0.2, P(a4) = 0.1, P(a5) = 0.1
a2 (0.4)a1 (0.2)a3(0.2) a4(0.1) a5(0.1)
a’4(0.2)
a’3(0.4)
a’1(0.6)
101000 0010 0011
(1)
111
100
0
0
Algorithm
• Entropy: 2.122 bits/symbol• Average length: 2.2 bits/symbol
101000 0010 0011a1 (0.2)a3(0.2) a4(0.1) a5(0.1)
a’4(0.2)
a’3(0.4)
a’1(0.6)(1)
111
100
0
0
a2 (0.4) 00
a2 (0.4)
a’2(0.6)
(1)
1110a3(0.2)a1(0.2)
a’1(0.4)0 1
1
1
011010a5(0.1)a4(0.1)
a’4(0.2)0 1
0
0
Which code is preferred? The one with minimum variance!
Exercisee h l o p t w
30.5% 13.4% 9.8% 16.1% 5% 22.8% 2.4%
e (30.5) h (13.4) o (16.1) p (5.0) w (2.4) l (9.8) t (22.8)
Length of Huffman Codes
• A source S with A = {a1,…,ak} and {P(a1),…,P(ak)}– Average codeword length:
• Lower and upper bounds
K
iii laPl
1
)(
1)()( SHlSH
entropy of the source
average code length
Extended Huffman Codes (1)
• Consider small alphabet and skewed probabilities– Example:
• Block multiple symbols together
symbol Prob. codeword
a 0.9 0
b 0.1 11 bit / letterNo compression!
symbol Prob. codeword
aa 0.81 0
ab 0.09 10
ba 0.09 110
bb 0.01 111 0.645 bit / letter
Extended Huffman Codes (2)
Another example:
H = 0.816 bits/symbol = 1.2 bits/symbol
H = 1.6315 bits/block = 0.816 bits/symbol = 1.7228 / 2 = 0.8614 bits/symbol
sizeblkSHlSH
1)()(
Adaptive Huffman Coding (1)
• No initial knowledge of source distribution• One-pass procedure• Based on statistics of encountered symbols
Maintain a Huffman Tree!
Adaptive Huffman Coding (2)
• Huffman tree• Node (id, weight)• Sibling property
– w(parent) = sum of w(children)– ids are ordered with non-
decreasing weights
weight: # of occurrence
Id: 1 2 3 4 5 6 7 8 9 10 11w: 2 3 5 5 5 6 10 11 11 21 32
Non-decreasing!
Adaptive Huffman Coding (3)• NYT (not yet transmitted) code
– w(NYT) = 0– Transmitted when seeing
a new letter– Smallest id in the tree
• Uncompressed code (ex: m letters)
Adaptive Huffman Coding: Encode (1)
Input: [a a r d v a r k] (Alphabet: 26 lowercase letters)Initial tree
a00000
Output: 00000
a1
1
NYT0
r10001
010001
NYT000
v1011
1
2
3
5
001011
Adaptive Huffman Coding: Encode (2)
Input: [a a r d v a r k] (Alphabet: 26 lowercase letters)
NYT00
d00011
Output: 0000010100010000011
Adaptive Huffman Coding: Encode (3)
1
2
3
5
Swap 47 and 48
3 2
5
Tree update
Adaptive Huffman Coding: Encode (4)
Swap 49 and 50
3 2
5
49 50
4748
45
43 44
46
Tree update
Adaptive Huffman Coding: Decode (1)
Input: 0000010100010000011001011…Initial tree
0000Not in the uncompressed codeGet one more bit
Output:
00000a
a
1a
a
Adaptive Huffman Coding: Decode (2)
Input: 0000010100010000011001011…
0NYT
1000Not in the uncompressed codeGet one more bit
00NYT
Output: aa
10001r
r ……
Arithmetic Coding
• Basic algorithm• Implementation
Small alphabetSkewed probability
Cases where Huffman Coding doesn’t work well
Letter Probability Codeword
a1 0.95
a2 0.02
a3 0.03
01110
H = -0.95*log(0.95)-0.02*log(0.02)-0.03*log(0.03)=0.335 bits/symbolAverage length = 0.95*1+0.02*2+0.03*2 = 1.05 bits/symbol
Average length = 1.222 bits/block = 0.611 bits / symbol
Huffman codes for large blocks
• # {codeword} grows exponential with block size– N symbols, group m symbols for a block =>
Nm codewords
• Generate codes for all sequences given a length m
• Not efficient!
Arithmetic Coding: Generate a tag
• View the entire sequence as a big block– Step 1: Map the sequence into a unique tagEx: A = {a1, a2, a3}, P(a1) = 0.7, P(a2) = 0.1, P(a3) = 0.2
Encode a1, a2, a3, …
– Step 2: Generate a binary code for the tag
0.0
1.0
0.70.8
a1
a2
a3
0.0
0.7
0.490.56
a1
a2
a3
0.49
0.56
0.5390.546
a1
a2
a3
0.546
0.56
0.55580.5572
a1
a2
a30.553
Arithmetic Coding: Interpret the tag
Ex: 0.553
0.0
1.0
0.70.8
a1
a2
a3
0.0
1.0
0.70.8
a1
a2
a3
Update the number(0.553-0.49)*(0.56-0.49)= 0.9
0.0
1.0
0.70.8
a1
a2
a3
a1
l(1) = 0.0u(1) = 0.7
a3
l(3) = 0.7+(0.8-0.7)*0.8u(3) = 0.7 +(0.8-0.7)*1.0
Update the number:(0.553-0.0)/(0.7-0.0)= 0.79
a2
l(2) = 0.0+(0.7-0.0)*0.7=0.49u(2) = 0.0+(0.7-0.0)*0.8=0.56
One more example
• A = {a1, a2, a3}, P(a1) = 0.8, P(a2) = 0.02, P(a3) = 0.18Encode a1, a3, a2, a1
0.0
1.0
0.80.82
a1
a2
a3
0.0
0.8
0.640.656
a1
a2
a3
0.656
0.8
0.77120.77408
a1
a2
a3
0.7712
0.77408
0.7735040.77356
a1
a2
a3
(0.7712+0.773504)/2= 0.772352
Ex: 0.772352
0.0
1.0
0.80.82
a1
a2
a3
(0.772352-0.0)/(0.8-0.0)= 0.96544
0.0
1.0
0.80.82
a1
a2
a3
(0.772352-0.656)*(0.8-0.656)= 0.808
0.0
1.0
0.80.82
a1
a2
a3
l(2) = 0.0+(0.8-0.0)*0.82=0.656u(2) = 0.0+(0.8-0.0)*1.0=0.8
l(1) = 0.0u(1) = 0.8
l(3) = 0.656+(0.8-0.656)*0.8=0.7712u(3) = 0.656+(0.8-0.656)*0.82=0.77408
(0.772352-0.7712)*(0.77408-0.0.7712)= 0.4
0.0
1.0
0.80.82
a1
a2
a3
Generating a binary code
• Use the binary representation of the tag– Truncate to bits– probability↗, interval↗, required bits↘Ex:
– Bounds:
Symbol Prob. Cdf Tag In binary Code
a1 0.5 0.5
a2 0.25 0.75
a3 0.125 0.875
a4 0.125 1.0
0.0
1.0
0.5
0.75
a1
a2
a40.875
a3
0.25
0.625
0.8125
0.9375
.0100
.1010
.1101
.1111
2
3
4
4
01
101
1101
1111An extreme case where the sequence has only one letter
lengthsequenceSHlSH
2)()(
Adaptive Arithmetic Coding
• A = {a1 , a2 , a3}
• Input sequence: a2 a3 a3 a2
0.0
1.0
0.33
0.67
a1
a2
a3
1/3
1/3
1/3
1/4
2/4
1/4
0.33
0.67
0.42
0.58
a1
a2
a3
0.60
0.63
1/5
2/5
2/5
0.58
0.67
a1
a2
a3
0.64
0.65
1/6
2/6
3/6
0.63
0.67
a1
a2
a3
1/7
3/7
3/7
0.64
0.65
a1
a2
a3
Implementation
• Two problems– Finite precision– Transmit the first bit after seeing the entire
sequence
Synchronized rescaling!
Incremental encoding!
Implementation: Encode (1)
• Incremental coding– Send the MSB when l(k) and u(k) have a common
prefix
• Rescaling– Map the half interval containing the code to [0, 1)
E1: [0, 0.5)E1(x) = 2xSend 0Left shift 1 bit
E2: [0.5, 1)E2(x) = 2(x-0.5)Send 1Left shift 1 bit
Implementation: Encode (2)
Example: a1, a3, a2, a1 (tag: 0.7734375)
0.0
1.0
0.80.82
a1
a2
a3
a1 a3 send 1
E1: [0, 0.5) -> [0, 1); E1(x) = 2xE2: [0.5, 1) -> [0, 1); E2(x) = 2(x-0.5)
a2
send 1
send 0
send 0
Implementation: Encode (3)
send 0
send 1
a1
Use 0.5send 10…0
0.7734375=(.1100011)2
How to stop? 1.Send the stream size2.Use EOF (00…0)
Implementation: Decode (1)Find the smallest interval (0.82-0.8=0.02) => 6 bits (2-6 < 0.02)
11000110…0.110001=0.765625decode a1
update code:(0.765625-0)/(0.8-0)=0.957decode a3
Input: 11000110…0
11000110…0
.100011=0.546875update code:(0.546875-0.312)/0.6-0.312=0.8155decode a2
11000110…0
11000110…0
11000110…0
11000110…0
11000110…0
Implementation: Decode (2)11000110…0
update code:(0.5-0.3568)/(0.54112-0.3568)=0.7769decode a1
.10=0.5
Enhancement (optional)
• One more mapping:– E3: [0.25, 0.75) -> [0, 1); E3(x) = 2(x-0.25)
How do we transfer information about an E3 mapping to the decoder?
E3 mapping
• E3E1
• E3E2
0.0
1.0
0.5
0.75
0.25
0.25
0.75
0.5
0.625
0.375
[¼, ½): send 0 1
0.0
1.0
0.5
0.75
0.25
0.25
0.75
0.5
0.625
0.375
[½, ¾): send 1 0
E3 mapping
• E3…E3E1
• E3…E3E2
m
m
0.0
1.0
0.5
0.75
0.25
0.25
0.75
0.5
0.625
0.375
0.375
0.625
0.5
0.5625
0.4375
…
[¼, ½): 01 [¼+⅛, ½): 011 [¼+⅛+…, ½): 011…1m
0.5
Send 1 0 0 … 0
Send 0 1 1 … 1
E3 mapping: EncodeExample: a1, a3, a2, a1 (tag: 0.7734375)
0.0
1.0
0.80.82
a1
a2
a3
a1 a3 send 1
E1: [0, 0.5) -> [0, 1); E1(x) = 2xE2: [0.5, 1) -> [0, 1); E2(x) = 2(x-0.5)E3: [0.25, 0.75) -> [0, 1); E3(x) = 2(x-0.25)
m = 1
a2
send 0
send 0
send 1 0 m = 0
E3 mapping: Encode
send 1
m = 1
a1
Use 0.5send 10…0
Output:11000110…0
E3 mapping: DecodeFind the smallest interval (0.82-0.8=0.02) => 6 bits (2-6 < 0.02)
11000110…0.110001=0.765625decode a1
update code:(0.765625-0)/(0.8-0)=0.957decode a3
Input: 11000110…0
11000110…0
.100011=0.5468752*(0.546875-0.25)=0.5938
m = 1
update code:(0.5938-0.124)/(0.7-0.124)=0.8155 decode a2
11000110…0 m = 0
11000110…0
11000110…0
11000110…0
m = 1
E3 mapping: Decode
m = 1
. 10…0 = 0.52*(0.5-0.25)=0.5update code:(0.5-0.2136)/(0.58224-0.2136)=0.7769 decode a1
Output:a1 a3 a2 a1
Next week
• In-class paper bidding• Decide how you distribute your points with
your partners before coming to the class!
