8/10/2019 MMC12 Intro

1/37

A

C

CA E

D DBE

A

C

CA E

D DBE

C

1. Introduction2. Fundamentals (Signal Processing,

Information Theorie)3. Speech Processing & Coding4. Audio Processing & Coding5. Still Image Coding (JPEG, etc.)6. Video Coding (MPEG, etc.)7. MPEG-4 Multimedia Framework, MPEG-78. 3D Video and Free Viewpoint Video

8/10/2019 MMC12 Intro

2/37

A

C

CA E

D DBE

C

C:

: = , , , ...

: C,

D

, ,

A

C

CA E

D DBE

Basic terms and principles of information theory andsignal processing for multimedia

Information theory, entropy coding

Communication channel

Signals Sampling Quantisation Frequency transforms Linear systems (filtering) Statistics Prediction

8/10/2019 MMC12 Intro

3/37

A

C

CA E

D DBE

Most important communication medium of humanity

Technical breaktroughs in this area always causedsignificant changes of society as a whole

Writing, printing, telephone, mobile phone, ...

Economy, sociology, culture, ...

Most important: understandability

Frequency range 300-3400 Hz sufficient

64 kbit/s in ISDN for speech transmision

A

C

CA E

D DBE

A

Quelle: Fellbaum

8/10/2019 MMC12 Intro

4/37

A

C

CA E

D DBE

Vowels, consonants, breaks, transitions

Models of speech generation (vocoder)

Modern speech codecs (CELP, G.728), 4-16 kbit/s

sch p r a ch s i g n a l

A

C

CA E

D DBE

Speech in generated by complex coordination of lung,vocal chords, edges/splits (neck, throat, mouth),

articulation organs (mouth, nose)

Lung

Vocal

Chords

Edges

Splits

Articula-tion Organs

a

1-a

Speech

8/10/2019 MMC12 Intro

5/37

A

C

CA E

D DBE

E.g. for speech synthesis but also for coding (GSM, CELP)

Energy

source

Impulsegenerator

Noise

generator

Linear

system

a

1-a

Speechsignal

A

C

CA E

D DBE

A

=

A =

A

=

E =

D

8/10/2019 MMC12 Intro

6/37

A

C

CA E

D DBE

A

OssicleBalanceorgan

Acousticnerve

Snail

Eustachiantube

EardrumAuditorycanal

A

C

CA E

D DBE

A

8/10/2019 MMC12 Intro

7/37

A

C

CA E

D DBE

A

A

C

CA E

D DBE

B

inaudible audible

8/10/2019 MMC12 Intro

8/37

A

C

CA E

D DBE

E1 A 3 / 3

Filterbank

Analyse

MDCT

dyn. FensterungSkalierer

Quantisierer

Huffman-

CoderMux

FFTCoder

Zusatzinfo

Mithr-

schwellen

Filterbank

Synthese

Inverse MDCT

dyn. Fensterung

Deskalierer

Dequantisierer

Huffman-

Decoder De-

MuxDecodier

Zusatzinfo

Compression e.g. to 128 kbit/s at ame quality, ratio 1:10

A

C

CA E

D DBE

2D matrix of picture elements (pixels)

Color representation e.g. RGB

8 2563 = 16,7

10

I 10 24 : 30

8/10/2019 MMC12 Intro

9/37

A

C

CA E

D DBE

C

Exploitation of properties of human visual system forefficent compression

Perception of color, brightness, contrast

JPEG: frequency transform and quantization infrequency domain => reduction of visual irrelevancy

A

C

CA E

D DBE

Quelle: [EGBECK]

E

8/10/2019 MMC12 Intro

10/37

A

C

CA E

D DBE

Quelle: [EGBECK]

cones

rods

A

C

CA E

D DBE

C

Quelle: [EGBECK]

8/10/2019 MMC12 Intro

11/37

8/10/2019 MMC12 Intro

12/37

A

C

CA E

D DBE

E

1:150

A

C

CA E

D DBE

New dimension: time, motion

Each single image is only visible for a short moment

Consecutive images are very similar

Changes due to motion of objects and camera (amongother things)

Key for efficient compression: motion estimation &

compensation

8/10/2019 MMC12 Intro

13/37

A

C

CA E

D DBE

E

A

C

CA E

D DBE

E

Without motion compensation With motion compensation

8/10/2019 MMC12 Intro

14/37

A

C

CA E

D DBE

Without motion compensation With motion compensation

E

A

C

CA E

D DBE

C B

D G (G)

P B B I B B P B B P

Prdiktion

Interpolation

B B P B B P B B I

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

a)

b) I

3

B

1

B

2

P

6

B

4

B

5

P

9

B

7

B

8

P

12

B

10

B

11

P

15

B

13

B

14

Bearbeitungsreihenfolge

Bildnummer

Bildtyp

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

B

8/10/2019 MMC12 Intro

15/37

A

C

CA E

D DBE

Entropy

Coding

Scaling & Inv.

Transform

Motion-Compensation

Quant.

Transf. coeffs

Motion

Data

Decoder

Motion

Estimation

Transform/Scal./Quant.-

InputVideo

Signal

Split intoMacroblocks

16x16 pixels

Output

VideoSignal

Hybrid Coding (MC+DCT)

DPCM

T-1

A

C

CA E

D DBE

/

8/10/2019 MMC12 Intro

16/37

A

C

CA E

D DBE

E .264/AC

1:47 1:165 1:495

Compression ratio

A

C

CA E

D DBE

1:21 1:98 1:429

E .264/AC

Compression ratio

8/10/2019 MMC12 Intro

17/37

A

C

CA E

D DBE

1:15 1:54 1:238

E .264/AC

Compression ratio

A

C

CA E

D DBE

C C

C

20 /

111001010

C C C

8/10/2019 MMC12 Intro

18/37

A

C

CA E

D DBE

A/

Which one is the original ?

A

C

CA E

D DBE

A/

Synthetictexture

Originaltexture

8/10/2019 MMC12 Intro

19/37

A

C

CA E

D DBE

E4

Multimedia Standard

Audio-visual Scene, consists of Audio Video (arbitrary shape) Still images 2D/3D computer graphics Text Interaction mechanisms

AV-Scenes are composed and rendered

A

C

CA E

D DBE

E4

8/10/2019 MMC12 Intro

20/37

A

C

CA E

D DBE

C C

B (B)

117

498 vertices 1940 vertices 7646 vertices

A

C

CA E

D DBE

C C

Original AFX-IC60,1 kbit/s

D3DMC62,7 kBit/s

8/10/2019 MMC12 Intro

21/37

A

C

CA E

D DBE

C C

AFX-IC at 109 kbit/s D3DMC at 108 kbit/s

A

C

CA E

D DBE

E7

Tremendous amount of multimedia is available andgrowing

Search for content gets more and more difficult

Automatic tools to assist search are necessary, searchenigines for the Internet

Metadata are tagged to multimedia data for contentdescription and classification

8/10/2019 MMC12 Intro

22/37

8/10/2019 MMC12 Intro

23/37

A

C

CA E

D DBE

3D

3D Video (3DV, also known as stereo) and Free ViewpointVideo (FVV) describe functionalities

Expand the users sensation beyond what is offered byclassical 2D video

They do not exclude each other, can be combined in a

single system

A

C

CA E

D DBE

Free Viewpoint Video

Same functionality as CGobjects free navigation, can be

viewed from anyviewpoint/direction

Integration into complete

scenes(virtual/augmented/real)

But: depict appearance,

motion, deformation of realworld objects

8/10/2019 MMC12 Intro

24/37

A

C

CA E

D DBE

Integrated Scene with FVV

A

C

CA E

D DBE

Free Viewpoint Video

8/10/2019 MMC12 Intro

25/37

A

C

CA E

D DBE

Free Viewpoint Video

A

C

CA E

D DBE

Free Viewpoint Video

8/10/2019 MMC12 Intro

26/37

A

C

CA E

D DBE

Free Viewpoint Video

A

C

CA E

D DBE

D (D)

Usage of N videos with depth informationInterpolation of virtual intermediate views along all cameras

(pair-wise switching)

8/10/2019 MMC12 Intro

27/37

A

C

CA E

D DBE

< 180

Noriginalcamera views

arbitraryview area

A

C

CA E

D DBE

3DV / Stereo Video

8/10/2019 MMC12 Intro

28/37

A

C

CA E

D DBE

Philips WoWvx autostereoscopic display

9 views at the same time http://www.business-sites.philips.com/3dsolutions/about/Index.html

3DV / Stereo Video

A

C

CA E

D DBE

Fehn

MERL

Holografika

USC

3DV / Stereo Video

8/10/2019 MMC12 Intro

29/37

A

C

CA E

D DBE

3DV/FVV Processing Chain

A B

1840:

1920:

1950:

1980:

C:!

8/10/2019 MMC12 Intro

30/37

2010 B?

3D

3D

C

3D C

8/10/2019 MMC12 Intro

31/37

B

C , 1838

A , 2011

EIC 2011, B, E

C D C

I

screen

accomodationvergence

8/10/2019 MMC12 Intro

32/37

A , 2011

EIC 2011, B, E

C

screen spacetheater

space

screen

painful

retinal rivalry areas

retinal rivalry

areas

Comfortable 3D

Painful 3Dvergence

D

A

F

E

3

= ?

8/10/2019 MMC12 Intro

33/37

A & C

A

E

: , ,

E D & C

8/10/2019 MMC12 Intro

34/37

8/10/2019 MMC12 Intro

35/37

D

A A

3D

, , ..

3D

C A

3D@

3D

3D

8/10/2019 MMC12 Intro

36/37

8/10/2019 MMC12 Intro

37/37

Same functionality asCG objects free navigation, can

be viewed from anyviewpoint/direction

Integration intocomplete scenes

(virtual/augmented/r

eal)

But: depictappearance, motion,

deformation of realworld objects

& C