Fraunhofer Institute for TelecommunicationsHeinrich-Hertz-Institut Berlin Einsteinufer 3710587 BerlinGermany
3D Video Formats and Coding Standards
+49 30 310 02 – [email protected] http://www.hhi.fraunhofer.de
Karsten Müller
02.11.2009© Fraunhofer HHI
Folie 2
Kars
ten
Mül
ler
Overview 3D Video
• Introduction, scenarios, requirements
• 3D video formats
• 3D coding standards
• Towards a new 3D video coding standard
02.11.2009© Fraunhofer HHI
Folie 3
Kars
ten
Mül
ler
3D Video Introduction
• Provision of left and right eye view• Added depth sensation • Resolved 2D viewing
ambiguity• Additional features
(e.g. free viewpoint, depth-controlled object insertion)
3D Video is more than adding 1 dimension to 2D! Source: Philips
02.11.2009© Fraunhofer HHI
3D video scenarios3D cinema• Increasing number of 3D productions• First studios start to release all productions also in 3D• Technology: stereoscopic 3D, glasses based
3D home entertainment• Different types of displays available: stereoscopic, auto-stereoscopic
with 2 … N views• Various technologies, input formats and display sizes• Glasses based systems may not be acceptable
3D mobile• auto-stereoscopic 2 view display with fixed viewing position• Good 3D viewing in spite of small display sizes
Slide 4
Philip
p M
erkl
e
02.11.2009© Fraunhofer HHI
Folie 5
Kars
ten
Mül
ler
Requirements: 3D processing chain
• Stereo / multiview capturing, depth provision
• Production format specification (e.g. SMPTE)
• Rectification, color correction, format conversion
• 3D format specification
• 3D video coding (e.g. MPEG)
• High quality intermediate view synthesis(e.g. 2 view + 2 depth N view)
• High-resolution stereo/multiview display
Capture
Post-processing
Coding
View Synthesis
ProductionFormat
TransportFormat
Transport Format
Display
Display Format
02.11.2009© Fraunhofer HHI
Folie 6
Kars
ten
Mül
ler
The plethora of 3D Video Formats
• Conventional stereo video (CSV)• Mixed resolution stereo (MRS)• Video plus depth (V+D)
• Multiview video (MVV)• Multiview video + depth (MVD)• Layered depth video (LDV)
• Depth-enhanced stereo (DES)
02.11.2009© Fraunhofer HHI
Slide 7
Philip
p M
erkl
e
assuming multiple views with color & depth…
Image Source: Interactive Visual Media Group of Microsoft Research
02.11.2009© Fraunhofer HHI
Slide 8
Philip
p M
erkl
e
Stereo video (CSV)
CSV
02.11.2009© Fraunhofer HHI
Slide 9
Philip
p M
erkl
e
Mixed resolution stereo (MRS)
MRS
02.11.2009© Fraunhofer HHI
Slide 10
Philip
p M
erkl
e
Video plus depth (V+D)
V+D
MVV
02.11.2009© Fraunhofer HHI
Slide 11
Philip
p M
erkl
e
Multiview video (MVV)
MVD
02.11.2009© Fraunhofer HHI
Slide 12
Philip
p M
erkl
e
Multiview video plus depth (MVD)
02.11.2009© Fraunhofer HHI
LDV
Slide 13
Philip
p M
erkl
e
Layered depth video (LDV)
02.11.2009© Fraunhofer HHI
DES
Slide 14
Philip
p M
erkl
e
Depth-enhance Stereo (DES)
02.11.2009© Fraunhofer HHI
Folie 15
Kars
ten
Mül
ler
3D Video Coding Standard(s)
• Advanced Video Coding (MPEG-4 AVC / H.264)– Simulcast– Stereo SEI– Auxilliary Picture Synthax
• MPEG-C part 3 (container format for V+D, e.g. with AVC coding)
• Multi-view Video Coding (MPEG-4 MVC / H.264)
All coding approaches are based on MPEG-4 AVC / H.264 standards family
02.11.2009© Fraunhofer HHI
Folie 16
Kars
ten
Mül
ler
Towards a new 3D Video Coding Standard
• MPEG develops a new 3D video coding standard
• Motivation for new standard:– Decouple production from coding format– MPEG-4 AVC/H.264 only optimized for 2D color video, but not for
depth information
3D components under consideration:– Transport format– 3D video coding methods– Intermediate view synthesis
02.11.2009© Fraunhofer HHI
Folie 17
Kars
ten
Mül
ler
Coverage of 3D Video Coding
Capture
Post-processing
Coding
View Synthesis
ProductionFormat
Transport Format
Transport Format
Display
Display Format
02.11.2009© Fraunhofer HHI
Folie 18
Kars
ten
Mül
ler
Constraints for 3DVC
• Consider capturing technology, i.e. maximal 2-3 recorded views
• Break linear dependency of coding bitrate from number of target views (e.g. MVC)
• Provide scene geometry data in general form, i.e. pixel-wise depth data
• Consider statistical properties of depth (and supplementary) data
• Consider new quality evaluation methods for intermediate views
• Provide high-quality view synthesis for continuous viewing range
• Decrease depth and coding errors
02.11.2009© Fraunhofer HHI
Folie 19
Kars
ten
Mül
ler
Summary
• 3D Video technology is maturing due to world wide development from capturing to display
• Although a variety of 3D/stereo formats exist,…• …only one underlying coding method with variations is
used: MPEG-4/AVC•
• For 3D video entertainment, new 3D Video coding beyond AVC/MVC is required and being developed
02.11.2009© Fraunhofer HHI
Folie 20
Kars
ten
Mül
ler
Acknowledements
• Special Thanks to:
– Philipp Merkle
