C. -Y. Tsai 2006/4/201Communication & Multimedia

Multiview Video CompressionMultiview Video Compression

Student: Chia-Yang TsaiStudent: Chia-Yang TsaiAdvisor: Prof. Hsueh-Ming HangAdvisor: Prof. Hsueh-Ming Hang

Institute of Electronics, NCTUInstitute of Electronics, NCTU

C. -Y. Tsai 2006/4/202Communication & Multimedia

OutlineOutline IntroductionIntroduction Coding methodsCoding methods PerformancePerformance ConclusionsConclusions

C. -Y. Tsai 2006/4/203Communication & Multimedia

ReferencesReferences

A. Smolic, P. Kauff, “Interactive 3-D video representation and coding technologies”, Proceedings of the IEEE, vol. 93,  no. 1, pp 98-110, Jan. 2005.

ISO/IEC JTC1/SC29/WG11, “Submissions received in CfP on Multiview Video Coding”, MPEG Document M12969, Bangkok, Thailand, January 2006.

ISO/IEC JTC1/SC29/WG11, “Multiview video compression using V frames”, MPEG Document M12828, Bangkok, Thailand, January 2006.

C. -Y. Tsai 2006/4/204Communication & Multimedia

IntroductionIntroduction

C. -Y. Tsai 2006/4/205Communication & Multimedia

Multivew= Multiple ViewpointsMultivew= Multiple Viewpoints

Applications Applications of multiviewof multiview 3D video 3D video Free Free

viewpoints viewpoints selectionselection

C. -Y. Tsai 2006/4/206Communication & Multimedia

Multivew= Multiple ViewpointsMultivew= Multiple Viewpoints

Reasons for multiview compressionReasons for multiview compression PC is powerful enoughPC is powerful enough Increasing network bandwidthIncreasing network bandwidth Future 3D videoFuture 3D video

C. -Y. Tsai 2006/4/207Communication & Multimedia

MPEG StandarizationMPEG Standarization

Call for proposal (N7567, Oct. 2005)Call for proposal (N7567, Oct. 2005) Proposal competition (M12969, Jan. 2006)Proposal competition (M12969, Jan. 2006)

NTT and Nagoya UniversityNTT and Nagoya University Thomson and University of Southern CaliforniaThomson and University of Southern California KDDI Corp.KDDI Corp. ETRI and Sejong University (=M12871)ETRI and Sejong University (=M12871) MERL (=M12828)MERL (=M12828) KBS and Yonsei University (=M12874)KBS and Yonsei University (=M12874) Fraunhofer-HHI (=M12945)Fraunhofer-HHI (=M12945) Technical University of BerlinTechnical University of Berlin

C. -Y. Tsai 2006/4/208Communication & Multimedia

Coding MethodsCoding Methods

Disparity compensated view prediction (DCVP)Disparity compensated view prediction (DCVP)View synthesis prediction (VSP)View synthesis prediction (VSP)

Hierarchical B framesHierarchical B frames

C. -Y. Tsai 2006/4/209Communication & Multimedia

Multiview Frame StructureMultiview Frame Structure

1 2 3 4 5 6 7

.

.

.

…..

time

view

C. -Y. Tsai 2006/4/2010Communication & Multimedia

Block diagramBlock diagram

Predictions based on H.264/AVC JM95Predictions based on H.264/AVC JM95

View i picture

Motion Compensation

Motion Estimation

Entropy Coding

Bitstream+ Quantization

IQuantization

+

Transform

ITransform

Deblocking Filter

Reference Picture

Store for View i

Reference Picture

Store for Other Views

Disparity/Illumination

Compensation

Disparity/Illumination Estimation

Intra Prediction

-

+

+

+

Mode Decision

C. -Y. Tsai 2006/4/2011Communication & Multimedia

Block diagramBlock diagram DecoderDecoder

C. -Y. Tsai 2006/4/2012Communication & Multimedia

DCVPDCVP DCVP= Disparity Compensated View DCVP= Disparity Compensated View

PredictionPrediction ProblemsProblems

High spatial correlations between High spatial correlations between different viewpointsdifferent viewpoints

SolutionSolution Prediction between viewpointsPrediction between viewpoints

C. -Y. Tsai 2006/4/2013Communication & Multimedia

DCVPDCVP DCVP= Disparity Compensated View PredictionDCVP= Disparity Compensated View Prediction

…..

I B B P B B I B B P B B I

I B B P B B I B B P B B I

I B B P B B I B B P B B I

I B B P B B I B B P B B I

I B B P B B I B B P B B I

P

B

B

C. -Y. Tsai 2006/4/2014Communication & Multimedia

VSPVSP

VSP= View Synthesis PredictionVSP= View Synthesis Prediction ProblemsProblems

Different viewpoints have different Different viewpoints have different depthdepth Rotation, translation speedRotation, translation speed

SolutionSolution Synthesis virtual images before real Synthesis virtual images before real

predictionprediction

C. -Y. Tsai 2006/4/2015Communication & Multimedia

VSPVSPtime

view

View Synthesis ViaView Interpolation

View Synthesis ViaView Warping

)(],,,[]1,,[)()(],,[ 1 cTyxtcDyxcAcRwvu

)}'(],,{[)'()'(]',','[ 1 cTwvucRcAzyx

R: Rotation matrixR: Rotation matrix

D: Depth informationD: Depth information

T: Translation matrixT: Translation matrix

A: Intrinsic matrixA: Intrinsic matrix

C

C’

C. -Y. Tsai 2006/4/2016Communication & Multimedia

How to get depths?How to get depths?

From camera recordFrom camera record From well-known computer vision algorithmsFrom well-known computer vision algorithms Block-based depth searchBlock-based depth search

||]',',,'[],,,[||),,,( min}

max...2

min,

min,

min{

yxtcIyxtcIyxtcD

Dstep

DDstep

DDDd

where || I [c,t,x,y] – I [c’,t,x’,y’] || denotes the average error between the block at (x,y) in camera c at time t

C. -Y. Tsai 2006/4/2017Communication & Multimedia

How to get depths?How to get depths?

Depths map:Depths map: Left: computer vision algorithmLeft: computer vision algorithm Right: block based depth searchRight: block based depth search

Compression result:Compression result: Depth information: 5-10% total bitratesDepth information: 5-10% total bitrates Left and right have equal performanceLeft and right have equal performance

C. -Y. Tsai 2006/4/2018Communication & Multimedia

Prediction structurePrediction structure

C. -Y. Tsai 2006/4/2019Communication & Multimedia

Hierarchical B picturesHierarchical B pictures

Hierarchical B picturesHierarchical B pictures Fully compatible to AVC Main profileFully compatible to AVC Main profile Non-dyadic decomposition is availableNon-dyadic decomposition is available

A B3 B2B1 AB3 B3 B3B2

L3 H1 H2H3H2 L3H1 H1 H1

GOP boundaries

AVC Main Profilecompatible base layer

MCTF enhancementlayer

C. -Y. Tsai 2006/4/2020Communication & Multimedia

Hierarchical B picturesHierarchical B pictures

C. -Y. Tsai 2006/4/2021Communication & Multimedia

PerformancePerformance

C. -Y. Tsai 2006/4/2022Communication & Multimedia

ExperimentsExperiments

C. -Y. Tsai 2006/4/2023Communication & Multimedia

ExperimentsExperiments

C. -Y. Tsai 2006/4/2024Communication & Multimedia

ConclussionConclussion

C. -Y. Tsai 2006/4/2025Communication & Multimedia

ConclussionConclussion

DCVP & VSP can reduce the correlation DCVP & VSP can reduce the correlation between viewpointsbetween viewpoints

Future workFuture work Depth search algorithmsDepth search algorithms Motion synthesisMotion synthesis

MCTFMCTF Correlation between temporal and viewpoints axisCorrelation between temporal and viewpoints axis

Rate-controlRate-control ScalabilityScalability Error protectionError protection Low delay issueLow delay issue

C. -Y. Tsai 2006/4/2026Communication & Multimedia

Thanks for your attention!Thanks for your attention!Any questions?Any questions?