Challenges and solutions for real-time immersive video

Dr. Oliver Schreer 1Short Course, SP&T, 14th of April 2005

Challenges and solutions for real-time immersive video communication

Part II - 14th of April 2005

Dr. Oliver Schreer Fraunhofer Institute for Telecommunications

Heinrich-Hertz-Institut, Berlin, Germany


Structure of the Short CoursePart I (13th of April 2005)

Introduction

Scope of immersive video communication

Quick tour into projective geometry

Part II (14th of April 2005)Quick tour into camera model and epipolar geometry

Analysis of real-time video streams

Part III (15th of April 2005)The concept of real-time immersive video conferencing

A quick tour into stereo image processing

Hybrid-recursive disparity estimation


Examples of Projective Transformations


The Projective Plane PP22

projective line withcorrespondence to affine planeprojective point with

correspondence to affine plane

ideal line or line at infinity

W=0ideal point or

point at infinity

affine plane PP22

W=1

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

WYX

m

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

0YX

m

projectiveplane PP22

X

YW

C

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛==

1YX

Wmmaffine

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

W00

l


OutlineCamera model

Epipolar geometry


Camera model

Epipolar geometry



The Object of Interest


The Pinhole Camera

Gemma Frisius, 1545De Radio Astronomica et GeometricaAlbrecht Dürer, 1525

Underweysung der Messung


The Pinhole Camera

( )vu,=m

( )www Z,Y,X=M


Camera Coordinate System

C

YC

ZC

XC

M

m


World Coordinate System

C

YC

ZC

XC

Yw

Zw

Xw

(R,t)

M

m


Sensor Coordinate System

C

YC

ZC

XC

M

m

c

y

xf

Yw

Zw

Xw

(R,t)


Image Coordinate System

C

YC

ZC

XC

M

m

v

u

v0

u0

c

y

xf

Yw

Zw

Xw

(R,t)


The perspective projection

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

====

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

1

~~~

11vu

~W

W

W

WWNCNC

C

C

N ZYX

MMMZYX

ss PPDAPAPAPm

Embedding of euklidian3D point in the

projective space PP33

Mapping of the point in theprojective space PP33 to the

projective plane PP22

Division by 3rd componentleads to the point on the affine

plane, the image plane


Non-linear Lens Distortion

radial distortion

distorted undistorted


OutlineCamera model

Epipolar geometry


Camera model

Epipolar geometry



Parallel Stereo Camera System

m1 m2

f f

M

ρ

Bu1

I1

C1

I2

u2 C2


Convergent Stereo Geometry

C2

m2

Π

C1

I1 I2

M

m1

(R,t)


The Epipoles

M

m1

C1 C2

m2

Π

I1 I2

e2e1

(R,t)


The Epipolar Lines

M

m1

C1 C2

m2

l1 l2

Π

I1 I2

e2e1

(R,t)


Convergent Stereo View and Epipolar Lines


Thank you for your attention in the first part!

Coffee break


OutlineCamera model

Epipolar geometry



The 3D Video Processing Chain

AnalysisAnalysis SynthesisSynthesis

TransmissionDecoding

Analysis•Segmentation•Tracking•Rectification•Depth analysis (stereo)

Encoding(MPEG-4, H.264)

Synthesis

Composition of thescene

Analysis•Segmentation•Tracking


Segmentation and Shadow Detection

change detection shadow detection in the YUV-space

background adaptation


Shadow in Colour SpacesColour in Human Perception:

huesaturationintensity

B

R

G

RGB

V

S H

HSV

YUV

U

V

Y

What is a shadow?Change of intensityChange of saturationConstant hue

Problems:YUV is common video formatColour space transf.Real-time constraints


Shadowed Areas in the YUV Space

Y/V-plane (blue)Y/U-plane (blue)

Y/V-plane (green)Y/U-plane (green)


Solution: Shadow Detection in YUV

U

V

Y

P

α

Approximation of Colour:

Shadow detection:Change of saturation and intensityConstant Hue

Influence of shadow to YUV-value:

Saturation: Distance to originHue: vector angle in UV-plane

Linear shift towards originConstant angle in the U/V-plane


Fast Decision SchemeInvestigate segmented object

Y smaller than BG ? Angle difference withinboundary ?

U

V

Y

Pshadowed

PBG

U

V

Pshadowed

PBG

α2

α1


Experimental Results

Pure Segmentation Shadow Detection


Texture Adaptive FeatheringProblems: corona, flickering


Transparency Masks

Composition into new scene

I = α⋅F + (1-α)⋅B

Segmentation

binary transparent


Usage of Delta Band

delta bandbinary maskoriginal image withcontour


Comparative Results


Avatar Animation - A Tele-Consulting System

Handsegmentation and Tracking

Filter

Feature Point Tracking Player

Video In

Character

2D Feature Conversion

Viseme ConversionViseme Generator

Audio In

Net

wor

k

Operator

Display

Customer Avatar Animation Parameter

(FAP’s, BAP’s)

Speaker Audio Grabber N

etw

ork

Inte

rface

Net

wor

k In

terfa

ce

Video Analysis


Segmentation and Tracking of HandsSegmentation based on skin colour

Specific colour range in UV-spaceindependent of human race

Bounding Boxes

Tracking of bounding boxesReduction of processing speed

Stabilisation of segmentation


Overview of the Approach

Region GrowingApproach

on high resolutionOriginal image

Subsampling + Tracking

on low resolution


Tracking of Hands in Critical Situations


Facial Feature Tracking

tracking of a few facial features

approximation of head rotation by relative motion of featurescompared to center of gravity of skin pixels


Experimental Result: Reconstructed Nick/Turn Angles

head nick and turn angle

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

1 frame

FAP

valu

e

nick angleturn angle


Animation of an Avatar


ConclusionCamera model provides a sufficient frame work for the imagingprocess of one camera

Epipolar geometry describes the relation between two cameras – theimaging process of two cameras

Advanced segmentation for real-time mixed reality applicationsrequires

Speed and robustness

Shadow detection

Alpha blending


..... thank you for your attention

END

Date post:	04-Feb-2022
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Challenges and solutions for real-time immersive video

Documents