1 Research & Innovation

An Industry Perspective on VVG Research

Oliver Grau

BBC Research & Innovation

VVG SUMMER SCHOOL '07

2 Research & Innovation

Overview

• Introduction– (Computer) Vision, Video & Graphics in media production

• Integration of (real) Video & Graphics• Production visualisation tools• Free-viewpoint video for Sport visualisation

• Summary

3 Research & Innovation

Introduction

• BBC is a public funded broadcaster producing different kinds of media. Mainly: Radio, TV and Online

• Video: One of the ‘Main businesses’• Graphics – used to:

– Add editorial value– Make programmes (visually) more exciting– Create new user’s experience

• Vision: Big toolbox to make things happen

4 Research & Innovation

Introduction

• Added editorial value

5 Research & Innovation

Introduction

• New user experience: 3D Virtual environment

CBBC Adventure Rock: www.bbc.co.uk/cbbc/adventurerock/

6 Research & Innovation

Introduction

7 Research & Innovation

Integration of video & graphics

• ‘Classical’ application: Weather forecast

• Studio with chroma-key facility

8 Research & Innovation

What makes an integrated ‘virtual scene’ looking ‘real’

Real Scene Virtual Scene

Depth perception

Camera perspective

Occlusions

Shadows

Reflections

9 Research & Innovation

Integration of video & graphics

• Virtual studio – inserting actor into virtual scene

Virtual background

Moving camera

Camera Tracking

10 Research & Innovation

3D reconstruction of dynamic scenes in the studio

• Virtual studios have limited features regarding optical interaction of real and virtual objects

• More realistic integration requires full optical interaction in 3D– Shadows + reflections– Occlusions– Free choice of camera angle

11 Research & Innovation

3D reconstruction of dynamic scenes in the studio

• 12 fixed, calibrated cameras with chroma-keying facility• Developed in the IST-ORIGAMI project

12 Research & Innovation

Overview

Block diagram

Volumetric visual hull

computation..

Input images,Camera parameters

Surface computationMarching cube

algorithm

Volumetric model Surface model

Texture mapping

13 Research & Innovation

Visual hull computation

• Visual hull computation from image silhouettes

Camera-1

Camera-2

Camera-3

14 Research & Innovation

Visual hull computation

• Visual hull computation from image silhouettes

Camera-1

Camera-2

Camera-3

15 Research & Innovation

Artefacts in Visual hull-based 3D reconstruction

• Fundamental feature: – only parts of the volume that are visible as background in at

least one of the silhouette images are taken out no concavities can be modelled

• Occlusion errors Phantom volumes Reduced by increasing number of cameras

• Approximation errors

• Sampling errors

16 Research & Innovation

Visual hull-based 3D reconstruction

• Sampling errors– Binary volumetric scene representation Quantisation noise

12 cameras6 cameras12 cameras

17 Research & Innovation

Artefacts in dynamic visual hulls

• “Moving edges” as effect of discrete voxel grid

C2

C1

object

reconstruction

(moving) edge

18 Research & Innovation

Improved visual hull computation: Super-sampling

– Sub-divide voxels by LSS levels

– Assign voxel a (pseudo-) continuous value

Camera-1

19 Research & Innovation

Improved visual hull computation

• Gaussian smoothing• (Optional) complexity reduction

20 Research & Innovation

Results 3D model using super-sampling (20000 triangles)

After Gaussian smoothing

Standard visual hull / marching cube

21 Research & Innovation

Results

• Video from ORIGAMI demo production

22 Research & Innovation

Results

• Non-photorealistic rendering

23 Research & Innovation

Re-lighting

Studio illumination map

3D scene model

irradiance

Specular component

Original image

Light-neutral image

24 Research & Innovation

Re-lighting