Image-Based Rendering from a Single Image

Kim Sang Hoon

Samuel Boivin – Andre Gagalowicz

INRIA

Introduction

To recover an approximation of BRDF of surface from a single image. (including specular, isotropic or anisotropic surfaces)

Hierarchical , interactive technique using error between the rendered image and the real image.

Background

Camera parameter (3D geometrical model, 2D image)[DeMenthon and Davis] Model-based object pose in 25 line of code – ECCV 92

BRDF Model:[Ward] Measuring and modeling anisotropic reflection – SIGGRAPH 92

Radiance map:[Tumblin and Rushmeier] Tone reproduction for realistic image – IEEE Computer Graphics and Applications November 1993.

How to find light pose from a single image

Note: Images record color, Radiance maps record brightness

Background and Related Work

1. Reflectance Recovery using a Specific Device.- Estimate the five parameters of anisotropic BRDF model. [Ward] Measuring and modeling anisotropic reflectance – SIGGRAPH 92

2. Reflectance Recovery from Several Images.- Method without Global Illumination.- Method with Global Illumination.

3. Reflectance Recovery from a Single Image.- Method without Global Illumination.- Method with Global Illumination.

- Radiosity-based Algorithm

Elements of Reflectance Recovery

Notion of Group

- Input : 3D geometrical model, a single image- Extraction of the object reflectance from the pixel. (by the projection of these objects in the image)- Problems (using a single image) -A lot of surfaces are not visible.

- Notion of Group- The object and the surface have a same reflectance property- Manual operation (Geometrical modeling process)

Elements of Reflectance Recovery

Reflectance Model and Data Description

- Image-Based Modeling (Alias | Wavefront’s Maya modeler)- Camera parameters [DeMenthon and Davis] Model-based object pose in 25 line of code – ECCV 92

- Photometric recovery method[Ward] Measuring and modeling anisotropic reflection – SIGGRAPH 92Five parameters for a complex BRDF: Diffuse ρd, Specular ρd, anisoptropic direction (x), anisotropic roughness (ax, ay)

Elements of Reflectance Recovery

3D Geometrical Modeling

Geometrical model - object, camera, light sources posesPhothmetric model - reflectance, light source intensity

Synthetic imageusing a classical rendering Software (Phoenix – global illumination software)

Overview

of the

Algorith

m

Inverse Rendering from a Single Image

Case of perfect diffuse surface

- Diffuse reflectance : the average of radiances covered by the projection of the group in the original image.

- Textured surface (using a pure diffuse) : Create a good visual approximation.

Inverse Rendering from a Single Image

Case of perfect diffuse surface

Error between the original and the rendered image

))(1(

))(1(

)(

)(ˆ

j

j

j

j

new

org

new

orgj

PTaver

PTaver

Baver

Baver

Where, B : average radiance P : pixels covered by the projection of object j in the

original image. T( ) is camera transfer function ( - correction function)

• Camera transfer function : To convert light input into electrical (analog or digital) signals.

Inverse Rendering from a Single Image

Diffuse reflectance ρd of object j is proportional to the average radiance B

The function f () eliminates problems by smaller object.Textures are not take into account – only consider a diffuse reflectance parameter ρd .

The radiances, the emittances and the full geometry (form factors) Solve radiosity equation for the reflectance.

Inverse Rendering from a Single Image

Case of perfect and non-perfect specular surface

Diffuse hypothesis failed considered as a perfect mirror.Perfect specular surface

-The easiest case to solve (ρd =0, ρs = 1), Need not iterationNon-perfect specular surface

- Require iteration to obtain an optimum ρs

Inverse Rendering from a Single Image

Case of both diffuse and specular surfaces with no roughness

Inverse Rendering from a Single Image

Case of isotropic surfaces

Recover Diffuse (ρd), Specular (ρd) and roughness (a) using Ward’s BRDF model.

Case of anisotropic surfaces

- Most complicate case- Anisotropic model of Ward requires to minimize a function of 5 parameter

s. (Diffuse ρd, Specular ρd, anisoptropic direction (x), anisotropic roughness (ax, ay)

Inverse Rendering from a Single Image

Case of textured surfaces

- Extracting the texture from image is an easy task

[Wolberg] Digital Image Warping – IEEE Computer Society Press

- Consider that it already has received the energy from the light source. Otherwise, over-illuminated.- Radiosity texture : balances the extracted texture with an intermediate textur

e in order to minimize the error (the real and synthetic image)

- Case of perfect diffuse surfaceTexture is computed by an iterative method.- At the first, extract from the real image.- Synthetic image- Multiplied by the ratio (newly texture of synthetic / texture of real image)

Inverse Rendering from a Single Image

Case of textured surfaces

- The problems

- A texture including the shadows, the specular reflection and the highlight.

- It is extremely hard to solve using a single image.

Results

Results

Future Works

[Debevec] Efficient View-Dependent Image-Based Rendering with ProjectiveTexture-Mapping – EGRW 98

[Debevec] Rendering Synthetic Objects into Real Scenes - SIGGRAPH 98

[Debevec] Modeling and Rendering Architecture from Photographs –SIGGRAPH96

[Ward] Measuring and modeling anisotropic reflection – SIGGRAPH 92

Fixed Camera – Obtain multiple images (Different exposure, light poses)

1. Assume perfect diffuse surface – extract texture (iterative method) –Render image.

2. Find radiance map – Estimate BRDF – Render image.