A Personal Surround Environment:

Projective Display with Correction forDisplay Surface Geometry and Extreme Lens Distortion

Tyler Johnson, Florian Gyarfas, Rick Skarbez, Herman Towles and Henry Fuchs

Department of Computer Science

University of North Carolina at Chapel Hill

IEEE VR– March 14, 2007

A Personal Surround Environment2

Multi-Projector Display

A Personal Surround Environment3

Wide-FOV Projectors

Conventional Projector Fisheye-Lens Projector

Conventional vs. fisheye lens projector

A Personal Surround Environment4

Outline

Distortion due to Non-Planar SurfacesLens Distortion Correction for ProjectorsResultsContributions and Future Work

A Personal Surround Environment5

Non-planar Display Surfaces

P

A Personal Surround Environment6

Geometry Compensationvia Ray Tracing

Display Surface

Viewer

Projector

Projector Image

A Personal Surround Environment7

Classic Remapping Algorithm2-Pass Rendering (Raskar ’98)Pass 1:

Render application scene

Pass 2:Remap Pass-1 Image

Viewer

Projector

A Personal Surround Environment8

2-Pass RenderingP

Pass-1 Pass-2

Display Surface

GeometryPinhole Projection

Pinhole Projection

Assumes no lens

distortion!

A Personal Surround Environment9

Distortion with Wide-FOV Lenses

Input Image Projected Result

A Personal Surround Environment10

Previous Applications ofWide-FOV Projection

Elumens VisionStation

Hemispherical display screen with fisheye-lens projector for immersive display.Static projector/display surface relationship.Distortion correction based on optical/mechanical design specifications.

A Personal Surround Environment11

Previous Applications ofWide-FOV Projection

Konieczny ’05, “A Handheld Flexible Display System”

Allows exploration of volume data using a deformable hand-held screen.Dynamic display surface.

A Personal Surround Environment12

Our Goals

Our goals were to providecamera-based calibration for wide-FOV projectorsgeometric correction for arbitrary display surfacessupport for tracked viewers

A Personal Surround Environment13

3-Pass Rendering

Pass-1 Pass-2

Display Surface

GeometryPinhole Projection

Pinhole Projection

Lens Distortion Correction

A Personal Surround Environment14

3-Pass Rendering Issuesfor Wide-FOV Lenses

Fisheye lenses are highly non-linearPinhole approximation in Pass-2 leads to sampling issues during lens distortion correction in Pass-3.

A Personal Surround Environment15

Sampling Artifacts

Display Surface

P

3-Pass Rendering

All Passes at Projector Resolution (1024 x 768)

A Personal Surround Environment16

Pass-3 Sampling Issues

Fisheye and pinhole lenses have very different spatial sampling characteristics

Leads to sampling issues during lens distortion correction.

Fisheye lens

(output)

Pinhole lens

(input)

Under-sampled! Over-

sampled!

Over-sampled!

A Personal Surround Environment17

Sampling Artifact Reduction

Pass 1-2: 4x Projector Resolution (4096x3072)

Pass 3: (1024x768)

P

Display Surface

A Personal Surround Environment18

A New Approach

Remove the pinhole approximation of the wide-FOV lens.

Use the actual lens model of the projector.

Allows simultaneous compensation for lens distortion and non-planar surface geometry.

A Personal Surround Environment19

Changes to Correction Pipeline

Pass-1 Pass-2

Display Surface

GeometryPinhole Projection

Pass-3

Pinhole Projection

Lens Distortion Correction

Projector Lens Model

Use actual projector lens model to avoid

resampling artifacts!

A Personal Surround Environment20

A New Approach

Display Surface

Viewer

Projector

Pre-compute where projector rays intersect

the display surface!

Projector Image

A Personal Surround Environment21

Implementation

Pre-compute the projector to display surface (ray) mapping

Store this projector-resolution, 2D to 3D map on GPU as a floating-point texture.Static map - independent of viewer location.

New Pixel ShaderAt each projector pixel:• Look up <x,y,z> vertex on the display surface.• Project the vertex into ideal image to obtain

output color.

A Personal Surround Environment22

Results

All Passes at Projector Resolution (1024x768)

Display Surface

P

A Personal Surround Environment23

Calibration

Capture structured light patterns with camera-pair

Calibrate projector• Linear calibration followed by

non-linear optimization incorporating projector lens model

Reconstruct surface geometry• RANSAC-based plane fitting

technique [Quirk, EDT 06]• General reconstruction

techniques can also be used

A Personal Surround Environment24

Viewer Tracking

Could apply simple 2D-2D mappingApplies only to stationary viewer!

Our approach is independent of the viewer location

Viewer location can be updated each frame

Pass-1 Combined Pass-2 & -3 Remapping

2D - Mapping

Only applicable for Stationary

Viewer

A Personal Surround Environment25

Heterogeneous 2-Projector Display

Also applicable to conventional projectors

Hybrid Display with Conventional and Fisheye-lens

Projectors

Fisheye-lens

Projector

Display Surface

Conventional

Projector

A Personal Surround Environment26

Results

Video

A Personal Surround Environment27

Contributions

Single-pass geometric remapping algorithmCorrecting for both extreme lens distortion and display surface geometryMinimizes sampling and FOV issues with fisheye lenses

Simple GPU shader program implementationStatic 2D:3D mapping stored as 3D floating point texture

Demonstrated applicabilityPersonal surround environment with a single fisheye lens projectorConventional or Wide-FOV fisheye AND Heterogeneous displays

Applicable to real-time VR applications with user-head tracking

A Personal Surround Environment28

Future Work

Methods to further reduce sampling artifacts

Mip-mapped Pass-1 textures for geometry optimized sampling.

Application of more general lens models[Kannala, IEEE TPAMI, Aug. 06]

Real-time, continuous calibrationAccommodate projector or display surface changes while the application running.Dynamic vs. static Pass-2 re-map function.

A Personal Surround Environment29

Thank You

Funding support: ONR N00014-03-1-0589Funding support: ONR N00014-03-1-0589DARWARS Training Superiority program DARWARS Training Superiority program

VIRTE – Virtual Technologies and Environments programVIRTE – Virtual Technologies and Environments program

Elumenati LLC for loan of fisheye-lens projectorElumenati LLC for loan of fisheye-lens projector