Seeing 3D from 2D Seeing 3D from 2D ImagesImages

How to make a 2D How to make a 2D image appear as image appear as

3D!3D!►Output and input is Output and input is

typically 2D Imagestypically 2D Images► Yet we want to Yet we want to

show a 3D world!show a 3D world!►How can we do this?How can we do this?

We can include We can include ‘cues’ in the image ‘cues’ in the image that give our brain that give our brain 3D information about 3D information about the scenethe scene

These cues are These cues are visual depth cuesvisual depth cues

Visual Depth CuesVisual Depth Cues

► Cues about the 3Cues about the 3rdrd dimension – total of 10dimension – total of 10

► Monoscopic Depth Cues Monoscopic Depth Cues (single 2D image) [6](single 2D image) [6]

► Stereoscopic Depth Stereoscopic Depth Cues (two 2D images) Cues (two 2D images) [1][1]

► Motion Depth Cues Motion Depth Cues (series of 2D images) [1](series of 2D images) [1]

► Physiological Depth Physiological Depth Cues (body cues) [2]Cues (body cues) [2] Hold a finger upHold a finger up

Monoscopic Depth CuesMonoscopic Depth Cues► InterpositionInterposition

An occluding object is closerAn occluding object is closer► ShadingShading

Shape and shadowsShape and shadows► SizeSize

The larger object is closerThe larger object is closer► Linear PerspectiveLinear Perspective

Parallel lines converge at a Parallel lines converge at a single pointsingle point

Higher the object is Higher the object is (vertically), the further it is(vertically), the further it is

► Surface Texture GradientSurface Texture Gradient More detail for closer More detail for closer

objectsobjects► Atmospheric effects Atmospheric effects

Further away objects are Further away objects are blurrier and dimmerblurrier and dimmer

► Images from Images from http://ccrs.nrcan.gc.ca/resource/tutor/stereo/chap2/chapter2_5_e.phphttp://ccrs.nrcan.gc.ca/resource/tutor/stereo/chap2/chapter2_5_e.php

Monoscopic Depth CuesMonoscopic Depth Cues► InterpositionInterposition

An object that occludes another is An object that occludes another is closercloser

► ShadingShading Shape info. Shadows are included Shape info. Shadows are included

herehere► SizeSize

Usually, the larger object is closerUsually, the larger object is closer► Linear PerspectiveLinear Perspective

parallel lines converge at a single parallel lines converge at a single pointpoint

► Surface Texture GradientSurface Texture Gradient more detail for closer objectsmore detail for closer objects

► Height in the visual fieldHeight in the visual field Higher the object is (vertically), the Higher the object is (vertically), the

further it isfurther it is► Atmospheric effects Atmospheric effects

further away objects are blurrierfurther away objects are blurrier► BrightnessBrightness

further away objects are dimmerfurther away objects are dimmer

Stereoscopic Display IssuesStereoscopic Display Issues

► StereopsisStereopsis► Stereoscopic Display Stereoscopic Display

TechnologyTechnology► Computing Computing

Stereoscopic ImagesStereoscopic Images► Stereoscopic Display Stereoscopic Display

and HTDs.and HTDs.►Works for objects < Works for objects <

5m. 5m. Why?Why?

StereopsisStereopsisThe result of the two slightly different views of the world that our laterally-displaced eyes receive.

Retinal DisparityRetinal Disparity

f1f2

Left Eye Right Eye

Retinal disparity =

If both eyes are fixated If both eyes are fixated on a point, on a point, ff11, in space:, in space: Image of Image of ff11 is focused at is focused at

corresponding points in corresponding points in the center of the the center of the foveafovea of each eye. of each eye.

ff22, would be imaged at , would be imaged at points in each eye that points in each eye that may at different may at different distances from the distances from the foveafovea..

This difference in This difference in distance is the retinal distance is the retinal disparity.disparity.

Retinal DisparityRetinal Disparity► If an object is farther than If an object is farther than

the fixation point, the retinal the fixation point, the retinal disparity will be:disparity will be:

Positive valuePositive value Uncrossed disparityUncrossed disparity Eyes must uncross to fixate Eyes must uncross to fixate

the farther object.the farther object.► If an object is closer than If an object is closer than

the fixation point, the retinal the fixation point, the retinal disparity will be:disparity will be:

Negative Negative Crossed disparityCrossed disparity Eyes must cross to fixate Eyes must cross to fixate

the closer object.the closer object.► An object located at the An object located at the

fixation point or whose fixation point or whose image falls on image falls on corresponding points in the corresponding points in the two retinae has:two retinae has:

Zero disparity (in focus)Zero disparity (in focus)► Question: What does this Question: What does this

mean for rendering mean for rendering systems?systems?

f1f2

Left Eye Right Eye

Retinal disparity =

Convergence AnglesConvergence Angles

i

f2

f1

D1

D2a b

c d

1

+a+c+b+d = 180+a+c+b+d = 180

+c+d = 180+c+d = 180

-- = a+(-b) = = a+(-b) = 1+1+2 = Retinal 2 = Retinal DisparityDisparity

2

Miscellaneous Eye FactsMiscellaneous Eye Facts

► StereoacuityStereoacuity - the - the smallest depth that smallest depth that can be detected can be detected based on retinal based on retinal disparity.disparity.

►Visual DirectionVisual Direction - - Perceived spatial Perceived spatial location of an object location of an object relative to an relative to an observer.observer.

HoroptersHoropters► Map out what points Map out what points

would appear at the would appear at the same retinal disparity.same retinal disparity.

► Horopter - the locus of Horopter - the locus of points in space that fall points in space that fall on corresponding points on corresponding points in the two retinae when in the two retinae when the two eyes binocularly the two eyes binocularly fixate on a given point in fixate on a given point in space (zero disparity).space (zero disparity).

► Points on the horopter Points on the horopter appear at the same appear at the same depth as the fixation depth as the fixation point. (can’t use point. (can’t use stereopsis.stereopsis.

► What is the shape of a What is the shape of a horopter?horopter?

Vieth-Mueller Circle

f1

f2

Stereoscopic DisplayStereoscopic Display

►Stereoscopic images are easy to do Stereoscopic images are easy to do badly, hard to do well, and impossible badly, hard to do well, and impossible

to do correctly.to do correctly.

Stereoscopic Stereoscopic DisplaysDisplays

► Stereoscopic display Stereoscopic display systems presents systems presents each eye with a each eye with a slightly different slightly different view of a scene.view of a scene. Time-parallel – 2 Time-parallel – 2

images same timeimages same time Time-multiplexed – 2 Time-multiplexed – 2

images one right images one right after anotherafter another

Time Parallel Stereoscopic Time Parallel Stereoscopic DisplayDisplay

Two ScreensTwo Screens► Each eye sees a Each eye sees a

different screendifferent screen► Optical system directs Optical system directs

correct viewcorrect view► HMD stereoHMD stereo

Single ScreenSingle Screen► Two different images Two different images

projectedprojected► Images are polarized at Images are polarized at

right anglesright angles► User wears polarized User wears polarized

glassesglasses

Passive Passive Polarized Polarized ProjectionProjection

► Linear PolarizationLinear Polarization Ghosting increases Ghosting increases

when you tilt headwhen you tilt head Reduces brightness of Reduces brightness of

image by about ½image by about ½ Potential Problems Potential Problems

with Multiple Screenswith Multiple Screens

► Circular PolarizationCircular Polarization Reduces ghosting Reduces ghosting Reduces brightness Reduces brightness Reduces crispnessReduces crispness

Problem with Linear Problem with Linear PolarizationPolarization

► With linear polarization, With linear polarization, the separation of the the separation of the left and right eye left and right eye images is dependent on images is dependent on the orientation of the the orientation of the glasses with respect to glasses with respect to the projected image. the projected image.

► The floor image cannot The floor image cannot be aligned with both be aligned with both the side screens and the side screens and the front screens at the the front screens at the same time. same time.

Time Multiplexed DisplayTime Multiplexed Display

► Left and right-eye Left and right-eye views of an image views of an image are computed are computed

► Alternately displayed Alternately displayed on the screenon the screen

► A shuttering system A shuttering system occludes the right occludes the right eye when the left-eye when the left-eye image is being eye image is being displayeddisplayed

Stereographics Shutter Stereographics Shutter GlassesGlasses

Screen ParallaxScreen Parallax

Pleft – Point P projected screen location as seen by left eyePright – Point P projected screen location as seen by right eye

Screen parallax - distance between Pleft and Pright

P

Left eye position

Right eye position

Pleft

Pright

Pright

Pleft

P

Display Screen

Object with positive parallax

Object with negative parallax

Screen Parallax Screen Parallax (cont.)(cont.)

f1

p

id

Left eyepoint

Right eyepoint

Projection

Plane

D

p = i(D-d)/Dp = i(D-d)/D

where p is the amount of screen where p is the amount of screen parallax for a point, f1, when parallax for a point, f1, when projected onto a plane a projected onto a plane a distance d from the plane distance d from the plane containing two eyepoints. containing two eyepoints.

i is the interocular distance i is the interocular distance between eyepoints and between eyepoints and

D is the distance from f1 to D is the distance from f1 to the nearest point on the the nearest point on the plane containing the two plane containing the two eyepointseyepoints

d is the distance from the d is the distance from the eyepoint to the nearest point eyepoint to the nearest point on the screenon the screen

How to create correct left- How to create correct left- and right-eye viewsand right-eye views

► What do you What do you need to specify need to specify for most for most rendering rendering engines?engines? EyepointEyepoint Look-at PointLook-at Point Field-of-View Field-of-View

or location of or location of Projection Projection PlanePlane

View Up View Up DirectionDirection

P

Left eye position

Right eye position

Pleft

Pright

Pright

Pleft

P

Display Screen

Object with positive parallax

Object with negative parallax

Basic Perspective Projection Basic Perspective Projection Set Up from Viewing Set Up from Viewing

ParamentersParamentersY

Z

X

Projection Plane is orthogonal to one of the major axes (usually Z). That axis is along the vector defined by the eyepoint and the look-at point.

What doesn’t workWhat doesn’t work

•Each view has a different projection plane

•Each view will be presented (usually) on the same plane

What Does WorkWhat Does Work

i i

Setting Up Projection Setting Up Projection GeometryGeometry

Look at pointEyeLocations

Look at points

EyeLocations

No

Yes

Visual Angle SubtendedVisual Angle Subtended

Screen parallax is measured in terms of visual angle. This is a screenindependent measure. Studies have shown that the maximum anglethat a non-trained person can usually fuse into a 3D image is about

1.6 degrees. This is about 1/2 the maximum amount of retinal disparityyou would get for a real scene.

Accommodation/ Accommodation/ ConvergenceConvergence

Display Screen

Position Dependence Position Dependence (without head-tracking)(without head-tracking)

Interocular DependanceInterocular Dependance

F

Modeled Point

Perceived Point

Projection Plane

True EyesModeled Eyes

Obvious Things to DoObvious Things to Do

►Head trackingHead tracking►Measure User’s Interocular DistanceMeasure User’s Interocular Distance

Another ProblemAnother Problem

►Many people can not fuse stereoscopic Many people can not fuse stereoscopic images if you compute the images images if you compute the images with proper eye separation!with proper eye separation!

►Rule of ThumbRule of Thumb: Compute with about ½ : Compute with about ½ the real eye separation.the real eye separation.

►Works fine with HMDs but causes Works fine with HMDs but causes image stability problems with HTDs image stability problems with HTDs (why?)(why?)

Two View Points with Head-TrackingTwo View Points with Head-Tracking

Projection Plane

Modeled Point

Perceived Points

Modeled Eyes

True Eyes

GhostingGhosting

►Affected by the amount of light Affected by the amount of light transmitted by the LC shutter in its off transmitted by the LC shutter in its off state.state.

►Phosphor persistencePhosphor persistence►Vertical screen position of the image.Vertical screen position of the image.

Time-parallel stereoscopic Time-parallel stereoscopic imagesimages

► Image quality may also be affected byImage quality may also be affected by Right and left-eye images do not match in Right and left-eye images do not match in

color, size, vertical alignment.color, size, vertical alignment. Distortion caused by the optical systemDistortion caused by the optical system ResolutionResolution HMDs interocular settingsHMDs interocular settings Computational model does not match Computational model does not match

viewing geometry.viewing geometry.

Motion Depth CuesMotion Depth Cues

► Parallax created by relative head position Parallax created by relative head position and object being viewed.and object being viewed.

►Objects nearer to the eye move a greater Objects nearer to the eye move a greater distancedistance

► (Play pulfrich video without sunglasses)(Play pulfrich video without sunglasses)

Physiological Physiological Depth CuesDepth Cues

► Accommodation Accommodation – – focusing adjustment focusing adjustment made by the eye to made by the eye to change the change the shapeshape of of the lens. (up to 3 m)the lens. (up to 3 m)

► ConvergenceConvergence – – movement of the eyes movement of the eyes to bring in the an to bring in the an object into the same object into the same location on the retina location on the retina of each eye.of each eye.

SummarySummary

►Monoscopic – Interposition is strongest.Monoscopic – Interposition is strongest.►Stereopsis is very strong.Stereopsis is very strong.►Relative Motion is also very strong (or Relative Motion is also very strong (or

stronger).stronger).►Physiological is weakest (we don’t even Physiological is weakest (we don’t even

use them in VR!)use them in VR!)►Add as neededAdd as needed

ex. shadows and cartoonsex. shadows and cartoons

Pulfrich EffectPulfrich Effect

►Neat trickNeat trick►Different levels of illumination require Different levels of illumination require

additional time (your frame rates differ additional time (your frame rates differ base of amount of light)base of amount of light)

►What if we darken one image, and What if we darken one image, and brighten another?brighten another?

►http://dogfeathers.com/java/pulfrich.html

►www.cise.ufl.edu/~lok/multimedia/www.cise.ufl.edu/~lok/multimedia/videos/pulfrich.avivideos/pulfrich.avi