1

Perception, Illusion and VR

HNRS 299, Spring 2008Lecture 8

Seeing Depth

2

How can we see in depth?

Problem: When light is focused on the retina to form an image, we go from 3 dimensions in the world to 2 dimensions on the image. We have lost the depth dimension.

Question: How, then, can we see depth?

Answers:Monocular cues to depth: from texture, motion, shading, etc.

Binocular stereopsis: This uses the slightly different images on the two retinae to compute relative depth.

3

Monocular cues to Depth When we view a scene with one eye, we still interpret the 3D scene fairly accurately.

The brain uses cues to the distance to estimate how far away things are. Some of these are listed here:

•Height in the visual field•Size of familiar objects (and relative object size)•Texture gradients•Linear perspective•Shading and shadows•Relative Motion (motion parallax)

4

Height in the Visual Field

In normal images, things that are farther away often have images that are higher in the visual field.

The visual system uses this height in the visual field as a cue to distance.

We then adjust our interpretation of the size of the object based on this distance.

(Note: familiar object size also gives a cue to depth here).

5

Relative object size•Recall that with perspective projection, the image size of a nearby object is larger than for a distant object.

•Therefore, 2 similar objects of different sizes may be perceived at different distances.

•Demo: http://psych.hanover.edu/krantz/art/rel_size.html

•This cue works better when the objects are familiar and have known sizes (see previous slide).

6

Texture Gradients•When we view a textured surface, the nearby texture is larger and coarser and the more distant texture is more fine grained. The texture forms a gradient from course to fine.•This is a consequence of perspective projection.

Demo: http://psych.hanover.edu/krantz/art/texture.html

7

Linear PerspectiveBecause the size of images decreases with distance, the gap between 2 parallel lines will become smaller as the lines recede into the distance. This is known as linear perspective.

Our visual system uses this convergence of lines as a cue to distance.

Demo: http://psych.hanover.edu/krantz/art/linear.html

8

Shape from ShadingShading in an image can give cues to the 3D shape of an object.The brain appears to assume the light source is above the scene.

In these images, when the shadow is on the bottom, we see bumps.When it is on the top, we see indentations.

9

Motion ParallaxAs we learned in the motion lecture, for a moving observer, the images of nearby objects move faster than the images of distant objects.

The difference in image speed, known as motion parallax, can signal depth differences.

Demo: http://psych.hanover.edu/krantz/MotionParallax/MotionParallax.html

Structure from motion: Relative motion in the image can also be used to determine the 3D shape of objects.

Demo: http://www.viperlib.org/(General depth images, page 7, rotating cylinder).

10

Moving Shadows

Combining motion and shadows can lead to some interesting perceptions of motion in depth.

Demo: http://www.viperlib.org/General Depth Images, page 1.

The only difference in these images is the position of the shadow.

11

Binocular Stereo

When viewed with a red filter over the left eye and a cyan filter over the right eye, this image shows a 3 dimensional view of Mars.

The fact that our two eyes see the world from different positions allows the brain to use differences in the image to compute depth. This is known as stereopsis or stereo vision.

A 3D Mars Rover

12

Binocular Stereo

•The image in each of our two eyes is slightly different.

•Images in the plane of fixation fall on corresponding locations on the retina.

•Images in front of the plane of fixation are shifted outward on each retina. They have crossed disparity.

•Images behind the plane of fixation are shifted inward on the retina. They have uncrossed disparity.

13

Crossed and uncrossed disparity

crossed (positive) disparity

uncrossed (negative) disparity

plane of fixation

1

2

Disparity is the amount that the image locations of a given feature differ in the two eyes.

14

Stereo processingTo determine depth from stereo disparity:

1) Extract the "features" from the left and right images

2) For each feature in the left image, find the corresponding feature in the right image.

3) Measure the disparity between the two images of the feature.

4) Use the disparity to compute the 3D location of the feature.

15

The Correspondence problem•How do you determine which features from one image match features in the other image? (This problem is known as the correspondence problem).

•This could be accomplished if each image has well defined shapes or colors that can be matched.

•Problem: Random dot stereograms.

Left Image Right Image Making a stereogram

16

Random Dot Stereogram

17

Problem with Random Dot Stereograms

•In 1980's Bela Julesz developed the random dot stereogram.

•The stereogram consists of 2 fields of random dots, identical except for a region in one of the images in which the dots are shifted by a small amount.

•When one image is viewed by the left eye and the other by the right eye, the shifted region is seen at a different depth.

•No cues such as color, shape, texture, shading, etc. to use for matching.

•How does the brain know which dot from left image matches which dot from the right image?

•Auto-stereograms (magic eye) are versions of random dot stereograms.

18

Some neurons are sensitive to disparity

Some neurons in V1 and MT are sensitive to the disparity of a stimulus. In V1, most neurons prefer a disparity near zero.

Tuned Excitatory Tuned Inhibitory

19

Near and Far cells

Some cells are broadly tuned for disparity, preferring either near objects or far objects.

20

Testing for Disparity tuningMovie: http://www.viperlib.org/(General depth images, page 5)

Torsten Wiesel tests a cell for sensitivity to depth.