1

Advanced Topics in Virtual Advanced Topics in Virtual RealityRealityTae Soo Yun

Dept. of Digital Contents

Dongseo University

Fall 2002

based on notes from

Soon Ki Jung, KNU

Wohn, KAIST

……

2

Table of ContentsTable of Contents

1. Introduction : What is VR ?2. Psychological and Cognitive Issues3. VR System Anatomy4. Virtual Perception5. Interaction6. Virtual Worlds: Representation, Creation and Sim

ulation7. Virtual Worlds: Rendering8. Networked VR Systems and Shared Virtual Envir

onements9. Image-based Virtual Reality10.Augmented Reality

3

H-sensor

perception

cognition

motion control

H-effector

Conceptual Model of VRConceptual Model of VR

V-sensor

V-effector

P-effector

P-sensor

L-effector

L-sensor

output device

input device

sensing

action

joystickmousetrackerTV cameramicrophone

2D - selector pick locator etc.3D - navigator manipulator etc.

V-humanV-vehicleV-bikeV-handetc.

sensing

action

HumanVirtual environment

Basic model of perception & Basic model of perception & cognitioncognition

H-sensor

perception

cognition

motion control

H-effectoraction

Human Natural environment

sensing

knowledge

2-1. Human Visual System2-1. Human Visual System

2-2. Human Auditory System

2-3. Other Perceptual Systems2-3-1. Tactile

2-3-2. Vestibular

2-3-3. Kinesthetic

2-3-4. Olfactory

2-3-5. Gustatory

2-4. Human Motor System

2-5. Cognitive System

2-6. Presence and Reality2-6. Presence and Reality

6

2-1. Human Visual System2-1. Human Visual System

1. Outline of the Visual System

2. Eye

3. Properties of Eye

4. Properties of Visual System

5. Spatial Perception

7

1. Outline of the Visual System1. Outline of the Visual System

Properties Bandwidth Sensitivity

Visual Path Eye Retina Optic nerve Lateral Geniculate Nucleus Visual cortex

8

2. Eye2. Eye

Cornea ( 각막 ) Iris ( 조리개 ) – pupil opening 제어

Pupil opening ( 동공 ) 지름 2-8 mm, 통과하는 빛의 양 결정 렌즈의 지름이 변하므로 depth-of field 가 변함

Lens ( 수정체 ) 렌즈의 두께를 변화시킴으로써

focal length 변화 비교적 느림 (400 msec from near

vision to far vision)

Retina ( 망막 ) 약 10 neural layers

9

receptor

10

3. Properties of Eye System3. Properties of Eye System

Accomodation and depth of fieldAccomodation and depth of field Thresholding intensity Spectral sensitivity Dynamic range Dark adaptation Visual fieldVisual field Visual acuityVisual acuity Spectral response Eye movements

11

Properties of Eye System (cont.)Properties of Eye System (cont.)

accomodationadjust the focus by changing the lens curvature.Eye measure the amount of “blur”.Accomodation does not occur in the graphic scene.

Eye 의 근육이 이완되어 있는 상태에서는 depth of field 가 6 m ~ 무한대이므로 다른 노력 없이 in-focus 가능 . 동공의 지름 = 8 mm: depth of field = 6m – 무한대 동공의 지름 = 4 mm: depth of field = 3.5m – 무한대 동공의 지름 = 2 mm: depth of field = 2.3m - 무한대

물체가 6m 이내에 들어오면 focus 를 맞추기 위해 근육을 수축하여 렌즈의 곡률을 변화시킴 .

물체까지의 거리가 1m 일 때 : 동공의 지름 = 4 mm: depth of field = 0.8m - 1.4m 동공의 지름 = 2 mm: depth of field = 0.7m - 1.8m

12

Properties of Eye System (cont.)Properties of Eye System (cont.)

dynamic range

인간이 sense 하는 intensity 의 dynamic range = 1:10^13 그러나 한 순간에서의 operating range = 1: 100 Discriminating power = 1%

dark adaptation

어두운 환경에 갑자기 들어 갈 때 감지 할 수 있는 빛의 최소량은 시간에 따라 감소 .

Cone -- 처음 7-8 분간 빨리 반응 . Rod -- Adaptation 은 느리나 더 적은 빛에서도 반응 . Cone 보다 약 100 배

sensitive 함 . 따라서 , 어두운 곳에서는

• Color perception 이 잘 안됨• 붉은 색보다는 푸른색이 눈에 잘 띔• Resolution 이 떨어짐

13

Properties of Eye System (cont.)Properties of Eye System (cont.)

visual field visual angle visual field

• one eye: 150 degrees horizontal, 120 degrees vertical

• two eyes: 200 degrees 120

• overlapping field: +/- 30 ~ 40 degrees

14

s

d

n r

Nodal point

Visual Angle

)d/sarctan(

Properties of Eye System (cont.)

15

Visual field of a normal human observer.

Properties of Eye System (cont.)

16

visual acuity ability to identify / discriminate features.

• 20/20 vision = -      정상인이 20 ft 떨어진 곳에서 식별할 수 있는 정도를 피실험자도 20 ft거리에서 식별 .

• 20/100 vision = 정상인의 100ft에서의 식별 능력과 피실험자의 20ft에서의 식별 능력이 같음 .

1) recognition acuity– Feature 를 식별할 수 있는 능력– 최대 1' (100/100 vision)

2) detection acuity– Single line 을 detect 할 수 있는 능력– 최대 2'

3) resolution acuity– Double lines 을 detect 할 수 있는 능력– 최대 30’’

17

spatial resolution obtained from visual acuity 4,800 x 3,800

Spatial frequency response Sine grating 의 frequency 를 바꿔가며 인간이

감지할 수 있는 최소의 contrast ( 즉 , sine grating 의 amplitude) 를 측정한다 .

최대의 sensitivity 는 4 - 5 Hz 에서 발생 .

18

Eye MovementsEye Movements

1) Nystagmus ( 안진 )

• Very fast (30 - 100 Hz) , Very small ( 약 17")2) Saccades

• Drift 를 보정하는 운동 , 또한 target 이 움직일 때 추적하는 운동 .

• 초당 3 - 5 회 발생 .1 초에 100msec 은 saccade 운동 , 900msec 은 정지 .

• Movement 중에는 image capture 하지 않음 .• Very fast (40° in 100 msec)

3) Drift

• Fixation point 가 target 으로부터 조금씩 벗어나는 현상 .• Slow (3'/sec)

4) Pursuit move

• 자발적인 운동으로서 물체가 움직일 때 계속 추적하는 운동 . Eyeball 의 움직임 range

• Yaw: ±50° , Pitch: 40° above, 60° below. , Roll: 6°

19

4. Properties of Visual System4. Properties of Visual System

Feature detection Edge detection Corner detection

Motion Short-term motion -- Detection

• up to 10’ –15’visual arc,up to 80 – 100 msec interval Long-term motion -- Matching Apparent motion = perception of motion without stimulus c

ontinuity

Color perception Physical stimuli vs. Color perception

20

5. Spatial Perception5. Spatial Perception

Non-pictorial depth cues Accomodation

• 유효거리 : 2-3m StereopsisStereopsis

• convergenceconvergence• disparitydisparity

Motion parallax

Pictorial depth cues size consistency perspective arial perspective luminance shading light and shadow color texture occlusion

21

Stereopsis (binocular vision)= the process that “integrate two images into one perception” binocular rivalry

• Visual system suppresses one image when two images are very different

• Caused by the luminance difference and others• Human has the dominant eye

binocular convergence• 약 6m – 9m 까지 유효

binocular disparity• Disparity 가 1° 이하일 때는 상당히 정확한 식별 가능 .• 50 cm 거리에서는 0.05 mm 분별 ,5 m 거리에서는 4 mm

분별• Disparity 가 1° 이상일 때는 상대적인 거리만 판별 .• Disparity 가 5° 이상이 되면 matching 불능

22

MotionMotion

motion parallax 약 50m 까지 유효

depth from motion Kinetic depth effect Stereo-kinetic depth illusion

23Fixated rod Disparate rod

Left EyeRight Eye

Foveas

Disparateimages offar rod

Fixation point

24

2-2. Human Auditory System2-2. Human Auditory System

1. Characteristics of SoundSound wave = the change of atmospheric pressureAtmospheric pressure = 10^6 barDetectable range of ear = 0.0002 bar - 200 bar

2. Anatomy of Auditory System outer, middle, and inner ears.

3. Psycho-acoustics: Auditory Perception physical -- sensitivity, discriminability psychophysical -- loudness, pitch, timbre precedence effect, cocktail party effect

4. Spatial Hearing

25

26

Sound localization Sound localization

azimuth, elevation, distance

Azimuth perception is due to … Time difference Intensity difference Monaural cues derived from the pinna( 귓바퀴 ) Head/sound movements

Interaural time difference Max time difference = 660 msec

(time for reaching from one ear to the other) Min discriminating time = 10 sec Effective for low frequency range

27

Sound localization (cont.)Sound localization (cont.)

Interaural intensity difference Max intensity difference = 20 dB (due to head occlusion) Min discernable intensity difference = 0.7 dB Effective for high frequency range

Cues derived from the outer ear (especially pinna( 귓바퀴 )). HRTF (Head related transfer function)

Distance perception is due to … Intensity cues Spectral changes Auditory parallax

28

2-3. Other Perceptual Systems2-3. Other Perceptual Systems

1. Tactile sense of touch, pain, cold, warmth. Five different skin receptors are responsible for the basic

senses

2. Kinesthetic movements and relative positions of body parts.

3. Vestibular sense of acceleration, orientation, rotational motion.

4. Olfactory sense of smell.

5. Gustatory sense of taste.

29

Tactile PerceptionTactile Perception

Sense of touch Basic sensing unit = pressure

Five different skin receptors are responsible for the basic senses: Touch Pain Cold Warmth

Usage How to design the tactile display system ?

30

31

Kinesthetic PerceptionKinesthetic Perception

Kinesthesia = awareness of the movements and relative position of the various body parts Internal sensing of kinematic (position, angle) and dynamic

(speed, force, weight) status Due to the sensory receptors in joints, muscle and skin Joints closer to body (proximal joints) are more sensitive th

an distal joints (such as fingers)

Usage How to design the force feedback system? How to present a virtual hand ?

• Visual perception + kinesthetic perception

32

Vestibular (equilibratory) perceptionVestibular (equilibratory) perception

Vestibular sense = sense of balance and position Acceleration Orientation Rotational motion

Usage How to design the motion platform ?

33

OlfactoryOlfactory

Smell classification ? Not yet!

34

GustatoryGustatory

Basic taste qualities Salty, sour, sweet, bitter

Hey, are any of these related to computer science, even remotely?

2-4. Human Motor Systems2-4. Human Motor Systems

H-sensor

perception

cognition

motion controlmotion control

H-effectorH-effector

V-sensor

V-effector

P-effector

P-sensor

L-effector

L-sensor

output device

input device

sensing

action

joystickmousetrackerTV cameramicrophone

V-humanV-vehicleV-bikeV-handetc.

sensing

action

HumanVirtual environment

36

Theoretical basis for planning, modifying and executing motor movements.

1. Reaction time

2. Motor control movement time as the time for processing information Fitts’ law: movement time = a + b log(2·D/W)

3. Kinematic properties

4. Dynamic properties

37

Reaction timeReaction time

Donder’s experiments Reaction A

• One stimulus -> one response Reaction B

• Multiple stimuli -> multiple responses (e.g., at a traffic light)

Reaction C• Multiple stimuli -> single response (e.g., name calling in

the class)

Reaction time = nervous system conduction time + time for mental identification + time for mental selection

38

Reaction time (cont.)Reaction time (cont.)

Information theoretical approach Reaction time information amount (Hick’s law) Reaction time # of alternatives

Determinants of reaction time Stimulus-response compatibility Practice

Speed-accuracy trade-off

39

Motor controlMotor control

Fitts’ law Movement time = a + b log(2D/W) … is an application of information theory to the problem of

motor control (Motion generates information)

Motor program Program that generates motion by accepting the control pa

rameters (e.g., pianist, athletes)

40

Kinematic propertiesKinematic properties

Eye movement

Head movement 600°/sec for yaw, 300°/sec for pitch and roll

Body movement

Locomotion

41

Dynamic propertiesDynamic properties

Force

Torque

2-5. Cognitive System2-5. Cognitive System

H-sensor

perception

cognitioncognition

motion control

H-effector

V-sensor

V-effector

P-effector

P-sensor

L-effector

L-sensor

output device

input device

sensing

action

joystickmousetrackerTV cameramicrophone

V-humanV-vehicleV-bikeV-handetc.

sensing

action

HumanVirtual environment

43

Why do we care about? Designing interface – human can transmit information only

at a finite rate. Developing the virtual perception.

We are interested in … Capacity Structure

Disciplines on human cognition Philosophy – thinking as a “logical” system Psychology – biologically (or neuro-physiologically)-based

model Human factors – cognition as information processing AI – cognition as symbolic manipulation

44

MemoryMemory

Sensory memory Holds information from eyes and ears Iconic

Short-term memory (working memory) Limited capacity (5-9 items)

Long-term memory Permanent storage No guarantee for accessibility

45

AttentionAttention

Selective attention Ability to focus (or ignore) some events

Divided attention Ability to do more than one thing at the time

46

Mental workloadMental workload

Mental workload vs. physical workload

Measuring the mental workload Subjective opinion: rating scale, questionnaires Spare mental capacity: based on the assumption that

human channels have the upper bound Primary-task method: based on the assumption that

increasing mental load will cause a decrement in primary-task performance

2-6. Presence and Reality2-6. Presence and Reality

1. VR: the medium

2. Tele-presence and Virtual presence

3. AIP cube

4. Model P

5. Measuring reality

6. Philosophical considerations

48

1. VR: The medium1. VR: The medium

The word medium means to be between two ends/extremes.

In the case of human communication, it is the process by which ideas are manifested and then experienced by people.

49

The content of HC medium is..The content of HC medium is..

Examples of virtual worlds implemented in various media: Imagination : Day dream / Mental model Imagination/Toys : Child playing with a doll Storytelling Cave-painting Novel (non-interactive fiction) Maps Illusions (of magic) : Pulling rabbit out of a hat Song Motion Picture (non interactive) Animation Puppetry Ham Radio Interactive-Fiction (via book) Interactive-Fiction (via computer) E-mail Newsgroups (via internet) Multimedia Interactive-Fiction

…….

50

Communicating ideas through a mediumCommunicating ideas through a medium

51

Different physical media …Different physical media …

Can be used for the same HC medium Painting

Oil on canvas Pigment on wall Bits in memory …

Written Word Novel Poem Short Story Textbook …

Virtual Reality HMD w/ Glove CAVE w/ Wand …

52

Filling in some detailsFilling in some details

53

Many paths from a composition Many paths from a composition to an experienceto an experience

54

Feedback to a performanceFeedback to a performance

55

Multiple types of feedbackMultiple types of feedback

56

A recorded experience …A recorded experience …

loses the ability for feedback

57

Interface to the Virtual WorldInterface to the Virtual World

All media must provide a means by which the recipient can gain access to the virtual world.

We call this the "User Interface" (though typically that term is thought of only for computer graphics programs).

The goal is (or should be) to find the interface that impedes the flow of information the least.

"The ultimate interface" is considered by some as the prime goal of VR -- the invisible interface.

But such an interface won't take full advantage of all the possibilities of VR, and in many cases may not be the best solution.

Some virtual worlds can be presented through different media. There are often changes that need to be made to the virtual world, and

especially the interface. Often a virtual world was created for a particular medium, taking advantage of

the strengths of that particular medium. Thus a significant part of the participatory experience may be lost when translated to a different medium.

When moving from a non-interactive to an interactive medium, one significant change is that more of the world must be defined.

58

Who is the author of an experience ?Who is the author of an experience ?

Relatively easy to say for a novel (though not absolute)

Becomes more difficult for interactive media We will use the term(s) "creator"/"composer" for

those who create the manifestation of the medium to avoid confusion.

In one sense, the recipient is always involved in authoring the experience.

59

NarrativeNarrative

Most media present an immotive (non-changing) narrative Virtual reality is among the media that provide an interractive n

arrative. The "narratives" of most interactive experiences tend to be bori

ng (For these, it is the interaction itself that is interesting.) Some VR (and other interactive media) have experiences create

d that are meant to tell an interesting story. The former are called "undirected narratives," and the latter "di

rected narratives." An experience with a story with a specific goal has elements that direct the user toward that goal.

60

GenreGenre

As with other media such as motion pictures and novels, many instances of work in the medium can be grouped into similar experiences. We call these groupings genres.

In virtual reality, there are already a handful of common genres: scientific visualization competitive game operation training manufacturing procedure analysis product prototyping interactive story historical site recreation

Often a particular genre will be associated with similar styles of interface, such as how one travels through the virtual world.

61

2. Presence and reality2. Presence and reality

Issues What is reality? What does it mean by “real”? What does determine “the sense of presence”? How to effectively generate the “suspension of disbelief”?

Why care? For conceptual and analytic framework

Has been the ultimate goal of philosophy. Has been (implicitly) studied in literature, theatrical

arts, film.

62

Presence? No comprehensive theory Mental manifestation

Ex. assumption/expectation on the virtual world, previous experience, duration of exposure.

Subjective measure seems inevitable.

[World] -> (selective) -> sense ->perception -> cognition -> [mental model

attention of the world]

63

Tele-presence and virtual presenceTele-presence and virtual presence

Tele-presence ?

64

Determination of presenceDetermination of presence

INDEPENDENT VARIABLES DEPENDENT MEASURES

Fig 3. Experimental determination of presence,

learning efficiency, and performance.

extent of

sensory

information

control of

sensors

PRINCIPAL

DETERMINANTS

OF PRESENCE ability to

modify

environment

task difficulty MAJOR

TASK

VARIABLES

degree of

automation

sense of presence subjective rating objective measures

training efficiency

task performance

3. AIP Cube3. AIP Cube

Zeltzer (1992)

A model for describing, categorizing, comparing various VEs, rather than what constitutes the sense of presence.

Three components autonomy interaction presence

autonomy and Interactionautonomy and Interaction

Autonomy the ability of a computation model to act and react to simul

ated events and stimuli

• 0 : passive, geometric model

• 1 : most sophisticated, knowledge based virtual agent

• 0.x : physics-based model

Interaction the degree of access to model parameters at runtime

• 0 : "batch" processing - no interaction at runtime

• 1 : comprehensive, realtime access to all parameters

presencepresence

A rough, lumped measure of the number and fidelity of available sensory input and output channels

Measure of the Selective Fidelity

– must consider the degree of match between the sensory data and mental model.

– may consider sensory substitution (e.g., auditory output, instead of haptic).

autonomy, interaction, presenceautonomy, interaction, presence

(0,0,0)

(1,0,0) (1,1,0)

(0,1,0)

(0,1,1)(0,0,1)

(1,0,1)

Autonomy

Interaction

Presence

DigitalShakespeare

Task LevelGraphical Simulation

ConventionalAnimationSystemsca. 1990

"Virtual Reality"

(1,1,1)

Interesting possibilitiesInteresting possibilities

0 0 0 batch processing of simple models on plotter0 0 1 non-interactive virtual tour0 1 0 animation systems0 1 1 typical virtual environment1 0 0 high precision simulation 1 0 1 "Virtual Theater"1 1 0 MUD(?)1 1 1 truly a Virtual Reality

Autonomy

InteractionPresence Typical System

71

Well’s model of presenceWell’s model of presence

immersive: The participant is completely surrounded by the VE.

interactive: The participant's action affect the VE, and vice versa.

intuitive: The participant communicates with the VE using natural action

72

Lavroff’s model of presenceLavroff’s model of presence

Immersion Manipulation Navigation

4. Model P4. Model P

Perception visual auditory tactile etc.

Interaction self environmental social

Model geometry kinematics dynamics behavioral cognitive emotional

Presence = f (perception, interaction, model)

5. Measuring reality / sense of presence5. Measuring reality / sense of presence

(1) Psychological and subjective measures

(2) Psychophysical measures

(3) Physiological measures

(4) Performance measures

(5) Reflex response

76

(1) Psychological and subjective measures procedure

i) Scale rating along a uni-directional axis.

ii) Compute the psychological distance

Ex. NASA TLX scale mental load = f (mental demand, physical demand,

temporal demand, performance, effort)

Ex. Presence assessment

Ex. factorial studies

77

78

79

80

Ex. Discrimination between a real and a virtual worlds

P (judged “real” | actually real)

P (judged “real” | actually virtual)

idea: image quality virtual < real

virtual = real + noise

81

(2) Psycho-physical measures measures more “local” parameters. Types of classic problems

1) Sensory threshold: to determine whether the stimulus exists

2) Recognition: to determine what the stimulus is

3) Discrimination: to determine whether a stimulus is different from another

4) Scale: to determine how different Scaling = assignment of numbers to stimuli

1) Nominal – give numbers

2) Ordinal – give rank

3) Interval – determine the category and its interval

4) Ratio – give ratio

82

(3) Physiological measures

classes Cardiovascular – heart rate, blood pressure Respiratory – respiration rate for oxygen Nervous – brain and muscle activities

- EEG (electro-encephalo-graph) measurements Sensors – visual/hearing acuity, blink rate blood chemistry – catecholamine ( 신경전달 작용을 하는

호르몬 )

83

(4) Performance measures Examples

# of errors time spent accuracy

Assumption: presence = f (performance) or f (behavior)

But, sometimes we may intentionally decrease presence in order to increase performance. Ex. Air traffic display – exo-centric display, rather than ego-centric

Ex. Geometric enhancement

84

(5) Reflex response

Response to unexpected / threatening stimuli. E.g. collision avoidance, driving

Socially-conditioned response E.g. – say “Hi”

effects of prolonged exposure

85

6. Philosophical considerations6. Philosophical considerations

Theories on reality Plato’s idea Leibniz’s possible worlds

• God has multiple versions of world.

• He has chosen one of them, which is our world. Goodman’s multiple worlds

• There are many possible worlds. We experience one of them, which is activated by certain specific theory. (theory-dependent world)

Popper’s three worlds

• Things can exist in three ways. Physical worlds, mental worlds and artificial worlds

86

Questions?Questions?Questions?Questions?