VR2.0: Making Virtual Reality Better Than Reality?

Slide 1

Making Virtual Reality better than Reality?Gordon WetzsteinStanford UniversityIS&T Electronic Imaging 2017

www.computationalimaging.org

Next-generation computing platforms will be wearable and provide seamless IO interfaces.

Phones today are already wearable, but all interaction with the digital world happens on the phone and is confined to that small window.

Future IO interfaces will augment our entire world with digital and interactive content. They would free us from the constraints of having to interact with dedicated devices and allow for natural interaction with an augmented reality.2

Personal Computere.g. Commodore PET 1983Laptope.g. Apple MacBookSmartphonee.g. Google Pixel

AR/VRe.g. Microsoft Hololens

???

Todays AR and VR displays look clumsy; they are bulky and power hungry. But if we have learned one thing throughout the last 50 years in Silicon Valley, then it is that room-sized computer will soon fit in your pocket. The compute power of a phone today is more than 1,300 times higher than that of Apollo 11 s computer, which is what put man on the moon in 1969.

We can think of next-generation AR systems as a natural continuation of wearable computers today.3

A Brief History of Virtual Reality

183819682012-2017

StereoscopesWheatstone, Brewster, VR & AR Ivan SutherlandVR explosionOculus, Sony, HTC, MS, NintendoVirtual Boy

1995

VR 2.0

But AR/VR displays are really nothing new.4

Where we are nowIFIXIT teardown

Recent progress in VR has been largely driven by the cellphone industry. If you look at whats inside a modern HMD, you will see mostly cellphone components. They have enables low-cost high-resolution display, inertial measurement units, and other components that are key to making AR & VR work.

Basically, you just need to snap on some lenses on your phone.5

6

7

Magnified Display

ddf

we are looking at a really big plane floating somewhere in front of usis this the type of visual signal that we want to see? what we are used to seeing in reality?8

Real World:Vergence & Accommodation Match!

When we look around the real world, a true 3D scene, our eyes verge and accommodate to the same distance. Well because the end goal of the two systems is the same, they cheat a little and help each other out. Having these systems linked is great when looking from my finger to you, but it actually causes serious complications when we put on VR headsets.

15

Current VR Displays:Vergence & AccommodationMismatch

for people with normal vision

clearly the vergence-accommodation conflict is a big problem for people with normal visionbut how many people actually have normal vision?16

Presbyopia[Katz et al. 1997]68%age 80+43%age 4025%Hyperopia[Krachmer et al. 2005]Myopia41.6%[Vitale et al. 2009]How Many People Have Normal Vision?all numbers of US population

all numbers percent of US popolation

17

4D / 25cmOptical InfinityNormal visionNearsighted/myopicFarsighted/HyperopicPresbyopicFocal range (range of clear vision)Modified from Pamplona et al, Proc. of SIGGRAPH 2010Nearsightedness & Farsightedness

Computational Near-eye Displays

Q1: Can computational displays effectively replace glasses in VR/AR?

Q2: How to address the vergence-accommodation conflict for users of different ages?

Q3: What are (in)effective near-eye display technologies?

possible solutions: gaze-contingent focus, monovision, multiplane, light field displays,

20

Q1: Can computational displays effectively replace glasses in VR/AR?



possible solutions: gaze-contingent focus, monovision, multiplane, light field displays,

21

Magnified DisplayDisplayLensFixed Focus

ddf

For this purpose, we propose a new display mode that we dub adaptive focus, that is capable of shifting the distance to the magnified image dynamically in one of two ways.22

Adaptive FocusMagnified DisplayDisplayLens

actuator vary d

This mode is capable of changing the distance to the virtual in one of two ways. We can either swap out the conventional static glass lens that current VR displays have with a focus-tunable lens, capable of changing its focal length. By varying the focal length the virtual image position will shift.The other method, is to use conventional optics, but change the distance between the microdisplay and the lens. Varying the position will cause the virtual image to shift as well.Introduce shifting microdisplay first23

Adaptive FocusMagnified DisplayDisplayLens

focus-tunablelens vary f

For this purpose, we propose a new display mode that we dub adaptive focus. This mode is capable of changing the distance to the virtual in one of two ways. We can either swap out the conventional static glass lens that current VR displays have with a focus-tunable lens, capable of changing its focal length. By varying the focal length the virtual image position will shift.The other method, is to use conventional optics, but change the distance between the microdisplay and the lens. Varying the position will cause the virtual image to shift as well.Introduce shifting microdisplay first24

Adaptive Focus - History

M. Heilig Sensorama, 1962 (US Patent #3,050,870)P. Mills, H. Fuchs, S. Pizer High-Speed Interaction On A Vibrating-Mirror 3D Display, SPIE 0507 1984S. Shiwa, K. Omura, F. Kishino Proposal for a 3-D display with accommodative compensation: 3DDAC, JSID 1996S. McQuaide, E. Seibel, J. Kelly, B. Schowengerdt, T. Furness A retinal scanning display system that produces multiple focal planes with a deformable membrane mirror, Displays 2003S. Liu, D. Cheng, H. Hua An optical see-through head mounted display with addressable focal planes, Proc. ISMAR 2008

manual focus adjustmentHeilig 1962automatic focus adjustmentMills 1984deformabe mirrors & lensesMcQuaide 2003, Liu 2008

Padmanaban et al., PNAS 2017

heres out take on adaptive focus display hardware others try to build smaller and smaller displays, we probably built the worlds biggest VR display here26


27


28


29


30


31


32

33

34

35

36

at ACM SIGGRAPH 2016

EyeNetra.com

eyenetra37

at ACM SIGGRAPH 2016

participants of the study, 152 totalEyeNetra.com

38

Participants - Prescription

Padmanaban et al., PNAS 2017n = 70, ages 21-64

How sharp is the target? (blurry, medium, sharp)Is the target fused? (yes, no)

4D (0.25m)3D(0.33m)2D(0.50m)1D(1m)Four simulated distances

Task

A target was presented randomly at 4 different stereoscopic distances, and after 4 seconds, the users were simply asked to indicate how sharp it was, and whether it was fused (i.e., not double)

Keep in mind that these four distances are simulated stereoscopically and not optically, so the virtual image distance was always the same. This is consistent with how conventional near-eye displays work. We selected a virtual image distance of 1.3D, also consistent with conventional systems.

--

Target ~6cm tall, appeared for 4 seconds

Screen at 1.3m (or 0.77 D)40

farnear1D1m2D0.5m3D0.3m4D0.25mDistancemedium 0Relative sharpnesssharp 1blurry -1

VR uncorrectedVR correctedResults - SharpnessPadmanaban et al., PNAS 2017

First Ill show you the sharpness ratings. Each user repeated these measures with and without refractive correction

Note that distances are shown in Diopters, or inverse meters, so low values are far and high values are near41

farnear

farnear1D1m2D0.5m3D0.3m4D0.25mDistancemedium 0sharp 1blurry -1Relative sharpness


Here is the mean and standard error of these ratings. Remember, everyone normally wears correction, so its not surprising that the target does not appear particularly sharp42

farnear



And here are the ratings for trials when the correction was turned on. The correction substantially and significantly increased perceived sharpness, but the ratings were not sharp 100% of the time.

Its hard to know whether this means that the VR correction wasnt fully effective, or whether even people wearing their typical correction experience some amount of blur in VR

---

Logistic regressions indicated significant main effects for both condition and distance. The odds ratios for correction were 4.05 (95% confidence interval(ci)=3.255.05) and 1.54 (ci=1.201.98) for sharpness and fusibility, respectively. The distance odds ratios were 0.77 and 0.21 (all ps0.01).

43

Mean = 0.63Mean = 0.60farnear


VR uncorrectedVR correctednormal correctionResults - SharpnessPadmanaban et al., PNAS 2017

We found that the VR correction was essentially as effective as peoples normal correction at providing clear vision of the virtual targets. Here Im just indicating the overall average sharpness scores for these two conditions, which were very similat44

far

farnear1D1m2D0.5m3D0.3m4D0.25mDistance

VR uncorrectedVR corrected1Proportion fused0.80.60.40.20Results - FusionPadmanaban et al., PNAS 2017

Remember, we also asked people whether or not the target appeared fused45

far

farnear1D1m2D0.5m3D0.3m4D0.25mDistance

VR uncorrectedVR corrected1Proportion fused0.80.60.40.20Results - FusionPadmanaban et al., PNAS 2017

46

Computational Near-eye DisplaysQ1: Can computational displays effectively replace glasses in VR/AR?



possible solutions: gaze-contingent focus, monovision, light field displays,

47

vergenceaccommodationConventional Stereo / VR Display

The stereoscopic depth cues in VR can drive the vergence response to arbitrary distances, but to focus on the image, the eyes much always accommodate to the same fixed distance of the virtual image.

heres a diagram of how the eyes are forced to focus at the optical, or virtual image, distance of the display in an HMD, while the stereoscopic distance can be adjusted with binocular disparities.

48

Visual discomfort (eye tiredness & eyestrain) after ~20 minutes of stereoscopic depth judgments (Hoffman et al. 2008; Shibata et al. 2011)

Degrades visual performance in terms of reaction times and acuity for stereoscopic vision (Hoffman et al. 2008; Konrad et al. 2016; Johnson et al. 2016)Consequences of Vergence-Accommodation Conflict

49

vergenceaccommodationRemoving VAC with Adaptive Focus

And heres a diagram of the idealized result with focus tunable optics, were now the optical and stereoscopic distances are matched, as they should be in the natural environment

50

Follow the target with your eyes

4D (0.25m)0.5D(2m)

Task

Target ~6cm tall, The stimulus was shown for 4.5 cycles, and the response gain was calculated as the relative amplitude between the response and stimulus for the 3 cycles directly following a 0.5 cycle buffer (shown).

0.125 Hz sinuisoid (8 seconds per cycle), 0.5 4D n = 24 dynamic and 59 conv

Dont measure vergence

51

Stimulus

Padmanaban et al., PNAS 2017Accommodative Response

Relative Distance [D]

Time [s]

To examine how well our system can drive natural accommodative responses, we conducted an experiment in which we measured participants accommodation state as they tracked a target oscillating in depth.

The target moved slowly in depth (at 0.125 Hz for 4.5 cycles sinusoidally) between 0.54 D and was scaled to be 6.2 cm in height.

First, Ill show you the average accomodative response of peoples eyes in a conventional HMD

-----

A Grand Seiko WAM-5500 autorefractor is integrated into the near-eye display system. The autorefractor uses built- in near infrared illumination (NIR) and a NIR camera to determine the users accommodative state. The illumination pattern is close to invisible to the user. Accommodation is recorded at roughly 45 Hz with an accuracy of 0.25 D and directly transmitted to the computer that controls the visual stimulus. The accuracy of the autorefractor is verified using a Heine Ophthalmoscope Trainer model eye (C-000.33.010).

52

StimulusAccommodationn = 59, mean gain = 0.29Padmanaban et al., PNAS 2017Accommodative Response


Time [s]

People did actually change their accommodation slightly in response to the stimulus distance, but the average response rain was only ~0.3.

N = 53

Stimulus

Padmanaban et al., PNAS 2017Accommodative Response


Time [s]

Next lets see what the accommodative responses were in the focus tunable HMD54

StimulusAccommodationPadmanaban et al., PNAS 2017Accommodative Response


Time [s]n = 24, mean gain = 0.77

Here we found that accommodative responses were larger on average, with an average gain of almost 0.8.

A gain of 0.8 is pretty typical for the accommodative gain to physical stimuli, so this results suggests that the system was effectively able to drive natural accommodative responses.55

Duane, 1912

Nearest focus distanceAge (years)816243240485664724D (25cm)8D (12.5cm)12D (8cm)Presbyopia0D (cm)16D (6cm)

1000 subjects56

Presbyopia

There is no cure for presybopia, in the sense that there is no treatment to restore the full accommodative range. Corrective lenses are designed to increase the range of distances over which people can experience clear vision, and these include bifocals, monovision lenses, and multifocal lenses57

Padmanaban et al., PNAS 2017Do Presbyopes Benefit from Dynamic Focus?

Gain

Age

58


Gain

Age

conventional

59


Gain

Age

conventionaldynamic

60


Gain

Age

conventionaldynamic

Response for Physical StimulusHeron & Charman 2004

61

farnearfarnearPadmanaban et al., PNAS 2017Age-dependent Fusion

Percent Fused

We also asked participants to rate the apparent sharpness and ability to fuse the same target at 4 different distances

----

A logistic regression with age, condition, and distance showed significant main effects of distance and condition. The distance odds ratio was 0.56 (ci=0.460.69), and the ratio for the dynamic condition was 0.60 (ci=0.48 0.75) (ps

Date post:	21-Feb-2017
Category:	Engineering
Upload:	stanfordcomputationalimaging
View:	167 times
Download:	0 times

VR2.0: Making Virtual Reality Better Than Reality?

Engineering