AR Devices: Grand Challenges - Khronos Group · AR Devices: Grand Challenges and how open standards...

© Khronos® Group Inc. 2019 - Page 1

AR Devices: Grand Challengesand how open standards can help

Neil TrevettKhronos President, NVIDIA VP Developer Ecosystems

[email protected] | @neilt3d

Spatial Computing Workshop

mailto:[email protected]


Real World

Virtual World

RealReality

AugmentedReality

VirtualReality

Disruptive?

Virtual Reality has already ‘emerged’steady growth in

GamingLBE

TrainingMedical

Etc.

Mixed Reality is the same thing as Augmented Reality!

Mobile Phone/Tablet - based AR

See-through

optics AR

AR on today’s mobile phones good for demos and short experiences

See-through AR may eventually replace today’s mobile phones

AR/VR (XR) Adoption


AR in Enterprise Today = $$

www.http://thearea.org/

Guided Task

Design Insights

Data Visualization

Remote Expert

Tablets and HoloLens already widely deployed in industrial and enterprise use cases

Less sensitive to cost and social factors that may hold back consumer AR

http://www.http/thearea.org/


OpticsResolution, brightness, field of view, focus accommodation, eyebox

SensorsPosition tracking, occlusion, lighting, eye-tracking, scene semantics, UI

Power ConsumptionAll day continuous

operation in small, socially acceptable form factor

InteroperabilityAcceleration API standards

Connectivity standardsContent standards

Content discoverability

See-Through AR Grand Challenges


Real World Stereoscopic Display

Vergence–accommodation Conflict

Wide Field Of View Varifocal Near-Eye Display Using See-ThroughDeformable Membrane Mirrors, Proc. of IEEE VR 2017

http://telepresence.web.unc.edu/files/2017/01/Dunn_2017_TVCG_MembraneAR.pdf

Focus accommodation needed: 1) Avoid fatigue/nausea

2) Realistic mixing of real and virtual imagery

Focus Accommodation

http://telepresence.web.unc.edu/files/2017/01/Dunn_2017_TVCG_MembraneAR.pdf


Daqri Smart glasses

HUDMonocular, no tracking

Single Focal Plane ARStereo, 6DOF positional tracking

Commercial See-Through Display Evolution

Magic Leap One$2,295

Nreal Light - CES 2019

Varifocal ARFocus Accommodation

?

Socially acceptable form factor - 85gSingle USB-C wire to processor pack1080P per eye52 degree FOVThree hour battery life$999

How can we delivervarifocal displays in a

mobile device

HoloLens 2 – MWC 2019Self-contained1080P per eye52 degree FOV

2-3 hour battery lifeHand and voice UI

$3,500

Vuzix Blade $999

Google Glass 2 $999


True Holographic Display ResearchThrowing massive compute at overcome physical constraintsCreate light field using dynamic holograph Spatial Light Modulator

Reflect into eye using (static) see-through, holographic elementWith enough compute can adjust focal depth of every pixel

Can also correct for optical path aberrations & vision correction

Uses LOTS of GPU compute cycles to enable compact size and wide-FOVAcceleratable with standard GPU APIs

Maimone, et al., “Holographic Near-Eye Displays for Virtual and Augmented Reality”, SIGGRAPH 2017https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/holo_author.pdf

https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/holo_author.pdf


Reducing Compute Load

4%

73%96%

iPhone 7Plus

27" DesktopMonitor

2016 VRHMD

Column1

20°

PeripheralPeripheral

Fove

al

Eye TrackingReduces rendering load - foveal rendering

Enables realistic focus cuesGaze based UI

Most HMD Pixels are Peripheral

Varjo VR-1February 2019

Hig-end XR HMDS beginning to use eye tracking

HoloLens 2

Magic Leap One











Real-time GPU-accelerated Lighting CalculationsHigh-Quality Reflections, Refractions, and Caustics in Augmented Reality and

their Contribution to Visual CoherenceP. Kán, H. Kaufmann, Institute of Software Technology and Interactive Systems,

Vienna University of Technology, Vienna, Austriahttps://www.youtube.com/watch?v=i2MEwVZzDaA

Position tracking

Eye-tracking

3D Reconstruction

Environmental lighting

Occlusion

Scene semantics

Natural UI

Environmental SensingSolved

Unsolved

Environmental Occlusion

Machine Learning needed as problems

become ‘fuzzier’

https://dl.acm.org/citation.cfm?id=3275041

More compute cycles needed for inferencing

https://www.youtube.com/watch?v=i2MEwVZzDaA

https://dl.acm.org/citation.cfm?id=3275041











Silicon Acceleration

Everything runs on CPU

Power Efficiency and Battery LifeOffload Processors – such as GPUs - are

FASTER AND more power EFFICIENT than CPUs

Developers need graphics and compute APIs to efficiently access acceleration processors

Pokémon Go AR processing runs

on CPU


Khronos Mission

Software

Silicon

Khronos is an open, member-driven industry consortium developing royalty-free standards, and vibrant ecosystems, to harness the power of silicon acceleration

for demanding graphics rendering and computationally intensive applications

Khronos members are industry leaders from around the world that join to safelycooperate - to advance their own businesses and the industry as a whole


Active Khronos Standards


Vulkan – Now Available (Almost) Everywhere

XR-Related FeaturesNOW

Multi-GPU supportMultiview Rendering

Context priorityFront buffer rendering

IN DISCUSSIONVariable Rate Rendering

Tiled rendering (beam racing)XR Devices


The Metaverse will be the Web!

https://xkcd.com/1367/

https://xkcd.com/1367/


ContentJavaScript, HTML, CSS, ...

WebGL Stack

JavaScript Middleware

CSS

JavaScript HTML5

Browser provides WebGL3D engine alongside other HTML5

technologies - no plug-in required

OS Provided Drivers WebGL uses native OpenGL

or OpenGL ES or Angle = OpenGL ES over DX

Content downloaded from the Web

Middleware provides accessibility for non-expert programmers

E.g. three.js library

Low-level WebGL API provides a powerful foundation for a rich

JavaScript middleware ecosystem

Reliable WebGL relies on work by

both GPU and Browser Vendors

->Khronos has the

right membership to enable that cooperation

http://code.google.com/p/angleproject/


GPU

OpenCL for Heterogeneous Acceleration

FPGA DSP

Custom Hardware

GPUCPUCPUCPU

Application

Heterogeneous Compute Resources

OpenCL provides a programming and runtime framework for programming heterogeneous compute resourcesOpenCL C kernels are compiled and distributed across available processors under programmer control

Many mobile SOCs and Embedded Systems becoming increasingly heterogeneous

Autonomous vehicles, Vision and inferencingAugmented and Virtual Reality

Arm Compute Library

clDNN

OpenCL-accelerated Machine Learning Inferencing Compilers

SYCL-DNN

OpenCL-accelerated Machine Learning Libraries

Android NNAPI

TI Deep Learning Library (TIDL)VeriSilicon

Huawei Intel

Intel Synopsis MetaWare EV

OpenCL Hardware Implementations

OpenCL-accelerated Vision and Imaging Libraries


Khronos OpenVX and NNEF for Inferencing

VisionNode

VisionNode

VisionNode

Downstream ApplicationProcessing

NativeCamera Control CNN Nodes

NNEF Translator converts NNEF representation into OpenVX Node Graphs

OpenVX High-level graph-based abstraction for Portable, Efficient Vision ProcessingOptimized OpenVX drivers created and shipped by processor vendors Can be implemented on almost any hardware or processorGraph can contain vision processing and NN nodes – for global optimization

Hardware Implementations

Performance comparable to hand-optimized, non-portable codeReal, complex applications on real, complex hardwareMuch lower development effort than hand-optimized code

Pow

er E

ffic

ienc

y

Computation Flexibility

Dedicated Hardware

GPUCompute

Multi-coreCPUX1

X10

X100

VisionDSPs













glTF – The JPEG of 3D!

Efficient, reliable transmissionBring 3D assets into 1000s of

apps and engines

glTF 2.0 – June 2017Native AND Web APIsPhysically Based Rendering

Metallic-Roughness and Specular-Glossiness

glTF 1.0 – December 2015Primarily for WebGL

Uses GLSL for materials

glTF spec development on open GitHub – get involved!

https://github.com/KhronosGroup/glTF

Compact to TransmitSimple and Fast to LoadDescribes Full Scenes

Runtime NeutralOpen and Extensible

https://github.com/KhronosGroup/glTF


Sony 3D Creator

Oculus

Paint 3D

Collada2gltf FBX2glTF

gltf-vscode

glTF-validatorglTF-asset-generator

ExperienceApps / Engines

DiscoverRepositories

Drive DemandUsers

CreateTools

Windows Mixed Reality Home

Modo

Mixed Reality Viewer

3D BuilderPrep for 3D printing

OBJ2GLTF

glTF-Toolkit

Titania

glTF Reference Viewer

3D Live Object

Game Engines

Web Engines

3D Apps and Engines

VR / AR Apps and Engines

Productivity and Social Apps

Dedicated 3D Authoring Tools

Authoring Tools that Export 3D

VR / AR Authoring Tools

3D Scanning Tools

Convertors and Optimizers

Validation and Reference Tools

Ecosystem


Khronos 3D Commerce Exploratory GroupKhronos 3D Commerce Exploratory GroupRetail AND Technology companies making virtual 3D

product representations possible on an industrial scale

Open to any company under NDA during exploratory phasehttps://www.khronos.org/exploratory/3d-commerce/

https://www.khronos.org/exploratory/3d-commerce/




OpenXR – Solving VR/AR Fragmentation

Proprietary Engine

VR App

1

AR App

2

ARApp

4

VRApp

3

VR Device

1

VR Device

2

VR Device

3

VR Device

5

VR Device

4

After OpenXR Wide interoperability of XR apps and devices

VRApp

1

ARApp

2

ARApp

4

VRApp

3

Proprietary Engine

VR Device

1

VR Device

3

VR Device

5

VR Device

2

VR Device

4

Before OpenXRXR Market Fragmentation

Device Integration Layer

Application Interface

WebXR WebXR


Companies Publicly Supporting OpenXR

OpenXR is a collaborative designIntegrating many lessons from proprietary ‘first-generation’ XR API designs


Khronos APIs for XR

High-performance, low-latency 3D rendering and composition*

MultiviewContext priority

Front buffer renderingTiled rendering (beam racing)

Variable rate rendering

* OpenXR can be used with other 3D APIssuch as Direct3D, OpenGL and OpenGL ES

Display, composition and

optical correction parameters

Application or Engine

Cross-platform access to XR HMDs and sensors

XR application lifecycleFrame timing and display compositionSensor tracking and pose calculationInput device discovery and events

Haptics Control


OpenXR Win-Win-Win

XR VendorsCan bring more applications onto their platform by leveraging the OpenXR content ecosystem

XR ISVsCan easily ship on more platforms for increased market reach

XR End-UsersCan run the apps they want on their system – reducing market confusion and increasing consumer confidence


Bringing XR to the WebWeb XR Apps

WebXR

Native XR Apps

System-exposed AR Capabilities

3D Engines 3D Engines

Close ongoing collaboration between

WebXR and OpenXR

Khronos providing the foundation for 3D and

XR in the Web and native stacks

Future versions of OpenXR will include

cross-platform extended AR functionality


XR and 5GLeveraging High Bandwidth and Low Latency

Wireless mobile device with display and sensors

Image: Nreal

MEC Server

Content RequestsLocation

Apps3D Assets

Rich Sensor Data

Generated Augmentations &

Scenes

OpenXR APIs can hide the 5G round trip from applications

MEC (Multi-access Edge Computing) Server 1. Processes sensor data, including machine learning for environmental

lighting, occlusion, scene semantics, object reconstruction and UI2. Generates imagery from 3D models, including stereo, foveal

rendering, ray-tracing, optics pre-distortion, varifocal processing


Any Questions?• Opportunities for VR and AR are here today!

- Mainly in enterprise today, disruptive consumer AR coming

• Standards will be key to pervasive XR- Compute, XR and content interoperability

• 5G can help accelerate XR deployment- Offload significant processing from mobile XR devices- Lightweight devices with high-bandwidth, low-latency link to Edge Computing

• Khronos is creating cutting-edge royalty-free open standards- For 3D, VR/AR, Compute, Vision and machine learning- Any company welcome to join and help steer the direction of the industry

• More Information- www.khronos.org- [email protected] @neilt3d

Date post:	25-May-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

AR Devices: Grand Challenges - Khronos Group · AR Devices: Grand Challenges and how open standards...

Documents