© Khronos® Group Inc. 2018 - Page 1
Overview and AR/VR Roadmap
Neil TrevettKhronos President
NVIDIA VP Developer [email protected] | @neilt3d
© Khronos® Group Inc. 2018 - Page 2
Khronos Connects Software to Silicon
Software
Silicon
Khronos is an International Industry Consortium creating royalty-free, open standards to enable software to access hardware acceleration for
3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing
© Khronos® Group Inc. 2018 - Page 3
What is An Open Standard?
Bad Standards - Overprescribes implementation details
- Forces lowest common denominator - Stifles innovation
-> Commoditization
Good Standards- Prescribes ONLY interoperability- Enables implementation diversity
– Encourages innovation-> Differentiation
An INTEROPERABILITY STANDARD enables two entities to COMMUNICATE.
E.g. Software <-> Hardware
A Truly OPEN standardIs not controlled by a single company – but by the whole industry
Is freely available to use by any company without royalty paymentsHas a well defined IP Framework to protect standard AND member’s IP
Standards Grow Markets…By reducing consumer confusion and increasing capabilities and usability
… Reduce Costs …By sharing development between
many companies and driving volume
… Accelerate Time to MarketWith well-proven functionality,
testing and interoperability
© Khronos® Group Inc. 2018 - Page 4
The Need for AR/VR Standards
Vertically Integrated One-off Solutions in 1990s High-volume Consumer Platforms Today!
Consumer platforms have insatiable need for content to drive demand - can’t afford silo’d content!
Consumer confidence – will my device run the content I want? Avoid Betamax vs. VHS Syndrome!
VR/AR will build on a constellation of standards
Interoperability standards so content can run on all VR devices!
Compete on adding user value –not restricting user choice!
HMD Connectivity Verticals and Security
VR Video Formats Acceleration APIs XR in the Browser
Wireless and 5G
© Khronos® Group Inc. 2018 - Page 5
Standards for AR and VR
High-performance, low-latency 3D Graphics
Download 3D object and scene data
Vision and sensor processing -including neural network
inferencing for machine learning
Portable interaction with VR/AR sensor, haptic and
display devices
© Khronos® Group Inc. 2018 - Page 6
Vulkan and New Generation GPU APIs
Modern architecture | Low overhead| Multi-thread friendlyEXPLICIT GPU access for EFFICIENT, LOW-LATENCY,
PREDICTABLE performance
Vulkan Porting Tools
Non-proprietary, royalty-free open standard ‘By the industry for the industry’Portable across multiple platforms - desktop and mobile
© Khronos® Group Inc. 2018 - Page 7
Vulkan – Now Available Everywhere
VR-Related FeaturesNOW
Multi-GPU supportMultiview Rendering
Context priorityFront buffer rendering
IN DISCUSSIONVariable Rate Rendering
Tiled rendering (beam racing)
© Khronos® Group Inc. 2018 - Page 8
The Metaverse will be the Web!
https://xkcd.com/1367/
© Khronos® Group Inc. 2018 - Page 9
ContentJavaScript, HTML, CSS, ...
WebGL Stack
JavaScript Middleware
CSS
JavaScript HTML5
Browser provides WebGL3D engine alongside other HTML5
technologies - no plug-in required
OS Provided Drivers WebGL uses native OpenGL
or OpenGL ES or Angle = OpenGL ES over DX
Content downloaded from the Web
Middleware provides accessibility for non-expert programmers
E.g. three.js library
Low-level WebGL API provides a powerful foundation for a rich
JavaScript middleware ecosystem
Reliable WebGL relies on work by
both GPU and Browser Vendors
->Khronos has the
right membership to enable that cooperation
© Khronos® Group Inc. 2018 - Page 10
WebGL Deployment – WebGL 2.0 is Here!
http://caniuse.com/#feat=webgl
93.26% Globally
65.77% Globally
WebGL 1.0
WebGL 2.0
WebGL 2.0 will reach WebGL 1.0 levels of availability
when Safari and Edge ship enabling a new class of
Web-based AR/VR 3D Apps!
Pervasive, portable access to OpenGL ES 2.0-class mobile graphics
OpenGL ES 3.0-based desktop-class graphics comes to the Web!
© Khronos® Group Inc. 2018 - Page 11
Khronos Standards for AR and VR
High-performance, low-latency 3D Graphics
Download 3D object and scene data
Vision and sensor processing -including neural network
inferencing for machine learning
Portable interaction with VR/AR sensor, haptic and
display devices
© Khronos® Group Inc. 2018 - Page 12
glTF – The JPEG of 3D!
Efficient, reliable transmissionBring 3D assets into 1000s of
apps and engines
glTF 2.0 – June 2017Native AND Web APIsPhysically Based Rendering
Metallic-Roughness and Specular-Glossiness
glTF 1.0 – December 2015Primarily for WebGL
Uses GLSL for materials
glTF spec development on open GitHub – get involved!
https://github.com/KhronosGroup/glTF
Compact to TransmitSimple and Fast to LoadDescribes Full Scenes
Runtime NeutralOpen and Extensible
© Khronos® Group Inc. 2018 - Page 13
glTF Ecosystem
Apps and Engines
Creation Tools
Repositories
Sony 3D Creator
Oculus
Paint 3D
Collada2gltf FBX2glTFgltf-vscode
glTF-validator glTF-asset-generator
Experience
Discover
DriveDemand
Create Windows Mixed Reality Home
Users
Modo
Mixed Reality Viewer3D Builder
Prep for 3D printing
© Khronos® Group Inc. 2018 - Page 14
glTF Roadmap• glTF manages its roadmap very carefully – complexity is the enemy
- Mission #1: ensure widespread, consistent, reliable usage
• Optional Draco Mesh Compression Extension Shipping Now- Adoption enabled with open source encoders and decoders
• Extensions and projects in discussion…- Draco compressed Point clouds, Universal compressed textures, LODs and streaming- Nexgen PBR materials, Advanced animations (e.g. Avatars and Face emoji), Metadata
• Developing reference viewer for visual consistency across web and native engines
Mesh Compression
Ratios
Stable Core Spec
Domain-specificextensions stay as
extensions
New widely needed functionality ships first as
extensions
Integrate extensions into new core spec only when:1) Widespread need is confirmed by the industry
2) Widespread reliable implementation is enabled (e.g. open source)
© Khronos® Group Inc. 2018 - Page 15
Khronos Standards for AR and VR
High-performance, low-latency 3D Graphics
Download 3D object and scene data
Vision and sensor processing -including neural network
inferencing for machine learning
Portable interaction with VR/AR sensor, haptic and
display devices
© Khronos® Group Inc. 2018 - Page 16
Vision and Inferencing Runtimes
Machine Learning Acceleration
Desktop and Cloud GPU/TPU Acceleration
Diverse Inferencing Acceleration Hardware
Training on Desktop and Cloud
Neural Net Training Frameworks
Neural Net Training Frameworks
Neural Net Training FrameworksTraining
Frameworks
Deployment on Embedded Devices
OptimizationTrained
Networks
Vision and Inferencing Applications
Training Data Sets
ORLiveData
© Khronos® Group Inc. 2018 - Page 17
OpenVX – Vision and Inferencing Acceleration Wide range of vision and inferencing hardware architectures so
OpenVX provides a high-level Graph-based abstraction for portability->
Enables Graph-level optimizations!Can be implemented on almost any hardware or processor!
->
Portable, Efficient Vision and Inference Processing!
Shipping Implementations
VisionNode
VisionNode Vision
Node
Downstream ApplicationProcessing
NativeCamera Control CNN Nodes
NNEF Translator converts NNEF representation into OpenVX Node Graphs
An OpenVX graph mixing CNN nodes with traditional vision nodes
© Khronos® Group Inc. 2018 - Page 18
GPU
OpenCL – Unique Heterogeneous Runtime
FPGA DSP
Custom Hardware
GPUCPUCPUCPUGPU
Growing number of optimized OpenCLvision and inferencing librariesVision: OpenCV, Halide, VisioncppMachine Learning: Xiaomi MACE, Arm Compute LibraryLinear Algebra: clDNN, clBlast, ViennaCL
Application orInferencing Run-time
GPU APIs
OpenCL is the only industry standard for low-level heterogeneous computePortable control over memory and parallel task execution
“The closest you can be to your embedded accelerator and still be portable”
Application orInferencing Run-time
© Khronos® Group Inc. 2018 - Page 19
Khronos Standards for AR and VR
High-performance, low-latency 3D Graphics
Download 3D object and scene data
Vision and sensor processing -including neural network
inferencing for machine learning
Portable interaction with VR/AR sensor, haptic and
display devices
© Khronos® Group Inc. 2018 - Page 20
XR = AR + VR
V1.0 - focused on VR
V
AAfter 1.0 – equal focus on AR
© Khronos® Group Inc. 2018 - Page 21
OpenXR – Solving VR/AR Fragmentation
Proprietary Engine
VR App
1
AR App
2
ARApp
4
VRApp
3
VR Device
1
VR Device
2
VR Device
3
VR Device
5
VR Device
4
After OpenXR Wide interoperability of XR apps and devices
VRApp
1
ARApp
2
ARApp
4
VRApp
3
Proprietary Engine
VR Device
1
VR Device
3
VR Device
5
VR Device
2
VR Device
4
Before OpenXRXR Market Fragmentation
Device Integration Layer
Application Interface
WebXR WebXR
© Khronos® Group Inc. 2018 - Page 22
OpenXR Architecture
Portable AR/VRInput Device Discovery
Multiple Sensor TrackingDevice Events
Pose NormalizationHaptics Control
Optical Corrections
OpenXR doesn’t replace AR/VR runtimes!
It enables those runtimes to use PORTABLE APIs to
expose their functionality
© Khronos® Group Inc. 2018 - Page 23
Companies Publicly Supporting
OpenXR is a collaborative designIntegrating many lessons from proprietary ‘first-generation’ API designs
© Khronos® Group Inc. 2018 - Page 24
Epic ‘Showdown’ VR Demo at SIGGRAPH
Demo runs portably across StarVR and Microsoft Windows Mixed Reality
headsets through the OpenXR APIs via an Unreal Engine 4 plugin https://www.youtube.com/watch?v=FCAM-3aAzXg&t=17250s
© Khronos® Group Inc. 2018 - Page 25
Mobile Augmented Reality Libraries
Pose Tracking Yes Yes
Plane Detection and Tracking (Hori. & Vert.) Yes Yes
Image Recognition and Tracking Yes Yes
3D Object Scanning, Recognition and Tracking Yes No
Ambient light level and temperature Yes Yes
Environment Light Probes Yes No
Environment Reflective Texturing Yes No
Link to Neural Net-based Object Detection Yes Yes
Access to Point Cloud Yes Yes
Camera Intrinsics Yes Yes
Multi-user and persistent cloud-based anchors Yes Yes
Multi-user Viewing Yes No
Face tracking (iPhone X) Yes No
OS Availability iOS Android and iOS
ARKit
Encapsulated Vision-based Functionality
Also leveraging motion sensors
3rd party AR Libraries typically use
ARKit/ARCore if available or
implement own tracking and
functionality if not
© Khronos® Group Inc. 2018 - Page 26
Bringing VR and AR to the WebWeb XR Apps
WebXR
Native XR Apps
System-exposed AR Capabilities
3D Engines 3D Engines
Close ongoing collaboration between
WebXR and OpenXR
Khronos providing the foundation for 3D and
XR in the Web and native stacks
Future versions of OpenXR will include
cross-platform extended AR functionality
© Khronos® Group Inc. 2018 - Page 27
Summary• Khronos is creating cutting-edge royalty-free open standards
- For 3D, VR/AR, Compute, Vision and machine learning
• Join Khronos – any company is welcome!- Influence the direction of industry standards- Get early access to draft specifications
• More Information- www.khronos.org- [email protected] @neilt3d