Open Standards for 3D and AR on the Web - Khronos Group · Open Standards for 3D and AR on the Web...

© Copyright Khronos Group 2017 - Page 1

Open Standards for

3D and AR on the WebNeil Trevett | Khronos President

NVIDIA VP Developer Ecosystem

[email protected] | @neilt3d

AWE, May 2017

mailto:[email protected]


Khronos Mission

Software

Silicon

Khronos is an International Industry Consortium of over 100 companies creating

royalty-free, open standard APIs to enable software to access hardware acceleration for

3D graphics, Virtual and Augmented Reality, Parallel Computing,

Neural Networks and Vision Processing


Khronos Standards Ecosystem

VR, Vision, Neural Networks- VR/AR system portability

- Tracking and odometry

- Scene analysis/understanding

- Neural Network inferencing

Real-time 2D/3D- Cross-platform gaming and UI

- VR and AR Displays

- CAD and Product Design

- Safety-critical displays

3D for the Web- Real-time apps and games in-browser

- Efficiently delivering runtime 3D assets

Parallel Computation- Machine Learning acceleration

- Embedded vision processing

- High Performance Computing (HPC)


Augmented Reality - Native Acceleration

Vision

sensor(s)

Tracking and

Positioning

Geometric scene

reconstruction

Semantic scene

understanding

(Neural Networks)

Generate Low Latency

3D Augmentations for

display by optical system

Interact with sensor, haptic

and display devices

Download 3D

augmentation object

and scene data

AR Application

How can we bring

ACCELERATED AR to

the Web?


glTF – Cross-Platform 3D Assets

glTF – OpenGL Transmission FormatEfficient transmission of 3D assets

Laugh Engine running on Vulkanhttps://github.com/jian-ru/laugh_engine

Now glTF 2.0 has PBR!Cool, portable materials

Rendering API independence

Coming to Microsoft Office!

glTF enables portable,

efficient, sophisticated 3D

assets for all 3D applications

https://github.com/jian-ru/laugh_engine


Khronos 3D APIs already

powering VR rendering

The standard for desktop VR today

On millions of mobile VR devices

Powering WebVR in browsers

The future of explicit, low-latency VR

Khronos APIs for Virtual Reality

But What About

INPUT Hardware for AR/VR..Device discovery

Multiple sensor tracking

Device Events

Haptics

Parameters for optics corrections

etc. etc…

OpenXR!Cross-Platform, portable AR/VR


OpenXR – Solving AR/VR Fragmentation

Proprietary

Engine

VR App

1

VR App

2

VRApp

4

VRApp

3

VR Device

1

VR Device

2

VR Device

3

VR Device

5

VR Device

4

After OpenXR Wide interoperabilityof

VR apps and devices

VR App

1

VR App

2

VRApp

4

VRApp

3

Proprietary

Engine

VR Device

1

VR Device

3

VR Device

5

VR Device

2

VR Device

4

Before OpenXRVR Market

Fragmentation

Device Layer

Application Interface


OpenXR Working Group Members

Design work started in December 2016

Typically 12-18 months to develop a V1.0 specification


Pervasive WebGL 1.0• WebGL on EVERY major desktop and mobile browser

http://caniuse.com/#feat=webgl

http://caniuse.com/#feat=webgl


WebGL 2.0 Spec• Enhanced visual quality, performance, features

- Instancing Multiple render targets| Uniform buffers | Transform feedback

- Multisampled Renderbuffers| 3D textures | NPOT textures

- More texture formats | Occlusion queries | Vertex array objects

- Sampler objects | Sync objects | Fragment depth | Primitive restart | …

• WebGL 2.0 now available in Chrome/Firefox!

- Chrome: Releasing to desktop and Android

- Firefox: Released on all platforms

- Edge/Safari: plan to ship WebGL 2.0

• Desktop support at 60% - rising fast http://webglstats.com/webgl2

WebVR Running Over WebGL 2.0

https://www.khronos.org/registry/webgl/specs/latest/2.0/

http://webglstats.com/webgl2

https://drive.google.com/open?id=0BzudLt22BqGRS2ZtNGxtdm5aemM


WebGL Timeline

2003

1.02004

1.12007

2.02012

3.02014

3.1

DriverUpdate

SiliconUpdate

SiliconUpdate

DriverUpdate

Compute Shaders

32-bit integers and floats

NPOT, 3D/depth textures

Texture arrays

Multiple Render Targets

Programmable

Vertex and

fragment shadersFixed function

Pipeline

2015

3.2

SiliconUpdate

Tessellation and geometry shaders

ASTC Texture Compression

Floating point render targets

Debug and robustness for security

Epic’s Rivalry demo using full Unreal Engine 4on mobile

https://www.youtube.com/watch?v=jRr-G95GdaM

2011

WebGL 1.0

GDC 2017!

WebGL 2.0

Conformance Testing is vital for

Cross-Platform ReliabilityWebGL 2.0 conformance tests are very thorough

10x more tests than WebGL 1.0 tests

NL +

Possible Compute Shader Extension

‘WebGL Next’

https://www.youtube.com/watch?v=jRr-G95GdaM


Vulkan Explicit GPU Control

GPU

High-level Driver

AbstractionLayered GPU Control

Context managementMemory allocation

Full GLSL compiler

Error detection

ApplicationSingle thread per context

GPU

Thin DriverExplicit GPU Control

ApplicationMemory allocation

Thread managementSynchronization

Multi-threaded generation of command buffers

Multiple Front-end

CompilersGLSL, HLSL etc.

Loadable debug and

validation layers

Vulkan 1.0 provides access to

OpenGL ES 3.1 / OpenGL 4.X-class GPU functionality

but with increased performance and flexibility

SPIR-V pre-compiled shaders

Complex drivers

cause overhead

and inconsistent

behavior across

vendors

Always active

error handling

Full GLSL

preprocessor and

compiler in

driver

OpenGL vs.

OpenGL ES

Resource management

offloaded to app:

low-overhead, low-latency

driver

Consistent behavior:

no ‘fighting with driver

heuristics’

Validation and debug layers

loaded only when needed

SPIR-V intermediate

language: shading language

flexibility

Multi-threaded command

creation. Multiple graphics,

command and DMA queues

Unified API across all platforms with feature set

flexibility

Vulkan = high performance

and low latency 3D – Ideal

for VR/AR applications


Vulkan – Portable New Generation GPU API

Cross Platform

7

All Major GPU Companies shipping Vulkan Drivers – for Desktop and Mobile Platforms

Mobile, Embedded and Console Platforms Supporting Vulkan

Embedded LinuxAndroid TVAndroid 7.0 Nintendo Switch

Vulkan support

in Unity 5.6

Vulkan support in V5.4

Late May 2017

DOTA 2 on Vulkan

port of Source 2

DOOM shipping on

Vulkan port of id Tech 6

Initial Vulkan mobile

renderer in 4.12

Vulkan support

since V1.8

http://www.amd.com/


WebGL and WASM (Web Assembly)

C/C++ Code with

Native API calls

Emscripten

Clang/LLVM

JVM

WASM Modules

+ JS Glue Code

JavaScript

JavaScript

calls WASM

Modules

WASM modules

access Web APIs

(like WebGL)

through JS call-outs

For calling Web APIs in WASM Modules there needs to be: 1) a native API for WASM source code

2) a JS API for run-time invocation

3) a simple mapping between them to minimize JS Glue Code

JavaScript WebAPI

WebGL Next will need:1) a portable native APIs that can run on any system

2) a JS binding of that API for use via JavaScript

3) a simple mapping between the two

Mozilla’s Obsidian WebGL Next Proposal: 1) defines as Vulkan Subset as a portable native API

2) binds the Vulkan subset to JavaScript

3) for a simple mapping between the two


Vulkan Portability – Call for Participation

‘WebGL Next’- Lift ‘Portability API’ to

JavaScript and use in

WebAssembly native code

- Nexgen graphics and

GPU compute for the Web

API

Overlap

Analysis

Portable ‘Vulkan

Subset’ API

Specification

A Portability Solution needs to address

APIs and shading languages

Open source compilers/translators for

shading and intermediate languages

MIR

DX IL

GLSL

MSL

HLSL

Vulkan Portability SolutionC/C++ Portability API Library

+ Shading Language tools

All open source

Use Feature

Sets to remove

non-portable

functionality

Use Extensions

to add

functionality

e.g. security and

robustness for

the Web

Vulkan is non-proprietary and is

already designed to be portable

No single API

on all systems


OpenVX - Portable, Efficient Vision Processing• OpenVX developers express a graph of vision and Neural Network operations (‘Nodes’)

- Using a C API

• Graphs isolate the programmer from hardware and enable significant optimizations

- Implementers can optimize under the high-level graph abstraction

• Nodes can be executed on any hardware or processor coded in any language

- Including Vulkan or WebGL Next Compute Shaders

VisionNode

VisionNode

VisionNodeCNN Nodes

OpenVX graph mixing CNN nodes with traditional vision nodes

‘WebGL Next’

‘WebVX’??

Possible Native StackC API to Build Graph

Nodes accelerated by Vulkan

Potential Web Stack??JS API to Build Graph

Nodes accelerated by WebGL Next


Augmented Reality - Web Acceleration

Vision

sensor(s)

Tracking and

Positioning

Geometric scene

reconstruction

Semantic scene

understanding

(Neural Networks)

Generate Low Latency

3D Augmentations for

display by optical system

Interact with sensor, haptic

and display devices

Download 3D

augmentation object

and scene data

AR Application

‘WebGL Next’

‘WebVX’

‘WebGL Next’


Please Get Involved!• Information on all these standards at Khronos

- www.khronos.org

• Khronos is driving to new levels of community engagement

- Specifications and resources increasingly open for industry input and feedback

• Any company or organization is welcome to join Khronos

- For a voice and a vote in any of these standards

• If joining is not possible – ask about a Khronos Advisory Panel

- Free of charge – enables design reviews, requirements and contributions

• Neil Trevett

- [email protected]

- @neilt3d

http://www.khronos.org/

Date post:	02-Apr-2018
Category:	Documents
Upload:	lephuc
View:	218 times
Download:	2 times

Open Standards for 3D and AR on the Web - Khronos Group · Open Standards for 3D and AR on the Web...

Documents