32
GTC Raheel Khalid 09/15/16 1
GTC Raheel Khalid 09/15/16
1
Who am I?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
2
Raheel Khalid - @rkhalid890
A game industry and operating systems veteran that serves as the Chief Engineer for Verizon Labs. Using years of experience in building large scale game engines and graphics platforms I work to architect and set the technical vision for Verizon's future in VR & AR streaming services.
Speaker bio
Why is Verizon here?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
3
Phone Company? Maybe we deliver your cable service?
Image - Pavin Trikutam - https://unsplash.com/photos/71CjSSB83Wo
Why is Verizon here? and talking about VR…?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
4
Phone Company? Maybe we deliver your cable service?
Image - Pavin Trikutam - https://unsplash.com/photos/71CjSSB83Wo
What we do
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
5
wireless fios enterprise streaming CDN big data security communications
advertising internet of things monitoring
Historically, verizon has been known as a phone company. (picture of person on old phone) Over the years we have slowly diversified (show growth and new business units, fios, enterprise, iot, vdms. Finally show advertising, aol, content) All of this eventually comes together (show how each business unit connects).
Image from verizon stock
and now…
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
6
advertising digital media lifestyles entertainment news
Historically, verizon has been known as a phone company. (picture of person on old phone) Over the years we have slowly diversified (show growth and new business units, fios, enterprise, iot, vdms. Finally show advertising, aol, content) All of this eventually comes together (show how each business unit connects).
Image from Jordan McQueen https://unsplash.com/search/friends?photo=88XM5Al3AXg
So why is Verizon here?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
7
Powered dozens of devices and services Evolved with our network to provide solutions and platforms Helped businesses innovate and get their products to consumers
Over years we have supported dozens of computing platforms and as technology grew we supported it with a network, content, digital advertising and more.
So why is Verizon here?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
8
The bridge between content, advertising, streaming, retail and the network make Verizon uniquely poised to create an ecosystem for VR/AR creation and distribution
Over years we have supported dozens of computing platforms and as technology grew we supported it with a network, content, digital advertising and more.
And VR?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
9
As the world gets closer to VR and AR we find ourselves needing to master rendering and computer vision in ever smaller form factors.
This brings new demands to silicon and ultimately the network.
“It is potentially the final computing platform. This is generalising, but if you have perfect virtual reality, you don't have to perfect much else.” Palmer Luckey, Oculus founder
That brings us to the present… If you look at how certain technology has trended desktops -> laptops -> smartphones -> tablets -> wearables. We are reaching another plateau and we are about ready for a new computing platform.
palmer luckey said ”Virtual Reality will be the final computing platform” much of the news media and masses agreed...
Image Source - https://hd.unsplash.com/photo-1468322931232-d3cb4c7da350 Aliis Sinisalu
Virtual, augmented and mixed reality today
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
10
Right now we are seeing a boom of VR consoles, platforms and content. This has come in the form of 360 videos, static games and most impressively roomscale vr (show examples of each).
We have even started to see examples of augmented reality where we can enhance, diminish, replace or remove things in the real world
Virtual, augmented and mixed reality today
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
11
Right now we are seeing a boom of VR consoles, platforms and content. This has come in the form of 360 videos, static games and most impressively roomscale vr (show examples of each).
We have even started to see examples of augmented reality where we can enhance, diminish, replace or remove things in the real world
So how do we get there?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
12
Augmented and Virtual Reality has a base requirement of low latency render, compute and network transmission
To create a reality bending future there are key problems that we need to solve. Since the mid 90’s the speed and development of GPU technology has greatly outpaced what CPU’s do today.
Image Source - https://unsplash.com/photos/NDZQLKiaCSI Yolanda Sun
So how do we get there?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
13
We now have a technological dependency on high resolution rendering, high refresh rate and loads of computer vision.
Low resolution and lag are no longer just annoying, they are a deal breaker
Talk about client side vs cloud render and compute
This is because we had a need to render realer and realer (show pac man to photorealistic render). (tie this to NVIDIA)
Image Source - Joey Kyber https://unsplash.com/search/fast?photo=45FJgZMXCK8
So how do we get there?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
14
It is generally accepted that the greatest acceptable time it takes to update the screen from a user’s movement is roughly 20ms. This is referred to as motion-to-photon latency.
Typical displays on VR/AR devices have a refresh rate of between 60Hz and 90Hz or 16.66ms to 11.11ms. This is the time to update pixels on a screen.
The fastest networks in the US provide 40-200ms latency round trip.
Talk about client side vs cloud render and compute
This is because we had a need to render realer and realer (show pac man to photorealistic render). (tie this to NVIDIA)
VR and AR has pushed the boundaries of what is minimally acceptable. Instead of the age old 60hz in games and 29.97 fps in film we now need to hit 90 and 120hz refresh. From 1080p to 4k to 4k per eye for resolution. And now we have newer constraints of motion to photon latency, field of view, stereoscopy and hardware size (show death to the dork helmet).
Image Source - Joey Kyber https://unsplash.com/search/fast?photo=45FJgZMXCK8
So how do we get there?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
15
Compute and render must either be performed on the CPU, GPU or Cloud. A combination of the three is the most likely.
We need far more compute than we possibly imagined and this is probably not an option for the long term.
Talk about client side vs cloud render and compute
This is because we had a need to render realer and realer (show pac man to photorealistic render). (tie this to NVIDIA)
Image source: pc world
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
16
So how do we get there?
There is a fundamental need on the network and the GPU to handle rending and compute for the foreseeable future
Source: blogs.nvidia.comSource: ericsson 2014
The other need that grew was on the network. We needed more bandwidth and less latency. (show bandwidth changes) (tie this to verizon)
Currently, these big problems in our industry exist in the form of computer vision (gpu), render (gpu) and transmission (network constraint). To get to the real future we need to solve essential problems in bandwidth, latency and graphics.
How can we solve these problems
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
17
For the near term, we need to push form factor to gain adoption. This means that compute and rendering needs to move off glass.
Due to size constraints and the need for a headset or eyewear to be truly accepted, it can’t be a dork helmet. We need to create eyewear that is natural and as a result we need better, smaller gpu’s and a big chunk of the gpu compute needs to off the headset into nearby compute units (show pictures of pocket packs, backpacks) or the cloud (show pictures of cloud GPU).
Image source - https://unsplash.com/search/glass?photo=mlQ_4Ff6YSM - mel baylon
How can we solve these problems
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
18
Due to size constraints and the need for a headset or eyewear to be truly accepted, it can’t be a dork helmet. We need to create eyewear that is natural and as a result we need better, smaller gpu’s and a big chunk of the gpu compute needs to off the headset into nearby compute units (show pictures of pocket packs, backpacks) or the cloud (show pictures of cloud GPU).
Image source - https://unsplash.com/search/glass?photo=mlQ_4Ff6YSM - mel baylon
What did Verizon Labs want to do?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
19
Bridge content, advertising, network, enterprise and retail with a new unified initiative where we can build the future of our company from scratch.
We set out with the challenge of defining a unifying play between our various business units where we wanted to combine our strengths as a top tier cellular and fios network, our 3rd
largest digital advertising network, stable of content companies and partners, premier streaming solution for media providers and other business and develop a platform for the future.
We set out to build a solution that powers virtual and augmented reality and leverages our strengths
What did Verizon Labs want to do?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
20
• A multi-platform solution for streaming VR, AR and 360 video experiences • Help users create 3d and 2d content using easy to use tools • Live-stream or sync your VR & AR solutions with all your users • Monetize your content and connect advertisers with potential consumers through relevant ads inside and
outside your headset
We see the future of VR and AR as starting with top tier digital experiences in the film of passive and interactive storytelling and games and the aspects of AR that includes computer vision, changing the world around you and sharing them.
We believe the future is in VR and AR. That it is the next computing platform.
To make this a reality we need to enable creators to create worlds and tell stories as easily as possible.
Envrmnt is a cloud solution and a rendering engine that allows content creators to build interactive worlds and stream them to customers. It is intended to be a turn-key approach to building a mesh-based world or streaming an environment in the form of 360 video or volumetric scene data and gives content creators a way to easily monetize those experiences.
Image Source - Eric Azares
What did Verizon Labs want to do?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
21
• Accommodate all capture solutions and media • Allow for projection or augmentation • Live-stream and live-stitch • Build worlds and connect your audience • Bring traditional advertising into the real world • View your content anywhere • Deliver it fast and seamlessly • Much, much more!
We set out then, to build a feature set as a suite of tools that allows traditional storytellers to move their media forward
Image Source - Eric Azares
Where did we address those pesky bottlenecks?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
22
• Accommodate all capture solutions and media • Allow for projection or augmentation• Live-stream and live-stitch• Build worlds and connect your audience • Bring traditional advertising into the real world • View your content anywhere • Deliver it fast and seamlessly• Much, much more!
To build a robust set of features that delivered high quality across platforms we strategically separated tasks that were sequential and parallel
Many aspects were optimized away using GPU’s either on device or in the cloud.
Image Source - Eric Azares
Rendering
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
23
GPU providers are doing their part.
Nvidia VR Works is providing the following functionality: • Lens Matched Shading • Single Pass Stereo • MultiRes Shading • VR SLI • Context Priority • Front Buffer Rendering • Positional Audio
VR Works
Rendering
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
24
Insert benchmarks here
From VR Works SDK we utilized the following functionality: • Lens Matched Shading• Single Pass Stereo
VR Works
Live stitching
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
25
Our stitching pipeline utilized CUDA and GPU parallelization in order to optimize our stitching pipeline
4x decoding speed 10x speed for Warping, Blending and Un-distortion in comparison to CPUs
10x overall performance gain vs similar class CPU
Auto stitching is a feature that allows film production to use many different cameras for single spherical view. In a live workflow there is quite a bit of compute required to transform, align, feature match and blend these views into a single very high res video.
(compare the need for high resolution for each view and eye)
All operations of stitching - decoding, undistort, warping and blending performed way better on GPU. call out decoding
Live stitching
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
26
All operations of stitching - decoding, undistort, warping and blending performed way better on GPU. call out decoding
Encoding
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
27
While incorporating VR Works we observed the following
Insert benchmarks here
VR Works
Demo!
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
28
Demo
Where do we think there is still work to be done?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
29
As network technology develops on it becomes more of a possibility to offload GPU compute tasks to the cloud.
Future enhancements of low latency high bandwidth connections should consider optimizations for graphics processing
Delivery, cdn, 5G
Image Source - Tomy Tong - https://unsplash.com/search/cell-phone?photo=Wa6LnHpFhIM
So to recap!
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
30
• A multi-platform solution for streaming VR, AR and 360 video experiences • Help users create 3d and 2d content using easy to use tools • Live-stream or sync your VR & AR solutions with all your users • Monetize your content and connect advertisers with potential consumers through relevant ads inside and
outside your headset
We see the future of VR and AR as starting with top tier digital experiences in the film of passive and interactive storytelling and games and the aspects of AR that includes computer vision, changing the world around you and sharing them.
We believe the future is in VR and AR. That it is the next computing platform.
To make this a reality we need to enable creators to create worlds and tell stories as easily as possible.
To enable these experiences it is vital to maximize the use of GPU compute for render, vision, and other tasks that can be parallelized. In a perfect future this implies offloading from consumer hardware to GPU’s in the cloud and on the edge.
Image Source - Eric Azares
Thank you!
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
31
• Contact Raheel Khalid • Email - [email protected] • Twitter - @rkhalid890
We’re Hiring!• Video Encoding Algorithm Architect • 3D Computer Vision and SLAM Engineers • Volumetric Reconstruction Engineers • Numerical Mathematicians / Algorithms Engineers • CUDA Experts • OpenGL Rendering Experts • Located in NJ, NY, & San Francisco with satellite teams all over the US!
Image Source - Eric Azares
Never know if you need a generic placeholder slide…
