15
PROJECT TANGO AND TEGRA K1 BRINGING SPATIAL AWARENESS TO MOBILE Neil Trevett (Vice President of Mobile Ecosystems, NVIDIA) Wil Braithwaite (Senior Applied Engineer, NVIDIA)
PROJECT TANGO AND TEGRA K1 BRINGING SPATIAL AWARENESS TO MOBILE
Neil Trevett (Vice President of Mobile Ecosystems, NVIDIA) Wil Braithwaite (Senior Applied Engineer, NVIDIA)
© Copyright NVIDIA 2014 - Page 2
Visual Computing = Graphics AND Vision Processing
Real-time GPU Compute Research project on CUDA-enabled laptop
High-Quality Reflections, Refractions, and Caustics in Augmented Reality and their Contribution to Visual Coherence
P. Kán, H. Kaufmann, Institute of Software Technology and Interactive Systems, Vienna University of Technology, Vienna, Austria
https://www.youtube.com/watch?v=i2MEwVZzDaA
Imagery
Data
Vision Processing
Graphics Processing
Enhanced sensor capability deepens
the interaction between real and
virtual worlds
https://www.youtube.com/watch?v=i2MEwVZzDaA
© Copyright NVIDIA 2014 - Page 3
Mobile Visual Computing Enables New Experiences
Augmented Reality
Face, Body and Gesture Tracking
Computational Photography and
Videography
3D Scene/Object Reconstruction
Need for advanced sensors and the GPU throughput to process them
© Copyright NVIDIA 2014 - Page 4
Tango – Developer Platform for Advanced Sensors
NVIDIA Tegra K1 processor 4GB of RAM 128GB of storage 1080p display Stock Android 4.4 WiFi, Bluetooth LE and 4G LTE 120 degree front camera 4MP Camera and Depth Sensor Motion-tracking Camera $1,024
Google ATAP Project to seed and inspire developers
- similar to Google Glass
Tango is intended to inspire and catalyze innovative applications
© Copyright NVIDIA 2014 - Page 5
RGB/IR Sensor + Flash
IR Projector
Wide-Angle Odometry
What is special about Tango? Knows much more about its environment than normal mobile devices Advanced visual sensors – and Tegra K1 350GFlop GPU to process them It can compute its position (‘pose’) It can sense objects and construct 3D models of them
© Copyright NVIDIA 2014 - Page 6
Visual Odometry Accurate Pose Estimation
Position (x,y,z) Pointing direction (i, j, k, rotation) Uses feature tracking and SLAM
Accuracy:
© Copyright NVIDIA 2014 - Page 7
Depth Sensor Depth information for many points on a 2D picture for:
3D geometry and model (Mesh) Body position and movement Object recognition
Mantis Vision Structured Light projector iR image captured by 4MP Camera
GPU accelerated processing Analyses structured pattern deformation to extract depth information
© Copyright NVIDIA 2014 - Page 8
Tango Tablet Hardware Architecture No external image processors needed
Reduces cost, power, latency
Odometry 30 FPS ~3-5ms GPU processing/frame GPU load: ~15%
Depth Decoding 5 FPS ~13-15ms GPU processing/frame GPU load: ~8%
Sensor Hub
Camera RGBiR 4MP
Camera VGA BW
Wide Angle
Tegra K1
MEMS Sensors
IR Projector
CSI
CSI
SPI
Strobes
Camera Front
120 Degree
CSI
© Copyright NVIDIA 2014 - Page 9
Tango Software Pipeline GPU processed sensor data made available to applications
Java interfaces or native access, together with 3D engines such as Unity
NVIDIA VisionWorks can be used for native vision processing Library of GPU accelerated vision primitives Use OpenVX for optimized execution of graphs of primitives
Time-stamping
Laser Control
Inertial Sensor Stack
Sensor Hub SW
Camera Core HAL V3
CUDA Depth
Decoding
CUDA
Feature Tracking
Odometry
Client
Depth Camera Client
Camera Control Use standard Android APIs GPU Sensor Processing
© Copyright NVIDIA 2014 - Page 10
Metaio - IKEA Furniture Catalog Select catalog item, and display in own home
1 million users in Europe
Depth camera enhances experience Absolute measurements and ground plane detection with no marker Environmental lighting Occlusion
© Copyright NVIDIA 2014 - Page 11
Matterport - Interior Space Capture Real Estate, Training, Historical Preservation Use dedicated camera now
Uses Kinect 5 degree drift after 360 degrees
Tango sensors will enable widespread adoption
© Copyright NVIDIA 2014 - Page 12
DotProduct3D 3D Model Capture Integrated sensors increase quality and make 3D scan capture devices ubiquitous
© Copyright NVIDIA 2014 - Page 13
3D Descriptor Database
With Consumer Capture – 3D Will Go Social!
Object Capture
Upload, View, Share and Comment
Inspire and Motivate
3D Printing (e.g. shapeways.com)
Social Loop Photos -> Facebook Videos -> YouTube
3D -> ?
Manufacturers provide 3D Object Descriptors - much
more information than Amazon-style 2D search
Search
© Copyright NVIDIA 2014 - Page 14
Demos!
© Copyright NVIDIA 2014 - Page 15
So Where Do I Get A Tango? Sign up at the Tango Website
https://www.google.com/atap/projecttango/#devices
Talk to NVIDIA Neil Trevett [email protected]
https://www.google.com/atap/projecttango/mailto:[email protected]
Slide Number 1Visual Computing = Graphics AND Vision ProcessingMobile Visual Computing Enables New ExperiencesTango – Developer Platform for Advanced SensorsWhat is special about Tango?Visual OdometryDepth SensorTango Tablet Hardware ArchitectureTango Software PipelineMetaio - IKEA Furniture CatalogMatterport - Interior Space CaptureDotProduct3D 3D Model CaptureWith Consumer Capture – 3D Will Go Social!Demos!So Where Do I Get A Tango?
