Heterogeneous Computing in 3D Perception Chris Slaughter, CEO Jeff Mahler, Computer Vision
Architecture
Video Games / AR
Visual Effects
Robotics
Architecture: Convenient 3D capture from a moving camera
Video Games / AR: Texture capture, live feed augmentation
Robotics: Capture, semantic interpretation, perception for manipulation
Visual Effects: Fast 3D capture for compositing, animation, lighting effects
All of these problems are
3D Perception. Perception - awareness of the elements of environment through physical sensation
3D Perception - awareness of the elements of environment through physical sensation in 3D
Input Output
3D perception tasks can be done in
Real Time.
3D Rendering
Application Scene Cull Draw Rendering
3D Perception
Cognition Scene Understanding
Infer Estimate Sensors
Today’s Agenda
Techniques • Motion Tracking • 3D Capture • Global Refinement • 3D Image Processing
Applications •Bomb Detection •Lynx A Camera
Future of Heterogeneous Computing!
Motion Tracking
Data Source Representation
Data Correspondence
Error Minimization
Objective Linearization A
x
b
-
||Ax – b||2
Normal Equations Parallel Reduction
A
x
b
- AT AT
||ATAx – ATb||2
Color
Position
Curvature
Orientation
3D Capture
Data Source Representation
Tracking
Refinement
Techniques
System Representation Tracking Refinement
KinectFusion1 Volumetric Projective Point-
Plane ICP Truncated Signed
Distance
Dense Tracking and Mapping2 Volumetric / 2.5D
Lucas-Kanade Alignment
None
Patchwork3 2.5D Projective Point-
Plane ICP Kernelized
Stochastic Descent
1. Richard Newcombe / Microsoft; 2. Richard Newcombe; UT Austin & Lynx Labs
Tracking
Refinement
Texture Reads (4x)
Surface Writes (1x)
Global Refinement
Raw Data Post Processing
Trajectory Pose Graph
Reprojection Error
Fast Data Correspondences
Sparsity in Adjacency Matrix
3D Image Processing
Anisotropic Diffusion
Fast Digital Inpainting
Harris Corner Detection
λ1 indicates the presence of an edge in the x-direction
λ2 indicates the presence of an edge in the y-direction
Bomb Detection
Pre-Captured 3D Model
Live Dense 3D Feed
- =
No Bomb Bomb
Lynx A Camera
Conclusion
• Many emerging applications for 3D perception
• Heterogeneous computing enables real-time applications
• GPU architecture is ideal for 3D
perception processing