Copyright © 2016 Your NVIDIA 1
NVIDIA VisionWorks, a Toolkit for
Computer Vision Elif Albuz
May 3, 2016
Copyright © 2016 Your NVIDIA 2
Computer Vision
Intelligent Video Analytics
Augmented Reality
Robotics Drones Autonomous Driving
Computational Photography
Copyright © 2016 Your NVIDIA 3
Computer Vision Application Development
Concept
Reference Implementation
Productization
Port to target & optimize
Copyright © 2016 Your NVIDIA 4
• Deliver high performance, robust
computer vision primitives
• Ease development of computer
vision applications on Tegra
embedded platforms
• Accelerate prototype to product cycle
VisionWorks Motivation
Depth Map
Optical Flow
Corner detection
Copyright © 2016 Your NVIDIA 5
VisionWorks at a Glance
Thread-safe API
Documentation, tutorials, sample software pipelines that teach use of primitives and framework
CUDA accelerated library
(OpenVX primitives + NVIDIA extensions + Plus Algorithms)
Flexible framework for seamlessly adding user-defined primitives. Interoperability with OpenCV
Copyright © 2016 Your NVIDIA 6
VisionWorks Toolkit Software Stack
CUDA Acceleration Framework
OpenVXTM API
VisionWorks
API Extensions
OpenVXTM Framework
VisionWorks
Framework
Extensions
VisionWorks
SfM
NVIDIA
Khronos
VisionWorks
Core Library
Source
Samples
VisionWorks Source Samples Feature Tracking, Hough Transform, Camera
Hist Eq., Stereo Sample
NVXIO
Multimedia
Abstraction
VisionWorks-
Plus
VisionWorks
Object Tracker . . .
VisionWorks
CUDA API
Copyright © 2016 Your NVIDIA 7
JETSON TK1 Pro JETSON TK1
VisionWorks Supported Platforms
Ubuntu Linux 12.04,
Ubuntu Linux 14.04,
Windows 8
Drive PX JETSON TX1
Automotive Embedded Desktop
Copyright © 2016 Your NVIDIA 8
VisionWorks Primitives
IMAGE
ARITHMETIC Absolute Difference
Accumulate Image
Accumulate Squared
Accumulate Weighted
Add/ Subtract/ Multiply +
Channel Combine
Channel Extract
Color Convert +
CopyImage
Convert Depth
Magnitude
MultiplyByScalar
Not / Or / And / Xor
Phase
Table Lookup
Threshold
FLOW & DEPTH Median Flow
Optical Flow (LK) +
Semi-Global Matching
Stereo Block Matching
IME Create Motion Field
IME Refine Motion Field
IME Partition Motion Field
GEOMETRIC
TRANSFORMS Affine Warp +
Warp Perspective +
Flip Image
Remap
Scale Image +
FILTERS BoxFilter
Convolution
Dilation Filter
Erosion Filter
Gaussian Filter
Gaussian Pyramid
Laplacian3x3
Median Filter
Scharr3x3
Sobel 3x3
FEATURES Canny Edge Detector
FAST Corners +
FAST Track
Harris Corners +
Harris Track
Hough Circles
Hough Lines
ANALYSIS Histogram
Histogram Equalization
Integral Image
Mean Std Deviation
Min Max Locations
NVIDIA
extensions
All OpenVX
Primitives
+ type/mode extension by NVIDIA
NVIDIA extension primitives
Copyright © 2016 Your NVIDIA 9
Feature
Tracker Stereo Depth
Extraction
OpenCV-NPP-OpenVX Interop
Hough Lines &
Circles
• + Video stabilization
• + Iterative Motion Estimation/Flow
and other platform specific samples (available only on certain platforms)
• Camera Capture, OpenGL interop, Video playback
VisionWorks Sample Applications
Copyright © 2016 Your NVIDIA 11
• VisionWorks provides 3 levels of API
• VisionWorks OpenVX™ Immediate Mode
• Standard specified heterogeneous compute API with individual
function calls
• VisionWorks OpenVX™ Graph Mode
• Heterogeneous compute API with graph optimizations
• Extensible with user defined nodes
• VisionWorks CUDA API
• Direct CUDA API for advanced CUDA developers
VisionWorks Programming Model
Copyright © 2016 Your NVIDIA 12
• VisionWorks OpenVX Immediate mode API enables developers to come up to speed fast
• Started with original OpenCV implementation of VideoStabilization application
• Ported to VisionWorks Immediate Mode (prefixed as nvxu)
• Outcome: Video stabilization application is accelerated by 4.75x
(including the overhead for Mat to vx_image conversions)
VisionWorks OpenVX™ Immediate Mode
Video Stabilization
Color Conversion
Optical Flow
Stabilized
frames
Cv::Mat to Vx_image
Process pts & Find
Homography
Warp Perspective
OpenCV image
Source Feature
detection
Image Pyramid
1.3x
2.3x
2.6x
10.1
x 1.5x 8.4x
Copyright © 2016 Your NVIDIA 13
OpenVX - fusion
• OpenVX Graph enables further optimizations
• Buffer reuse
• Kernel fusion
• Developer controlled CPU-GPU execution
• Efficient use of streaming and CUDA textures
VisionWorks OpenVX™ Graph Enabled
Structure From Motion
Color Conversion
Channel Extract
Optical Flow
Harris Track
Image Pyramid
Point
Clouds
Frame Capture/
Processing
Camera Motion
Estimation
Two-view Triangulation
Scale Estimation
Frame
Source
Copyright © 2016 Your NVIDIA 14
• VisionWorks CUDA API enables developer with low-level access
• Developer manages the data allocations and transfer
• Developer identifies efficient scheduling and pipelining
• Note that this API operates on the GPU, and does not allow switching between CPU and
GPU
VisionWorks CUDA API
Feature Tracking
YUV
frame
Gray
frame
Camera/image/video
Input data Rendering/Output
nvxcuColor Convert
nvxcuChannel Extract
nvxcuOpticalFlowPyrLK
nvxcuHarris Track
nvxcuGaussian Pyramid
RGB
frame as
CUDA
buffer
Array of
keypoints
Copyright © 2016 Your NVIDIA 17
• VisionWorks Toolkit enables demos and applications, shortens
productization time
• OpenVX immediate mode enables quick port from other libraries
• OpenVX graph mode brings advanced optimizations, hiding overheads of
the target platforms
• VisionWorks CUDA API enables users with lower level access
• Heterogeneous API which enables switching from GPU to CPU version of
the pipeline is very powerful, reducing productization time
Conclusion
Copyright © 2016 Your NVIDIA 18
• http://www.embedded-vision.com/
• https://www.khronos.org/openvx/
• https://developer.nvidia.com/embedded/visionworks
Resources