Post on 03-Jan-2016
transcript
ComputerVisionLabSeoul National University
Keyframe-Based Real-Time Camera Tracking
Young Ki BAIK
Vision seminar : Mar. 5. 2010
Computer Vision Lab.Computer Vision Lab.
ComputerVisionLabSeoul National University
Reference
Keyframe-Based Real-Time Camera Tracking
- ICCV 2009- Zilong Dong, Guofeng Zhang, Jiaya Jia Hujun Bao
ComputerVisionLabSeoul National University
Outline
Introduction
System overview
Offline map building
Online camera tracking
Demonstration
Conclusion
Opitimal keyframe selection
Keyframe recognition using vocabulary tree
How to use multiple thread
ComputerVisionLabSeoul National University
What is the purpose?Real-time camera tracking
Known internal camera parameters The map of environment is given…
6-DOF camera pose estimation!!
ComputerVisionLabSeoul National University
System overviewReal-time camera tracking
Novel keyframe selection algorithm
How to build a map
How to estimate the camera pose
Offline mapping using the conventional MVG method
Online camera tracking
Efficient key frame recognition method
Parallel computing framework
ComputerVisionLabSeoul National University
Offline mapping stage
reference images
Image Acquisition Video data for the target environment
ComputerVisionLabSeoul National University
Offline mapping stage
Image Acquisition
Feature Extraction and Matching
SIFT descriptors
ComputerVisionLabSeoul National University
Offline mapping stage
Image Acquisition Conventional MVG
Feature Extraction and Matching
3D Map Building
ComputerVisionLabSeoul National University
Offline mapping stage
Keyframe set
Image Acquisition Keyframe selection
Feature Extraction and Matching
3D Map Building
Optimal Keyframe Selection For efficiency of map management
ComputerVisionLabSeoul National University
Offline mapping stage
Image AcquisitionVocabulary tree
Feature Extraction and Matching
3D Map Building
Optimal Keyframe Selection
Vocabulary Tree Construction
For fast keyframe selection
in tracking stage…
Contribution!!
ComputerVisionLabSeoul National University
Why keyframe?
Which one is good for a map?
To solve the scalability problem of the map
All references Keyframe
A small number of …
ComputerVisionLabSeoul National University
Optimal Keyframe Selection
select manually…
using heuristic method…
select optimal keyframe
Containing many salient features as possible
Reducing non-distinctiveness in matching
Features should be distributed evenly in keyframes
ComputerVisionLabSeoul National University
Optimal Keyframe Selection
Problem formulation… to find optimal
keyframes
Completeness Term Redundancy Term
weight
Saliency of features
Distribution of features
Distinctiveness
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Definition
Reference images
Keyframes
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Definition
Feature clusters
Reference image set
2D image position
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Definition
Set of superior features
~Long-term surviving (or representative) features in reference
images
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Saliency
~
The larger s(χ) means the higher saliency!
DoG map
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Distribution
The smaller d (χ)→ Features are well distributed !!
Feature density
31x31 window
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
=3 : controls the sensitivity of feature density…: the superior feature set in the keyframe
set F
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionRedundancy Term
NormalizationThe number of
keyframes which contain feature χ
Small Er(F) indicates low redundancy!
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
Let = 0Initialization
~
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Add Keyframe
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Add Keyframe
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Check
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Add Keyframe
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Add Keyframe
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Check
ComputerVisionLabSeoul National University
Vocabulary Tree Construction
For fast keyframe recognition…
Root node
List
Branch = 8, Tree depth = 5
All FMean patch
All feature descriptor
ComputerVisionLabSeoul National University
Vocabulary Tree Construction
To build each node…
Root node
List
Clustering by K-means method
Branch = 8, Tree depth = 5
ComputerVisionLabSeoul National University
Vocabulary Tree Construction
To build each node…
Root node
List
Clustering by K-means method
List List
Computing mean patchBuilding keyframe list
Branch = 8, Tree depth = 5
ComputerVisionLabSeoul National University
Vocabulary Tree Construction
Finally…
Root node
List
Clustering by K-means method
List List
Computing mean patchBuilding keyframe list
List List List List
Branch = 8, Tree depth = 5
ComputerVisionLabSeoul National University
Online tracking stage
SIFT Feature Extraction
SIFT feature
Input image
ComputerVisionLabSeoul National University
Online tracking stage
SIFT Feature Extraction Vocabulary tree
Keyframe Recognition
ComputerVisionLabSeoul National University
Online tracking stage
SIFT Feature Extraction
Keyframe Recognition
Keyframe-based matching
Input image Selected F
ComputerVisionLabSeoul National University
Online tracking stage
SIFT Feature Extraction
Keyframe Recognition
Keyframe-based matching
Camera Pose Estimation
3-point algorithm
3-D map
2-Dpoint
ComputerVisionLabSeoul National University
Keyframe recognition
With SIFT patches of input image…
Root node
List List
List List List List
Matching input patch & mean
patches
Vocabulary tree Key frame voting memory
ComputerVisionLabSeoul National University
Keyframe recognition
With SIFT patches of input image…
Root node
List List
List List List List
Key frame voting memory
Move patch And Update voting memory
ComputerVisionLabSeoul National University
Keyframe recognition
With SIFT patches of input image…
Root node
List List
List List List List
Key frame voting memory
Select keyframe with largest voting value
ComputerVisionLabSeoul National University
Processing time
Process time per frame with a single thread…
ComputerVisionLabSeoul National University
Processing time
Using multiple-thread with pipeline
SIFT Feature
Extraction
Time axis
Select Keyframe
Keyframe
Matching
SIFT Feature
ExtractionPose
Estimation
Thread#1
Thread#2
Thread#3
Thread#4
Select Keyframe
Keyframe
Matching Pose Estimation
20 fps
First frame istime consuming!
ComputerVisionLabSeoul National University
Demonstration
ComputerVisionLabSeoul National University
Conclusion
Real-time camera tracking is presented !!
The optimal keyframe selection algorithm
Real-time tracking is presented!
Minimizing the energy formulation!
Fast keyframe recognition using vocabulary tree
Multiple thread with pipeline structure
LimitationIncremental updating of VT is not considered…
Proposed process is only available for known environment.
ComputerVisionLabSeoul National University
Q & A