IIIT
Hyd
erab
ad
Geometry Directed Browser for Personal Photographs
Center for Visual Information Technology
IIIT Hyderabad
Aditya Deshpande, Siddharth Choudhary, P J Narayanan, Kaustav Kundu, Krishna Kumar Singh, Aditya Singh, Apurva Kumar
IIIT
Hyd
erab
ad
We use SfM and other 3D computer vision techniques to provide intuitive Geometry Directed Photo Browsing.
Photo-Browsing
• Digital Photography - No hard copy - Capture photographs and relive later on display device
• Photo-Browsers are tools to view digital photographs. E.g. Windows Photo Viewer, iPhoto, FSpot, KSquirrel etc.
•Photo Browsing model has notevolved much.
IIIT
Hyd
erab
ad
Related Work
• Face Detection & Tagging on Social Networking Sites.
• [Zhang et al. MM’03], Automatic annotation of family albums.
• [Davis et al. MM’05], Additional contextual data viz. time of capture, geo-tag, indoor/outdoor scene, co-occurring faces.
Above techniques only improve photo-browsing experience of social engagements.
IIIT
Hyd
erab
ad
Our Goal
• Apart from social engagements, a large chunk of user’s personal photographs consist of tourist places & monuments.
• [Snavely et al. IJCV’08, SIGGRAPH’06] (Photosynth) - CPC Storage, local reconstruction to add new camera’s
• Choudhary et al., Li et al., Sattler et al., Irschara et al. etc. - Localize new query images w/o exhaustive search.
We combine SfM-Reconstruction + Localization to provide intuitive browsing of user photos in 3D space of the monument.
IIIT
Hyd
erab
ad
Assumptions
• Our target platform is an off-the-shelf laptop or a desktop.
• User is expected to click around 5-50 photographs for a particular monument.
• The system should localize these user photographs in areasonable time.
• The system should provide a smooth visualization / transitions of all user photos and ~105 points of the monument.
IIIT
Hyd
erab
ad
System Design(1) Heavy SfM Reconstruction done offline in the cloud
(2) GDBPackage : reconstruction + addnl. information downloaded to local disk
(3) User uploads personal photo’s through a camera / phone
(4) System registers user’sphotos to the point cloud andprovides 3D visualization.
IIIT
Hyd
erab
ad
System Block Diagram
GDBPackage
User Photos
RegistrationModule
VisualizationModule
System is divided in two parts : 1. Registration / Localization Module2. Visualization Module
2 1
EstimatedCamera’s
IIIT
Hyd
erab
ad
Localizing User Photo’s
• Trivial if photograph is taken from GPS enabled deviceand is geo-tagged!
• What if no geo-tag information?
• Two Localization Approaches :Image based search in a geo-tagged Image Dataset[Panda et al.] Geo-locate digital heritage site photos.
Using structure information in SfM Dataset[Irschara et al. CVPR’09], match to nearby similar images.[Li et al. ECCV’10], visibility prioritized 3D-2D matches. [Sattler et al. ICCV’11, ECCV’12], visual words to find 2D-3D matches.
IIIT
Hyd
erab
ad
Localization - Choudhary et al.
• [Choudhary et al. ECCV’12]- Triangulate a seed point in the user photograph.- Further 3D-2D search is guided by visibility probabilities.- Find ~20 independent matches.- Use RANSAC to estimate camera parameters.
Probability Guided 3D-2D correspondence
3D Position
Up Vector
View Direction
IIIT
Hyd
erab
ad
Advantages of Localization Method
• Data for Localization is stored in GDBPackage : (1) Cover Set (2) Visibility Matrix (3) Bi-Partite Visibility Graph
• CPC images need not be stored, data requirements are minimal.
• The method is fast and localizes images at the rate of 1sec/photo.
IIIT
Hyd
erab
ad
Non-Localizable Photographs
• In some cases the images lack sufficient monument geometryfor localization to work :- Occluded by people.- Noisy images of nearby scenery/smaller monuments.- Zoomed in images of smaller monument structures etc.
Zoomed In View of Small Structure
(Pantheon Dataset)
Completely Occludedby People
(Colosseum Dataset)
IIIT
Hyd
erab
ad
Non-Localizable Photographs
• Photographs have time of capture stored in their EXIF-tags.
• A non-localized image is placed at a position that is weightedaverage of its immediate known predecessor and immediateknown successor in time.
• Similarly, linear interpolation is also done for the view-direction vector to get the complete camera pose.
• The above method will not give the exact location, but placing it in temporal neighborhood suffices for display purposes.
IIIT
Hyd
erab
ad
Visualization Module
• 3D Viewer
• Mouse Navigation
• Button Navigation
• Add Screenshot
• Delete Path
• Generate Photo-Tour
• 2D Viewer
IIIT
Hyd
erab
ad
3D Photo Browser : Geometry Directed Photo-Browsing
• Initial Mode : 3D Model and small preview (thumbnails) of user photographs.
• Select Mode : Animate to clicked photo and detailed view.
• Linear quaternion interpolation of Rotation Matrix for smoothtransitions between images.
• Smooth transitions give a feel of the geometric space of the monument.
IIIT
Hyd
erab
ad
3D Photo Browser : Generating Custom Photo Tours
• User can save the current viewpoint (“Add Screenshots”)
• Once a set of viewpoints are saved, he can smoothly animateover viewpoints. (“Generate Photo-Tour / Animate Path”)
• User can delete the viewpoints and generate a new photo-tour.
• Photo-Tours are a good way to creatively view personalphoto’s taken at a tourist place.
IIIT
Hyd
erab
ad
ResultsMonument # Photos # Registered
PhotosReg. Time
(secs per photo)Colosseum 24 21 1.01Colosseum 19 14 0.97Pantheon 35 22 1.13
Stone Chariot(Hampi)
17 17 1.07
(a) Localization Module
(b) Visualization Module
IIIT
Hyd
erab
ad
Conclusion and Future Work
• Minimal System Requirements.
• Intuitive 3D Visualization of User Photographs.
• Pipeline for 3D personal photo-viewing from SfM reconstruction.
• Port our system to a mobile phone and have a touch/gesture interface.
3D Photo-Viewing & Localization App
IIIT
Hyd
erab
ad
Thank you. Questions?More Results
(a) Hampi Dataset (Stone Chariot)
(b) Pantheon Dataset
IIIT
Hyd
erab
ad
Platform Details
Item Specification
CPU Intel ® CORE ™ i5
Clock Speed 2.44GHZ
RAM 4GB
GPU Intel ® HD Graphics Accelerator