Creating Photo Sequences Using Angled Graphs

J. ArmourMAS 531 Final ProjectFall 2010

Main Idea Large photo collections of the same event

or the same object or place are common with the proliferation of camera phones, digital photography, and easy storage.

Imagine being able to take such a data base and automatically sort them and output a smooth transition movie of that object or event.

Input: Unsorted Photo Collection

Calculate: Angled Graph

Output: Smooth Image Sequence

Additional Applications Sporting event imagery. Paparazzi photography. Tourist or security camera photography of same

location. Data taken of the same scene with varying time of

day, time of year, weather conditions, dynamic content, etc.

Use this method to decompose a video and remove camera jitter, but keep the appearance of smooth transitions frame to frame.

Use the graph to find circuits, creating infinite video loops (specific data sets needed with repeative motion, nearly identical backgroud. Example: flying bird)

MIT Dome, Summer vs. Winter

Current Methods Photo Tourism work uses online image collections along with

structure from motion and planar proxy techniques to generate an browse-able photo collection. Method first builds and understanding of a scene and then picks out sequences

of images based on a series of selected camera poses. SIFT, KLT, SURF, and GIST methods can all be used for matching

images based on features. Methods are computationally intensive and not robust to certain changes in

imagery, for example changes in viewing angle > 20 degrees.

Isomap, PCA, and MDS methods, involving image manifolds, have also been used to work with image sorting.

Future work needed comparing angled graph method to these previous methods .

Overview of the Process Step 1: Determine image feature vectors based on image histogram, gradients,

colors, etc. Step 2: Use feature vectors to calculate weighted graph of image distances.

Given image m,n, have Fm, Fn , then:

Step 3: Calculate angles between adjacent edge of weighted graph.

)()( nmT

nmnmmn FFWFFwFFd

Step 4: Convert angles and edge pairs to a new weighted graph, G*

Step 5: Threshold the G* to reduce the number of possible paths in the data, keeping only the “straightest” paths

Step 6: Calculate possible minimal paths in remaining data.

Step 7: Turn straightest path into an image sequence.

Contributions We introduce a novel graph

structure to summarize the relationship between sets of 3 images and the smoothness of their transitions.

We use this information to compute global optimal paths in the imagery starting from a given image, forming a image sequences.

Algorithm involves entirely image-based processing.

Does not reconstruct structure, shape, or geometry.

Method is near real-time. Can handle non-geometric changes

such as camera focus. Extracts paths through images (camera poses, lighting conditions, etc.)

without expensive pre-processing steps to build scene understanding. Can handle a 360⁰ angular viewing of objects.

Sample Weighted Graph G*

Further Thoughts

What Worked: Implemented the algorithm except step 6. What Failed: Lack of understanding of shortest path algorithms and

the correct way to find global minimum paths. Hoping to have sorted out possible coding errors by Tuesday.

Acknowledgements: Project idea is 100% derived out of Camera Culture Group SIGGRAPH 2010

submission entitled “Personal to Shared Moments with Angled Graphs of Pictures”. Figure above is their initial vision for the project.

Previous work done by Prof. Gabriel Taubin, Kevin Chiu, Tom Cuypers, and Ramesh Raskar.

My Contribution: Implemented the idea in Matlab based on the rough draft paper, took more data sets for testing, implemented variation on initial

Limitations Scalability could be a problem as method involves computation

of n2 image distances, n3 angles from feature vectors of size m x 1 for each of the n images in a data set. Included figure showing quadratic trend in computation time needed

(without the shortest path step implemented) versus number of frames.

Shortest path calculations, in our case Floyd-Warshall, are generally O(n3). We’ve thresholded the results to try to

make up for this, but need to look into the actual cost of this step.

Further testing needed to understand robustness of the method to feature vector contents and weights.

Future Work Modification of system (weights or feature

vectors) to allow images from different cameras to be sorted.

Testing for robustness of system to different variations in data: ie lighting conditions, horizontal/vertical camera translation, camera rotation, zoom, focus, differing camera, etc.

Comparison of robustness of the system to that of previous methods.

Comparison of memory and computational cost of system to that of previous method.

Test system on data taken of same object at different angles but with static background. Example: Google Embryo data.

Look at intersections of paths, creating smooth transitions to data varying in different ways (eg. if using stanford face data, find head shaking left to right and smoothly transition into nodding up and down.)

Summary

We’ve proposed a new method for sorting images using angular information

This allows us to determine nearest-neighbor relationships.

We use these nearest-neighbors to create videos with smoothest possible image transition from a given data sets.

We suggested some possible applications and future work needed.