Yuping Lin and Gérard Medioni

Introduction Method

Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic

Synchronization of multiple video streams Conclusion

Input: Multiple UAV video streams Position of moving objects in each video

stream

Goal: Synchronize using a common moving object

Register UAV streams to a global reference image (a map), then

Synchronize the streams using the unique path of a common moving object on the map

Introduction Method

Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic

Synchronization of multiple video streams Conclusion

Input: Global reference image (Map) UAV stream The homography of the first frame of the

UAV stream to the map

ISSUES

UAV images and the map are different in terms of viewpoints, sensors and time of capture

Direct matching is difficult

APPROACH

Given the homography of the first UAV frame to the map,

Two step registration Consecutive UAV image

registration, then UAV to Map registration

Introduction Method

Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic

Synchronization of multiple video streams Conclusion

Method: extract features in each frame Establish feature correspondences

between consecutive images estimate the transformation

ISSUES

Features should be descriptive for matching and sufficient to give good transform estimation

Feature matching

Transform estimation

APPROACH

SIFT feature extraction 128 dimension feature

descriptor Avg. 2000 features in each

image Nearest neighbor

matching Avg. 1000 matches in each

pair of images RANSAC

Avg. 600 inliers in each pair of images

Illustration

Illustration

Illustration

Illustration

MiH ,1

Illustration

MiH ,1 1,,1', iiMiMi HHH

Problem: error is accumulated

i

kkkMiiMiMi HHHHH

1,1,01,,1,

Introduction Method

Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic

Synchronization of multiple video streams Conclusion

Method: Perform local search for

correspondences between the UAV image and the map

ISSUES

UAV images are very different from the map, SIFT features cannot always match

APPROACHES

Sample points in the map

For each point, locally search for the most similar image patch in the UAV image

Use Mutual Information as similarity measurement

Illustration

Illustration

',MiH

Illustration

',MiH

H

Illustration

Illustration

',, MiMi HHH

Introduction Method

Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic

Synchronization of multiple video streams Conclusion

Method: Perform consecutive UAV image

registration and UAV to Map registration iteratively▪ Consecutive UAV image

registration produce good initials for UAV to Map registration▪ Register the partial local

mosaic to the map

ISSUES

Correspondences in a single frame are not enoughRegistration is unstable

APPROACH

Multiple frames in a time window forms a partial local mosaic which spans a larger region and provides more correspondences More robust Smooth transition

ISSUES

Correspondences in a single frame are not enoughRegistration is unstable

APPROACH

Multiple frames in a time window forms a partial local mosaic which spans a larger region and provides more correspondences More robust Smooth transition

Result

Register single frame

Register partial local mosaic

Illustration

Illustration

Illustration

Illustration

Result

Introduction Method

Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic

Synchronization of multiple video streams Conclusion

Input: UAV image sequences of different views, different frame rates, but capture the same area and overlap in time

An moving object on the ground plane which serves as a “clock” to synchronize the sequences

The moving object should generate a single path on the map

Use sequence alignment algorithm to synchronize the UAV streams

Two steps to register an UAV image to the map Register each frame to its previous frame

to derive an initial estimate Register UAV image to the map to derive

Limitations Initial estimate should be given Unable to recover from a bad estimate

MH ,0',MiH

',MiH

MiH ,