Reconstructing Camera Positions from

Visibility Information

Elif Tosun

Honors Thesis Presentation

Advisor: Ileana Streinu

May 1st, 2002

Outline

Motivation General Problem Some Definitions Main Problems of Interest Background in Rigidity Solutions in 2D Further Work Summary

Motivation: MIT City Scanning Project Goal: To reconstruct a 3D model of an

urban environment using numerous images.

Method:– Robot with a mounted hemispherical camera– Images acquired at different *unknown*

positions of an urban environment– Images put together to get “image feature

points” using methods in Computer Vision and Computer Graphics.

Motivation: MIT City Project

A corner may be an image feature point

Method (Cont’d):– Slopes between cameras and image features

are inferred from use of hemispherical cameras with known local orientations.

– Once the set of slopes are obtained, solution for camera and image feature positions is based on solving a linear system of equations

However… A linear system of equations may be over or under

determined. This information is NOT known a priori Therefore, system *sometimes* may fail to give a

solution. A SUBSET of edges that give a unique solution is

needed.

“… determination of the minimal set of distinct pairs needed for a non-degenerate solution is significantly more complicated and depends entirely on the topology of the adjacency graph. ” [Antone-Teller, 2000]

The Problem Given: Visibility information as a set of

slopes at n positions of a hemispherical camera

Find: Absolute coordinates of camera locations and positions of image features

Why? May be used to reconstruct a model of environment.

Problem? Over- or under-determined system of equations. Need a subset.

This thesis focuses on: Reformulating this problem and

modeling it based on its geometric/combinatorial aspects

Exploring it in 2 and 3 dimensions Isolating several feasible sub problems

and giving algorithmic solutions Providing implementations for some

General Camera Registration Problem Formulation using “direction networks”

Data set:

3 cameras ,4 image feature points, slope known for 13 pairs

Direction Network:

7 (=3+4) vertices

13 edges

Direction Network?

A direction network is a graph G=(V, E) together with slope information for each edge:D = (V, E, S) with

S = {se: se is the slope information for an edge e E}

General Camera Registration ProblemGiven a direction network D and the location of a subset of vertices V’ of V,find the location of all vertices v V of G.

Notes: 1. We don’t treat image features and camera

positions differently. Both sets together form the set of vertices V of G.

2. Location of a subset of vertices is needed to fix a frame of reference.

Definitions Realization of a direction network: A

geometric embedding of G of D. – i.e. set of points P s.t. if (ij)E then the

slope between points pi and pj is sij

Definitions Realization of a direction network: A

geometric embedding of G of D. – i.e. set of points P s.t. if (ij)E then the

slope between points pi and pj is sij

x-axis

y-axis

(0,0)

Definitions Mixed “direction and length” network: A direction network

of the following form:

MD = (V, v1, E1, E2, S, L) where:

E1 = set of edges for which slopes are given

E2 = set of edges for which lengths are given

S = {se: se is the slope for edge e E1}

L = {le: le is the length for edge e E2}

v1 = a unique vertex that is pinned down (at origin)

How to solve for an MD system

To solve for a mixed “direction and length” network…

A) Associate a linear equation to each slope information…

i.e. for each slope sij between points

pi=(xi, yi) and pj=(xj, yj):

(yj-yi) – sij(xj-xi) = 0

B) And associate a quadratic equation to each

length information

i.e. for each length lij between points

pi=(xi, yi) and pj=(xj, yj):

(xj-xi)2 + (yi-yj)2 = lij2

A particular case…The particular case of an MD that we will be

considering is the following:

There exists an edge e, s.t. – both the length (le) and the slope information

(se) is known

– one end point of e is pinned down

– e is the only edge in E2

i.e. it’s as if two vertices are pinned down:

le

se So, we’re only dealing with a linear system of

equations

More Definitions

Tight direction network: A direction network where there is a unique solution to the system.

Loose direction network: A direction network where the system is under-determined.

Loose direction network: A direction network where the system is under-determined.

It has a parallel redrawing!

Over-determined direction network: A direction network where there are more edges than needed to have a unique solution:

Non-realizable direction network: A direction network with no solution where the system is over-determined and the constraints are incompatible

Main Problems

Problem 1: Given a direction network, decide whether it’s tight or not.

Problem 2: Given a loose direction network, find a set of edges, that needs to be added, so that it becomes tight. - if possible

Problem 3: Given an over-determined direction network:– Find a set of edges that need to be

removed so that it becomes tight – if possible

– Find the set of edges to be removed to get ‘the best’ tight framework.

– Get an approximation of ‘the best’ tight framework.

Why “if possible” for Problems 2 and 3:– The data set might not have the edges

necessary for tightness

– The given direction network might be partially over- or under-determined

All induced sub-graphs are under-determined

Why “if possible” for Problems 2 and 3:– The data set might not have the edges

necessary for tightness

– The given direction network might be partially over- or under-determined

All induced sub-graphs are under-determined

Under-determined sub-graph

No edge removal can make it tight

Why “if possible” for Problems 2 and 3:– The data set might not have the edges

necessary for tightness

– The given direction network might be partially over- or under-determined

All induced sub-graphs are under-determined

Over-determined sub-graph

No edge addition can make it tight

Background in RigidityGeometric Rigidity 2 dual categories:

– Fixed Edge Lengths– Fixed Directions (“parallel redrawing”)

Duality in 2D: Properties of one hold for the other as well!

Fixed Length Rigidity:Definitions A framework is a graph G=(V, E)

together with length information for each edge:

F = (V, E, L) with

L = {le: le is the length

information for

an edge e E}le

A minimally rigid framework

A flexible framework

(infinitesimal rigidity – velocities)

Fixed Length Rigidity:Definitions

Fixed Length Rigidity:Definitions A minimally rigid framework

A flexible framework

(infinitesimal rigidity – velocities)

Fixed Length Rigidity:Definitions An over-braced framework

An unrealizable framework

Fixed Length vs. Fixed Direction Framework Minimally Rigid

Flexible

Direction Network Tight

Loose

Fixed Length vs. Fixed Direction

Over-braced

Unrealizable

Over-determined

Unrealizable

Fixed Length vs. Fixed Direction Find a non-trivial set

of velocities Rigidity Matrix (M)

(2nxm). Info on derived from

point coordinates

Find point coordinates

Parallel Redrawing Matrix (M) (2nxm).

Info on slopes

Fixed Length vs. Fixed Direction Solution: Mv = b

Why? (fixed rods, perpendicular velocity vectors)

(pi-pj).(vi-vj) = 0

Solution: Mv = b

Why? (line equations)

(xi-xj)mij –(yi-yj) = 0

V = column vector of unknowns, b = column vector of 0s.

mij

pi

pj

Fixed Length vs. Fixed Direction Solution: Mv = b

Why? (fixed rods, perp. velocity)

(pi-pj).(vi-vj) = 0

Solution: Mv = b

Why? (line equations)

(xi-xj)mij –(yi-yj) = 0

V = column vector of unknowns, b = column vector of 0s.

mij

pi

pj

Rigidity in the Plane:Combinatorial Rigidity Based on underlying graph, no

numeric data Laman’s Theorem:Given a minimally rigid graph G=(V, E) with n

vertices, m edges:(i) m = 2n-3(ii) For every sub graph G’ of k vertices has <= 2k-3 edges

Rigidity in the Plane:Combinatorial Rigidity Based on underlying graph, no

numeric data Laman’s Theorem:Given a minimally rigid graph G=(V, E) with n

vertices, m edges:(i) m = 2n-3(ii) For every sub graph G’ of k vertices has <= 2k-3 edges

Generic Rigidity & Tightness A generic minimally rigid framework(or

tight dir. network) : minimally rigid (or tight) for generic positions of points.

Generic Non-generic

Generic Rigidity & Tightness A generic minimally rigid framework(or

tight dir. network) : minimally rigid (or tight) for generic positions of points.

Generic Non-generic

Solution to Problem 1(Decision Problem)

Geometric Approach :Using parallel redrawing matrix

If there exists a solution to Mv=BCheck dimension, if =0, TIGHTif > 0, UNDER-DETERMINED

Else UNREALIZABLE

Given a direction network, decide if

tight or not

Decision ProblemGeometric Approach (Cont’d) PROBLEM:

In case of NOISE, geometric approach may return: Unrealizable(although if erroneous edges are not used, there might be a solution)

So : we need a subset of these edges necessary and sufficient for a unique solution

Use Combinatorial Approach to get a subset, then use Geometric Approach

Solution to Problem 1(Decision Problem) Combinatorial ApproachMain Idea: - Tight direction network has a unique

solution *most* of the time- Underlying graph of a tight D.N. is a minimally

rigid graph- Find a way of checking if a given graph is

minimally rigid or not

Decision Problem:Combinatorial Approach

Sugihara’s Algorithm, based on bipartite matchings– Associate a bipartite graph to original– Modify it for each edge and check for

complete bipartite matching

Mathematica Notebook

Decision Problem:Combinatorial Approach: Sugihara’s Alg.

Complexity

O(n1.5).O(n2) =O(n 3.5)

Solution to Problem 2(Extension) Problem: Given an under-determined

direction network, find a set of edges to extend it to a tight graph (if possible).

Extension ProblemThe Algorithm

Natural Approach: A Greedy AlgorithmAdd one edge at a time, use “decision”

algorithm to test

Proof of Correctness!

Matroid Theory and Rigidity

Definition:

A matroid is an ordered pair M=(S, l) satisfying the following conditions

1. S is a finite non empty set

2. l is a non empty family of subsets of S called the independent subsets, s.t. if B l and AB, then A l

3. If A l, and |A|<|B|, then there exists an elt

x (B-a) s.t. A {x} l

Matroid Theory and Rigidity2 Important Properties:1. Result by Sugihara:

“Frameworks demonstrate matroidal properties.”

2. “For matroidal structures, greedy algorithms return optimal solutions correctly”

Because of these two properties, our Greedy Approach returns a correct solution

Solution to Problem 3(Extraction) Given an over-determined direction

network, find a set of edges to be removed to get a tight one(if possible).

Extraction: Part 1The Algorithm Greedy AlgorithmRemove one edge at a time, use

“decision” algorithm to testCorrectness – based on Matroidal

Property

For Extension and Extraction Problems: How about if we assign weights to

edges? Weights can mean…

– Easiest to sample– Most reliable

Slight modification in code They help in recognizing a better/best

tight direction network

Extraction Problem : Part 2 Connection b/n Combinatorial and

Geometric Rigidity The Problem: Given a tight subset of a

data set, find another (better?, w/out a non-generic embedding?) tight subset.

Data Original Output New Output

B

C

D

AB

C

D

A

C

D

A

Further Work Motivating problem is in 3D. Our algorithms…

– Emphasis not on efficiency but to show that they are implementable for the sake of experimentation

…and implementations– Emphasis was on underlying theory. Used

what was already available to us (Mathematica)

Summary Reformulation of Problem

Summary Reformulation of Problem

Underlying Theory

Summary Reformulation of Problem

Underlying Theory

Implementations

Credits

Seth Teller, for suggesting the problem

Ruth Haas, Brigitte Servatius, Jack Snoeyink, Ileana Streinu, and Walter Whiteley , for their ideas and references

Special Thanks…

Ileana Streinu Joseph O’Rourke Smith CS Dept Friends and Family

QUESTIONS

?