Computación y Sistemas Vol. 6 No. 4 pp. 293 - 299@2003, CIC - IPN. ISSN 1405-5546 Impresoen México

Qualitative Interpretation of Camera Motion in theProjection of Individual Frames of an Image StreamInterpretacion Cualitativa del Movimiento de la Camara mediante elAnalisis de

Projecciones de los Cuadros de una Secuencia de Imagenes

Joaquin Salas

CICATA-IPN

Jose Siurob lO, Col AlamedaQueretaro, Qro. CP 76040

E-mail: salas@ieee.org

Article received on March 19. 2002: acceoted on March 03._2003

Abstract

In this study, we are interested on some special .kindof camera movements that ideally take place on a ftatsurface and the image projections perpendicular to thesemovements. We analyze interpretation of motion for acamera heading forward, panning around its optical cen-ter and translating perpendicular to its optical axis. Wepresent simulations and experiments with real data. Ourresults show that it is possible to obtain qualitative infor-mation about the ramera motion's nature.

Keywords:Camera Motion, Qualitative Interpretation, Image

Sequence, Flatland

Resumen

En este documento, estudiamos una clase especial demovimientos de la cámara que idealmente se desarrollanen una superficie plana y las proyecciones de imagen per-pendiculares a esos movimientos. De esta forma, anali-zamos la interpretación de movimientos de una cámaracuando ésta avanza en dirección de su eje local, giraalrededor de su centro óptico y se traslada perpendicu-lar a su eje local. l'yIostromos sim'ul<1cionesy experimen-tos con datos reales. Nuestros resultados muestran quela aproximación permite obtener información cualitativasobre la naturaleza del movimiento de la cámara.

Keywords:Movimiento de la Cámara, Interpretación

Cualitativa, Secuencias de Imágenes, Mundo Plano

1 Introduction

Computer Vision is concerned with the tridimensionalinterpretation a scene from a sequence of images. For ORething, this problem is important because in theory, thereis an infinity number of objects that can produce thesame image, e.g., varying the object size and the distancefrom the camera to the object. In particular, we inves-tigate Borneuses of projections of individual frames thatare part themselves of an image stream. In a way simi-lar to images, projections are not uBique. Nonetheless, inthis document, we explore some limits where this notioncan be challenged in practice. Projections are well knowncompact representations of images[Jain et aL, 1995](seeFig. 4(a) and its projection in Fig. 2(a)). In a compactimage stream the variations of projections of individualframes are smalL Thus providing almost uBique charac-teristics to a particular camera trajectory.

In Borne cases, ramera motion is solved along withscene structure. Nevertheless, computing structure frommotion has been shown to be an extremely difficultproblem. However, significatively advances have beendone in the area. For instance, Tomasi[Tomasi, 1991]developed an optimal solution for the case of ortographicprojection. Under perspective projection, extreme caremust be given to computing the intrinsic and extrinsicramera parameters. Even then, the solution is brittleand numerically unstable. Today's state of the artincludes making Euclidean reconstruction from basi-rally uncalibrated cameras[Kahl and Heyden, 2001].The two dominant approaches are factorization-like methods[Zhang and Tomasi, 1999] for weak-perspective and iterative solutions with Kalmanfiltering[Kim et aL, 1997]. The former requires solv-ing sequential matching while the latter involves theproblem of establishing a good initial starting point for

293

J, Salas: Qualitative Interpretation of Camera Motion in the Projection of Individual Frames otan Image Stream

p

Imageplan"

Object

q

l¿/"-8

X // {C,/} ,

.,// Ca'mera ancf~ <' Ir:wge

f\)}/ referente." frames

Figure 1: Imaging object points in flatbnd. An object with corners p and q is placed in the world. The worldreference system is centered in {W}. A camera, with reference system {C}, has its optical center in x. The camerafocallength is f. The image's reference system is placed in {I}. The angle e is the angular difference between {C}and the image reference system {l}.We would llke to verify how object points are imaged as the camera llaves inits workplace.

bundle adjustment. Duric and Rivlin [Duric et aL, 2000]analyze what happen with the histogram of normaloptical flow when the camera rotates, translates in thedirection of the optical axis, perpendicular to the opticalaxis and around the axis perpendicular to the opticalaxis. Since obtaining complete structure from motionhas shown to be difficult and error prone, we claim thatit is worth pursuing trying to gather at least partialand qualitative information about the nature of cameramotion.

In §2, we study ideal projection of object points fordifferent types of camera motion. Next in §3, we presentan scheme to track features along the projected individ-ual frames in an image stream. Then in §4, we presentsome experimental results with both ideal and real data.Finally, we conclude with some remarks and discussionabout research directions.

2 lmaging in Flatland

We are interested on a special kind of camera move-ments that ideally take place on a flat surface andthe image projectiolls perpendicular to these move-ments. Under these circumstances, the projection

294

process may be described analytically by the Radontransform[Kak and Slaney, 1988], given by

P"((t) = l:l: I(x, y)8 (x cos 'Y+ y sin 'Y - t) dx dy(1)

where I(x, y) describes the image, 8(x, y) is a delta func-tion, x cos 'Y + y sin 'Y - t is a collection of parallel raysthat forms an angle 'Ywith the y-axis, and t is a distancealong the projection. From now on, we will focus on thecase where 'Y = O. Let us consider the situation where a

robot llaves while a vision system grabs images with avery small timestamp difference between frames. In thissection, we review how world points will be imaged innoise free flatland(see Fig. 1), Unless stated all coordi-Dates are expressed in the world reference system {W},Suppose that an object with corners p and q is placedin the world. A camera, with reference system {C}, hasits optical center in x. The camera focallength is f. Fi-nally, the image's reference system is placed in {l}. Wewould like to verify how object points are imaged as thecamera llaves in its workplace. The line between pointsp and x is given by

rp+(l-r)x=t (2)

J. Salas: Qualitative Interpretation o, Camera Motion in the Projection o, Individual Frames o, an Image Stream

6'10'

,~'1

'\\ ~J\. 2

J lit

I ':) V (105i

09

, 10 12 " 16 ",..,""

Olo "' '00 '"' ;re 251J IIJ

P""P"""""J",

(a) Projection of the image in Fig.4(a)

o.,

07

06

05

(b )Result of evaluating Eq. (12)in plot (a)

(c) The value of the maximumin (b) are ordered from larger tosmaller and divided by their cu-mulative sum:

Figure 2: Features are found in the projection of images. They are chosen among the larger values of Eq. (12). Athreshold is established based on their cumulative sumo

where the points in the segment px are the points to forwhich r E [0,1]. On the other baTId, the image plane isdefined by the line

p = [s + x¡T u

Here u = [cose,sine]T is the vector in the direction ofthe shortest point from the image plane and both thecamera {C} and the world {W} reference systems; p isthe shortest distance from {W} to the image plane; s is apoint in the image plane; and e is the angular differencefrom {C} and the image reference system {l}. Both Hilesintersect when s + x = t. That is when r equals

r = p - xT U[p - xJTu

This point is given in world coordinates. To get theprojection in image coordinates, the following transfor-mation applies

pI = T¿T&pW

where TfI refers tq a homogenous transform involvinga rotation and a translation such that pA = TfIpB =R~pB + tA. Therefore, given a known camera motion,we have a way to express a feature'sprojection. Oth-erwise stated, when the camera llaves perpendicular toits optical axis, the projection Ul (x) of an object pointp = (x,z) is Ul(X) = k1x, where k1 = f/z. Whenthe camera llaves along the direction of its optical axisis U2(Z) = k2/Z, where k2 = Ix. Finally, when pansaround its optical center it is U3(e) = f cot e.

3 Motion Tracking

(3)

Shi and Tomasi[Shi and Tomasi, 1994] studied the prob-lem of tracking two-dimensional image features fromframe to frame using a Newton-Raphson type of search.In our case, the problem is simplified sílice we trackone dimensional features. The following developmentis largely based on Shi and Tomasi for the case of onedimensional features. Let J(x) and l(x) be two consec-utive frame projections. The dissimilarity between cor-responding features separated a distance d can be mea-sured by .

E(d) = l(J(X+d)-I(x))2dX(6)

(4)where F is a small interval ayer which similarity issought. Th"eterm J(x+d) can be expressed by the linearterms, neglecting the second and higher arder terms, ofits Taylor's expansion as

J(x + d) "" J(x) + doJ(x)ox (7)

(5) Thus,oJ(x + d) ~ oJ(x)

od ~~

The derivative of E(d) is

(8)

o~~) = 2l (J(x + d) - l(x)) OJ(~d+ d) dx(9)

Replacing J(x + d) and its derivative by their approxi-mation, we have

oE(d) "" 2 r(J(x) + oJ(x) d - I(X)) oJ(x) dx (10)od JF ox OX

295

J. Salas: Qua/itative Interpretation of Camera Motion in the Projection of Individual Frames of an Image Stream

0.5

m

'/

//

{/

\\~ +--<--+---t

. .1!¡.'5..~ .1

-25

-3.5

_,lo 4

'"mi,"

(a)

/~

(c ~

.5o 0.5 15 2 25 3 15 4 45 5

~ 211,,2115

¡, 1

195

'9

1851 1 3 , 5 , 7 , 9 10 11

~.h

(c)'10'35

~25

r

/ r//~~"'.C-.

. / /

W / /1

~2/ /l/

..,~

(d)

11 1 3 4 5' a 9 10 11."

-1O 05 1 15 1 2.5 3 15 , 4.5 5

""".

(e) (f)

Figure 3: Matching image profile features. In 3(a), 3(b) and 3(c) we found the correspondence for a feature after 7iterations. In 3(a) the upper line shows the displacement at each iteration. The line below it shows the accumulative

displacement. The final displacement is -4.4875 pixels. In 3(b) we show how the error is decreasing. The final erroris 1.123 millions. In 3(c) we show graphically how both curves converge iteration after iteration to the circled curve.

In 3(d), 3(e) and 3(f) we found the correspondence for another feature after 4 iterations. In 3(d) the upper line showsthe displacement at each iteration. The line below it shows the cumulative displacement. The final displacement is-2.9648 pixels. In 3(e) we show how the error is decreasing. The final error is 717,370. In 3(f) we show graphicallyhow both curves converge iteration after iteration. The rightmost curve is the best fit to the circled curve.

The mínimum error yields when the derivative of E(d)equals zero. Therefore d can be expressed as

where

and

The value of z is a good reference about how easy itis to track a feature. That is, when its value is smallthe displacement is large and convergence may be poor.Contrariwise, when z is large, the iteration tends to con-verge.

296

Sílice non linear factors become important under mostsituations, Eq.(l1) has to be replaced by the followingiterative formulation

dk+1 = dk + d (14)

4 Experimental Results

The equationsoutlinedin §2 can be used to get insightabout the projection of object points in flatland underdifferent types of camera motion. Suppose that there isan object with points p = (-4, 40]T and q = (4,40]T.The focallength is one unit. In Fig. 4 there are Borneresulting plots. In Fig. 4(b) the center of projection wasmoved between [0,3jT and [0,30jT units. The focal axiscoincides with the direction of motion. The imaged ob-ject size is inversely proportional to the distance between

.","

l1r

" \I,! \

\\-01 ; 4o 1 2 il,,"'""

(b).d

9, i\

H \

d = z-le (11)

z= L(a)r dx (12)

e = l (J(x) - I(x)) a) dx (13)

Jo Salas: Qualitative Interpretationof CameraMotionin the Projectionof IndividualFrames of an Image Stream

Heading

Forward

Panning

around

its optical

center

Translating

perpen-

dicular to

its optical

axis

A few frames in the

sequence

(a)

(e)

(i)

Ideal imaged feature

position

;/

/

/0/-~--

~---~',.".,

(b)

\ \

... \

"\ \\\\!W, ,

.~

Projectionstream Feature tracking

,.

\\

Df Jj;!/ A~ .¡KÁyo .ro' ;'r,.- ~,.f - ,,0....-

f".r '" ift:,.o(1" ""/;;;...~

¡"'~~;~!¡ .L r¡~ -,

'""""'t':"v;..¡.~

~

'>I'-'.ro</j:;"+{~f:...", .f' o' "'.VI,,!'. "".

~' O,'v

. 01 '" \"" ", o... '\, r'"'\ 'Y'", ..~,¡.~~ ;:

o' ,( . . . . ~ ~ K a

1.,

(e) (d)

Do

Figure 4: Simulation and experiments with real data to analyze the qualitative interpretation of motion for a cameraheading forward, panning around its optical center and translating perpendicular to its optical axiso

2fJ7

(f) The darkerline shows the

(g) (h)width betweenextremes

:;¡I

])/

//

// /

//

//

, //

:II,[]I.JOCI

In

(j) (k) (1)

J. Salas: Qualitative Interpretation of Camera Motion in the Projection of Individual Frames of an Image Stream

'1'

,"\

,

:¡!',!

1\

11

11'

the camera and the object. In Fig. 4(f) we rotated thecamera between 50 and 130 degrees.

The center of projection is fixed at x = [O,3]T. Inthis range the variation is almost linear and may beconfused with camera motion perpendicular to the op-tical axis. In 4(j) the camera moves laterally betweenXi = [-3, ojT and xi = [3, O]T. The focal axis is per-pendicular to the direction of motion. As reported byBolles[Baker and Bolles, 1988], since the image size de-

pends on the distance between the camera and the objectand hence remains constant through the displacementunder a predefined motion, the extended base line mayprovide both a way to a robust numerical solution and

simple algorithm for tasks such as occluding boundarydetection, stereo analysis and others.

Given the simulation results, we gather some experi-mental data. In Fig. 4, we show three dense sequences.Figure 4(c) corresponds to images of the scene shown inFig. 4(a). It shows a sequence of 153 intensity profilesof a camera heading forward. Physically, the camera ad-vanced about 120crps. There was not strict control onthe distance betweeÍl frames. Indeed the variations in

orientations are remarkable. Nevertheless, the camerahad an overall forward trajectory with its optical axisapproximately in its direction of motion. Here, we ob-serve how the features diverge as the camera gets closer.Figure 4(g) corresponds to images of the scene shown inFig. 4(e). It presents a 82 intensity profiles when thecamera rotates about 90 degrees. Again, the angle be-tween frames is not equal. Also there is not warrantythat the optical center coincides with the center of ro-

tation. Finally, 4(k) corresponds to images of the sceneshown in Fig. 4(i). It presents a dense sequence of 82intensity profiles where we imaged in a direction perpen-dicular to the direction of motion. The camera advancedabout 90cm.

The intensity profile ofthe image in Fig. 4(a) is shownin Fig. 2(a). Then in Fig. 2(b), we show the function zwhen the size of the window F is 10 pixels. The valuesof the maximum sorted by decreasing arder as a per-

centage of the cumulative sum are given in Fig. 2(c). Agood feature tends to be present when there is abruptchange in the profile intensity values. In the rest of theexperiments, we consider a good feature to those whichcumulative value are below 99% of the sum of the featurevalues.

Now, we may attempt to track a feature from line toline. In Fig. 3, we show a couple features of a given lineand its tracking in the next hile. In 3(a), 3(b) and 3(c) wefound the correspondence after 7 iterations. In 3(a) theupper line shows the displacement at each iteration. The

line below it shows the accumulative displacement. The

final displacement is -4.4875 pixels. In 3(b) we show how

11

11,

298

the error is decreasing. The final error is 1.123 millions.In 3(c) we show graphically how both curves convergeiteration after iteration to the circled hile. In 3(d), 3(e)and 3(f) wefound the correspondencefor feature 1after 4iterations. In 3(d) the upperJine shows the displacementat each iteration. The line below it shows the cumulativedisplacement. The final displacement is -2.9648 pixels.In 3(e) we show how the error is decreasing. The finalerror is 717,370. In 3(f) we show graphically how bothcurves converge iteration after iteration. The rightmostcurve is the best fit to the circled curve. At this point, weare in the position to track all the selected features from

one frame to the following. Figure 4(d), 4(h) and 4(1)show the result of tracking through the image stream.The lines are computed automatically by tracking themost promising features. These lines show clearly thatit is possible to infer, at least qualitatively, the nature ofcamera motion from the projections of individual frames.

Conclusion

In this document, we show that it is possible to quali-tatively interpret camera motion from the projection ofindividual frames in an image stream. This interpreta-tion is made for the cases where the camera is movingin the direction of its optical axis, around its opticalcenter, and perpendicular to its optical axis. Given animage streams with these type of camera motion, wepresented a tracking scheme to follow features along theimage sequence. It is possible to observe clearly how thedifference from a rotation and a translation perpendicu-lar to the direction of motion are generate very similarimaging features. The adds to the common believe thatstructure from motion is very sensitive in nature.

In this study, we show that useful information can beprocessed efficiently due to the compact representationof images and the smooth variation of the cumulativesum between frames. Further work aim to organize theredundant visual perception, and to quantitatively inter-pret camera motion, and to use this information to 10-calize the camera in the workspace to allow visual basednavigation.

Acknow ledges

The author wants to thank to Prof. Carlo Tomasi forall his ideas, help and support and to the reviewers fortheir comments and suggestions. This work was partiallysupported with a grant from CEGEPI~IPN.

J. Salas: Qualitative Interpretation of Camera Motion in the Projection of Individual Frames of an Image Stream

References

Baker,H. and BoIles,R. (1988).GeneralizingEpipolar-PlaneImageAnalysis on the SpatiotemporalSurface. In ComputerVision and Pattern Recognition, pages 2-9.

Durie, Z., Rivlin, E., and Rosenfeld, A. (2000). Qualitative

DescriptionofCamera Motion ftom Histograms ofNormal Flow.In IEEE Internationai Conference on Pattern Recognition,

volume~3,pages 194-198.

Jain, R., Kasturi, R., and Schunck, B. (1995). Machine VrSion,McGraw Hill.

Kahl, F. and Heyden,A. (2001). EucJidean Reconstruction andAuto-Calibration from Continuous Motion.

In Internationai Conference on Computer Vision, volume 2,

pages 572-577.

Kak,A.-c. and Slaney, M. (1988). Principies ofComputerized

Tomographic Imaging. IEEE Press.

Kim, E.T.,Han, J.-K, and Kim,RM. (1991).AKalman-Filtering

Method for 3-D Camera Motion Estimation from Image

Sequences. In IEEE Internationai Conference on Image

Processing, volume 3, pages 630-633.

Shi, J. and Tomasi, C. (1994). Good Features to Track.In IEEE Conference on Computer Vision and PatternRecognition,pages 593-600.

Tomasi, C. (1991). Shape and Motion from Image Streams: a

Factorization Method. PhD thesis, Carnegie Mellon

University. CMU-CS-91-172.

Zhang, T. and Tomasi, C. (1999). Robust and ConsistentCameraMotionEstimation.InIEEE Conferenceon ComputerVision and Pattern Recognition, pages 164-170.

Joaquín Salas, obteined his Doctor degree in Informatics from ITESM campus Monterrey in 1996. From then on,he has been ajJiliated with CICATA-IPN. He has been visiting scholar or invited profesor at Xerox PARe, OregonState University. Universidad Autónoma de Barcelona, Stanford University, and the Ecole Nationale Superiore desTelecommunications. He has published 17 articles in international journals and congress on the general topic of

image analysis. Since 1995, he has ryeenmember ofMexico's National System ofResearchers. He wasfoundingPresident of IEEE Querétaro, section and chairman of the IEEE 7th Mexico's National Minirobotics Contest

299