Improved Local Flow

Shaogang Gong *

Robotics Research GroupDepartment of Engineering Science

University of Oxford

In order to understand the parallel computation of optic flow,we introduce here a novel algorithm for computing the flowfield at certain locations in the image. These locations are onthe gradient edges, which Brady calls seeds [1] . Our CurveMotion Constraint Equation provides an additional constraintto fully and locally estimate the flow field at seeds. Our ini-tial computational experiments have used the improved localflow as the initial input to Hildreth's algorithm. Now we areexploring an algorithm to perform the whole computation inparallel.

1 IntroductionHorn and Schunck's motion constiaint equation provides oneconstraint on the optic flow vector at any position in an image.This raises the "aperture problem" in the two-dimensionalapparent motion computation. In order to fully constrain orestimate the optic flow, they assumed that the physical worldis locally smooth everywhere, as is the three-dimensional mo-tion field. As optical projection preserves smoothness into thetwo-dimensional motion field, and also because of the similar-ity between the two-dimensional motion field and the two-dimensional optic flow, the optic flow is locally smooth ev-erywhere in the image. By using the smoothness assumptionto regularise the flow field (to achieve global minimisation),they suggested that the aperture problem can be overcome [6].Under egomotions in which the camera moves against a staticenvironment, the motion field does seem to be smooth, as doesthe optic flow. But the flow field in a general situation whichcontains at least one moving object against a static back-ground, will not be smooth. Also, depth discontinuities giverise to the motion field discontinuities [15], [3], [2], [16]. In-stead of smoothing the flow field in two-dimensions, Hildreth'sscheme only smooths the flow along one-dimensional curvescorresponding to the zero crossings of the image. The argu-ments for doing so are based on: physiological evidence [5], thenumerical conditioning of the motion constraint equation and,the physical adequacy for assuming smoothness along edgesrather than everywhere in the image. Hildreth's approachis supported by some experimental results [5], [3]. Thoughit is clearly one of the best schemes proposed for estimat-ing optic flow (at least along curves), it is inherently sequen-tial [3]. Alternatively, Scott's Four Line Method [15] com-putes a dense optic flow similar to that of Horn and Schunck,but he argues that the flow field across a motion boundarywon't be smoothed out, and furthermore, the match proce-dure used in the scheme is local instead of a global regulari-

*The author acknowledges the support of the Royal Society andGEC Hirst Research Laboratories.

sation, which overcomes the sequential computation problemassociated with Hildreth's scheme. In practice, the matchscheme employed in Scott's algorithm tends to propagate theflow field into the static background region which has anylocal edge structure. This gives the effect of a "wake" sur-rounding moving objects, though it employs some novel ideaswhich combines different smoothness assumptions accordingto the different intensity structures into a single scheme [3].

Imposing observational assumptions, such as smoothness,about the physical world, are aimed to overcome the aper-ture problem by introducing another constraint. Most peoplesimply apply regularisation to achieve a global minimisation.There are hardly ideas about trying to find a local constraintwhich can fully estimate the optic flow only applying localcomputation. Koenderinck has carried out some theoreticalinvestigations from biological point of view [8], [7], Theoreti-cally as well as practically, we have been inspired by Nagel'swork [10], [9], [11].

The work represented in this paper is to understand the de-gree of constraint on the optic flow computation at the seedlocations as we will describe later [1]. We claim that at theseed locations, we can fully estimate the optic flow locally.Similar work has been carried out by Nagel in his second or-der Taylor series expansion of the intensity function. Nagelshowed that at "grey value corners", the full flow can be com-puted by his second order equation [10]. More recently, Nagelshows that employing the smoothness assumption along theedges as Hildreth does, or other kinds of 'oriented smooth-ness' assumptions are implicitly employing the higher order ofTaylor expansion of the intensity function [11]. But, Nagel'smethod concerns how to compute the flow field in a two-dimensional local region. As we said before, in general, thereare flow discontinuities across the image, we believe that try-ing to estimate the two-dimensional flow field before knowingthe boundaries of the flow discontinuity will risk the dangerto cross these boundaries in the computation. This will blurthe flow across the different objects which carry on differentmotions. Therefore, we attempt to understand here the localconstraint at the locations along the edges.

2 Seeds and its constraintsThe Seeds are locations of two-dimensional constraint [1], ex-amples of which include Nagel's "grey value corners", or otherkinds of models based on the change of second order deriva-tives of the intensity function [4], [12], etc. At seed loca-tions, we have a two-dimensional constraint on the flow vectorwhich means that in theory, we can fully estimate the two-dimensional flow locally. This observation has been notedpreviously [9], [11], [1], etc.

Instead of trying to recover the optic flow field everywherein the image by a single mechanism, we believe that there

129

ALVEY VISION CONFERENCE, pp. 129-135, MANCHESTER, ENGLAND, SEPTEMBER 1988

are different schemes for estimating the flow associated withdifferent intensity structures. In the image, there are locithat offer differing constraints, namely points within regionsof smooth change or constant intensity value, points of one-dimensional intensity discontinuity which are often associatedwith image edges, and points of two-dimensional discontinuityof the intensity which are seeds, are distinguish. Furthermore,according to the degree of the constraint, there are degrees oflocality. This means when the constraint is decreases, theassociated condition for locally estimating the full flow alsodecreases. There are loci in the image at which we can com-pletely and locally compute the flow without global assump-tions. But, as the constraint decreases, we need a differentschemes for estimating the flow in those less constrained lo-cations which can not be done locally. Therefore, the parallelcomputation of two-dimensional apparent motion can not beemployed throughout all stages of the computation. This sug-gests that the computation of the optic flow is a multiple levelmechanism in the sense that different levels are associatedwith a certain degree of well-conditioning as well as of paral-lelism. The question of parallel computation of visual motionis to maximally employ the degree of parallelism, which aredifferent in different stages of the computation, rather thantrying to employ a single parallel computation mechanism allthe way through the motion estimation. Similar suggestionsare to be found in physiological and psychological experiments[13], [14].

3 Mathematical structure of theedge flow

We study the conditions in which we can locally compute bothcomponents of the optic flow (full flow) along an edge.

We start from the motion constraint equation (m.c.e):

VI • /i + /, = 0 (1)which derives from an assumption of temporally constant

intensity. VI is the first order spatial gradient vector, (i is theoptic flow vector, which is defined as:

li= { dx/dt dy/dt ] T , (2)and /t is the temporal gradient at a pixel. The relationship

between the spatial and temporal gradients and the optic flowvector is then:

I,(3)

where N = VI/||V/|| is the unit vector in the direction ofthe intensity gradient.

As we said before, equation 3 imposes one local constrainton the two-dimensional optic flow vector. If the norm of spa-tial gradient ||V/|| is small, this computation will be poorly-conditioned. This is the numerical argument in favour of edgemotion estimation. Therefore, we restrict our attention tothose image loci which have high first-order spatial gradient,edges ( V I ) . At such an edge point, an edgel, the intensitygradient is orthogonal (in the image plane) to the (tangent tothe) edge.

To understand the mathematical structure of the flow fieldalong an edge, we examine an edge in a temporal sequence ofimages. In one representation, edgels are addressed by theirimage coordinates (x, y); and in another representation, thesame point is accessed by its distance s along the edge. Addingthe time parameter t, we denote the quantities that feature inour analysis as follows:

I(x, y, t) : intensity at a point in the image./(s, t) : intensity of an edgel.y(s, t) : an edge.x(s, t) : x coordinate of a point on the edge.y(s, t) : y coordinate of a point on the edge.VI : spatial gradient of a pixel.N : unit vector in the gradient direction.T : unit vector orthogonal to N.H : velocity vector of a pixel.H : Hessian matrix of a pixel.

The position along an edge is given by s, at any instanttime U, and at any particular point on the edge, we have:

In other words:

T = [ dx/ds dy/ds ] T(4)

After these statements have been cleared, it can be shownwhat the tangential component of the flow vector along anedge should be. First we have the Taylor expansion of theoptic flow along an edge, which is:

As) = p{s) + ^ -As + -Q^ • O(As3)

Similarly, the Taylor expansion of VI and It along an edgeshould be:

VI(s + As) = VI(s)

I,(s + As) = /,(«) + | ^ A s +

O(As3)

+ O(As3)

Ignore the triple and higher order of As in the above equa-tions, we have:

H(s + As) = p(«) + |^As + P^as as'

VI(s + As) = VI(s)ds

dh

as2

(5)

(6)

a2/,As2

Now, consider the motion constraint equation in the neigh-bourhood of a point at distance s along the edge, which is:

VI(s + As) • n(s + As) + It(s + As) = 0 (8)

Substitute equations 5, 6 and 7 into this neighbourhoodm.c.e; then, ignoring the triple and higher order of As gives:

= o

The partial derivatives of intensity and velocity with re-spect to a spatial coordinate in two-dimensions are constant.Therefore, in order to satisfy this equation, the coefficients ofzero, first and second order in As should respectively be 0.This leads to:

VI • fi + It = 0 (9)

130

1. At first, equation 15 can be more explicitly written as:

ds ^ ds(10)

= 0 (11)

Equation 9 is the motion constraint equation. Equations 10and 11 provide two new relationships between the intensityand the flow along an edge. What do equations 10 and 11tell us ? After some mathematical manipulations on the firstorder equation 10, we derive the following:

(TTHN)(N • n) + (TTHT)(T • ti) - (TTH/i) = 0 (12)

Similarly, from the second order equation 11, we obtain:

(TTHN){(TTH/i) + (VI t • T)} = 0 (13)

where VI^ is the temporal gradient of the spatial gradient.Clearly, equation 12 provides no additional information. How-ever, equation 13 links the tangential and normal componentsof the optic flow along the edge. This will be seen in moredetail from the following.

From equation 13, we have either:

T T H N = 0 or (TTH/i) + (VI t • T) = 0

If (TTHN) is zero, then either HT is parallel to T or, itis 0. Now:

= (TTHN)N + (T T HT)T = HT

and so in general, (HT) is not parallel to T, nor is it equalto 0. In other words, generally we have the Curve MotionConstraint Equation (c.m.c.e) as following:

(TTHN)(N • ti) + (TTHT)(T • fi) = - ( V I t • T) (14)

Is there any special case for which T T H N vanishes? As:

(TTHN)N + (TTHT)T = HT

which means if,

T T HN = 0 =4- HT = (T T HT)T

As we have:

* = - ! K 7 7 F ( T T H T )

which K is the curvature of the edge, therefore:

HT =-/c||V7||T

If HT is parallel to T, T T H N equals to 0. In fact, we foundanalytically that this situation happens along a circular edge.This may explain why we can not compute a circular object'srotation about its centre, for which all the full flow vectors onthe edge only have tangential component.

Except for the case that HT is parallel to T, furthermore,being 0 — in other words, edge is a straight line, also makesT T H N vanishing. This gives:

0 = [ dh/ds dly/ds ] T = HT (15)

It says that both dlx/ds and dly/ds are equal to zero,which gives the following:

- — hxJy - hylx— nrr Til J J _J T| |V7| | [ ixyiy — Jyj^i

which means:

Ixxly — lxylx == 0, *xyly lxylx = 0

This obviously gives us: Ixxlyy = fly Therefore, wherethe edge is straight, we have:

detH = 0

2. Secondly, as:

(TTHM) + (VI t • T) = (VI t • T) jt 0

which means where the edge is straight, c.m.c.e doesn'thold.

By this stage, we have determined a relationship betweenthe edge's optic flow and the corresponding intensity struc-ture. This can be briefly summaried as:

conditions if we assume: in general, the second order deriva-tives of the intensity function exist locally, then we have:

1. To require an edge being straight, means that thedeterminant of the local Hessian matrix equals tozero.

2. Theoretically, we can estimate both componentsof optic flow locally along edges wherever the edgeis not straight, though we still can not estimatethe circular objects rotating according to its cen-tre. Practically, we need to concern ourselves withthe numerical condition of the c.m.c.e. It is verysimilar to the situation that arises in using them.c.e. For the m.c.e, it is judged by the norm ofthe gradient. For the c.m.c.e, it is judged by detH.In fact, detH = K1K2(EG - F2), in which E, Fand G are the components of the first fundamen-tal form [12], which suggested that the intensitysurface shape can be basically classified into threetypes based on the value of detH. That is:

detH < 0, hyperbolic point;detH > 0, elliptic point;detH. = 0, parabolic point.

The parabolic points are associated with thestraight edges in the intensity surface, but bothhyperbolic and elliptic points are associated withseed locations.

3. Furthermore, where the detH equals to zero, theoptic flow estimation reverts to be an under-determined problem.

What can deduce about the c.m.c.e ? First of all, we canwrite the c.m.c.e. in a more general form as: <j>(T • /i) +¥>(N • n) + A = 0. So the equation gives the correct solutionin two kinds of extreme situations (figure 1).

Secondly, we see that Hessian matrix plays a central rolehere. In practice, we only can compute the T • n where H iswell-conditioned. As H is associated with the surface curva-ture of the intensity function [17], [10], [12], large Hessian de-terminant is associated with the loci of local maximum surface

131

v (N.v)N (N.v)N

(T.v)T

a moving object

(T.v)T

a moving object

when |v| = N.v,so: T.v = 0

when |v| = T.v,so: N.v = 0

Figure 1: Two extreme situations for the flow flied com-putation.

The function represents two aspects of error sources in thetangential flow computation, where e expresses the degree ofconfidence in the computation of the Hessian matrix itself, andthe determinant of Hessian represents the degree of confidencein using the c.m.c.e. In general, t should be the conditionnumber of the Hessian matrix:

where ( equals to 1 corresponding to well-conditioned, andoo corresponding to singular. Also, in general, /? should besmaller than a, as the error in the computation of secondorder derivatives are bigger than in the first order derivatives.The discrete function for the first, second and third term inthe functional of equation 17 are as follows:

curvature on the intensity surface, which are the loci of two-dimensional constraint. Therefore, they correspond to loci ofseeds, where we can give a full value for the optic flow. Thoughit seems to be similar to Nagel's grey value corners, we derivedhere an explicit connection between the structure of intensityon the edge and the constraint for computing the optic flow atthese locations. This connection generalises Nagel's methodby giving explicit detail about not only dealing with the flowof two-dimensional constraint loci — grey value corners; butalso dealing with the flow of one-dimensional constraint loci— edges, and about the degree of numerical reliability forlocal flow computation on these edges.

4 Improved local flow for edge'sflow

Our first application of c.m.c.e is to give an improved localflow as the initial data for Hildreth's scheme. First we givean improved model which is based on the Hildreth's originalapproach but consider the new local information from thec.m.c.e.

In general, there will be error in both the local computa-tion of N • /J, which is denoted as iix, and T • \i denoted as/iT. This is caused by image quantisation error and imagenoise, etc. Therefore, in practice, we require only that theflow obey these local constraints approximately. Combiningthis approximate requirement with the general smoothness as-sumption along the edge, we have the following functional tobe minimised:

0

+« [[

(17)

The parameter a is a weighting factor which expresses thedegree of confidence in the local normal flow from the motionconstraint equation. /? is another weighting factor for the localtangential flow. We set a to be a constant, but set /? to bea function of the local Hessian matrix which is varying alongthe edge.

„ detH

1 = 2

02 = a

03 =detK

From ©i, ©2 and ©3, we have the discrete formula forequation 17 as:

= ©i + ©2 + ©3 (18)

Now, the question of minimising equation 17 leads to thequestion of finding a set of x and y components of the flow,which minimise the discrete function $ in equation 18. Thisgives a set of 2n linear equations, which are:

_2L_ = o, £L- = 0, 1 < i < n.

to be solved. This leads to:

[4 + 2a{Nx,)2

(19)

1 < i < n.

and,

[4 + 2a(Nyi)

- 2 ^ , . , + [2<xNx,Ny,

These two equations constitute an improved model relativeto Hildreth's original scheme [5]. It uses more local informa-tion both from m.c.e. and c.m.c.e. to give a faster algorithm.

132

' " T T T

normal flow tangential flow

first pixture deforming 2 pixels

traslation - 2 pixels down

full flow from c.m.c.e. full flow from Hildreth's

Figure 4: Synthetic rotation — 3° clockwise.

%<

normal flow tangential flow

rotation - 3 degrees clockwise general motion

Figure 2: Synthetic image sequences for the experimentof c.m.c.e.

normal flow tangential flow

full flow from c.m.ce.

Figure 3: Synthetic translationdirection.

full flow from Hildreth's

2 pixels down the y

5 Experimental resultsIn order to show that our model works on practical images,we show the results of some initial experiments. First we givesome results which are for testing the c.m.c.e. Secondly, weshow some results from our improved version of Hildreth'salgorithm.

We first explain the pictures we use here. The aim of thispart of the experiment is to show whether the equation worksbased on the degree of curving on the edges which we haveshown theoretically. Therefore, we try to use a kind of imagewhich has less influence from other aspects, such as noise inthe image which leads to errors in edge detection and so on.This leads us to use a sequence of synthetic ellipse imageswhich has different curvature along its boundary which is go-ing to give us an edge that can be used for the experimenthere. The image sequences include translation, rotation anddeformation (see figure 2). The size of images is 128 X 128.

The resulting flows are shown in figures 3, 4, 5 and 6, whicheach figure also shows a result from Hildreth's method, butall of them are about fifteen times faster. We note that mosttime consuming in the computation spends on computing allthe derivatives for which we are using relatively large masks(7x7 and 9x9) because the computation of the second orderderivatives are very sensitive in general. On the other hand,

full flow from cm.ce. full flow from Hildreth's

Figure 5: Synthetic deformation — 2 pixels in all direc-tions.

the computing time in Hildreth's method is mostly spent inthe iteration stage which is inherently sequential. Thereforewe can further speed up the seed motion computation consid-erably and easily by using a hardware image processing array.In figure 3, we have a translation in which the local normalflow of the loci of seed are useless. The recovered local tan-gential flow complement the local full flow rather well. Thisassociates with the two extreme situations we mentioned be-fore. Figure 4 and 5 give the results from a rotation (threedegrees clockwise). The tangential flow gives a reasonably ac-curate contribution to the full flow at the seed loci. Finally,figures 5 and 6 show two flows associated with a motion of twopixels deforming in all directions, and a motion of rotation inthree degrees clockwise and translation of two pixels in both xand y directions. In these more general situations, the c.m.c.estill gives a quite reasonable local tangential flow.

We have demonstrated the adequacy on sequences of syn-thetic images. But these are images that have simple struc-ture. For real images, the situation is going to be changed

"--"hT-H

normal flow tangential flow

full flow from cm.ce. full flow from Hildreth's

Figure 6: Synthetic general motion — translating 2 pixelsin both x and y directions; also rotating 3° clockwise.

133

a better understanding of the parallel computation of visualmotion.

first frame second frame

flow field

Figure 7: Results from the improved algorithm.

simply because on one hand there are lots of noises both fromthe original image, and from the image quantisation; and onthe other hand, the computation of second order derivativesare well known to be sensitive to noise. Therefore, we considerthe local tangential flow as an approximate local constraint forthe full flow computation, as in equation 17 rather than as aprecise calculation. The result from applying our improvedHildreth's method on the real image sequence is shown in fig-ure 7. This result didn't show us a great change in the timeconsuming (about half the iterations as Hildreth's does). Thereason can be explained as there are not enough seeds alongthe edges; also we don't take their tangential flow as boundaryconditions which turns out that the seed's contributions areburied in or smoothed by the non-seed's normal flow. Thisleads to our following consideration of an alternative way forapplying the c.m.c.e.

Currently, we compute a fairly good approximation to thelocal tangential flow at the seed loci, or we compute the tan-gential flow everywhere on the edge by setting the threshold ofdetH to zero, which will give us a very low confidence aboutthe tangential flow at the loci of low detH.. Consequently,our current investigations start from finding how to build upthe local tangential flow all along the edge by propagating,instead of directly using equation 14, the tangential flow atthe loci of seed to the loci of low detH. This should give us amore confident tangential flow at those loci of detH. Althoughthese propagated tangential flows along the edge still can onlybe approximate, they will have greater confidence. We arecurrently exploring the possibility of using wave-diffusion pro-cessing (based on the assumption that the change of tangentialflow is continuous along the edge) to give us a fast propaga-tion along the edge. This processing will again be inherentlyparallel. After we have the tangential flow everywhere on theedge, we combine them locally with the normal flow, to giveus a very close approximation to this edgel's full flow. By thisstage, we only need a few final iterations to smooth the wholeflow globally to overcome the initial error that derives fromnoise rather than from insufficient constraint. In this way, weuse the local computation as much as we can to minimise theinvolvement with the global computation. Therefore, most ofour computation can be done in parallel. The reasons for do-ing this is to understand how we can propagate the reliableoptic flow into less constrained regions and at the same timehow much parallelism can be achieved. The ultimate pointhere is to understand the multiple level optic flow computa-tion structure. To answer the questions such as how muchparallelism is associated with a certain level's computation,how we can maximally apply it, and finally, how much par-allelism is in the whole optic flow computation will bring us

AcknowledgementsI thank my supervisor, Mike Brady, for providing the initialideas in this work, and for very helpful discussions and sugges-tions. Thanks to Bernard Buxton for very valuable commentsboth in the mathematic and in the context of the paper. Alsothanks to David Murray, Guy Scott, Andrew Zissermen andall the others in Vision Lab. for help in various ways.

References[1] J.M. Brady. Seeds of Perception. In Proceedings of

The Third Alvey Vision Conference, pages 259-267, Uni-versity of Cambridge, Cambridge, England, September1987.

[2] L.S. Davis, Z. Wu, and Sun H. Contour Based MotionEstimation. Computer Vision, Graphics and Image Pro-cessing, 23:313-326, 1983.

[3] Shaogang Gong. Parallel Computation of Visual Motion.MSc. Report, Oxford University, England, July 1987.

[4] Haralick, R.M. Digital Step Edges from Zero-crossingsof Second Directional Derivatives. IEEE Trans. PatternAnal, and Machine Intell, PAMI-6(l):58-68, 1984.

[5] E.C. Hildreth. The Measurement of Visual Motion. MITPress, Cambridge, Mass., 1984.

[6] B.K.P. Horn and B.G Schunck. Determining OpticalFlow. Artifical Intelligence, 17:185-203, 1981.

[7] Koenderink, J.J. Optic Flow. Vision Research,26(l):161-180, 1986.

[8] Koenderink, J.J. and Doom, A.J. Local Structure ofMovement Parallax of the Plane. J. Opt. Soc. Am.,66(7):717-723, 1976.

[9] H.H Nagel. An Investigation of Smothness Constraintsfor the Estimation of Displacement Vector Fields fromImage Sequences. IEEE Trans. Pattern Anal, and Ma-chine Intell., PAMI-8(5):565-593, 1986.

[10] H.H Nagel. Displacement Vectors Derived from SecondOrder Intensity Variations in Image Sequences. Com-puter Vision, Graphics and Image Processing, 21:85-117,1983.

[11] H.H Nagel. On the Estimation of Optical Flow: Rela-tions between Different Approaches and Some New Re-sults. Artifical Intelligence, 33:299-324, 1987.

[12] J. Alison Noble. The Geometric Structure of Images.MSc. Report, Oxford University, July 1987.

[13] Ramachandran, V.S. Apparent Motion of SubjectiveSurfaces. Perception, 14:127-134, 1985.

[14] Ramachandran, V.S. and Anstis, S.M. Perceptual Or-ganization in Multistable Apparent Motion. Perception,14:135-143, 1985.

[15] G.L Scott. The Four Line Method of Locally EstimatingOptical Flow. Technical Report, University of Sussex,England, 1986.

[16] A.M. Waxmanand K. Wohn. Contour Evolution, Neigh-bourhood Deformation and Global Image Flow: Pla-nar Surfaces in Motion. The International Journal ofRobotics Research, 4(3):95-108, 1985.

[17] R.J. Woodham. Analysing Images of Curved Surfaces.Artificial Intelligence, 17:117-140, 1981.

134