Detection of motion fields under spatio-temporal non-uniform illumination

Lin Zhang*, Tatsunari Sakurai, Hidetoshi Miike

Graduate School of Science and Engineering, Yamaguchi University, 2557 Tokiwadai, Ube 755, Japan

Received 5 September 1997; received in revised form 5 January 1998; accepted 5 May 1998

Abstract

In actual scene analysis, the influence of non-ideal conditions such as non-uniform illumination should be taken into account. Theconventional methods for the estimation of motion fields are violated in this situation. In this study, two approaches are proposed to extractreliable motion fields under spatio-temporal non-uniform illumination. These are an extended constraint equation with spatio-temporal localoptimization and a pixel-based temporal filtering. Experiments have been made to confirm the performance of the proposed methods and toclarify the difference of characteristics between them.q 1999 Elsevier Science B.V. All rights reserved.

Keywords:Motion field; Non-uniform illumination; Spatio-temporal local optimization; Temporal filtering

1. Introduction

In the computer vision [1] fields, many studies have beencarried out with the aim of obtaining information on thethree-dimensional (3D) environment from image sequence.One of the most important problems is to determine opticalflow, which is the distribution of apparent velocities ofmoving brightness patterns in an image sequence [2,3,18].Optical flow results from relative motion between a cameraand objects in the scene.

A number of different approaches to determine opticalflow have been proposed including gradient-based, corre-lation-based, energy-based and phase-based methods. Arecent survey is due to Barron et al. (1994) [5], where thedifferent approaches were compared on a series of syntheticand real images. In the actual scene analysis, however, theperformance of conventional methods is not satisfactory.There exists the influence of non-ideal conditions in theactual scene. For example:

non-uniform illumination [8];occlusions [9];multiple optical flow [10,11];non-rigid motion of object [12]; anddiffusion [13].

If we want to obtain a reliable optical flow, we shouldtake such problems into account. Recently, our research

group have proposed two methods for determiningmotion fields from sequential images under spatially or tem-porally non-uniform illumination [8,14]. The methods arebased on the extended conservation equation, which isobtained by observing the total brightness change in afixed local closed area. One of the methods assumesspatially non-uniform illumination and a stationary motionfield. The other method assumes temporally non-uniformillumination and local constancy of motion vectors. Withthese methods, we can determine 2D motion fields of fluidflow under spatially or temporally non-uniform illuminationseparately [8].

On the other hand, in the ordinary approach, noise reduc-tion and contrast enhancement of images are based on two-dimensional (2D) space filtering [4]. For these purposes, wecan introduce frequently digital filtering with 2D FastFourier Transform (FFT) and non-linear filters such as amedian filter. These space domain approaches are effectivefor static image processing; however, it is usually difficult toremove the influence of non-uniform illumination (spatialand temporal) in a dynamic image sequence.

In this paper, we develop a new algorithm to cope withtwo co-existing conditions of non-uniform illumination. Wetested two approaches. The first method introduces theextended constraint equation with spatio-temporal localoptimization. The second method introduces a new methodof temporal filtering that enables the reduction of theinfluence of non-uniform illumination. The performanceof the proposed methods is confirmed by the use of syntheticand real image sequences.

0262-8856/99/$ - see front matterq 1999 Elsevier Science B.V. All rights reserved.PII: S0262-8856(98)00111-5

* Corresponding author. E-mail:zhang@sip.eee.yamaguchi-u.ac.jp

Image and Vision Computing 17 (1999) 309–320

2. Background of extended constraint equation

An extended constraint equation is derived from a con-servation law of total brightness [14,16,17] in a fixed smallregiondSas illustrated in Fig. 1.

]

]t

∫dS

f dS¼ ¹ rdCf ~n·~ndCþ

∫dS

fdS, (1)

wheref(x,y,t) is a spatio-temporal brightness distribution ofsequential images,dS is a fixed local observation area,dC isthe contour surroundingdS, ~n ¼ (nx, ny) is the motion vectorto be determined,~n is the unit vector normal todC andpointing outwards, andf is the rate of creation (or annihila-tion) of brightness at a pixel indS. The creation termincludes increasing or decreasing brightness on the imageplane under influence of non-uniform illumination. Eq. (1)is reduced to a differential formula [14] in two dimensions:

]f]t

¼ ¹ fdiv(~n) ¹ ~n·grad(f ) þf: (2)

Under the assumptiondiv(~n) ¼ 0 [16] andf ¼ 0 [17], Eq. (2)coincides with the basic constraint equation of the gradient-based method [7]:

]f]t

¼ ¹ ~n·grad(f ) ¼ ¹ nx]f]x

¹ ny]f]y:

In this study, we adopt the following relationship for thedetermination of motion fields:

]f]t

¼ ¹ ~n·grad(f ) þ f: (3)

This relationship is reduced from Eq. (2) under the assump-tion of div(~n) ¼ 0. This assumption requires a rigid objectmotion perpendicular to the camera optical axis. Since thisconservation equation contains the creation term of bright-ness, it is possible to estimate the effects of non-uniformillumination for the detection of motion fields.

Nomura et al. (1995) [8] introduced an assumption ofseparability of non-uniform illumination. They assumedthe spatio-temporal brightness distributionf ðx; y; tÞ isf ðx; y; tÞ ¼ rðx; y; tÞ·gðx; y; tÞ, where r(x,y,t) represents theeffect of non-uniform illumination andg(x,y,t) is the bright-ness distribution under uniform illumination. For the first

situation, the illumination is assumed to be only spatiallynon-uniform and constant with respect to time (r ¼ r(x,y),]r/]t ¼ 0). The reduced relationship is [8]

]f]t

þ ~n·grad(f ) ¼ fq��������������n2

x þ n2y

q, (4)

whereq(x,y) is an unknown constant. If the velocity field isconstant locally with respect to time (dt), the motion vector~n and the unknown constantq(x,y) are determined by mini-mizing the following error function (with temporal localoptimization: TLO [14]) with the non-linear least-squaresmethod (for example, the Newton–Raphson method):

E¼∑dt

(ft þ nxfx þ nyfy ¹ fq��������������n2

x þ n2y

q)2: (5)

As in the second situation, the illumination is assumed to beonly temporally non-uniform and constant with respect tospace (r ¼ r(t), grad(r) ¼ 0). The reduced relationship is [8]

]f]t

þ ~n·grad(f ) ¼ f·]r(t)=]t

r(t)¼ fw(t), (6)

wherew(t) is an unknown constant. If the velocity field isconstant locally with respect to space (dS ¼ dx·dy), themotion vector~n and the unknown constantw(t) are deter-mined by minimizing the following error function with thelinear least-squares method (with spatial local optimization:SLO [15]):

E¼∑dx

∑dy

(ft þ nxfx þ nyfy ¹ fw)2: (7)

Here, the parametersdx and dy represent the width of thelocal neighborhooddS in x andy direction.

3. Proposed methods

3.1. A spatio-temporal local optimization

The extended constraint equation for gradient-basedmethod contains the creation term of brightness. Nomuraet al. (1995) [8] separated the term into two different con-ditions of illumination (spatially non-uniform illuminationor temporally non-uniform illumination) and determinedthis term under respective conditions.

In the actual scene, however, there is the case of two co-existing conditions of non-uniform illumination. In this sec-tion, we propose a new model, which fuses two conditionsof non-uniform illumination. We introduce the extendedconstraint equation with spatio-temporal local optimization[19].

The parameterf(x,y,t) of Eq. (3) represents the influenceof non-uniform illumination. Under the assumption that theillumination is only spatially non-uniform, f1 ¼

f��������������n2

x þ n2y

qq(x,y). On the other hand,f2 ¼ fw(t) when we

assume that the illumination is only temporally non-uniform, where q(x,y) represents the core effect ofspatially non-uniform illumination andw(t) represents that

Fig. 1. A schematic explanation of variables appearing in the conservationequation.

310 L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

of non-stationary illumination. Now we assume the spatio-temporal brightness distributionf(x,y,t) is under spatio-temporal non-uniform illumination

f (x,y, t) ¼ r(x,y, t)·g(x,y, t) ¼ r1(x,y)·r2(t)·g(x,y, t),

where r 1(x,y) represents the effect of spatial non-uniformillumination, r 2(t) represents the effect of temporal non-uniform illumination andg(x,y,t) is the virtual brightnessdistribution under uniform illumination. From Eq. (3) thef(x,y,t) is represented by the following equation [see Eq.(A12) in Appendix A]:

f(x,y, t) ¼f1(x,y, t) þ f2(x,y, t)

¼ f (x,y, t)��������������n2

x þ n2y

qq(x,y) þ w(t)

� �:

(8)

When the unknown variablesnx, ny are assumed to be con-stant in a local spatial–temporal volumedV ¼ dx·dy·dt, Eq.(8) can be reduced:

f ¼ f (x,y, t)(cq(x, y) þ w(t)),

wherec¼��������������n2

x þ n2y

q¼ const. We also assume that the image

functionf(x,y,t) varies rapidly with respect to time and spacecompared to the effect of non-uniform illuminationq(x,y)or w(t). Thus, we obtain again a simplified relationship:

]f]t

¼ ¹ ~n·grad(f ) þ fw9(x, y, t), (9)

wherew9ðx; y; tÞ ¼ cq(x,y) þ w(t) is regarded as a constantparameter in the local volumedV. Since this equation con-tains both effects of spatially non-uniform illumination andnon-stationary illumination, it seems possible to manipulatethe effects of every condition of non-uniform illuminationunder the above assumptions.

The above assumptions seem to be valid in a small spatio-temporal neighborhooddV ¼ dx·dy·dt. For the determinationof the three unknown variables in Eq. (9),~n ¼ (nx, ny) andw9,we introduce the assumption that the components of motionvector and unknown variables are constant with respect totime and space in a local volume ofdV:

w9 ¼ const

nx ¼ const

ny ¼ const

in dV:

8>><>>: (10)

Motion vector~n and the unknown constantw9 can be deter-mined by minimizing the following error function with thelinear least-squares method,

E¼∑dx

∑dy

∑dt

(ft þ nxfx þ nyfy ¹ fw9)2: (11)

We tentatively call this approach ‘the extended constraintequation with spatio-temporal local optimization’ (E-STO).Since this method can manipulate many equations effec-tively compared with SLO and TLO, it is expected to deter-mine more accurate motion fields. The comparison ofESTO, SLO and TLO is summarized in Table 1. It is alsopossible to introduce smoothness constraints or the regular-ization approach to solve Eq. (9).

3.2. The pixel-based temporal filtering

In this paper, we also introduce a new approach based ona pixel-based image sequence processing [20]. For a sequen-tial image, temporal change of the brightness at each pixel isregarded as a time series. The proposed method is based ondigital Fourier Transform and digital filtering under theassumption of local constancy of statistical characteristicsof the time series. We evaluate temporal development of thespectrum within a finite time at each pixel by Fast FourierTransform. After a digital filtering of the spectrum, inversetransformation is carried out at each pixel site to createfiltered image sequences (see Fig. 2).

Let a temporal brightness change of raw imagesequence at a pixel coordinate (x,y) be f(x,y,t). We definea local time-domain digital Fourier Transform within a localtime-window dT at aroundt. The reduced instantaneousspectrumF(x,y,t;qK) can be represented by

F(x,y, t; qK) ¼∑t þ dT=2

t ¹ dT=2

f (x,y, t)exp( ¹ iqKt),

whereqK ¼ 2pk/dT. After a digital filtering of the spectrum,inverse transformation is carried out to create filtered imagefac(x,y,t):

fac(x,y, t) ¼∑

Fac(x, y, t; qK)exp(iqKt),

where Facðx; y; t; qKÞ ¼ F(x,y,t;qK Þ 0). By shifting thetime-window step-by-step, we can create filtered image

Table 1The comparison of the proposed method (ESTO) with the method of assumption of separability of non-uniform illumination proposed by Nomura et al. (1995)[8]

Non-uniformillumination condition

Constraint equation Error function (E) Solution

Spatial non-uniform]f]t

þ ~n·grad(f ) ¼ f (x,y, t)��������������n2

x þ n2y

qq(x,y)

∑dt ft þ nxfx þ nyfy ¹ fq

��������������n2

x þ n2y

q� �2temporal local optimization(TLO)

Temporal non-uniform]f]t

þ ~n·grad(f ) ¼ f (x,y, t)w(t)∑

dx

∑dy (ft þ nxfx þ nyfy ¹ fw)2 spatial local optimization

(SLO)Proposed method: Spatio-temporal non-uniform

]f]t

þ ~n·grad(f ) ¼ f (x,y, t)w9(x, y, t)∑

dx

∑dy

∑dt (ft þ nxfx þ nyfy ¹ fw9)2 spatio-temporal local

optimization (ESTO)

311L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

sequence (ac-image). From the raw image sequence, asequential image is created to enhance the brightness ofmoving object and because that the DC-component isremoved in the filtered image sequence, it is also expectedto reduce the influence of non-uniform (spatial and

temporal) illumination. In the previous report [20], we pro-posed this method mainly for the motion enhancement. Inthe report, we test the performance of the temporal filteringas a preprocessing tool for the detection of motion fields.

4. Experiment

In this section, we try to apply the proposed methods todetermine motion fields under non-uniform illumination(containing non-stationary illumination with respect totime and non-uniform illumination with respect to space).We compare the proposed methods with the conventionalmethods and discuss the performance of the proposed meth-ods. The experimental data include two synthetic imagesequences and one real image sequence.

4.1. Experimental images analysis

4.1.1. Yosemite sequence1 (synthetic data)Fig. 3 shows a snapshot of synthetic image sequence

(Yosemite sequence). The image sequence has a resolutionof 316 3 252 pixels. The brightness is quantified into 256steps. TheYosemite sequenceis a complex test scene. In thescene, the cloud has a translational motion with a speed of2 pixels/frame, while the speed in the lower left is about 4–5 pixels/frame. However, the brightness of cloud changeswith respect to time and space. The landscape (mountains,valley, etc.) moves against depth direction. Then, motionfield expands. Namely, the motion field has divergencecharacteristics(div(~n) Þ 0). This sequence is challengingbecause of the range of velocities, occluding edges betweenthe mountains and at the horizon, divergence and non-uniform illumination. Fig. 4 represents the theoreticalmotion fields of Fig. 3.

4.1.2. Rotated Yosemite sequenceunder non-uniformillumination (synthetic data)

RotatedYosemite sequenceis created from a static image(the first frame ofYosemite sequence) shown in Fig. 3. Theexample of theRotated Yosemite sequence f(x,y,t) simulat-ing spatially and temporally non-uniform illumination iscreated by the equation:

f (x,y, t) ¼ r1(x,y)·r2(t)·g(x, y, t),

where

r1(x,y) ¼ e¹

(x¹ x_center)2

j2x

þ(y¹ y_center)2

j2y

� �represents the effect of spatially non-uniform illumination,r2(t) ¼ sinð pt

t_sizeÞ represents the effect of temporally non-uni-form illumination andr(x,y,t) ¼ r 1(x,y)·

Fig. 2. A schematic explanation of a pixel-based temporal filtering.

Fig. 3. A simulation image sequence (Yosemite sequence).

1 The image sequence of the Yosemite sequence is obtained from the ftp-site of ftp.csd.uwo.ca.

312 L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

r 2(t) represents the effect of spatially and temporally non-uniform illumination. g(x,y,t) is obtained by rotating thestatic image (see Fig. 3) clockwise around its center axiswith constant angular velocity (0.8 degree/frame). The sizeof the Rotated Yosemite sequenceis 2003 150 pixels and50 frames, the center axis is atx_center¼ 100,y_center¼75, we takejx ¼ 100,jy ¼ 75 andt_size¼ 50. Fig. 5 showsthe 10th and 20th frames of the images. The theoreticalmotion fields are shown in Fig. 6.

4.1.3. Toy car sequence (real data)In order to confirm the usefulness of the proposed

methods, we took sequential images of toy car motions ona floor affected by spatially and temporally non-uniformillumination through a TV camera with a samplingfrequency of 30 Hz. The size of theToy car sequenceis

236 3 110 pixels and 40 frames. Brightness is quantifiedinto 256 steps. The toy car moves from lower left to upperright. The spatially non-uniform illumination is from 27065 to 13606 10 lux and the temporally non-uniform illumi-nation is from 456 2 to 7656 5 lux. Fig. 7 shows the 2ndand 32nd frames of theToy car sequence.

4.2. Experimental results

In this section, we report the quantitative performance ofconventional methods and proposed methods on the syn-thetic image sequences and also show the motion fieldsproduced by the methods on the real image sequence.

4.2.1. Comparison of conventional methods with the E-STO(proposed method 1)

We try to determine motion fields by use of the followingmethods.

First, we use the conventional gradient-based method(Horn and Schunck [7]).

Fig. 4. Theoretical motion fields of theYosemite sequence.

Fig. 5. The 10th and 20th frames of theRotated Yosemite sequence.

Fig. 6. Theoretical motion fields of theRotated Yosemite sequence.

Fig. 7. The 2nd and 32nd frames of theToy car sequence.

313L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

• Spatial Global Optimization (SGO):Horn and Schunck [7] combined the basic constraint

equation with a global smoothness term to constrain theestimated motion field~n ¼ (nx,ny), minimizing

∫∫nx

]f]x

þ ny]f]y

þ]f]t

� �2

þ a2(k=nxk22 þ k=nyk

22)

� �dxdy:

We useda ¼ 0.5 and 100 iterations as suggested by Barron[5] in testing below.

The original method described by Horn and Schunck [7]used first-order difference to estimate intensity derivatives.Because this is a relatively crude form of numerical dif-ferentiation and can be the source of considerable error,following Barron [5], we implemented the method withspatio-temporal Gaussian presmoothing (jxy ¼ j t ¼ 1.5)and four-point central differences for differentiation (withmask coefficients 1/12(¹1,8,0,¹8,1)). The results ofobtained motion fields by using modified Horn and Schunckmethod is shown in Fig. 8a–c.

Second, we use the conventional gradient-based methodof local optimization.

• Spatio-Temporal Local Optimization (STO)We also tested conventional gradient-based method with

spatio-temporal local optimization. We assume that motionvector is constant with respect to time and space in a localvolume of 53 5 pixels and 8 frames. The result of obtainedmotion fields is shown in Fig. 9a–c.

Third, we use the proposed method 1 (E-STO) based onEq. (11). We assume that the motion vector~n and w9 areconstant in a local volume of 53 5 pixels and 8 frames. Theresult of obtained motion fields is shown in Fig. 10a–c.

The motion fields obtained by the conventional gradient-based methods (with SGO and STO) have serious errors atthe fields under non-uniform illumination where thebrightness distribution changes temporally and spatially(e.g. at the cloud position in theYosemite sequence).When we apply the extended constraint equation underthe assumption of Eq. (11), motion fields obtained at theseplaces are apparently improved. However, for theYosemitesequence, reduction of the error at foreground mountainsurface is not satisfactory. Because the texture of fore-ground mountain surface is only pinstriped, it is easy toencounter the aperture problem. When we try to determinemotion fields by the gradient method, we have to considerthe aperture problem, which cannot be solved locally. If thepinstriped texture area is larger than the observation area, itis hard to obtain the correct motion fields with the localoptimization method. In general, it is not possible tocompute true velocity and direction by the observationwithin a small neighborhood (a local area). In order to over-come the shortage, it seems effective to introduce a spatialpresmoothing (e.g. Gaussian spatial presmoothing) based onthe image sequence (see Fig. 13). We also can introducethe global optimization techniques [7] and hierarchicalapproaches [6].

4.2.2. Comparison of conventional method with raw imagesand with temporal filtered images (proposed method 2)

In Figs. 11 and 12, we demonstrate a different approach toremove the influence of non-uniform illumination in motion

Fig. 8. (a) Motion field ofYosemite sequencedetermined by SGO: Horn andSchunck (modified) (conventional gradient-based method). (b) Motion fieldof Rotated Yosemite sequencedetermined by SGO: Horn and Schunck(modified) (conventional gradient-based method). (c) Motion field ofToycar sequencedetermined by SGO: Horn and Schunck (modified) (conven-tional gradient-based method).

314 L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

analysis. The raw image sequences to be analysed are alsoYosemite sequence, Rotated Yosemite sequenceandToy carsequenceshown in Figs. 3, 5, and 7, respectively. Examplesof temporally filtered image sequence (ac-image) obtained

by the temporal filtering described in Section 3.2 are shownin Fig. 11a–c.

We applied the conventional gradient-based method withSTO to obtain motion fields from the temporal filtered

Fig. 9. (a) Motion field ofYosemite sequencedetermined by STO (conven-tional gradient-based method). (b) Motion field ofRotated Yosemitesequencedetermined by STO (conventional gradient-based method). (c)Motion field of Toy car sequencedetermined by STO (conventionalgradient-based method).

Fig. 10. (a) Motion field ofYosemite sequencedetermined by the proposedmethod 1 (E-STO). (b) Motion field ofRotated Yosemite sequencedeter-mined by the proposed method 1 (E-STO). (c) Motion field ofToy carsequencedetermined by the proposed method 1 (E-STO).

315L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

image sequence (ac-image, see Fig. 11a–c). Fig. 12a–cshows the analysed motion fields. By comparison amongthe analysed motion fields (Figs. 8, 9, and 12), the error ofmotion vector estimation at the places (non-uniformillumination) are reduced apparently in Fig. 12. Finally,for theYosemite sequencewe introduced a Gaussian spatialpresmoothing to cope with the aperture problem. Theconventional gradient-based method (with STO) is appliedto the temporal filtered image sequence (ac-image, seeFig. 11a) by introducing the Gaussian spatial presmoothing

(jxy ¼ 1.0) simultaneously. The result is shown in Fig. 13.Apparently, the obtained motion fields confirm to us thebetter performance. For more detailed and quantitativeevaluation see Table 2 and Table 3 in Section 4.3.

Fig. 11. (a) A temporal filtered image sequence: ac-image (fromYosemitesequence, dT ¼ 8). (b) A temporal filtered image: ac-image (fromRotatedYosemite sequence, dT¼ 16). (c) A temporal filtered image: ac-image (fromToy car sequence,dT ¼ 2).

Fig. 12. (a) Motion field ofYosemite sequencedetermined by STO: Appliedto ac-images (conventional gradient-based method). (b) Motion field ofRotated Yosemite sequencedetermined by STO: applied to ac-images (con-ventional gradient-based method). (c) Motion field ofToy car sequencedetermined by STO: applied to ac-images (conventional gradient-basedmethod).

316 L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

4.2.3. Application of the ESTO to the filtered ac-imageWe also test the situation to apply the ESTO method to

the temporal filtered ac-image (combination of proposedmethod 1 with proposed method 2). The results are shownin Fig. 14a–c. Because the influence of non-uniform illumi-nation has been removed from the raw images in the ac-image, the improvement of the combination of proposedmethod 1 with proposed method 2 is not very apparent,comparing it with the proposed method 2. But comparingFig. 14a with Fig. 13, the motion fields obtained at the cloudarea (upper left) are improved slightly.

4.3. Error measurement

We tested the proposed algorithms on the synthetic andreal images. For the experiment on the synthetic imageswhere the correct motion fields are known, we can use theangular measure of error used by Barron et al. (1994) [5] to

Fig. 13. Motion field ofYosemite sequencedetermined by STO: appliedto ac-images and Gaussian spatial presmoothing images (conventionalgradient-based method).

Table 2Summary of theYosemite sequencemotion fields results

Algorithm Ave. Error S.D. Frames1 G_s_s2 G_t_s3

Horn and Schunck (original) 31.69 31.18 2 No NoHorn and Schunck (modified) 9.78 16.19 15 Yes YesUras et al. 8.94 15.61 15 Yes YesNagel 10.22 16.51 15 Yes YesAnandan 13.36 15.64 2 No NoSingh 10.44 13.94 3 No NoConventional STO 10.18 16.40 8 No NoTLO (based on Eq. (5)) 14.37 20.23 8 No NoSLO (based on Eq. (7)) 10.49 17.64 3 No NoProposed method 1(ESTO: based on Eq. (11)) 8.47 15.32 8 No NoProposed method 2(STO: based on ac-images) 8.05 14.94 8 No NoProposed method 2 with G_s_s(STO: based on ac-images) 5.70 11.15 8 Yes NoProposed method 3 with G_s_s(ESTO: based on ac-images) 5.59 11.24 8 Yes No

The error comparison of the proposed methods with other techniques reported by Barron et al. (1994) [5]. All compared methods provide 100% density ofmotion vector.

Notes: 1 Frames represents the input frames of images the technique required.2 G_s_s represents the Gaussian spatial presmoothing used.3 G_t_srepresents the Gaussian temportal presmoothing used.

Table 3Summary of theRotated Yosemite sequencemotion fields results

Algorithm Ave. Error S.D. Frames1 G_s_s2 G_t_s3

Horn and Schunck (original) 11.99 11.06 2 No NoHorn and Schunck (modified) 10.57 9.53 15 Yes YesConventional STO 3.91 4.41 8 No NoProposed method 1(ESTO: based on Eq. (11)) 2.61 2.91 8 No NoProposed method 2(STO: based on ac-images) 2.47 2.16 8 No NoProposed method 3(ESTO: based on ac-images) 2.46 2.43 8 No No

The error comparison of the proposed methods with Horn and Schunck technique. The all compared methods provides 100% density of motion vector.Notes: 1 Frames represents the input frames of images the technique required.2 G_s_s represents the Gaussian spatial presmoothing used.3 G_t_s

represents the Gaussian temporal presmoothing used.

317L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

evaluate the results. It is favorable to compare our methodswith other techniques [5]. They measure the error betweenthe correct velocity~nc ¼ (nx, ny) and the estimate~ne ¼ (n̂x, n̂y)as the angle between the unit vectors in 3D space,

~n3 ; 1=����������������������n2

x þ n2y þ 1

q� �(nx, ny, 1):

The angular error between the correct vector~n3c and theestimate~n3e is

wE ¼ arccos(~n3c·~n3e): (12)

The error comparison of the proposed methods with othertechniques reported by Barron et al. [5] is summarized inTables 2 and 3, which list the average and standard devia-tion of the angular error. The comparison with TLO andSLO (see Table 1) is also carried out. The proposed method3 records the best performance.

5. Conclusions

In this paper, we proposed two methods to determinemotion fields from an image sequence under non-uniformillumination. The first method is based on the extendedconstraint equation from the conservation law of totalbrightness in a fixed observation area. Thus, it is possibleto estimate the effects of non-uniform illumination and truemotion fields. Since we adopted the spatio-temporal localoptimization, we obtained high resolution and high reliabil-ity of the determined motion fields compared to the conven-tional gradient method. The performance of the proposedmethod was confirmed by the analysis of two syntheticimage sequences and one real image sequence.

As the second method, we propose a different approach toremove the influence of non-uniform illumination. Thealgorithm is based on a local temporal filtering. From anoriginal image sequence, a dynamic scenes (ac-image) iscreated, which is defined in a local time-domaindT ataroundt. By experimentation, the reduction of the influenceof non-uniform illumination is also confirmed.

It is confirmed that the influence of non-uniform illumi-nation is reduced by the proposed methods. However, forthe aperture problem, the two methods proposed are noteffective because of the local approach. For theYosemitesequencethe mountain area (lower-left front region in theraw image sequence) has a parallel high-contrast pattern ofpinstripe. To cope with this aperture problem, we can intro-duce approaches such as regularization method. For asimple approach, we tested the Gaussian spatial presmooth-ing combined with the proposed temporal filtering. Theperformance of the filter is clearly demonstrated in Fig.13. Thus, the proposed methods are hopeful for complicatedactual scene analysis.

An advantage of the first method can be the possibility toevaluate the non-uniform illumination quantitatively.Further investigation considering the neglected term(div(~n)) and testing global approaches are expected.

Acknowledgements

The authors wish to thank Dr A. Nomura, Professor E.

Fig. 14. (a) Motion field ofYosemite sequencedetermined by ESTO:applied to ac-images and Gaussian spatial presmoothing images (proposedmethod 1þ proposed method 2). (b) Motion field ofRotated Yosemitesequencedetermined by ESTO: applied to ac-images (proposed method1þ proposed method 2). (c) Motion field ofToy car sequencedeterminedby ESTO: applied to ac-images (proposed method 1þ proposed method 2).

318 L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

Yokoyama and Dr Y. Mizukami for their critical commentsand helpful discussions. This work partly supported by theSasakawa Scientific Research Grant from The JapanScience Society.

Appendix A The model of spatio-temporal non-uniformillumination

We assume the spatio-temporal brightness distributionf(x,y,t) is

f (x,y, t) ¼ r1(x,y)·r2(t)·g(x,y, t), (A1)

where r 1(x,y) represents the effect of spatial non-uniformillumination, r 2(t) represents the effect of temporal non-uniform illumination andg(x,y,t) is the virtual brightnessdistribution under uniform illumination. Now the followingequation is adopted for determining motion field under non-uniform illumination:

]f]t

þ ~n·grad(f ) ¼ f: (A2)

Substituting Eq. (A1) we obtain

](r1r2g)]t

þ ~n·grad(r1r2g) ¼f: (A3)

Expansion of the left side of Eq. (A3)

gr2]r1

]tþ ~n·grad(r1)

� �þ gr1

]r2

]tþ ~n·grad(r2)

� �þ r1r2

]g]t

þ ~n·grad(g)� �

¼ f:

(A4)

Where we assume thatg(x,y,t) obeys the equation (becauseof uniform illumination)

]g]t

þ ~n·grad(g) ¼ 0: (A5)

Then we obtain the relationship

gr2]r1

]tþ ~n·grad(r1)

� �þ gr1

]r2

]tþ ~n·grad(r2)

� �¼ f

(A6)

where r 1(x,y) represents the effect of spatial non-uniformillumination, r 2(t) represents the effect of temporal non-uniform illumination. Then

r1 ¼ r1(x, y),]r1

]t¼ 0,

r2 ¼ r2(t), grad(r2) ¼ 0: (A7)

Substituting Eq. (A6) with Eq. (A7) we obtain

gr2·~ngrad(r1) þ gr1·]r2

]t¼ f, (A8)

fr1

·~n·grad(r1) þfr2

·]r2

]t¼ f: (A9)

Thenf is expressed as

f ¼ f ~n·grad(r1)

r1þ f·

]r2=]tr2

: (A10)

Here we introduce a vector

~p(x,y) ¼ grad(r1(x,y))=r1(x, y):

Then Eq. (A10) is expressed as

f ¼ f ~n·~pþ f·]r2=]t

r2¼ f l~nk~plcosa þ f·

]r2=]tr2

¼ f��������������n2

x þ n2y

ql~plcosa þ f·

]r2=]tr2

:

(A11)

wherea(x,y) is the angle between~n(x,y) and ~p(x, y). Withthe assumption of spatio-temporal optimization(]~n=]x¼

]~n=]y¼ ]~n=]t ¼ 0 in dV ¼ dx·dy·dtÞ, the term~p(x,y), aðx; yÞand

]r2(t)=]tr2(t)

are also constant indV. If the symbolq(x,y) and w(t) areused as unknown constants, Eq. (A11) is rewritten as

f(x,y, t) ¼ fq(x, y)��������������n2

x þ n2y

qþ fw(t), (A12)

whereq(x,y) ¼ l~plcosa,

w(t) ¼]r2=]t

r2:

References

[1] B.K.P. Horn, Robot vision. MIT Press, Cambridge, MA, 1986.[2] A. Verri, T. Poggio, Motion field and optical flow qualitative proper-

ties, IEEE Trans. Pattern Anal. Machine Intell. 11 (1989) 490–498.[3] F. Bergholm, S.A. Carlsson, ‘Theory’ of optical flow, CVGIP:

Graphic Models and Image Processing 53 (2) (1991) 171–188.[4] B. Jahne, Digital Image Processing, Springer-Verlag, Berlin, 1995,

pp. 53–230.[5] J.L. Barron, D.J. Fleet, S.S. Beauchemin, Systems and experiment—

performance of optical flow techniques, Intern. J. Comput. Vis. 12 (1)(1994) 43–77.

[6] F. Glazer, G. Reynolds and P. Anandan, Scene matching by hierar-chial correlation, IEEE Computer Society, 432-441, 1983.

[7] B.K.P. Horn, B.G. Schunck, Determining optical flow, Artifical Intell.17 (1981) 185–203.

[8] A. Nomura, H. Miike, K. Koga, Determining motion fields under non-uniform illumination, Pattern Recog. Letters 16 (1995) 285–296.

[9] E. Dubois and J. Konrad, Estimation of 2D motion fields from imagessequences with application to motion compensated processing,Motion Analysis and Image Sequence Processing, Kluwer AcademicPublishers, Dordrecht, pp. 53–87, 1993.

[10] J.A. Lees, C.S. Novak, B.B. Clark, An automated technique forobtaining cloud motion from geosynchronous satellite data usingcross correlation, Journal of Applied Meterology 10 (1971) 118–132.

[11] M.J. Black, The robust estimation of multiple motions: Parametric andpiecewise-smooth flow fields, Comput. Vision and Image Understand-ing 63 (1) (1996) 75–104.

[12] A. Verri, F. Girosi, V. Tore, Differential techniques for optical flow, J.Opt. Soc. Am. A7 (1990) 912–922.

319L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320

[13] A. Nomura, H. Miike and E. Yokoyama, Detecting motion anddiffusion from a dynamic image sequence, Trans. of the Institute ofElectronics Engineers Japan,115, 3, 4003, 1995 (in Japanese).

[14] A. Nomura, H. Miike, K. Koga, Field theory approach for determiningoptical flow, Pattern Recog. Letters 12 (1991) 183–190.

[15] J.K. Kearney, W.B. Thompson, D.L. Boley, Optical flow estimation:an error analysis of gradient-based methods with local optimization,IEEE Trans. Pattern Anal. Machine Intell 9 (1987) 229–244.

[16] N. Cornelius and T. Kanade, Adapting optical flow to measure objectmotion in reflectance and X-ray image sequence, Proc. ACMSIGGRAPH/SIGGART Interdisciplinary Workshop on Motion: Repre-sentation and Perception, Toronto, Ontario, Canada, pp. 145–153, 1983.

[17] J. Aisbett, Optical flow with an intensity-weighted smoothing, IEEETrans. Pattern Anal. Machie Intell. 11 (1989) 512–522.

[18] A. Singh, Optical Flow Computation: A Unified Perspective, IEEEComputer Society Press, Los Alamitos, California, 1991.

[19] T. Hara, T. Kudou, H. Miike and E. Yokoyama, A. Nomura, Recover-ing 3D-shape from motion stereo under non-uniform illumination,IARP MVA, pp. 241–244, 1996.

[20] H. Miike, T. Sakurai, L. Zhang and H. Yamada, Motion enhancementand visualization of dynamic streamline by pixel-based time-domainfiltering of image sequence (submitted).

320 L. Zhang et al. / Image and Vision Computing 17 (1999) 309–320