TR2001-413, Dartmouth College, Computer Science

Differential Elastic Image Registration

Senthil Periaswamy and Hany FaridDepartment of Computer Science

Dartmouth CollegeHanover NH 03755

We have applied techniques from differential motion estimation to the problem of automaticelastic registration of medical images. This method models the mapping between images asa locally affine but globally smooth warp. The mapping also explicitly accounts for varia-tions in image intensities. This approach is simple and highly effective across a broad rangeof medical images. We show the efficacy of this approach on several synthetic and clinicalimages.

1

1. Introduction

There are a variety of methods for medical imageregistration (see [9, 10, 26, 19, 17] for general sur-veys). Differential registration techniques, how-ever, are often cited as being ineffective, and assuch have received little attention (exceptions in-clude [4, 16, 20, 21]). This is unfortunate as dif-ferential motion techniques have been quite effec-tive in the Computer Vision community (e.g., [15,18, 25, 1, 14, 2, 13, 5, 12, 6, 7, 24]).

Here we present an effective technique for elas-tic image registration built upon a differential frame-work. This technique models the mapping be-tween images as a locally affine but globally smoothwarp, and explicitly accounts for variations in im-age intensities. The resulting registration is sim-ple and automatic. Results from several syntheticand clinical images are shown.

2. Methods

We formulate the problem of image registrationbetween a source and target image within a dif-ferential (non-feature based) framework. This for-mulation borrows from various areas of motionestimation. We first outline the basic computa-tional framework, and then discuss several im-plementation details that are critical for a success-ful implementation.

2.1. Local affine model

Denote f(x, y, t) and f(x, y, t − 1) as the sourceand target images, respectively. 1 We begin byassuming that the image intensities between im-ages are conserved, and that the motion betweenimages can be modeled locally by an affine trans-form:

f(x, y, t) =

f(m1x + m2y + m5, m3x + m4y + m6, t− 1),(1)

where m1, m2, m3, m4 form the 2 × 2 affine ma-trix, and m5, m6 the translation vector. These pa-

1We adopt the slightly unconventional notation of denot-ing the source and target image with a temporal parame-ter t. This is done for consistency within our differentialformulation.

rameters are estimated locally for each small spa-tial neighborhood, but for notational conveniencetheir spatial parameters are dropped. In order toestimate these parameters, we define the follow-ing quadratic error function to be minimized:

E(~m) =∑

x,y∈Ω

[f(x, y, t)−

f(m1x + m2y + m5,

m3x + m4y + m6, t− 1)]2 , (2)

where ~mT = (m1 . . . m6 ), and Ω denotes asmall spatial neighborhood. Since this error func-tion is non-linear in its unknowns, it cannot beminimized analytically. To simplify the minimiza-tion, we approximate this error function using afirst-order truncated Taylor series expansion:

E(~m) ≈

∑

x,y∈Ω

(f(x, y, t)−

[f(x, y, t) +

(m1x + m2y + m5 − x)fx(x, y, t) +

(m3x + m4y + m6 − y)fy(x, y, t)−

ft(x, y, t)])2 , (3)

where fx(·), fy(·), ft(·) are the spatial/temporalderivatives of f(·). This error function further re-duces to:

E(~m) =∑

x,y∈Ω

[ft(x, y, t)−

(m1x + m2y + m5 − x)fx(x, y, t)−

(m3x + m4y + m6 − y)fy(x, y, t)]2 .(4)

Note that this quadratic error function is now lin-ear in its unknowns, ~m. This error function maybe expressed more compactly in vector form as:

E(~m) =∑

x,y∈Ω

[

k − ~cT ~m]2

, (5)

where the scalar k and vector ~c are given as:

k = ft + xfx + yfy (6)~cT = (xfx yfx xfy yfy fx fy ) , (7)

where again, for notational convenience, the spa-tial/temporal parameters of fx(·), fy(·), and ft(·)

2

are dropped. This error function can now be min-imized analytically by differentiating with respectto the unknowns:

dE (~m)

d~m=

∑

x,y∈Ω

−2~c[

k − ~cT ~m]

, (8)

setting this result equal to zero, and solving for ~mto yield:

~m =

∑

x,y∈Ω

~c ~cT

−1

∑

x,y∈Ω

~ck

. (9)

This solution assumes that the first term, a 6 × 6matrix, is invertible. This can usually be guar-anteed by integrating over a large enough spa-tial neighborhood Ω with sufficient image con-tent. With this approach a dense locally affinemapping can be found between a source and tar-get image.

2.2. Intensity variations

Inherent to the model outlined in the previoussection is the assumption that the image intensi-ties between the source and target are unchanged.This assumption is likely to fail under a numberof circumstances. To account for intensity vari-ations, we incorporate into our model an explicitchange of local contrast and brightness [22]. Specif-ically, our initial model, Equation (1), now takesthe form:

m7f(x, y, t) + m8 =

f(m1x + m2y + m5, m3x + m4y + m6, t− 1),(10)

where m7 and m8 are two new (spatially varying)parameters that embody a change in contrast andbrightness, respectively. Note that these parame-ters have been introduced in a linear fashion. Asbefore, this error function is approximated witha first-order truncated Taylor series expansion toyield:

E(~m) =∑

x,y∈Ω

[

k − ~cT ~m]2

, (11)

where the scalar k and vector ~c are now given as:

k = ft − f + xfx + yfy (12)~cT = (xfx yfx xfy yfy fx fy − f − 1) ,(13)

Minimizing this error function is accomplishedas before by differentiating E(~m), setting the re-sult equal to zero and solving for ~m. The solutiontakes the same form as in Equation (9), with k and~c as defined above.

Intensity variations are typically a significantsource of error in differential motion estimation.The addition of the contrast and brightness termsallows us to accurately register images in the pres-ence of local intensity variations. It is possible, ofcourse, to fully explain the mapping between im-ages with only a brightness modulation. In thenext section we describe how to avoid such a de-generate solution.

2.3. Smoothness

Until now, we have assumed that the local affineand contrast/brightness parameters are constantwithin a small spatial neighborhood, Equation (11).There is a natural tradeoff in choosing the size ofthis neighborhood. A larger area makes it morelikely that the matrix

∑

x,y∈Ω ~c ~cT will be invert-ible, Equation (9). A smaller area, however, makesit more likely that the constancy assumption willhold. We can avoid balancing these two issues byreplacing the constancy assumption with a smooth-ness assumption [14]. That is, we assume that themodel parameters ~m vary smoothly across space.A smoothness constraint on the contrast/brightnessparameters has the added benefit of avoiding adegenerate solution where a pure brightness mod-ulation is used to describe the mapping betweenimages.

To begin, we augment the error function in Equa-tion (11) as follows:

E(~m) = Eb(~m) + Es(~m), (14)

where Eb(~m) is defined as in Equation (11) with-out the summation:

Eb(~m) =[

k − ~cT ~m]2

, (15)

with k and ~c as in Equations (12) and (13). Thenew quadratic error term Es(~m) embodies the smooth-ness constraint:

Es(~m) =8

∑

i=1

λi

[

(

∂mi

∂x

)2

+

(

∂mi

∂y

)2]

,(16)

3

where λi is a positive constant that controls therelative weight given to the smoothness constrainton parameter mi.

This error function is again minimized by dif-ferentiating, setting the result equal to zero andsolving, dE (~m) /d~m = dEb (~m) /d~m+dEs (~m) /d~m =0. The derivative of Eb(~m) is:

dEb (~m)

d~m= −2~c

[

k − ~cT ~m]

. (17)

The derivative of Es(~m) is computed by first ex-pressing the partials, ∂mi/∂x and ∂mi/∂y withdiscrete approximations [14], and then differenti-ating, to yield:

dEs (~m)

d~m= 2L(~m− ~m), (18)

where ~m is the component-wise average of ~m overa small spatial neighborhood, and L is an 8 × 8diagonal matrix with diagonal elements λi, andzero off the diagonal. Setting

dEb (~m) /d~m + dEs (~m) /d~m = 0, (19)

and solving for ~m at each pixel location yieldsan enormous linear system which is intractableto solve. As such, we express ~m in the followingform:

~m =(

~c ~cT + L)

−1 (

~c k + L~m)

, (20)

and employ an iterative scheme to solve for ~m [14].An initial estimate of ~m is determined using theclosed-form solution of Section 2.2. This solutionyields an initial estimate of ~m, from which a newestimate of ~m is obtained, Equation (20). This pro-cess is repeated, where on each iteration a newestimate of ~m is computed from the previous so-lution.

The use of a smoothness constraint has the ben-efit that it yields a dense locally affine but glob-ally smooth mapping. The drawback is that theminimization is no longer analytic. We have found,nevertheless, that the iterative minimization is quitestable and converges relatively quickly.

2.4. Implementation details

While the formulation given in the previous sec-tions is relatively straight-forward there are a num-ber of implementation details that are critical fora successful implementation.

First, in order to simplify the minimization, theerror function of Equation (15) was derived througha Taylor-series expansion. A more accurate es-timate of the actual error function can be deter-mined using a Newton-Raphson style iterative scheme [23].In particular, on each iteration, the estimated warpis applied to the source image, and a new warp isestimated between the newly warped source andtarget image. As few as five iterations greatly im-proves the final estimate.

Second, the required spatial/temporal deriva-tives have finite support thus fundamentally lim-iting the amount of motion that can be estimated.A coarse-to-fine scheme is adopted in order tocontend with larger motions [18, 3]. A Gaussianpyramid is built for both source and target im-ages, and the local affine and contrast/brightnessparameters estimated at the coarsest level. Theseparameters are used to warp the source image inthe next level of the pyramid. A new estimate iscomputed at this level, and the process repeatedthrough each level of the pyramid. The warps ateach level of the pyramid are accumulated yield-ing a single final warp.

Finally, the calculation of the spatial/temporalderivatives is a crucial step. Spatial/temporal deriva-tives of discretely sampled images are often com-puted as differences between neighboring samplevalues. Such differences are typically poor ap-proximations to derivatives and lead to substan-tial errors. In computing derivatives we employa set of derivative filters specifically designed formulti-dimensional differentiation [11]. These fil-ters significantly improve the resulting registra-tion.

3. Results

In all of the examples shown here, the source andtarget are 256 × 256, 8-bit grayscale images withintensity values scaled into the range [0, 1]. A three-level Gaussian pyramid is constructed for both

4

the source and target image. At each pyramidlevel a single global affine warp is first estimatedaccording to Equation (11), with Ω, the spatial in-tegration window, defined to be the entire image.Then, the local affine and contrast/brightness pa-rameters, ~m are estimated according to Equation (11),with Ω = 5×5 pixels. This estimate of ~m is used tobootstrap the smoothness iterations, Equation (20).In each iteration, λi = 1× 1011, i = 1, ..., 8 and mi

is computed by convolving with the 3 × 3 ker-nel (1 4 1 ; 4 0 4 ; 1 4 1)/20. After 40 itera-tions, the source is warped according to the fi-nal estimate, and this process is repeated 5 times.This entire process is repeated at each level of thepyramid. Although a contrast/brightness mapis estimated, it is not applied when warping thesource image. In order to minimize artifacts dueto the warping, we accumulate successive warpsand apply a single warp to the original sourceimage at each scale. In order to minimize edgeartifacts, all convolutions are performed with amirror-symmetric boundary. The temporal deriva-tives are computed using a 2-tap filter, and thespatial derivatives using a 3-tap filter. All of theseparameters were held fixed in all of the examplesshown here. In general we find that the particu-lar choice of these parameters is not crucial. Ourcurrent MatLab implementation requires approx-imately 25 minutes per image on a 1.2 GHz Linuxmachine.

To test our registration algorithm, we gener-ated synthetic data by applying a locally smoothwarp and contrast/brightness field to a target im-age. These smooth fields were randomly gener-ated by specifying a warp parameter at equallyspaced points along a coarse rectilinear grid. In-termediate values were interpolated using Book-stein’s thin-plate splines [8]. On average each pixelwas warped by ±8 pixels (not including a pos-sible global affine transform), the multiplicativecontrast variation was between 0.8 and 1.0, andthe additive brightness variation between 0.0 and0.2 (with image intensities in [0, 1]). Shown in Fig-ure 2(d)-(f) are examples of these warp fields.

Shown in Figure 2(a) and (b) is a syntheticallygenerated source, and target image, respectively.Shown in panel (c) is the source image after reg-

istration. Shown in panels (d) and (e) are the ap-plied contrast/brightness maps, and shown in pan-els (g) and (h) are the estimated maps. Note thatwhile there is a tradeoff between the estimatedcontrast and brightness maps, errors in this es-timate do not impact the estimated warp field.Shown in panel (f) is the synthetic warp as ap-plied to a rectilinear grid. Shown in panel (i) isthe result of applying the inverse of the estimatedwarp to panel (f). If the estimate was perfect, theresult should be a rectilinear grid. Notice that inthe areas of image content, this is nearly the case.

Shown in Figure 3 are results from four moresynthetically warped images. In each case, a dif-ferent random warp and contrast/brightness fieldwas applied to the source image. In each case,the registered source image is in good agreementwith the target image.

Shown in Figure 4 are results from four clini-cal cases. In each case, the source and target im-ages are either from different subjects, from sub-jects at different times, or from different modal-ities. Shown across each row are the source andtarget images, the registered source, and the es-timated warp. Even in the presence of signifi-cant intensity variations, the registered source isin good agreement with the target image.

And finally shown in Figure 1 are the resultsfrom an extreme and completely unrealistic syn-thetic warp, which we show to illustrate the ro-bustness and flexibility of our registration tech-nique. Unlike the previous examples, the modelused here is one of translation only (i.e., no affineor contrast/brightness terms), and the smooth-ness parameters on these translation terms wasreduced to 1 × 10−2. These small changes werenecessary to accommodate the extreme nature ofthe synthetic warp.

4. Discussion

We have presented an elastic registration algo-rithm built upon a differential framework. Ourregistration model incorporates both a geometricmapping that is locally affine and globally smooth,and contrast/brightness modulations that are glob-ally smooth. The simple differential estimation

5

source target

warped source

Figure 1: Shown is a target image, a source im-age exposed to an extreme warp, and the resultof our registration.

techniques, and the flexibility of the model hasproven to be highly effective across a broad rangeof medical images. We have tested our algorithmon a number of synthetic images generated ac-cording to the assumptions of our model. We havealso shown the efficacy of our algorithm on clin-ical images, thus suggesting that these assump-tions are reasonable.

Our current implementation suffers from a fewshortcomings. First, the registration of a 256 ×256 image requires approximately 45 minutes ona high-end PC. We are hopeful that optimizationof our algorithm and a C-based implementationwill significantly reduce this run-time. Our im-plementation is image-based and not yet able toregister volume data. The basic framework, how-ever, extends naturally to higher-dimensions andwe are currently working on a 3-D implementa-tion.

In general, we believe that differential estima-tion techniques can be quite elegant and power-ful. While we believe that the application to im-age registration is promising, it is unlikely that

this approach will supplant the multitude of ex-isting registration techniques. We expect, rather,that this approach will provide motivation for fur-ther investigation into differential methods as wellas their incorporation into other registration algo-rithms.

Acknowledgments

This work has been supported by NSF grants BCS-9978116 (SP), EIA-98-02068 (SP,HF), and NSF CA-REER Award IIS-99-83806 (HF).

References

[1] E.H. Adelson and J.R. Bergen. Spatiotempo-ral energy models for the perception of mo-tion. Journal of the Optical Society of America,2(2):284–299, 1985.

[2] J. K. Aggarwal and N. Nandhakumar. Onthe computation of motion from sequencesof images - a review. Proceedings of the IEEE,76(8):917–935, 1988.

[3] P. Anandan. A computational frameworkand an algorithm for the measurement of vi-sual motion. International Journal of ComputerVision, 2(3):283–310, 1989.

[4] D. Barber, W. Tindale, E. Hunt, A. Mayes,and H. Sagar. Automatic registration ofspect images as an alternative to immobi-lization in neuroactivation studies. Physicsin Medicine and Biology, 40:449–463, 1995.

[5] J.L. Barron, David J. Fleet, and S. S. Beau-chemin. Performance of optical flow tech-niques. International Journal of Computer Vi-sion, 12(1):43–77, 1994.

[6] J.R. Bergen, P. Anandan, K.J. Hanna, andR. Hingorani. Hierarchical model-based mo-tion estimation. volume 588, pages 237–252,1992.

[7] M. J. Black and P. Anandan. The robust esti-mation of multiple motions: Parametric and

6

piecewise-smooth flow fields. volume 63,pages 75–104, Jan. 1996.

[8] F. L. Bookstein. Principal Warps: Thin-platesplines and the decomposition of deforma-tions. IEEE Trans. Patt. Anal. Machine Intell.,11:567–585, 1989.

[9] L. Brown. A survey of image registra-tion techniques. ACM Computing Surveys,24(4):325–376, 1992.

[10] P.A. Van den Elsen, E.J.D. Pol, and M.A.Viergever. Medical image matching - a re-view with classification. IEEE Engineering inMedicine and Biology, 12(1):26–39, 1993.

[11] H. Farid and E.P. Simoncelli. Optimallyrotation-equivariant directional derivativekernels. In International Conference on Com-puter Analysis of Images and Patterns, pages207–214, Kiel, Germany, Sept. 1997.

[12] D.J. Fleet and A.D. Jepson. Computation ofcomponent image velocity from local phaseinformation. International Journal of ComputerVision, 5:77–104, 1990.

[13] D.J. Heeger. Optical flow using spatio-temporal filters. International Journal of Com-puter Vision, 1(4):279–302, 1988.

[14] B.K.P. Horn. Robot Vision. MIT Press, Cam-bridge, MA, 1986.

[15] B.K.P. Horn and B. G. Schunck. Determin-ing optical flow. Artificial Intelligence, 17(1-3):185–203, 1981.

[16] K.J. Friston, J. Ashburner, J.B. Poline, C.D.Frith, J.D. Heather, and R.S.J. Frackowiak.Spatial registration and normalization of im-ages. Human Brain Mapping, 2:165–189, 1995.

[17] H. Lester and S. Arridge. A survey of hier-archical non-linear medical image registra-tion. Pattern Recognition, Special issue on Im-age Registration, 32(1):129–149, 1999.

[18] B.D. Lucas and T. Kanade. An iterative im-age registration technique with an applica-tion to stereo vision. In Proceedings of the 7th

International Joint Conference on Artificial In-telligence, pages 674–679, Vancouver, 1981.

[19] J.B.A. Maintz and M.A. Viergever. A surveyof medical image registration. Medical ImageAnalysis, 2(1):1–36, 1998.

[20] J. Meunier, A. Guimond, C. Janicki, B. Im-bert, and J. Soucy. Computer Assisted Radiol-ogy, Vol. 1124 of Excerpta Medica - InternationalCongress Series, chapter Automatic 3D regis-tration of brain SPECT images, pages 187–192. Elsevier, Amsterdam, 1996.

[21] O. Nestares and D.J. Heeger. Robust mul-tiresolution alignment of mri brain volumes.Magnetic Resonance in Medicine, 43:705–715,2000.

[22] S. Negahdaripour and C-H Yu. A general-ized brightness change model for comput-ing optical flow. In Proceedings of the FourthICCV, volume 2, pages 2–11, Berlin, 1993.

[23] J. Shi and C. Tomasi. Good features totrack. In IEEE Computer Society Conference onComputer Vision and Pattern Recognition, June1994.

[24] E.P. Simoncelli. Handbook of Computer Visionand Applications, chapter Bayesian Multi-scale Differential Optical Flow, pages 397–420. 1999.

[25] A.B. Watson and A.J. Ahumada. Motion: Per-ception and representation, chapter A look atmotion in the frequency domain, pages 1–10. 1983.

[26] J. West, J. Fitzpatrick, M. Wang, B. Dawant,C. Maurer, R. Kessler, and R. Maciunas.Comparison and evaluation of retrospectiveintermodality image registration techniques.In Image Processing, volume 2710, pages 332–347, Newport Beach, Apr. 1996.

7

(a) source (b) target (c) warped source

(d) applied contrast (e) applied brightness (f) applied warp

(g) estimated contrast (h) estimated brightness (i) rectified warp

Figure 2: Complete results from a synthetic warp. Shown along the top are the source, target andestimated warped source. Shown in panels (d), (e) and (g), (h) are the applied and estimated con-trast/brightness maps. Shown in panel (f) is the applied warp. Shown in panel (i) is the inverse of theestimated warp applied to panel (f) - if the estimate was perfect, this result should be a rectilinear grid, asis nearly the case.

8

source target warped source rectified warp

Figure 3: Results from synthetic warps. Shown in each row is the source, target and estimated warpedsource. If the estimated warp was perfect, the last column should appear as a rectilinear grid, see alsoFigure 2.

9

source target warped source estimated warp

Figure 4: Results from clinical images with unknown registration. Shown in each row is the source, targetand estimated warped source. Shown in the last column is the estimated warp field.

10