Image Registration for Multi-exposure High Dynamic RangeImage Acquisition

Anna TomaszewskaSzczecin University of Technology

atomaszewska@wi.ps.pl

Radoslaw MantiukSzczecin University of Technology

rmantiuk@wi.ps.pl

ABSTRACT

We present a fully automatic method for eliminating misalignments between a sequence of hand-held photographstaken at different exposures. The key component of the technique is the SIFT method that is employed to searchfor key-points (or feature-points) in consecutive images.The key-points are used to find matrices, that transforma set of images to a single coordinate system and eliminate any global misalignments (including general planarhomography). We employ this technique to capture high dynamic range images from a set of photographs taken atdifferent exposures, where misalignments can cause blurring and artifacts, and prevent achieving high quality HDRimages. The proposed alignment technique works well for over- and under-exposed images and is not sensitiveto an image content. We present our implementation of the technique and the results of tests made for variety ofphotographs.

Keywords: image alignment, image registration, contrast domain, HDRcapture, High Dynamic Range Imaging.

1 INTRODUCTION

There is tremendous progress in the development andaccessibility of high dynamic range (HDR) imagingtechnology [RWPD05]. Modern image processingand graphics software becomes HDR enabled. AlsoHDR digital photography replaces low dynamic range(LDR) technologies. HDR photographs are of muchbetter quality and easier to be processed in a digitaldarkroom. Unfortunately, HDR cameras are still veryexpensive and not available for average users. On theother hand, taking HDR photographs seems to be legit-imate and crucial. In the near future LDR images maybecome almost obsolete due to the progress in LCDtechnology [SHS+04] and it will not be easy to displayLDR image correctly. LDR photographs will look paleand not interesting on HDR LCD displays.

The multi-exposure HDR capture technique [MP95]seems to be a good alternative to HDR cameras, whichcan be used to create an HDR image from photographstaken with a conventional LDR camera. The tech-nique uses differently exposed photographs to recoverthe response function of a camera. From the responsefunction, the algorithm creates an HDR image whose

Permission to make digital or hard copies of all or part ofthis work for personal or classroom use is granted withoutfee provided that copies are not made or distributed forprofit or commercial advantage and that copies bear thisnotice and the full citation on the first page. To copyotherwise, or republish, to post on servers or to redistributeto lists, requires prior specific permission and/or a fee.

Copyright UNION Agency – Science Press, Plzen, CzechRepublic

pixel values are proportional to the true radiance (orluminance) value of a scene. Because this techniquerequires multiple input photographs, there is a highlikelihood of misalignments between pixels in the se-quence of exposures due to moving of a hand-heldcamera (global motion) or dynamic object in a scene(local motion causing ghosting). It is crucial that mis-alignments between input photographs should be re-moved before fusing an HDR image.

Our work addresses the problem of global alignment(registration) of hand-held photographs taken with dif-ferent exposures. The method compensates misalign-ments caused by any movement of a camera. In partic-ular, general planar homography (or linear planar pro-jective transform) is considered so any linear transfor-mation of a point position between planes on imagesis recovered (see [HZ06] for details on the propertiesand limitations of planar homography). The key com-ponent of the technique is the scale-invariant featuretransform (SIFT) [Low04] that is employed to searchfor key-points (often called feature-points) in consecu-tive images. We modified the algorithm by automatiz-ing contrast threshold value calculation. The key-pointcorrespondences are used to find matrices, that map aset of images to a single coordinate system. The bilin-ear interpolation of the output images based on the cal-culated matrices completes the algorithm. The regis-tered photographs are fused using our implementationof multi-exposure HDR capture technique proposed byMitsunaga and Nayar [MN99]. Our alignment tech-nique is fully automatic and is not sensitive to imagecontent. It works correctly even for highly under- andover-exposed images. We modified registration algo-rithm so it can be applied not only for standard LDR

images but also for HDR images. We can use extendeddynamic range photographs stored in RAW format asan input to alignment algorithm. This way, less inputphotographs can be used to create high dynamic rangeimage. Moreover, we noticed that the accuracy of im-age registration is much better for input RAWs than forstandard LDR photographs.

In Section 2 we present a review of previous workrelated to image alignment for multi-exposure HDRcapture techniques. A detailed description of our SIFTbased HDR alignment technique is presented in Sec-tion 3. In Section 4 we show implementation of themethod followed by the results of the tests run onhand-captured photographs (Section 5). Section 6presents conclusions and discusses future work direc-tions.

2 RELATED WORKIn recent years significant progress has been made inthe development of algorithms that allow to captureHDR images using low dynamic range sensors (stan-dard LDR cameras) [MP95, DM97, MN99, RBS99].These algorithms retrieve high dynamic range infor-mation from a sequence of photographs. The authorssuggest using tripod to avoid camera movement andthey do not address the problem of eliminating mis-alignments.

The problem of image alignment and matching wasintensively studied during last years [ZF03, Bro92] butnot for registration of images of different exposures.The only solution that addressed exactly the prob-lem of capture of HDR photographs, was proposedin [War03, RWPD05]. The technique employs con-version of input photographs into percentile thresholdbitmaps. The bitmaps are analyzed and then alignedhorizontally and vertically using shift and differenceoperations over each image. In chapter 5 we comparethe results achieved with this algorithm to our method.

Kang et al. [KUWS03] described a technique forcreating high dynamic range video from a sequenceof altering light and dark exposures. A part of thetechnique is a HDR stitching process, which includesglobal and local alignment to compensate for pixel mo-tion. The stitching process can be also used to compen-sate for camera movement when creating an HDR stillfrom a series of bracketed still photographs. However,the presented technique seems to be suitable for videowhere there are no large differences between consecu-tive frames.

In [ST04] Sand and Teller present a global and lo-cal matching algorithm, which is robust to changes inexposure of photographs. The key idea behind thistechnique is to identify which parts of the image canbe matched without being confused by parts that aredifficult to match. Such assumption seems to be notvalid for images with large differences in exposures,

where there is usually not enough information for cor-rect matching. The technique was designed for match-ing two video sequences and was not tested on stillphotographs.

Recently, Cerman and Hlavac [CH06] presented analignment method based on unconstrained nonlinearoptimization. In this method, each image is linearizedusing the estimated camera response function andmultiplied by the exposure ratio. Then, a normalizeddifference summed across all corresponding pixelsis used to estimate misalignments. The method cancompensate global rotation and horizontal and verticalshifts.

There are a few techniques which computecamera response function based on misalignedphotographs [GN03, KP04]. However, thesemethods are not meant to create HDR images.The problem of removing ghosting artifacts in amulti-exposure sequence of photographs was alsoinvestigated [EAKR06, RWPD05] but proposed algo-rithms do not take into consideration a compensationof camera movements.

In the next section, we present our technique of re-moving global misalignments between photographs ina multi-exposure sequence. This technique can be ap-plied to still images and it compensates any cameramovement.

3 MISALIGNMENT COMPENSATIONOur modification of multi-exposure HDR capturemethod is presented in Figure 1. We align all inputphotographs to a selected reference (the image withthe best exposure) before fusing an HDR image ratherthan use a tripod as suggested in the previous methods.

The algorithm of our alignment technique is shownin Figure 2. The technique establishes correspon-dences between points, lines or other geometrical enti-ties in a set of images. We use a modified SIFT (ScaleInvariant Feature Transform) [Low04] algorithm toextract local features called key-points. They are lo-cated at scale-space maxima/minima of a difference ofGaussian function. The key-points descriptors accu-mulated in orientation histograms form the invariantdescriptors.

In the next step, these descriptors are used to findcorrespondences between key-points in the referenceimage and remaining images. The number of the cor-respondences is reduced by RANSAC [FB81] algo-rithm because only four pairs of key-points (8 coef-ficients) are needed to calculate transformation ma-trix. The RANSAC selects a set of key-points cor-respondences that are compatible with a homographybetween the images. It uses the singular value compo-sition (SVD) method to solve over-determined systemof linear equations required by the optimization algo-rithm.

Figure 1: A technique of capturing HDRphotographs with a hand-held camera based on

the image alignment and image fusing techniques.

The transformation matrix is computed by the directlinear transform (DLT) algorithm. This matrix is usedto transform a given image to the coordinates of thereference image:

x′ ∼ H · x, (1)

where x′ and x are points from destination andsource images respectively, andH is a transformationmatrix:

H =

h11 h12 h13

h21 h22 h23

h31 h32 h33

,

hi j are unknowns.

The algorithm computes the color values of consec-utive output pixels using bilinear interpolation of ap-propriate input pixels. This solution ensures that thereare no holes in an output image. Moreover a sub-pixelprecision of calculation improves the quality of images(decreases aliasing).

We noticed that the accuracy of the transforma-tion matrix can be decreased due to doubtful corre-spondences. The RANSAC algorithm chooses thebest pairs of key-points based on a random technique,which can fail. We improve the accuracy of the cor-respondences selection by choosing only these key-points which occur in all photographs simultaneously.

The key-points searching algorithm works in con-trast domain so it is intensity invariant. Therefore, wecan use it to find the similarities between images ofdifferent exposures, where intensity values can changesignificantly. The algorithm uses a multi-scale pyra-mid of Difference-of-Gaussian images to improve thekey-points detection (a larger neighborhood of a pixelis considered).

As has been mentioned above the key-searching al-gorithm operates also with RAW format as an inputdata. However, in order to improve the accuracy ofimage registration for such pictures we modified it.Instead of using constant contrast threshold value wecalculate if for each image separate in relation its dy-namics. This modification makes it possible to detectfar more key-points in RAW images in comparison toLDR ones. It enhances accuracy for pictures taken inbad light conditions. The results are presented in sec-tion 5.

4 IMPLEMENTATIONThe algorithm for multi-exposure image alignmenthas been implemented as a part ofpfstools 1 pack-age [MKM07]. The main components ofpfstoolsare programs for reading and writing images inall major HDR and LDR formats (e.g. OpenEXR,Radiance’s RGBE, etc.) and programs for basic imagemanipulation (rotation, scaling, cropping, etc.). Thetypical usage ofpfstools involves executing severalprograms joined by UNIX pipes. The first programtransmits the current frame or image to the next one inthe chain. The final program should either display animage or write it to a disk. An example of commandline is given below:

1 pfstools is distributed as Open Source under the GPL licenseand the project web page can be found athttp://pfstools.sourceforge.net/

Figure 2: The algorithm of the image alignment technique.

pfsin input1.jpg input2.jpg | pfsalign /

| pfshdrcalibrate | pfsout output.exr

Read the images input1.jpg and input2.jpg,align the images (pfsalign), create HDR image(pfshdrcalibrate) and write the output tooutput.exr.

Thepfsalign program (see the example above) is re-sponsible for recovering misalignments and transform-ing input images to a single coordinate space. Thisprogram was implemented in C++ and, like thepfs-tools package, can be run under Mac OS, Linux andWindows (with cygwin support).

The pfsalign consists of two main modules. Thefirst one looks for key-points and correspondences be-tween them, and then calculates transformation matrix.We used SVD solver implementation from numericalrecipes book [PTVF92] to aid the calculation of thetransformation matrix. The second module transforms

images to single coordinate space with sub-pixel pre-cision so that there are no holes in resulting images.

The pfshdrcalibrate program (see the exampleabove) fuses images of different exposures andcalculates a HDR radiance map. We implementedthe fusing algorithm proposed by Mitsunaga andNayar [MN99] which is based on polynomial approx-imation of the camera response function. We findthis algorithm very suitable, especially in case of theinput sequence of a limited number of photographs(e.g. 2-3). The polynomial function fills gaps in aresponse function so that it is possible to calculatecorrect radiance value even for pixels without goodrepresentation in the input photographs (e.g. pixelswith maximum value (255) in all photographs).

5 RESULTSTo test the usability of our multi-exposure image align-ment method, we have executed an extensive set oftests on photograph sequences. The main goal of the

tests was to check the accuracy of alignment for im-ages of short and long exposures (very light and verydark) and of different image content. All photographswere taken with hand-held digital camera (Canon 10Dwith Sigma 18-125 mm F3.5-5.6 DC lens) and en-compassed any possible combination of camera move-ments (shifts, rotations and zooming). To evaluatethe accuracy of the image alignment, we fuse a set ofaligned images into a HDR image and subjectively es-timate the quality of that image.

In the top row of Figure 3 a set of misaligned pho-tographs of the same scene is depicted (the set consistsof three photographs of resolution 1536x1024 pixelsstored in 8-bits JPEG format). The photographs werealigned using our algorithm (the second row) and thena HDR image was created (bottom row). The pho-tographs were aligned with sufficient accuracy so thatthere are no visible artifacts in the final HDR image.There are also no visible holes in the aligned images(from the middle row), as they were filled with bi-linearly interpolated pixel values. The algorithm re-quires about 450 seconds to align three presented pho-tographs on the CPU. The time of registration dependson an image content. For comparison, we also showa false HDR (the third row - left) which was fussedbased on misaligned images from the top row. In thethird row (right), the HDR image created byhdrgen(available athttp://www.anyhere.com/ ) programis presented. It can be seen in this figure that simpletransformations implemented inhdrgen are not enoughfor accurate image registration.

In Figure 4 we show a few additional results of im-age alignment. In these examples the capabilities ofremoving shifts, rotations, zooms and a combinationof these transformations are presented. The algorithmcorrectly recognizes even large movements of a cam-era.

Figure 5 presents another example of input image setwith very light (over-exposed) and very dark (under-exposed) photographs. One can see that despite a largedifference in exposure, sufficient number of key-pointswas found to establish a correspondence between im-ages and calculate transformation matrices. Moreover,there are no key-points in unsteady too dark or too lightareas.

Figure 6 depicts the arrangement and number ofkey-points detected for LDR and RAW images respec-tively. The pictures were taken in the same light con-ditions. In RAW images, in contrast to LDR pictures,features were detected also in poorly lit fragments ofimages. Due to that we have a more regular arrange-ment of key-points in a picture compared with LDRimages, and what follows we can calculate a more ac-curate transformation matrix.

We successfully tested our alignment technique formany photographs of different content. The technique

requires only four pairs of corresponding key-points tocalculate transformation matrix so it will give correctresults also for images with large non-textured areasand parallax effects.

The algorithm fails for images that contain many re-current textures occupying a large portion of an image.In such cases, many key-points have the same descrip-tors and RANSAC (more precisely SVD) cannot com-pute good (it means falling below an error threshold)key-point correspondences. We find this problem dif-ficult to solve based on the feature based alignmentmethod (our algorithm belongs to this group). For-tunately, such kinds of photographs are rather rare inpractice.

6 CONCLUSION AND FUTUREWORK

The major outcome of this work is an image regis-tration method that can be used to remove misalign-ments between images in a sequence of differentlyexposed photographs (in RAW and/or JPEG format).The method is fully automatic and compensates mis-alignments caused by any movement of a camera. Itworks in contrast domain so is less sensitive to imagecontent and changes in exposure. The method employsSWIFT algorithm to find key-points, RANSAC algo-rithm to choose the best key-point correspondencesand DLT technique to calculate transformation ma-trix. The input photographs are transformed to a sin-gle coordinate system with sub-pixel precision to avoidholes and aliasing artifacts. The results of this workfacilitate capturing HDR images based on the multi-exposure techniques. Our method simplifies acquisi-tion of HDR images by removing the main disadvan-tage of the multi exposure techniques, which is a ne-cessity of using a tripod.

We implemented and tested our method for a vari-ety of photographs. We find it effective in most casesbut several difficult images cause the method to fail.The solution of this problem is further optimization ofSIFT and RANSAC algorithms for adjusting them tohigh dynamic range input images. We also plan to es-timate the transformation matrix not only for planarhomography but also for more general 3D case withdepth reconstruction. Since our current implementa-tion does not deal with local changes, we intend toimplement a de-ghosting technique in the future. Weplan to speed-up the implementation based on the GPUprogramming to facilitate the practical usage of ourmethod.

ACKNOWLEDGMENTS

We would like to thank Piotr Ostiak for basic imple-mentation of the image matching algorithm. Specialthanks go to Rafal Mantiuk and Grzegorz Krawczyk

Figure 3: The HDR image captured with a hand-held camera. Toprow: misaligned photographs. Thesecond row from top: aligned photographs, the first photograph is a reference image. The third row: HDRimage created by fusing the misaligned photographs (left) and by hdrgen software (right) (the inset depicts

misalignment artifacts). Bottom row: HDR image created from aligned photographs. All HDR imageswere tone mapped using contrast domain operator. [MMS05])

Figure 4: The results of image alignment. Left column: reference photographs. Middle column:photographs before alignment. Right column: photographs after alignment (transformed to the coordinate

system of the reference photograph). From the top to the bottom row, the examples show shift, rotation,zoom, and complex camera movement (general homography).

for their helpful comments concerning this work andfor delivering thepfstools software.

REFERENCES

[Bro92] Lisa Gottesfeld Brown. A survey of imageregistration techniques.ACM Comput. Surv.,24(4):325–376, 1992.

[CH06] Lukas Cerman and Vaclav Hlavac. Exposuretime estimation for high dynamic range imag-ing with hand held camera. InProc. of Com-puter Vision Winter Workshop, Czech Republic,February 2006.

[DM97] P.E. Debevec and J. Malik. Recovering high dy-namic range radiance maps from photographs.In Proceedings of ACM SIGGRAPH 1997,pages 369–378, 1997.

[EAKR06] Ahmet Oguz Akyuz Erum Arif Khan and ErikReinhard. Ghost removal in high dynamic rangeimages. InProc. of IEEE International Confer-ence on Image Processing, October 2006.

[FB81] M. A. Fischler and R. C. Bolles. Randomsample consensus: A paradigm for model fit-ting with applications to image analysis and au-tomated cartography.Communications of theACM, 24(6):381Ð395, 1981.

[GN03] M.D. Grossberg and S.K. Nayar. Determin-ing the camera response from images: What isknowable? PAMI, 25(11):1455–1467, Novem-ber 2003.

[HZ06] Richard Hartley and Andrew Zisserman.Mul-tiple View Geometry in Computer Vision. Cam-bridge University Press, 2006.

[KP04] S.J. Kim and M. Pollefeys. Radiometric self-alignment of image sequences. InProc. of IEEEConf. on Computer Vision and Pattern Recogni-tion, 2004.

[KUWS03] Sing Bing Kang, Matthew Uyttendaele, SimonWinder, and Richard Szeliski. High dynamicrange video. ACM Trans. Graph., 22(3):319–325, 2003.

[Low04] David G. Lowe. Distinctive image features fromscale-invariant keypoints.Int. J. Comput. Vi-sion, 60(2):91–110, 2004.

[MKM07] Rafal Mantiuk, Grzegorz Krawczyk, and Ra-doslaw Mantiuk. High dynamic range imag-ing pipeline: Perception-motivated representa-tion of visual content. InTo be published inProc. of Human Vision and Electronic ImagingXII, IS&T/SPIE’s Annual Symposium on Elec-tronic Imaging (2007), SPIE Proceedings Se-ries, San Jose, California USA, January 2007.

Figure 5: Example photographs taken with a very short and very long exposures. Valid key-points aremarked as white dots. There are no key-points in too dark and too light areas.

Figure 6: An arrangement and number ofkey-points detected for LDR (top) and RAW

(bottom) images taken in the same light conditions.

SPIE.

[MMS05] Rafał Mantiuk, Karol Myszkowski, and Hans-Peter Seidel. A perceptual framework for con-trast processing of high dynamic range images.In APGV ’05: 2nd Symposium on Appied Per-ception in Graphics and Visualization, pages87–94, 2005.

[MN99] T. Mitsunaga and S. K. Nayar. Radiometric selfcalibration. pages 374–380, 1999.

[MP95] S. Mann and R. Picard. Being ‘Undigital’ withDigital Cameras: Extending Dynamic Range byCombining Differently Exposed Pictures.Proc.of IST’s 48th Annual Conference, pages 442–448, May 1995.

[PTVF92] William Press, Saul Teukolsky, William Vetter-ling, and Brian Flannery.Numerical Recipesin C. Cambridge University Press, Cambridge,UK, 2nd edition, 1992.

[RBS99] M. Robertson, S. Borman, and R. Stevenson.Dynamic range improvement through multipleexposures. InProceedings of the 1999 Interna-tional Conference on Image Processing (ICIP-99), pages 159–163, Los Alamitos, CA, Octo-ber 24–28 1999.

[RWPD05] Erik Reinhard, Greg Ward, Sumanta Pattanaik,and Paul Debevec.High Dynamic Range Imag-ing. Acquisition, Manipulation, and Display.Morgan Kaufmann, 2005.

[SHS+04] H. Seetzen, W. Heidrich, W. Stuerzlinger,G. Ward, L. Whitehead, M. Trentacoste,A. Ghosh, and A. Vorozcovs. High dynamicrange display systems.ACM Trans. on Graph.,23(3):757–765, 2004.

[ST04] Peter Sand and Seth Teller. Video matching.ACM Trans. Graph., 23(3):592–599, 2004.

[War03] Greg Ward. Fast, robust image registration forcompositing high dynamic range photographsfrom hand-held exposures.journal of graphicstools, 8(2):17–30, 2003.

[ZF03] Barbara Zitova and Jan Flusser. Image regis-tration methods: a survey.Image and VisionComputing, 21(11):977–1000, October 2003.