Journal of Visual Communication andImage Representation
journal homepage: www.elsevier.com/locate/jvci
Efficient feature-based image registration by mapping sparsified surfaces☆
Chun Pang Yunga,⁎, Gary P.T. Choib, Ke Chenc, Lok Ming Luia,⁎
a Department of Mathematics, The Chinese University of Hong Kong, Hong Kongb John A. Paulson School of Engineering and Applied Sciences, Harvard University, USAc Department of Mathematical Sciences, The University of Liverpool, United Kingdom
With the advancement in the digital camera technology, the use of high resolution images and videos has beenwidespread in the modern society. In particular, image and video frame registration is frequently applied incomputer graphics and film production. However, conventional registration approaches usually require longcomputational time for high resolution images and video frames. This hinders the application of the registrationapproaches in the modern industries. In this work, we first propose a new image representation method toaccelerate the registration process by triangulating the images effectively. For each high resolution image orvideo frame, we compute an optimal coarse triangulation which captures the important features of the image.Then, we apply a surface registration algorithm to obtain a registration map which is used to compute theregistration of the high resolution image. Experimental results suggest that our overall algorithm is efficient andcapable to achieve a high compression rate while the accuracy of the registration is well retained when com-pared with the conventional grid-based approach. Also, the computational time of the registration is significantlyreduced using our triangulation-based approach.
1. Introduction
In recent decades, the rapid development of the digital camerahardware has revolutionized human lives. On one hand, even mid-levelmobile devices can easily produce high resolution images and videos.Besides the physical elements, the widespread use of the images andvideos also reflects the importance of developing software technologyfor them. On the other hand, numerous registration techniques forimages and video frames have been developed for a long time. Theexisting registration techniques work well on problems with a moderatesize. However, when it comes to the current high quality images andvideos, most of the current registration techniques suffer from ex-tremely long computations. This limitation in software seriously im-pedes fully utilizing the state-of-the-art camera hardware.
One possible way to accelerate the computation of the registration isto introduce a much coarser grid on the images or video frames. Then,the registration can be done on the coarse grid instead of the high re-solution images or video frames. Finally, the fine details can be addedback to the coarse registration. It is noteworthy that the quality of thecoarse grid strongly affects the quality of the final registration result. Ifthe coarse grid cannot capture the important features of the images orvideo frames, the final registration result is likely to be unsatisfactory.
In particular, for the conventional rectangular coarse grids, since thepartitions are restricted in the vertical and horizontal directions, im-portant features such as slant edges and irregular shapes cannot beeffectively recorded. By contrast, triangulations allow more freedom inthe partition directions as well as the partition sizes. Therefore, it ismore desirable to make use of triangulations in simplifying the regis-tration problems.
In this work, we propose a two-stage algorithm for effective regis-tration of specially large images. In stage 1, a content-aware imagerepresentation algorithm to TRiangulate IMages, abbreviated as TRIM,is developed to simplify high quality images and video frames.Specifically, for each high quality image or video frame, we compute acoarse triangulation representation of it. The aim is to create a highquality triangulation on the set of the content-aware sample pointsusing the Delaunay triangulation. The computation involves a series ofsteps including subsampling, unsharp masking, segmentation andsparse feature extraction for locating sample points on important fea-tures. Then in stage 2, using coarse triangular representation of theimages, the registration is computed by a landmark-based quasi-con-formal registration algorithm [17] for computing the coarse registra-tion. The fine detail of the image or video frame in high resolution iscomputed with the aid of a mapping interpolation. Our proposed
https://doi.org/10.1016/j.jvcir.2018.07.005Received 27 July 2017; Received in revised form 6 January 2018; Accepted 6 July 2018
☆ This paper has been recommended for acceptance by Junzhou Huang.⁎ Corresponding authors.E-mail address: [email protected] (L.M. Lui).
Journal of Visual Communication and Image Representation 55 (2018) 561–571
framework may be either used as a standalone fast registration algo-rithm or also served as a highly efficient and accurate initialization forother registration approaches.
The rest of this paper is organized as follows. In Section 2, we re-view the literature on image and triangular mesh registration. Ourproposed method is explained in details in Section 3. In Section 4, wedemonstrate the effectiveness of our approach with numerous realimages. The paper is concluded in Section 5.
2. Previous works
In this section, we describe the previous works closely related to ourwork.
Image registration have been widely studied by different researchgroups. Surveys on the existing image registration approaches can befound in [39,6,16,38]. In particular, one common approach for guar-anteeing the accuracy of the registration is to make use of landmarkconstraints. Bookstein [1–3] proposed the unidirectional landmarkthin-plate spline (UL-TPS) image registration. In [13], Johnson andChristensen presented a landmark-based consistent thin-plate spline(CL-TPS) image registration algorithm. In [14], Joshi et al. proposed theLarge Deformation Diffeomorphic Metric Mapping (LDDMM) for re-gistering images with a large deformation. In [10,11], Glaunès et al.computed large deformation diffeomorphisms of images with pre-scribed displacements of landmarks.
A few works on image triangulations have been reported. In [8],Gee et al. introduced a probabilistic approach to the brain imagematching problem and described the finite element implementation. In[15], Kaufmann et al. introduced a framework for image warping usingthe finite element method. The triangulations are created using theDelaunay triangulation method [31] on a point set distributed ac-cording to variance in saliency. In [18,19], Lehner et al. proposed adata-dependent triangulation scheme for image and video compression.Recently, Yun [35] designed a triangulation image generator calledDMesh based on the Delaunay triangulation method [31].
In our work, we handle image registration problems with the aid oftriangulations. Numerous algorithms have been proposed for the re-gistration of triangular meshes. In particular, the landmark-driven ap-proaches use prescribed landmark constraints to ensure the accuracy ofmesh registration. In [34,22,33], Wang et al. proposed a combinedenergy for computing a landmark constrained optimized conformalmapping of triangular meshes. In [23], Lui et al. used vector fields torepresent surface maps and computed landmark-based close-to-con-formal mappings. Shi et al. [32] proposed a hyperbolic harmonic re-gistration algorithm with curvature-based landmark matching on tri-angular meshes of brains. In recent years, quasi-conformal mappingshave been widely used for feature-endowed registration [36,37,24,26].Choi et al. [5] proposed the FLASH algorithm for landmark alignedharmonic mappings by improving the algorithm in [34,22] with the aidof quasi-conformal theories. In [17], Lam and Lui reported the quasi-conformal landmark registration (QCLR) algorithm for triangular me-shes.
Contributions. Our proposed approach for fast registration of highresolution images or video frames is advantageous in the followingaspects:
(1) The triangulation algorithm is fully automatic. The importantfeatures of the input image are well recorded in the resulting coarsetriangulation.(2) The algorithm is fast and robust. The coarse triangulation of atypical high resolution image can be computed within seconds.(3) The registration algorithm for the triangulated surfaces by aBeltrami framework incorporates both the edge and landmark con-straints to deliver a better quality map as fine details are restored.By contrast, for regular grid-based approaches, the same landmarkcorrespondences can only be achieved on the high resolution image
representation.(4) Using our approach, the problem scale of the image and videoframe registration is significantly reduced. Our method can alter-natively serve as a fast and accurate initialization for the state-of-the-art image registration algorithms.
3. Proposed method
In this section, we describe our proposed approach for efficientimage registration in details.
3.1. Stage 1 – Construction of coarse triangulation on images
Given two high resolution images I1 and I2, our goal is to compute afast and accurate mapping →f I I: 1 2. Note that directly working on thehigh resolution images can be inefficient. To accelerate the computa-tion, the first step is to construct a coarse triangular representation ofthe image I1. In the following, we propose an efficient image triangu-lation scheme called TRIM. The pipeline of our proposed framework isdescribed in Fig. 1.
Our triangulation scheme is content-aware. Specifically, specialobjects and edges in the images are effectively captured by a segmen-tation step, and a suitable coarse triangulation is constructed with thepreservation of these features. Our proposed TRIM method consists of 6steps in total.
3.1.1. Subsampling the input image without affecting the triangulationquality
Denote the input image by I. To save the computational time fortriangulating the input image I, one simple remedy is to reduce theproblem size by performing certain subsampling on I. For ordinaryimages, subsampling unavoidably creates adverse effects on the imagequality. Nevertheless, it does not affect the quality of the coarse trian-gulation we aim to construct on images.
In our triangulation scheme, we construct triangulations based onthe straight edges and special features on the images. Note that straightedges are preserved in all subsamplings of the images because of thelinearity. More specifically, if we do subsampling on a straight line, thesubsampled points remain to be collinear. Hence, our edge-based tri-angulation is not affected by the mentioned adverse effects. In otherwords, we can subsample high resolution images to a suitable size forenhancing the efficiency of the remaining steps for the construction ofthe triangulations. We denote the subsampled image by ∼I . In practice,for images larger than ×1000 1000, we subsample the image so that it issmaller than ×1000 1000.
3.1.2. Performing unsharp masking on the subsampled imageAfter obtaining the subsampled image ∼I , we perform an unsharp
masking on ∼I in order to preserve the edge information in the finaltriangulation. More specifically, we first transform the data format ofthe subsampled image ∼I to the CIELAB standard. Then, we apply theunsharp masking method in [27] on the intensity channel of the CIELABrepresentation of ∼I . The unsharp masking procedure is briefly de-scribed as follows.
By an abuse of notation, we denote ∼I x y( , ) and I x y( , ) the
Fig. 1. The pipeline of our proposed TRIM algorithm for accelerating imageregistration via coarse triangulation.
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
562
intensities of the input subsampled image ∼I and the output image Irespectively, andG x y( , )σ the Gaussian mean of the intensity of the pixelx y( , ) with standard derivation . Specifically, G x y( , )σ is given by
∫≜∈
− − + −G x y
σ πe( , ) 1
2.σ u v
u x v yσ
( , ) Ω
( ) ( )2
2 22
(1)
We perform an unsharp masking on the image using the followingformula
= ∼ − ⎧⎨⎩
∗∼ ><
I x y I x y λ G I x y V x y θV x y θ
( , ) ( , ) ( , ) if ( , ) ,0 if ( , ) ,
σ s
s (2)
where
∫≜ ∼ −∼∈
V x yArea M
I u v I x y( , ) 1( )
( ( , ) ( , ))ss u v M mean( , )
2s (3)
and
∫∼ = ∼∈
I x yArea M
I u v( , ) 1( )
( , ).means u v M( , ) s (4)
Here, ∗ is the convolution operator and Ms is the disk with centerx y( , ) and radius s. The effect of the unsharp masking is demonstrated inFig. 2. With the aid of this step, we can highlight the edge informationin the resulting image I for the construction of the triangulation in thelater steps. For simplicity we set =s σ . In our experiment, we choose
= = =λ σ s0.5, 2, 2, and =θ 0.5. An analysis on the choice of theparameters is provided in Section 4.
3.1.3. Segmenting the imageAfter obtaining the image I upon unsharp masking, we perform a
segmentation in this step in order to optimally locate the mesh verticesfor computing the coarse triangulation. Mathematically, our segmen-tation problem is described as follows.
Suppose the image I has L intensity levels in each RGB channel.Denote i as a specific intensity level ⩽ ⩽ −i e i L( . . 0 1). Let C be a colorchannel of the image (i.e. ∈C R G B{ , , }), and let hi
C denote the imagehistogram for channel C, in other words, the number of pixels whichcorrespond to its i-th intensity level.
Define ≜piC h
NiC, where N represents the total number of pixels in
the image I . Then we have
∑ ∑=
∈
= ==
∈i
C R G B
p μi
C R G B
ip0,
{ , , }
1 and0,
{ , , }
.L
iC
TC
L
iC
(5)
Suppose that we want to compress the color space of the image I to lintensity levels. Equivalently, I is to be segmented into l classes
…D D, ,ClC
1 by the ordered threshold levels = … −x j l, 1, , 1jC . We define
the best segmentation criterion to be maximizing the inter-class in-tensity-mean variance. More explicitly, we define the cost
∑≜=
∈
−σj
C R G B
w μ μ1,
{ , , }
( ) ,Cl
jC
jC
TC 2
(6)
where the probability wjC of occurrence of a pixel being in the class Dj
C
and the intensity-mean μjC of the class Dj
C are respectively given by
=
⎧
⎨
⎪⎪⎪⎪
⎩
⎪⎪⎪⎪
∑ =∈
=
∑ = +∈
< <
∑ = +∈
=
=
⎧
⎨
⎪⎪⎪⎪⎪
⎩
⎪⎪⎪⎪⎪
∑ =∈
=
∑ = +∈
< <
∑ = +∈
=
−
−
−
−
w
iC R G B
p j
i tC R G B
p j l
i tC R G B
p j l
μ
iC R G B
j
i tC R G B
j l
i tC R G B
j l
0,{ , , }
if 1,
1,{ , , }
if 1 ,
1,{ , , }
if ,
and
0,{ , , }
if 1,
1,{ , , }
if 1 ,
1,{ , , }
if .
jC
tiC
jC
tiC
jC
LiC
jC
t ipw
jC
t ipw
jC
L ipw
1
1
1
1
jC
jC
jC
iC
jC
jC
iC
jC
iC
jC
(7)
Hence, we maximize three objective functions of each RGB channel
< <⋯< <=−
−
σ xargmax ({ } ),x x L
CjC
jl
111
ClC
1 1 (8)
where ∈C R G B{ , , }. Our goal is to find a set of = =−xx { }j
Cjl
11 such that
above function is maximized for each RGB channel.To solve the aforementioned segmentation optimization problem,
we apply the Particle Swarm Optimization (PSO) segmentation algo-rithm [9] on the image I . The PSO method is used in this segmentationoptimization problem for reducing the chance of trapping in local op-timums.
An illustration of the segmentation step is provided in Fig. 3. Afterperforming the segmentation, we extract the boundaries of the seg-ments. Then, we can obtain a number of large patches of area in each ofwhich the intensity information is almost the same. They provide uswith a reasonable edge base for constructing a coarse triangulation inlater steps.
3.1.4. Sparse feature extraction on the segment boundariesAfter computing the segment boundaries B on the image I , we aim
to extract sparse feature points on B in this step. For the final trian-gulation, it is desirable that the edges of the triangles are as close aspossible to the segment boundaries B , so as to preserve the geometricfeatures of the original image I. Also, to improve the efficiency for thecomputations on the triangulation, the triangulation should be muchcoarser than the original image. To achieve the mentioned goals, we
Fig. 2. An illustration of unsharp masking. Left: the input image. Right: theresulting image. The unsharp masking procedure helps preserving the edgeinformation of the input image to ensure that the vertices in unclear edges canalso be extracted.
Fig. 3. An illustration of the segmentation step for compressing the color spaceto achieve a sparse intensity representation. Left: the original image. Right: thesegmentation result.
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
563
consider extracting sparse features on the segment boundaries B anduse them as the vertices of the ultimate triangulated mesh.
Consider a rectangular grid table G on the image I . Apparently, thegrid table G intersects the segment boundariesB at a number of points.DenoteP as our desired set of sparse features. Conceptually,P is madeup of the set of points at which B intersect the grid G, with certainexceptions.
In order to further reduce the number of feature points for a coarsetriangulation, we propose a merging procedure for close points.Specifically, let gi j, be the vertex of the grid G at the i-th row and the j-thcolumn. We denotePi j,
1 andPi j,2 respectively as the set of points at which
B intersect the line segment +g gi j i j, , 1 and the line segment +g gi j i j, 1, . SeeFig. 4 for an illustration of the parameters.
There are 3 possible cases for Pi jk, , where =k 1, 2:
(i) If P =| | 0i jk, , then there is no intersection point between the line
segment and B and hence we can neglect it.(ii) If P =| | 1i j
k, , then there is exactly one intersection point pi j
k, be-
tween the line segment andB . We include this intersection point pi jk,
in our desired set of sparse features P .(iii) If P >| | 1i j
k, , then there are multiple intersection points between
the line segment and B . Since these multiple intersection points lieon the same line segment, it implies that they are sufficiently closeto each other. In other words, the information they contain aboutthe segment boundaries B is highly similar and redundant.Therefore, we consider merging these multiple points as one point.
More explicitly, for the third case, we compute the centre mi jk, of the
points in Pi jk, by
P= ∈m mean p.i jk
p p, { | }i jk, (9)
The merged point mi jk, is then considered as a desired feature point. In
summary, our desired set of sparse features is given by
P = ⋃⋃ p p m m{ , , , }.i j
i j i j i j i j,1
,2
,1
,2
(10)
An illustration of the sparse feature extraction scheme is given inFig. 4. However, one important problem in this scheme is to determinea suitable size of the grid G so that the sparse feature points are opti-mally computed. Note that to preserve the regularity of the extractedsparse features, it is desirable that the elements of the grid G are close toperfect squares. Also, to capture the important features as complete aspossible, the elements of G should be small enough. Mathematically,the problem can be formulated as follows.
Denote w as the width of the image I h, as the height of the image′I w, as the number of columns in ′G h, as the number of rows inG l, w as
the horizontal length of every element of G, and lh as the vertical lengthof every element of G. See Fig. 4 for a geometric illustration of lw and lh.We further denote p as the percentage of grid edges in G which intersectthe segment boundaries B , and n as the desired number of the sparsefeature points. Given the two inputs p and n, to find a suitable grid sizeof G, we aim to minimize the cost function
= −c l l l l( , ) | |w h w h2 (11)
subject to
= ′h h l(i) ,h (12)
= ′w w l(ii) ,w (13)
′ + ′ + ′ ′ =p w h w h n(iii) ( 2 ) . (14)
Here, the first and the second constraint respectively correspond tothe horizontal and vertical dimensions of the grid G, and the thirdconstraint corresponds to the total number of intersection points. Tojustify Eq. (14), note that
= += ′ ′ + + ′ ′ += ′ + ′ + ′ ′
h w w hw h w h
Total # of line segmentsTotal # of horizontal line segments Total # of vertical line segments
( 1) ( 1)2 .
(15)
Note that this minimization problem is nonlinear. To simplify thecomputation, we assume that ′ ′w h, are very large, that is, the grid G issufficiently dense. Then, from Eq. (14), we have
=′ + ′ + ′ ′
≈′ ′
= =( )( )
pn w h w h w h
l lwh
12
12
1
2 2.
wl
hl
w h
w h (16)
By further assuming that the grid G is sufficiently close to a square grid,we have ≈l lw h. Then, it follows that
≈ ≈pn
lwh
lpwhn2
,2
.ww
2
(17)
Similarly,
≈lpwhn
2.h (18)
To satisfy the integral constraints for ′w and ′h , we make use of theabove approximations and set
′ = ′ ≔⎢
⎣
⎢⎢⎢
⎥
⎦
⎥⎥⎥
=⎢
⎣⎢
⎥
⎦⎥h h h nh
pw2.
pwhn
0 2(19)
Similarly, we set
′ = ′ ≔⎢
⎣
⎢⎢⎢
⎥
⎦
⎥⎥⎥
=⎢
⎣⎢
⎥
⎦⎥w w w nw
ph2.
pwhn
0 2(20)
Finally, we take
Fig. 4. An illustration of our sparse feature extraction scheme. The chosensparse feature points are represented by the red dots. If the segment boundarydoes not intersect an edge, no point is selected. If the segment boundary in-tersects an edge at exactly one point, the point is selected as a feature point. Ifthe segment boundary intersects an edge at multiple points, the centre of thepoints is selected as a feature point.
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
564
=′
=′
l hh
l ww
and .h w0 0 (21)
To summarize, with the abovementioned strategy for the featurepoint extraction, we obtain a set of sparse feature points which ap-proximates the segment boundaries B . Specifically, given the inputs pand n, the rectangular grid G we introduce leads to approximately nregularly-extracted sparse feature points. An illustration of the sparsefeature extraction scheme is shown in Fig. 5 (left). In our experiments, pis set to be 0.2, and n is set to be 10% of the number of pixels in thesegmentation result. A denser triangulated representation can beachieved by increasing the value of p.
3.1.5. Adding landmark points to the vertex set of the desired coarsetriangulation
This step is only required when our TRIM algorithm is used forlandmark-constrained registration. For accurate landmark-constrainedregistration, it is desirable to include the landmark points in the vertexset of the coarse representations of the input image I. One of the most
important features of our coarse triangulation approach is that it allowsregistration with exact landmark constraints on a coarse triangular re-presentation. By contrast, the regular grid-based registration can onlybe achieved on very dense rectangular grid domains in order to reducethe numerical errors.
With the above mentioned advantage of our approach, we can freelyadd a set of landmark points PLM to the set of sparse features P ex-tracted by the previous procedure. In other words, the landmark pointsare now considered as a part of our coarse triangulation vertices:
P P= ⋃⋃ ∪p p m m{ , , , } .i j
i j i j i j i j LM,1
,2
,1
,2
(22)
Then, the landmark-constrained registration of images can be computedby the existing feature-matching techniques for triangular meshes. Theexisting feature detection approaches such as [12,21] can be applied forobtaining the landmark points.
3.1.6. Computing a Delaunay triangulationIn the final step, we construct a triangulation based on the set P of
feature points. Among all triangulation schemes, the Delaunay trian-gulation method is chosen since the triangles created by the Delaunaytriangulations are more regular. More specifically, if α and β are twoangles opposite to a common edge in a Delaunay triangulation, thenthey must satisfy the inequality
+ ⩽α β π. (23)
In other words, Delaunay triangulations always aim to minimize theformation of sharp and irregular triangles. Note that the regularity doesnot only enhance the visual quality of the resulting triangulation butalso lead to a more stable approximation of the derivatives on the tri-angles when applying various registration schemes. Therefore, we
Fig. 5. An illustration of computing a Delaunay triangulation on the extracted features. Left: the points obtained by the feature extraction step from Fig. 3. Middle: aDelaunay triangulation on the feature points. Right: the triangulation with a color approximated on each triangle.
Fig. 6. An illustration of quasi-conformal mappings. The maximal magnifica-tion and shrinkage are determined by the Beltrami coefficient μ of the map-pings.
Fig. 7. Several images and the triangulations by our TRIM algorithm. Top: the input images. Bottom: the resulting triangulations. The key features of the images arewell represented in our triangulations, and the regions with similar color can be represented by coarse triangulations.
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
565
Fig. 8. A bee image and the triangulations created by our TRIM algorithm and DMesh [35]. Left to right: The input image, the coarse triangulation created by TRIM,our TRIM coarse triangulation with a color approximated on each triangle, and the triangulation by DMesh [35].
Fig. 9. An butterfly image and the triangulations created by our TRIM algorithm and DMesh [35]. Left to right: The input image, the coarse triangulation created byTRIM, our TRIM coarse triangulation with a color approximated on each triangle, and the triangulation by DMesh [35].
Fig. 10. Two more examples created by our TRIM algorithm and Dmesh [35]. Our coarse triangulations capture the important features and closely resemble theoriginal images. Left: The input images. Middle: The triangulations by TRIM. Right: The triangulations by DMesh [35].
Fig. 11. Two triangulation examples byour TRIM algorithm for noisy images.Left to right: The noise-free images, thetriangulations computed by TRIM basedon the noise-free images, the noisyimages, and The triangulations com-puted by TRIM based on the noisyimages. Note that the important fea-tures of the images are preserved evenfor noisy images.
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
566
compute a Delaunay triangulation on the set P of feature points forachieving the ultimate triangulation T . An illustration of the con-struction of the Delaunay triangulations is shown in Fig. 5.
These 6 steps complete our TRIM algorithm as summarized inAlgorithm 1.
Algorithm 1: Our proposed TRIM algorithm for triangulatingimages
Input: An image I , the desired number of image intensity levels lfor segmentation, the desired number of feature points n, thesparse ratio p.Output: A coarse triangulationT that captures the main featuresof the image.
1 Subsample the input image I to a suitable size and denote the
result by ∼I2 Apply an unsharp masking on the subsampled image ∼I and denotethe result by I
3 Apply the PSO segmentation for compressing the color space of Ito l intensity levels, and extract boundaries B of the segments
4 Extract a set of sparse feature points P from the segmentboundaries B based on the parameters n and p
5 Add a set of extra landmark points PLM to P if necessary6 Compute a Delaunay triangulationT on the sparse feature pointsP .
It is noteworthy that our proposed TRIM algorithm significantlytrims down high resolution images without distorting their importantgeometric features. Experimental results are shown in Section 4 to de-monstrate the effectiveness of the TRIM algorithm.
3.2. Stage 2 – Registration of two triangulated image surfaces
With the above triangulation algorithm for images, we can simplifythe image registration problem as a mapping problem of triangulatedsurfaces rather than of sets of landmark points. Many conventionalimage registration approaches are hindered by the long computationaltime and the accuracy of the initial maps. With the new strategy, it iseasy to obtain a highly efficient and reasonably accurate registration ofimages. Our registration result can serve as a high quality initial mapfor various algorithms.
To preserve angles and hence the local geometry of two surfaces,rather than simply mapping two sets of points, conformal mappingsmay not exist due to presence of landmark constraints. We turn toconsider quasi-conformal mappings, a type of mappings which is clo-sely related to the conformal mappings. Mathematically, a quasi-con-formal mapping →f : satisfies the Beltrami equation
∂∂
=∂∂
fz
μ zfz
( )(24)
where μ (called the Beltrami coefficient of f) is a complex-valued func-tion with sup norm less than 1. Intuitively, a conformal mapping mapsinfinitesimal circles to infinitesimal circles, while a quasi-conformalmapping maps infinitesimal circles to infinitesimal ellipses (see Fig. 6).Readers are referred to [7] for more details.
In this work, we apply the quasi-conformal landmark registration(QCLR) algorithm (designed for general surfaces in [17]) to our coarsetriangulations of images. More explicitly, to compute a registrationmapping →f I I: 1 2 between two images I1 and I2 with prescribed pointcorrespondences
↔ = …p q i n, 1, 2, , ,i i (25)
where =p{ }i in
1 are a set of points on I1 and =q{ }i in
1 are a set of points on I2,we first apply our proposed TRIM algorithm and obtain a coarse
triangulationT1 on I1. Here, we include the feature points =p{ }i in
1 in thegeneration of the coarse triangulation, as described in the fifth step ofthe TRIM algorithm. Then, instead of directly computing f, we can solvefor a map T →∼f I: 1 2. Since the problem size is significantly reducedunder the coarse triangulation, the computation for
∼f is much moreefficient than that for f.
The QCLR algorithm makes use of the penalty splitting method andminimizes
T T T∫ ∫ ∫= ∇ + + −∼ ∼E ν f ν α ν ρ ν μ f( , ) | | | | | ( )|LM
split 2 2 21 1 1 (26)
subject to (i) =∼f p q( )i i for all = …i n1, 2, , and (ii) <∞ν‖ ‖ 1. Furtheralternating minimization of the energy ELM
split over ν and∼f is used.
Specifically, for computing∼fn while fixing νn and the landmark con-
straints, we apply the linear Beltrami solver by Lui et al.[25]. Forcomputing +νn 1 while fixing
∼fn , by considering the Euler–Lagrangeequation, it suffices to solve
− + + = ∼+αI ρI ν ρμ f( Δ 2 2 ) 2 ( ).n n1 (27)
From +νn 1, one can compute the associated quasi-conformal mapping∼
+fn 1 and then update +νn 1 by
← + −∼+ + + +ν ν t μ f ν( ( ) )n n n n1 1 1 1 (28)
for some small t to satisfy the landmark constraints (25).After computing the quasi-conformal mapping
∼f on the coarse tri-angulation, we interpolate once to retrieve the fine details of the re-gistration in the high resolution. Since the triangulations created by ourproposed TRIM algorithm preserves the important geometric featuresand prominent straight lines of the input image, the details of the re-gistration results can be accurately interpolated. Moreover, since thecoarse triangulation largely simplifies the input image and reduces theproblem size, the computation is significantly accelerated.
The overall registration procedure is summarized in Algorithm 2.Experimental results are illustrated in Section 4 to demonstrate thesignificance of our coarse triangulation in the registration scheme.
Algorithm 2: Feature-based registration via our proposed TRIMalgorithm
Input: Two images or video frames I1, I2 to be registered, with theprescribed feature correspondences.Output: A feature-matching registration mapping →f I I: 1 2.
1 Compute a coarse triangulation T1 of I1 using our proposed TRIMalgorithm (Algorithm 1). Here, we include the prescribed featurepoints on I1 in the generation of the coarse triangulation T1
2 Select landmark correspondences of the coarse triangulation T1
and the target image I2. Denote the landmark points on T1 and I2
by =p{ }i in
1 and =q{ }i in
1 correspondingly
3 Compute a landmark based quasi-conformal mapping T →∼f : 1
by the QCLR algorithm in [17]
4 Obtain f by∼f with a bilinear interpolation between Tj and Ij.
4. Experimental results
In this section, we demonstrate the effectiveness of our proposed tri-angulation scheme. The algorithms are implemented using MATLAB. Theunsharp masking step is done using MATLAB’s imsharpen. The PSOsegmentation is done using the MATLAB Central function segmenta-tion. For solving the mentioned linear systems, the backslash operator (⧹)in MATLAB is used. The test images are courtesy of the RetargetMe dataset[28] and the Middlebury Stereo Datasets [29,30]. The bird image iscourtesy of the first author. All experiments are performed on a PC with anIntel(R) Core(TM) i7-4500U [email protected] processor and 8.00GB RAM.
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
567
4.1. Performance of our proposed triangulation (Algorithm 1)
In this subsection, we demonstrate the effectiveness of our trian-gulation scheme by various examples.
Our proposed algorithm is highly content-aware. Specifically, re-gions with high similarities or changes in color on an image can beeasily recognized. As a result, the triangulations created faithfullypreserve the important features by a combination of coarse triangleswith different sizes. Some of our triangulation results are illustrated inFig. 7. For better visualizations, we color the resulting triangulations bymean of the original colors of corresponding patches. In Fig. 8, we applyour TRIM algorithm on a bee image. It can be observed that the regionsof the green background can be effectively represented by coarser tri-angulations, while the region of the bee and flowers with apparent colordifferences is well detected and represented by a denser triangulation.Fig. 9 shows another example of our triangulation result. The butterflyand the flowers are well represented in our triangulation result. Theabove examples demonstrate the effectiveness of our triangulationscheme for representing images in a simplified but accurate way. Somemore triangulation examples created by our TRIM algorithm are shownin Fig. 10. Fig. 11 shows some triangulation examples for noisy images.It can be observed that our TRIM algorithm can effectively computecontent-aware coarse triangulations even for noisy images.
We have compared our algorithm with the DMesh triangulator [35]in Fig. 8, Fig. 9 and Fig. 10. It can be observed that our triangulationscheme outperforms DMesh [35] in terms of the triangulation quality.Our results can better capture the important features of the images.Also, the results by DMesh [35] may contain unwanted holes while ourtriangulation results are always perfect rectangles. The comparisonsreflect the advantage of our coarse triangulation scheme.
To quantitatively compare the content-aware property of our TRIMmethod and the DMesh method, we calculate the average absolute in-tensity difference −I I‖ ‖N
1triangulated original 1 between the original image
Ioriginal (e.g. the left images in Fig. 10) and the triangulated imageItriangulated with piecewise constant color for each method (e.g. themiddle and the right images in Fig. 10), where N is the number of pixelsof the image. Table 1 lists the statistics. It is noteworthy that theaverage absolute intensity difference by TRIM is smaller than that byDMesh by around 30% on average. This indicates that our TRIM algo-rithm is more capable to produce content-aware triangulations.
Then, we evaluate the efficiency of our triangulation scheme forvarious images. Table 2 shows the detailed statistics. The relationshipbetween the target coarse triangulation size and the computational timeis illustrated in Fig. 12. Even for high resolution images, the compu-tational time for the triangulation is only around 10 s. It is noteworthythat our TRIM algorithm significantly compresses the high resolutionimages as coarse triangulations with only several thousand triangles.
It is noteworthy that the combination of the steps in our TRIM
Table 1The content-aware property of our TRIM algorithm and the DMesh method.
Fig. 12. The relationship of the desired coarse triangulation size and thecomputational time of our proposed TRIM algorithm.
Fig. 13. The triangulations created by our TRIM algorithm (left) and by the algorithm without the segmentation step (Right). The results show that the segmentationstep is crucial for achieving a coarse triangulation. Number of triangles produced (left to right): 923, 3612, 1496, 8685.
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
568
algorithm is important for achieving a coarse triangulation. More spe-cifically, if certain steps in our algorithm are removed, the triangulationresult will become unsatisfactory. Fig. 13 shows two examples of tri-angulations created by our entire TRIM algorithm and by our algorithmwith the segmentation step excluded. It can be easily observed thatwithout the segmentation step, the resulting triangulations are ex-tremely dense and hence undesirable for simplifying further computa-tions. By contrast, the number of triangles produced by our entire TRIMalgorithm is significantly reduced. The examples highlight the im-portance of our proposed combination of steps in the TRIM algorithmfor content-aware coarse triangulation.
We also analyze the sensitivity of the triangulated images to the
parameters in the unsharp masking step. Fig. 14 shows several trian-gulation results with different choice of λ σ s θ( , , , ). It can be observedthat the triangulation results are robust to the parameters.
4.2. Registration of two triangulated image surfaces (Algorithm 2)
In this subsection, we demonstrate the effectiveness of our proposedtriangulation-based method for landmark-based image registration. Inour experiments, the feature points on the images are extracted usingthe Harris–Stephens algorithm [12] as landmark constraints. Thelandmark extraction is fully automatic. More specifically, we use theMATLAB functions detectHarrisFeatures,extractFeaturesand matchFeatures on the images. For the teddy example, 132landmark pairs are generated using the above procedure. For the conesexample, 162 landmark pairs are generated.
For simplifying the image registration problems, one conventionalapproach is to make use of coarse regular grids followed by interpola-tion. It is natural to ask whether our proposed coarse triangulation-
Fig. 14. The triangulation results with different parameters λ σ s θ( , , , ) in the unsharp masking step. Top left: The original image. Top middle left to bottom right:results with different parameters.)
Fig. 15. Stereo landmark registration of two images of doll using different al-gorithms. (a): The source image. (b): The target image. (c): The prescribedfeature correspondences. (d): The registration result by the dense grid-basedapproach (4 pixels per grid). (e): The registration result via DMesh [35]. (f): Theregistration result by our TRIM-based method. (g): The intensity difference afterthe registration by the dense grid-based approach. (h): The intensity differenceafter the registration via DMesh [35]. (i): The intensity difference after theregistration by our TRIM-based method.
Fig. 16. Stereo landmark registration of two images of cones using differentalgorithms. Please refer to Fig. 15 for the description of the images.
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
569
based method produces better results. In Fig. 15, we consider a stereoregistration problem of two scenes. With the prescribed feature corre-spondences, we compute the feature-endowed stereo registration viathe conventional grid-based approach, the DMesh triangulation ap-proach [35] and our proposed TRIM method. For the grid-based ap-proach and the DMesh triangulation approach [35], we take the meshvertices nearest to the prescribed feature points on the source image assource landmarks. For our proposed TRIM method, as the landmarkvertices are automatically embedded in the content-aware coarse tri-angulation, the source landmarks are exactly the feature points detectedby the method in [12].
It can be observed that our triangulation-based approach produces amuch more natural and accurate registration result when compared
with both the grid-based approach and the DMesh triangulation ap-proach. In particular, sharp features such as edges are well preservedusing our proposed method. By contrast, the edges are seriously dis-torted in the other two methods. In addition, the geometry of thebackground in the scenes are well retained via our TRIM method butnot the other two methods. The higher accuracy of the registrationresult by our approach can also be visualized by the intensity differenceplots. Our triangulation-based approach results in an intensity differ-ence plot with more dark regions than the other two approaches. Theadvantage of our method over the other two methods is attributed tothe geometry preserving feature of our TRIM algorithm, in the sensethat the triangulations created by TRIM are more able to fit into com-plex features and have more flexibilities in size than regular grids. Also,the triangulations created by DMesh [35] do not capture the geometricfeatures and hence the registration results are unsatisfactory. They re-flect the significance of our content-aware TRIM triangulation schemein computing image registration. Another example is illustrated inFig. 16. Again, it can be easily observed that our proposed TRIM tri-angulation approach leads to a more accurate registration result.
To highlight the improvement in the efficiency by our proposedTRIM algorithm, Table 3 records the computational time and the errorof the registration via the conventional grid-based approach and ourTRIM triangulation-based approach. It is noteworthy that our proposedcoarse triangulation-based method significantly reduces the computa-tional time by over 85% on average when compared with the tradi-tional regular grid-based approach. To quantitatively assess the qualityof the registration results, we define the matching accuracy by
=− ∊
×A# pixels for which‖final intensity original intensity‖ is less than
Total # of pixels100%.1
(29)
The threshold ∊ is set to be 0.2 in our experiments. Our triangulation-
Table 4Comparison between our TRIM-based image registration, the Large Displacement Optical Flow (LDOF) [4] and the Spectral Log-Demons [20].
Images Size TRIM LDOF Spectral Log-Demons
Time (s) Matching accuracy (%) Time (s) Matching accuracy (%) Time (s) Matching accuracy (%)
Fig. 17. Different landmark correspondencesand the TRIM-based registration results for theteddy example in Fig. 15. Left: Only 1/2 of thelandmark pairs in Fig. 15 are randomly selectedfor computing the registration. Middle: Only 1/3of the landmark pairs are used. Right: Only 1/3of the landmark pairs are used, with 2% randomnoise added to the target landmark locations.The matching accuracies are respectively70.5%, 70.4%, 67.2%, which are very close to theoriginal result (70.7%).
Table 3The performance of feature-based image registration via our proposed TRIMcoarse triangulation method and the ordinary coarse grids. Here, the timesaving rate is defined by ×− 100%TRIMRegistration time via regular grids Registration time via
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
570
based method produces registration results with the matching accuracyhigher than that of the regular grid-based method by 6% on average.The experimental results reflect the advantages of our TRIM content-aware coarse triangulations for image registration.
We further compare our TRIM-based registration method with twoother state-of-the-art image registration methods, namely the LargeDisplacement Optical Flow (LDOF) [4] and the Diffeomorphic Log-Demons [20]. Table 4 lists the performance of the methods. It is note-worthy that our method is significantly faster than the two othermethods, with at least comparable and sometimes better matching ac-curacy.
Besides, we study the stability of the TRIM-based registration resultwith respect to the feature points detected. Fig. 17 shows the resultswith different feature correspondences, including a change in thenumber of landmark pairs and a change in the target landmark position.From the resulting triangulated images and the statistics on thematching accuracy, it can be observed that the deformation is stablewith respect to the choice of the feature points.
5. Conclusion and future work
In this paper, we have proposed a new image registration algorithm(Algorithm 2), which operates on content-aware coarse triangulationsto aid registration of high resolution images. The obtained algorithm iscomputationally efficient and capable to achieve a highly accurate re-sult while resembling the original image. It has two stages with stage 1obtaining content-aware coarse triangulations and stage 2 registeringtwo triangulated surfaces. Both algorithms can be used as standalonemethods: Algorithm 1 for extracting main features of images (com-pression) and Algorithm 2 for registering two surfaces (quality map-ping).
Our proposed method is advantageous for a large variety of regis-tration applications with a significant improvement of the computa-tional efficiency and registration accuracy. Our proposed method canalso serve as an effective initialization for other registration algorithms.In the future, we aim to extend our proposed algorithm to high di-mensions.
Acknowledgment
Lok Ming Lui is supported by HKRGC GRF (Project ID: 402413).
References
[1] F.L. Bookstein, The measurement of biological shape and shape change, Springer-Verlag: Lecture Notes Biomathematics 24 (1978) 1–191.
[3] F.L. Bookstein, Linear methods for nonlinear maps: procrustes fits, thin-platesplines, and the biometric analysis of shape variability, Brain Warping, AcademicPress, London, 1999, pp. 157–181.
[4] T. Brox, J. Malik, Large displacement optical flow: descriptor matching in varia-tional motion estimation, IEEE Trans. Pattern Anal. Mach. Intell. 33 (3) (2011)500–513.
[5] P.T. Choi, K.C. Lam, L.M. Lui, FLASH: Fast landmark aligned spherical harmonicparameterization for genus-0 closed brain surfaces, SIAM J. Imaging Sci. 8 (1)(2015) 67–94.
[6] W.R. Crum, T. Hartkens, D.L.G. Hill, Non-rigid image registration: theory andpractice, Br. J. Radiol. 77 (2004) 140–153.
[7] F. Gardiner, N. Lakic, Quasiconformal Teichmüller theory, Math. SurveysMonographs 76 (2000) American Mathematics Society.
[8] J.C. Gee, D.R. Haynor, M. Reivich, R. Bajcsy, Finite element approach to warping ofbrain images, Proc. SPIE (1994) 327–337.
[9] P. Ghamisi, M.S. Couceiro, J.A. Benediktsson, N.M.F. Ferreira, An efficient methodfor segmentation of images based on fractional calculus and natural selection,
Expert. Syst. Appl. 39 (16) (2012) 12407–12417.[10] J. Glaunès, M. Vaillant, M.I. Miller, Landmark matching via large deformation
diffeomorphisms on the sphere, J. Math. Imaging Vis. 20 (1) (2004) 179–200.[11] J. Glaunès, A. Qiu, M.I. Miller, L. Younes, Large deformation diffeomorphic metric
curve mapping, Int. J. Comput. Vis. 80 (3) (2008) 317–336.[12] C. Harris, M. Stephens, A combined corner and edge detector, Proceedings of the
4th Alvey Vision Conference, 1988, pp. 147–151.[13] H.J. Johnson, G.E. Christensen, Consistent landmark and intensity-based image
registration, IEEE Trans. Med. Imag. 21 (5) (2002) 450–461.[14] S.C. Joshi, M.I. Miller, Landmark matching via large deformation diffeomorphisms,
IEEE Trans. Image Process. 9 (8) (2010) 1357–1370.[15] P. Kaufmann, O. Wang, A. Sorkine-Hornung, O. Sorkine-Hornung, A. Smolic,
M. Gross, Finite Element Image Warping, Computer Graphics Forum 32 (2pt1)(2013) 31–39.
[16] A. Klein, J. Andersson, B.A. Ardekani, J. Ashburner, B. Avants, M.-C. Chiang,G.E. Christensen, D.L. Collins, J. Gee, P. Hellier, J.H. Song, M. Jenkinson, C. Lepage,D. Rueckert, P. Thompson, T. Vercauteren, R.P. Woods, J.J. Mann, R.V. Parsey,Evaluation of 14 nonlinear deformation algorithms applied to human brain MRIregistration, NeuroImage 46 (3) (2009) 786–802.
[17] K.C. Lam, L.M. Lui, Landmark and intensity based registration with large de-formations via quasi-conformal maps, SIAM J. Imaging Sci. 7 (4) (2014)2364–2392.
[18] B. Lehner, G. Umlauf, B. Hamann, Image compression using data-dependent tri-angulations, Int. Symp. Visualization Computer Graphics (2007) 351–362.
[19] B. Lehner, G. Umlauf, B. Hamann, Video compression using data-dependent trian-gulations, Computer Graphics Visualization (2008) 244–248.
[20] H. Lombaert, L. Grady, X. Pennec, N. Ayache, F. Cheriet, Spectral log-demons:Diffeomorphic image registration with very large deformations, Int. J. Comput. Vis.107 (3) (2014) 254–271.
[21] D. Lowe, Distinctive image features from scale-invariant keypoints, Int. J. Comput.Vis. 60 (2) (2004) 91–110.
[22] L.M. Lui, Y. Wang, T.F. Chan, P.M. Thompson, Landmark constrained genus zerosurface conformal mapping and its application to brain mapping research, Appl.Numer. Math. 57 (5–7) (2007) 847–858.
[23] L.M. Lui, S. Thiruvenkadam, Y. Wang, P.M. Thompson, T.F. Chan, Optimizedconformal surface registration with shape-based landmark matching, SIAM J.Imaging Sci. 3 (1) (2010) 52–78.
[24] L.M. Lui, K.C. Lam, S.T. Yau, X. Gu, Teichmüller mapping (T-map) and its appli-cations to landmark matching registration, SIAM J. Imaging Sci. 7 (1) (2014)391–426.
[25] L.M. Lui, K.C. Lam, T.W. Wong, X. Gu, Texture map and video compression usingBeltrami representation, SIAM J. Imaging Sci. 6 (4) (2013) 1880–1902.
[26] T.W. Meng, G.P.-T. Choi, L.M. Lui, TEMPO: Feature-endowed Teichmüller extremalmappings of point clouds, SIAM J. Imaging Sci. 9 (4) (2016) 1922–1962.
[27] A. Polesel, G. Ramponi, V.J. Mathews, Image enhancement via adaptive unsharpmasking, IEEE Trans. Image Process. 9 (3) (2000) 505–510.
[28] M. Rubinstein, D. Gutierrez, O. Sorkine, A. Shamir, A comparative study of imageretargeting, ACM Trans. Graph. (SIGGRAPH Asia 2010) 29 (5) (2010)160:1–160:10.
[29] D. Scharstein, R. Szeliski, High-accuracy stereo depth maps using structured light,IEEE Computer Soc. Conf. Computer Vision Pattern Recognition (CVPR) 1 (2003)195–202.
[30] D. Scharstein, C. Pal, Learning conditional random fields for stereo, IEEE ComputerSoc. Conf. Computer Vision Pattern Recognition (CVPR) (2007) 1–8.
[31] J.R. Shewchuk, Triangle: Engineering a 2D quality mesh generator and Delaunaytriangulator, Appl. Comput. Geometry: Towards Geometric Eng. 1148 (1996)203–222.
[32] R. Shi, W. Zeng, Z. Su, H. Damasio, Z. Lu, Y. Wang, S.T. Yau, X. Gu, Hyperbolicharmonic mapping for constrained brain surface registration, IEEE Conf. ComputerVision Pattern Recognition (CVPR) (2013) 2531–2538.
[33] Y. Wang, L.M. Lui, X. Gu, K.M. Hayashi, T.F. Chan, A.W. Toga, P.M. Thompson,S.T. Yau, Brain surface conformal parameterization using Riemann surface struc-ture, IEEE Trans. Med. Imag. 26 (6) (2007) 853–865.
[34] Y. Wang, L.M. Lui, T.F. Chan, P.M. Thompson, Optimization of brain conformalmapping with landmarks, Med. Image Comput. Comput. Assist. Interv. (MICCAI) I I(2005) 675–683.
[36] W. Zeng, X.D. Gu, Registration for 3D surfaces with large deformations using quasi-conformal curvature flow, IEEE Conf. Computer Pattern Recognition (CVPR) (2011)2457–2464.
[37] W. Zeng, L.M. Lui, X. Gu, Surface registration by optimization in constrained dif-feomorphism space, IEEE Conf. Computer Vision Pattern Recognition (CVPR)(2014) 4169–4176.
[38] J. Zhang, K. Chen, B. Yu, An efficient numerical method for mean curvature-basedimage registration model, East Asian J. Appl. Math. 7 (01) (2017) 125–142.
[39] B. Zitova, J. Flusser, Image registration methods: a survey, Image. Vision Comput.21 (11) (2003) 977–1000.
C.P. Yung et al. Journal of Visual Communication and Image Representation 55 (2018) 561–571
