O(1) Transposed Bilateral Filtering for OptimizationKenjiro Sugimoto∗, Keiichiro Shirai† and Sei-ichiro Kamata∗
∗Graduate School of Information, Production and Systems, Waseda University, Kitakyushu, Japan.E-mail: sugimoto@asagi.,[email protected]
†Faculty of Engineering, Shinshu University, Nagano, Japan.E-mail: [email protected]
Abstract—This paper presents an essential algorithm foroptimization-based image processing using the bilateral filter(BLF), called constant-time transposed BLF (O(1) TBLF). Someiterative solvers for optimization problems require a pair of filtersdefined as multiplying a filter matrix or its transpose to vectorizedimages. Since the BLF can be described as a matrix form, itspaired filter also exists, called a TBLF in this paper. BLF-basedoptimization achieves high smoothing performance; whereas, itrequires much high computational complexity due to iteratingboth BLF and TBLF many times. Hence, this paper designsan O(1) TBLF algorithm to accelerate the iterative process.Experiments show that our O(1) TBLF runs in low complexityregardless of its filter window size and works effectively forflash/no-flash image integration via BLF-based optimization.
I. INTRODUCTION
Edge-preserving smoothing has played a fundamental roleon image processing, computer vision, and computer graphicsrecent years. In particular, the bilateral filter (BLF) [1]–[3],which determines filter coefficients from two laterals: pixelposition and pixel intensity, has flourished in various appli-cations because of its clear concept and smoothing efficiency.The BLF has been widely extended in the literature to enhancethe smoothing efficiency or to reduce the computational com-plexity. The cross BLF [4] (XBLF), which is identical to thejoint BLF [5], is a natural extension of the BLF. In filteringa noisy target image, the BLF diminishes the smoothingefficiency due to determining its filter coefficients from thenoisy target; by contrast, the XBLF overcomes this problemby determining them from a guide image captured under adifferent photographic condition instead. Another algorithmicextension is the fast BLF [6], [7] including the constant-timeBLF (O(1) BLF) [8]–[11]. Both original BLF and XBLF oftenrequire unacceptable computational complexity in filteringhigh-resolution or high-dimensional images because the cost-per-pixel depends on the filter window size. On the otherhand, the O(1) BLF runs in O(1) time per pixel, easilygeneralized to the XBLF. We discuss this BLF family froma different viewpoint. For simplicity, both BLF and XBLF arecollectively-referred to as BLF in this paper.
Our motivation comes from optimization-based image pro-cessing. This handles image restoration tasks such as deblur-ring through formalizing the tasks as optimization problemsand solving the formalized problems by iterative solvers.The procedures of iterative solvers are generally described asmatrix equations that consist of several linear transforms tovectorized images. These transforms can also be interpreted
as a series of image filters. Thus, we basically have twoaspects to this operation: as a matrix form or as a filter form.Since the BLF is a time-varying filter, its filter matrix hasdifferent elements in different rows. Using this filter matrixexpression, we can easily formalize optimization problemsbased on the BLF. Importantly, the BLF-based optimizationhas been showing outstanding performance in several tasksrecent years [12], [13].
A severe problem for the BLF-based optimization is itstremendous computational complexity. This stems from thefollowing situation. Consider a filter matrix B that describesa filter to a vectorized image. In a recent convex optimizationas mentioned in [14], some iterative solvers require to apply Band B⊤ several times for each iteration1. Since the BLF canbe represented as B, its paired filter B⊤ also exists. We callit the transposed BLF (TBLF). Thus, iteratively solving BLF-based optimization problems requires a pair of the BLF and theTBLF. However, straightforwardly generating, transposing andmultiplying B⊤ highly cost for iterative solvers. Moreover, un-like the O(1) BLF, the computational complexity is not O(1)per pixel. We should perform the TBLF without explicitly-generating B⊤ in low computational complexity regardless ofthe filter window size.
This paper presents essential algorithms for acceleratingBLF-based optimization. We first derive the filter form ofthe TBLF to perform it without explicitly-generating B⊤. Wethen propose an O(1) TBLF to solve BLF-based optimizationproblems in acceptable computational complexity. As an appli-cation example, we discuss a BLF-based optimization problemfor flash/no-flash image integration [4], [5]. It is worth notingthat our approach is applicable to any optimization tasks witha similar mathematical model to this example.
II. OPTIMIZATION BASED ON BILATERAL FILTERING
This section clarifies our motivation through an example ofimage processing applications under our scenario.
A. Bilateral Filter
Consider smoothing a grayscale image by the BLF in across/joint fashion. Let x,y, z ∈ RN be vectorized images thatare a target image, a reference image, and a smoothed imagewhere N indicates the number of pixels. We use a notation [i]
1The matrix forms Bx and B⊤x correspond to the convolution filtering(f ⊗ g)(x) =
∑u f(u)g(x− u) and the correlation filtering (f ∗ g)(x) =∑
to denote the element at i-th row for a vector and use [i, j] todenote the element at i-th row of j-th column for a matrix.Let Ω ⊂ Z2 be an image domain, i the index of a pixel, andpi ∈ Ω a pixel position, and j ∈ Ni a neighboring pixelcentered at the pixel i. The BLF has the form
By[i, j] =gs(pj − pi) gr(y[j]− y[i])∑
k∈Nigs(pk − pi) gr(y[k]− y[i])
, (1)
where gs(·) is a spatial kernel, gr(·) is a range kernel. Notethat the subscript y of By indicates that the filter weightis determined from the image y. This filtering process isrepresented as z = Byx and, if x = y, it is identicalto the original BLF. The kernel functions gs(·) and gr(·)are generally defined to be Gaussian functions with standarddeviation σs and σr (called scale) in the form
gs(t) = exp
(−∥t∥222σ2
s
), gr(t) = exp
(− t2
2σ2r
), (2)
where ∥·∥2 denotes ℓ2-norm. The kernel size |Ni|, i.e., thewindow size of the filter, depends on σs.
B. Sample Model for Optimization
As a sample application under our scenario, we focus onflash/no-flash image integration [4], [5]. This task aims togenerate a noiseless no-flash images by compositing the basestructure component of a no-flash image and the texture detailcomponent of a flash image. Generally, the BLF has beenmainly used for smoothing and decomposition of imagescomponents. In fact, the XBLF was first proposed for this task.We describe the formulations for this application to clarify ourtarget problem. Note that our proposed algorithm is applicableto any optimization tasks with similar formulations.
In flash/no-flash image integration, x is a no-flash image thathas preferred color information but contains much noise, andy is a flash image that has unpreferred color information butcontains less noise where each color component is processedindependently. In order to enhance smoothing performance,the following filtering process is designed in [5]:
z = Byx+ (I−By)y, (3)
where I ∈ RN×N is an identity matrix. In (3), the first termrepresents structure components such as color or contour andthe second term represents texture components obtained bysubtracting the source image y from its smoothed image Byy.
C. Motivation
The aforementioned approach can be improved by extendingit to an convex optimization problem. Specifically
argminz∥By (z− x)∥22 + λ ∥(I−By) (z− y)∥22 , (4)
where λ is an influence rate for balancing both terms and thesecond term plays a role analogous to total variation (TV)using the BLF. The general model (4) as an optimizationproblem has some advantages over the original model (3): itguarantees existence and uniqueness of the optimal solution
and provides options such as adjustment of influence rate andflexibility of norm selection. Its solution can be obtained asfollows. Derivating (4) w.r.t. z and assuming it to be zero, weobtain
B⊤yBy + λ (I−By)
⊤(I−By)
z (5)
= B⊤yByx+ λ (I−By)
⊤(I−By)y.
This equation can be interpreted as a standard form Ax = b.Since A is a symmetric matrix, we can efficiently solve theproblem by an iterative solver such as the conjugate gradientmethod [15], [16]. For each iteration, the following operationof matrix A and a conjugate vector u are required:
Au :=B⊤
yBy + λ (I−By)⊤(I−By)
u. (6)
Obviously, the computational complexity of this iterative oper-ation is dominated by the two By and the two B⊤
y . Althoughour optimization approach will provide high smoothing perfor-mance, it requires tremendous computational complexity dueto multiplying By and B⊤
y many times. Hence, we acceleratethe operation of B⊤
y to solve the optimization problem inacceptable complexity. Importantly, BLF-based optimizationapproaches with similar formalizations [12], [13] have alsofaced to the same difficulty.
III. O(1) TRANSPOSED BILATERAL FILTERING
This section describes the filter form and the matrix formof the BLF and the TBLF. The important points are how toperform the TBLF without generating the B⊤ and how toaccelerate it to O(1) complexity per pixel.
A. Filter Form of Transposed Cross Bilateral Filter
We clarify differences between the BLF (z = Byx) andthe TBLF (z = B⊤
y x). Consider decomposing By into anumerator part and a denominator part by newly introducingfilter matrix Gy ∈ RN×N ,
Gy [i, j] = gs(pj − pi) gr(y[j]− y[i]), wy = Gy1,
where 1 ∈ RN is a column vector of all ones. The BLF isexpanded to
Byx =Gy ⊘
(wy1
⊤)x = (Gyx)⊘wy, (7)
where ⊘ denotes an element-wise division operator for a ma-trix or vector, and we used
A⊘ (bc⊤)
x = A(x⊘ c)⊘
b, and b, c ∈ RN , A ∈ RN×N . Similarly, the TBLF is
B⊤y x =
Gy ⊘
(wy1
⊤)⊤ x =G⊤
y ⊘(wy1
⊤)⊤x
=Gy ⊘
(1w⊤
y
)x = Gy (x⊘wy) , (8)
where Gy = G⊤y . Note that wy1
⊤ in (7) and 1w⊤y in (8)
represent matrices aligning wy in each column and w⊤y in
each row, respectively. An important difference between thematrix forms of the BLF and the TBLF is the target of dividingGyx by wy. The BLF divides the output image of Gyx; bycontrast, the TBLF divides the input image of Gyx in advance.
Filter forms, which we can operate without explicitly-generating filter matrices, are more efficient than matrix formsin terms of computational and space complexity. The filterform of the TBLF is derived from (8) by using intermediateimage s ∈ RN as
s[i] =x[i]∑
j∈Nigs(pj − pi) gr(y[j]− y[i])
, (9)
z[i] =∑j∈Ni
gs(pj − pi) gr(y[j]− y[i]) s[j]. (10)
Although this operation is identical to z = B⊤y x, we can
operate it without explicitly-generating entire B⊤y . Thereby,
a procedure of the TBLF is summarized as follows:
Algorithm 1 Transposed Bilateral Filter
1: function TBLF(x, y) ▷ x: Target, y: Reference2: s← x⊘ (Gy1) ▷ (9)3: z← Gys ▷ (10)4: return z
B. Constant-time Algorithm
The computational complexity of the aforementioned naivealgorithm depends on the filter window size, i.e., the spatialscale σs. In order to achieve faster optimization, we shouldimprove the naive TBLF to the constant-time one. An im-portant observation is that both (9) and (10) can execute inO(1) complexity per pixel. As [10], [11] reported, each ofthe numerator and the denominator in (1) can be executed inO(1). Obviously, (9) and (10) have the same forms as thenumerator and denominator and the division in (9) obviouslyruns in O(1). Due to space limitation, we have explained onlya summary of the procedure of our O(1) TBLF (see also [11]for the detail).
We modify the algorithm of [11] and then apply the sameapproach to the TBLF. In [11], the BLF is decomposedinto a bunch of the constant-time Gaussian filters such asthe recursive Gaussian filters [17], [18]. This paper appliestwo techniques to achieve more efficient performance. First,replacing the recursive Gaussian filtering to the constant-timeGaussian filter proposed in [19] provides a higher approximateaccuracy in lower computational complexity. Second, aggre-gating the cosine terms of approximate Gaussian kernels usedin [11] can reduce the number of the Gaussian filters by halfwithout a loss of accuracy. Specifically, in Eq. (16) of [11],the m-th and (N −m)-th cosine terms (m = 0, 1, . . . ,
⌊N2
⌋)
share the same cosine basis. These two modifications providea 4~5× faster computation than [11].
IV. EXPERIMENTS AND DISCUSSION
First of all, we evaluate the basic performance of the naiveBLF, the naive TBLF, and our O(1) TBLF. The test image is“lenna” (RGB with 512×512 pixels), treated as 64-bits floatingpoint RGB data with a normalized dynamic range, i.e., [0, 1].
1 2 4 8 1 6
Spatial scale σs
0 .1
1 .0
1 0 .0
Co
mp
uta
tio
na
l ti
me
[s
]
lenna (512x512 pixels, RGB), σr =0.1
Na ive BLF
Na ive TBLF
O(1) TBLF (ǫ=0.5)
0 .0 5 0 .1 0 0 .1 5 0 .2 0
Range scale σr
0
1
2
3
4
5
6
7
8
Co
mp
uta
tio
na
l ti
me
[s
]
lenna (512x512 pixels, RGB), σs =2.0
Na ive BLF
Na ive TBLF
O(1) TBLF (ǫ=0.5)
Fig. 1. Scale σs and σr versus computational time [s].
1 2 4 8 1 6
Spatial scale σs
0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
Ac
cu
rac
y;
PS
NR
[d
B]
lenna (512x512 pixels, RGB), σr =0.1
O(1)-TBLF (ǫ=0.5)
0 .0 5 0 .1 0 0 .1 5 0 .2 0
Range scale σr
0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
Ac
cu
rac
y;
PS
NR
[d
B]
lenna (512x512 pixels, RGB), σs =2.0
O(1)-TBLF (ǫ=0.5)
Fig. 2. Scale σs and σr versus approximate accuracy [dB].
We set window size M = 2 ⌈3σs⌉+ 1 for the naive BLF andthe naive TBLF, and ϵ = 0.5 and K = 1 for our O(1) TBLF.All the implementations are written in Matlab with MEX.Our test environment mounts Intel 2.67GHz CPU with 8GBmain memory. Figure 1 plots the spatial scale σs or the rangescale σr against the computational time [s]. Against σs, thenaive TBLF consumes O(M) complexity because M dependson σs; by contrast, our O(1) TBLF evidently achieves O(1)complexity. Against σr, even though our O(1) BLF somewhatdepends on σr, the computation is at least 3× faster than thenaive TBLF over a wide range of σr. The naive TBLF is nearly2× slower than the naive BLF due to its inefficient pipeline,i.e., (7) requires only one-pass processing but (8) does three-pass. Figure 2 plots σr and σs versus the approximate accuracy[dB], revealing that our O(1) TBLF can produce a sufficientapproximate accuracy regardless of σr and σs. Thus, our O(1)TBLF can replace the naive TBLF without a large loss ofaccuracy and can provide a faster filtering speed.
Figure 3 lists output images of the competitors and the20× amplified error between the naive TBLF and our O(1)TBLF for visual assessment. The BLF works as an edge-preserved smoother; on the other hand, the TBLF works assmoothing around edges, e.g., some Halo occurred aroundedges, but emphasizing the center core of edges or smallregions. Our O(1) TBLF produces some error around edgesas compared with the TBLF but the amount of errors issufficiently acceptable since the error image is 20× amplified.
Third, we observe the effectiveness of BLF-based optimiza-tion approaches to flash/no-flash image integration. The testimages are RGB with 400×476 pixels. Figure 4 shows theresults of well-known approaches. A key advantage of ourapproach is to strictly provide one optimum solution andto adjust the strongness/weakness of the texture component
Fig. 3. Filtered images (each left) and their subimages (each right). Parameter configuration is σs = 4.0, σr = 0.15, and ϵ = 0.5. Image (d) is the 20×amplified difference between (b) and (c) to facilitate visualization.
Fig. 4. Results of flash/no-flash image integration. The parameter configuration is σs = 4.5 and σr = 0.1. Image (f) is the difference between (d) and (e).
by changing λ. The computational time of our optimizationalgorithm is about 9 [s] (10 iterations × 0.9 [s] per iteration).Our O(1) TBLF achieved this acceptable time of iterativeapproaches.
V. CONCLUSIONS
This paper presented an O(1) TBLF as an essential algo-rithm for BLF-based optimization problems. Our experimentsvalidated the efficiency of our TBLF in terms of the com-putational complexity and the approximate accuracy over awide range of scale parameters. This performance enables usto make it possible solve BLF-based optimization problemsin acceptable computational complexity, as we validated inflash/no-flash image integration. Moreover, the simple model(4) can be extended to more highly-designed models, e.g., byadding terms of vector-valued total variation for color images,and can be solved by more advanced solvers such as theprimal-dual splitting [14]. Hence, our approach has a largepotential to contribute many image processing applications.
REFERENCES
[1] V. Aurich and J. Weule, “Non-linear Gaussian filters performing edgepreserving diffusion,” in Mustererkennung 1995, 17. DAGM-Symposium,1995, pp. 538–545.
[2] S. M. Smith and J. M. Brady, “SUSAN — a new approach to lowlevel image processing,” Int. J. Comput. Vis. (IJCV), vol. 23, no. 1, pp.45–78, 1997.
[3] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and colorimages,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Jan. 1998, pp.839–846.
[4] E. Eisemann and F. Durand, “Flash photography enhancement viaintrinsic relighting,” ACM Trans. Graph. (Proc. SIGGRAPH), vol. 1,no. 212, pp. 673–678, 2004.
[5] G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen, H. Hoppe, andK. Toyama, “Digital photography with flash and no-flash image pairs,”ACM Trans. Graph. (Proc. SIGGRAPH), vol. 1, no. 212, pp. 664–672,Aug. 2004.
[6] F. Durand and J. Dorsey, “Fast bilateral filtering for the display of high-dynamic-range images,” ACM Trans. Graph. (Proc. SIGGRAPH), pp.257–266, 2002.
[7] S. Paris and F. Durand, “A fast approximation of the bilateral filterusing a signal processing approach,” Int. J. Comput. Vis. (IJCV), vol.81, no. 1, pp. 24–52, Jan. 2009.
[8] Q. Yang, K.-H. Tan, and N. Ahuja, “Real-time O(1) bilateral filtering,”in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). June 2009,number 1, pp. 557–564, IEEE.
[9] S. Yoshizawa, A. Belyaev, and H. Yokota, “Fast Gauss bilateralfiltering,” Computer Graphics Forum, vol. 29, no. 1, pp. 60–74, Mar.2010.
[10] K. N. Chaudhury, D. Sage, and M. Unser, “Fast O(1) bilateral filteringusing trigonometric range kernels,” IEEE Trans. Image Process., vol.20, no. 12, pp. 3376–82, Dec. 2011.
[11] K. N. Chaudhury, “Acceleration of the shiftable O(1) algorithm forbilateral filtering and nonlocal means,” IEEE Trans. Image Process.,vol. 22, no. 4, pp. 1291–1300, Apr. 2013.
[12] T. Baba, R. Matsuoka, S. Ono, K. Shirai, and M. Okuda, “Flash/noflashimage integration using convex optimization,” in Proc. IEEE Int. Conf.Acoust. Speech Signal Process. (ICASSP), 2014, pp. 1194–1198.
[13] R. Morita, K. Shirai, and Y. Tanaka, “Retargeting pyramid using directdecimation,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process.(ICASSP), 2014, pp. 2813–2817.
[14] L. Condat, “A primal-dual splitting method for convex optimizationinvolving Lipschitzian, proximable and linear composite terms,” J.Optimization Theory and Applications, vol. 158, no. 2, pp. 460–479,2013.
[15] H. W. Engl, M. Hanke, and A. Neubauer, Regularization of inverseproblems, Springer, 1996.
[16] R. M. Hestenes and E. Stiefel, “Methods of conjugate gradients forsolving linear systems,” J. Research of the National Bureau of Standards,vol. 49, pp. 409–436, 1952.
[17] R. Deriche, “Fast algorithms for low-level vision,” IEEE Trans. PatternAnal. Mach. Intell., vol. 12, no. 1, pp. 78–87, 1990.
[18] L.J. van Vliet, I.T. Young, and P.W. Verbeek, “Recursive Gaussianderivative filters,” in Proc. Int. Conf. Pattern Recognition (ICPR), 1998,vol. 1, pp. 509–514.
[19] K. Sugimoto and S. Kamata, “Fast Gaussian filter with second-ordershift property of DCT-5,” in Proc. IEEE Int. Conf. Image Process.(ICIP), Sept. 2013, pp. 514–518.
