Jmrnal of Systems Engineering and Electronics, Vol.16, No.4, 2005, pp. 728-732

Midtisensor image fusion algdrithm using nonseparahle wavelet frame transfonn'̂

LiZhenhtui, Jing Zhongliang, Wang Hong & Sun Shaoyuan Inst, d Aerospace Infonnation and Contid, Schcwl of Electronic Information and Electrical Engineering,

Shanghai Jiaotong Univ., Shanghai 200030, P. R. China

(Received June 1, 2004)

Abstract: A imiltisenscn' image fusion algorithm is described uang 2-diniensional nonseparable wavelet frame (NWF)

transfcxm. The source multisenscM" images are first decomposed by the NWF transfcnm. Then, the NWF transfcam coef-

ficients ci the source images are combined into the composite NWF transform coefficients. Inverse NWF transfcnm is per-

fonned on the composite NWF transfain coefficients in order to obtain the intermediate fused image. Finally, intoisity

ac^ustinent is applied to the intermediate fused image in order to maintain the dynamic intensity range. Experimait re-

sults using real data show that the proposed algraithm works well in multisoisor image fusion.

Key words: multisensor, image fusion, image processing, iKxiseparable wavelet frame transform.

1· lNTR(H>UCnON

With the rapid improvement of sensor technology,

numerous multisensor data, which often contain c»m-

plementary and redundant information about the re-

gion surveyed, are obtained in many fields such as re-

nK>te sensing, medical imaging, machine vision and

military applications. Sensor fusion is increasingly be-

coming a prcanising research area. It can be divided

into signal, pixel, feature, and symbol levels. The

paper mainly addresses the problem of pixel level im-

age fusicxi. Through combining registered images

generated by different imaging systems, image fusion

can produce fused images that are more suitable for

the purposes of human vision perception, object de-

tection and automatic target recognition.

Multiresolution decomposition is widely used in

multisensor image fusion. Mtiltiresolution decomposi-

tion mainly includes pjn-amid transform^ ̂ '̂ ^ and

wavelet transform^^^^^. A pyramid structure is an ef-

ficient way to implement multiscale representation.

Each image in a pyramid is a low-pass filtered and

subsampled cc^y of the previous images. Wavelet

transform can also deocmpose a signal into several

components, each of which captures information pre-

sent at a given scale. According to the separable

2-dimensbnal ( D ) multiresolution analysis theory

(MRA), the discrete wavelet transform (DWT) of

an image used in many fusion schemes can break

down into 1-D wavelet decomposition on rows and

columns respectively. In practice, for 2-D signal

f(x,y)^:L^(R^), it carmot be processed separately

in X and y directions in most cases. The 2-D separa-

ble wavelet transform also yields a shift variant data

representation by the downsampUng process and is

not appropriate for multisensor image fusion. Non-

separable wavelet'̂ ^^ allows true processing of images.

Images are treated as areas instead of rows and

columns. The advantage of nonseparable wavelet is

having better frequency characteristics, directional

properties and more degree of freedom, resulting in

better design. By eliminating the decimator and in-

terpolator process and changing the filter coefficients

of nonseparable wavelet transform, we will obtain

nonseparable wavelet frame (NWF) transform"^^ .̂ All

the sub-bands after the decomposition using the NWF

transform will have the same size as the source im-

age. The NWF transform has the properties of shift-

invariance and aliasing free. In this paper, we

develop an approach based on the NWF transform for

the fusing of multisensor images. Experimental re-

sults indicate that the proposed method outperforms

* This prc^eav^^suppc^ed by the National Natural Sdence Foundation of China ( ω Specialized Research Fund for the Doctoral Program of Higher Education (20020248029); China Aviation Science Foundation (02D57003); Aerospace Si4)pOTting Technofcgy Foundation (2003 - 1. 3 0 2 ) ; EXPO Technologies Special Prcject of National Key TechncJogies R&D Procramme (2004BA908B07).

Multisensor image fusion algorithm using nonseparable wavelet frame transform 729

the approaches based on the pyramid transform and

the DWT transform.

2 . THE 2-D NWF TRANSFORM

The implication of nonseparable wavelet transform'̂ ^^

is similar to 1-D case. The low-pass component is re-

peatedly filtered and sub-sampled resulting in another

low-pass and another detail signal. However, sub-

sampling is not performed by retaining every second

colunm and row, as it is in the separable case, but

performed on a nonseparable sampling matrix. There

are many kinds of sampling matrixes. The sampling

matrix used in this paper is given by

The down-sampling process of an image / using the

sampling matrix D is defined as

/ | d ( K ) =f(D'K),K^Z^ (2)

where Ζ is the set of int^er; Κ is the pixel coordi-

nates; / ( · ) is the pixel intensity of image / .

The up-sampling process of an image / using the

sampling matrix D is defined as

/ ( D - i - K ) , i f K = D . / a n d / e z 2

0 , otherwise

(3)

By eliminating the resampling process and

changing the filter coefficients of nonseparable

wavelet transform, we will obtain the NWF trans-

form*-®̂ . The analysis and synthesis structure of the

NWF transform is shown in Fig. 1 . The NWF coeffi-

cients of an image are calculated as

lgi.i(K) = [Go]^d^MK)

i = 0 , 1 , - , N - 1 ( 4 )

where " * denotes the convolution between two sig-

nals; Ho and Gq are the 2-D analysis prototype fil-

ters; [ H q ] t D' and [Gq] t d ' are the dilated versions

of the low-pass filter Hq and the high-pass filter Gq;

/ o ( K ) = / ( K ) ; Ν is the total number of decomposi-

tion level. In Fig. 1 , Hi and Gi are the 2-D synthesis

prototype filters; [ Hi ] f d' and [ Gi ] f d' are the di-

lated versions of the low-pass filter Hi and the high-

pass filter G i .

fi

Fig. 1 The analysis and S3mthesis structure of the NWF transform

The design of the 2-D nonseparable filters is gen-

erally more difficult than the design of 1-D filters.

McClellan transformation'̂ ^^ has been shown to be a

useful technique for designing the 2-D nc»iseparable

filters. It allows t o transform a 1-D pΓotot)φe filter

into a 2-D zero phase FIR filters, and the 2-D filter

parameterized by the McQellan transformation has

the property of the 1-D protot5φe filter.

The NWF transform has two advantages: less

constraint on filters and translation invariance. After

Ν level NWF decomposition of an image f{x,y)y

we will get several high-pass sub-bands coefficients

\gi{x,y) \ i = \.,2,'"jN\ and one low-pass sub-

b a n d w h e r e \{x,y)\U.y)^Z^ is the

coordinates of image pixels. Each frequency band has

the same size as the source image.

3 . MULTISENSOR IMAGE FUSION USING

THE NWF TRANSFORM

In the later discussion, we use the symbol ^ix^y)

and / j ^ ( x , y ) , g f i x . j ' ) a n d / S ( x , y ) to denote

the NWF transform coefficients of the source images

A and Β respectively, and use the S5mibols gf (x ,3^)

a n d / n iocyy) todenc^e theoomposite NWF transfcnn

coefficients, v^ere 1 = 1 , 2 , — , N and {xyy)^Z^.

In Ref. [ 2 ] , a feature selection rule is used to

combine the gradient pyramid coefficients of the

source images to be fused. In this image fusicxi

scheme, we apply this feature selection rule to com-

bine the NWF coefficients of the source images.

Salient features are first identified in each source im-

age. The salience of a feature is computed as a local

energy in the neighborhood of a coefficient

E\{x,y)= Σ t t ; ( m , n ) · g f ( m , n ) ^ , ( m , f i ) 6 N ( x . > )

/ = A , B (5)

730 Li Zhenhua, Jing Zhongliang, Wang Hong & Sun Shaoyuan

where N(x,y) defines a window of the neighbor-

hood coefficients centered on the current coefficient

gi ( x ,3^). The size of the window is typically small,

i.e. 3 X 3. And w(m, n) is the weight satisfied

with Σ w ( m , w ) = 1 . (m,n)eN(x,y)

At a given resolution level ί , two modes of fus-

ing are used: selection and averaging. In order to de-

termine whether the selection or averaging mode will

be used, the match measure is calculated as

M^U.y) =

2 Σ Mm,n)'\ g^(m,n) l-l g f ( m , / i ) I

£ f ( x , y ) + £ f ( x , y )

(6)

Large match measure at a given position means that

the source coefficients are similar at that position,

vice versa. Mf^(x ,y) is compared to a threshold a.

If iVff^(x ,y) is less than or equal to a, then the co-

efficient with larger local energy is placed in the com-

posite transform while the coefficient with less local

energy is discarded. The selection mode is imple-

mented as

[ g f ( x , y ) , £ f ( x , y ) > E f ( x , y )

[ g f ( x , y ) , EtU,y)<EfU,y)

(7)

If Mf^ ( X . y ) is greater than a, the source coeffi-

cients in N ( x ,y) are similar and the weighted aver-

ages are calcidated from the coefficients of the source

images. In the averaging nKxie, the combined trans-

form coefficient is implemented as

^ ( x , y ) = a>f(x,y) · g f ( x , y ) +

a ) f ( x , y ) - g f ( x , y ) . (8)

where a/^(x ,y) and <of(x ,y) are the weights

\ω^, Et(x,y)^Ef(x,y)

a,f(x,y) = l-wHx,y) (9)

gfix,y) =

<^(x,y)

where (

For the fusion of the lowest frequency sub-bands

/ a and / β , we use the simple averaging method

J%(x,y) = 0.5 · f^(x,y) + 0.5 · f%(x,y)

(10)

After the ccwiposite NWF transform coefficients are

obtained, inverse NWF transform is performed to get

the intermediate fused image F ' . In order to ensure

the intensity range of the fused image is between 0 ~

255 (255 is the maximum intensity of a pixel in this

paper), finally we apply linear intensity transforma-

tion to modify the dynamic intensity range of the in-

termediate fused image. The pixel intensity in the

fused image F is defined as

max — mm

where max is the maximum intensity in image and

min is the minimtim intensity in image F ' .

4 . EXPERIMENT RESULTS AND CXJNCLIJSK^

The visual and infrared (IR) images (shown in Figs.

2(a) and 2 (b ) ) are used as the source images to be

fused for the experiment. We have compared our algo-

rithm to two traditicxial pyramid transform and discrete

wavelet transform based multisensor image fusicxi algo-

rithms^ '̂̂ .̂ Figure 2 shows the fuacei results.

Three objective evaluation methods are employed

to evaluate the performance of each image fusion algo-

rithm

(1) Entropy ( H )

Η = - I ] ^ F ( 0 l o g 2 M 0 (12)

where hf(i) is the normalized histogram of the fused

image F to be evaluated; L is the maximum value for

a pixel in the image, in our tests, L is equal to 255.

The entropy is used to measure the overall informa-

tion in the fused image. The larger the value is, the

better fusion results we get.

(2) Overall cross entropy (OCE)

OCE(A,B,F) = CE{A,F)^CE{B,F)

(13) where A and β are the source images; F is the fused image; C E ( A , F ) ( C £ ( B , F ) ) is the cross entropy of the source image A(B) and the fused image F

CEiA,F) = Σ / ι λ ( £ ) ί = 0

L

CE(B.F) = Σ / ΐβ(0

log2

log2

Mi) hpU)

hf(i)

(14)

(15)

where h^d), hsd) and hpd) are the normalized

histograms of the images A , β and F . The overall

cross entropy is used to measure the difference be-

tween the source images and the fused image. The

Multisensor image fusion algorithm using nonseparable wavelet frame transform 731

less the value is, the better fusion result we get. better fusion result we get. The spatial frequency is (3) Spatial frequency (SF) defined as Spatial frequency is used to measure the overall

. . , , , S F = / R F ^ + CF^ actmtylevel of an .mage. The larger the value is, the ^^ere RF is the row frequency

and CF is the column frequency

(16)

(17)

" VoT^ry! (Y - 1 ) Σ E (F (x 4 - 1 , ^ ) - Fu,y)) (18)

where X , Y are the width and height of the fused image F .

Quantitative evaluation is shown in Table 1. From this table, we can conclude that the perfor-

mance of our algorithm is better than that of these

traditional fusion algorithms'^^'^^. The proposed algo-rithm can be used to fuse images obtained by different types of sensors. Using our algorithm, most impor-tant information found in input images can be trans-ferred into the fused image.

(a) Visual image (b) IR image

( c ) Fused image by the pyramid

transform based algorithm'^'

(d) Fused image by the DWT

based algorithm'"

( e ) Fused image by

our algorithm

Fig. 2 Fusion results of the source visual and IR images

Table 1 Pnf ormance evaluatkm of

Η OCE SF

Algorithm in Ref. [2 ] 6.251 2 1.1011 7.789 7

Algorithm in Ref. [3 ] 6.356 I 1.531 9 9.388 9

Our algorithm 6.764 1 0.923 2 10.996 0

R E F E R E N C E S

^ 1 ] Toet A. Image fusion by a ratio of low-pass pyramid. Pat-

tern Recognition Letters , 1989. 9 (4 ) : 2 4 5 - 2 5 3 .

[2] Burt Ρ J. A gradient pyramid basis for pattern selective

image fusion. SID International Symposium, Boston,

Digest of Technical Papers. 1992: 4 6 7 - 4 7 0 .

[3J Η Li, Manjunath Β S, Mitra S K. Multi-sensor image fu-

sion using the wavelet transform. Graphical Models and

linage Processing. 1995, 57(3): 235^245.

[4] Varshney Ρ Κ, Chen Η Μ, Ramac LC, et al. Registration

and fusion of infrared and millimeter wave images for concealed

weapon detection. Pnx. of the IEEE International Confer-

ence on hnageProcessing, Japan, 1999, 3: 532—536.

[5] Chipman L J, Qrr TM. Wavelets and image fusion. Proc.

732 Li Zhenhua , Jing Zhongliang, Wang Hong & Sun Shaoyuan

of the IEEE International Conference on Image Process-

ing, Washington B.C., 1995 : 2 4 8 - 251.

[ 6 ] Chibani Y, Houacine A. On the use of the redundant

wavelet transfcMTn for multisensor image fusion. Proc. of

Int. Conf. on Electronics, Circuits and Systems, 2000:

4 4 2 - 4 4 5 .

[7 ] Kovacevic J, Vetterli M. Nonseparable two-and three-di-

mensional wavelets. IEEE Trans. on Signal Processing,

1995, 43 (5 ) : 1269-1273 .

[8 ] Pan J, Wang J. Texture segmoitation using sparable and

non-separable wavelet frames. lEICE Trans, on Funda-

mentaU, 1999, E 8 2 - A ( 8 ) : 1463-1474.

[9 ] Kovacevic J, Vetterii M. NonsQ)arable multidimensional

perfect reconstruction filter banks and wavelet bases for

RT. IEEE Trans, on Information Theory, 1992 , 38

( 2 ) : 5 3 3 - 5 5 5 .

Li Ztienliiia was bom in 1976. He received the B. S.

and M. S. degrees in control science and control engi-

neering from Shandong University in 2002. He is

currently ptirsuing the Ph. D. in control science and

control engineering at Shanghai Jiaotong University.

His research interests include multisensor data fu-

sion, image fusion, and pattern recognition.

Jing Zhongliang was bom in 1960. He received the

B. S . , M. S. and Ph. D. degrees from Northwestem

Polytechnical University, Xi'an, in 1983, 1988 and

1994, respectively, all in electronics and information

technology. His research interests are intelligent in-

formation processing, information fusion, target

tracking, stochastic neuro-fuzzy systems, high per-

formance motion control, and aerospace control and

information processing.

Wang Hong was bom in 1972. Now he is a Ph. D.

candidate in the Institute of Aerospace Information

and Control, School of Electronics and Information

Technology, Shanghai Jiaotong University. His re-

search interests include image fusion, mtiltiscale anal-

ysis and wavelet transform.

Sun Shaoyuan received M. S. and Ph. D. d ^ e e s

from the School of Elec. Eng. & Opto. Tech. , Nan-

jing University of Science and Technology, in 1999

and in 2002, respectively. And she is currendy a

px)stdoctorai fellow in the Institute of Aerospace In-

formation and Control, Shanghai Jiaotong Universi-

ty. She is interested in image processing, image and

data fusion, night vision and renxjte sensing.