Independent feature analysis for image retrieval

Jing Peng a, Bir Bhanu b,*

a Department of Computer Science, Oklahoma State University, Stillwater, OK 74078, USAb Center for Research in Intelligent Systems, University of California, Bourns Hall A220, Riverside, CA 92521, USA

Abstract

Content-based image retrieval methods based on the Euclidean metric expect the feature space to be isotropic. They

su�er from unequal di�erential relevance of features in computing the similarity between images in the input feature

space. We propose a learning method that attempts to overcome this limitation by capturing local di�erential relevance

of features based on user feedback. This feedback, in the form of accept or reject examples generated in response to a

query image, is used to locally estimate the strength of features along each dimension while taking into consideration

the correlation between features. This results in local neighborhoods that are constricted along feature dimensions and

that are most relevant, while elongated along less relevant ones. In addition to exploring and exploiting local principal

information, the system seeks a global space for e�cient independent feature analysis by combining such local infor-

mation. We provide experimental results that demonstrate the e�cacy of our technique using both simulated and real-

world data. Ó 2001 Elsevier Science B.V. All rights reserved.

Keywords: Feature analysis; Image retrieval

1. Introduction

The rapid advance in digital imaging technolo-gy makes possible the widespread use of image li-braries and databases. This in turn demandse�ective means for access to such databases. It hasbeen well documented that simple textual anno-tations for images are often ambiguous and inad-equate for image database search. Thus, retrievalbased on image ``content'' becomes very attractive(Cox et al., 1996; Flickner et al., 1995; Ma andManjunath, 1996; Minka and Picard, 1997; Penget al., 1999; Rui et al., 1997). Generally, a set of

features (color, shape, texture, etc.) are extractedfrom an image to represent its content. As such,image database retrieval becomes a K nearestneighbor (K-NN) search in a multidimensionalspace de®ned by these features under a givensimilarity metric.

Simple K nearest neighbor search, as an imageretrieval procedure, returns the K images closest tothe query. Obviously this involves the issue ofmeasuring the closeness or similarity between twoimages. The most common measure of the simi-larity between two images, represented by theirfeature vectors x and y, is the distance be-tween them. If the Euclidean distance D�x; y� �����������������������������Pq

i�1�xi ÿ yi�2q

is used, then the K closest imagesto the query xQ are computed according tofx j D�x; xQ�6 dKg, where dK is the Kth orderstatistic of fD�xi; xQ�gN

i�1. Here N is the number of

www.elsevier.nl/locate/patrec

Pattern Recognition Letters 22 (2001) 63±74

* Corresponding author. Tel.: +1-909-787-3954; fax: +1-909-

787-2425.

E-mail addresses: jpeng@cs.okstate.edu (J. Peng), bhanu@-

cris.ucr.edu (B. Bhanu).

0167-8655/00/$ - see front matter Ó 2001 Elsevier Science B.V. All rights reserved.

PII: S 0 1 6 7 - 8 6 5 5 ( 0 0 ) 0 0 1 0 0 - 8

images in the database. The major appeal forsimple K-NN search methods resides in theirability to produce continuous and overlappingrather than ®xed neighborhoods, and to use adi�erent neighborhood for each individual queryso that, to the extent possible, all points in theneighborhood are close to the query.

One problem with the Euclidean metric, how-ever, is that it does not take into account thein¯uence of the scale of each feature variable inthe distance computation. Changing the scale of afeature dimension in di�erent amounts alters theoverall contribution of that dimension to thedistance computation, hence its in¯uence in thenearest neighbors retrieved. This is usually con-sidered undesirable. An additional limitation isthat the use of Euclidean distance, while simplecomputationally, implies that the input featurespace is isotropic. However, isotropy is often in-valid and irrelevant features might hurt retrievalperformance. Finally, feature relevance dependson the location where the query is made in theinput feature space. Capturing such relevanceinformation is a prerequisite for constructingsuccessful retrieval procedures for image data-bases.

In this paper, we propose a novel method thatprovides a solution to the problems discussedabove. With this method image retrieval system isable to learn di�erential feature relevance in ane�cient manner that takes into account the cor-relation between features, and that is highlyadaptive to query locations. In addition, by accu-mulating experience obtained at each iteration thesystem is capable of, to the extent possible, con-tinuously improving its retrieval performance.

2. Previous work

Minka and Picard (1997) select features basedon feedback from user but all features are treatedwith equal importance. Ma and Manjunath (1996)use hybrid neural networks to partition the featurespace into clusters but no user feedback is used tore®ne retrievals and all features are treated asequally important. Rui et al. (1997) in their MARSsystem use a simple query shifting mechanism that

attempts to improve retrieval performance byadaptively moving the input query toward relevantretrievals and, at the same time, away from irrel-evant ones. Similarity computation remains ®xedthroughout the retrieval process. While MARS hasbeen shown to improve retrieval performance insimple tasks, it is clear that in many problems themere shifting of the query is insu�cient to achievedesired goals. PicHunter (Cox et al., 1996) is asystem based on Bayesian relevance feedback.PicHunter de®nes a set of actions that a user mighttake and attempts to estimate the probabilities ofthe actions the user will take. Based on actualactions taken, the system estimates the probabilityof each image in the database being the target. Amajor concern with PicHunter is that probabilityestimates rely heavily on simplifying assumptionsthat are often invalid in practice. Furthermore, itsimage similarity measure is nonadaptive. Peng etal. (1999) use probabilistic feature relevancelearning for content-based image retrieval thatcomputes ¯exible metrics for producing retrievalneighborhoods that are elongated along less rele-vant feature dimensions and constricted alongmost in¯uential ones. The technique has shownpromise in a number of image database applica-tions (Peng and Bhanu, 1999). It, however, be-comes less appealing in situations where featurerelevance can only be captured by examining sev-eral feature variables simultaneously. In this pa-per, we develop techniques that attempt toovercome this limitation.

3. Local feature relevance

In a two class (1/0) classi®cation problem, theclass label y 2 f0; 1g at query x is treated as arandom variable from a distribution with theprobabilities fPr�1 j x�; Pr�0 j x�g. We then havef �x��: Pr�y � 1 j x� � E�yjx�. To predict y at x,f �x� is ®rst estimated from a set of training datausing techniques based on regression, such as theleast-squares estimate. The Bayes classi®er canthus be applied to achieve optimal classi®cationperformance. In image retrieval, however, the``label'' of x is known, which is 1 (positive image)in terms of the notation given above. All that is

64 J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74

required is to exploit di�erential relevance of theinput features to image retrieval. In the absence ofany variable assignments, the least-squares esti-mate for f �x� is E�f � � R f �x�p�x�dx, where p�x�is the joint density. Now given only that x isknown at dimension xi � zi. The least-squares es-timate becomes E�f j xi � zi� �

Rf �x�p�x j xi � zi�

dx. Here p�x j xi � zi� is the conditional density ofthe other input variables.

In image retrieval, f �z� � 1, where z is thequery. Then ��f �z� ÿ 0� ÿ �f �z� ÿ E�f j xi � zi��� �E�f j xi � zi� represents a reduction in error be-tween the two predictions. We can now de®ne ameasure of feature relevance at query z asri�z� � E�f j xi � zi�. The relative relevance can beused as a weighting scheme. The following expo-nential weighting scheme:

wi�z� � exp�Tri z�� �Xq

l�1

,exp�Trl�z�� �1�

is employed in this paper. Here T is a parameterthat can be chosen to maximize (minimize) thein¯uence of ri on wi. A discussion on the choice ofa value for T can be found in (Peng et al., 1999).From (1), we obtain the weighted Euclidean dis-

tance D�x; y� ����������������������������������Pq

i�1 wi�xi ÿ yi�2q

.

In order to estimate (1), one must ®rst computeE�f j xi � zi�. The retrieved images with relevancefeedback from the user can be used as trainingdata to obtain estimates for E�f j xi � zi�, hence(1). Let fxj; yjgK

1 be the training data, where xj

denotes the feature vector representing the jth re-trieved image, and yj is either 1 (positive image) or0 (negative image) marked by the user as the classlabel associated with xj. Since there may not beany data at xi � zi, the data from the vicinity of xi

at zi are used to estimate E�y j xi � zi�, a strategysuggested in (Friedman, 1994). That is,

E y j xi� � zi� �PK

j�1 y1j j xji ÿ zi j 6Xÿ �PK

j�1 1 j xji ÿ zi j 6Xÿ � ; �2�

where 1��� is 1 if its argument is true, and 0 oth-erwise. X can be chosen so that there are su�cientdata for the estimation of E�f j xi � zi�. In thispaper, X is chosen such that

XK

j�1

1 j xji

ÿ ÿ zi j6X� � C; �3�

where C6K is a constant. It represents a trade-o�between bias and variance and should be chosen tolie between 1 and K.

4. Feature decorrelation using local principal infor-

mation

In order for Eq. (1) to be e�ective, features mustbe independent. However, this condition canhardly be met in practice. Often, there is a degreeof correlation among the input features. Our goalhere is to seek a space into which to project data sothat the feature dimensions coincide with the ei-genvectors of the space, whereby feature relevancecan be estimated at the query along individualdimensions independently. The novelty here is inchoosing the space using pooled local principalinformation.

We begin with linear multivariate analysis.Given a set of q-dimensional data fxjgn

j�1, the ei-genspace is the space spanned by eigenvectors ontowhich the feature dimensions representing theprojected data are statistically uncorrelated. Let �xdenote the mean for the observed data. Then thisspace corresponds to the eigenspace of the datapoints xj ÿ �x, or the local covariance matrix at thequery.

In the context of image retrieval, the basic ideais to compute xj ÿ �x at a given query using onlylocal information, and then perform the principalcomponent analysis for the local scatter matrix.Speci®cally, let fxj�i�gn

1 be the set of n nearestneighbors obtained at query i, and �x�i� be the as-sociated mean vector. Also let

S�i� � 1

n

Xn

j�0

xj�i��

ÿ �x�i��

xj�i��

ÿ �x�i��t

�4�

be the scatter matrix at query i, where t denotestranspose. We compute the space spanned by theeigenvectors of S�i� (for query i) onto which thefeature relevance analysis of the projected data canbe performed independently along each dimension.

J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74 65

The overall algorithm is summarized in Fig. 1.Here K denotes the number of images returned tothe user, and M is an adjustable procedural pa-rameter. In general, M � K. We call this algo-rithm adaptive feature relevance estimation(AFRE). Note that n is the number of data pointsused to compute the local scatter matrix, whereasM represents the number of data points projectedonto the eigenspace within which feature relevancecomputation is carried out.

The algorithm just described computes the localscatter matrix S�i� for each given query i. As such,it is capable of computing a neighborhood that ishighly adaptive to query locations, thereby sig-ni®cantly improving retrieval performance. On theother hand, a potential weakness of the method isthat the estimated covariance matrix centered atthe given query might be far di�erent from the truecovariance matrix that characterizes actual datadistributions due to the nature of computation,which is based solely on limited local samples. Weseek a method here to overcome this limitation by®nding a space that is close to the eigenspace of theaverage local scatter matrices, S�i�s, over all que-ries.

If we denote by U an orthonormal basis for theq-dimensional space, we can obtain the space byminimizing the following total residual sum ofsquares

R�U� �XN

i�1

Xn

j�1

�~xtj�i�~xj�i� ÿ ~xt

j�i�UU t~xj�i��;

where ~xj�i� � �xj�i� ÿ �x�i��, n is the number of lo-cal retrievals, and N is the number of queries. Wecan obtain the minimum of the above equation bymaximizing trU tfPN

i�1 S�i�gU , where tr representsthe trace operation of a matrix. This can be solvedby ®nding the eigenvectors of the following matrix�S �PN

i�1 S�i�=N , which is the average scatter ma-trices over all queries. The main bene®t of aver-aging is that it reduces variance due to skewed localinformation, thereby improving overall estimationaccuracy. If we add �S � �S � �S�i� ÿ �S�=�l� 1�;l � l� 1 to Fig. 1, immediately after step 4, andreplace S�i� by �S in step 5, we obtain the learningfeature relevance estimation (LFRE). Here both �Sand l are initialized to zero.

While LFRE demonstrates improvements overAFRE on the problems we examine here, it is still

Fig. 1. The AFRE algorithm.

66 J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74

a rather computationally intensive process. Wepropose to approximate the average scatter matrixwithout compromising the level of achievableperformance. The basic assumption is that �S canbe reasonably estimated from a few representativelocal S�i�s. Speci®cally, we approximate �S by in-crementally combining the local S�i�'s computedfor the queries seen so far, similar to the way it iscomputed in LFRE. However, the estimationprocess stops when �S becomes su�ciently accu-rate. Subsequent feature relevance estimates arecarried out in the space spanned by the eigenvec-tors of �S. In this paper, we measure the accuracyof �S using a matrix norm m. That is, we say that �Sis accurate if m��St�1 ÿ �St� is su�ciently small.While other measures exist, we do not intend toaddress this issue further in the rest of this paper.

If we replace line 4 in AFRE (Fig. 1) by thefollowing, and S�i� by �S in line 5, we arrive at theapproximate learning feature relevance estimation(A-LFRE) algorithm (Fig. 2). Here �S0 is initializedto 0, m��� represents the matrix norm operator, andd is a constant parameter input to the algorithm.

5. Empirical results

In the following we compare three competingretrieval methods using both real and simulateddata. (a): The probabilistic feature relevancelearning (PFRL) algorithm (Peng et al., 1999)coupled with the exponential weighting scheme (1).Note that this algorithm, unlike the ones describedabove, computes local feature relevance in the

Fig. 2. The A-LFRE algorithm.

Fig. 3. A simple two class problem with substantial within class covariance between the two input features.

J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74 67

original feature space. It does not perform featuredecorrelation. (b): The AFRE algorithm describedabove. Again, the algorithm is coupled with theexponential weighting scheme (1). (c): The LFREalgorithm described above. Similar to (a) and (b)above the algorithm is coupled with the exponen-tial weighting scheme (1). Note that there is afourth method, the simple (unweighted) K-NNmethod, that is being compared against implicitly.The ®rst retrieval by all the three methods is basedon the unweighted K-NN method. Also, in all theexperiments, the performance is measured usingthe following average retrieval precision (Peng etal., 1999; Rui et al., 1997):

precision � Positive retrievals

Total retrievals� 100%:

In all the experiments the input features are ®rstnormalized. The normalization is performed alongeach feature dimension over the entire data set insuch a way that the normalized feature values liebetween 0 and 1. It simply removes some of theartifacts due to di�erent scales of variables thatare generally considered undesirable in the ab-sence of any additional information. The numberof retrieved images, K, at each iteration is set to 20that provide relevance feedback. As far as therelative performance of AFRE, LFRE and PFRLis concerned, the choice of K will not alter thequalitative behaviors of the three techniques ob-served in the databases, provided that K is not toosmall.

5.1. Simulated data experiments

5.1.1. The problemIn the simulated data experiments we use a two

class problem with substantial within class co-variance between the two features, as shown inFig. 3. There are 250 data points in each class. Thisproblem clearly favors algorithms that ®rst rotatethe feature dimensions so that they coincide withthe eigenvectors of a sample covariance matrix,and then perform feature relevance analysis. Inthese experiments, each data point is selected as aquery, and the average retrieval precisions by thethree competing methods are reported.

5.1.2. ResultsIn these experiments, the procedural parameters

T (1), C (3), n and M input to the algorithms undercomparison were determined empirically thatachieved the best performance. They were set to 12and 25 (PFRL, parameters n and M are not ap-plicable to PFRL), 12, 22, 80 and 400 (AFRE),and 14, 24, 80 and 400 (LFRE), respectively. Fig.4(a) plots the performance of the three algorithms:PFRL, AFRE and LFRE as a function of itera-tion. Both AFRE and LFRE demonstrate per-formance improvement over PFRL, as expected,and more so by LFRE because of its averaginge�ect.

Fig. 4. E�ect of feature decorrelation on retrieval. (a) Perfor-

mance of PFRL, AFRE and LFRE. (b) Performance of A-

LFRE.

68 J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74

We carried out an additional experiment to ex-amine the performance of the A-LFRE algorithm(Fig. 2) on the same problem shown in Fig. 3. Inthis experiment, 400 data points were uniformly

randomly selected, each of which was used as aquery. This process was repeated 10 times, and theaverage retrieval precisions by LFRE and A-LFREwere reported in Fig. 4(b). Note that in these ®rst

Fig. 5. Sample images from MIT database.

J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74 69

experiments, we compared the results achieved by a®xed number of updates to �S, as in the A-LFREalgorithm, against that achieved by a total update,as in the LFRE algorithm. The results show that

the signi®cant level of performance can be achievedby A-LFRE with fewer �S updates. The proceduralparameters used in this experiment were 14 (T), 24(C), 80 (n) and 400 (M) for both algorithms.

Fig. 5. (Continued)

70 J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74

5.2. Real data experiments

5.2.1. Database 1The data, taken from the UCI repository

(Murphy and Aha, 1995), consist of images thatwere drawn randomly from a database of sevenoutdoor images. The images were hand segmentedby the creators of the database to classify eachpixel. Each image is a region. There are sevenclasses: brickface, sky, foliage, cement, window,path and grass, each having 330 instances. Thus,there are totally 2310 images in the database.These images are represented by 19 real valuedattributes. These features are basically statisticalmoments and line counts. For further details, seeMurphy and Aha (1995).

5.2.2. Database 2The data are obtained from MIT Media Lab. 1

There are total 640 images of 128� 128 in thedatabase with 15 classes. The number of images ineach class varies from 16 to 80. The images in this

database are represented by 8 Gabor ®lters (2scales and 4 orientations), giving rise to a 16-di-mension feature space. The mean and S.D. of themagnitude of the transform coe�cients are used asfeature components after being normalized by thestandard deviations of the respective features overthe images in the entire database. Fig. 5 showssample images from the MIT dataset.

5.2.3. ResultsFor both the problems, each image in the dat-

abase is selected as a query and top 20 (corre-sponding to parameter K in the algorithmsdescribed above) nearest neighbors are returnedthat provide necessary relevance feedback. Notethat only negative images (that are di�erent fromthe query) need to be marked in practice. Also, Mwas set to 400 in AFRE and LFRE in these ex-periments. The average retrieval performance bythe three competing algorithms is plotted in Fig. 6.

The ®rst iteration in Fig. 6 shows the averageretrieval precision obtained without any relevancefeedback. That is, it is the result of applying thesimple K-NN method using unweighted Euclideanmetric. The second iteration and beyond show the

Fig. 5. (Continued)

1 Whitechapel.media.mit.edu/pub/VisTex.

J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74 71

average retrieval precision obtained after learninghas taken place. That is, relevance feedback ob-tained from the previous retrieval is used to esti-mate local feature relevance, hence a newweighting. The procedural parameters T (1), C (3)and n input to the algorithms under comparisonwere determined empirically that achieved the bestperformance. They are by no means exhaustive.For the UCI image database, they were set to 13and 19 (PFRL, parameter n is not applicable toPFRL), 13, 21 and 200 (AFRE), and 13, 27 and200 (LFRE), respectively; while for the MIT imagedatabase, they were set to 13 and 20 (PFRL, againn is not applicable to PFRL), 15, 19 and 50(AFRE), and 14, 19 and 70 (LFRE), respectively.

It can be seen from Fig. 6 that LFRE demon-strates performance improvement across the tasksand over both PFRL and AFRE. However, theimprovement is most pronounced on the MIT dataset. The reason is that features based on Gaborwavelet ®lters exhibit a degree of correlation be-cause these ®lters partially overlap. On the otherhand, the features representing the UCI data setare less correlated. Overall, the results show con-vincingly that feature decorrelation plays an im-portant role in improving feature relevanceestimates.

An additional experiment was also carried outto examine the performance of the A-LFRE algo-rithm (Fig. 2). In this experiment, 600 images arerandomly chosen from the MIT database as queryimages. We ran both LFRE and A-LFRE on thisdatabase and obtained the average retrieval preci-sions on the query sets. We repeated this processfor 10 times, and plotted the average precisionsover the 10 runs at iterations 1, 2, 3, 4 and 5 in Fig.7. Note that instead of computing a matrix norm tomeasure the accuracy of �S, as a ®rst approximationA-LFRE simply computes a ®xed number of up-dates to �S, after which �S is ®xed throughout.

The plots in Fig. 7 show that after only a fewupdates of the average scatter matrix A-LFREapproached the level of performance obtained byLFRE. Furthermore, A-LFRE did so with far lesscomputation than that required by LFRE, therebydemonstrating its computational e�ciency andadvantage in practical applications. We performedsimilar experiments on the UCI database, where2000 images are randomly selected as query im-ages. We omit the details of the experiments exceptstating that similar results to that of the MITdatabase were obtained.

5.2.4. DiscussionsOne might contemplate to use a covariance

matrix computed from a set of samples to decor-relate the entire database o�-line, and then toperform feature relevance estimate in the trans-formed feature space using the techniques pre-sented in this paper. There may be several reasonsagainst such an idea. The most important one iswhen the database is dynamic, then an o�-lineoperation may not be feasible. In one experiment

Fig. 6. E�ect of feature decorrelation on retrieval. (a) Perfor-

mance of PFRL, AFRE and LFRE on the UCI data set. (b)

Performance of PFRL, AFRE and LFRE on the MIT data set.

72 J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74

the entire UCI data set is used to compute thescatter matrix. We then project the original datainto the eigenspace of the matrix and performcorresponding feature relevance estimates. Weobtained the following average retrieval precisionsat iterations 1, 2, 3, 4 and 5: 91.14, 94.99, 96.07,96.70 and 96.86, respectively. These results andthose obtained earlier (the UCI data set in Fig.6(a)) clearly favor the on-line techniques proposedin this paper. We obtained similar results on theMIT database.

Two databases used here have a relatively smallnumber of data in each class. The e�ectiveness ofAFRE and LFRE in databases where each classmay contain a large number of images dependscritically on the e�ectiveness of PFRL (Peng et al.,1999) in such databases, since both AFRE andLFRE can be viewed as a special case of PFRLthat computes feature relevance in the eigenspaceof a scatter matrix. For a detailed empirical eval-uation of PFRL in such databases, please see Pengand Bhanu (1999).

6. Conclusions

This paper presents a novel method for learningthe di�erential feature relevance for a given query

that takes into consideration the correlation be-tween features. In addition to exploring and ex-ploiting local discriminant information, the systemseeks a global space for e�cient independent fea-ture analysis by combining such local information.Furthermore, by accumulating experience ob-tained at each iteration the system is capable of, tothe extent possible, continuously improving itsretrieval performance. The experimental resultspresented demonstrate the potential for substan-tial improvements in both the technique presentedhere and simple K-NN search.

Acknowledgements

This work was supported in part by DARPA/AFOSR grant F49620-97-1-0184 at the Universityof California at Riverside. The contents of theinformation do not necessarily re¯ect the positionor the policy of the US Government.

References

Cox, I., Miller, M., Omohundro, S., Yianilos, P., 1996.

PicHunter: Bayesian relevance feedback for image retrieval.

In: Proceedings of the International Conference on Pattern

Recognition. Vienna, Austria, pp. 361±369.

Fig. 7. Performance of the A-LFRE algorithm on the MIT database.

J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74 73

Flickner, M., et al., 1995. Query by image and video content:

the QBIC system. IEEE Computer, pp. 23±31, September

1995.

Friedman, J.H., 1994. Flexible metric nearest neighbor classi-

®cation. Technical Report, Department of Statistics, Stan-

ford University.

Ma, W.Y., Manjunath, B.S., 1996. Texture features and

learning similarity. In: Proceedings of the IEEE Computer

Society Conference on Computer Vision and Pattern

Recognition. San Francisco, pp. 425±430, June 1996.

Minka, T.P., Picard, R.W., 1997. Interactive learning with

a ``society of models''. Pattern Recognition 30 (4), 565±

581.

Murphy, P.M., Aha, D.W., 1995. UCI repository of machine

learning databases. www.cs.uci.edu/�mlearn/MLReposito-

ry.html..

Peng, J., Bhanu, B., Qing, S., 1999. Probabilistic feature

relevance learning for content-based image retrieval. Com-

puter Vision and Image Understanding 75 (1/2), 150±164.

Peng, J., Bhanu, B., 1999. Feature relevance estimation for

image databases. In: Proceedings of the Fifth International

Workshop on Multimedia Information Systems, pp. 12±19.

Rui, Y., Huang, T.S., Mehrotra, S., 1997. Content-based image

retrieval with relevance feedback in MARS. In: Proceedings

of the IEEE International Conference on Image Processing.

Santa Barbara, California, October 1997, pp. 815±818.

74 J. Peng, B. Bhanu / Pattern Recognition Letters 22 (2001) 63±74