Available online at www.sciencedirect.com

www.elsevier.com/locate/patrec

Pattern Recognition Letters 29 (2008) 637–646

An active feedback framework for image retrieval

Tao Qin a,1, Xu-Dong Zhang a, Tie-Yan Liu b,*, De-Sheng Wang a,Wei-Ying Ma b, Hong-Jiang Zhang b

a Department of Electronic Engineering, Tsinghua University, Beijing 100084, PR Chinab Microsoft Research Asia, No. 49 Zhichun Road, Haidian District, Beijing 100080, PR China

Received 17 January 2006; received in revised form 29 April 2007Available online 15 December 2007

Communicated by R. Manmatha

Abstract

In recent years, relevance feedback has been studied extensively as a way to improve performance of content-based image retrieval(CBIR). Since users are usually unwilling to provide much feedback, the insufficiency of training samples limits the success of relevancefeedback. In this paper, we propose two strategies to tackle this problem: (i) to make relevance feedback more informative by presentingrepresentative images for users to label; (ii) to make use of unlabeled data in the training process. As a result, an active feedback frame-work is proposed, consisting of two components, representative image selection and label propagation. For practical implementation ofthis framework, we develop two coupled algorithms corresponding to the two components, namely, overlapped subspace clustering andmulti-subspace label propagation. Experimental results on a very large-scale image collection demonstrated the high effectiveness of theproposed active feedback framework.� 2007 Elsevier B.V. All rights reserved.

Keywords: Active learning; Image retrieval; Clustering; Relevance feedback

1. Introduction

The success of content-based image retrieval (CBIR) isgreatly limited by the gap between low-level features andhigh-level semantics. In order to reduce this gap, relevancefeedback has been introduced from the domain of textualdocument retrieval. Relevance feedback iteratively refinesthe retrieval results by learning from user-labeled examples.Although relevance feedback is an effective approach, itsuffers from the fact that users do not like to label toomany images, even if this is helpful to improve the retrieval

0167-8655/$ - see front matter � 2007 Elsevier B.V. All rights reserved.

doi:10.1016/j.patrec.2007.11.015

* Corresponding author. Tel.: +86 10 62617711; fax: +86 10 62555337.E-mail addresses: qinshitao99@tsinghua.edu.cn (T. Qin), zhangxd@

tsinghua.edu.cn (X.-D. Zhang), tyliu@microsoft.com (T.-Y. Liu),wangdsh_ee@tsinghua.edu.cn (D.-S. Wang), wyma@microsoft.com(W.-Y. Ma), hjzhang@microsoft.com (H.-J. Zhang).

1 This work was performed when the author was an intern at MicrosoftResearch Asia.

accuracy. As a result, the examples we could get during thefeedback process are very limited.

To cope with this problem, we propose the followingtwo approaches in this paper: (i) to make user’s feedbackmore informative by presenting representative images tothe users (the definition of ‘‘representative images” willbe given in Section 4). In such a way, labeled exampleswill contain more information. (ii) To leverage unlabeleddata in the training phase, the number of which couldbe much more than that of the few labeled images. Corre-spondingly, in this paper, an active feedback framework isproposed, with two novel components named representa-tive image selection and label propagation. In particular,we further develop two algorithms, i.e., overlapped sub-space clustering and multi-subspace label propagation torealize these two components in this paper. It is noted thatthese two algorithms are not independent, but are highlycoupled and can be jointly optimized. Experimental resultson a very large-scale image collection demonstrated the

Query Submission

Single or Multiple examples

Retrieval

Output the ranked list for all images in the database

RepresentativeImage Selection

Overlapped SubspaceClustering

User LabelingRelevant/non-relevant

Label Propagation

Multi -SubspaceLabel Propagation

LearningAny supervised learningmethods. In our implementation,Rank SVM is used

Fig. 1. Proposed active feedback framework for CBIR.

638 T. Qin et al. / Pattern Recognition Letters 29 (2008) 637–646

high effectiveness of the proposed active feedbackframework.

The rest of this paper is organized as follows. Section 2reviews related work on relevance feedback based CBIR.The new active feedback framework for CBIR is presentedin Section 3. The technical details of two new units (repre-sentative image selection and label propagation) of theframework are given in Sections 4 and 5. In Section 6,experimental results are reported to show the effectivenessof the proposed active feedback framework. Concludingremarks are given in Section 7.

2. Related work

The early relevance feedback algorithms for CBIR,which were borrowed from the field of textual documentretrieval, include query refinement (Rui et al., 1998) andre-weighting (Rui et al., 1998). Rui and Huang (1999) com-bined these two approaches to minimize the total distancebetween the positive examples and the refined query pointwith a refined similarity metric. However, because the posi-tive images may distribute dispersively in the feature space,it is difficult to retrieve them directly based on low-levelfeature similarity, either refined or not.

To overcome the disadvantages of the early relevancefeedback algorithms, statistical learning technologieshave been applied in recent years. The representativeworks include Bayesian inference (Su et al., 2001;Vasconcelos and Lippman, 1999), Boosting (Tieu andViola, 2000) and support vector machines (SVM) (Chenet al., 2001; Jing et al., 2003; Tong and Chang, 2001).Due to its clear mathematical formation and wellfounded theory, SVM has attracted wide attention inthe literature. So we also take SVM as an example toillustrate our proposed framework. The proposed meth-odologies, however, can be applied to Boosting andBayesian inference as well. Chen et al. (2001) estimatedthe distribution of positive examples with a one-classSVM, and returned images with largest probabilities asrelevant images. They avoided estimating the distributionof the negative examples which is very complex and dif-ficult to model. Tong and Chang (2001) proposed aSVM-based active learning scheme. They provided theusers with the most informative images to label anduse SVM to learn a hyperplane that separates the posi-tive examples from the negative ones. The most informa-tive images in their paper are those images closest to theclassification boundary, for which SVM has lowest confi-dence. Such a data selection strategy is reasonable andmay lead to faster convergence of relevance feedback.Experiments in (Jing et al., 2003) showed that two-classSVM as a classifier outperforms one-class SVM as a dis-tribution estimator for image retrieval.

Although the statistical learning algorithms have beenproved to be effective, their successes in CBIR are limited.The key problem is that the training samples (user’s labels)are often not sufficient to ensure the performance of the

learning machine. As pointed out by Donoho (2000), moreor less the learning theory for data analysis is based on theassumption of D < N (where D is the feature dimensionand N is the number of samples). However, in the case ofimage retrieval, training samples are often much smallerthan the dimension of features. In other words, we face atypical ‘‘insufficient training sample” problem.

To improve the performance of relevance feedback, wehave to address this issue. There have been some meaning-ful attempts in this direction. Wu et al. (2000) tried to solvethis problem with transduction method. Transduction(Vapnik, 1998) adopts a discriminative model, and maxi-mizes its margins on both labeled and unlabeled data, pro-vided that the labeled samples are classified as correctly aspossible. The disadvantage of their work is that to find theoptimal decision boundary requires solving a mixed integerprogramming problem that is NP-complete. Chang et al.(2003) suggested enlarging the training set by recursive sub-space co-training (Lai and Chang, 2002). They providedeach training sample set with distinct subspace views toboost the pool of the negative examples. However, thismethod cannot handle positive examples.

3. Active feedback framework

To address the problem of insufficient training exam-ples, we add two new processing units to the traditional rel-evance feedback pipeline, so as to formulate a new activefeedback framework, which is shown in Fig. 1.

The four units, query submission, retrieval, user labelingand learning, are inherited from the previous works. Thetwo new units, representative image selection and labelpropagation are our key contributions in this paper. Theadvantages of this framework include:

(1) Benefiting from representative images selection, a fewrepresentative images are delivered to users for label-ing. It can not only make the labeling work of users

T. Qin et al. / Pattern Recognition Letters 29 (2008) 637–646 639

bring most information but also keep the mass oflabeling tasks very little. It is an effective and efficientway to label images.

(2) Since the relationship among images can be easilyobtained, we can propagate the labels from thelabeled set to some unlabeled images (not all the unla-beled data). By doing so, we expand the training setand make the obtained classifier robust to the noisyunlabeled data, since we only use the high-qualityunlabeled data instead of all the unlabeled data. Notethat such a strategy is very different from (Wu et al.,2000), which makes use of all the unlabeled images,and is thus sensitive to noise.

In fact, from a broader point of view, some previousmethods can also be classified into these two units. Forexample, the most positive and most informative schemesin SVM active learning (Tong and Chang, 2001) may beconsidered as two methods for representative imagesselection. Transduction (Wu et al., 2000) and co-training(Chang et al., 2003; Lai and Chang, 2002) algorithms areboth aiming at utilizing the unlabeled images. However,the differences between our approaches and those worksare: (i) they have not formulated explicit concepts ofeither representative images selection or label propaga-tion. (ii) In their philosophy, the representative imagesselection and unlabeled data integration are consideredisolated. In contrast, in our framework, these two unitsare not independent of each other. Actually they are clo-sely interconnected and we try to treat them from a glo-bal-optimization viewpoint.

As the other components have been extensively studiedin previous works, in the following two sections, we focuson the two new components.

4. Representative image selection

In this section, we first give the formal definition ofrepresentative image. Then, we present the process ofrepresentative image selection, which consists of two sub-phases: feature subspace partitioning and overlapped sub-space clustering.

4.1. Representative images

When the training set is small, the training performanceis very sensitive to the effectiveness of each training exam-ple. That is, the statistical characteristics of the labeledimages will highly affect the performance of the CBIR sys-tem. On one hand, if these images are too similar to eachother, there will be too much redundancy which decreasesthe information capacity; while on the other hand, if thereare little consistency among them, the learning algorithmwill encounter great difficulty in training a reasonable clas-sifier. In order to make the training more smoothly, weshould provide some representative images for user label-ing, which should have the following two properties.

(i) The images should have consistency. Here, the con-sistency means that these images should have similarbehavior in training the classifier.

(ii) The images should not contain too much redun-dancy.

To guarantee the first property, we select representativeimages from a sub image set with consistent characteristicsinstead of the whole image database. We call this sub set as‘‘estimated possibly positive image set (EPPIS)”. That is,EPPIS contains a subset of images, which are most likelyrelevant to the query. Note that EPPIS is a dynamic imagecollection, which will change after each iteration of user’sfeedback. In the beginning of retrieval, EPPIS containsthose images nearest to the query point (with some distancemeasurement). After the learning machine is trained, it willbe used to test all the samples in the whole image collec-tion. Only those images classified to be positive with highconfidence will be included in the next-round EPPIS.

To guarantee the second property, we introduce the fol-lowing definitions first.

Definition 1 (Element–set distance). Given a finite setX = {x1, x2, . . ., xN} with N elements in it, for a subsetY = {y1, y2, . . ., yM} of X, the distance between someelement x 2 X and Y is defined as

dðx;YÞ ¼ minyi2Y

dðx; yiÞ ð1Þ

where d(x, yi) is the element–element distance.

For d(x, yi), we can adopt any distance metrics in theoriginal feature space, such as Euclidean (Carson et al.,1999; Rui et al., 1997), Minkowski (Swain and Ballard,1991; Voorhees and Poggio, 1998) and quadratic (Hafner,1995; Niblack, 1993) distances; or use kernel functions,such as Gaussian, polynomial and sigmoid kernels (Burges,1998). We believe it should be better to choose different dis-tances than one deterministic distance for differentapplications.

In the view of information theory, Definition 1 can beexplained as follows. If we use an element in Y to takethe place of x, the minimal information loss will bed(x, Y). In other words, if we treat Y as a code book,d(x, Y) displays the residue when using Y to encode x.

Definition 2 (Set–set distance). Given a set X = {x1, x2, . . .,xN} and the element–set distance, for two subsetsY, Z � X, the distance from Y to Z is defined as

dðY;ZÞ ¼Xy2Y

dðy;ZÞ ð2Þ

Note that the set–set distance is not symmetric:d(Y, Z) 6¼ d(Z, Y). This is easy to understand: if Y � Zand Y 6¼ Z, we have d(Y, Z) = 0 while d(Z, Y) > 0. Simi-larly, d(Y, Z) displays the information loss when encodingY with the code book Z. Actually d(Y, Z) is a decreasingfunction of Z: if Z1 � Z2, d(Y, Z1) P d(Y, Z2). Intuitive

Table 1Feature subspace partitioning algorithm

Input: EPPIS X = {x1, x2, . . ., xN}, in which xi is a NF-dimensionalvector

Output: partitioning of NF features(i) Quantize each feature into several bins, and compute the v2 sta-

tistic between any two features so as to get an NF � NF matrix.Initially, treat each feature as a subspace;

(ii) If there exist features in two subspaces which are dependent,merge these two subspaces;

(iii) Go to (ii) until the remained subspaces can no longer be merged.And then output the final subspace partitioning

640 T. Qin et al. / Pattern Recognition Letters 29 (2008) 637–646

explanation is that the bigger the code book is, the less theinformation loss will be.

With the above two definitions, representative image set,which is the collection of representative images in CBIR,can be defined as below.

Definition 3 (Representative image set). The NR-elementrepresentative image set R of EPPIS is

R ¼ arg minYfdðEPPIS;YÞjY � EPPIS;NY ¼ NRg ð3Þ

where NY is the number of elements in Y.

That is to say, we choose a subset of EPPIS as the rep-resentative image set, to which EPPIS has the smallestset–set distance. From the view of coding theory, the repre-sentative image set is the best code book with NR elementsfor EPPIS, and it has the minimum information loss toencode EPPIS.

4.2. Feature subspace partitioning

According to Definition 3, if one first partitions theEPPIS into NR clusters, the centroid of each cluster willbe the representative image. In the following discussions,we use f(�) to indicate a specific clustering algorithm.

As well known, image features are of high dimensions.When a user searches the database, his/her focuses on dif-ferent feature subspaces are not equal to each other. Insome cases, color may be the dominant subspace, whilein some other cases shape may be more important. Forexample, blue color is the dominant subspace when usersearches for sky images; and shape is the dominant sub-space when user searches for cars images. To better modeluser’s retrieval behavior, we firstly partition the image fea-tures into subspaces, give each subspace different weight,and then select representative images for each subspaceseparately. Here, our assumption for subspace partitioningis that the features in a same subspace should share somestatistical consistency. To model this, we treat every feature(e.g. one dimension of color moment) as a discrete randomvariable (taking values over the whole EPPIS). Since mostimage features are not discrete, we quantize them beforeadopting K. Pearson v2 statistic (Mario, 2003) to test thestatistical dependency between any two features. Specifi-cally, for two features F1 and F2,

v2ðF 1; F 2Þ ¼X

f

Xg

� ðP ðF 1 ¼ f ; F 2 ¼ gÞ � P ðF 1 ¼ f ÞP ðF 2 ¼ gÞÞ2

P ðF 1 ¼ f ÞP ðF 2 ¼ gÞð4Þ

If v2(F1, F2) is small than 3.84 (as widely used in the litera-ture, Mario, 2003), they are regarded as dependent and putinto the same subspace.

The details of feature subspace partitioning algorithmare given in Table 1 (where NF is the dimensionality ofimage features):

As aforementioned, the different subspace may not beequally important. We introduce the concept of subspaceweight to address this point. Suppose there are totally N

subspaces {C(n)}, where n = 1, 2, . . ., N. L(n) is the set con-taining the projections of all the positive examples (labeledby user feedback) on C(n). If L(n) is empty, set all the sub-space weights X(n) = 1/N; else calculate the subspace weightof C(n) by

XðnÞ ¼ 1

cðnÞ

�XN

k¼1

1

cðkÞð5Þ

where cðnÞ ¼P

x2LðnÞ;y2LðnÞ ðdðnÞðx; yÞÞ2; and dðnÞð; Þ denotes

the element–element distance metric for subspace C(n).Note that the weight of a subspace displays the attentionthe user pays on this subspace. Bigger the weight, the userhas more interest in this subspace.

4.3. Overlapped subspace clustering

After partitioning the feature space and calculating theweight for each subspace, we can cluster images in eachsubspace. The number of clusters for each subspace is pro-portional to its weight. Here we show an example. Supposewe get two feature subspaces with weight 0.6 and 0.4 sepa-rately, and we want to select 10 representative images fromthe EPPIS. Firstly, we partition EPPIS into six clusters inthe first subspace and four clusters in the second subspace.Secondly, the image nearest to the centroid of each clusteris selected as representative images.

The detailed clustering algorithm is shown in Table 2.Note that if we cluster images in each subspace indepen-

dently, we may select a same representative image from twodifferent subspaces. To avoid this problem, in step (iii) wedo clustering from subspace associated with the largestweight to subspace associated with the smallest weight.Suppose the subspace C(1), C(2), . . ., C(N) is ordered by des-cent weight. Starting from C(1), suppose we have alreadyselected representative image set for C(n) (denoted byR(n)). Then for C(n+1), after a representative image isselected, it will be projected back onto all C(m) (m 6n + 1) to see whether it is close enough (with the ele-ment–element distance d(m)(,)) to any representative imagesin R(m). If so, delete it and split the cluster for C(n+1) withthe largest average element–element distance into twonew clusters and update the representative images set

Table 2Overlapped subspace clustering algorithm

Input: EPPIS, partitioning C(1), C(2), . . ., C(N) of the whole featurespace and the corresponding weight X(1), X(2), . . ., X(N) for eachfeature subspace

Output: representative image set(i) Sort N subspaces by weight in a descent order: X(i1) P

X(i2) P ,. . ., P X(iN).(ii) Allocate the cluster number for each subspace according to its

weight. The total cluster number equals K, the number ofimages for users to label in each iteration. Roughly speaking,the cluster number for C(in) will be [X(iN)K] (where [x] is the inte-ger part of x). It is possible that

PNin¼1½XðinÞK� < K. In such case

the extra ðK �PN

in¼1½XðinÞK�Þ clusters are assigned to C(i1). In

such a way, we get the final assignment of cluster numbers{K(in)}, n = 1, 2, . . ., N.

(iii) For each subspace C(in), use clustering algorithm f(�) to generateK(in) clusters and select the representative images set R(in).

(iv) Get the final representative image set R ¼SN

n¼1RðnÞ:

T. Qin et al. / Pattern Recognition Letters 29 (2008) 637–646 641

accordingly. In this case, there will be K(n+1) + 1 clustersfor C(n+1), but the number of representative images is stillK(n+1). This process continues until the selection of R(n+1)

becomes stable.To summarize, the main idea of our algorithm is to

select the representative images through different subspac-es: (i) subspaces with different user attention are assignedwith different weights, thus represented by different numberof clusters and images; (ii) clusters for different subspacesare overlapped in sense that an image can belong to differ-ent clusters in different subspaces; (iii) the selected imagescan represent their cluster well in sense of Definition 3;(iv) the representative images are not too close to eachother in any subspace. In such a way, this algorithm cancapture the user attention and handle the nonidentityamong different subspaces well.

In fact, the output of the above clustering process is notonly some images for user labeling, but also the basis oflabel propagation, which will be introduced in the nextsection.

5. Label propagation

After the clustering process, a set of representativeimages are selected and returned to the users for their feed-backs. As a result, each of the representative images will geta label. In this section, we discuss how to propagate theselabels to the whole EPPIS set based on the clusters gener-ated by overlapped subspace clustering. This process canhelp to solve the insufficient training sample problem byenlarging the training set. Firstly, we introduce the generalconcept of label propagation; and then we propose themulti-subspace label propagation algorithm.

Fig. 2. Structure of multi-subspace label propagation.

5.1. Concept of label propagation

Although there are many ways for the user to supply hisfeedback in literature, such as goodness/badness, ranking

and explicit (Ortega-Binderberger and Mehrotra, 2003),empirical studies have shown that users typically give verylittle feedback and that the flexibility of multiple levels ofrelevance is too burdensome (Jansen et al., 2000). As aresult, the most popular mode for the user to label imagesconverged to the binary approach: an image is either rele-vant (positive) or not (negative). Maybe we can regard bin-ary labels on the representative images as reasonable;however, it will not be suitable to do with the unlabeledimages in the same deterministic manner. Our idea here isto estimate a fuzzy relevance r 2 (�1, +1) for the unlabeledimages based on the binary labels (+1 and �1) of thelabeled images. More specifically, (i) only if labeled imageL and unlabeled image U have some kind of similarity,should we propagate the label of L to U; (ii) user’s atten-tion may focus on different subspaces of an image for dif-ferent queries, so the propagation should be carried outsubspace-wise. Based on these ideas, we propose the‘‘multi-subspace label propagation” algorithm.

5.2. Multi-subspace label propagation

The proposed algorithm is built on top of overlappedsubspace clustering method. Its structure is shown inFig. 2. There exist multiple paths to propagate from alabeled image L to an unlabeled image U. These propaga-tion paths are in different subspaces and summed with cor-responding subspace weights. Specifically, in the path ofC(n), whether the label of L will be propagated to U

depends on the relationship between their projections onC(n). Only if their projections are in the same cluster, wewill use the following mechanism to propagate the label.

Suppose the projections of L and U (denoted by L(n) andU(n)) on C(n) are both in the ith cluster c

ðnÞi , and the set

LðnÞi and U

ðnÞi contain the projections of all the labeled

and unlabeled images that fall in cðnÞi respectively. Then,

we have LðnÞ 2 LðnÞi and U ðnÞ 2 U

ðnÞi . L’s propagation

Table 3Multi-subspace label propagation algorithm

Suppose there are totally N subspaces {C(n)}, n = 1, . . . , N. For eachC(n), K(n) clusters are generated: c

ðnÞi ; i ¼ 1; . . . KðnÞ. L

ðnÞi contains all

the projections of the labeled images on C(n) that fall in cðnÞi . Then,

the estimated relevance for an unlabeled images U (whose projectionon C(n) is U(n)) is

rðUÞ ¼PN

n¼1

PKðnÞ

i¼1

Pl2L

ðnÞi

XðnÞ � xðnÞi ðlðnÞ;U ðnÞÞ � RðlÞ� �

ð7Þ

where R(l) is the binary label of the representative image whoseprojection on C(n) is l(n)

642 T. Qin et al. / Pattern Recognition Letters 29 (2008) 637–646

influence (called element weight) on U in this subspace pathis determined by

xðnÞi ðLðnÞ;U ðnÞÞ ¼1

dðnÞðLðnÞ;U ðnÞÞPu2U

ðnÞi

1dðnÞðLðnÞ;uÞ

ð6Þ

Let the element weight between two images whose projec-tions are not in the same cluster be zero, the above processcan be formulated as in Table 3.

We would like to point out the following two propertiesof our algorithm:

1. Labels can be propagated through different subspacepaths. Hence, the propagation may happen more thanonce between two images.

2. There is the case that an unlabeled image is propagatedwith a positive relevance in one subspace path, whilewith a negative relevance in another. For example, whena user wants to find the images with white flowers, anunlabeled image with red flowers may get a positive rel-evance by the subspaces of texture and shape but nega-tive relevance by the color subspaces. This is reasonableand can help to capture user’s attention.

After propagating the labels to each unlabeled image inEPPIS, both labeled and unlabeled, will be used to trainthe retrieval engine. Because the relevance of an unlabeledimage is not binary but distributed in (�1, 1), rank-SVM(Joachims, 2002) algorithm is adopted in our paper to ful-fill the corresponding training task. Since the unlabeledimages in EPPIS are much more than the labeled ones,to remove the fear that the labeled images are overwhelmedby unlabeled images, we only use those unlabeled imageswhose relevance r satisfies |r| > 0.3 for training in ourexperiments.

With the proposed clustering, propagation and rank-SVM, we have developed an effective approach to solvethe insufficient training sample problem in relevance feed-back. Experiments in the next section showed that ourapproach can improve the retrieval performance by much.

2 In fact, our additional experiments show that which clusteringalgorithm we used does not lead to much difference of the final retrievalaccuracy.

6. Experiments

In this section, we tested the effectiveness of proposedactive feedback framework. Firstly, we describe the bench-mark of our experiments; secondly, comprehensively exper-

iments on a collection with 5000 images are introduced;thirdly, we give an intuitive example to show the advantageof our framework; at last, we report the performance of theproposed active feedback framework on a large collectionwith more than 60,000 images.

6.1. Experiment setup

To avoid the image collection bias, we used two subsetsof COREL as our benchmark. There are totally 50 catego-ries in the first subset with 100 images in each category.There are 542 categories with total 60,196 images in thesecond subset, and each category has 50–150 images. Sincethe first subset is relatively small, we investigated the per-formance of the two new units of our framework compre-hensively on this subset. We only test the performance ofthe whole framework on the second subset.

In our automatic testing, the category label serves as theground truth: the images in the same category of the queryare treated as relevant/positive images, which is the samewith previous works.

For feature selection, we totally used 384 image features:256 HSV histogram, 9 Luv moment, 104 wavelet texturedescriptors (Chang and Kuo, 1993) and 15 MRSAR(Mao and Jain, 1992) descriptors.

We use the retrieval precision in each step of iteration asthe performance measurement, defined as the percentage ofthe positive images in all the retrieval results. That is

Precision ¼ relevant images retrieved in top N returns

Nð7Þ

6.2. Results on the first image collection

For this subset, an exhaustive test scheme was adopted:all the 5000 images were used as query and five feedbackswere simulated. The average performance among all these5000 images was calculated to indicate the performanceof the retrieval algorithms. For the clustering algorithm,we adopted K-mean with Euclidean distance metric forsimplicity.2 For all the SVM methods used in this sub sec-tion, we adopted the Gaussian kernel with default param-eter setting in SVMlight (Joachims, 1999).

In the first experiment, we tested the impact of EPPIS

size on the performance of the proposed algorithms.The top-20 retrieval accuracy with varying EPPIS size

was listed in Fig. 3. From the results, we could get thefollowing conclusions: (i) the performance of our algo-rithm was not sensitive to the EPPIS size. While theEPPIS size increased from 50 to 200, the retrieval perfor-mance only changed less than 2%. (ii) Just as discussed in

Fig. 3. Performance comparison with different EPPIS sizes.

Fig. 5. Comparison of the retrieval systems with and without labelpropagation.

T. Qin et al. / Pattern Recognition Letters 29 (2008) 637–646 643

Section 4, if the EPPIS size was too small or too large,the performance would drop, although not much. (iii)The best setting of the EPPIS size in Fig. 3 is about100 images. Considering this is a top-20 case, we usedthe EPPIS size of 5 � K (K is the number of the labeledimages in each iteration) in the following experiments(including Sections 6.2 and 6.3) while comparing to otherreference algorithms.

In the second experiment, we tested the added-values ofthe two new units in our framework. First we inserted therepresentative images selection unit to a SVM-based rele-vance feedback framework. The comparison algorithmsincluded (Tong and Chang, 2001; Jing et al., 2003), alsoSVM-based frameworks. The main difference was that weselected the representative images by overlapped subspaceclustering (OSC) while they selected the most informativeand the most positive images for the users to label. Theaverage top-20 accuracy was shown in Fig. 4. From this fig-ure, we can see that our method was about 5% more accu-rate than the most informative selection scheme and 10%better than most positive scheme after five iterations. Thatis to say, the representative image selection unit (with OSCalgorithm) is effective.

Fig. 4. Comparison of different image selection schemes.

To test the label propagation unit, we used the followingtwo retrieval schemes. The first one was ‘‘OSC + UserLabeling + SVM”, while the second was ‘‘OSC + UserLabeling + MSLP (multi-subspace label propaga-tion) + rank-SVM”. The average top-20 performance waslisted in Fig. 5. We found that MSLP improved the

Fig. 6. Retrieval accuracy (top-20).

Fig. 7. Retrieval accuracy (top-30).

Fig. 8. The final results of SVM-MP.

644 T. Qin et al. / Pattern Recognition Letters 29 (2008) 637–646

retrieval performance by another 5%, validating the effec-tiveness of label propagation.

In the third experiment, we tested the overall perfor-mance of the proposed active feedback framework. Forsimplicity, we named our framework by ‘‘SCLP (subspaceclustering and label propagation)” in following descrip-tions. We compared our SCLP framework with severalprevious works, including the most informative SVM(SVM-MI) (Tong and Chang, 2001), the most positiveSVM (SVM-MP) (Jing et al., 2003) and Rui’s re-weightingmethod (Rui et al., 1998). Figs. 6 and 7 showed the resultsof the average top-20 and top-30 retrieval accuracy respec-tively. For all cases, both SVM-MP and SVM-MI hadhigher performance than Rui’s method, while SCLP alwaysperformed the best. For the top-20 case, SCLP outper-formed SVM-MP and SVM-MI by about 10%; for thetop-30 case, SCLP leaded to more than 13% higheraccuracy.

6.3. An intuitive example

In previous subsection, we investigated the performanceof the framework by statistical average of many queries. Inthis subsection, we will have a look at the final retrievalresults of a specific query to give some intuitive impressabout the framework.

Fig. 8 showed the final results of SVM-MP after fiveiterations for a query ‘‘lion”, which was in the red solidbox3 of the figure. As seen, the two images in blue dashed

3 For interpretation of color in Fig. 8, the reader is referred to the webversion of this article.

box were irrelevant to the query. However, since the lowlevel visual feature of these two images were very similarto other relevant images, they were returned after fiveiterations.

Fig. 9 showed the final results of our framework (SCLP)after 5 iterations for the same query in the red solid box.4

All the returned images were relevant to the query. Com-paring with Fig. 8, we could see that the two images in bluedashed box were a little different with other images: theyhad green background. Because of the green background,they were not very closed to other relevant images, andso SVM-MP could not treat them as relevant images. Onthe contrary, the two algorithms in our framework werebased on subspace, and they could discover the similaritybetween these two images and the query. That is, ourframework could find their similarity in texture subspaceand their similarity in green color subspace (since the queryimage also had some green background). So the multi-sub-space label propagation algorithm could propagate thepositive label to these two images from subspaces, andretrieve them as relevant images. However, since the mostpositive images in SVM-MP were very similar to each otherand the query, SVM-MP algorithm could not retrieve theimages with similarity only in some subspaces.

6.4. Results on the second image collection

For this subset with 60,196 images, we randomlyselected 1000 images as queries and 5 feedbacks for each

4 For interpretation of color in Fig. 9, the reader is referred to the webversion of this article.

Fig. 9. The final results of our active feedback framework.

Table 5Relative improvement of SCLP

Iteration Top-20 Top-30

Improvementover SVM-MP (%)

Improvementover SVM-MI (%)

Improvementover SVM-MP (%)

Improvementover SVM-MI (%)

1 5.69 5.69 10.53 10.532 11.87 11.18 17.89 16.673 12.13 11.05 17.89 18.854 10.85 10 15.38 17.655 10.82 9.03 12.23 14.67

T. Qin et al. / Pattern Recognition Letters 29 (2008) 637–646 645

query were simulated. Similarly to the previous subsection,the average performance among these 1000 images was cal-culated to measure the performance of the retrieval algo-rithms. We also adopted K-mean with Euclidean distancemetric for clustering and Gaussian kernel for SVM.

Since Rui’s method (Rui et al., 1998) performed not sowell as other methods in Section 6.2, in this subsectionwe only compared our active feedback framework withthe most informative SVM (SVM-MI) (Tong and Chang,2001) and the most positive SVM (SVM-MP) (Jing et al.,2003).

Table 4 showed the results of the average top-20 andtop-30 retrieval accuracy. For the top-20 case, our activefeedback framework performed best, with 3.3% improve-ment over SVM-MP and 2.8% improvement over SVM-MI after five iterations. For the top-30 case, our frameworkperformed best again, with 2.3% improvement over SVM-MP and 2.7% improvement over SVM-MI after fiveiterations.

Due to the large scale of this image collection, one cansee that the retrieval accuracies of all these algorithms were

Table 4Retrieval accuracy

Iteration Top-20 Top-30

SCLP SVM-MP SVM-MI SCLP SVM-MP SVM-MI

0 0.115 0.115 0.115 0.071 0.071 0.0711 0.130 0.123 0.123 0.084 0.076 0.0762 0.179 0.160 0.161 0.112 0.095 0.0963 0.231 0.206 0.208 0.145 0.123 0.1224 0.286 0.258 0.260 0.180 0.156 0.1535 0.338 0.305 0.310 0.211 0.188 0.184

much lower than those reported in the previous subsection.Correspondingly, the absolute improvements are also smal-ler on this collection. However, if we have a look at the rel-ative improvement of SCLP as shown in Table 5, we canfind that SCLP still greatly outperformed SVM-MP andSVM-MI.

To summarize, for our proposed framework, it is easy toselect internal parameters because the retrieval perfor-mance is not very sensitive to the EPPIS size. Both thetwo new units contribute to the whole performance of theactive feedback framework. Tested on two general datasets, our framework was proved to have higher retrievalaccuracy than those reference algorithms examined in thispaper.

7. Conclusions

In this paper, an active feedback framework has beenproposed to handle the insufficient training sample

646 T. Qin et al. / Pattern Recognition Letters 29 (2008) 637–646

problem for content based image retrieval. In this frame-work, two new units, representative image selection (withoverlapped subspace clustering algorithm) and label prop-agation (with multi-subspace label propagation algorithm)were developed. As the two algorithms share the idea ofsubspace partitioning and clustering, they can not onlyhandle the insufficient training sample case, but also cap-ture the user’s attention on specific sub feature spaces whenretrieving the image database. Tested on general large scaleimage databases, the framework has demonstrated verypromising retrieval accuracy.

Acknowledgement

Special thanks should be given to Guang Feng, BinGao, Qiankun Zhao, Huimin Yan and Huaiyuan Yangfor their sincerely helps.

References

Burges, C., 1998. A tutorial on support vector machines for patternrecognition. Data Min. Knowl. Discov. 2 (2), 121–167.

Carson, C., Thomas, M., Belongie, S., Hellerstein, J.M., Malik, J., 1999.Blobworld: A system for region-based image indexing and retrieval. In:Proc. 3rd Internat. Conf. on Visual Information and InformationSystem (VISUAL’99), Amsterdam, The Netherlands, June 1999.

Chang, E., Li, B., Wu, G., Goh, K.S., 2003. Statistical learning foreffective visual information retrieval. In: Proc. IEEE Internat. Conf.on Image Processing (ICIP’03), Barcelona, September 2003, pp.609–612.

Chang, T., Kuo, C.-CJ., 1993. Texture analysis and classification withtree-structured wavelet transform. IEEE Trans. Image Process. 2 (4),429–441.

Chen, Y., Zhou, X.S., Huang, T.S., 2001. One-class SVM for learning inimage retrieval. In: Proc. IEEE Internat. Conf. on Image Processing(ICIP’2001), Thessaloniki, Greece, October 7–10, 2001, pp. 815–818.

Donoho, D.L., 2000. High-dimensional data analysis: The curses andblessings of dimensionality. American Math Society Lecture – MathChallenges of the 21st Century.

Hafner, J. et al., 1995. Efficient color histogram indexing for quadraticform distance functions. IEEE Trans. Pattern Anal. Machine Intell. 17(7), 729–736.

Jansen, B.J., Spink, A., Saracevic, T., 2000. Real life, real users and realneeds: A study and analysis of users queries on the web. Inform.Process. Manage. 36 (2), 207–227.

Jing, F., Li, M.J., Zhang, H.J., Zhang, B., 2003. Support vector machinesfor region-based image retrieval. In: Proc. IEEE Internat. Conf. onMultimedia & Expo.

Joachims, T., 1999. Making large-scale SVM learning practical. In:Scholkopf, B., Burges, C., Smola, A. (Eds.), Advances in KernelMethods – Support Vector Learning. MIT-Press.

Joachims, T., 2002. Optimizing search engines using click through data.In: Proc. ACM Conf. on Knowledge Discovery and Data Mining(KDD), ACM, pp. 133–142.

Lai, W.C., Chang, E., 2002. Hybrid learning schemes for multimediainformation retrieval. In: Proc. IEEE Pacific Rim Conf. on Multi-media, December 2002, pp. 556–563.

Mao, J.C., Jain, A.K., 1992. Texture classification and segmentation usingmultiresolution simultaneous autoregressive models. Pattern Recogni-tion 25 (2), 173–188.

Mario, F.T., 2003. Elementary Statistics, Ninth Edition.Niblack, W., et al., 1993. Querying images by content, using color, texture,

and shape. In: Proc. SPIE Conf. on Storage and Retrieval for Imageand Video Database, vol. 1908, April 1993, pp. 173–187.

Ortega-Binderberger, M., Mehrotra, S., 2003. Relevance feedback inmultimedia databases. Handbook of Video Databases: Design andApplications. CRC Press.

Rui, Y., Huang, T.S., Ortega, M., Mehrotra, S., 1998. Relevancefeedback: A power tool in interactive content-based image retrieval.IEEE Trans. Circ. Syst. Video Technol. 8 (5), 644–655.

Rui, Y., Huang, T.S., 1999. A novel relevance feedback technique in imageretrieval. In: Proc. 7th ACM Conf. on Multimedia. 1999, pp. 67–70.

Rui, Y., Huang, T.S., Mehrotra, S., 1997. Content-based image retrievalwith relevance feedback in MARS. In: Proc. Internat. Conf. on ImageProcessing, pp. 815–818.

Su, Z., Zhang, H J., Ma, S., 2001. Relevant feedback using a Bayesianclassifier in content-based image retrieval. In: Proc. SPIE ElectronicImaging 2001, San Jose, CA.

Swain, M.J., Ballard, D.H., 1991. Color indexing. Internat. J. Comput.Vision 7 (1), 11–32.

Tieu, K., Viola, P., 2000. Boosting image retrieval. In: Proc. IEEE Conf.on Computer Vision and Pattern Recognition, pp. 228–235.

Tong, S., Chang, E., 2001. Support vector machine active leaning forimage retrieval. In: Proc. 9th ACM Conf. on Multimedia, Ottawa,Canada, pp. 107–118.

Vapnik, V., 1998. Statistical Learning Theory. Wiley.Vasconcelos, N., Lippman, A., 1999. Learning from user feedback in

image retrieval systems. In: Proc. 13th Conf. on Neural InformationProcessing Systems (NIPS’99).

Voorhees, H., Poggio, T., 1998. Computing texture boundaries fromimages. Nature 333, 364–367.

Wu, Y., Tian, Q., Huang, T.S. 2000. Integrating unlabeled images forimage retrieval based on relevance feedback. In Proc. 15th Internat.Conf. on Pattern Recognition (ICPR’2000), vol. I, pp. 21–24.