IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014 1 Cost-effective features for re-identification in camera networks Syed Fahad Tahir and Andrea Cavallaro Abstract—Networks of smart cameras share large amounts of data to accomplish tasks such as re-identification. We propose a feature selection method that minimizes the data needed to represent the appearance of objects by learning the most appro- priate feature set for the task at hand (person re-identification). The computational cost for feature extraction and the cost for storing the feature descriptor are considered jointly with feature performance in order to select cost-effective good features. This selection allows us to improve inter-camera re-identification while reducing the bandwidth that is necessary to share data across the camera network. We also rank the selected features in the order of effectiveness for the task to enable a further reduction of the feature set by dropping the least effective features when application constraints require this adaptation. We compare the proposed approach with state-of-the-art methods on the i- LIDS and VIPeR datasets and show that the proposed approach considerably reduces network traffic due to inter-camera feature sharing while keeping the re-identification performance at an equivalent or better level compared with the state of the art. Index Terms—Smart camera networks, person re- identification, data reduction, cost of features, feature selection. I. I NTRODUCTION S MART camera networks are composed of nodes that perform image processing locally and aim to transfer the minimum amount of data over the network to accomplish collaborative tasks such as object detection and tracking [1], [2]. The challenges involved in smart camera networks include reducing the amount of data to be processed and shared across the network, real-time operation and energy efficiency [3], [4]. Data reduction can be achieved by using image com- pression [5], [6] and metadata reduction, which are generally obtained via trade-offs between data-transmission rate and distortions [2]. An important example of the use of metadata (features) is for describing objects such as people for their re-identification across the network [7]–[9]. Multiple object features are com- bined and shared among cameras to improve performance [8]– [16]. Gabor and Schmid filters that define two kernels applied to the luminance channel to extract textures may be used, The authors are with the Centre for Intelligent Sensing, Queen Mary Univer- sity of London, E1 4NS, London, UK. (email: [email protected]; [email protected]). S.F. Tahir was supported by the Erasmus Mundus Joint Doctorate in Interactive and Cognitive Environments, which is funded by the Education, Audiovisual & Culture Executive Agency (FPA n ◦ 2010-0012). This work was partially supported by the Artemis JU and UK Technology Strategy Board as part of the Cognitive & Perceptive Cameras (COPCAMS) project under GA number 332913. Copyright (c) 2014 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending an email to [email protected]. followed by learning and probabilistic approaches for feature optimization [14]–[17]. The histogram of the RGB [12], [13] and the HSV [10] color space are also used as features. Alternatively, the two chrominance channels of the YUV space are used [18] with a Gaussian Mixture Model to encode the most relevant color clusters. Features from multiple color spaces (RGB, YCbCr, and SV) and texture types (Gabor and Schmid) may be concatenated to increase their discriminative power [14]–[16]. However, after a certain number of feature concatenations, any additional features might decrease the re- identification performance. This problem can be addressed by feature selection. However, very little work has been done to consider the cost of a feature, such as, for example, the computational time for its extraction and the amount of data that is necessary for its storage. We argue that the cost of a feature should be considered jointly with its performance, namely its ability to represent and to discriminate an object for the task at hand. The cost constraint in feature selection becomes particularly important when the cost varies signifi- cantly across features to be shared among nodes of a smart camera network. In this paper, we propose a Cost-and-Performance-Effective (CoPE) feature selection method that combines the cost of using features with their performance in order to identify the most inexpensive feature subset for person re-identification in a smart camera network. To the best of our knowledge, the proposed approach is the first to apply explicit feature selection for re-identification while considering the cost of features. Instead of optimizing the combined contribution of the best set of features, the most discriminative, well-performing and cost-effective features are selected by evaluating each feature individually and then by ranking the selected features based on their contribution to the task, thus making the approach scalable in transmitting data over the network. CoPE is evaluated in the person re-identification task using Direct Distance Minimization (DDM). The results are compared with two existing re-identification approaches, namely PRDC [15] and ASFI [17], and five feature selection approaches: Fisher score [19], Information gain [20], mRMR [21], ReliefF [22] and Bi-clusters [23]. We also demonstrate that CoPE features further improve the performance of learning methods for re- identification with rankSVM [14] and AdaBoost [16] by reduc- ing the feature dimensions, the training time and by improving their effectiveness. The software of the proposed method is available at http://www.eecs.qmul.ac.uk/~andrea/software.htm. The paper is organized as follows. Section II discusses the related work on feature selection and the features used in person re-identification. Section III analyzes the performance
Transcript
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014 1
Cost-effective features forre-identification in camera networks
Syed Fahad Tahir and Andrea Cavallaro
Abstract—Networks of smart cameras share large amounts ofdata to accomplish tasks such as re-identification. We proposea feature selection method that minimizes the data needed torepresent the appearance of objects by learning the most appro-priate feature set for the task at hand (person re-identification).The computational cost for feature extraction and the cost forstoring the feature descriptor are considered jointly with featureperformance in order to select cost-effective good features. Thisselection allows us to improve inter-camera re-identification whilereducing the bandwidth that is necessary to share data acrossthe camera network. We also rank the selected features in theorder of effectiveness for the task to enable a further reductionof the feature set by dropping the least effective features whenapplication constraints require this adaptation. We comparethe proposed approach with state-of-the-art methods on the i-LIDS and VIPeR datasets and show that the proposed approachconsiderably reduces network traffic due to inter-camera featuresharing while keeping the re-identification performance at anequivalent or better level compared with the state of the art.
Index Terms—Smart camera networks, person re-identification, data reduction, cost of features, feature selection.
I. INTRODUCTION
SMART camera networks are composed of nodes thatperform image processing locally and aim to transfer the
minimum amount of data over the network to accomplishcollaborative tasks such as object detection and tracking [1],[2]. The challenges involved in smart camera networks includereducing the amount of data to be processed and shared acrossthe network, real-time operation and energy efficiency [3],[4]. Data reduction can be achieved by using image com-pression [5], [6] and metadata reduction, which are generallyobtained via trade-offs between data-transmission rate anddistortions [2].
An important example of the use of metadata (features) isfor describing objects such as people for their re-identificationacross the network [7]–[9]. Multiple object features are com-bined and shared among cameras to improve performance [8]–[16]. Gabor and Schmid filters that define two kernels appliedto the luminance channel to extract textures may be used,
The authors are with the Centre for Intelligent Sensing, Queen Mary Univer-sity of London, E1 4NS, London, UK. (email: [email protected];[email protected]). S.F. Tahir was supported by the ErasmusMundus Joint Doctorate in Interactive and Cognitive Environments, which isfunded by the Education, Audiovisual & Culture Executive Agency (FPA n2010-0012). This work was partially supported by the Artemis JU and UKTechnology Strategy Board as part of the Cognitive & Perceptive Cameras(COPCAMS) project under GA number 332913.
Copyright (c) 2014 IEEE. Personal use of this material is permitted.However, permission to use this material for any other purposes must beobtained from the IEEE by sending an email to [email protected].
followed by learning and probabilistic approaches for featureoptimization [14]–[17]. The histogram of the RGB [12], [13]and the HSV [10] color space are also used as features.Alternatively, the two chrominance channels of the YUV spaceare used [18] with a Gaussian Mixture Model to encodethe most relevant color clusters. Features from multiple colorspaces (RGB, YCbCr, and SV) and texture types (Gabor andSchmid) may be concatenated to increase their discriminativepower [14]–[16]. However, after a certain number of featureconcatenations, any additional features might decrease the re-identification performance. This problem can be addressed byfeature selection. However, very little work has been doneto consider the cost of a feature, such as, for example, thecomputational time for its extraction and the amount of datathat is necessary for its storage. We argue that the cost ofa feature should be considered jointly with its performance,namely its ability to represent and to discriminate an objectfor the task at hand. The cost constraint in feature selectionbecomes particularly important when the cost varies signifi-cantly across features to be shared among nodes of a smartcamera network.
In this paper, we propose a Cost-and-Performance-Effective(CoPE) feature selection method that combines the cost ofusing features with their performance in order to identify themost inexpensive feature subset for person re-identification ina smart camera network. To the best of our knowledge, theproposed approach is the first to apply explicit feature selectionfor re-identification while considering the cost of features.Instead of optimizing the combined contribution of the bestset of features, the most discriminative, well-performing andcost-effective features are selected by evaluating each featureindividually and then by ranking the selected features basedon their contribution to the task, thus making the approachscalable in transmitting data over the network. CoPE isevaluated in the person re-identification task using DirectDistance Minimization (DDM). The results are compared withtwo existing re-identification approaches, namely PRDC [15]and ASFI [17], and five feature selection approaches: Fisherscore [19], Information gain [20], mRMR [21], ReliefF [22]and Bi-clusters [23]. We also demonstrate that CoPE featuresfurther improve the performance of learning methods for re-identification with rankSVM [14] and AdaBoost [16] by reduc-ing the feature dimensions, the training time and by improvingtheir effectiveness. The software of the proposed method isavailable at http://www.eecs.qmul.ac.uk/~andrea/software.htm.
The paper is organized as follows. Section II discusses therelated work on feature selection and the features used inperson re-identification. Section III analyzes the performance
2 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014
and the cost of features, defines their combination strategy anddiscusses the implementation in multi-camera networks underconstrained environments. In Sec. IV, the proposed approachis validated and compared with existing feature selection andre-identification approaches using the challenging i-LIDS andVIPeR datasets. Finally, Sec. V draws conclusions.
II. RELATED WORK
In this section, we discuss relevant feature selection ap-proaches and the features used in the re-identification, whilehighlighting properties and inadequacies of current featureselection and re-identification approaches.
A. Feature selection
Feature selection aims at finding the most important featuresand their combinations for effectively describing and matchingobjects [24], [25]. Feature selection is an NP-hard problem.Approaches based on heuristics exist, which approximate thesolution by exploiting problem-specific properties. Selectionapproaches produce a subset of features [26] and reduceredundancies among features [27]. Feature selection is alsoan important pre-processing step in machine learning to avoidover-fitting and to increase the effectiveness of learning. Fea-tures can be selected either based on group performance oron their individual performance [28]. The set of individuallyselected features may not collectively provide good classifica-tion performance due to the lack of information about featurecorrelation, while individual weak features may provide strongdiscriminatory power in a group [23]. However, individuallyselected features can perform well in constrained environmentswhen some features need to be discarded adaptively becauseof user requirements or application constraints with resource-constrained devices; whereas in the case of feature grouping,the removal of a single feature may significantly reduce theeffectiveness of the whole feature set.
A method for ranking of selected features according to theircontribution to the tasks is presented in [29]. The similaritymeasures between each feature and all the rest are added togenerate a feature score. The highest-scoring feature is selectedand the process is repeated to choose the next relevant feature.The feature importance and similarity between features canbe exploited with a greedy selection method [27] or boostedregression trees can be applied [30]. A hierarchical featureselection method is developed by using rankSVM along witha quality measure to predict the number of selected features[31]. The best-first search can be used to partition the featuresinto subsets that are then combined to maximize the definedinformation retrieval measures [25]. The coherence betweensubgroups of data can also be used to rank features [23].An approach based on cooperative game theory evaluatesthe power of each feature individually and within groups[32]. The structural similarity between data before and afterfeature selection is maintained and topological neighborhoodinformation is used for computing the structural similarity[33]. An unsupervised feature ranking algorithm can discoverBi-clusters that are used to evaluate feature inter-dependencies,
Structural Similarity XFeature Cooperation XHierarchical Clustering X XGame Theory XCo-ordinate Ascent XKernel Class Separability XRandom forest XBi-clusters XmRMR XReliefF XDistance Discriminant X X
Dataset
Text Retrieval XMedical Data XUCI ML Benchmarks X X X X X X XLETOR 4.0 XHandwriting Images XCarnegie Mellon Datasets XBio-Informatics X XUCI regression XSurveillance Videos X X
Evaluation Performance X X X X X X X X X X X X Xcriteria Cost X
separability of instances and feature ranking [23]. This ap-proach inherits some characteristics from ranking and wrap-pers, which use learning methods for feature selection andare classifier-dependent. A minimum-redundancy maximum-relevance (mRMR) based approach can be combined with awrapper method to select a more compact subset from thecandidate features [21]. A kernel-based feature selection cri-terion incorporates the kernel trick with the class separabilitymeasures [34], where the kernel parameters are automaticallytuned by maximizing kernel class separability criteria. Featureselection based on a distance discriminant method converts thesearch problem of feature selection into feature ranking. Theapproach achieves feature selection performance comparableto exhaustive-search methods with a lower computationalcomplexity [35]. Finally, hierarchical clustering is applied toselect the optimal feature subset [28]. Table I summarizes andcompares state-of-the-art feature selection methods.
B. Person re-identification
Various feature types have been used for image-basedperson re-identification [7], [8]. Features can be extractedfrom the image using a Region Covariance Descriptor (RCD)[36] to preserve shape, location and color information alsousing a multi-scale quadtree descriptor [37]. An unsupervisedclustering approach based on appearance attributes can beused to mine Attribute Sensitive Feature Importance (ASFI)[17], which are then combined with global features. Randomforests can be used to group images of the same personinto sub-clusters based on color and texture features. Featuresrelevant to each sub-cluster are weighted to improve the re-identification rate. A spatio-temporal relationship is learned tofind the probability of matching a person from one camerato another, coupled with an adaptive Brightness TransferFunction (BTF) to handle illumination changes [38].
Local Binary Patterns (LBP) are used to describe spatial pat-terns using normalized color intensities, followed by a pixel-based thresholded color distance to depict structural informa-tion [39]. The Mean Riemannian Covariance Grid [40] is usedto generate a human signature from the detected persons usingLBP on the head regions [41]. Histograms of Oriented Gra-
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014 3
dients (HOGs), the Scale Invariant Feature Transform (SIFT)and weighted HSV histograms can be used for shape, textureand chromatic content to build a discriminative signature alongwith a weighted distance measure [42]. The mean color valuesfrom small image regions can be combined with the histogramof LBP to represent an image, and then pairwise sampledifferences are learned for re-identification [43]. Multi-shotre-identification approaches combine the information frommultiple instances of the same object [11]. HSV histogramsof the image-epitome group extract patches from multi-imageswith similar properties. Alternatively, the image can be dividedinto small components and the difference from an existingbag of components is considered. The difference vector isrepresented as a descriptor of the image [44]. For a detailedanalysis of the state of the art in person re-identification andrelated taxonomy the reader is referred to [7]–[9].
Existing re-identication approaches exploit features for im-proving the re-identication rate without considering constraintson resource utilization, which significantly vary between fea-tures. This limits their usability and scalability in real-worldapplications. Unlike existing methods, the proposed approach,CoPE, selects good-performing less-expensive features.
III. COST-AND-PERFORMANCE-EFFECTIVE (COPE)FEATURE SELECTION
Let C = CnNn=1 be a network of N cameras andPn = PmnMm=1 be a set of M persons in the Field ofView (FoV) of Cn. Each Pmn is represented by a feature setFmn = frmnRr=1 containing R feature types fr, wherer = 1 · · ·R.
To characterize a feature we consider a performance vectorand a cost vector. The performance vector, Πr, measuresthe discriminating ability of fr in person re-identification;whereas the cost vector, Ψr, measures the extraction time andthe storage size associated to fr. Πr and Ψr are iterativelycombined to generate cost-effective well-performing features.
A. Feature performanceLet a training set be composed of P1 = Pk1Mk=1 and
P2 = Pm2Mm=1, where M is the number of persons visiblein both C1 to C2. The same value of k and m representsthe same person. The feature sets Fk1 = frk1Rr=1 andFm2 = frm2Rr=1 are extracted from person Pk1 and Pm2,respectively. We measure the performance of a feature byanalyzing the similarity of the representation of the sameperson in two views using feature fr as well as the similaritywith the other M − 1 people.
We define the performance vector Πr representing theperformance of fr on M persons. We measure the similaritybetween two instances frk1 and frm2 of fr by a relative matchingdistance D(·), which receives as input a feature pair andreturns the feature similarity drmk between Pk1 and Pm2:
drmk = D(frk1, frm2). (1)
For each Pm2, we measure M distances from P1. Each drmkis then normalized (0 ≤ drmk ≤ 1) as
drmk =drmk −mink d
rmk
maxk drmk −mink drmk, (2)
where mink drmk and maxk d
rmk are, respectively, the mini-
mum and the maximum distances of Pm2 from P1 using fr.The set of M normalized distances drm = drmkMk=1 containsone distance corresponding to the same person in C1 and C2
(drmm: correct matching distance) and M − 1 distances ofPm2 from the instances of other persons in C1 (Ωr
m: the setof incorrect matching distances).
In the ideal case, a feature fr is considered well-performingfor Pm2 if the distance in fr between the correct matchingpair is smaller than the minimum value of distances in Ωr
m.However, in real-world re-identification scenarios the idealcondition is fulfilled only for a limited number of people.Therefore, we relax the criterion and use the median valueof the incorrect distances, Ωr
m, and measure the performancescore Πrm as
Πrm =drmm
Ωrm
. (3)
The condition drmm < Ωrm leads to 0 ≤ Πrm < 1 in
Eq. 3. The smaller Πrm, the better the performance. Πrm ≥ 1indicates that fr performs poorly. For each fr, we define theperformance vector Πr using M persons as
Πr = ΠrmMm=1, (4)
where each element Πrm corresponds to the performancescore of fr for a single person in the training data. fr withminm Πrm ≥ 1 for all M persons are discarded beforeperforming the feature selection thus resulting in R ≤ Rremaining features. We then define the R ×M performancematrix ∆ as
∆ =[Πrm
]R×M , (5)
where r = 1, . . . , R and m = 1, . . . ,M . Πr is the rth rowrepresenting the performance of fr for P2, while χm is themth column representing the performance comparison of Pm2
for R features. ∆ is further analyzed jointly with the cost offeatures as discussed next.
B. Feature cost
We define the cost vector of feature fr by considering twocomponents, namely its storage size, βrmn, and the computa-tional time for feature extraction, Γrmn. The range of bothcomponents can vary largely across different fr. To avoidone of the two components dominating the other for featureselection in case of large values (e.g. large storage size orhigh computational time), we define the cost vector, Ψr, asthe inverse of the average of the two cost components:
Ψr = (ΨΓr,Ψβr) =
αMNN∑n=1
M∑m=1
Γrmn
,(1− α)MNN∑n=1
M∑m=1
βrmn
, (6)
where α ∈ [0, 1] is a weight that helps accounting for caseswhen one constraint is more important than the other (e.g.when a limited storage space is available with no constraintson the computational time, or vice versa). In this paper weconsider the byte as the unit for βrmn and the millisecond
4 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014
Feature storage
Feature extraction
Select features
Cost evaluation
Performance evaluation
Y
Γ𝑟𝑚𝑛, β𝑟𝑚𝑛
Ψr
𝐟𝑟𝑚𝑛
P𝑚𝑛
𝚷r
Fig. 1. Block diagram of the proposed Cost-and-performance-effective(CoPE) feature selection approach.
(ms) as the unit for Γrmn. We measure the magnitude of thecost vector by calculating the Euclidean norm ||Ψr||, where||Ψr|| ∈ [0, 1]. The larger ||Ψr||, the cheaper the feature.Note that as this cost score is obtained by combining twoindependent components in Eq. 6, new cost constraints can beincluded as additional independent components of the vector.
C. Feature selection
We perform a competitive feature selection such that theleast costly features exhibiting the best performance are se-lected by exploiting Ψr and Πr of each fr (Fig. 1). We definea vector V that contains the elements Πrm ≤ 1 from ∆ sortedin ascending order. We divide V into bins Ii, where 1 ≤ i ≤ Rand each bin contains M performance scores such that in thebest case a single feature with the best performance for all theM persons can be selected in a single iteration. A set Φir isdefined which contains the performance scores Πrm within Iifor each fr.
Figure 2 shows an example of performance matrix ∆, andhighlights the vector V, the bin Ii and the set Φir. Weiteratively traverse each bin Ii until all performance scores inV are exploited for feature selection. Cost is considered jointlywith performance to select a cheaper feature when comparableresults can be obtained by the features in the set. We calculatethe combined importance score Sir of each fr within Ii as
Sir =|Φir| ||Ψr||
Πr
, (7)
where ||Ψr|| is the Euclidean norm and Πr is the median ofthe values in Πr. |Φir| is the cardinality of Φir, which repre-sents the number of persons for which fr has the performancescores within Ii. Sir combines the cost with the performancewithin the bin Ii such that fr with the least cost, the highestoverall performance and the maximum number of Πrm withinthe interval Ii gets the maximum importance score Sir. As thebins are sorted in decreasing order of performance, the bestperforming feature can be selected as
r∗ = arg maxr
Sir, (8)
where r∗ is the ID of the feature with the highest combinedimportance score Sir.
Let Y12 be the list of selected features for C1 and C2.If fr
∗/∈ Y12 then fr
∗is appended in Y12. The list Y12 is
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
f1
f2
f3
f4
f5
f6
Performance score
Featu
re type
Pir1
Pir2
Pir3
Pir4
Pir5
Pir6
Pir7I1
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
f1
f2
f3
f4
f5
f6
ratio
fea
ture
s
Pir1
Pir2
Pir3
Pir4
Pir5
Pir6
Pir7
Πr1
Πr2
Πr3
Πr4
Πr5
Πr6
Πr7
V
Φ15
Φ12
Φ11
f1
f2
f3
f4
f5
f6
Fig. 2. Example of performance matrix ∆ containing performance scorevalues Πrm (horizontal axis) obtained for R features (vertical axis) and Mpersons (color coded), where r = 1 · · · 6, m = 1 · · · 7, R = 6 and M = 7.V contains Πrm in the range [0, 1]. The bin Ii within V contains M valuesof Πrm. The values within Ii are spread among R features such that eachfeature has a set Φir containing Πrm and
∑Rr=1 |Φir| = M . The bin I1
for i = 1 is illustrated in the figure. For each feature fr we define a set Φ1r ,e.g. when r = 5, f5 contains Φ15 = Π53,Π57,Π54, |Φ15| = 3 and∑6
r=1 |Φ1r| = 7; f4 is discarded because minm Π4m ≥ 1; person m = 6is discarded because minr Πr6 ≥ 1.
progressively filled with fr∗
in order of importance. However,repeating the selection process of the next best feature mayresult in performance overlapping. Performance overlappingoccurs when the performance scores of data points (persons)already considered in the previously selected features are usedto measure the importance score Sir of a new feature. Toavoid performance overlapping, we remove from V all theperformance scores χm of a person Pmn for which at least oneperformance score Πrm exists in the set Φir∗ of the selectedfeature fr
∗. Person Pmn is included in the set Z∗ containing
the persons that have already taken part in the selection offr
∗, given as Z∗ ∪ Pmn ∀ Πrm ∈ Φir∗ . We then repeat
the process for selecting the next best feature. Each selectedfeature is now representative of a unique subset of data and aselected new feature now increases the diversity in the featureset by covering a wider range of data.
Feature selection continues within the same bin Ii until allperformance scores have been utilized for the selection. Thenwe move to the next bin in V. The algorithm stops when allpersons in the training data are exhausted (|Z∗| = M ) orwhen all features are selected (〈Y12〉 = R, where 〈·〉 countsthe elements in the list). In the former case we obtain a subsetof features. In the latter case the method returns the completefeature set with features ranked in order of importance.
Note that because the selected features in Y12 are ranked bydecreasing importance, the feature set can be further reducedby dropping the IDs of the least important features should theconstraints of the application become more restrictive. CoPEis summarized in Algorithm 1.
D. Discussion
Feature selection using CoPE is performed once usingtraining data when a camera network is set-up. Then each
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014 5
Algorithm 1 CoPE feature selectionM : total number of persons;R : number of features;Cn : nth camera in the network;fr : rth feature in the feature set;Pmn : mth person in Cn;Ψr : cost vector of fr ;Πrm : performance score value for Pmn using fr ;Πr : performance vector for fr ;∆ : performance matrix;χm : R performance scores for Pmn;V : vector containing sorted values of Πrm ≤ 1 from ∆ ;Ii : ith bin with values from V;Sir : combined importance score of fr in Ii;Ynq : list of selected features for Cn and Cq ;Z∗ : set of people taking part in the selection;Φir : set of Πrm in V within Ii for fr ;〈·〉 : number of elements in the list;| · | : cardinality of a set;
1: Z∗ = φ , Ynq = φ2: while |Z∗| ≤M or 〈Ynq〉 ≤ R do3: while 1 ≤ i ≤ R do4: for r = 1 to R do5: Φir = Πrm in V within Ii for fr
6: end for7: for r = 1 to R do8: calculate Sir using Eq. 79: end for
10: get r∗ using Eq. 8 . ID of selected feature11: if fr
∗/∈ Ynq then
12: append fr∗
to Ynq
13: end if14: remove χm from V; ∀ Πrm ∈ Φir∗
15: Z∗ = Z∗ ∪ Pmn; ∀ Πrm ∈ Φir∗
16: remove Πr∗ from V17: end while18: end while
camera locally stores the list of selected features Ynq for eachneighboring camera Cq . Cn and Cq are neighbors if an objectexiting the FoV of a camera is expected to enter the FoVof the second camera without passing through the FoV of athird camera. If a new camera is added to the network, thetraining is performed pair-wise between the new camera and itsneighbors only [45]. Note that features selected for a camerapair may not always be appropriate for another camera pairbecause of differences in illumination conditions and camerapose with respect to the targets. This approach, developedfor camera pairs, is appropriate for distributed multi-camerasettings where each camera communicates with its neighborswithout a central control unit.
A performance matrix ∆ is generated for each camera pair,while the cost vector Ψr already takes into account N camerasand therefore remains the same. N cameras in the extremecase form a complete graph, where each camera has N − 1neighbors. The time complexity of feature selection for such anetwork is N(N + 1)/2 times that of the feature selection fora camera pair. In the case of multiple neighboring cameras, itis also possible that the locally stored lists of selected featuresmay together result in the extraction of the complete featureset. Such a scenario may occur in the case of neighbors locatedfar away and reduce the benefits of feature selection.
When machine learning is used for re-identification, twotraining phases are involved, namely the training for feature
selection and the training for learning the re-identificationmodel. Each camera stores the trained models (weights) inaddition to the selected feature IDs between (neighboring)camera pairs. The inclusion of learning models with the CoPEfeature selection is independent of the feature selection itself,and there is an increase in the storage cost (fixed) due to thelocal storing of the trained models (and not because of thefeature selection).
IV. EVALUATION AND COMPARISONS
We evaluate CoPE on the initial feature set as in [14]–[17]. We test the re-identification capabilities with existingapproaches based on DDM, namely Bhattacharyya distance[10], [16], L1-Norm [9] and Chi-square distance. CoPE withDDM is further compared with recent re-identification ap-proaches such as Probabilistic Relative Distance Compari-son (PRDC) [15] and Attribute-Sensitive Feature Importance(ASFI) [17]. We further compare CoPE with the followingfeature selection methods: Fisher score [19], Information gain[20], mRMR [21], ReliefF [22] and Bi-clusters [23] for re-identification using DDM (Bhattacharyya) and learning ap-proaches (rankSVM [14] and AdaBoost [16]). Finally, weperform a cost-performance analysis for CoPE.
A. Evaluation criteria
We consider three validation criteria, namely cost of fea-tures, re-identification rate and feature budgeting. The cost offeatures is calculated for the initial feature set and then for theselected features to analyze improvements in data reductionand computational time. In addition, we evaluate the trainingtime for feature selection. The re-identification rate for theassociation methods is compared with the initial feature set andthen with the selected features using the Cumulative MatchingCharacteristics (CMC) curves [16]. The curves show the truetarget rates for given false target rates. The overall performanceis also evaluated using the Area Under the CMC Curves(AUC). Finally, we consider feature budgeting in constrainedenvironments to analyze the scalability of CoPE and the effectsin terms of cost and performance of further feature reductions.
B. Experimental setup
The datasets for the evaluation are VIPeR [16] and i-LIDS [46]. These datasets present a mix of characteristicssuch as outdoor and indoor settings, variations in viewingangle, occlusions and illumination changes. The VIPeR dataset(outdoors) contains 632 pairs of persons from arbitrary view-points. The images of people in VIPeR are not occluded,but present significant appearance changes. In the i-LIDSdataset, C1 is Camera 1 and C2 is Camera 3. These twocameras represent non-overlapping views with considerableillumination changes. We represent each person with a singleimage per camera. In total, we have 348 people that go fromC1 to C2. Single images of 124 people are manually extractedwhile exiting C1 when the person is completely in the FoVof C1 and the first image of a person on reappearance when(s)he is completely in C2. People can be partially visible due
6 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014
Fig. 3. Examples of upper-body stripes (cropped region) extracted basedon the location of the head (green bounding box) Eq. 9 in (a) VIPeR and(b) i-LIDS. (c) Comparison of re-identification rates using the initial featureset from the full-body patch (solid lines) and the upper-body stripe (dashedlines), without feature selection, using i-LIDS (M = 174). For associationthree DDM approaches are compared.
to occlusions. For the remaining 224 people, we utilize theexisting ( [9], [15], [40])images of persons extracted from i-LIDS videos and select one image per person per camera, suchthat no person is repeated.
The head and the upper-body are the most frequentlyvisible and recognizable parts of a person (Fig. 3 (a, b))in case of surveillance settings and multiple people in thescene. We assume that the person detection phase is solvedusing a head detector [47], [48] resulting in a bounding boxBmn = (x, y, w, h), for the head of person Pmn in Cn, wherex and y are the x-y coordinates of the top left corner, w isthe width and h is the height of the bounding box. From agiven bounding box Bmn (in this paper manually defined),a vertical stripe of the upper-body is generated using theapproach described in [9]:
Pmn = f(Bmn) = [x+ w/4, y + h/4, w/2, h ∗ 2]. (9)
The area of the defined upper-body shape is 8 times smallerthan that of normalized full-body patch.
We use the commonly employed color and texture features[14]–[17] for the comparison. Existing approaches use a 2784-dimensional feature vector by dividing the full-body personimage into a set of stripes (6) and then concatenating thecorresponding features from each stripe. We reduce the sizeof the object representation by extracting features from thedefined upper-body shape only (better suited for crowdedscenes). Each feature is a 12-bin histogram of a color channelor a filtered image of the defined shape. Nine color channels(R, G, B, Y, Cb, Cr, H, S, V) from RGB, YCbCr and HSVcolor spaces are used. We divide the upper-body patch intotwo halves and extract the features. Instead of concatenatingthe extracted features, we add the corresponding bins. Thisoperation results in a single feature set with twice the weightto the upper region of the defined shape. For texture, 8 Gaborand 13 Schmid filters are applied on the Y channel of thedefined shape. The parameters of the Gabor and Schmidfeatures are indicated in Fig. 4. Figure 3 (c) shows there-identification results in i-LIDS for the complete featureset extracted from the defined upper-body and the full-bodypatches without feature selection. It can be observed that abetter re-identification rate can be achieved with the upper-
body shape in the case of occlusions and crowd.The data generated for each feature is encoded using the
lossless data compression algorithm ‘deflate’, which combinesLZ77 and Huffman coding. The average storage size andcomputational cost of a single feature (per person) is calculatedfor each camera. For each dataset, we apply the two-fold crossvalidation. The experiments are carried out using Matlab 7.11on a 3.3 GHz dual core desktop system with 3 GB of RAM.
C. Analysis and discussion
Figure 4 shows the storage size βr and the extraction timeΓr of the 30 features used. A fixed number of bins is usedfor histograms. However βr varies between 29 and 56 bytesbecause the data encoding is applied before the feature storage.Γr of the feature extraction varies between 16 and 60 ms. Theoverall computational time and the storage size required by asingle camera Cn for R features and M persons is given byTn as
Tn =
M∑m=1
R∑r=1
Γrmn,
M∑m=1
R∑r=1
βrmn
. (10)
For the comparisons, we measure the normalized cost E of theselected feature sets from the cost score Ψr (Eq. 6) as
E =
fr∗∈Ynq∑
1/||Ψr∗ ||R∑r=1
1/||Ψr||, (11)
where Ynq is the list of selected features. The set of 30features has the maximum cost Emax = 1. We consider 316and 174 persons in VIPeR and i-LIDS, respectively.
Figure 5 shows the analysis of CoPE on i-LIDS withthree selection criteria: (i) feature selection as a functionof performance only keeping the cost fixed (CoPE-FC); (ii)feature selection considering both cost and performance of afeature (CoPE); and (iii) feature selection as a function ofcost only while keeping the performance fixed (CoPE-FP).The cost of the selected features is the highest for CoPE-FC, since the selection is carried out based on performanceonly. However, in the absence of cost constraints CoPE-FC is also able to achieve the highest re-identification rateusing DDM. Both the cost and the re-identification rate ofthe selected features are reduced with CoPE. When varyingthe cost parameter α in CoPE, while the composition of theselected features remains similar their order changes (top threerows of the table in Fig. 5 (a)). Since cost and performanceare independent in the feature selection, varying α does notaffect the performance of a feature. The selected features mayvary based on the requirement of a system controlled by α,i.e. for good-performing features with a limited extraction timeα = 1, and for features with limited storage size and goodperformance α = 0. Note that a limited extraction time maynot imply a higher storage size (and vice versa). The smallestcost for CoPE is obtained when there is an equal contributionof computational time and storage size (α = 0.5).
In CoPE-FP, although performance is not used for featureselection, in order not to obtain a sorted list of all features
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014 7
Sto
rag
e si
ze
60
50
40
30
20
10
0
Ex
traction tim
e
0
10
20
30
40
50
60Storage size (bytes) Extraction time (ms)
Fig. 4. Storage size and the extraction time for each feature in the initial feature set (listed on the horizontal axis with their IDs and parameters). For Gabor(γ, θ, λ, σ2) and Schmid (σ, τ) filters, γ is the aspect ratio, θ is the angle in radians, λ the wavelength of the sinusoidal factor, σ the standard deviation andτ the number of cycles. The vertical axes represent the storage size (bytes) and the feature extraction time (ms) for a single person within a camera FoV.
Fig. 5. Analysis of feature selection with varying selection criteria: CoPE (both cost and performance), CoPE-FC (fixed cost, performance only) and CoPE-FP(fixed performance, cost only), and with varying α values: α = 0 (black), α = 0.5 (blue), α = 1 (magenta) in i-LIDS (M = 174) in terms of (a) selectedfeatures, (b) normalized cost E, (c) CMC curves, and (d) AUC of CMC curves.
based on cost, we remove the people from the trainingdata for which the selected minimum cost feature has goodperformance so that the algorithm stops when all the peoplein the training data are exhausted. For α = 0 the order ofselection is controlled by the storage size, while for α = 1the features with the shortest extraction times (color features)are selected first (see Fig. 4 for time and size). An interestingcase is when the features with IDs 8 and 9 are selected for allthree values of α, since these features have both the shortestcomputational time and the smallest storage size. In contrast,the feature with ID = 1 has the shortest extraction time and alarge storage size. This makes it the first feature with α = 0.5and 1, while it is not selected with α = 0. The order ofperformance of the selected features for the three criteria is asfollows CoPE-FP < CoPE < CoPE-FC. The rest of theevaluation is performed for α = 0.5, in order to have an equalcontribution from the storage size and the extraction time.
1) CoPE vs all-features: Table II shows the storage size andthe computational time for the features extracted from eachperson observed in one camera using Eq. 10. We comparethe results of the initial feature set with that of the threenon-unique sets of selected features obtained by CoPE using
TABLE IISTORAGE SIZE, COMPUTATIONAL TIME AND NORMALIZED COST OF THE
INITIAL FEATURE SET PER CAMERA USED IN EXISTINGRE-IDENTIFICATION APPROACHES COMPARED WITH COPE FEATURES
OBTAINED FOR THREE SIMILARITY MEASURES (IN EQ. 1).
Dataset Distance Total Feature IDs as in Fig. 4 Size Time Cost Eas in Eq. 1 features (KB) (sec) (Eq. 11)
three similarity measures: Bhattacharyya distance, L1-Normand Chi-square distance in Eq. 1. For VIPeR, the number ofselected features are 6, 6 and 8, respectively, for the threesimilarity measures that reduce the storage size per camera to11%, 18% and 24% of the total size (466.43 KB) of the initial30 features. In i-LIDS, 10, 8 and 9 features are selected for thethree similarity measures that respectively reduce the storagesize to 31%, 23% and 27% of the storage requirement for theinitial feature set (256.83 KB). Similarly, the computational
8 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014
Fig. 6. Person re-identification comparison using CMC curves representing true target rate for the top 30% of false target rate, and the AUC of the CMCcurves in VIPeR (M = 316) and i-LIDS (M = 174) datasets. (a-d) CoPE features selected using three similarity measures, namely Bhattacharyya distance(blue), L1-Norm (red) and Chi-square (green) in Eq. 1, compared with the initial complete feature set using DDM approaches for re-identification. (e-h)Existing re-identification approaches: PRDC [15] (magenta) and ASFI [17] (green) using the complete feature set are compared with DDM (Bhattacharyya)[16] using CoPE features (blue) and the complete feature set (black).
time of feature extraction per camera is reduced significantly.In the VIPeR dataset, the computational time is reducedto 12%, 10% and 19% for the three similarity measures,respectively. In the case of i-LIDS, the computational timeis reduced to 27%, 20% and 23%. It can also be observedthat the normalized cost E of the selected CoPE features isreduced more in VIPeR than in i-LIDS because mostly thecolor features are selected in VIPeR. The color features arefast to extract with less or comparable storage size (Fig. 4) andperform better than texture features. In VIPeR, we reduce thecost E of the feature set to 20%, while in i-LIDS we reduceit to 33% of the initial feature set.
Figure 6 (a-d) compares the re-identification rate for thethree DDM approaches with the state of the art. In a DDMapproach, two persons are considered correctly matched for re-identification, if their obtained feature sets have the minimummatching distance between them. The performance of theselected features is measured in terms of improvement of there-identification rate of DDM approaches compared to that ofusing the initial feature set. CMC curves highlight the truetarget rate for the first 30% of false target rates (the mostimportant part of CMC for evaluation). In VIPeR, a higher re-identification rate is obtained using the selected features. Forexample, at 20% false target rate in the CMC curves, the truetarget rate is above 65% for selected features compared to theinitial feature set with true target rates between 40% to 50% forall the three measures. Due to the limited illumination changesbetween cameras, mostly color features are selected (Table II).In i-LIDS, both color and texture features are selected. The re-identification results for association using the selected featuresare improved and in some points are comparable to that of
using all features. The AUC shows that the features selectedusing all the three similarity measures have overall betterperformance than that of the initial feature set. The highestre-identification rate is obtained when the features are selectedusing the Bhattacharyya distance. Therefore, in the followingexperiments we use the Bhattacharyya distance as a similaritymeasure while comparing with existing re-identification andfeature selection approaches.
Figure 6 (e-h) shows the performance comparison of DDM(Bhattacharyya) using CoPE with two recent state-of-the-artre-identification approaches: PRDC [15] and ASFI [17]. Theextracted features from the upper-body patch are given as inputto PRDC and ASFI. In both i-LIDS and VIPeR, a betteror comparable re-identification performance is achieved byCoPE with less storage and computational requirements. CMCcurves show a higher re-identification rate for CoPE especiallyat lower false target rates. CoPE outperforms PRDC and ASFI(AUC in the case of VIPeR), with a cost of 20% of theinitial feature set used in these methods. In i-LIDS comparableresults can be observed at 33% the cost. The results ofPRDC and ASFI in the original papers used the full-bodypatches and a large 2784-dimensional feature set. Thereforewe also include a comparison with the reported results whileperforming the CoPE feature selection on the larger feature setand full-body patches. Figure 7 shows the cost-performancecomparison on ViPER, which has less occlusions and thusjustifies the use of the full patch for person description. It canbe observed from the CMC curves and the AUC that CoPEselected features with DDM show a better re-identificationrate than ASFI with a 73% reduction in the storage size anda 77% reduction in the extraction time. Finally, the use of
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014 9
Fig. 7. Person re-identification comparison using full-body patches and 2784-dimensional feature vector on VIPeR (M = 316). (a) CMC curves and (b)AUC of CMC curves obtained by existing re-identification approaches: PRDC[15] (magenta) and ASFI [17] (green). The results are compared with CoPEusing DDM (cyan) and PRDC (blue) for association. (c) The storage size andthe computational time of the extracted features.
the CoPE features as input to PRDC further improves the re-identification rate of PRDC at 24% the cost of the completefeature set.
2) CoPE vs feature selection methods: We compare CoPEwith 5 existing feature selection and ranking methods, namelyFisher score [19], Information gain [20], mRMR [21], ReliefF[22] and Bi-clusters [23]. Since these are single-objectivefeature selection approaches, for comparision we performfeature selection using the performance only while keepingthe cost fixed (CoPE-FC). The similarities between the featurepairs obtained using Eq. 1 along with the assigned labels ascorrect/incorrect matches are given as input to the featureselection methods. Feature selection methods return a rankedlist of features and a weight vector in the case of Fisher score,Information gain and ReliefF methods, while mRMR and Bi-clusters return only a ranked feature list.
Table III shows the training time for feature selectionand the obtained features ranked in order of importancefor re-identification. Training time is useful to understandthe feasibility of the single time set-up off-line process andbecomes crucial as the size of the network increases. Thetraining time is measured using 316 and 174 people in theVIPeR and i-LIDS, respectively. With VIPeR, CoPE takes0.76 seconds, 5 times less than the next shortest trainingtime by the Fisher score. ReliefF requires the maximum time(124.70 seconds) for training, while Bi-cluster could not betrained for VIPeR even after 25 days. With i-LIDS, the trainingtime of Bi-clusters is nearly 20 hours. Therefore, in a largercamera network Bi-clusters may not be applicable for featureselection. CoPE and CoPE-FC take 0.30 and 0.20 seconds,respectively. The Fisher score takes 0.15 seconds. As thedataset size almost doubles from i-LIDS to VIPeR, the timerequirement for Fisher Score is increased by nearly 24 times,whereas others are only 3 times longer. With the smallestratio and minimum training time CoPE is desirable for featureselection in a camera network.
In Table III, each selection approach returns a differentranking order of features, since there exists no unique feature
subset to solve the same task. If two features show an identicalperformance then any of the two can be selected. In performinga cost-aware feature selection, CoPE returns a subset of well-performing cost-effective features until any further addition inthe cost of features does not improve performance. In VIPeR,most feature selection methods including CoPE and CoPE-FCreturn similar sets with color features in the top ranks. CoPEand CoPE-FC returns the same set of 6 features because of thesimilarity in the selection procedure. In i-LIDS, 10 features areselected by CoPE, while 13 features are selected by CoPE-FC.We fix the number of selected features for the existing methodsto be equal to the number of features selected by CoPE-FC(a comparison with varying number of selected features canbe seen in Fig. 11). We pick the top 13 features in i-LIDSand the top 6 features in VIPeR from the ranked features ofthe existing approaches. Figure 8 (a, d) shows the normalizedcost E (Eq. 11) of the obtained selected features. Even afterfixing the number of selected features, E for CoPE featuresremains the smallest. mRMR features show the highest costin both datasets, while those of Fisher score, Information gainand Bi-clusters have costs comparable with that of CoPE-FC.In VIPeR, the CoPE feature set contains all color featuresbecause of the limited illumination changes, while in i-LIDSboth color and texture features are selected. CoPE selects thecolor features first and then the texture, resulting in the leastE of 0.15 and 0.30 in VIPeR and i-LIDS, respectively.
Figure 8 (b, c, e, f) shows the re-identification performanceof the selected features using DDM (Bhattacharyya) as theassociation method. In both VIPeR and i-LIDS, the selectedfeatures using CoPE and CoPE-FC reach the highest re-identification rate. In i-LIDS, CoPE-FC reaches the highestperformance in the absence of the cost constraints. Unlike theexisting approaches based on overall performance only, CoPEselects features by iteratively relaxing the performance scoreSir, thus achieving cost as well as performance advantages.
Finally, features are selected on one dataset and tested on theother to analyze the amount of degradation in the results. Wecompare CoPE with two feature selection approaches, namelyFisher score and Information gain, which have the highestperformance in the cross validation within the same dataset.Fig. 9 shows the cross-data robustness of selected features. InVIPeR, the performance of CoPE is degraded less compared tothe other two methods. The performance of features selectedusing VIPeR deteriorates at a greater rate in i-LIDS, whichis a more challenging dataset. The results are degraded at acomparable rate for all the feature selection approaches, sincealmost the same 6 color features are selected by the threefeature selection approaches.
3) CoPE with learning models: The top-ranked selectedfeatures are used as input to the two learning methods, namelyrankSVM [14] and AdaBoost [16] for re-identification, whichapply implicit feature selection by weighting the feature set.In these cases, feature selection may be used to removepoorly performing features as a pre-processing step to improvethe effectiveness of learning methods. Since the features arerearranged and weighted within the specific learning method,the order of selection is not important and only the differencein the selected features affects the performance. We compare
10 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014
TABLE IIITHE TRAINING TIMES AND THE RANKING ORDER OF FEATURES FOR RE-IDENTIFICATION USING FISHER SCORE [19], INFORMATION GAIN [20], MRMR[21], RELIEFF [22] AND BI-CLUSTERS [23] AS FEATURE SELECTION METHODS COMPARED WITH COPE AND COPE-FC USING VIPER (M = 316) AND
Fig. 8. (a, d) Normalized cost, (b, e) CMC curves and (c, f) AUC of CMC curves obtained for re-identification by applying DDM to the features selectedusing the Fisher score [19] (cyan), Information gain [20] (magenta), mRMR [21] (black), ReliefF [22] (yellow) and Bi-clusters [23] (green), CoPE (blue) andCoPE-FC (red) on (Top row) VIPeR (M = 316) and (Bottom row) i-LIDS (M = 172).
5 10 15 20 25 30
10
20
30
40
50
60
70
80
False target rate (%)
Tru
e ta
rget
rat
e (%
)
Fisher scoreInformation gainCoPEFisher score CDInformation gain CDCoPE CDRandom
(a) CMC VIPeR
5 10 15 20 25 30
10
20
30
40
50
60
70
80
False target rate (%)
Tru
e ta
rget
rat
e (%
)
Fisher scoreInformation gainCoPEFisher score CDInformation gain CDCoPE CDRandom
(b) CMC i-LIDS
Fig. 9. Cross Data (CD) performance comparison in re-identification for thetop-two performing existing feature selection approaches, namely Fisher score[19] (cyan) and Information gain [20] (magenta); and CoPE (blue). The CMCcurves are obtained by (a) feature selection on i-LIDS (M = 174) and testingon VIPeR (M = 316) and by (b) feature selection on VIPeR (M = 316)and testing on i-LIDS (M = 174).
the performance of RankSVM and AdaBoost with their defaultsettings. The comparisons are performed with and withoutfeature selection keeping the same settings, which may notbe optimal. However, the improvement in the results can beobserved after the feature selection by the proposed approach.
RankSVM assigns relative weights to the input featuresbased on the combined contributions in the feature set. Fig-ure 10 (a-d) shows that rankSVM has a better re-identificationrate for both VIPeR and i-LIDS using the features selectedby CoPE compared to those from existing feature selectionmethods. The variation in re-identification rates using theselected features from different approaches is smaller in VIPeRthan in i-LIDS because mostly the same color features areselected (Table III). With i-LIDS, the features selected bydifferent methods (and the re-identification rate) vary in theircomposition. The best performance of CoPE-FC in the truetarget rate (CMC curves) is almost 15% higher than that ofmRMR at the same false target rate, followed by CoPE with aslightly smaller re-identification rate because of the additionalcost constraints. However CoPE remains higher than existingfeature selection approaches. Also the obtained AUCs arehighest for CoPE and CoPE-FC.
AdaBoost combines multiple weak classifiers/features toimprove the matching performance. Figure 10 (e-h) shows theperformance for AdaBoost. In both VIPeR and i-LIDS, the
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014 11
Fig. 10. CMC curves and AUC of CMC curves for re-identification using learning methods: (a-d) rankSVM [14] and (e-h) AdaBoost [16] applied to thefeatures selected by CoPE (blue) and CoPE-FC (red) and existing methods: Fisher score [19] (cyan), Information gain [20] (magenta), mRMR [21] (black),ReliefF [22] (yellow) and Bi-clusters [23] (green), using (a, b, e, f) VIPeR (M = 316); and (c, d, g, h) i-LIDS (M = 174).
features selected by CoPE have an overall better or comparablere-identification rate than existing feature selection methods.In VIPeR, similarly to the rankSVM, CMC curves show asmaller re-identification rate variation among existing methodsbecause of the limited number of selected features (i.e. only6). In i-LIDS, the variation in performance between CoPE andexisting feature selection methods is high as the number ofselected features is increased (up to 13). AdaBoost has a betterlearning ability in i-LIDS than in VIPeR. The performanceon the CMC curves, especially in the starting part, showsthat CoPE and CoPE-FC are able to remove noisy featuresmore effectively than existing feature selection methods thusresulting in a better re-identification rate. In Fig. 10 (g), theCMC curve for ReliefF shows a marginal improvement of upto 2% in true target rate between 20% and 25% of false targetrates at the expense of more costly features than that of CoPE(Table: III). In CoPE because of the cost constraints, we mayobserve a drop in the performance in a few instances in favorof cost reduction and an overall performance improvement.Overall CoPE-FC remains the highest (AUC) followed byCoPE and ReliefF features.
Since learning algorithms are dependent on the training datain addition to the selected features, in challenging scenariosthe performance of learning methods can be reduced. Inre-identification, a single person may exhibit several poseand illumination changes, while we can only extract a fewpatches thus resulting in an under sampled data representation[15]. For example, in VIPeR (CMC curves in Fig. 6 (a) incomparison with Fig. 10 (a, e)), the performance of learningmethods is slightly reduced. In Fig. 6 (a) we can see that afterCoPE feature selection the performance is improved (almostdouble compared to using the initial feature set). A furtherimprovement through a learning method will require a more
robust training set.4) CoPE and feature budgeting: In a constrained environ-
ment, a further reduction of the feature set might be necessary.In such cases the performance needs to be reduced in apredictable manner (feature budgeting). Figure 11 shows thecost vs performance comparison of feature selection methodsfor re-identification using DDM. The performance is measuredusing the area under the first half of the CMC curves and thecost is measured using Eq. 11. In CoPE, a consistent increasein performance and cost can be observed with the additionof each new feature. Since the most important features forre-identification are selected first, the rate of improvement inthe performance is high at the beginning and the performancemonotonically increases as the cost increases, the most desir-able behavior in feature budgeting.
In VIPeR (Fig. 11 (a)), the performance of Fisher score andInformation gain becomes constant after selection of up to 9features due to minimal weighting to the lower ranked features.However, the low ranked features keep increasing the cost ofthe feature set. Such feature selection represents the majorityof data with similar properties while neglecting the featureswith discriminating capability for small amounts of data. ThemRMR feature selection produces a monotonically decreasingperformance after reaching a high performance point becauseof the ranking only strategy. Since VIPeR requires up to 6 dis-criminant features as selected by CoPE, the additional featuresresult in redundant information and the performance decreases(mRMR) or remains constant (Fisher and Information gain),while the cost increases. In i-LIDS dataset (Fig. 11 (b)), Fisherscore shows a non-monotonically increasing performance atthe start and while selecting the second feature shows ahigher performance than CoPE because of the selection ofa comparatively costly feature (with Feature ID=4). However,
12 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014
Fig. 11. Cost vs performance analysis of CoPE (blue star) and existing feature selection methods: Fisher score [19] (cyan plus), Information gain [20](magenta triangle), mRMR [21] (black circle), ReliefF [22] (yellow triangle) and Bi-clusters [23] (green circle) using DDM (Bhattacharyya) for re-identificationon (a) VIPeR and (b) i-LIDS. Features are added in order of decreasing performance. Solid lines show the number of selected features equal to those generatedwith CoPE. Dotted lines extend the cost vs performance comparison for the feature excluded by CoPE. The cost (horizontal axis) is measured using (Eq. 11)and the performance (vertical axis) is measured as the area under the first-half of the CMC curves. At each marker point a new feature is added.
as new features are added, the performance starts decreasing,while CoPE preserves a balance between cost and perfor-mance, which results not only in a monotonically increasingperformance but also in the highest performance with thesmallest cost when the same number of features are used. Thespecific feature (with ID=4) is selected by CoPE at a later stagewhen its cost justifies the performance. A non-monotonicallyincreasing performance is observed in the Information gainand Bi-clusters, however their performance is lower than thatof CoPE as the cost increases.
This evaluation shows how CoPE can select, in the correctorder, less expensive and well-performing features. Improvedor comparable performance compared to existing selectionapproaches is achieved for DDM and learning methods forre-identification with cost-effective features.
V. CONCLUSIONS
We proposed a feature selection approach that identifiesthe most appropriate features for person re-identification. Theamount of data stored for each feature and the computationaltime for its extraction are used jointly with their performanceto generate an overall feature score. The best features areselected in a defined range of scores to reduce the performanceoverlap, a measure of similarity among features. A furtherreduction of the selected features is made possible to accountfor additional operational constraints (e.g. limited resources).The proposed method decreases both the amount of datagenerated per feature set and the amount of time needed forthe extraction of the selected feature set, up to 80% in VIPeRand up to 70%, in i-LIDS dataset without compromisingthe re-identification rate compared to existing re-identificationapproaches (PRDC [15] and ASFI [17]). We also demonstratedthat CoPE improves the performance of other learning-basedre-identification approaches such as those based on rankSVM[14] and AdaBoost [16] compared to the existing featureselection methods. Future work includes implementing the
proposed approach on actual smart cameras and to extract thehead locations using the trained detectors.
REFERENCES
[1] H. Aghajan and A. Cavallaro, Multi-Camera Networks: Principles andApplications. Elsevier, 2009.
[2] V. Sulic, J. Pers, M. Kristan, and S. Kovacic, “Efficient feature dis-tribution for object matching in visual-sensor networks,” IEEE Trans.Circuits Syst. Video Technol., vol. 21, no. 7, pp. 903 –916, 2011.
[3] Y. Charfi, N. Wakamiya, and M. Murata, “Challenging issues in visualsensor networks,” IEEE Wireless Commun. Mag., vol. 16, no. 2, pp. 44–49, 2009.
[4] S. Soro and W. Heinzelman, “A survey of visual sensor networks,”Advances in Multimedia, vol. 09, pp. 1–21, 2009.
[5] D.-U. Lee, H. Kim, M. H. Rahimi, D. Estrin, and J. D. Villasenor,“Energy-efficient image compression for resource-constrained plat-forms,” IEEE Trans. Image Process., vol. 18, no. 9, pp. 2100–2113,2009.
[6] V. Lecuire, C. Duran Faundez, and N. Krommenacker, “Energy efficientimage transmission in sensor networks,” International Journal of SensorNetworks, vol. 4, no. 1, pp. 37–47, 2008.
[7] R. Vezzani, D. Baltieri, and R. Cucchiara, “People re-identification insurveillance and forensics: a survey,” ACM Computing Surveys, vol. 46,no. 2, pp. 1–37, 2013.
[8] G. Doretto, T. Sebastian, P. Tu, and J. Rittscher, “Appearance-basedperson reidentification in camera networks: problem overview andcurrent approaches,” Journal of Ambient Intelligence and HumanizedComputing, vol. 2, no. 2, pp. 127–151, 2011.
[9] R. Mazzon, S. F. Tahir, and A. Cavallaro, “Person re-identification incrowd,” Pattern Recognition Letters, vol. 33, no. 14, pp. 1828–1837,2012.
[10] M. Farenzena, L. Bazzani, A. Perina, V. Murino, and M. Cristani,“Person re-identification by symmetry-driven accumulation of localfeatures,” in Proc. CVPR, San Francisco, USA, June 2010.
[11] L. Bazzani, M. Cristani, A. Perina, and V. Murino, “Multiple-shotperson re-identification by chromatic and epitomic analyses,” PatternRecognition Letters, vol. 33, no. 7, pp. 898–903, May 2012.
[12] Y. Cheng, W. Zhou, Y. Wang, C. Zhao, and S. Zhang, “Multi-camera-based object handoff using decision-level fusion,” in Int. congress onImage and Signal Processing, Tianjin, China, October 2009.
[13] O. Javed, K. Shafique, Z. Rasheed, and M. Shah, “Modeling inter-camera space-time and appearance relationships for tracking across non-overlapping views,” Comput. Vis. Image Understand., vol. 109, no. Issue2, pp. 146 – 162, February 2008.
[14] B. Prosser, W.-S. Zheng, S. Gong, and T. Xiang, “Person re-identificationby support vector ranking,” in Proc. BMVC, Aberystwyth, UK, August2010.
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. X, NO. Y, 2014 13
[15] W.-S. Zheng, S. Gong, and T. Xiang, “Re-identification by relativedistance comparison,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35,no. 3, 2013.
[16] D. Gray and H. Tao, “Viewpoint invariant pedestrian recognition withan ensemble of localized features,” in Proc. ECCV, Marseille, France,October 2008.
[17] C. Liu, S. Gong, C. C. Loy, and X. Lin, “Person re-identification: Whatfeatures are important?” in Proc. ECCV, Firenze, Italy, October 2013.
[18] K. Jeong and C. Jaynes, “Object matching in disjoint cameras usinga colour transfer approach,” Springer Journal of Machine Vision andApplications, vol. 19, no. 5, pp. 88–96, September 2008.
[19] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, 2nd ed.John Wiley & Sons, New York, 2001.
[20] T. M. Cover and J. A. Thomas, Elements of Information Theory. Wiley,1991.
[21] H. Peng, F. Long, and C. Ding, “Feature selection based on mu-tual information: criteria of max-dependency, max-relevance, and min-redundancy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 8, pp.1226–1238, 2005.
[22] M. Robnik-Sikonja and I. Kononenko, “Theoretical and empirical analy-sis of relieff and rrelieff,” Machine Learning, vol. 53, no. 1-2, pp. 23–69,Oct. 2003.
[23] Q. Huang, D. Tao, X. Li, L. Jin, and G. Wei, “Exploiting local coherentpatterns for unsupervised feature ranking,” IEEE Trans. Syst., Man,Cybern. B, vol. 41, no. 6, pp. 1471–1482, 2011.
[24] E. Guldogan and M. Gabbouj, “Feature selection for content-basedimage retrieval,” Signal, Image and Video Processing, vol. 2, no. 3,pp. 241–250, 2008.
[25] V. Dang and W. B. Croft, “Feature selection for document rankingusing best first search and coordinate ascent,” in Proc. SIGIR Workshop,Geneva, Switzerland, July 2010.
[26] R. Kohavi and G. H. John, “Wrappers for feature subset selection,”Artificial Intelligence, vol. 97, no. 1, pp. 273–324, 1997.
[27] X. Geng, T. yan Liu, T. Qin, and H. Li, “Feature selection for ranking,”in Proc. SIGIR, Amsterdam, Netherlands, July 2007.
[28] T. Yan and Y. Hou, “An unsupervised feature selection method based ondegree of feature cooperation,” in Proc. IEEE FSKD, Shanghai, China,July 2011.
[29] H.-L. Wei and S. A. Billings, “Feature subset selection and ranking fordata dimensionality reduction,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 29, no. 1, pp. 162–166, 2007.
[30] F. Pan, T. Converse, D. Ahn, F. Salvetti, and G. Donato, “Featureselection for ranking using boosted trees,” in Proc. ACM CIKM, NewYork, USA, November 2009.
[31] G. Hua, M. Zhang, Y. Liu, S. Ma, and L. Ru, “Hierarchical featureselection for ranking,” in Proc. World Wide Web, Raleigh NC, USA,April 2010.
[32] X. Sun, Y. Liu, J. Li, J. Zhu, H. Chen, and X. Liu, “Feature evalua-tion and selection with cooperative game theory,” Pattern Recognition,vol. 45, no. 8, pp. 2992–3002, 2012.
[33] S. Mitra, P. P. Kundu, and W. Pedrycz, “Feature selection using structuralsimilarity,” Information Science, vol. 198, no. 0, pp. 48–61, 2012.
[34] L. Wang, “Feature selection with kernel class separability,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 30, no. 9, pp. 1534–1546, 2008.
[35] J. Liang, S. Yang, and A. Winstanley, “Invariant optimal feature se-lection: A distance discriminant and feature ranking based solution,”Pattern Recognition, vol. 41, no. 5, pp. 1429–1439, 2008.
[36] O. Tuzel, F. Porikli, and P. Meer, “Region covariance: A fast descriptorfor detection and classification,” in Proc. ECCV, 2006, pp. 589–600.
[37] W. Ayedi, H. Snoussi, and M. Abid, “A fast multi-scale covariancedescriptor for object re-identification,” Pattern Recognition Letters,vol. 33, no. 14, pp. 1902 – 1907, 2012.
[38] K.-W. Chen, C.-C. Lai, P.-J. Lee, C.-S. Chen, and Y.-P. Hung, “Adaptivelearning for target tracking and true linking discovering across multiplenon-overlapping cameras,” IEEE Trans. Multimedia, vol. 13, no. 4, pp.625 –638, 2011.
[39] G. Lian, J.-H. Lai, C. Suen, and P. Chen, “Matching of trackedpedestrians across disjoint camera views using ci-dlbp,” IEEE Trans.Circuits Syst. Video Technol., vol. 22, no. 7, pp. 1087 –1099, 2012.
[40] S. Bak, E. Corvee, F. Bremond, and M. Thonnat, “Multiple-shot humanre-identification by mean riemannian covariance grid,” in Proc. IEEEAVSS, Klagenfurt, Austria, September 2011, pp. 179 –184.
[41] E. Corvee, S. Bak, and F. Bremond, “People detection and re-identification for multi surveillance cameras,” in Proc. VISAPP, February2012, pp. 82–88.
[42] N. Martinel and C. Micheloni, “Re-identify people in wide area cameranetwork,” in Proc. CVPR Workshop, June 2012, pp. 31 –36.
[43] M. Hirzer, P. M. Roth, and H. Bischof, “Person re-identification by effi-cient impostor-based metric learning,” in Proc. IEEE AVSS, September2012, pp. 203 –208.
[44] R. Satta, G. Fumera, and F. Roli, “Fast person re-identification basedon dissimilarity representations,” Pattern Recognition Letters, vol. 33,no. 14, pp. 1838–1848, 2012.
[45] S. Khan and M. Shah, “Consistent labeling of tracked objects in multiplecameras with overlapping fields of view,” IEEE Trans. Pattern Anal.Mach. Intell., vol. 25, no. 10, pp. 1355–1360, 2003.
[46] i LIDS, “Home office multiple camera tracking scenario definition(uk),” 2008. [Online]. Available: http://www.homeoffice.gov.uk/science-research/hosdb/i-lids/(lastaccessed:June,2013)
[47] S. Gong, T. Xiang, and S. Hongeng, “Learning human pose in crowd,”in Proc. ACM Multimedia, Firenze, Italy, October 2010.
[48] H. Xu, P. Lv, and L. Meng, “A people counting system based on head-shoulder detection and tracking in surveillance video,” in Proc. IEEEICCDA, Qinhuangdao, China, June 2010, pp. 394–398.
Syed Fahad Tahir received his MS degree in Com-puter System Engineering from Ghulam Ishaq KhanInstitute of Engineering Sciences and Technology(GIKI) in 2006 and BS degree in Computer Sciencesfrom National University of Computer and EmergingSciences (FAST-NU) in 2004. He served as Manager(Technical) in the Centre of Excellence in Scienceand Applied Technologies, a Government ResearchOrganization of Pakistan. Since 2011, he has beenwith Queen Mary University of London, UK, andAlpen-Adria Universitat Klagenfurt, Austria, as a
PhD researcher under the supervision of Prof. A. Cavallaro and Prof. B.Rinner. He was a NESCOM Fellow during his MS studies and was awardedErasmus Mundus fellowship for his Double Doctorate. He has more than 6journal and conference publications in the areas of re-identification in multi-camera networks, pattern analysis and object matching.
Andrea Cavallaro received his Ph.D. degree in elec-trical engineering from the Swiss Federal Instituteof Technology, Lausanne, in 2002, and the Laurea(summa cum laude) degree in electrical engineer-ing from the University of Trieste in 1996. He isProfessor of multimedia signal processing and theDirector of the Centre for Intelligent Sensing, QueenMary University of London, U.K. He was a ResearchFellow with British Telecommunications in 2004and 2005 and was awarded the Royal Academy ofEngineering Teaching Prize in 2007; three Student
Paper Awards on target tracking and perceptually sensitive coding at theIEEE and the Best Paper Award at the IEEE AVSS 2009. He is an AreaEditor for the IEEE SIGNAL PROCESSING MAGAZINE and an AssociateEditor for the IEEE TRANSACTIONSON IMAGE PROCESSING. He is anelected member of the IEEE Signal Processing Society, Image, Video, andMultidimensional Signal Processing Technical Committee, and a Chair of itsAwards Committee. He served as an elected member of the IEEE SignalProcessing Society, Multimedia Signal Processing Technical Committee, asan Associate Editor for the IEEE TRANSACTIONS ON MULTIMEDIA andthe IEEE TRANSACTIONS ON SIGNAL PROCESSING, and as a GuestEditor for seven international journals. He was a General Chair for theIEEE/ACM ICDSC 2009, BMVC 2009, M2SFA2 2008, SSPE 2007, and theIEEE AVSS 2007. He was a Technical Program Chair of the IEEE AVSS2011, the European Signal Processing Conference (EUSIPCO 2008), andWIAMIS 2010. He has published more than 130 journal and conferencepapers, one monograph on Video Tracking (Wiley, 2011) and three editedbooks: Multi-Camera Networks (Elsevier, 2009), Analysis, Retrieval andDelivery of Multimedia Content (Springer, 2012), and Intelligent MultimediaSurveillance (Springer, 2013).
