Date post: | 20-Dec-2015 |
Category: |
Documents |
View: | 216 times |
Download: | 0 times |
11
A Novel Log-based Relevance Feedback Technique A Novel Log-based Relevance Feedback Technique in Content-based Image Retrievalin Content-based Image Retrieval
Steven Steven Chu-Hong Hoi & Michael R. LyuChu-Hong Hoi & Michael R. Lyu
Department of CSEDepartment of CSE
The Chinese University of Hong KongThe Chinese University of Hong Kong
Shatin, Hong Kong SARShatin, Hong Kong SAR
{chhoi, lyu}@cse.cuhk.edu.hk{chhoi, lyu}@cse.cuhk.edu.hk
ACM Multimedia 200412th Annual Conference, October 10 -16, 2004
New York City, Columbia University
22Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
OutlineOutline
Introduction & MotivationIntroduction & Motivation
Log-based Relevance FeedbackLog-based Relevance Feedback
Soft Label Support Vector MachineSoft Label Support Vector Machine
Experimental ResultsExperimental Results
Conclusions and Future WorkConclusions and Future Work
33Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
IntroductionIntroduction
Content-based Image Retrieval (CBIR)Content-based Image Retrieval (CBIR)– Attract much interest, studied for many yearsAttract much interest, studied for many years– AAn important componentn important component in multimedia retrieval in multimedia retrieval– Query based on low-level visual content: color, texture,Query based on low-level visual content: color, texture,
shape, etc. shape, etc.
QBE
Challenge: the semantic gap between low-level features and high-level conceptsChallenge: the semantic gap between low-level features and high-level concepts
44Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
IntroductionIntroduction
Relevance Feedback (RF) in CBIRRelevance Feedback (RF) in CBIR– A powerful technique, attack the semantic gap problemA powerful technique, attack the semantic gap problem– Using interactive mechanisms, soliciting users’ interactions, Using interactive mechanisms, soliciting users’ interactions,
learning users’ high-level conceptslearning users’ high-level concepts– Boosting retrieval performance effectivelyBoosting retrieval performance effectively– Many popular techniques: MARS, QEX, MindReader, OptiMany popular techniques: MARS, QEX, MindReader, Opti
mizing learning, SVM (active), Boosting, etc. mizing learning, SVM (active), Boosting, etc.
ProblemsProblems– Regular relevance feedback techniques: a lot of Regular relevance feedback techniques: a lot of timestimes of feed of feed
backback which will which will cost much time cost much time and and make users boringmake users boring
55Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
MotivationMotivation
Can users’ feedback logs information be used to Can users’ feedback logs information be used to improve the regular relevance feedback? improve the regular relevance feedback?
Relevance Feedback
Users’ Feedback Logs
Problem
?
66Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
LRF: Log-based Relevance FeedbackLRF: Log-based Relevance Feedback
Problem FormulationProblem Formulation– Construct a Relevance Matrix: Construct a Relevance Matrix: RMRM
– Each log session: (Each log session: (N N = + ) = + ) NN images are marked: relevant & irrelevant instances images are marked: relevant & irrelevant instances
– Values: relevant (+1), irrelevant (-1), unknown (0)Values: relevant (+1), irrelevant (-1), unknown (0)
Log Sessions
Image samples in the image database
1 -1 1 -1 -1 0 1 -1 -1 11
-1 1 -1 -1 -1 -1 -1 1 -1-10
77Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Log-based Relevance Feedback (cont’d)Log-based Relevance Feedback (cont’d)
Relationship MeasurementRelationship Measurement– For each given session For each given session k k , if the image , if the image i i is marked as is marked as
‘‘relevantrelevant’ (positive) and the image ‘’ (positive) and the image ‘jj’ is marked as ’ is marked as ‘‘irrelevantirrelevant’ (negative), then the elements are represented as’ (negative), then the elements are represented as
RM (RM (k, ik, i) = 1 ) = 1 andand RM ( RM (k, jk, j) = -1) = -1– For every two images: For every two images: ii and and jj, their relationship can be , their relationship can be
measured by a modified correlation function:measured by a modified correlation function:
88Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
LRF AlgorithmLRF Algorithm
Collection of training SamplesCollection of training Samples– Regular relevance feedbackRegular relevance feedback
Learn only with a limited number of training samplesLearn only with a limited number of training samples
Cannot achieve good performance without enough training samplesCannot achieve good performance without enough training samples
– Idea: finding more samples based on Idea: finding more samples based on NN initial samples initial samples
– For an initial positive sample For an initial positive sample ii, the relevance degrees between , the relevance degrees between every image sample every image sample jj of the database are computed by a soft label of the database are computed by a soft label function:function:
– By ranking the soft label values, we can collect a number of samples By ranking the soft label values, we can collect a number of samples with larger soft label values corresponding to the sample with larger soft label values corresponding to the sample i.i.
99Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
LRF Algorithm (cont’d)LRF Algorithm (cont’d)
The learning issue of the algorithmThe learning issue of the algorithm– Based on the initial marked samples and the log Based on the initial marked samples and the log
information, we can collect a large number of positive and information, we can collect a large number of positive and negative training samples associated with soft labels which negative training samples associated with soft labels which represent their confidence degrees.represent their confidence degrees.
– Problem: how to develop the algorithm to learn the data associated with soft labels ?
Proposed Solution: Soft Label Learning – Soft Label Support Vector Machine (SLSVM)
1010Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
SLSVM: Soft Label Support Vector MachineSLSVM: Soft Label Support Vector Machine
Problem FormulationProblem Formulation
SVM
1
11
1
11
0.5
1111Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
SLSVM (cont’d)SLSVM (cont’d)
1212Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
SLSVM (cont’d)SLSVM (cont’d)
1313Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Experimental ResultsExperimental Results
DatasetsDatasets– Images selected from COREL image CDsImages selected from COREL image CDs– 20-Category: 2000 image instances20-Category: 2000 image instances– 50-Category: 5000 image instances50-Category: 5000 image instances– Each category contains a specific semantic meaningEach category contains a specific semantic meaning
1414Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Experimental Results (cont’d)Experimental Results (cont’d)
Image RepresentationImage Representation– Color Moment Color Moment
9-dimension9-dimension
– Edge Direction Histogram Edge Direction Histogram 18-dimension18-dimension
Canny detector, 18 bins of 20 degreesCanny detector, 18 bins of 20 degrees
– Wavelet-based texture Wavelet-based texture 9-dimension9-dimension
Daubechies-4 wavelet, 3-level DWTDaubechies-4 wavelet, 3-level DWT
9 subimages are selected to generate the feature9 subimages are selected to generate the feature
1515Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Experimental Results (cont’d)Experimental Results (cont’d)
Log FormatLog Format– Define a Log Session (LS) as a basic log unit, that Define a Log Session (LS) as a basic log unit, that
corresponds to a relevance feedback roundcorresponds to a relevance feedback round– Each log session contains 20 images marked by usersEach log session contains 20 images marked by users
Log CollectionLog Collection– Collect logs from 10 usersCollect logs from 10 users– Non-noisy logs: 100 LSNon-noisy logs: 100 LS– Noisy logs: Noisy logs:
20-Category: 103 LS, 7.2% noise20-Category: 103 LS, 7.2% noise
50-Category: 138 LS, 8.1% noise50-Category: 138 LS, 8.1% noise
1616Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Experimental Results (cont’d)Experimental Results (cont’d)
Compared SchemesCompared Schemes– EU (Euclidean distance - baseline)EU (Euclidean distance - baseline)– RF_QEX (QEX: query expansion)RF_QEX (QEX: query expansion)
Multiple instance sampling, pick Multiple instance sampling, pick NN nearest samples recursively nearest samples recursively
– RF_SVMRF_SVMRegular relevance feedback by SVMRegular relevance feedback by SVM
– LRF_QEXLRF_QEXSimilar to RF_QEX, but we pick the samples weighted by soft Similar to RF_QEX, but we pick the samples weighted by soft labels in our framework (the larger the label, the smaller the labels in our framework (the larger the label, the smaller the distance)distance)
– LRF_SLSVMLRF_SLSVM
1717Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Experimental Results (cont’d)Experimental Results (cont’d)
SettingsSettings– Same Kernels: e.g. RBF kernelSame Kernels: e.g. RBF kernel– Evaluation metric: Evaluation metric:
Average Precision = # of relevance / # of returned Average Precision = # of relevance / # of returned – Automatic evaluation: Automatic evaluation:
Taking average precision over 200 query executionsTaking average precision over 200 query executions
1818Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Experimental Results (cont’d)Experimental Results (cont’d)
Performance ComparisonPerformance Comparison
1919Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Experimental Results (cont’d)Experimental Results (cont’d)
Performance ComparisonPerformance Comparison
2020Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Experimental Results (cont’d)Experimental Results (cont’d)
Performance ComparisonPerformance Comparison
2121Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
ConclusionsConclusions
In this paper we proposed a new scheme to study In this paper we proposed a new scheme to study users’ feedback logs for improving the performance users’ feedback logs for improving the performance of regular relevance feedback in CBIR.of regular relevance feedback in CBIR.We introduce the soft label learning concept and We introduce the soft label learning concept and developed a modified SVM technique, i.e. Soft Label developed a modified SVM technique, i.e. Soft Label SVM, to construct the algorithm for log-based SVM, to construct the algorithm for log-based relevance feedback.relevance feedback.We evaluate our proposed method compared with We evaluate our proposed method compared with traditional techniques and demonstrate promising traditional techniques and demonstrate promising results.results.
2222Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Limitations & Future WorkLimitations & Future Work
The proposed LRF with SLSVM algorithm still suffers The proposed LRF with SLSVM algorithm still suffers performance drop when many noisy logs are appeared.performance drop when many noisy logs are appeared.Much noise may be involved when the scale of the image Much noise may be involved when the scale of the image database is increased. database is increased. When the number of log sessions is large, the dimension of the When the number of log sessions is large, the dimension of the relevance matrix may be a problem.relevance matrix may be a problem.Training time of SLSVM need be considered for large scale Training time of SLSVM need be considered for large scale datasets.datasets.
Open questions:Open questions:Can we work out more effective Soft Label Learning techniques Can we work out more effective Soft Label Learning techniques in the future?in the future?Can we include some noise filtering techniques into our Can we include some noise filtering techniques into our framework?framework?
2323Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
Thank You!Thank You!
Q & AQ & A
2424Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
References (part)References (part)
[He & King 2003] X. He, O. King, W.-Y. Ma, M. Li, and H. J. Zhang. Learning a semantic space from user’s relevance feedback for image retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 13(1):39–48, Jan. 2003.
[Huang & Zhou 2001] T. S. Huang and X. S. Zhou. Image retrieval by relevance feedback: from heuristic weight adjustment to optimal learning methods. In Proceedings of IEEE International Conference on Image Processing (ICIP’01), Thessaloniki, Greece, Oct. 2001.
[Hong & Huang 2000] P. Hong, Q. Tian, and T. Huang. Incorporate support vector machines to content-based image retrieval with relevant feedback. In Proc. IEEE International Conference on Image Processing (ICIP’00), Vancouver, BC, Canada, 2000.
[Rui & Huang 1999] Y. Rui and T. S. Huang. A novel relevance feedback technique in image retrieval. In Proc. ACM Multimedia (MM’99), pages 67–70, Orlando, Florida, USA, 1999.on Image Processing (ICIP’00), Vancouver, BC, Canada,
[Tong & Change 2001] S. Tong and E. Chang. Support vector machine active learning for image retrieval. In Proceedings of the ninth ACM international conference on Multimedia, pages 107–118. ACM Press, 2001.
2525Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
AppendixAppendix
Kernel ComparisonKernel Comparison
2626Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004
CBIRCBIR