Post on 01-Jan-2016
description
transcript
February, 262007
Content-Based Image Retrieval
Saint-Petersburg State University
Natalia Vassilievanatalia@ntc-it.ru
Il’ya Markovilya.markov@gmail.com
Alexander Dolnikalexander.dolnik@gmail.co
m
February, 262007
Our team
Natalia Vassilieva Alexander Dolnik Ilya Markov Maria Teplyh Maria Davydova Dmitry Shubakov Alexander Yaremchuk
February, 262007
General problems
Semantic gap between system and human mode of image analysis Specific of human visual perception How to catch semantics of an image
Signature calculation and response time
Combining different features and metrics
February, 262007
Image retrieval system
General goal: an image retrieval system
that is able to process natural language query that is able to search among annotated and non-annotated
images that takes into account human visual perception that processes various features (color, texture, shapes) that uses relevance feedback for query refinement, adaptive
search
How to minimize “semantic gap”?
semantic low-level featuressemantic gap
February, 262007
CBIR : Traditional approachin
dexati
on
retr
ieval
signaturecalculation database
signaturecalculation
comparisonresult
image
query
Relevance feedback: query refinement
fusion of results: independent search by different features
color space partition according to human perception
auto-annotation
annotations refinement
multidimensional indexing (vp-tree)
February, 262007
Research directions
Color space partition according to human visual perception
Correspondence between low-level features and semantics: auto-annotation
Fusion of retrieval result sets
Adaptive search: color and texture fusion
Using relevance feedback
February, 262007
Human visual perception: colors
Experiments with color partition: HSV space
(H=9; S=2; V=3) – 72 %(H=11; S=2; V=3) – 66%(H=13; S=2; V=3) – 63%(H=15; S=2; V=3) – 60%
Compare partitions of different spaces (RGB, HSV, Lab)
February, 262007
Research directions
Color space partition according to human visual perception
Correspondence between low-level features and semantics: auto-annotation
Fusion of retrieval result sets
Adaptive search: color and texture fusion
Using relevance feedback
February, 262007
Auto-annotation
Natalia Vassilieva, Boris Novikov. Establishing a correspondence between low-level features and semantics of fixed images. In Proceedings of the Seventh National Russian Research Conference RCDL'2005, Yaroslavl, October 04 - 06, 2005
Training set selection
Color feature extraction for every image from the set
Similarity calculation for every pair of images from the set
Training set clustering
Basis color features calculation: one per every cluster
Definition of basis lexical features
Correspondence between basis color features and basis lexical features
February, 262007
Examples
city, night, road, river snow, winter, sky, mountain
February, 262007
Retrieve by textual query
N. Vassilieva and B. Novikov. A Similarity Retrieval Algorithm for Natural Images. Proc. of the Baltic DB&IS'2004, Riga, Latvia, Scientific Papers University of Latvia, June 2004
Image database is divided into clusters
Search for appropriate cluster by textual query using cluster’s annotations
Browse the images from the appropriate cluster
Use relevance feedback to refine the query
Use relevance feedback to reorganize the clusters and assign new annotations
February, 262007
Feature extraction: color
Color: histograms
Color: statistical approachFirst moments for color distribution (every channel) and covariations
February, 262007
Feature extraction: texture
Texture: use independent component filters that results from ICA
H. Borgne, A. Guerin-Dugue, A. Antoniadis
“Representation of images for classification with independent features”
Image I1
Image I2
…
N filtres
dist(I1,I2) = KLH(H1i , H2i)Σi=1
N
February, 262007
Research directions
Color space partition according to human visual perception
Correspondence between low-level features and semantics: auto-annotation
Fusion of retrieval result sets
Adaptive search: color and texture fusion
Using relevance feedback
February, 262007
Fusion of retrieval result sets
How to merge fairly? How to merge efficiently? How to merge effectively?
Fusion of weighted lists with ranked elements:
(x11, r1
1), (x12, r1
2), … , (x1n, r1
n)ω1
(x21, r2
1), (x22, r2
2), … , (x2k, r2
n)ω2
(xm1, rm
1), (xm2, rm
2), … , (xml,
rml)
ωm
… ?
February, 262007
Supplement fusion– union textual results (textual viewpoints )
Collage fusion– combine texture (texture viewpoint) & color
results (color viewpoint)– different color methods (different color
viewpoints)
Ranked lists fusion: application area
February, 262007
Search by textual query in partly annotated image database
Ranked lists fusion: application area
Textual query
TextResult1, textrank1
TR2, tr2,
...
…tr1
…tr2
…
by annotations
conte
nt-
base
d
Result
February, 262007
commutative propertyassociative propertyvalue of result object's rank independent of
another object's ranks
Examples:COMBSUM, COMBMIN, COMBMAX merge functions
Three main native fusion properties
February, 262007
normalization & delimitation property
conic property attraction of current object for mix result
depend on value of function g(rank, weight) ≥ 0 ;
snare condition:
Additional native fusion properties
February, 262007
g monotonically decreases with fixed weight parameter
g monotonically decreases with fixed rank parameter
g must satisfy boundaries conditions: g( 0, w ) > 0 if w != 0 g( r, 0 ) = 0
Conic properties, function g
February, 262007
Fusion formula
where
Ranked lists fusion: Formulas
February, 262007
All lists are sorted by object id
Using step by step lists merging (object id priory)
If object_id1 not equal object_id2 => some object is absent in one of the lists
Ranked lists fusion: Algorithm
List 1
List 2
Result list
Currentobject_id2
Currentobject_id1
February, 262007
Viewpoint should provide some “valuable” information. Retrieval system's performance at least should be better than a random system.
Information is not fully duplicated. There should be partial disagreement among viewpoints.
Ranked lists fusion: Experiments
Necessary conditions:
February, 262007
Roverlap && Noverlap conditions
Intercomparison of methods– Classical methods: COMBSUM, COMBMIN,
COMBMAX – Probability methods: probFuse– Random method: random values that satisfied
to merge properties.
Ranked lists fusion: Experiments
Parameters:
February, 262007
Research directions
Color space partition according to human visual perception
Correspondence between low-level features and semantics: auto-annotation
Fusion of retrieval result sets
Adaptive search: color and texture fusion
Using relevance feedback
February, 262007
Adaptive merge: color and texture
Hypothesis:
Optimal α depends on features of query Q. It is possible to distinguish common features for images that have the same “best” α.
Dist(I, Q) = α*C(I, Q) + (1 - α)*Т(I, Q),
C(I, Q) – color distance between I and Q;T(I, Q) – texture distance between I and Q; 0 ≤ α ≤ 1
February, 262007
Adaptive merge: experiments
February, 262007
Estimation tool
Web-application
Provides interfaces for developers of search-methods
Uses common measures to estimate search methods: Precision Pseudo-recall
Collects users opinions – > builds test
database
February, 262007
Datasets
Own photo collection (~2000 images)
Subset from own photo collection (150 images)
Flickr collection (~15000, ~1.5 mln images)
Corel photoset (1100 images)
February, 262007
Research directions
Color space partition according to human visual perception
Correspondence between low-level features and semantics: auto-annotation
Fusion of retrieval result sets
Adaptive search: color and texture fusion
Using relevance feedback