16 November 2011 1 CIARP 2011 1
The dissimilarity representation for structural pattern recognition
CIARP, Pucón, Chile, 15-18 November 2011
Robert P.W. Duin, Delft University of Technology (In cooperation with Elżbieta Pȩkalska, Univ. of Manchester)
Pattern Recognition Lab
Delft University of Technology, The Netherlands
PRLab.TUDelft.nl
Introduction
16 November 2011 3 CIARP 2011
Preview
3
A B
?
How to classify structures given examples?
A B
By the dissimilarity representation: An extension of template matching, based on generalized of kernels
16 November 2011 4 CIARP 2011
Real world objects and events
4
Images Spectra Time signals Gestures
shapes
How to build a representation? Features Structure
16 November 2011 5 CIARP 2011 5
Blob Recognition
446 binary images, varying size, e.g.: 100 x 130 Andreu, G., Crespo, A., Valiente, J.M.: Selecting the toroidal self-organizing feature maps (TSOFM) best organized to object recogn. In: ICNN. (1997) 1341–1346.
Shape classification by weighted-edit distances (Bunke) Bunke, H., Buhler, U.: Applications of approximate string matching to 2D shape recognition. Pattern recognition 26 (1993) 1797–1812
BACK BREAST DRUMSTICK THIGH-AND-BACK WING
16 November 2011 6 CIARP 2011 6
Colon Tissue Recognition
normal pathological ???
16 November 2011 7 CIARP 2011
Volcano / Seismic Signal Classification
Volcano-Tectonic Long Period 150 000 events (1994 – 2008) 5 volcanos 40 stations 15 classes
J. Makario, INGEOMINAS, Manizales, Colombia
M. Orozco-Alzate, Nat. Univ. Colombia, Manizales
R. Duin, TUDelft
M. Bicego, Univ. of Verona, Italy
Cenatav, Havana, Cuba
7
16 November 2011 8 CIARP 2011 8
Gesture Recognition
Is this gesture in the database?
16 November 2011 9 CIARP 2011 9
Pattern Recognition System
Representation Generalization Sensor
B
A
B
A
perimeter
are
a
perimeter
area
Feature Representation
Representation
16 November 2011 11 CIARP 2011 11
Compactness
The compactness hypothesis is not sufficient for perfect classification as dissimilar objects may be close. class overlap probabilities
Representations of real world similar objects are close. There is no ground for any generalization (induction) on representations that do not obey this demand.
1x
2x
(perimeter)
(area)
A.G. Arkedev and E.M. Braverman, Computers and Pattern Recognition, 1966.
16 November 2011 12 CIARP 2011 12
True Representations
no probabilities needed, domains are sufficient!
1x
2x
(perimeter)
(area)
Similar objects are close and
Dissimilar objects are distant.
16 November 2011 13 CIARP 2011 13
Pattern Recognition System
Representation Generalization Sensor
B
A
B
A
perimeter
are
a
perimeter
area
Feature Representation
16 November 2011 14 CIARP 2011 14
Pattern Recognition System
Representation Generalization Sensor
B
A
B
A
pixel_1
pix
el_
2
Pixel Representation
16 November 2011 15 CIARP 2011 15
Pattern Recognition System
Representation Generalization Sensor
B
A
B
A
D(x,xA1)
D(x
,xB
1)
Dissimilarity Representation
16 November 2011 16 CIARP 2011
Structural Representation
16
X = (x1, x2, .... , xk) Y = (y1, y2, .... , yn)
A
D
E B
F
C
E
D
C
B F
Strings
Graphs
16 November 2011 17 CIARP 2011
Goldfarb’s Evolving Transformation System (ETS)
17
- Generate for each class how the objects could evolve from common primitive objects. - Test how new objects could most easily evolve from the generated trees.
Lev Goldfarb 1984: features dissimilarities PE spaces
1995: vector spaces are not good for representing concepts concepts need a structural representation
ETS
Goldfarb et al., What is a structural representation? Tech. rep. 2004 http://www.cs.unb.ca/tech-reports/documents/tr04-165.pdf
16 November 2011 18 CIARP 2011
Structural Representation
18
A
D
E B
F
C
E
D
C
B F
How to generalize? Distances!
A B
Dissimilarities
16 November 2011 20 CIARP 2011 20
Examples Dissimilarity Measures
A B
Dist(A,B):
a A, points of A b B, points of B d(a,b): Euclidean distance
D(A,B) = max_a{min_b{d(a,b)}}
D(B,A) = max_b{min_a{d(b,a)}}
Hausdorff Distance (metric): DH = max{max_a{min_b{d(a,b)}} , max_b{min_a{d(b,a)}}}
Modified Hausdorff Distance (non-metric): DM = max{mean_a{min_b{d(a,b)}},mean_b{min_a{d(b,a)}}}
max B
A
max
B
A
D(A,B) ≠ D(B,A)
Dubuisoon & Jain, Modified Hausdorff distance for object matching, ICPR12, 2004,, voll 1, 566-568.
16 November 2011 21 CIARP 2011 Representation
Dissimilarities – Possible Assumptions
1. Positivity: dij 0
2. Reflexivity: dii = 0
3. Definiteness: dij = 0 iff objects i and j are identical
4. Symmetry: dij = dji
5. Triangle inequality: dij < dik + dkj
6. Compactness: if the objects i and j are very similar then dij < d.
7. True representation: if dij < d then the objects i and j are
very similar.
8. Continuity of d.
Me
tric
16 November 2011 22 CIARP 2011
The class of problems
• Compact
• Uniquely labelled
22
In a dissimilarity representation such classes are separable by any positive definite distance measure
16 November 2011 23 CIARP 2011 23 December 2012 23
D = 0.3 D = 0.3
D = 1
David W. Jacobs, Daphna Weinshall and Yoram Gdalyahu, Classification with Nonmetric Distances: Image
Retrieval and Class Representation, IEEE Trans. Pattern Anal. Mach. Intell, 22(6), pp. 583-600, 2000.
16 November 2011 24 CIARP 2011 24
Euclidean - Non Euclidean - Non Metric
B
C
A
D
1 0
1 0
10
5 .1
5 .1
5 .1
B
C
A
D
1 0
1 0
1 0
5 .8
5 .8
5 .8
B
C
A
D
1 0
1 0
10
4
4
4
16 November 2011 25 CIARP 2011 25
Non-metric distances
14.9
7.8 4.1
object 78
objec t 419
object 425
Bunke’s Chicken Datase t
D(A,C)A
B
C
D (A,C) > D(A,B ) + D(B ,C)
D(A,B ) D(B,C)
A
B–
x
A B
AB
C
Weighted-edit distance for strings Single-linkage clustering
2
B
2
A
2
BAB)J(A,
0C)J(A, largeB)J(A,
B)J(A,smallB)J(C,
Fisher criterion
16 November 2011 26 CIARP 2011 26
Intrinsicly Non-Euclidean Dissimilarity Measures Single Linkage
Distance(Table,Book) = 0
Distance(Table,Cup) = 0
Distance(Book,Cup) = 1
D(A,C)A
B
C
D (A,C) > D(A,B ) + D(B ,C)
D(A,B ) D(B,C)
Single-linkage clustering
16 November 2011 27 CIARP 2011 27
Intrinsicly Non-Euclidean Dissimilarity Measures Invariants
Object space
Non-metric object distances due to invariants
A
B
C
D(A,C) > D(A,B) + D(B,C)
16 November 2011 28 CIARP 2011
Indefinite Metric and the 1NN rule
28
Indefinite metric: dij = 0 for objects i and j that are not identical
Possibly different labels
Template matching and 1-NN rule may fail!
Dissimilarity Representation
16 November 2011 30 CIARP 2011 30
Alternatives for the Nearest Neighbor Rule
77 76 75 74 7372 71
67 66 65 64 63 62 61
57 56 55 54 53 52 51
47 46 45 44 43 42 41
37 36 35 34 33 32 31
27 26 25 24 23 22 21
17 16 15 14 13 12 11
T
ddddddd
ddddddd
ddddddd
ddddddd
ddddddd
ddddddd
ddddddd
D
Dissimilarities dij between
all training objects
Training set B
A
) d d d d d d d (dx7x6x5x4x3x2x1x
Unlabeled object x to be classified
1. Dissimilarity Space 2. Embedding
Pekalska, The dissimilarity representation for PR. World Scientific, 2005.
16 November 2011 31 CIARP 2011 31
Alternative 1: Dissimilarity Space
77 76 75 74 7372 71
67 66 65 64 63 62 61
57 56 55 54 53 52 51
47 46 45 44 43 42 41
37 36 35 34 33 32 31
27 26 25 24 23 22 21
17 16 15 14 13 12 11
T
ddddddd
ddddddd
ddddddd
ddddddd
ddddddd
ddddddd
ddddddd
D
) d d d d d d d (dx7x6x5x4x3x2x1x
r1 r2 r3
Dissimilarities
Selection of 3 objects for representation
B
A
r1(d1)
r2(d4)
r3(d7)
Given labeled training set
Unlabeled object to be classified
16 November 2011 32 CIARP 2011 32
Prototype Selection: Polygon Dataset
The classification error as a function of the number of selected prototypes. For 10-20
prototypes results are already better than by using 1000 objects in the NN rules.
Pekalska et al., Prototype selection for dissimilarity-based classification, Pattern Recognition, 2006, 189-208.
16 November 2011 33 CIARP 2011
Dissimilarity space properties
33
• Euclidean by postulation
• Dissimilarity character not used
• Any classifier may be used
• May be filled by additional training objects • (just a limited set of objects needed for representation)
• Control of computational complexity
16 November 2011 34 CIARP 2011 34
Alternative 2: Embedding
Training set
B
A Dissimilarity matrix D X
Is there a feature space for which Dist(X,X) = D ?
1x
2x
Position points in a vector space such that their Euclidean distances D
16 November 2011 35 CIARP 2011 35
Embedding
16 November 2011 36 CIARP 2011 36
(Pseudo-)Euclidean Embedding
mm D is a given, imperfect dissimilarity matrix of training objects.
Construct inner-product matrix:
Eigenvalue Decomposition ,
Select k eigenvectors: (problem: Lk< 0)
Let k be a k x k diag. matrix, k(i,i) = sign(Lk(i,i))
Lk(i,i) < 0 Pseudo-Euclidean
nm Dz is the dissimilarity matrix between new objects and the training set.
The inner-product matrix:
The embedded objects:
JJDB(2)
2
1 11
m
1IJ
TQQB L
2
1
kkQX L
)JD-J(DB)2(T
n
1(2)
z2
1
z11
kkkz
2
1
QBZ L
16 November 2011 37 CIARP 2011 37
PES: Pseudo-Euclidean Space (Krein Space)
If D is non-Euclidean, B has p positive and q negative eigenvalues.
A pseudo-Euclidean space ε with signature (p,q), k =p+q, is a non-
degenerate inner product space k = p q such that:
q
1pj
jj
p
1i
iipq
Tyxyxyxy,x
pp
pqI0
0I
)y,x(d)y,x(dyx,yx)y,x(d2
q
2
p
2
Pekalska, The Dissimilarity Representation for Pattern Recognition. Foundations and Applications. World Scientific, Singapore , 2005
16 November 2011 38 CIARP 2011 38
Pseudo Euclidean Space
22
ij i jd x x
2 22 p p q q
ij i j i jd x x x x
Pseudo Euclidean embedding D {Xp,Xq}
Euclidean embedding D X
‘Positive’ and ‘negative’ space, Compare Minkowsky space in relativity theory
16 November 2011 39 CIARP 2011
PE-space embedding properties
39
• A square matrix with dissimilarities is needed (all training (+ test objects) needed for representation)
• Projection of new objects is difficult
• Densities are not (yet) well defined
• Distance to a classifier is inappropriate for classification
16 November 2011 40 CIARP 2011
PE-Space classifiers
40
• kNN, Parzen, Nearest Mean As object distances can be computed (are known)
• LDA, QDA As PE inner possibly product definitions cancel they can be computed, interpretation … ?
• SVM
May get a result (indefinite kernel), possibly not optimal
• Others ??
16 November 2011 41 CIARP 2011 41
Examples Dissimilarity Measures
Matching new objects to various templates: class(x) = class(argminy(D(x,y))) Dissimilarity measure appears to be non-metric.
A.K. Jain, D. Zongker, Representation and recognition of handwritten digit using deformable templates, IEEE-PAMI, vol. 19, no. 12, 1997, 1386-1391.
16 November 2011 42 CIARP 2011 42
Three Approaches Compared for the Zongker Data
Dissimilarity Space equivalent to Embedding better than Nearest Neighbour Rule
Pekalska, The Dissimilarity Representation for Pattern Recognition. Foundations and Applications. World Scientific, Singapore , 2005
16 November 2011 43 CIARP 2011 43
Ball Distances
- Generate sets of balls (classes) uniformly, in a (hyper)cube; not intersecting.
- Balls of the same class have the same size.
- Compute all distances between the ball surfaces.
-> Dissimilarity matrix D
Duin et al., Non-Euclidean dissimilarities: Causes and informativeness, SSSPR 2010, 324-333.
16 November 2011 44 CIARP 2011 44
Balls3D
10 x ( 2-fold crossvalidation of 50 objects per class )
16 November 2011 45 CIARP 2011 45
Representation Strategies
Avoiding the PE space
2 2
ij p i jd d ( x , x )
X { [ X p , X q ], } 2 2 2
ij p i j q i jd d ( x , x ) d ( x , x )
As it is
Correcting
Associated space
Dissimilarity Space: X = D
Positive space
Negative space 2 2
ij q i jd d ( x , x )
pX X
qX X
2 2 2
ij p i j q i jd d ( x , x ) d ( x , x ) Pseudo Euclidean Space X {X p , X q}
Additive Correction 2 2
ij ijd d c , i j X E m b ed d in g (D )
Classifiers to be developed further
16 November 2011 46 CIARP 2011 28 October 2011 Ups and Downs in Pattern Recognition
46
Informative
Extremely Informative
Not Informative
+- + - Is the PE Space Informative?
16 November 2011 47 CIARP 2011
Multiple Dissimilarity Matrices
47
Dissimilarity Measure 1:
Dissimilarity Measure 2:
Dissimilarity Measure 3:
16 November 2011 48 CIARP 2011
Averaging of dissimilarity matrices
48
• Three procedures for graph matching compared on the Coil dataset: 4 classes (objects), 72 images per class.
• Classification errors for 25 times 10-fold crossvalidation.
CoilDelftDiff Graphs are compared in the eigenspace with a dimensionality determined by the smallest graph in every pairwise comparison by the JoEig Approach [1]. CoilDelftSame Dissimilarities in 5D eigenspace derived from the two graphs by the JoEig approach [1]. CoilYork Dissimilarities are found by graph matching, using the algorithm of Gold and Ranguranjan [2]
[1] Lee & Duin, An inexact graph comparison approach in joint Eigenspace, SSSPR 2008. [2] Gold & Rangarajan, A graduated assignment algorithm for graph matching. PAMI 18(4), 1996
Data NEF 1-NN 1-NND SVM-1
CoilDelftDiff 0.13 0.48 0.44 0.40
CoilDelftSame 0.03 0.65 0.41 0.39
CoilYork 0.26 0.25 0.37 0.33
Averaged 0.37 0.22 0.24
Dissimilarity space
Examples
16 November 2011 50 CIARP 2011 50
Example: Chickenpieces (H. Bunke, Bern)
446 binary images, varying size, e.g.: 100 x 130 Andreu, G., Crespo, A., Valiente, J.M.: Selecting the toroidal self-organizing feature maps (TSOFM) best organized to object recogn. In: ICNN. (1997) 1341–1346.
Shape classification by weighted-edit distances (Bunke) Bunke, H., Buhler, U.: Applications of approximate string matching to 2D shape recognition. Pattern recognition 26 (1993) 1797–1812
BACK BREAST DRUMSTICK THIGH-AND-BACK WING
16 November 2011 51 CIARP 2011 51
Chickenpieces: Various Dissimilarity Measures
Best classification result is for a very non-Euclidean dissimilarity measure !
Pekalska et al., On not making dissimilarities Euclidean, SSSPR 2004, 1145-1154.
16 November 2011 52 CIARP 2011
Chickenpieces: classification errors
52
Different dissimilarity measures
16 November 2011 53 CIARP 2011
Flow Cytometry
53
612 histograms in 3 classes
Nap & van Rodijnen, Atrium Hospital, Heerlen
intensity
intensity intensity
16 November 2011 54 CIARP 2011
Flow Cytometry: classification errors
54
Data Source
NEF 1-NN 1-NND SVM-1
Tube 1 0.27 0.38 0.38 0.30
Tube 2 0.27 0.37 0.37 0.29
Tube 3 0.27 0.38 0.40 0.27
Tube 4 0.27 0.42 0.42 0.30
Averaged 0.24 0.27 0.20 0.11
Dissimilarity space
16 November 2011 55 CIARP 2011
Bio-crystallization
55
image size: 2114 x 2114 Different food products / quality 2 classes, 54 examples/class
Busscher et al., Standardization of the iocrystallization Method for Carrot Samples, Biological Agriculture and Horticulture, 2010, Vol. 27, pp. 1–23
16 November 2011 56 CIARP 2011
Bio-crystallization: Dissimilarity Measures
56
Originals
Gauss L2
Laplace Abs Histogram L1
Laplace L2
16 November 2011 57 CIARP 2011
Bio-crystallization: classification errors
57
Dissimilarity Measure
NEF 1-NN 1-NND SVM-1
Gauss 0 0.329 0.266 0.106
Laplace 0 0.229 0.313 0.125
Laplace Histogram 0.067 0.107 0.172 0.072
Averaged 0.004 0.114 0.166 0.057
Dissimilarity space
16 November 2011 58 CIARP 2011 58
Gesture Recognition
Is this gesture in the database?
16 November 2011 59 CIARP 2011
Gesture Recognition
59
20 signs (classes), 75 examples/sign Distance measure: DTW
Dissimilarity Space PCA
Lichtenauer et al. Sign language recognition by combining statistical DTW and independent classification. PAMI 2008, 2040–2046.
16 November 2011 60 CIARP 2011
Application: Graphs
60
x2
x4
x5
x3
x1
Graph with feature nodes
Taken from: Ren, Aleksic, Wilson, Hancock, A polynomial characterization of hypergraphs using the Ihara zeta function, Pattern Recognition, 2011, 1941-1957
16 November 2011 61 CIARP 2011
Interpolating structural and feature space dissimilarities
61
W.R.Lee et al., Bridging Structure and Feature Representations in Graph Matching, IJPRAI,2011, accepted
Structure only (no features)
Features only (no structure)
{x1 x2 x3 x4 x5}
Conclusion
16 November 2011 63 CIARP 2011 23 December 2012 63 WP3 Final Report
The Bridge
between structural and statistical pattern recognition offered by the dissimilarity representation
………
is a toll bridge,
to be paid by solving the non-Euclidean problem
………
The dissimilarity space may settle the fare
The Toll Bridge
