Canonical correlation analysis (CCA)
Proposed method: SemiCCA
[WeBCT8.14] International Conference on Pattern Recognition (ICPR2010)
SemiCCA: Efficient semi-supervised learning of canonical correlationsAkisato Kimura (1),Hirokazu Kameoka (1),Masashi Sugiyama (2),Takuho Nakano (1,3),
Eisaku Maeda (1),Hitoshi Sakano (1),Katsuhiko Ishiguro (1)
(1) NTT Communication Science Laboratories, NTT Corporation, Japan(2) Tokyo Institute of Technology, Japan (3) the University of Tokyo, Japan
<Abstract> Semi-supervised variant of canonical correlation analysis (CCA)1. Incorporating additional unpaired samples for mitigating overfitting
even when the number of paired samples is quite limited2. Can be computed efficiently (just by solving a single eigenvalue problem)
smoothly bridges the eigenvalue problems of CCA and PCA
Linear projection
Linear projection
Observation 1 Observation 2
Maximizing correlations
Multi-label predictionFeature vectors Sets of class labels
Multi-modal relationshipImage features Audio features
Paired
Unpaired
Application to automatic image annotation
Problem :
The number of paired samplesis often limited overfitting
(1) CCA: a method of finding bases: maximizing the correlationamong projected vectors
(Mean = 0 for bravity)
(Covariance matrices of paired samples)
(2) Taking derivatives of Lagrangean with respect to , we obtain
Matrix form
PCA for paired& unpaired samples in each domain
SemiCCAFormulated as a generalized eigenvalueproblem ----- almost the same as that for CCA
Experiments with artificial data
Projection basis for discrimination
Discriminant boundary
Linear discriminant function
Framework Calculus
Paired
Framework
How to incorporate unpaired samples ?Needs some assumptions for the nature of unpaired samples A global structure in each domain revealed
by unpaired samples would be consistent with co-occurrence information.
Calculus
Subspace estimated by CCA in case all unpaired samples in X and Y are paired
Subspace estimated by semiCCA from both unpaired and paired samples
paired samplesunpaired samples in Xunpaired samples in YSubspace estimated
from paired samples
Smoothlybridging
: CCA for paired samples
: PCA for paired & unpaired samplesin each domain
: inherits both the propertiesglobal structure in each domainco-occurrence information
Relationships among previous researchesMultivariate analysis
CCA
FDAMLR
PCA
MLR := multiple linear regression,FDA := Fisher linear discriminant analysis
Semi-supervised variant
SemiCCA
SELF[Sugiyama 2009]SemiMLR?
SemiPCA?
Semi-supervised variant of CCA• Graph Laplacian regularization
[Blaschko 2008]• Tikhonov regularization
[Hardoon 2004]• SemiCCA = generalization of
Tikhomov regularization(when considering kernelization)
Contact: Akisato Kimura (NTT CS Labs, [email protected])
Method [Nakayama 2008]SemiCCA is applied here.
ConditionsDataset: VOC2008/2009 dataset
(removed all the bounding boxes)Labeled images = 500 (from 2008 training)Unlabeled images = 13743
(4096 from 2008 training,9647 from 2009 training/test)
Test images = 500 (from 2008 training)
Result
ConditionsSamples: 10000 samples drawn from a Gaussian topic model
--- Latent variable Z : drawn from the normal Gaussian--- Gaussian parameters for p(X|Z) and p(Y|Z) were drawn
randomly from the normal Gaussian for each trial--- Dimension: X=15, Y=20, Z=10
Generating unpaired samples:--- remove several Y samples with the following linear
discriminant function
Average evaluation score
Average trade-off parameter
Histograms of trade-off parameters
Discriminant boundary
Evaluation measure: Weighted sum of cosine distances--- “True” eigenvectors and eigenvalues: