ACM Multimedia 2007ACM Multimedia 2007
Guo-Jun Qi, Guo-Jun Qi, Xian-Sheng HuaXian-Sheng Hua, Yong Rui, Jinhui Tang, Tao Mei and , Yong Rui, Jinhui Tang, Tao Mei and Hong-Jiang ZhangHong-Jiang Zhang
Microsoft Research AsiaMicrosoft Research Asia
September 25, 2007September 25, 2007
MotivationMotivation Correlative Multi-Label AnnotationCorrelative Multi-Label Annotation Modeling correlationsModeling correlations Learning the classifierLearning the classifier Connections to Gibbs Random FieldConnections to Gibbs Random Field
Experiments Experiments Live DemoLive Demo
2
How many images and videos in the How many images and videos in the world?world?
3
May 2007: 500
millionsAug. 2007 : 1
billion2000 images
/minute
Sep. 2007 : 84
millions
70 - 80’ Manual Labeling
90’ Pure Content Based (QBE)
Now Automated Annotation
Year
Manual
Automatic
Learning-Based
1970 1980 1990 2000
Now Automated Annotation
Learning-Based
Modeling and
Learning
Classifier
Training samples
Features
Learning-based video annotation schemes
Person
Grass
Tree
Building
Road
Face
New sampleLake?
A typical strategy – A typical strategy – Individual Concept Individual Concept DetectionDetection
Annotate multiple concepts separatelyAnnotate multiple concepts separately
6
Low-Level Features
Outdoor Face PersonPeople-
MarchingRoad
Walking- Running
-1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1
7
√ Person√ Street√ Building
× Beach× Mountain
√ Crowd√ Outdoor√ Walking/Running
√ Marching? Marching
Low-Level Features
Outdoor Face PersonPeople-Marchin
gRoad
Walking- Running
-1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1
8
Low-Level Features
Outdoor Face PersonPeople-Marchin
gRoad
Walking- Running
-1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1
Concept Model Vector
Score Score Score Score Score Score
Concept Fusion
Another typical strategy – Another typical strategy – Fusion-BasedFusion-Based Context Based Concept fusion (CBCF)Context Based Concept fusion (CBCF)
9
Low-Level Features
Outdoor Face PersonPeople-
MarchingRoad
Walking- Running
-1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1
Concept Fusion
Concept Model Vector
Score Score Score Score Score Score
10
Our strategy – Our strategy – Integrated Concept Integrated Concept DetectionDetection
Correlative Multi-Label Learning (CML)Correlative Multi-Label Learning (CML)
-1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1
Low-Level Features
OutdoorPeople-
MarchingRoadFace Person
Walking- Running
11
Multi-Label Annotation
No correlation
Has Correlations, but uses a second step
Model concepts and correlations in one step
Individual Detectors
Fusion Based
Integrated
1st Paradigm
2nd Paradigm
3rd Paradigm
Our strategy – Our strategy – Integrated Concept Integrated Concept DetectionDetection
Correlative Multi-Label Learning (CML)Correlative Multi-Label Learning (CML)
13
How to model concepts and the How to model concepts and the correlations among concept in a single correlations among concept in a single stepstep
NotationsNotations
14
Modeling concept and correlations Modeling concept and correlations simultaneouslysimultaneously
1 1
15
6.0,5.0,4.0,3.0,2.0,1.0x
1:,1:,1:,1:,1: treecarbeachroadperson y
02.002.0
1.0001.0
01.0
12,2
12,2
11,2
11,2
13,1
13,1
12,1
12,1
11,1
11,1
NoYesconceptfeature
/,
-
Modeling concept and correlations Modeling concept and correlations simultaneouslysimultaneously
1 1
16
6.0,5.0,4.0,3.0,2.0,1.0x
1:,1:,1:,1:,1: treecarbeachroadperson y
0010
00011,1
3,11,1
3,11,1
3,11,13,1
112,1
112,1
1,12,1
1,12,1
，，
NYNYConceptConcept
/,/2,1
-
Modeling concept and correlationsModeling concept and correlations
17
12 KDK
Learning the classifierLearning the classifier
Misclassification Error
Loss function
Empirical risk
Regularization
Introduce slackvariables
Lagrange dual
Find solution by SMO
18
Connection to Gibbs Random FieldConnection to Gibbs Random Field
Define a random field
19
Rewrite the classifier
is a random field
consists of all adjacent sites, that is, this RF is fully connected
Define energy functionDefine GRF
Connection to Gibbs Random FieldConnection to Gibbs Random Field
Rewrite the classifier
20
Define energy function
Intuitive explanation of CML
Define a random field
is a random field
consists of all adjacent sites, that is, this RF is fully connected
Define GRF
ExperimentsExperiments TRECVID 2005 dataset (170 hours)TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite)39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%)Training (65%), Validation (16%), Testing (19%)
21
ExperimentsExperiments TRECVID 2005 dataset (170 hours)TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite)39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%)Training (65%), Validation (16%), Testing (19%) CML (CML (MAP=0.290MAP=0.290) improves IndSVM () improves IndSVM (MAP=0.246MAP=0.246) 17% and CBCF ) 17% and CBCF
((MAP=0.253MAP=0.253) 14%) 14%
22
CMLCBCFSVM
SVM CML ↑ 17%CBCF CML ↑14%
ExperimentsExperiments TRECVID 2005 dataset (170 hours)TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite)39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%)Training (65%), Validation (16%), Testing (19%) CML (CML (MAP=0.290MAP=0.290) improves IndSVM () improves IndSVM (MAP=0.246MAP=0.246) 17% and CBCF ) 17% and CBCF
((MAP=0.253MAP=0.253) 14%) 14%
23
CMLCBCFSVM
SVM CML ↑ 131%CBCF CML ↑128%
ExperimentsExperiments TRECVID 2005 dataset (170 hours)TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite)39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%)Training (65%), Validation (16%), Testing (19%) CML (CML (MAP=0.290MAP=0.290) improves IndSVM () improves IndSVM (MAP=0.246MAP=0.246) 17% and CBCF ) 17% and CBCF
((MAP=0.253MAP=0.253) 14%) 14%
24
CMLCBCFSVM CMLCBCFSVM CMLCBCFSVM
ExperimentsExperiments TRECVID 2005 dataset (170 hours)TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite)39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%)Training (65%), Validation (16%), Testing (19%) CML (CML (MAP=0.290MAP=0.290) improves IndSVM () improves IndSVM (MAP=0.246MAP=0.246) 17% and CBCF ) 17% and CBCF
((MAP=0.253MAP=0.253) 14%) 14%
25
26
Correlative Multi-Label Video AnnotationCorrelative Multi-Label Video Annotation A new paradigm for multi-label annotationA new paradigm for multi-label annotation Models correlations and concepts Models correlations and concepts
simultaneouslysimultaneously Has a close connection to Gibbs Random FieldHas a close connection to Gibbs Random Field
27
Multi-Instance Multi-Label AnnotationMulti-Instance Multi-Label Annotation Exploit correlations among concepts and among Exploit correlations among concepts and among
instances at the same timeinstances at the same time Not only can get image/frame level annotation, Not only can get image/frame level annotation,
but also can get region level annotationbut also can get region level annotation
28
Sky
MountainWater
Sands
Scenery
29
Correlative Multi-Label Video AnnotationCorrelative Multi-Label Video Annotation A new paradigm for multi-label annotationA new paradigm for multi-label annotation Models correlations and concepts Models correlations and concepts
simultaneouslysimultaneously Has a close connection to Gibbs Random FieldHas a close connection to Gibbs Random Field
30
