Image Classification for Automatic Annotation Jianping Fan Department of Computer Science University...

Image Classification for Automatic Annotation

Jianping FanDepartment of Computer Science

University of North Carolina at Charlotte

http://www.cs.uncc.edu/~jfan

Input Image Salient Objects

Visual FeaturesColor histogram, Tamura texture, Locations …….

Feature Extraction

Object-Based Approach

Feature Extraction

Image-Based Approach

Image Representation

Salient Objects

Color histogram

Tamura TextureShape

Images

Color histogramWavelet Texture histogramInterest Point set

Feature-Based Image Representation

Feature dimension

Feature dimension

Feature dimension Curse of Dimensionality


Tree-Based Database Indexing


Tree-Based Database Indexing

Overlapping between Different nodes

Nearest Neighbor Search

What’s the solution?


Where idea for tree-based indexing come from?

Library:

12,000,000 books

I get it!

Too easy!11!

Books in Library

Natural Sciences Social Sciences

DancingComputer Science

ElectricalEngineering

Computer Languages Researches

Database Multimedia



What is the solution?

Concept Hierarchy or Ontology

Hierarchical Concept Organization

Concept Ontology

Hierarchical Concept Organization

Concept Ontology

Beach

Water & Sky

Water & Sand Water, Sand, Sky, …

Feature Subset 1

Feature Subset 2

Feature Subset 9

Atomic Image Concept

Different Patternsof Co-Appearances of Salient Objects

Feature Subsets

122

nCM nn

i

in

6. Classifier Training for Atomic Image

Concepts

6. Classifier Training for Atomic Image Concepts

Curse of Dimensionality # samples needed increase with #

dimensions (generally exponentially) . Human labeling is expensive Some features are redundant

Proposal Joint SVM boosting and feature

selection

Boosting SVM Classifier TrainingPCA PCA PCA PCA

Subspace 1 Subspace 2 Subspace 3 … Subspace N

Higher Level Classifier

Weak Classifier 1

SV

M

…

Boosting for optimal combination

Weak Classifier 2 Weak Classifier 3 Weak Classifier N

SV

M

SV

M

SV

M

•Less training samples due to dimension reduction

•Reuse training results on low-level concepts

•More selection opportunities compared to filter and wrapper

Low-level classifiers

High-level classifier

High dimensional feature space


Kernel-Based Data Warping

)(X

Kernel Function: )()(),( jT

iji XXXX


Kernel for Color Histogram

N

i ii

ii

vu

vuvu

1

22 )(

2

1),(

Statistical Image Similarity

Kernel

/),(2),( vuevu


Wavelet Filter Bank Kernel


Wavelet Filter Bank Kernel

n

iijii yhxh

eyx 1

2 /))(),((

),(

n

i

yhxh iiiie1

/))(),((2


Interest Point Matching Kernel


Interest Point Matching Kernel


Multiple Kernel Learning

1

^

),(),(i

ii yxyx 11

i

i

SVM Image Classifier

M

llll bXXYgXf

1

^

),(sin)(


Dual Problem

M

ll

hh cw

1

2

122

1min

Subject to:

lh

lkilMl bXwY

1)(,,0:

11

6. Classifier Training for Atomic Image

Concepts


Garden Scene

Beach Scene

Some Results

High-Level Image Concept Modeling

Inter-Concept Similarity Modeling

Nature Scene

Garden Beach Flower View

Nature Scene: Larger Hypothesis Space & Large Variations of Visual Properties!

Garden, Beach, Flower view: Different but share common visual properties!

7. Classifier Training for High-Level Image Concepts

Challenging Problems

Error Transmission Problems

Training Cost Issue

Knowledge Transferability and Task Relatedness Exploitation

Error Transmission Problem


The classifiers for low-level image concepts cannot recover the errors for the classifiers of high-level image concepts!

Error Transmission Problem


Outdoor


Errors for the classifiers of atomic image concepts may be transmitted to the classifiers for the high-level image concepts!


Training Cost Issue Multiple Hypotheses

garden

outdoor

beach flower view

Large Diversity of Contents


Knowledge Transferability & Task Relatedness Exploitation

Outdoor


They are different but strongly related!

Multi-Task Learning

Which tasks are strongly related?

How to quantify the task relatedness?

How to integrate such task relatedness for training large-scale related image classifiers?

7. Classifier Training for High-Level Image

Concepts

Related Learning Tasks

Nature Scene


They are different but strongly related!

Concept Ontology can provide a good environment for multi-task learning!


Concepts

Related Learning Tasks


Concepts


Relatedness Modelling

bXWXf TjC j

)( jj VWW 0

garden

outdoor

beach flower view

0W: Common Prediction Structure


Joint Objective Function

C

j

N

i

C

jjij WCV

C1 1 1

2

02

21min

Subject to:

ijijjijNi

Cj bXVWY 1)(: 011

Dual Problem


C

j

C

j

N

i

C

h

N

ljlihjhjljlihih

N

iij XXKYY

1 1 1 1 11

),(2

1max

Subject to:

0,0:1 1

11

C

j

N

iijijij

Cj

Ni YC

Biased Classifier Training


m

ll

Tl bXWYWW

1

2

0 )](1[2

1min

Dual Problem

m

l

m

h

m

ll

Tllh

Tlhlhl XWYXXYY

1 1 10 )1(

2

1min

Subject to:

0,0:1

1

l

m

lll

ml YC

Common Prediction Structure

Nature Scene


Common Visual Properties


Concepts

)()()(1

1

XfXpXHjk C

C

jjC

1

1

))(exp(

))(exp()( C

jC

C

j

Xf

XfXp

j

j

Hierarchical Boosting


Concepts

Biased Classifier for Parent Node


ll

m

ll XYWW

1

0

bXWXf TCk

)(

Hierarchical Boosting to Generate Classifier for Parent Node


)()()(1

1

XfXpXHjk C

C

jjC

1

1

))(exp(

))(exp()( C

jC

C

j

Xf

XfXp

j

j

Performance Evaluation


Performance Evaluation


Advantages of Hierarchical Boosting

Handling inter-concept similarity via multi-task learning

Reducing training cost

Enhancing discrimination power of the classifiers


Concepts

8. Hierarchical Image Classification

Overall Probability

Parent Node

Children Node 1 Children Node 2 Children Node C

Path 1 Path 2 Path C

)( kC

)( kC

)( 1C )( 2C )( cC

CjCCC jkk ,....1|)(max)()(

Some Results

8. Hierarchical Image

Classification

10. Query Result Evaluation

Allow Users to See Global View!


Allow Users to See Similarity Direction!


Allow Users to Zoom into Images of Interest!

10. Query Result Evaluation: Red Flower

Allow Users to Select Query Example Interactively!

10. Query Result Evaluation: Sunset

Allow Users to Look for Particular Images!

11. Training Image Observation



Date post:	17-Jan-2016
Category:	Documents
Upload:	kelley-king
View:	215 times
Download:	0 times

Image Classification for Automatic Annotation Jianping Fan Department of Computer Science University...

Documents