Image Classification for Automatic Annotation Jianping Fan Department of Computer Science University...

transcript

Image Classification for Automatic Annotation

Jianping FanDepartment of Computer Science

University of North Carolina at Charlotte

http://www.cs.uncc.edu/~jfan

Input Image Salient Objects

Visual FeaturesColor histogram, Tamura texture, Locations …….

Feature Extraction

Object-Based Approach

Feature Extraction

Image-Based Approach

Image Representation

Salient Objects

Color histogram

Tamura TextureShape

Images

Color histogramWavelet Texture histogramInterest Point set

Feature-Based Image Representation

Feature dimension

Feature dimension Curse of Dimensionality

Tree-Based Database Indexing

Overlapping between Different nodes

Nearest Neighbor Search

What’s the solution?

Where idea for tree-based indexing come from?

Library:

12,000,000 books

I get it!

Too easy!11!

Books in Library

Natural Sciences Social Sciences

DancingComputer Science

ElectricalEngineering

Computer Languages Researches

Database Multimedia

What is the solution?

Concept Hierarchy or Ontology

Hierarchical Concept Organization

Concept Ontology

Hierarchical Concept Organization

Concept Ontology

Water & Sky

Water & Sand Water, Sand, Sky, …

Feature Subset 1

Feature Subset 2

Feature Subset 9

Atomic Image Concept

Different Patternsof Co-Appearances of Salient Objects

Feature Subsets

nCM nn

6. Classifier Training for Atomic Image

Concepts

6. Classifier Training for Atomic Image Concepts

Curse of Dimensionality # samples needed increase with #

dimensions (generally exponentially) . Human labeling is expensive Some features are redundant

Proposal Joint SVM boosting and feature

selection

Boosting SVM Classifier TrainingPCA PCA PCA PCA

Subspace 1 Subspace 2 Subspace 3 … Subspace N

Higher Level Classifier

Weak Classifier 1

Boosting for optimal combination

Weak Classifier 2 Weak Classifier 3 Weak Classifier N

•Less training samples due to dimension reduction

•Reuse training results on low-level concepts

•More selection opportunities compared to filter and wrapper

Low-level classifiers

High-level classifier

High dimensional feature space

Kernel-Based Data Warping

Kernel Function: )()(),( jT

iji XXXX

Kernel for Color Histogram

Statistical Image Similarity

Kernel

/),(2),( vuevu

Wavelet Filter Bank Kernel

iijii yhxh

2 /))(),((

yhxh iiiie1

/))(),((2

Interest Point Matching Kernel

Multiple Kernel Learning

),(),(i

ii yxyx 11

SVM Image Classifier

llll bXXYgXf

),(sin)(

Dual Problem

Subject to:

lkilMl bXwY

1)(,,0:

6. Classifier Training for Atomic Image

Concepts

Garden Scene

Beach Scene

Some Results

High-Level Image Concept Modeling

Inter-Concept Similarity Modeling

Nature Scene

Garden Beach Flower View

Nature Scene: Larger Hypothesis Space & Large Variations of Visual Properties!

Garden, Beach, Flower view: Different but share common visual properties!

7. Classifier Training for High-Level Image Concepts

Challenging Problems

Error Transmission Problems

Training Cost Issue

Knowledge Transferability and Task Relatedness Exploitation

Error Transmission Problem

The classifiers for low-level image concepts cannot recover the errors for the classifiers of high-level image concepts!

Error Transmission Problem

Outdoor

Errors for the classifiers of atomic image concepts may be transmitted to the classifiers for the high-level image concepts!

Training Cost Issue Multiple Hypotheses

garden

outdoor

beach flower view

Large Diversity of Contents

Knowledge Transferability & Task Relatedness Exploitation

Outdoor

They are different but strongly related!

Multi-Task Learning

Which tasks are strongly related?

How to quantify the task relatedness?

How to integrate such task relatedness for training large-scale related image classifiers?

7. Classifier Training for High-Level Image

Concepts

Related Learning Tasks

Nature Scene

They are different but strongly related!

Concept Ontology can provide a good environment for multi-task learning!

Concepts

Related Learning Tasks

Concepts

Relatedness Modelling

bXWXf TjC j

)( jj VWW 0

garden

outdoor

beach flower view

0W: Common Prediction Structure

Joint Objective Function

jjij WCV

C1 1 1

Subject to:

ijijjijNi

Cj bXVWY 1)(: 011

Dual Problem

ljlihjhjljlihih

iij XXKYY

1 1 1 1 11

Subject to:

0,0:1 1

iijijij

Biased Classifier Training

Tl bXWYWW

0 )](1[2

Dual Problem

Tlhlhl XWYXXYY

1 1 10 )1(

Subject to:

Common Prediction Structure

Nature Scene

Common Visual Properties

Concepts

)()()(1

XfXpXHjk C

))(exp(

))(exp()( C

Hierarchical Boosting

Concepts

Biased Classifier for Parent Node

ll XYWW

bXWXf TCk

Hierarchical Boosting to Generate Classifier for Parent Node

)()()(1

XfXpXHjk C

))(exp(

))(exp()( C

Performance Evaluation

Advantages of Hierarchical Boosting

Handling inter-concept similarity via multi-task learning

Reducing training cost

Enhancing discrimination power of the classifiers

Concepts

8. Hierarchical Image Classification

Overall Probability

Parent Node

Children Node 1 Children Node 2 Children Node C

Path 1 Path 2 Path C

)( 1C )( 2C )( cC

CjCCC jkk ,....1|)(max)()(

Some Results

8. Hierarchical Image

Classification

10. Query Result Evaluation

Allow Users to See Global View!

Allow Users to See Similarity Direction!

Allow Users to Zoom into Images of Interest!

10. Query Result Evaluation: Red Flower

Allow Users to Select Query Example Interactively!

10. Query Result Evaluation: Sunset

Allow Users to Look for Particular Images!

11. Training Image Observation

Image Classification for Automatic Annotation Jianping Fan Department of Computer Science University...

Documents