RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR
SEGMENTATION AND DIMENSION REDUCTION OF FEATURES
Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu
Department of Computer Science
Kent State University
September, 2009 Kent State University 2
Data Mining and Knowledge Management
• Processing multimedia objects
• Defining and extracting features
• Feature dimension reduction
• Multimedia data retrieval
• Knowledge representation and management
September, 2009 Kent State University 3
Current Tasks
• Off-line data training– Segment images – batch mode– Find region of interest (ROI)– Interface with feature extraction and analysis– Feature domain processing
September, 2009 Kent State University 4
Current Tasks (cont.)
• Users Interfaces– Reading user-input images– Segmentation– Find ROI– Feature extraction of ROI– Compare with trained data in repository– Return data (images) satisfying certain criteria
September, 2009 Kent State University 5
Data Training
Image &Feature DataRepository
Segmentation Finding ROI Interface Feature ExtractDimension Reduction
Sending Images for Processing Store Feature Data back
Image Domain Feature Domain
September, 2009 Kent State University 6
Image Domain Procesisng
• Segmentation – Color VQ, Texture based image segmentation
• Find ROI – ROI occupies large area– ROI locates near the image center– ROI contains homogenous texture
September, 2009 Kent State University 7
Color-Texture SegmentationApplications
• Identify Regions of Interest (ROI) in a scene
• Image classification
• Image annotation
• Object based image and video coding
September, 2009 Kent State University 8
Color-Texture SegmentationCurrent Limitations
• Many existing techniques work well on homogeneous color regions, while natural scenes are rich in color and texture.
• Many texture segmentation algorithms require the estimation of texture model parameters, which is a difficult problem and often requires a good homogeneous region for robust estimation.
September, 2009 Kent State University 9
Color-Texture SegmentationAdvantage of Color VQ and Texture
based segmentation • Does not attempt to estimate a specific
model for a texture region.
• Tests for the homogeneity of a given color-texture pattern, which is computationally more feasible than estimation of model parameters.
September, 2009 Kent State University 10
Color-Texture SegmentationTwo-Step Process
• Color Quantization– Performed in the color space without
consideration of spatial distribution of colors.– Label each pixel with a quantized color to form
a class-map.
• Spatial Segmentation– Performed on the class-map
September, 2009 Kent State University 11
Color-Texture SegmentationColor Quantization
• Use Peer Group Filtering
• As a result, coarse quantization can be obtained while preserving the color information in the original images.
• Usually 10-20 colors are needed in the images of natural scenes.
September, 2009 Kent State University 12
Color-Texture SegmentationCriteria for Good Segmentation
WWT
W
C
i
C
i ZziiW
ZzT
SSSJ
S
mzSS
mzS
i
/)(
Define
class. same the tobelonging points of variance total theis
and
Let
1 1
2
2
September, 2009 Kent State University 13
Color-Texture Segmentation-A Criterion for Good Segmentation
• When the color classes are more separated from each other, J is getting larger.
• If all color classes are uniformly distributed over the entire image, J tends to be small.
September, 2009 Kent State University 14
Color-Texture SegmentationA Criterion for Good Segmentation
• Now let us recalculate J over each segmented region instead of the entire class-map and define the average by
• A segmentation which can minimize J is considered a good segmentation.
k
Kk JMN
J1
September, 2009 Kent State University 15
Color-Texture Segmentation-Spatial Segmentation
• Seed Determination
• Seed Growing
• Region Merge
September, 2009 Kent State University 16
Color-Texture Segmentation-Spatial Segmentation
September, 2009 Kent State University 17
ROI Determination
• Find ROI – Mechanism– Pixel closer to the center contributes more
weight to the region it belongs to.
– Region with more pixels tends to get higher weight
dw
1
September, 2009 Kent State University 18
Results of Image Domain Processing
• Results of Color Quantization
• Results of Finding ROI
Results of Image Domain Processing
March, 2004 Kent State University 19
V1 = 500, V2 = 1, V3 = 0.5 AutoV1 = 500, V2 = 1, V3 = 0.5
Results of Image Domain Processing
March, 2004 Kent State University 20
V1 = 500, V2 = 1, V3 = 0.5 Auto
March, 2004 Kent State University 21
September, 2009 Kent State University 22
Interface with Feature Domain
• Find the rectangle circumscribing the ROI
• Store its coordinate information into to a temporary file for feature domain’s use.
September, 2009 Kent State University 23
Feature Domain(Overview)• Two Stages:
– Feature Extraction
– Dimension Reduction (DR)
Image &Feature DataRepository
Interface
Store Feature Data back
Image Domain Feature ExtractDimension Reduction
Feature Domain
September, 2009 Kent State University 24
Implementations
• Acquire ROI information from the image domain
• Extract features based on Gabor Filter and color histogram on HSV space
• Integrate two feature spaces
• Reduce the high feature dimensions to a very low number
September, 2009 Kent State University 25
Implementations (cont.)
• Calculate the similarity measurement between the query object and the objects in the image repository
• Search the similar images in the repository based on similarity index
• Output the corresponding retrieval images
• Knowledge extraction
September, 2009 Kent State University 26
Feature Extraction Algorithm
• Gabor Filter Feature– One of the most important wavelets with multi-
scale and multi-resolution – Mainly reflect texture information
• Color histogram on HSV space– Provide color features
September, 2009 Kent State University 27
Gabor Filter Concept
• A complete but non-orthogonal basis wavelet set
• A significant aspect: localized frequency description – composed of space information
September, 2009 Kent State University 28
Gabor(cont.)
• A two dimensional Gabor function g(x, y) and its Fourier transform G(u, v) can be written as:
jWx
yxyxg
yxyx
22
1exp
2
1),(
2
2
2
2
2
2
2
2)(
2
1exp),(
vu
vWuvuG
yvxu andwhere 2/1,2/1
September, 2009 Kent State University 29
Gabor(cont.)
• Let g(x, y) be the mother Gabor wavelet, then this self-similar filter dictionary can be obtained by appropriate dilations and rotations of g(x, y) through the generating function
number. scale themeans
ns.orientatio ofnumber total theis and , where
)cossin(),sincos(
int,,1),,(),(
m
Knππ/θ
yxayandyxax
nmayxGayxgmm
mmn
September, 2009 Kent State University 30
Color Histogram in HSV Space
• HSV color space includes – Hue (H)– Saturation (S)– Value (V or Lightness)
• Only consider Hue and saturation information, since the lightness of pictures is very sensitive to the surrounding conditions.
September, 2009 Kent State University 31
HSV space Figure
September, 2009 Kent State University 32
HSV space bands
• Design bands in the HSV space– 8 hue bands – 4 saturation bands, – Total 32 sub-spaces
• Compute color histogram feature in each sub-space to form 32 feature dimensions eventually
September, 2009 Kent State University 33
Feature Integration
• Normalize both Gabor filter and HSV color histogram features
• Set a weight factor to balance two feature spaces. Usually Gabor filter features will have the bigger weight value.
September, 2009 Kent State University 34
DR Algorithm
• Disadvantages in the high dimension space– The computational complexity arise sharply– The database indexing becomes difficult
• Principal Component Analysis (PCA) – PCA seeks to reduce the dimension of the data
by finding a few orthogonal linear combinations (Principal Component “PC”)
September, 2009 Kent State University 35
DR implementation
• Original feature dimensions– Gabor filter features: 6*5*2 = 60– HSV color histogram features: 4*8 = 32– Total dimensions: 92
• Feature dimensions after DR– 10 ~15 dimensions
September, 2009 Kent State University 36
Simulation Results in the Feature Domain
• We randomly select 11 query pictures as the test samples in this report.
• At each query time, at most 14 retrieval pictures are retrieved.
• The minimum square error method is served as the similarity measurement.
• The value in the tables as below means the positive pictures out of the 14 retrieval pictures.
September, 2009 Kent State University 37
Performance between different feature extraction techniques
• the integration of Gabor Filter and HSV color Histogram
gains the better performance. • See pictures in detail. Click here
Query pic#
1 2 3 4 5 6 7 8 9 10 11
Gabor 6 7 7 4 12 1 1 2 4 3 2
HSV 8 2 9 1 2 3 1 1 4 2 3
Integrated 10 5 11 4 12 3 3 2 5 2 4
September, 2009 Kent State University 38
Performance between with and without DR applied
• The performance after DR applied slightly degrades on average in comparison to the results before DR takes on
stage • See pictures in detail. Click here
Query pic#
1 2 3 4 5 6 7 8 9 10
11
Integrated
10 5 11 4 12 3 3 2 5 2 4
DR 9 6 5 5 12 2 1 1 4 3 2
September, 2009 Kent State University 39
More Simulations
• Performance between different weight used
• Performance between different dimensions retained after DR
September, 2009 Kent State University 40
Final Integration Results
• Simulation results when both the image domain and the feature domain are used
• See the detail pictures, Click here
September, 2009 Kent State University 41
Integration
• UAV media capture and analysis
• WWW based media analysis
• Vehicle based media capture and analysis
September, 2009 Kent State University 42
Future ResearchExtended to video objects
• Object based video coding
• Non-object based video coding
• Video indexing
• Knowledge extraction and management
September, 2009 Kent State University 43
Future ResearchData Fusion
• Multimodality medical imaging
• CT – Structural information
• PET – Functional information
• Fusion
• Knowledge management