CBIR Based Expert Systeman Overview

Subhash ChandDec. 21, 2010

I n d i a n A g r i c u l t u r a l S t a t i s t i c s R e s e a r c h I n s t i t u t e

Introduction.

CBIR Techniques

Implementations.

Conclusion.

Today’s Agenda

Able to design Expert System:

◦ beyond the language bar

◦ Image retrieval (Color, Taxture, Histogram etc.)

Obsective

IntroductionFarmers always need satisfactory and easy advice from expert system, especially from agricultural expert system. To get the advice from an expert system, it should have enough knowledge about the domain. Gathering enough knowledge and representing it in a machine understandable format is time consuming and difficult job. Also, representing each and every kind of knowledge is still a research issue.

Multimedia integrate image, movie, sound and text in any combination. The use of images in human communication is hardly new – our cave-dwelling ancestors painted pictures on the walls of their caves, and the use of maps and building plans to convey information almost certainly dates back to pre-Roman times. Images now play a crucial role in fields as diverse as medicine, journalism, advertising, design, education and entertainment.

Is Multimedia Solution?

Multimedia expert systems provide "a comprehensive set of resources to enhance the presentation and communication of expertise to support users' needs within a particular area of expertise."

Since, a single picture is worth a thousand words, it will be a good idea to acquire knowledge also in images rather than only text. Image is an easy way of communication without any boundary of language. Hence there is a need for building an expert system with content based image retrieval which could acquire and deliver the knowledge by searching the image having the similar features that is searched by the user.

Can CBIR Expert System help?

Content-based image retrieval (CBIR), also known as query by image content (QBIC) and content-based visual information retrieval (CBVIR) is to describe experiments into automatic retrieval of images from a database, based on the colors and shapes.

The term has been used to describe the process of retrieving desired images from a large collection on the basis of syntactical image features. The techniques, tools and algorithms that are used originate from fields such as statistics, pattern recognition, signal processing, and computer vision.

What is CBIR ?

"Content-based" means that the search will analyze the actual contents of the image rather than the  metadata such as keywords, tags, and/or descriptions associated with the image.

The term 'content' in this context might refer to colors, shapes, textures, or any other information that can be derived from the image itself. CBIR is desirable because most web based image search engines rely purely on metadata and this produces a lot of garbage in the results. Also having humans manually enter keywords for images in a large database can be inefficient, expensive and may not capture every keyword that describes the image. Thus a system that can filter images based on their content would provide better indexing and return more accurate results.

What does mean Content in CBIR ?

11

Applications◦Art Collections e.g. Fine Arts Museum of San Francisco◦Medical Image Databases CT, MRI, Ultrasound, The Visible Human◦Scientific Databases e.g. Earth Sciences◦Trade Mark◦The World Wide Web◦Expert Systems

12

What is a query in CBIR?◦an image you already have ◦a rough sketch you draw ◦a symbolic description of what you want e.g. an image of a man and a woman on a beach

Demo

Caliph & Emir

MPEG-7 is a multimedia content description standard developed by MPEG (Moving Picture Experts Group). It was standardized in ISO/IEC 15938 (Multimedia content description interface).

A Descriptor represents a Feature of audio-visual material whereas a Feature here is understood as "a distinctive characteristic of the data [that] signifies something to somebody"

MPEG-7 Visual Descriptors :Part three of the ISO/IEC 15938-3 specifies standardized methods for the description of visual content - that are still images, videos and 3D models. The MPEG-7 Visual Descriptors are grouped in five categories by the features of color, texture, shape, motion and localization as well as a domain specific face recognition tool.

Definations

Color Feature Descriptors ◦The color description tools include four

descriptors that represent different aspects of color: representative colors (DominantColor), color distribution (ScaleableColor) and spatial distribution of colors (ColorLayout and ColorStructure) and may be extracted from images of arbitrary shape.

Low Level Features of Image

DominantColor: fits the targets for content based retrieval of either the whole image or arbitrary defined shapes by defining a set of characterizing dominant colors in that area. It contains fields (among others) representing the percentage of pixels having that associated color in the specified region, as well as a spatial coherency per dominant color attribute. This descriptor is most suitable for representing local features where a small amount of colors is enough to characterize the provided information in the region of interest but it also is applicable for whole images for example, flag or color trademark images.

ScalableColor: This descriptor depicts a color histogram in the HSV color space, encoded by a Haar transformation (which allows the sorting of data by frequency) and is used to find and compare images by their color characteristics.

ColorLayout: Is used for high-speed image retrieval and browsing using image-to-image and sequence-to-sequence matching - and even sketch-to-image matching, by specifying a spatial distribution of colors. This Descriptor makes use of the DCT12 the discrete cosine transformation to compute its YDC, CbDC, CrDC, YAC, CbAC and CrAC coefficients. This compact representation of color layout information allows visual signal matching functionality combined with high retrieval efficiency at very small computational costs and therefore even is utilizable on mobile terminal applications with strictly limited hardware and software restrictions.

ColorStructure: This descriptor is similar to the one of a color histogram in the meaning of describing color content - but further also pays respect to the structure of its content. Unlike the color histogram it uses a structuring element, which is composed of 8x8 connected pixels and instead of characterizing the relative frequency of individual image samples it characterizes the frequency of these structuring elements. In that way it is possible to distinguish between two images having an identical amount of the same color, though the structure of the groups of pixels having that amount is different. The main areas of usage are image-to-image comparison in domains of still-image retrieval – even in arbitrary shaped and possibly non-connected regions.

Textures in MPEG-7 are understood in the meaning of patterns which are used to describe surface characteristics of objects. Similarity of patterns here is measured in terms of intensity of certain colors in an image or by the degree of uniformity of a structure compared to a certain environmental point. The three standardized Visual Texture Descriptors TextureBrowsing, HomogeneousTexture and EdgeHistogram are very important tools for similarity retrieval and browsing.

Texture Feature Descriptors

HomogeneousTexture: Homogeneous texture has emerged as an important visual primitive for searching and browsing through large collections of similar looking patterns. An image can be considered as a mosaic of homogeneous textures so that these texture features associated with the regions can be used to index the image data.

TextureBrowsing: This descriptor has the function to represent a perceptual characterization of a texture, in terms of regularity (irregular, slightly regular, regular, and highly regular), coarseness (fine, medium, coarse, very coarse) and directionality (the dominant direction(s) characterizing the texture alignment.

EdgeHistogram: Edges play an important role for image perception. The EdgeHistogram tool distinguishes five types of edges in local image regions, four directional (vertical, horizontal, 45 degree, 135 degree) and a non- directional one. This information is provided for every sub-image, which is de.ned by dividing the image space into 16 non-overlapping parts. The Descriptor is able to retrieve images with similar semantics.

24

Using Relevance Feedback

◦ The CBIR system should automatically adjust the weight that were given by the user for the relevance of previously retrieved documents

◦ Most systems use a statistical method for adjusting the weights.

Content Based Image Retrieval (CBIR)

Techniques1. Color Correlogram2. Color Edge Display Descriptor (CEDD)3. Fuzzy Color and Texture Color (FCTH)

Color Correlogram: The color correlogram describes the global distribution of local spatial correlations of colors and Integrates both color information and space information. Color correlogram of an image is a table indexed by color pairs, where the k-th entry for (i,j) specifies the probability of finding a pixel of color j at a distance k from a pixel of color i in the image. Such an image feature turns out to be robust in tolerating large changes in appearance of the same scene caused by changes in viewing positions, changes in the background scene, partial occlusions, camera zoom that causes radical changes in shape, etc. This technique is suitable for the system as image taken by farmer may have variation in image’s properties.

Color Edge Display Descriptor: extraction technique is based on algorithm. The unit associated with the extraction of color information is called Color Unit. Similarly, the Texture Unit is the unit associated with the extraction of texture information. The CEDD histogram is constituted by 6 regions, determined by the Texture Unit. Each region is constituted by 24 individual regions, originating from the Color unit.

Fuzzy Color and Texture Color (FCTH): extraction technique is based on algorithm, in which the histogram is constituted by 8 regions, as these are determined by the fuzzy system that takes decision with regards to the texture of the image. Each region is constituted by 24 individual regions, as these results from the second fuzzy system. Overall, the output that results includes 8 × 24 = 192 bins. Based on the content of the bins the respecting final histogram is produced.

Basic Components of CBIR• Feature Extractor

• Create the metadata

• Query Engine

• Calculate Similar

Can Use Java API called LIRE http://www.semanticmetadata.net/lire

Implementation

Typical System Design Diagram of Multimedia Enabled Expert System using CBIR

Feature Extraction

Search for Image

Indexing Mechanism

Database Handler

Database

Delete ImageInsert Image

UserFarmerDomain Expert User

Interface

Image Processing

Database Management

Typical Flow Chart of the Multimedia CBIR based Expert System

Yes

No

Image

Image Feature Extraction

XML Document

Indexing

Searching

Visualisation

Storage

Retrieval on textual descriptors and /or similarity

measures of features

No image found; Submiteed to domain

experts to characterize the image

Uploading

CEDD

FCTH

Correlogram

Project◦ Lire◦ http://www.anaktisi.net/

Image Search Engine◦ http://www.gazopa.com◦ http://mipai.esuli.it/ ◦ http://www.tineye.com/ compare

Demo

Case Study

Sample image

Results

1. Results_Corologrm2. Result_CEDD3. ResultFCTH

Now, We have many features (too many) Not all features are always important. Can choose one or combination of many

techniques CBIR based application can be integrated with

existing text based expert system. It’s wide acceptability beyond the regional

language bar. Where images are involved in your research.

Conclusion

http://en.wikipedia.org/wiki/List_of_CBIR_engines http://orpheus.ee.duth.gr/anaktisi/ http://http://cvit.iiit.ac.in/projects/cbir/ various

approach of CBIR Huang, J.; Kumar, S. R.; Mitra, M.; Zhu, W. &

Zabih, R. "Image Indexing Using Color Correlograms", IEEE Computer Society(2007).

Refferences

Thanks

Project◦ http://www.semanticmetadata.net/◦ http://www.anaktisi.net/

Image Search Engine◦ http://www.gazopa.com◦ http://mipai.esuli.it/ ◦ http://www.tineye.com/ compare

Experience the following and expand your ideas