Texture Analysis and Texture Analysis and Synthesis – SeminarSynthesis – Seminar

Semester B, 2006-2007Semester B, 2006-2007

Presented by Eyal GilstronPresented by Eyal Gilstron

Lecture 2: Co – Occurrence MatricesLecture 2: Co – Occurrence MatricesContext Sensitive Texture NibContext Sensitive Texture Nib AA

By Thomas Malzbender and Susan Spach, Proceedings of By Thomas Malzbender and Susan Spach, Proceedings of

Computer Graphics International, June '93, pp.151-163.Computer Graphics International, June '93, pp.151-163.

The problem we want to solve The problem we want to solve

Soccer

texture

Problem:Problem:

We have an Image which has a interesting We have an Image which has a interesting texture and we want to remove other texture and we want to remove other object in interactive matter (i.e. manual object in interactive matter (i.e. manual eraser behavior).eraser behavior).

ExampleExample::

History and old techniquesHistory and old techniques

Lets go back in time and try to Lets go back in time and try to remember ..remember ..

HistogramHistogram

Image processing – Lesson 2 – Hagit Hal-Or

'Houston, We Have a Problem!' 'Houston, We Have a Problem!'

Two equal histograms may represent two very Two equal histograms may represent two very dissimilar images.dissimilar images.

Histograms is a point operation so we just knowHistograms is a point operation so we just know

how many pixs we have in each gray scale color,how many pixs we have in each gray scale color,

It’s not a geometric operation.It’s not a geometric operation.

Maybe we need another tool which can Maybe we need another tool which can represent also spatial data (point & geometric).represent also spatial data (point & geometric).

Remember this problem!Remember this problem!

GLCM and GCMGLCM and GCM

Before we will get to GCM, we will talk about Before we will get to GCM, we will talk about a simpler algorithm, GLCM.a simpler algorithm, GLCM.

GLCMGLCM

The idea behind GLCM is to describe the The idea behind GLCM is to describe the texture as a matrix of “pair gray level texture as a matrix of “pair gray level probabilities”.probabilities”.This allows us to know which gray-level This allows us to know which gray-level pairs, a texture is more dominant are more pairs, a texture is more dominant are more dominant and which are less, and may dominant and which are less, and may allow us (hopefully) to stochasitcally grow allow us (hopefully) to stochasitcally grow the texture by randomly deciding pixel the texture by randomly deciding pixel colors according to those probabilities.colors according to those probabilities.

GLCM – HardcoreGLCM – Hardcore

I is our image.

Δ = {Δx, Δy} , this some offset chosen in advance.

C Δ= a matrix.

i1, i2 – these are the indices to our matrix, which are also grey-level values.

n – width and height of our image.

#A – size of group A (|A|)

The divisor of this fraction is the number of all pairs of pixels in which the second pixel is Δx pixels to the right and Δy pixels to the bottom of the first one (we will call all of these pixels ALL_GROUP).

The dividend of this fraction is the number of all pixels in ALL_GROUP which satisfy the requirement that the gray-level of the first pixel is i1 and the gray-level of the second pixel is i2 (we will call it SOME_GROUP).

We can now say that the C Δ[i1][i2] is the probability for two pixels at ALL_GROUP to also be in SOME_GROUP.

GLCM – Why is it useful?GLCM – Why is it useful?

The matrix seen here can be used to find out The matrix seen here can be used to find out probabilities of some texture features to appear, probabilities of some texture features to appear, which allows us to get an image of how the which allows us to get an image of how the texture “behaves”. texture “behaves”.

Example: probability of two pixels having a Example: probability of two pixels having a difference in gray-level d: take the sum of a difference in gray-level d: take the sum of a diagonal line in the matrix which starts from diagonal line in the matrix which starts from GLCM[0][d] to find out the probability of two GLCM[0][d] to find out the probability of two pixels having difference d between them.pixels having difference d between them.

GCM – A generalizationGCM – A generalization

GCM – A generalizationGCM – A generalization

The idea represented in GLCM is a good The idea represented in GLCM is a good idea when we want to work with idea when we want to work with probabilities of gray levels of pixels.probabilities of gray levels of pixels.Sometimes we are interested in something Sometimes we are interested in something else, sometimes we don’t even consider else, sometimes we don’t even consider an image as a collection of pixels with an image as a collection of pixels with gray-levels, but as “something else”.gray-levels, but as “something else”.GCM allows us to formally define GCM allows us to formally define “something else”.“something else”.

GCM - ComponentsGCM - Components

p – a collection of attributes representing an “image p – a collection of attributes representing an “image element”. element”. Examples:Examples:pppixelpixel = {position, gray-level} (“simple” pixels) = {position, gray-level} (“simple” pixels)ppedgeedge={position, intensity, direction} (output of edge ={position, intensity, direction} (output of edge detection)detection)S – a function which gets two p-elements and returns S – a function which gets two p-elements and returns either true or false. either true or false. Examples: (f1, f2 are “objects” of “type” p)Examples: (f1, f2 are “objects” of “type” p)sspixelpixel(f1, f2) = true (f1, f2) = true f2.position – f1.position = f2.position – f1.position = (Δx, Δy)sedge(f1, f2) = true abs(f2.position – f1.position) ≤ 20)A – some attribute of p which “interests” us.

GCM - HardcoreGCM - Hardcore

f1, f2, … fm – all elements which compose the picture, whether they are pixels, edge data, or anything else.

v1, v2 – two values which we desire the A (“interesting” property) of two f-elements which satisfy S to have, accordingly.

Divisor – Count of pairs which satisfy S (size of ANY_GROUP).

Dividend – Count of (fi, fj) pairs which satisfy S as well have the A property of the fi equal to v1 and the A property of fj equal to v2 (size of SOME_GROUP)

Once again, this formula is the probability of two elements which satisfy S to also saitsfy some requirement about their “interesting properties” (A).

Note: the GCM will usually be a square matrix whose height/width equals the total number of possible values for A (problem: what do we do when A is not discrete?)

GCM – Generalization of GLCMGCM – Generalization of GLCM

It is easy to see that GCM is a generalization of It is easy to see that GCM is a generalization of GLCM, in fact, this has already been showed to GLCM, in fact, this has already been showed to you before, let’s look at the following definitions:you before, let’s look at the following definitions:

pppixelpixel = {position, gray-level} (“simple” pixels) = {position, gray-level} (“simple” pixels)

sspixelpixel(f1, f2) = true (f1, f2) = true f2.position – f1.position = f2.position – f1.position = (Δx, Δy)=Δ

AApixelpixel=gray-level=gray-level

If you go back to the GCM formula you can see If you go back to the GCM formula you can see that if we define (p, s, a), we get GLCM.that if we define (p, s, a), we get GLCM.

Example on boardExample on board....

GCM – using it to grow a patternGCM – using it to grow a pattern

Let’s say we have some texture called “the Let’s say we have some texture called “the green texture”, and also some object green texture”, and also some object which hides parts of the “texture” called which hides parts of the “texture” called “the blue object”.“the blue object”.

GCM – using it to grow a patternGCM – using it to grow a pattern

Let’s assume that S is some geometric Let’s assume that S is some geometric function (it’s based on distance of function (it’s based on distance of positions). We can take pieces of the positions). We can take pieces of the “green texture” and build a GCM from “green texture” and build a GCM from them. (in the example below, the pieces them. (in the example below, the pieces are marked in red)are marked in red)

GCM – using it to grow a patternGCM – using it to grow a pattern

We can then stochastically replace points We can then stochastically replace points of “the blue object” with points generated of “the blue object” with points generated based on the probabilities in the GCM. based on the probabilities in the GCM. Let’s say we have two points which satisfy Let’s say we have two points which satisfy S, one is in “the green texture” and one is S, one is in “the green texture” and one is in “the blue object”. We can replace the A in “the blue object”. We can replace the A property of S (let’s assume A is ‘color’) property of S (let’s assume A is ‘color’) with something which would fit “the green with something which would fit “the green texture”.texture”.

GCM – using it to grow a patternGCM – using it to grow a pattern

How do we select something which would fit “the green texture”.How do we select something which would fit “the green texture”.We have two pixels, one in “the green texture” and one in “the blue We have two pixels, one in “the green texture” and one in “the blue object”.object”.We take the i-th line in the GCM where i is the A value of the pixel in We take the i-th line in the GCM where i is the A value of the pixel in “the green texture”.“the green texture”.We then randomly choose a value for the pixel in “the blue object”:We then randomly choose a value for the pixel in “the blue object”:We will choose 0 with a probability of GCM[i][0]We will choose 0 with a probability of GCM[i][0]We will choose 1 with a probability of GCM[i][1]We will choose 1 with a probability of GCM[i][1]We will choose 2 with a probability of GCM[i][2]We will choose 2 with a probability of GCM[i][2] etc.etc.

A Fast Method to Determine Co-Occurrence A Fast Method to Determine Co-Occurrence Texture Features Using A Linked List Texture Features Using A Linked List

ImplementationImplementation

One of the basic rules in practical computer sciences is: “if it can’t One of the basic rules in practical computer sciences is: “if it can’t be done fast, it can’t be done at all”.be done fast, it can’t be done at all”.GLCM matrices have two characteristics:GLCM matrices have two characteristics:1) They are big.1) They are big.2) They are sparse.2) They are sparse.Since they are big, in most cases we do not have to calculate the Since they are big, in most cases we do not have to calculate the entire matrix, but can instead “lazy-evaluate” only the parts we entire matrix, but can instead “lazy-evaluate” only the parts we need. It would still be smart to store the results of those “lazy need. It would still be smart to store the results of those “lazy evaulations”. However, eventually we may still have to store a large evaulations”. However, eventually we may still have to store a large amount of data in the memory.amount of data in the memory.Luckily, the matrices are also usually sparse, which means that Luckily, the matrices are also usually sparse, which means that instead of storing the matrix, we can only store the coordinates and instead of storing the matrix, we can only store the coordinates and values of places in the matrix which are not zero. This can be values of places in the matrix which are not zero. This can be possibly done via a linked list (possibly done via a linked list (in my opinionin my opinion, for better , for better performance it would be wise to use a hash map).performance it would be wise to use a hash map).

Color Co–Occurrence Matrix - CCMColor Co–Occurrence Matrix - CCM

Malzbender and Spach techniqueMalzbender and Spach technique

““Our technique is based on mimicking a second Our technique is based on mimicking a second order statistics of samples of scanned textures, order statistics of samples of scanned textures, typically taken from photographs of real textures typically taken from photographs of real textures occurring in nature. We use Color Co-occurring in nature. We use Color Co-Occurrence Matrices to capture and optimized Occurrence Matrices to capture and optimized these second order statistics. This technique is these second order statistics. This technique is shown to provide interactive response and good shown to provide interactive response and good performance over a broad range of stationary performance over a broad range of stationary textures, from periodic to stochastic.”textures, from periodic to stochastic.”

Color Co–Occurrence Matrix - CCMColor Co–Occurrence Matrix - CCM

The CCM is a four dimensional matrix given by:The CCM is a four dimensional matrix given by:

Color Co–Occurrence Matrix - CCMColor Co–Occurrence Matrix - CCM

Color Co–Occurrence Matrix - CCMColor Co–Occurrence Matrix - CCM

To illustrate, the following figure shows how graycomatrix To illustrate, the following figure shows how graycomatrix calculates the first three values in a CCM. In the output calculates the first three values in a CCM. In the output CCM , element (1,1) contains the value 1 because there CCM , element (1,1) contains the value 1 because there is only one instance in the input image where two is only one instance in the input image where two horizontally adjacent pixels have the values 1 and 1, horizontally adjacent pixels have the values 1 and 1, respectively. glcm(1,2) contains the value 2 because respectively. glcm(1,2) contains the value 2 because there are two instances where two horizontally adjacent there are two instances where two horizontally adjacent pixels have the values 1 and 2. Element (1,3) in the pixels have the values 1 and 2. Element (1,3) in the CCM has the value 0 because there are no instances of CCM has the value 0 because there are no instances of two horizontally adjacent pixels with the values 1 and 3. two horizontally adjacent pixels with the values 1 and 3. graycomatrix continues processing the input image, graycomatrix continues processing the input image, scanning the image for other pixel pairs (scanning the image for other pixel pairs (i,ji,j) and ) and recording the sums in the corresponding elements of the recording the sums in the corresponding elements of the CCM . CCM .

Naive code by Eyal GilstronNaive code by Eyal Gilstron::Delta function = takes value of 1 if and only if it’s arg is zeroDelta function = takes value of 1 if and only if it’s arg is zero// // For each entry in the CCM we do thisFor each entry in the CCM we do this::

for for ((xx==0;x<X;x0;x<X;x++) ++) {{for for ((yy==0;y<Y;y0;y<Y;y++) ++) {{ if if ((((II[[xx][][yy] == ] == LhLh) ) && && ((II[[xx++d_xd_x][][yy++d_yd_y] == ] == LrLr))))

CCM CCM [[d_xd_x][][d_yd_y][][LhLh][][LrLr] ++] ++;; }}}} K K = (= ((X (X - - absabs((d_xd_x) * () * (Y Y - - absabs((d_yd_y))));;CCM CCM [[d_xd_x][][d_yd_y][][LhLh][][LrLr] /= ] /= K;K;

Why this normalization?Why this normalization?

K K = (= ((X (X - - absabs((d_xd_x) * () * (Y Y - - absabs((d_yd_y))));;

K represents the no’ of pixs pairs which K represents the no’ of pixs pairs which has the (d_x, d_y) distance within X*Y has the (d_x, d_y) distance within X*Y window. window.

Take a look at Fig 1Take a look at Fig 1::

Y direction histogram – slice of CCMY direction histogram – slice of CCM

Why do we have negative values?Why do we have negative values?

d_x

d_y

Lh

Lr -5

-4

d_x = -4, d_y = -5

Color QuantizationColor Quantization

CCM size can be extremely large when we CCM size can be extremely large when we use colors instead of grayscale:use colors instead of grayscale:

(2 * d_XMAX +1) * (2 * d_YMAX +1) * N * N(2 * d_XMAX +1) * (2 * d_YMAX +1) * N * N

N = number of chosen colors (“This color N = number of chosen colors (“This color {L1 .. LN} are indices of color lookup table {L1 .. LN} are indices of color lookup table of 24 bit colors”)of 24 bit colors”)

If we use all 24 bit possible colors CCM it’s If we use all 24 bit possible colors CCM it’s unpractical.unpractical.

Color Quantization continue Color Quantization continue

Actually we can have small set of colors Actually we can have small set of colors and still retain excellent texture image and still retain excellent texture image quality.quality.There are many methods for picking There are many methods for picking subset of colors.subset of colors.We choose the K-means (avg) clusteringWe choose the K-means (avg) clustering

(Lim 90)(Lim 90)(see more (see more ""מערכות דימות וצבעמערכות דימות וצבע"" course by course by

Hagit Hal-Or)Hagit Hal-Or)

Texture NibTexture Nib

Question: What is a Texture Nib?Question: What is a Texture Nib?

Answer: Texture Nib is an application.Answer: Texture Nib is an application.

Question: So how does it work?Question: So how does it work?

Answer: patience my friends.Answer: patience my friends.

Texture Nib ApplicationTexture Nib Application

User point of view:User point of view:

We have a texture image that contains a We have a texture image that contains a crushed soda can, we want to remove the crushed soda can, we want to remove the soda can and refill with the texture.soda can and refill with the texture.

So we pick a window from the image with So we pick a window from the image with the texture that will be the reference.the texture that will be the reference.

We define a ‘nib’ cursor and we start scroll We define a ‘nib’ cursor and we start scroll the soda can until it is removed.the soda can until it is removed.

How the nib texture algorithm worksHow the nib texture algorithm works??

For using the nib texture the user defines a nib For using the nib texture the user defines a nib size (window) and CCM size.size (window) and CCM size.

First, the program fills the nib window with noise First, the program fills the nib window with noise then it updates the CCM in the relevant entries then it updates the CCM in the relevant entries and after it calculates the Euclidian distance and after it calculates the Euclidian distance between the destination (base texture), and the between the destination (base texture), and the current nib texture, to determine how close we current nib texture, to determine how close we are. are.

Iteratively the program changes the colors in the Iteratively the program changes the colors in the nib until we minimize the error. nib until we minimize the error.

Define matricesDefine matrices

CCMd = Our destination (see texture nib)CCMd = Our destination (see texture nib)

CCMw = Working CCMCCMw = Working CCM

CCMt = Our try CCM – result of color CCMt = Our try CCM – result of color changing according to CCMwchanging according to CCMw

Changes in CCMt matrix

Why?

Remember?

Error calculationError calculation

Are we getting closer?Are we getting closer?

Which means: ΔE = Δt - Δw

Step by step ..Step by step ..

L1:L1:

Changing (x,y) pix with color Lt instead of LcChanging (x,y) pix with color Lt instead of Lc

For each possible offsets we update CCMt in all relevant entries:For each possible offsets we update CCMt in all relevant entries:

for (d_x=0;d_x<d_xMax;d_x++)for (d_x=0;d_x<d_xMax;d_x++)

for(d_y=0; …) {for(d_y=0; …) {

Changes …Changes …

}}

Calculate d_E (Euclidian distance from CCMt to CCMw)Calculate d_E (Euclidian distance from CCMt to CCMw)

If (d_E is “If (d_E is “goodgood” enough) then {” enough) then {

update the Nib with the new colorupdate the Nib with the new color

update Eupdate E

}}

Else continue doing L1Else continue doing L1

1) Philoos, There is a half bee near your food..

2) Damn!, I think It’s in my stomach..

3) Never mindIt’s protein.

More examples ..More examples ..

Small world ..Small world ..

Request from Malzbender:Request from Malzbender:

Dear MrDear Mr. . MalzbenderMalzbender,,My name is Eyal Gilstron I'm B.Sc student in computer My name is Eyal Gilstron I'm B.Sc student in computer sciences at Haifa University in Israel.sciences at Haifa University in Israel.I just recently finished reading your 'A Context Sensitive I just recently finished reading your 'A Context Sensitive Nib' article and I would really appreciate if you send me a Nib' article and I would really appreciate if you send me a working code or more information regarding it. working code or more information regarding it. My best regards,My best regards,Eyal GilstronEyal GilstronHaifa UniversityHaifa UniversityIsraelIsrael

What a small world ..What a small world ..

Eyal,Eyal,  The texture nib work is very old and the code I have I wrote on a HP The texture nib work is very old and the code I have I wrote on a HP

UX system. I've attached what I have, but don't know if it will UX system. I've attached what I have, but don't know if it will compile as is. I do think there is a possibility of a larger publication compile as is. I do think there is a possibility of a larger publication with this sort of approach. No interactive texture synthesis has been with this sort of approach. No interactive texture synthesis has been presented at Siggraph yet. Also, there is a person in Haifa who I presented at Siggraph yet. Also, there is a person in Haifa who I have worked with who has interest in this work, his name is Jacov have worked with who has interest in this work, his name is Jacov Hel-Or. I would recommend contacting him, bright guy.Hel-Or. I would recommend contacting him, bright guy.

  regards, regards, TomTom

At last ..At last ..

The End.The End.Bye Bye..