Texture characterization and image segmentationusing Haar patterns and evolutionary processes

Radu-Mihai Coliban1, Mihai Ivanovici1, Iuliu Szekely21Machine Intelligence and Vision Laboratory, Dept. of Electronics and Computers, Transilvania University, Brasov, Romania

2Dept. of Electrical Engineering, Sapientia University, Cluj-Napoca, Romaniacoliban.radu@unitbv.ro, mihai.ivanovici@unitbv.ro, gszekely@ms.sapientia.ro

Abstract—Images are composed of objects exhibiting texturedsurfaces. Anisotropic or irregular textures lacking a deterministicrepetitive aspect are difficult to characterize or model. However,such textures may be composed of various textons that can bemodelled as specific functions. Starting from this hypothesis,we propose a texture characterization approach based on Haarpatterns, combined with an evolutionary approach for the ap-proximate segmentation of the region of interest in color images.The segmentation map is given by the probability of finding aspecified pattern and the average distance to discovered solutionsand we show that this can be a reasonable estimate of theregion of interest. Determining the characteristics of a certainregion allows for image segmentation, classification and analysisor further interpretation of image content. In this paper, we showthe results we obtain both on synthetic textures and on paintings.

Index Terms—extended color Haar-like features, evolutionaryalgorithms, image segmentation

I. INTRODUCTION

The number of image segmentation approaches is very large[1], but only several frameworks have been accepted by thescientific community as generating accurate and predictableresults. Segmentation using active contour models is basedon variational calculus and is commonly used to extract thecontour of an object with a priori known characteristics [2].In the case of watershed segmentation, which is based onmathematical morphology, the gray-level image is viewed as atopographic relief [3]. Multi-scale approaches use a pyramidalstructure for representing the image at different resolutionlevels [4]. Graph-based approaches use graph representationsof images, which unfortunately may explode in size withthe increase of image spatial resolution [5]. Another set ofapproaches are based on the local description of texture,e.g. Haralick texture features [6]. The recent trend in imagesegmentation is over-segmenting the image on purpose, thenperforming the fusion of similar regions [7]. However, all theaforementioned approaches aim at determining regions withsimilar properties.

With the increase of image spatial and spectral resolution,all exhaustive approaches may become very time consuming.Moreover, certain regions may exhibit a large variability ofthe composing elements or textons, making them difficult tosegment. In this paper we present an approach which integrateslow level feature extraction using color extended Haar-likefeatures into an evolutionary algorithm framework, with the

aim of determining the regions that contain a certain patternof interest. The motivation behind choosing such a frameworkis the fact that using a deterministic and exhaustive approachmay take a lot of time, especially for high resolution imagesand a large variability of the features of interest. We choseto develop a probabilistic alternative that does not explore theentire image and feature space, in order to identify regionswith certain characteristics. Because of this choice, the resultswill be rough estimates of the regions of interest. In addition,purely deterministic approaches may fail when applied toobjects with a high variability in shape, color or texture.Our solution offers also the possibility of identifying irregulartextured regions composed of specific textons.

Genetic algorithms use the principles of living organisms’evolution and selection to explore a very large set of candidatesolutions. This is an appropriate approach when a set of criteriafor the characterization of a convenient solution can be aggre-gated in a fitness function, providing the means for sortingthe candidates and selecting the most fit ones. Evolutionaryalgorithms can be successfully used in segmentation whenthere is a need to identify a set of similar objects in animage, but with a large variability. The segmentation result willconsist of the areas with the highest probability of containingthe sought objects.

Various approaches have been proposed for image seg-mentation and object detection using genetic algorithms. Forinstance, the authors of [8] create cascades of classifiers usinggenetic algorithms for the purpose of object detection innoisy ultrasound images. In [9] distributed genetic algorithmsare used for the segmentation of video frames, in order toextract moving objects out of MPEG-4 sequences. In [10],a series of basic image processing algorithms are integratedin a fuzzy-genetic framework in order to detect buildings inhigh-resolution satellite images.

In this paper we show results of rough segmentation usingan evolutionary method, which aims at the rapid identificationof regions that contain a Haar-like pattern of interest.

II. OUR APPROACH

Fig. 1(a) and (b) depicts two synthetic texture images com-posed of the same square pattern, arranged in regular fashion inone case and irregular in the other. Both images are affected byGaussian noise of zero mean and standard deviation σ = 10in the RGB space. In order to characterize the textures, we

986978-1-4799-5183-3/14/$31.00 ' 2014 IEEE

(a) Texture 1 (b) Texture 2

(c) Co-occurrence matrix 1 (d) Co-occurrence matrix 2

(e) Granulometry 1 (f) Granulometry 2

Fig. 1. Noisy texture images and their associated co-occurrence matricesand granulometries.

computed both the co-occurrence matrix in RGB [11], for aneighborhood distance of one pixel (d = 1) on the horizontaldirection (θ = 0◦) and the gray-level granulometry [12] forthe two images. The white pixels in the co-occurrence matriximages correspond to non-zero values in the matrices; the twomatrices are similar to a certain degree, and moreover, donot allow a compact characterization of the two textures. Thetwo granulometry vectors underline the distinction between thetwo textures, however the spatial information is lost and thecharacterization is too compact. Various other approaches totexture characterization exist [13]. Our aim is to use Haar-likefeatures in an evolutionary algorithm framework in order tocharacterize the textures, including color, shape, pattern size,orientation and spatial information.

The core principle of our approach is emitting hypothesesabout the appearance (including size, color, shape, position,orientation) of the textons of interest inside the analyzedimages, and then verifying the validity of those hypotheses andrefining them in a series of iterations of evolutionary processes,thus leading to solutions that best suit the image. The processis started by randomly generating a number of hypothesizedobjects, forming the initial population. The improvement andrefining of the solutions through evolutionary processes is

done via three basic operations: selection, recombination andmutation. Using a genetic approach requires a way to encodehypotheses, quantify their fitness and reproduce the fittestones. The similarity of the genetic algorithm to real-lifebiological processes can be achieved to various degrees, suchthat more complex implementations generate solutions with alarge variability, while simpler implementations can be fasterand more specific. Our approach falls in the second category,aiming at rapidly detecting color Haar-like patterns in theimage for both texture characterization of the region of interestand rough segmentation.

A. Hypothesis encoding

First, we need to find a way of encoding a hypothesis intonumbers. The code of an object is named genotype, while theresulting form is called phenotype. The fields of the hypothesisare hereby named chromosomes, which are further divided intosmaller fields of information, called genes. Each gene encodesfor a trait, or process of generating traits, while one of allpossible forms of a gene is called an allele.

For the encoding of the hypothesis we use color extendedHaar-like features based on the ones defined by [14], presentedin Fig. 2. Each hypothesis hi is of the following format: atposition (x, y) in the image there exists a rectangle of width w,height h, color c1 and orientation α, surrounded by a borderof size b and color c2. Colors are represented as 24-bit RGB.Therefore the encoding represents the concatenation of allthese fields into a single string hi = [x, y, w, h, b, α, c1, c2].Given the large variability of the searched pattern, one canextend these features in order to match various applicationpurposes [15].

Fig. 2. Illustration of a hypothesis (color extended Haar-like feature).

B. Generating the initial population

The creation of the initial population consists in randomlygenerating P hypotheses of the form hi, with each field havinga particular distribution. The positions (x, y) are uniformlygenerated such that the entire image is covered. The values forthe other fields of the hypotheses are generated using Gaussiandistributions, with the means and variances chosen empirically,such that the resulting hypotheses are similar to the textonsof interest in the image under consideration. The choice ofstandard deviation for each distribution is made as to ensurethe satisfactory variability of the hypotheses.

In Fig. 3 we show an ideal synthetic image I with whitebackground and four objects of interest represented by grey

987

rectangles and a population of P = 4 hypotheses (h1, ...,h4),overlaid on the original image.

Fig. 3. Illustration of a population overlaid on the image I.

C. Selection based on fitness

The second phase of the evolutionary process is analyzingthe initial population and quantifying the correctness of thesolution, i.e. assessing the match between the hypothesis hi

(sought pattern) and the original image I, through the compu-tation of a fitness function. A natural choice is to construct thefitness function on a distance between the image correspondingto hypothesis hi and the corresponding (underlying) region ofimage I.

Computing the distance can be achieved through variousapproaches: computing the mean squared error, projecting aregion of the analyzed image on a basis of functions repre-senting the searched pattern, applying a hit-or-miss transform(e.g. the color hit-or-miss transform for simple patterns [16][17]), etc.

A first version of the fitness function we developed wasbased the computation of a distance function d(hi, I) asa sum of ∆E distances [18] in the CIELAB color spacebetween each pixel of the hypothesis and the correspondingpixel in the image. The distance is used to assess the visualsimilarity between the pattern and the region onto which it issuperimposed:

d(hi, I) =1

A (hi)

∑(u,v)

∆E(C (hi(u, v)), I(x+u, y+v)) (1)

where (u, v) ∈ Supp(hi), A (hi) represents the area, definedas the number of pixels that comprise the image correspondingto the hypothesis; C is the color of the pattern, either c1 or c2depending on the shape. The normalization of the sum is usedin order to reduce the influence of the size of the analyzedHaar-like feature on the value of the distance.

The fitness function is then defined as follows:

f(hi, I, T ) =

{ 1d(hi,I)

, d(hi, I) ≤ T0 , otherwise

(2)

where T is a configurable threshold for the acceptance levelof matching.

After measuring the fitness of all the existing hypotheses,the selection phase takes place, which chooses a set of N

hypotheses from the total of P for future reproduction. Thegoal is the improvement of the existing solutions, meaning thatthe fittest ones must reproduce, while the less fit should beeliminated. The overall fitness function has to be maximized:

arg maxi,f(hi,I,T ) 6=0

N∑i=1

f(hi, I, T ) (3)

This can be done either through a stochastic process ora deterministic one. The former requires randomly selectingsolutions based on their relative fitness, such that betterhypotheses have greater chances of reproducing; the latterconsists of simply selecting the best N solutions. Statistically,better solutions would reproduce more often for both methods.We chose the deterministic approach, with the number ofselected solutions being a percentage of all hypotheses in thepopulation.

After performing some preliminary experiments, we con-cluded that the discrimination capabilities of the fitness func-tion (2) were not of an acceptable level (see Fig. 8), andwe propose an improved version of the fitness function. Abetter evaluation of the similarity between the pattern and theunderlying image region is achieved by using the threshold forpixel-wise matching. Thus, we define the matching functionδ(u, v, T ) between the pixel at coordinates (u, v) in thehypothesis and the corresponding pixel in the image as:

δ(u, v, T ) =

{1,∆E(C (hi(u, v)), I(x+ u, y + v)) ≤ T0, otherwise

(4)Then we define the fitness function as:

f(hi, I, T ) =1

A (hi)

∑(u,v)

δ(u, v, T ) (5)

The function is constructed such that a value of 0 for ahypothesis indicates that there are no matches at the pixellevel, as opposed to the sub-image level in the previousversion, between the hypothesis and the corresponding regionof the image given the acceptance level T ; this ensures a betterevaluation of the match between the two patterns. In the initialselection, if more than 10% of the hypotheses have an non-zeroassociated fitness function, then the best fit 10% hypothesesare retained. No hypothesis with a fitness function value of 0is allowed to pass to the next iteration.

D. Evolution

The evolution of the solutions and, thus, their refinement, isdone using recombination and mutation of solutions. Throughthe recombination of similar solutions, the amount of variationis increased, while still preserving vital information. Thisserves to combine features of different hypotheses, in the hopethat the result would be a fitter solution. Such combinationsof encoding strings are facilitated by the existence of well-separated fields inside the strings, each one encoding specificsets of traits of the phenotype, or processes of the phenotypegenerator.

988

Fig. 4. The recombination of two hypotheses hi and hj .

At the moment of recombination, each parent solutiondonates its chromosomes, resulting in an offspring whichhas pairs of homologous chromosomes. Two problems arise:(i) how to choose which hypotheses recombine from theentire population and (ii) how to select which chromosomefrom a pair to donate to the offspring. We chose to usea deterministic approach, in which all N hypotheses in ageneration are combined pairwise, resulting in a number ofN2 hypotheses (including the original ones). The positionand shape information (x, y, w, h, b, α) is inherited from oneparent, while the color information (c1, c2) is inherited fromthe other. The recombination of two hypotheses is illustratedin Fig. 4.

Mutation means randomly altering the values of variousfields of the hypothesis encoding. The amount of mutation isan important parameter of the algorithm, as few mutations pergenotype can lead to a longer, yet more stable, progression,while higher rates of mutation preserve too little of theinformation for selection to achieve its goal of generatinga better batch of hypotheses. We chose to apply a singlemutation at each iteration, consisting in altering one bit of arandomly-chosen field from a randomly-selected hypothesis.

The selection step after recombination and mutation isslightly different from the initial selection. As recombinationyields multiple solutions per position, there is a possibility thata large number of solutions centered in the same point of theimage could have the best fitness, and thus be selected for thenext iteration, at the expense of other viable solutions. Thus,we employ a superposition filter for the selection phase, inwhich at each position which contains solutions, only the bestfit hypothesis is kept, while the others are eliminated fromfurther reproduction; this is based on the assumption that thepatterns of interest cannot overlap in a given spatial area.

At a first glance, the complexity of such an algorithm mightseem unnecessarily high, but these processes can prove impor-tant when creating a suitable environment for the developmentof effective solutions for such general problems as objectdetection. Precisely what evolutionary processes can achievein the context of object detection is still an open question andthe recipe for an optimal implementation is currently unknown.

E. Pseudo-code

The algorithm can be summarized through the pseudo-codegiven below. After the initial population is generated, the initialselection phase takes place using a threshold value T = v.Hypotheses with a fitness function value of 0 are eliminatedfrom the population. If the number of candidate solutions thatremain is large, only the best 10 % are retained, in order to

continue processing the fittest available solutions. Then, foreach iteration, recombination and mutation take place, withthe elimination of superimposed results; the threshold willbe halved for the selection phase, such that the algorithmconverges with the best possible solutions. The algorithm stopswhen T reaches a smaller value than a given ε.

Algorithm 1 Our approachGenerate initial population of P hypothesesT ← vCompute fitness function for each hypothesisif More than 10% have a non-zero fitness then

Select 10% best fitelse Select hypotheses with non-zero fitnessend ifT ← T/2while T > ε do

RecombinateMutateApply superposition filteringSelect hypotheses with non-zero fitnessT ← T/2

end while

III. EXPERIMENTAL RESULTS

For the proof-of-concept of our approach we have chosento use Haar-like features of reduced complexity, generatingunrotated square patterns, thus eliminating the variability oftwo fields in the hypotheses (w = h and α = 0). Figures5 and 6 depict the results of iterating our approach on thetwo texture images with an initial population of P = 200 andthreshold T = 2. The selected hypotheses are highlighted. Itmay be noticed that for Texture 1, the best fit 16 solutionsat the third iteration fit on most of the squares (75%). Thefact that not all squares are covered is due to the randomnessintrinsic to our approach. In the case of Texture 2, the best fit3 solutions fit on the three squares in the image (100%).

Our aim in the case of the 500 × 500 crop from ’Adele’ byGustav Klimt (Fig. 7(a)) is to identify the region containingyellow squares with a red border, in order to perform a roughsegmentation and texture characterization. The first results(Fig. 8) were obtained using the initially developed fitnessfunction (2), with an initial threshold T = 30; the initial popu-lation of 200 Haar-like features and the hypotheses that remainafter selection are depicted, along with the results of two fulliterations. It may be noticed that the selected hypotheses tendto focus outside the area of interest, indicating that the fitnessfunction is not properly defined for the objective. However,this version of the fitness function could be successfully usedin the case of the two texture images, due to the fact that thebackground is strongly different to the pattern in that case.

The images in Figures 9 and 10 depict the results of iteratingthe evolutionary algorithm with the improved version of thefitness function (5), using an initial population P = 200 andan initial threshold T = 6. From the initial selection phase,

989

(a) Initial population (b) Initial selection, T = 2

(c) It. 1, T = 1 (d) Best fit 16, It. 3, T = 0.25

Fig. 5. Experimental results for Texture 1.

(a) Initial population (b) Initial selection, T = 2

(c) It. 1, T = 1 (d) Best fit 3, It. 3, T = 0.25

Fig. 6. Experimental results for Texture 2.

it is evident that the overall functioning of the algorithmis improved, as the solutions are inside or near the areaof interest. Given the probabilistic nature of our approach,different runs will generate different results, as reflected bythe two figures. There are only two solutions in Fig. 9(d), ofwhich one fits perfectly over the corresponding pattern in theimage, while Fig. 10(d) exhibits five less accurate solutions.

Results for a 400 × 400 crop of a design from the CollierCampbell archive (Fig. 7(b)) are presented in Fig. 11; the aimis to identify the yellow squares with a green border. The initial

(a) Adele (b) Collier Campbell

Fig. 7. Test images.

(a) Initial population (b) Initial selection, T = 30

(c) It. 1, T = 15 (d) It. 2, T = 7.5

Fig. 8. Experimental results for Adele using the initial fitness function (2).

settings for the algorithm were P = 200 and T = 6. Mosthypotheses selected at the end of the third iteration are nearthe textons of interest, although there is also one erroneousmatch, where the hypothesis is fit over a square with a blueborder.

In order to assess the consequences of choosing an evo-lutionary approach over a deterministic one, we also imple-mented an algorithm which computes the fitness function (5)at each pixel location in the image for a single hypothesis,of similar characteristics to the textons of interest. The resultsare depicted in the form of pseudo-images (Fig. 12), in whichwe plotted the values of the fitness function at each location;the values were scaled such that the white pixels correspondto the largest values of the function.

The results for the synthetic textures (Fig. 12 (a) and (b))clearly indicate the superiority in accuracy of the deterministicapproach. The results for Adele and Collier Campbell (Fig.12 (c) and (d)) are somewhat similar to those given by theevolutionary approach, in the sense that areas that contain thetextons of interest are highlighted, along with false matches,

990

(a) Initial population (b) Initial selection, T = 6

(c) It. 1, T = 3 (d) It. 2, T = 1.5

Fig. 9. Experimental results for Adele.

(a) Initial population (b) Initial selection, T = 6

(c) It. 1, T = 3 (d) It. 2, T = 1.5

Fig. 10. Experimental results for Adele.

as a consequence of the larger variablility of the textons.The advantage of using the evolutionary approach is broughtup when analyzing the run time of the algorithms. The runtime for the deterministic algorithm on each synthetic textureimage was of over 6500 seconds, while the run time forAdele and Collier Campbell was of over 1700 and 1100seconds, respectively. In contrast, the average evolutionaryalgorithm run times, with an initial population P = 200, areof approximately 40 seconds for the synthetic textures andabout 12 seconds for Adele and Collier Campbell, includingdisk write operations at each iteration. We implemented both

(a) Initial population (b) Initial selection, T = 6

(c) It. 1, T = 3 (d) Best fit 15, It. 3, T = 0.75

Fig. 11. Experimental results for Collier Campbell.

(a) Texture 1 (b) Texture 2

(c) Adele (d) Collier Campbell

Fig. 12. Experimental results using the deterministic approach.

approaches in MATLAB and the tests were conducted usinga 1.6 GHz CPU machine.

The highlighted area in Fig. 13(a) represents a rough seg-mentation of the area of interest in the Adele image, consistingof the pixels situated at a maximum Euclidean distance of 50from the solutions in Fig. 9. This representation can be adaptedas a function of detected pattern size. The figure also depictsa representation of the two solutions h1 and h2 as nodes ina graph. Information on the size of each Haar-like featureis coded in the corresponding vector’s length. Such a graphcould be used as a compact representation of the solutions,

991

(a) Rough segmentation and graphrepresentation

(b) Manual segmentation

Fig. 13. Segmentation result and manual segmentation.

allowing for the indexing of the segmented area for variousruns of the evolutionary algorithm and retrieval from an imagedatabase. We also include a manual segmentation in Fig. 13(b)for reference purposes.

IV. CONCLUSIONS

In this article we present a proof-of-concept evolution-ary framework for texture characterization and rough imagesegmentation based on the assumption that the region ofinterest can be modeled as a sum of specific patterns. Ourapproach uses color extended Haar-like features and evolution-ary processes. We show how this approach can be used for apreliminary analysis of paintings, by searching simple patterns,i.e. rectangles of specified color range. The results of thisimage information mining approach are rough segmentationmaps, together with the associated graph for textural regioncharacterization.

Given the probabilistic nature of our approach and the waythe evolutionary processes work, there are two uncertaintiesrelated to the solutions: (i) the absolute fitness between theimage and the sought pattern is not known and the relativelybest solutions are chosen at a certain iteration of the algorithm;(ii) the exact position of the best solution is not known, thuswe are looking at the closest solutions within a specifiedrange. Once the rapid and rough segmentation is performed byusing our approach, one may use dedicated tools for a betterdiscovery of the pattern of interest. Moreover, our approachis an attempt to determine a pattern spectrum of the regionof interest. For future work, the approach can be developedfurther for the characterization of the entire image for indexingpurposes.

REFERENCES

[1] M. Ivanovici, N. Richard, and D. Paulus, “Color image segmentation,”in Advanced Color Image Processing and Analysis, C. Fernandez-Maloigne, Ed. Springer New York, 2013, pp. 219–277.

[2] M. Kass, A. W., and D. Terzopoulos, “Snakes: Active contour models,”International Journal of Computer Vision, vol. 1, no. 4, pp. 321–331,1988.

[3] S. Beucher, “Watersheds of functions and picture segmentation,” Acous-tics, Speech, and Signal Processing, IEEE International Conference onICASSP ’82., vol. 7, pp. 1928 – 1931, May 1982.

[4] A. Rosenfeld, Some pyramid techniques for image segmentation. Lon-don, UK: Springer-Verlag, 1986, pp. 261–271.

[5] A. K. Sinop and L. Grady, “A seeded image segmentation frameworkunifying graph cuts and random walker which yields a new algorithm,”in Computer Vision, IEEE International Conference on. Los Alamitos,CA, USA: IEEE Computer Society, 2007, pp. 1–8.

[6] R. Haralick and L. Shapiro, “Image segmentation techniques.” ComputerVision, Graphics, & Image Processing, vol. 29, no. 1, pp. 100–132, 1985.

[7] A. Levinshtein, A. Stere, K. N. Kutulakos, D. J. Fleet, S. J. Dickinson,and K. Siddiqi, “Turbopixels: Fast superpixels using geometric flows.”IEEE Trans. Pattern Anal. Mach. Intell., pp. 2290–2297, 2009.

[8] J. Masek, R. Burget, J. Karasek, V. Uher, and S. Guney, “Evolutionaryimproved object detector for ultrasound images,” in Telecommunicationsand Signal Processing (TSP), 2013 36th International Conference on,2013, pp. 586–590.

[9] S. W. Hwang, E. Y. Kim, S. H. Park, and H. J. Kim, “Object extractionand tracking using genetic algorithms,” in Image Processing, 2001.Proceedings. 2001 International Conference on, vol. 2, 2001, pp. 383–386 vol.2.

[10] E. Sumer and M. Turker, “An adaptive fuzzy-genetic algorithm approachfor building detection using high-resolution satellite images,” Comput-ers, Environment and Urban Systems, vol. 39, pp. 48 – 62, 2013.

[11] R. Haralick, M. Shanmugam, and I. Dinstein, “Textural features forimage classification,” IEEE Transactions on Systems, Man, and Cyber-netics, vol. SMC-3, no. 6, p. 610621, 1973.

[12] P. Soille, Morphological Image Analysis: Principles and Applications.Springer-Verlag, 2002.

[13] M. Petrou and P. Sevilla, Dealing with Texture. John Wiley and SonsLtd., 2006.

[14] R. Lienhart and J. Maydt, “An extended set of haar-like features forrapid object detection,” in Image Processing. 2002. Proceedings. 2002International Conference on, vol. 1, 2002, pp. I–900–I–903 vol.1.

[15] S. Vural, Y. Mae, H. Uvet, and T. Arai, “Multi-view fast object detectionby using extended haar filters in uncontrolled environments,” PatternRecognition Letters, vol. 33, no. 2, pp. 126 – 133, 2012.

[16] J. Weber and S. Lefevre, “Spatial and spectral morphological templatematching,” Image and Vision Computing, vol. 30, no. 12, pp. 934 – 945,2012.

[17] A. Ledoux, N. Richard, and S. Capelle-Laize, “Color hit-or-miss trans-form,” in Signal Processing Conference (EUSIPCO), 2012 Proceedingsof the 20th European, aug. 2012, pp. 2248–2252.

[18] R. Seve, “Practical formula for the computation of CIE 1976 huedifference,” Color Research and Application, vol. 21, no. 4, pp. 314–314, 1996.

992