Unsupervised Surface Reflectance Field Multi-segmenter Michal Haindl 1(B ) , Stanislav Mikeˇ s 1 , and Mineichi Kudo 2 1 The Institute of Information Theory and Automation of the Czech Academy of Sciences, Prague, Czech Republic {haindl,xaos}@utia.cz 2 Graduate School of Engineering, Hokkaido University, Sapporo, Japan [email protected]Abstract. An unsupervised, illumination invariant, multi-spectral, multi-resolution, multiple-segmenter for textured images with unknown number of classes is presented. The segmenter is based on a weighted combination of several unsupervised segmentation results, each in dif- ferent resolution, using the modified sum rule. Multi-spectral textured image mosaics are locally represented by eight causal directional multi- spectral random field models recursively evaluated for each pixel. The single-resolution segmentation part of the algorithm is based on the underlying Gaussian mixture model and starts with an over segmented initial estimation which is adaptively modified until the optimal num- ber of homogeneous texture segments is reached. The performance of the presented method is extensively tested on the Prague segmentation benchmark both on the surface reflectance field textures as well as on the static colour textures using the commonest segmentation criteria and compares favourably with several leading alternative image segmentation methods. Keywords: Unsupervised image segmentation · Textural features · Illu- mination invariants · Surface reflectance field · Bidirectional texture function 1 Introduction Segmentation is the fundamental process which partitions a data space into meaningful salient regions. Image segmentation essentially affects the overall performance of any automated image analysis system thus its quality is of the utmost importance. Image regions, homogeneous with respect to some usually textural or colour measure, which result from a segmentation algorithm are anal- ysed in subsequent interpretation steps. Texture-based image segmentation is area of intense research activity in recent years and many algorithms were pub- lished in consequence of all this effort. These methods are usually categorised c Springer International Publishing Switzerland 2015 G. Azzopardi and N. Petkov (Eds.): CAIP 2015, Part I, LNCS 9256, pp. 261–273, 2015. DOI: 10.1007/978-3-319-23192-1 22
Abstract. An unsupervised, illumination invariant, multi-spectral,multi-resolution, multiple-segmenter for textured images with unknownnumber of classes is presented. The segmenter is based on a weightedcombination of several unsupervised segmentation results, each in dif-ferent resolution, using the modified sum rule. Multi-spectral texturedimage mosaics are locally represented by eight causal directional multi-spectral random field models recursively evaluated for each pixel. Thesingle-resolution segmentation part of the algorithm is based on theunderlying Gaussian mixture model and starts with an over segmentedinitial estimation which is adaptively modified until the optimal num-ber of homogeneous texture segments is reached. The performance ofthe presented method is extensively tested on the Prague segmentationbenchmark both on the surface reflectance field textures as well as onthe static colour textures using the commonest segmentation criteria andcompares favourably with several leading alternative image segmentationmethods.
Keywords: Unsupervised image segmentation · Textural features · Illu-mination invariants · Surface reflectance field · Bidirectional texturefunction
1 Introduction
Segmentation is the fundamental process which partitions a data space intomeaningful salient regions. Image segmentation essentially affects the overallperformance of any automated image analysis system thus its quality is of theutmost importance. Image regions, homogeneous with respect to some usuallytextural or colour measure, which result from a segmentation algorithm are anal-ysed in subsequent interpretation steps. Texture-based image segmentation isarea of intense research activity in recent years and many algorithms were pub-lished in consequence of all this effort. These methods are usually categorised
[18] as region-based, boundary-based, or as a hybrid of the two. Different pub-lished methods are difficult to compare because of lack of a comprehensive anal-ysis together with accessible experimental data, however available results indi-cate that the ill-defined texture segmentation problem is still far from beingsatisfactorily solved. Spatial interaction models and especially Markov randomfields-based models are increasingly popular for texture representation [4,18],etc. Several researchers dealt with the difficult problem of unsupervised segmen-tation using these models see for example [10,15,17] or [5,7,12]. The concept ofdecision fusion [14] for high-performance pattern recognition is well known andwidely accepted in the area of supervised classification where (often very diverse)classification technologies, each providing complementary sources of informationabout class membership, can be integrated to provide more accurate, robust andreliable classification decisions than the single classifier applications.
Similar advantages can be expected and achieved [12] also for the unsuper-vised segmentation applications. However, a direct unsupervised application ofthe supervised classifiers fusion idea is complicated with unknown number ofdata hidden classes and consequently a different number of segmented regions insegmentation results to be fused. This paper exploits above advantages by com-bining several unsupervised segmenters of the same type but with different fea-ture sets. It introduces a novel eight-directional generative multispectral texturerepresentation and invariant features capable to discriminate surface reflectancefield type of textures, i.e., bidirectional texture function (BTF) textures with afixed or small range of viewing angle.
2 Combination of Multiple Segmenters
The proposed method (MW3AR8i) combines segmentation results from differ-ent resolution. We assume to down-sample input image Y into M differentresolutions Y (m) =↓ιm Y with sampling factors ιm m = 1, . . . , M identical inboth horizontal and vertical directions and Y (1) = Y . Local surface reflectancefield texture for each pixel Y
(m)r in resolution m is represented the 3D simul-
taneous causal autoregressive random field model (CAR) parameter space Θ(m)r
(5) and modeled by the Gaussian mixture model (6),(7).
2.1 Single-Resolution Texture Model
Static smooth multi-spectral textures require three dimensional models for ade-quate representation. We assume that single multi-spectral textures can belocally modelled using a 3D simultaneous causal autoregressive random fieldmodel (CAR). This model can be expressed as a stationary causal uncorrelatednoise driven 3D autoregressive process [11]:
Yr = γXr + er , (1)
where γ = [A1, . . . , Aη] is the d × dη parameter matrix, Ai∀i ∈ Icr are d × d
parametric matrices, d is the number of spectral bands, Icr is a causal neigh-
borhood index set with η = card(Icr) and er is a white Gaussian noise vector
Unsupervised Surface Reflectance Field Multi-segmenter 263
with zero mean and a constant but unknown covariance, Xr is a correspond-ing vector of the contextual neighbours Yr−s where s ∈ Ic
r and r, r − 1, . . .is a chosen direction of movement on the image index lattice I. The selectionof an appropriate CAR model support (Ic
r ⊂ I) is important to obtain goodtexture representation but less important for segmentation. The optimal neigh-bourhood as well as the Bayesian parameters estimation of a CAR model canbe found analytically under few additional and acceptable assumptions usingthe Bayesian approach (see details in [11]). The recursive Bayesian parameterestimation of the CAR model is [11]:
γTr−1 = γT
r−2 +V −1
x(r−2)Xr−1(Yr−1 − γr−2Xr−1)T
(1 + XTr−1V
−1x(r−2)Xr−1)
, (2)
where Vx(r−1) =∑r−1
k=1 XkXTk +Vx(0). Local texture for each pixel is represented
by eight parametric vectors. Each vector contains local estimations of the CARmodel parameters. These eight models have identical contextual neighbourhoodIcr but they differ in their major movement direction (↓, ↑,→,←,↘,↖,↗,↙),
i.e.,γT
r ={γt
r, γbr , γ
rr , γl
r, γdr , γ−d
r , γar , γ−a
r
}T. (3)
The parametric space γ (Section 2.2) is subsequently smooth out, rearrangedinto a vector and its dimensionality is reduced using the Karhunen-Loeve featureextraction (γ).
2.2 Illumination Invariant Textural Features
We assume that two images Y , Y of the same texture and view position differingonly in illumination can be linearly transformed to each other:
Yr = B Yr ,
where Yr, Yr are multispectral pixel values at position r and B is sometransformation matrix dependent on an illumination. This linear formula is validfor changes in brightness and illumination spectrum, with surfaces including bothLambertian and specular reflectance. We have proven [20] that the followingfeatures are illumination invariant for each CAR model:
The illumination invariant feature vector (3) for every pixel r has the form:
γTr =
{tr{At
1}, νt1,1, . . . , ν
t1,C , . . . , tr{A−a
η }, . . . , ν−aη,1 , . . . , ν
−aη,C
}T
. (4)
264 M. Haindl et al.
2.3 Mixture Based Segmentation
Multi-spectral texture segmentation is done by clustering in the CAR parameterspace Θ defined on the lattice I where
Θr = [γr, ζr]T (5)
is the modified local parameter vector (3) computed for the lattice location r.The vector ζr contains both spatial coordinates r1, r2 and local colour values.We assume that this parametric space can be represented using the Gaussianmixture model (GM) with diagonal covariance matrices due to the previousCAR parametric space decorrelation. The Gaussian mixture model for CARparametric representation at the m-th resolution (m = 1, . . . , M) is as follows:
p(Θ(m)r ) =
K(m)∑
i=1
p(m)i p(Θ(m)
r | ν(m)i , Σ
(m)i ) , (6)
p(Θ(m)r | ν(m)
i , Σ(m)i ) =
|Σ(m)i |− 1
2
(2π)d2
e − (Θ(m)r −ν
(m)i
)T (Σ(m)i
)−1(Θ(m)r −ν
(m)i
)2 . (7)
The mixture model equations (6),(7) are solved using a modified EM algorithm.
Initialization. The algorithm is initialised using ν(m)i , Σ
(m)i statistics for
each resolution m estimated from the corresponding thematic maps in twosubsequent steps:
1. refining directionν(m−1)i
(∀Θ
(m−1)r : r ∈↑ Ξ
(m)i
), Σ
(m−1)i
(∀Θ
(m−1)r : r ∈↑ Ξ
(m)i
)
m = M + 1,M, . . . , 2 i = 1, . . . , K(m) ,2. coarsening direction
ν(m)i
(∀Θ
(m)r : r ∈↓ Ξ
(m−1)i
), Σ
(m)i
(∀Θ
(m)r : r ∈↓ Ξ
(m−1)i
)
m = 2, 3, . . . ,M i = 1, . . . , K(m) ,
where Ξ(m)i ⊂ I ∀m, i, and the first initialisation thematic map Ξ
(M+1)i is
approximated by the rectangular subimages obtained by regular division of theinput texture mosaic. All the subsequent refining step are initialised from thepreceding coarser resolution up-sampled thematic maps. The final initialisationresults from the second coarsening direction where the gradually coarsening seg-mentations are initialised using the preceding down-sampled thematic maps. Foreach possible couple of components the Kullback-Leibler divergence
D (p(Θr | νi, Σi) || p(Θr | νj , Σj)) =∫
Ω
p(Θr | νi, Σi) log(
p(Θr | νi, Σi)p(Θr | νj , Σj)
)
dΘr
is evaluated and the most similar components, i.e.,
{i, j} = arg mink,l
D (p(Θr | νl, Σl) || p(Θr | νk, Σk))
Unsupervised Surface Reflectance Field Multi-segmenter 265
are merged together in each initialisation step. This initialisation results in Kini
subimages and recomputed statistics νi, Σi . Kini > K where K is theoptimal number of textured segments to be found by the algorithm. Two stepsof the EM algorithm are repeating after initialisation. The components withsmaller weights than a fixed threshold (pj < 0.02
Kini) are eliminated. For every
pair of components we estimate their Kullback-Leibler divergence. From themost similar couple, the component with the weight smaller than the thresholdis merged to its stronger partner and all statistics are actualised using the EMalgorithm. The algorithm stops when either the likelihood function has negligibleincrease (Lt − Lt−1 < 0.01) or the maximum iteration number threshold isreached.
2.4 Resulting Mixture Probabilities
Resulting mixture model probabilities are mapped to the original fine resolutionimage space for all m = 1, . . . , M mixture sub-models ((6)(7)). The M cooper-ating segmenters deliver their class response in the form of conditional probabili-ties. Each segmenter produces a preference list based on the mixture componentprobabilities of a particular pixel belonging a particular class, together with aset of confidence measurement values generated in the original decision-makingprocess.
Single-Segmenters Correspondence. Single-resolution segmentation resultscannot be combined without knowledge of the mutual correspondence betweenregions in all different-resolution segmentation probabilistic mixture componentmaps (K1×∑M
m=2 Km combinations). Mutual assignments of two probabilisticmaps are solved by using the Munkre’s assignment algorithm [12] which findsthe minimal cost assignment
g : A → B,∑
α∈A
f(α, g(α))
between sets A, B, |A| = |B| = n given the cost function f(α, β), α ∈ A, β ∈B. α corresponds to the fine resolution probabilistic maps, β corresponds todown-sampled probabilistic maps and f(α, β) is the Kullback-Leibler divergencebetween probabilistic maps. The algorithm has polynomial complexity insteadof exponential for the exhaustive search.
Final Parametric Space. The parametric vectors representing texture mosaicpixels are assigned to the clusters based on our modification of the sum ruleaccording to the highest component probabilities, i.e., Yr is assigned to the clus-ter ωj∗ if [9]
πr,j∗ = maxj
∑
s∈Ir
ws
(M∑
m=1
p2(Θ(m)r−s | ν(m)
j , Σ(m)j )
∑Mi=1 p(Θ(i)
r−s | ν(i)j , Σ
(i)j )
)
,
266 M. Haindl et al.
where ws are fixed distance-based weights, Ir is a rectangular neighbourhoodand πr,j∗ > πthre (otherwise the pixel is unclassified). The area of single clusterblobs is evaluated in the post-processing thematic map filtration step. Regionswith similar statistics are merged. Thematic map blobs with area smaller than agiven threshold are attached to its neighbour with the highest similarity value.
3 Experimental Results
The algorithm was tested on natural wooden bidirectional texture function(BTF) mosaics from the Prague Texture Segmentation Data-Generator andBenchmark (http://mosaic.utia.cas.cz) [6]. The benchmark test mosaics layoutsand each cell texture membership are randomly generated and filled with BTFtextures from the large UTIA BTF database. The BTF wood measurements aremapped on the randomly generated spline surface. These tested BTFs have 3spectral bands (d = 3) but the segmenter can handle any number of bands.
The benchmark ranks segmentation algorithms according to a chosen crite-rion. There are implemented twenty seven most frequented evaluation criteriacategorised into four criteria groups – region-based [6], pixel-wise [6], clusteringcomparison criteria, and consistency measures [6]. The region-based [6] perfor-mance criteria mutually compare ground truth (GT) image regions with thecorresponding machine segmented regions (MS). The pixel-wise criteria groupcontains the most frequented classification criteria such as the omission and com-mission errors, class accuracy, recall, precision, etc. Finally the last two criteriasets incorporate the global and local consistency errors [6] and three clusteringcomparison criteria.
Table 1 compares the overall benchmark performance of the proposed algo-rithm MW3AR8i with the Voting Representativeness - Priority Multi-ClassFlooding Algorithm (VRA-PMCFA) [8,16], Segmentation by Weighted Aggre-gation (SWA) [19], Efficient Graph-Based Image Segmentation (EGBIS) [3],Factorization-based texture SEGmenter (FSEG) [21], HGS [13], Edge Detectionand Image SegmentatiON (EDISON) [1], JSEG [2], Deep Brain Model (DBM)[8], respectively. The table criteria are averaged over 10 experimental mosaics.
MW3AR8i ranks second (average rank 3.05) over all benchmark criteria,slightly worse than the overall winner of the ICPR 2014 Unsupervised ImageSegmentation Contest [8] - the VRA-PMCFA method.
These results illustrated in Figs. 1-3 and Table 1 demonstrate very goodpixel-wise, correct region segmentation, missed error, noise error, and underseg-mentation properties of our method. For most the pixel-wise criteria our methodis among the best ones while. Our oversegmentation value is the second worstfrom all the compared methods what offers a large space for further improvementby better future post-processing.
Figs. 2,3 and show five selected 1024× 1024 experimental benchmark mosaicscreated from four to twelve naturalBTF textures.The last four or five rows on thesefigures demonstrate comparative results from the eight alternative algorithms.Three methods (VRA-PMCFA, FSEG, DBM) participated in the ICPR contest.
Unsupervised Surface Reflectance Field Multi-segmenter 267
Fig. 1. Performance curves (vertical axis - f(threshold), horizontal axis - threshold,details in http://mosaic.utia.cas.cz) of correct segmentation, undersegmentation, over-segmentation, and F-measure, respectively.
Unsupervised Surface Reflectance Field Multi-segmenter 269
BTF mosaic
ground truth
VRA-PMCFA
MW3-AR8i
SWA
EGBIS
Fig. 2. BTF mosaic, ground truth, and segmentation results, respectively.
270 M. Haindl et al.
ground truth
FSEG
HGS
EDISON
JSEG
DBM
Fig. 3. Ground truth, and segmentation results, respectively.
Unsupervised Surface Reflectance Field Multi-segmenter 271
The contest used the large size (80 textural mosaics) unsupervised Colour bench-mark without noise degradation and with linear region borders. The contest crite-rion was the average rank over all benchmark criteria.
Hard natural BTF textures were chosen rather than synthesised (for exam-ple using Markov random field models) ones because they are expected to bemore difficult for the underlying segmentation model. The fourth row on Fig. 2demonstrates solid behaviour of our MW3AR8i algorithm but also infrequentalgorithm failures producing the oversegmented thematic map for some textures.Such failures can be reduced by a more elaborate post-processing step.
The SWA [19], EGBIS [3], FSEG [21], HGS [13], EDISON [1], JSEG [2],and DBM algorithms on these data performed mostly worse as can be seen intheir corresponding rows on Figs. 2,3 some areas are undersegmented while otherparts of the mosaics are oversegmented. The best six method’s performance isillustrated also on Fig. 1.
4 Conclusions
We proposed a significant improvement of our previously published unsupervisedmulti-segmenter [9]. The MW3AR8i segmenter is computationally efficient androbust method for unsupervised textured image segmentation with unknownnumber of classes based on the underlying CAR and GM texture models. Thealgorithm is reasonably fast, despite of using the random field type data repre-sentation, due to its efficient recursive parameter estimation of the underlyingmodels and therefore is much faster than the usual Markov chain Monte Carloestimation approach required for the Markovian image representations. Usualdrawback of most segmentation methods is their application dependent param-eters to be experimentally estimated. Our method requires only a contextualneighbourhood selection and two additional thresholds. The method’s perfor-mance is demonstrated on the extensive benchmark tests on both natural texturemosaics as well as on BTF mosaics. It performs favourably compared with eightalternative segmentation algorithms. Detailed experimental results are availablein http://mosaic.utia.cas.cz.
Acknowledgments. This research was supported by the Czech Science Foundationproject GACR 14-10911S.
References
1. Christoudias, C., Georgescu, B., Meer, P.: Synergism in low level vision. In:Kasturi, R., Laurendeau, D., Suen, C. (eds.) Proceedings of the 16th InternationalConference on Pattern Recognition, vol. 4, pp. 150–155. IEEE Computer Society,Los Alamitos (2002)
2. Deng, Y., Manjunath, B.: Unsupervised segmentation of color-texture regions inimages and video. IEEE Transactions on Pattern Analysis and Machine Intelligence23(8), 800–810 (2001)
4. Haindl, M.: Texture synthesis. CWI Quarterly 4(4), 305–331 (1991)5. Haindl, M., Mikes, S.: Unsupervised texture segmentation using multispectral mod-
elling approach. In: Tang, Y., Wang, S., Yeung, D., Yan, H., Lorette, G. (eds.)Proceedings of the 18th International Conference on Pattern Recognition, ICPR2006, vol. II, pp. 203–206. IEEE Computer Society, Los Alamitos (2006)
6. Haindl, M., Mikes, S.: Texture segmentation benchmark. In: Lovell, B.,Laurendeau, D., Duin, R. (eds.) Proceedings of the 19th International Confer-ence on Pattern Recognition, ICPR 2008, pp. 1–4. IEEE Computer Society, LosAlamitos (2008)
7. Haindl, M., Mikes, S.: Model-based texture segmentation. In: Campilho, A.C.,Kamel, M.S. (eds.) ICIAR 2004. LNCS, vol. 3212, pp. 306–313. Springer,Heidelberg (2004)
8. Haindl, M., Mikes, S.: Unsupervised image segmentation contest. In: Proceed-ings of the 22nd International Conference on Pattern Recognition, ICPR 2014,pp. 1484–1489. IEEE Computer Society CPS, Los Alamitos, August 2014. http://mosaic.utia.cas.cz/icpr2014/
9. Haindl, M., Mikes, S., Pudil, P.: Unsupervised hierarchical weighted multi-segmenter. In: Benediktsson, J.A., Kittler, J., Roli, F. (eds.) MCS 2009. LNCS,vol. 5519, pp. 272–282. Springer, Heidelberg (2009)
10. Haindl, M.: Texture segmentation using recursive Markov random field parameterestimation. In: Bjarne, K.E., Peter, J. (eds.) Proceedings of the 11th Scandina-vian Conference on Image Analysis, pp. 771–776. Pattern Recognition Society ofDenmark, Lyngby (1999)
11. Haindl, M.: Visual data recognition and modeling based on local markovian mod-els. In: Florack, L., Duits, R., Jongbloed, G., Lieshout, M.C., Davies, L. (eds.)Mathematical Methods for Signal and Image Analysis and Representation, Com-putational Imaging and Vision, chap. 14, vol. 41, pp. 241–259. Springer, London(2012). doi:10.1007/978-1-4471-2353-8 14
12. Haindl, M., Mikes, S.: Unsupervised texture segmentation using multiple seg-menters strategy. In: Haindl, M., Kittler, J., Roli, F. (eds.) MCS 2007. LNCS,vol. 4472, pp. 210–219. Springer, Heidelberg (2007)
13. Hoang, M.A., Geusebroek, J.M., Smeulders, A.W.: Color texture measurement andsegmentation. Signal Processing 85(2), 265–275 (2005)
14. Kittler, J., Hojjatoleslami, A., Windeatt, T.: Weighting factors in multiple expertfusion. In: Proc. BMVC, pp. 41–50. BMVA (1997)
15. Manjunath, B., Chellapa, R.: Unsupervised texture segmentation using markovrandom field models. IEEE Transactions on Pattern Analysis and Machine Intel-ligence 13, 478–482 (1991)
16. Panagiotakis, C., Grinias, I., Tziritas, G.: Natural image segmentation based ontree equipartition, bayesian flooding and region merging. IEEE Transactions onImage Processing 20(8), 2276–2287 (2011)
17. Panjwani, D., Healey, G.: Markov random field models for unsupervised segmenta-tion of textured color images. IEEE Transactions on Pattern Analysis and MachineIntelligence 17(10), 939–954 (1995)
Unsupervised Surface Reflectance Field Multi-segmenter 273
18. Reed, T.R., du Buf, J.M.H.: A review of recent texture segmentation and featureextraction techniques. CVGIP-Image Understanding 57(3), 359–372 (1993)
19. Sharon, E., Galun, M., Sharon, D., Basri, R., Brandt, A.: Hierarchy and adaptivityin segmenting visual scenes. Nature 442(7104), 719–846 (2006)
20. Vacha, P., Haindl, M.: Image retrieval measures based on illumination invarianttextural mrf features. In: CIVR 2007: Proceedings of the 6th ACM internationalconference on Image and video retrieval, pp. 448–454. ACM Press, New York (2007)
21. Yuan, J., Wang, D.: Factorization-based texture segmentation. Tech. Rep.OSU-CISRC-1/13-TR0, The Ohio State University, Columbus (2013)
