Segmentation Notes II Regional Approach: Global Auto Threshold Methods (The most used methods are in bold)
Huang
Intermodes
IsoData
Li
Max Entropy
Mean << only used for a starting guess
Min Error(I)
Minimum
Moments
Otsu
Percentile
Renyi Entropy
Shanbhag
Triangle
Yen
All implemented in ImageJ
Elements and definitions:
Consider pixels at coordinates (x, y), let I(x, y) be a discrete, NM scalar image with graylevel attribute u= I(x,y).
The size or cardinality of image I(x,y) is defined as:
1 1
( ', ') ( , ) 0 0
( ) 1 1N M
x y I x y y x
card I NM
We define the Iverson bracket [P], with P any boolean proposition:
1 if is true[ ]
0 if is false PP
P
For example [(21) < 0.1] = 0, [2+1 = 41]=1, and [i = j]=ij.
The oposite (logic or boolean complement) of the expression [u > uT] is [u uT] and the oposite of [u = uT] is [u uT], or [u< uT ¤ u > uT2]. The oposite of [u uT1 ⁄ u < uT2] is [u< uT1 ¤ u uT2]. In C/C++ code, the equivalent boolean expressions for the last pairs of Iverson brackets are written, respectively, as:
(u>u_T) and (uu_T), (u==u_T) and (u!=u_T), ((u>=u_T1)||(u<u_T2)) and ((u<u_T1)&&(u u_T2).
We consider boolean segmentation labels: 0 for the backround (“false”) and 1 for the foreground or objects of interest (“true”).
A global binary threshold is indicated, for example, as unew = [u>uthreshold]. In practice, for display, we remap the boolean label “1” to white (graylevel u255 =255): unew = 255[u>uthreshold].
In general, in the following expressions, a non-boolean label ulabel may be included by changing “[P]” to “ulabel[P]”, for any proposition P on segmentation criteria (not only on graylevels).
Testing all methods (44 mosaic montage)
Original
Mean thresholding method (global)
This method selects as threshold the mean of the grey level distribution:
1 1
0 0
1where ( , ) ( , )
[ ],N M
meany x
new mean
NMu x y I x y
u u u
This threshold rarely works properly and rather by coincidence. However, some other methods use this as a first-guess threshold. In general, the robust means, the median and other measures of a representative value are better adapted for local thresholds. Glasbey, CA (1993), "An analysis of histogram-based thresholding
algorithms", CVGIP: Graphical Models and Image Processing 55: 532-537.
Huang's fuzzy thresholding method.
This uses Shannon's entropy function (one can also use Yager's entropy function). Details in: Huang, L-K & Wang, M-J J (1995), "Image thresholding by maximizing the
index of nonfuzziness of the 2-D grayscale histogram", Pattern Recognition 28(1): 41-51, <http://portal.acm.org/citation.cfm?id=638891>
Intermodes This assumes a bimodal histogram. The histogram is iteratively smoothed using a running average of size 3, until there are only two local maxima: j and k. The threshold t is then computed as (j+k)/2. Images with histograms having extremely unequal peaks or a broad and flat valley are unsuitable for this method. method Prewitt, JMS & Mendelsohn, ML (1966), "The analysis of cell images",
Annals of the New York Academy of Sciences 128: 1035-1053, <http://www3.interscience.wiley.com/journal/119758871/abstract?CRETRY=1&SRETRY=0>
See also a full description in: http://www.cs.tut.fi/~ant/histthresh/
IsoData
Iterative procedure based on the isodata algorithm of: Ridler, TW & Calvard, S (1978), "Picture thresholding using an iterative
selection method", IEEE Transactions on Systems, Man and Cybernetics 8: 630-632, <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4310039>
The procedure divides the image into object and background by taking an initial threshold; then, the averages of the pixels at or below the threshold and pixels above are computed. The averages of those two values are computed, the threshold is incremented and the process is repeated until the threshold is larger than the composite average. That is:
threshold = (average background + average objects)/2. Several implementations of this method exist. The default method of ImageJ is almost the same that the one by Ridler et al.
Li Implements Li's Minimum Cross Entropy thresholding method based on the iterative version (2nd reference below) of the algorithm. Li, CH & Lee, CK (1993), "Minimum Cross Entropy Thresholding", Pattern
Recognition 26(4): 617-625 Li, CH & Tam, PKS (1998), "An Iterative Algorithm for Minimum Cross
Entropy Thresholding", Pattern Recognition Letters 18(8): 771-776 Sezgin, M & Sankur, B (2004), "Survey over Image Thresholding Techniques
and Quantitative Performance Evaluation", Journal of Electronic Imaging 13(1): 146-165, <http://citeseer.ist.psu.edu/sezgin04survey.html>
MaxEntropy Implements Kapur-Sahoo-Wong (Maximum Entropy) thresholding method: Kapur, JN; Sahoo, PK & Wong, ACK (1985), "A New Method for Gray-Level
Picture Thresholding Using the Entropy of the Histogram", Graphical Models and Image Processing 29(3): 273-285.
MinError(I) An iterative implementation of Kittler and Illingworth's Minimum Error thresholding. The ImageJ implementation seems to converge more often than the original. Nevertheless, sometimes the algorithm does not converge to a solution. In that case a warning is reported to the log window and the result defaults to the initial estimate of the threshold which is computed using the Mean method. The Ignore black or Ignore white options might help to avoid this problem.
Kittler, J & Illingworth, J (1986), "Minimum error thresholding", Pattern Recognition 19: 41-47.
Minimum Similarly to the Intermodes method, this assumes a bimodal histogram. The histogram is iteratively smoothed using a running average of size 3, until there are only two local maxima. The threshold t is such that yt−1 > yt <= yt+1. Images with histograms having extremely unequal peaks or a broad and flat valley are unsuitable for this method. Prewitt, JMS & Mendelsohn, ML (1966), "The analysis of cell images",
Annals of the New York Academy of Sciences 128: 1035-1053, <http://www3.interscience.wiley.com/journal/119758871/abstract?CRETRY=1&SRETRY=0>
Moments Tsai's method attempts to preserve the moments of the original image in the thresholded result. Tsai, W (1985), "Moment-preserving thresholding: a new approach",
Computer Vision, Graphics, and Image Processing 29: 377-393, <http://portal.acm.org/citation.cfm?id=201578>
Otsu's method: Example of a histogram, shape-based image thresholding
This algorithm assumes that the image to be thresholded contains two classes of pixels (e.g. foreground and background) then calculates the optimum threshold separating those two classes so that their combined spread (intra-class variance) is minimal.
In Otsu's method we exhaustively search for the threshold that minimizes the intra-class variance, defined as a weighted sum of variances of the two classes in function of a varying threshold T:
2 2 2
1 1 2 2( ) ( ) ( ) ( ) ( )w T T T T T
Weights ωi are the probabilities of the two classes separated by a
threshold T and 2 ( )b T are the variances of these classes. Otsu shows that minimizing the intra-class variance is the same as maximizing inter-class variance:
22 2 21 2 1 2( ) ( ) ( ) ( ) ( ) ( )b wT T T T T T
which is expressed in terms of class probabilities ωi and class means μi which in turn can be updated iteratively. This idea yields an effective algorithm:
Algorithm 1. Compute histogram and probabilities of each intensity level.
2. Set up initial ωi (0) and μi (0) (e.g., ωi (0)=1 for all i). 3. Step through all possible thresholds T = 1,..., maximum intensity.
1. Update ωi and μi
2. Compute 2 ( )b T
4. Desired threshold corresponds to the maximum 2 ( )b T .
Figure 1. An example image thresholded using Otsu's algorithm.
Nobuyuki Otsu (1979). "A threshold selection method from gray-level histograms". IEEE Trans. Sys., Man., Cyber. 9: 62–66.
Percentile
Assumes the fraction of foreground pixels to be 0.5.
Doyle, W (1962), "Operation useful for similarity-invariant pattern recognition", Journal of the Association for Computing Machinery 9: 259-267, doi:10.1145/321119.321123, <http://portal.acm.org/citation.cfm?id=321119.321123>
See here for an excellent slide presentation and the original Matlab code: http://www.cs.tut.fi/~ant/histthresh/
RenyiEntropy
Similar to the MaxEntropy method, but using Renyi's entropy instead.
Kapur, JN; Sahoo, PK & Wong, ACK (1985), "A New Method for Gray-Level Picture Thresholding Using the Entropy of the Histogram", Graphical Models and Image Processing 29(3): 273-285.
Shanbhag
Shanbhag, Abhijit G. (1994), "Utilization of information measure as a means of image thresholding", Graph. Models Image Process. (Academic Press, Inc.) 56 (5): 414--419, ISSN 1049-9652, DOI 10.1006/cgip.1994.1037.
Triangle
This is an implementation of the Triangle method:
Zack GW, Rogers WE, Latt SA (1977), "Automatic measurement of sister chromatid exchange frequency", J. of Histochem. Cytochem. 25 (7): 741–53, PMID 70454, <http://www.jhc.org/cgi/pmidlookup?view=long&pmid=70454>
Modified from from Johannes Schindelin plugin Triangle_Algorithm in
http://wbgn013.biozentrum.uni-wuerzburg.de/ImageJ/triangle-algorithm.html
See also: http://www.ph.tn.tudelft.nl/Courses/FIP/noframes/fip-Segmenta.html#Heading118
The Triangle algorithm, a geometric method, cannot tell whether the data is skewed to one side or another, but assumes a maximum peak (mode) near one end of the histogram and searches towards the other end. This causes a problem in the absence of information of the type of image to be processed, or when the maximum is not near one of the histogram extremes (resulting in two possible threshold regions between that max and the extremes). Here the algorithm was extended to find on which side of the max peak the data goes the furthest and searches for the threshold within that largest range.
Yen
Implements Yen's thresholding method from:
Yen JC, Chang FJ, Chang S (1995), "A New Criterion for Automatic Multilevel Thresholding", IEEE Trans. on Image Processing 4 (3): 370-378, ISSN 1057-7149, doi:10.1109/83.366472, <http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=366472>
Sezgin, M & Sankur, B (2004), "Survey over Image Thresholding Techniques and Quantitative Performance Evaluation", Journal of Electronic Imaging 13(1): 146-165, <http://citeseer.ist.psu.edu/sezgin04survey.html>
Segmentation Notes III
Local Approach: Adaptive Threshold methods (or Auto Local Thresholds) Elements and definitions (see also the Regional Approach above):
Consider a local, shift-invariant window W centered at (x,y). Circular windows are preferred. In that case, we define W as a ball of radius R:
2 2( , ) ( ', ')( ', '), ( , )R x y x y Rx yW x y B x y
Note: we may also use the set of relative offsets:
W ={(x’, y’) | ||(x’,y’)|| R2}.
The size or cardinality of window W is defined as:
( ', ') ( , )
( ) 1x y W x y
card W
for a rectangular NM window, card(W)=NM.
The radius R, and W in general (its shape, if not circular) are chosen according to the size of details to be taken into account for thrsholding. A non-shift invariant W is possible to obtain not only adaptive thresholds, but also to use adaptive windows.
Some local or adaptive thresholds are equivalent to a preprocessing or filtering step before a global thresholding method; for example the local mean is equivalent to background remotion by a circular-window convolution acting as high-pass filter and then using intermodal thresholding. A Gaussian filter may work still better, because of its spatial domain and spatial-frequency domain properties:
I(x, y) = I A(x, y) Ibkg (x, y), with [0,1] (usually 0.5)
and: Ibkg (x,y) I A (x, y)*((2 2)exp( (x2+y2)/(2 2))),
where determinates the size of those details to be blended with the background.
Auto Local Threshold Methods:
Local Mean
Local Median
Local MidGrey
*Local Trimmed and Sigma Means
Bernsen
Niblack
Sauvola * Excepting the trimmed mean, all methods are implemented in ImageJ
Testing all methods (original image at left-bottom, Sauvola gave white):
Local Mean thresholding method
This method selects the threshold as the mean umean(x,y) of the local greyscale distribution:
( ', ') ( , )
1
( )
[ ],
where ( , ) ( ', ')new mean
meanx y W x ycard W
u u u
u x y I x y
A variation of this method uses the umeanC, where C is a constant. The size of W (diameter, in a circular window) may be determined as less than half the size of features of interest, that are to be thresholded as foreground, following the Nyquist criterium for data sampling.
Parameter: the C value.
Note: When using a set of offsets W ={ (x’, y’) | ||(x’, y’)||2 R2}, rather than W(x,y), the equation for the mean should read:
( ', ')
1
( )( , ) ( ', ')mean
x ycardu x y I x x y y
WW
An advantage of the relative-coordinate notation is that W alone indicates a shift invariant window, and dependence on position, for a non-shift invariant window, may be then explicitly written as an attributed window or kernel W(x,y).
Note that the mean, in a circular window of radius R, is equivalent to a convolution with a radial rectangular function R( ||(x,y)|| ).
Median thresholding method
The median and the trimmed means are based on order-statistics sets Uos. To define Uos we take all pixel intensities u=I(x,y) in a local window W and sort them from umin to umax and obtain an ordered set Uos
={un}, with un un+1, and n=1,…, Nos. The reordered values un are called order statistics. Then we take the central order statistics:
)
)
/2 ( 2 /2
( 1 /2
1/2 [( mod 2) 0] ( )
[( mod 2) 1] ( )os os
os
medianosN N
osN
u u if N evenu
u if N odd
The median method selects the threshold as the median of the local greyscale distribution:
[ ]new medianu u u
A variation of this method uses the umedianC, where C is a constant.
Parameter: the C value.
Trimmed and Sigma Means
A trimmed-mean value is obtained by not using the first k and the last k order statistics of the local greyscale distribution, that is, by using the Nos–2k central order statistics of Uos. For each pixel at (x,y), we have a threshold selected as:
1
1
[ ],
where ( , )2
new trimmed meanN k
trimmed mean nos n kN
u u u
u x y uk
A variation of this method uses (utrimmed meanC), where C is a constant.
Parameters: the k and the C values.
The sigma mean is obtained by averaging only those pixel values u that lie within a two-sigma interval of the central value I(x,y), in a given neigborhood.
2( ', ') ( ', ')
1 ( ', ')( )
where ( ' ') | 2 ( , ) 2
sigma meanx y W I x y M
u u I x ycard M
M u I x y u I x y u
Then, the sigma-mean threshold is:
[ ]new sigma meanu u u
A variation of this method uses (usigma meanC), where C is a constant.
Parameters: the and the C values.
MidGrey thresholding method
This method selects the threshold as the mid-grey of the local greyscale distribution:
max min
min max
[ ],
where ( , ) ,2
and min ( , ), max ( , )
new midgray
midgray
u u
u u uu u
u x y
u W x y u W x y
A variation of this method uses (umidgrayC), where C is a constant.
Parameter: the C value.
Bernsen's thresholding method.
We define, in a local window W centered on (x,y):
min maxmin max
( , )2, where min max ( , )mid_gray
x y
u uu u W x y
then, the Bernsen’s threshold is defined as:
[ 128] if ( )[ ]
mid_gray local_contrast contrast_thresholdnew
mid_gray
u u < uu
u u otherwise
Parameter: the contrast threshold. The default value is 15. Any number, different than 0, will change the default value. The method uses a user-provided contrast threshold. The threshold is set at the midgrey value (the mean of the minimum and maximum grey values in the local window). If the local contrast (max-min) is below the contrast threshold the neighbourhood is considered to consist only of one class. In language C/C++, the above expression is implemented by the code:
if ( local_contrast < contrast_threshold ) pixel_u = ( mid_gray >= 128 ) ? object : background else pixel_u = (pixel >= mid_gray ) ? object : background
Bernsen, J (1986), "Dynamic Thresholding of Grey-Level Images", Proc. of the 8th Int. Conf. on Pattern Recognition
Sezgin, M & Sankur, B (2004), "Survey over Image Thresholding Techniques and Quantitative Performance Evaluation", Journal of Electronic Imaging 13(1): 146-165, <http://citeseer.ist.psu.edu/sezgin04survey.html>
Niblack's thresholding method:
2
( ', ') ( , )
( ', ')
( )
[ ( )]
where ( , )mean
new mean stddev
x y W x ystddev
u I x y
card W
u u u k u
u x y
Parameter 1: is the k value. The default value is 0.2 for bright objects and -0.2 for dark objects. Any other number than 0 will change the default value.
Niblack, W (1986), An introduction to Digital Image Processing. Prentice-Hall.
Sauvola's thresholding method:
This is a variation of Niblack's method, with
11 stddev
new mean
uu
rku u
Parameter 1: is the k value. The default value is 0.5. Any other number than 0 will change the default value.
Parameter 2: is the r value. The default value is 128. Any other number than 0 will change the default value
Sauvola, J & Pietaksinen, M (2000), "Adaptive Document Image Binarization", Pattern Recognition 33(2): 225-236,
Hybrid Segmentation Methods Hybrid segmentation methods refers to either:
Using both global and local information to decide the pixel label. Combining two or more global (or local) methods.
Method Combination via Linear blending
(A) Before binarization, or for preprocessing
For graylevel image operators we have the linear combination or alpha-blending of the pixel attribute outputs from methods A and B:
(1 ) method A method Bu u u
0 1 : interpolates between method outputs A and B,
1/2 : gives the average u1/2 of methods A and B,
0 : extrapolates method A, in “oposite direction” to method B, 1 : extrapolates method B, in “oposite direction” to method A.
(B) For binary image operators:
For binary operators, such as global and local thresholds, Marquez [in preparation] proposes the morphological interpolation and extrapolation of binary images (linear morphing):
0
( , )(1 ) ( ( , )) ( ( , ))d method A method A
I x yinterior L I x y I x y
D D
with 0 1 : to interpolate between method outputs A and B,
1/2 : I1/2(x,y) is the average of methods A and B,
0 : to extrapolate method A, in “oposite direction” to method B, 1 : to extrapolate method B, in “oposite direction” to method A.
where we define:
the 4-connected boundary components of a binary image I(x, y) as:
8( ,́ )́ ( , )
( , ) ( ( , ) 1) ( ( ', ') 0) , ( , ) dom( ) ,x y x y
I x y I x y I x y x y I
N
where the 8-adjacent neighborhood of point (x,y) is the set:
8 ( 1, ), ( 1, 1), ( , 1), ( 1, 1),( 1, ), ( , 1), ( 1, 1), ( 1, 1)
( , ) x y x y x y x yx y x y x y x y
x y
N
The 8-connected boundary is obtained by using the 4-adjacent neighborhood of point (x, y):
4 ( 1, ), ( , 1), ( 1, ), ( , 1)( , ) x y x y x y x yx y N ,
The 8-connected boundary components are also obtained from the morphological internal gradient, defined in terms of the mathematical morphology erosion operator and the structuring element E4:
8 4\ ( ).f f f E
the signed Euclidean Distance Transform:
A
11
interior( )exterior( )
( ( )) {( , ( )) | sgn min }
sgn if Iif I
I d d
q
pp
D p p p p q
and the Isodensity (isolevel) sets/contours of a graylevel image J:
0( ) ( , )| ( )0u=uL J u u J u p p
Evaluating Segmentation Methods
Expert Evaluation:
The correct segmented image is performed manually by an expert.
“Blind” Evaluation:
A reference image is used for comparison, or other method is used as reference.
To evaluate a segmentation method, there exist many ways; one is to count correct segmented pixels with respect to a reference:
Correct Segmentation Accuracy (CSA):
total number of correct segmented pixels( dom ) = total number of pixels at the image
CSAcard I
When comparing segmented regions (either between two methods, or from one method and an expert reference), the Dice Coefficient is a measure often used to compare in general two regions. It is proportional two the number of shared pixels over the total number of pixels in two regions:
2 ( )( ) ( )
a b
a b
card R RDC
card R card R
Another way to compare results, is to subtract the segmented image Iα using the method α, from the original image I (or a reference image), to obtain any the following error measures: Mean-squared difference msd(I, I ):
2
( , ) dom( )
1, , ,
(dom( ) ) x y I
msd I I I x y I x ycard I
where, for a N×M image I, we have card(dom(I)) = NM.
Number of sites of disagreement nsd( I, I), given a tolerance :
( , ) dom( )
1, , ,
(dom( )) x y I
nsd I I I x y I x ycard I
Largest absolute difference ldif( I, I) :
( , ) dom( )
, max , ,x y I
ldiff I I I x y I x y
Total signed difference tdiff( I, I),
( , ) dom( )
, , ,x y I
tdiff I I I x y I x y
Total unsigned difference tudiff( I, I), (or SAD: Sum of Absolute Differences):
( , ) dom( )
, , , ,x y I
tudiff I I SAD I I I x y I x y
For binary shape components, defined as I=interior(I), and I=interior(I) an non-overlapping measure comes from the count of pixels of their simetric difference, normalized by the total pixels in their union or in the reference set:
( ) ( )cardoverlapU
card
I I I I
I II I
c c
( ) ( )cardoverlap
card
I I I I
I II
c c
Morphological Mean-squared Error (or Difference)
In [Márquez2006] it is proposed a shape-based error or difference measure of the Euclidean Distance Transforms (or fields) from the binary segmented images:
2
( , ) dom( )
1, , ,
(dom( ) ) x y I
mmsd I I I x y I x ycard I
D D
A morphological SAD, a morphological sdiff, etc., are also possible, in a similar fashion.
The above measures can be taken only on a ROI or subset of interest, for example along a contour or boundary kI :
2
( , )
1, , ,
( ))k
kk x y I
Imsd I I I x y I x ycard I
where the restriction of f to A, denoted by f |A indicates to take only those values of f, restricted to the set A (that is, f is only defined on (dom f) … A). A qualitative assessment of segmentation errors can be performed by visualizing the (signed or squared) differential images (or fields):
, , ,diffI x y I x y I x y
2, , ,sdiffI x y I x y I x y
, , ,diff x y I x y I x y D D D
2, , ,sdiff x y I x y I x y D D D
Another comparison method, focused on the boundaries, and suggested by [Marquez2010], consists in isolating the signed attribute values of D(I(x,y)) along the boundary of I(x,y), and viceversa, obtaining attributed boundaries:
, ,, or ,I x y I x yI x y I x y D D
A display in color scale of attributed boundaries is especially useful in 3D. Note that, ideally, we should have, in a signed EDT:
( , )
,abs , 0x y I
I x yI x y
D
Consider two binary images I , I. We define in general the following measures,
restricted to the boundary of image I :
( , )
,
1( )
( , ) ,fx y
x yI
Icard Imatch I I g f I x y
D
where scalar function f() is either the absolute value abs(), the square power sq() = ()2, the inverse inv() = ()1, or the identity, and g() is, respectively, the identity, the square
root sqr() = ()1/2, the inverse ()1 or the identity. Other functions f , g could be defined to weight differently the distance between shape components of binary images I , I. Then, two binary images I, I are the same, and are perfectly aligned if
( , )absmatch I I = ( , )sqmatch I I =0. To simplify, in the following, and without
loss of generality, we chose f() = sq(). If I, I are not aligned (even if equal), a
registration process would minimize either measure, and if I, I are different, we consider the following similarity measures:
( , ) min , ( )sqsimilarity I I match I I θ
θT
T
where T is a geometric transform with a set of parameters .
ideally, ( , ) ( , )
, ,, abs , 0x y x yI I
x y x yI II x y I x y
D D
abs( , )
,
1( )
( , ) abs ,x y
x yI
Icard Imatch I I I x y
D
( , )
,
1( )
( , ) ,x y
x yI
Icard Ishift I I I x y
D
sq( , )
,
21( )
( , ) ,x y
x yI
Icard Imatch I I I x y
D
inv
( , ),
11 1( ) ,
( , )x y
x yI I
card I I x ymatch I I
D
Finally, a comparison between two method results , is possible, taking, in the above equations, I as the reference image I.
Signed Variances There is a number of variance-like descriptors, where some difference of pixel values and the local mean is taken into account. This allows to define a parametric image where the pixel attribute indicate the signed difference, using a special color scale for negative and positive values. Examples of such descriptors are the signed variance and the signed standard deviation (see equation (5)):
21var sgn( )i, j i, j i, j i-k, j-l i, j
k,l W
I Icard W
(10a)
21sgn( )i, j i, j i, j i-k, j-l i, j
k,l W
I Icard W
(10b)
[ 0]+1 01 0
where sgn( ) = 1 xif xif x
x
