Sections 10 1Sections 10.1 Edge linking 10.2.7 (very briefly) … · “How did Andy Warhol get his...

INF 4300 14.10.09Image segmentationImage segmentation

“How did Andy Warhol get his inspiration?”Sections 10 1Sections 10.1

Edge linking 10.2.7 (very briefly)10.410.5

10.6.1A S S lbAnne S. Solberg

Anne Schistad Solberg, Department of InformaticsAnne Schistad Solberg, Department of Informatics

Today

• Segmentation approaches

1. Region growing1. Region growing

2. Region split and merge

3. Watershed

4. Edge-based segmentation

5. Segmentation by motion

Assumed known:

1. Edge detection, point and line detection (10.2.1-10.2.6)1. Edge detection, point and line detection (10.2.1 10.2.6)

2. Segmentation by thresholding (10.3)

Anne Schistad Solberg, Department of Informatics

SegmentationSegmentation

Segmentation is separation of one or more regions orSegmentation is separation of one or more regions or objects in a image based on a similarity criterion

A region in an image can be defined by its border (edge) or its interior, and the two representations are equal.

If you know the interior, you can always define the border, and vice versa.and vice versa.

Because of this, image segmenation approaches cantypically be divided into two categories, edge and region based methodsbased methods.


What is segmentation?What is segmentation?

What if you wanted to group all the pixels corresponding to thecar in this image into one group Would that be easy?


car in this image into one group. Would that be easy?

What is a region?A region R of an image I is defined as a connected homogenousA region R of an image I is defined as a connected homogenous subset of the image with respect to some criterion such as gray level or texture

A segmentation of an image f is a partition of f into severalA segmentation of an image f is a partition of f into several homogeneous regions Ri, i=1,….m

An image f can be segmented into regions Ri such as:

P(Ri) is a logical predicate defined over all points in Ri. It must be true for all pixels inside the region and false for pixels in other regions.

T i R d R i hb if th i i f t d


Two regions Ri and Rj are neighbors if their union forms a connected component.

Segmentation approaches

• Pixel-based segmentation: each pixel is segmentedbased on grey-level values, no contextualg y ,information (information from other neighboringpixels). Example: thresholding.

• Region-based segmentation: takes into accountgrey-levels from neighboring pixels by eitheri l di i il i hb i i l ( i i )including similar neighboring pixels (region growing), split-and-merge, or watershed segmentation.

• Edge based segmentation: Detect edges and link• Edge-based segmentation: Detect edges and link the edges together to form contours.


Need repetition of thresholding?

• Read section 10.3 in Gonzalez and Woods.

• English foils on thresholding can be found atEnglish foils on thresholding can be found at

http://www.uio.no/studier/emner/matnat/ifi/INF3300/h06/undervisningsmateriale/thresholding 2006.pdf/undervisningsmateriale/thresholding_2006.pdf


R i d b d hRegion vs. edge-based approaches

• Region based methods are robust because• Region based methods are robust because– Regions cover more pixels than edges and thus you have more

information available in order to characterize your regionWhen detecting a region you could for instance use texture which is not– When detecting a region you could for instance use texture which is not easy when dealing with edges

– Region growing techniques are generally better in noisy images where edges are difficult to detectedges are difficult to detect

• The edge based method can be preferable because:– Algorithms are usually less complex– Edges are important features in a image to separate regions

• The edge of a region can often be hard to find because of noise or occlusions


occlusions• Combination of results may often be a good idea

Edge-based Segmentation• A large group of methods based on edge information.• We only mention the simplest methods here.

• Rely on edges found in an image by edge detecting operators (discontinuities in gray level, color, texture, etc.)

• Image resulting from edge detection cannot be used as a• Image resulting from edge detection cannot be used as a segmentation result

• Post processing steps must follow to combine edges into edge p g p g gchains to represent the region border

• The more prior information used in the segmentation process, th b tt th t ti lt b bt i dthe better the segmentation results can be obtained

• The most common problems of edge-based segmentation are:– edge presence in locations where there is no border


– edge presence in locations where there is no border – no edge presence where a real border exists

Edge-based Segmentation

1. Edge detection (Gradient, Laplacian, LoG, Canny filtering) g)

2. Edge linking – Local Processing

• strength of the response of the gradient

• direction of the gradient vector

• Edges in a predefined neighborhood is linkedEdges in a predefined neighborhood is linked

if both magnitude and direction criteria is satisfied

– Global Processing via Hough Transform (later)


Why is more than a gradient operatorWhy is more than a gradient operator needed for edge segmentation?

• Most images will produce a very complicated edge map under the Sobel filter.O l l ill th di t it d b• Only rarely will the gradient magnitude be zero.

• Calculating an approximation to the gradient vector in an image will generally not tell you were the salient edges are.


Method 1

Russian ground to air missiles near Havana, Cuba. The imagewas taken by an American U2 aircraft on the 29th of August 1962.


Method 1

Magnitude of gradient image, min=0, max=1011


Method 1

Threshold at 250


Method 1

Threshold at 350


Method 1

• What if we assume the following:– All gradient magnitudes above a strict threshold are assumed to g g

belong to a bona fide edge.

– All gradient magnitudes above an unstrict threshold and connected to a pixel resulting from the strict threshold are also assumed to p gbelong to real edges.

• Hysteresis thresholding – Canny’s edge detection ( INF 2310)(see INF 2310).


Method 1

Result of hysteresis, are we really impressed?


Method 2

• One evident problem is the thickening of edges.

• One simple method for reducing this is based onOne simple method for reducing this is based on using directional information provided by the edge detector.

• The method is described in the next slide.


Method 2

• Quantize the edge directions into eight (or four) directions.

• For each gradient magnitude pixel with nonzero magnitude, inspect its two neighboring pixels in the directions from point 1directions from point 1.

• If the edge magnitude of any of these neighbors are higher than that under consideration, mark it for g ,deletion.

• When all pixels have been scanned, delete those k d f d l timarked for deletion.


Edge based SegmentationEdge-based Segmentation exampleLeft: Input imageRight: G_y

gg

Left: G_xRight: Result after Right: Result after edge linking


Edge based segmentation

• Advantages– Similar approach as human segment an image

W k ll i i ith d t t b t bj t d– Works well in images with good contrast between object and background

• Disadvantages– Do not work well on images with smooth transitions and low

contrast– Sensitive to noise– Edge linking is not trivial


Region growing• Grow regions by recursively including the neighboring pixels that G o eg o s by ecu s e y c ud g t e e g bo g p e s t at

are similar and connected to the seed pixel.• Connectivity is needed not to connect pixels in different parts of

the image.

• Similarity measures - examples: – Difference in grey levels for regions with homogeneous grey levels– Texture features for textured regions

Similary: grey level differenceCannot be segmented using greylevel similarity


y g y level similarity.

Region growing• Starts with a set of seeds (starting pixels)Sta ts t a set o seeds (sta t g p e s)

– Predefined seeds– All pixels as seeds– Randomly chosen seedsy

• Region growing steps (bottom-up method)– Find starting points– Include neighboring pixels with similar features (grey-level, texture,

color) A similarity measure must be selected.Two variants: 1 Select seeds from the whole range of grey levels in the image Grow1. Select seeds from the whole range of grey levels in the image. Grow

regions until all pixels belong to a region.2. Select seed only from objects of interest (e.g. bright structures). Grow

regions only as long as the similarity criterion is fulfilled.g y g y

• Problems:– Not trivial to find good starting points


Not trivial to find good starting points– Need good criteria for similarity

Region growing example: WeldRegion growing example: Weld inspection

X-ray image of defective weld SeedsCriteria used:

• Seeds: f(x,y) = 255

X ray image of defective weld

↓Seeds

↓( ,y)

• P(R) = TRUE if – | seed gray level – new | g y

pixel gray level | < 65

and

– New pixel must be 8-connected with at least one pixel in the regionpixel in the region

Result of region growing →


Similarity measure

• Intensity difference within a region (from a pixel to a seed or the mean of a region)

• Within a Maximum and Minimum value

• Distance between mean value of the regions (specially for region merging or splitting)(specially for region merging or splitting)

• Variance or standard deviation within a region• Variance or standard deviation within a region

• Difference in another feature, e.g. texture


Difference in another feature, e.g. texture

Region merging techniques

• One region based type of segmentation methods is the so called region merging method.g g g

• Very simple method. Initialization is done as follows:– Start by giving all the pixels a unique label (a possible variation is

to give groups of 2 and 2 or 4 and 4 pixels a unique label).

– All pixels in the image are assigned to a region.



• The rest of the algorithm is as follows:– In some predefined order, examine the neighbor regions of all p g g

regions and decide if the predicate evaluates to true for all pairs of neighboring regions.

– If the predicate evaluates to true for a given pair of neighboring p g p g gregions then give these neighbors the same label. The predicate is the similarity measure (can be defined based on e.g. region mean values or region min/max etc.).

– Continue until no more mergings are possible.

– Upon termination all region criteria will be satisfied.



• An interesting case for region merging is the image to the rightto the right.

• The aim is to separate the apples from the background.Thi i f• This image poses more of a challenge than you might think.



• We run a standard region merging procedure where all i l i iti ll ipixels initially are given a

unique label.

• If neighboring regions have g g gmean values within 10 gray levels they are fused,

• Regions are considered• Regions are considered neighbors in 8-connectivity.




How Andy Warhol really did it…



• A caveat:

• Remember that initialization is critical segmentationRemember that initialization is critical, segmentation results will in general depend on the initialization.

• The order in which the regions are treated will alsoThe order in which the regions are treated will also influence the result, in the images in the next slide the right image was flipped upside down before it was fed to the merging algorithm.



The right image was flipped upside down before being fedto the region merging algorithm, notice the differences

b t th t


between the two.

Split and merge

• Separate the image in regions based on a given similarity measure. Then merge regions based on y g gthe same or a different similarity measure

• The method is also called "quad tree" division

1. Set up some criteria for what is a uniform area (ex mean, i t t t )variance, texture, etc)

2. Start with the full image and split it in to 4 sub-images.

3. Check each sub-image. If not uniform, divide into 4 new sub-images

4. After each iteration, compare neighboring regions end merge if uniform according to the similarity measure.


g y


Split and merge exampleSplit and merge example


W t h d t ti th idWatershed segmentation – the idea

L k t th i t hi f ith b th ll d• Look at the image as a topographic surface, with both valleys and mountains

• Assume that there is a hole in each minima and the surface is• Assume that there is a hole in each minima and the surface is immersed into a lake

• The water will enter through the holes at the minima and flood theThe water will enter through the holes at the minima and flood the surface

• To avoid the water coming from two different minima to meet, a dam gis build

• Final step: the only thing visible of the surface would be the dams

• The walls between the dams are called the watershed lines


Watershed segmentationD t t t d i ith i il l (i t it )• Detect connected regions with similar value (intensity )

• Every minimum correspond to a region

M tl b t h d(i )• Matlab: watershed(im, conn)


Watershed segmentation

Water graduallyrise

At the pointwhere tworegions

Negative distance transform

regions flood, dams are built

Watershed transform of D



Original Original, Original image

Original, topographicview

Water rises Water nowand fill thedark background

Water nowfills one ofthe dark regions


The twoThe twobasins areabout to meet, dam constructionco st uct ostarts

Final segmentationresult

Video: http://cmm.ensmp.fr/~beucher/wtshed.html


• Can be used on images derived from:

– The intensity image

– Edge enhanced image

– Distance transformed image

• Thresholded image. From each foreground pixel, compute the distance to a background pixel.

– Gradient of the image

M t di t i• Most common: gradient image


Watershed algorithmL t ( ) b th i t i ( ft di t• Let g(x,y) be the input image (often a gradient image).

It has several local minima Let M M be the• It has several local minima. Let M1,…MR be thecoordinates of the regional minima.

• Let C(M ) be a set consisting of the coordinates of all• Let C(Mi) be a set consisting of the coordinates of all points belonging to the catchment basin associatedwith the regional mimimum Mi. g i

• Let T[n] be the set of coordinates (s,t) where g(s,t)<t{ }ntsgtsnT <= ),(|),(][

This is the set of coordinates lying below the plane g(x,y)=n

This is the candidate pixels for inclusion into the catchment basin, butwe must take care that the pixels do not belong to a different

{ }g ),(|),(][


we must take care that the pixels do not belong to a differentcatchment basin.

Watershed algorithm cont.

• The topography will be flooded with integer floodincrements from n=min-1 to n=max+1.

• Let Cn(Mi) be the set of coordinates of point in thecatchment basin associated with Mi that are floodedi

at stage n. This must be a connected componentand can be expressed as Cn(Mi) = C(Mi)∩T[n] (onlyth ti f T[ ] i t d ith b i M )the portion of T[n] associated with basin Mi)

• Let C[n] be the union of all flooded catchments at stage n: Rstage n:

)(]1[max

and

)(][1

i

R

in

R

i

MCC

MCnC

U

U=

=+

=


)(]1[max1

ii

MCC U=

+

C 1[M1] C [M ]Dam constructionStage n 1: two basins They form Step n-1C[n 1]

Cn-1[M1] Cn-1[M2]

Stage n-1: two basins. They form separate connected components.

To consider pixels for inclusion in

Step n 1C[n-1]

basin k in the next step (after flooding), they must be part of T[n], and also be part of the connected Step n

qp

component q of T[n] that Cn-1[k] is included in.

Use morphological dilation iteratively

p

Use morphological dilation iteratively

The dilation of C[n-1] is constrained to q

The dilation can not be performed on pixels that would cause two basins to be merged (form a single connected g ( gcomponent)

Watershed algorithm cont.I iti li ti l t C[ i 1] T[ i 1]• Initialization: let C[min+1]=T[min+1]

• Then recursively compute C[n] from C[n-1] as:L t Q b th t f t d t i T[ ]– Let Q be the set of connected components in T[n].

– For each component q in Q, there are three possibilities:

1. q∩C[n-1] is empty – new minimum

Combine q with C[n-1] to form C[n]

2. q∩C[n-1] contains one connected component of C[n-1]

q lies in the catchment basin of a regional minimumq lies in the catchment basin of a regional minimum

Combine q with C[n-1] to form C[n]

3. q∩C[n-1] contains more than one connected component of C[n-1]

q is then on a ridge between catchment basins, and a dam must be built to prevent overflow.

Construct a one-pixel dam by dilating q∩C[n-1] with a 3x3


structuring element, and constrain the dilation to q.

Watershed alg.

• Constrain n to only existing intensity values of g(x,y) (obtain them from the histogram)( g )



Challenge: over-segmentation

Image IGradient magnitude image (g)image (g)

Watershed of g W t h d f of g Watershed of

smoothed g

Using the gradient image directly can cause over-segmentation because of noise and small irrelevant intensity changes

Improved by smoothing the gradient image or using markers

Solution: Watershed with markers

A marker is a connected component in the image A marker is a connected component in the image

Can be found by intensity, size, shape, texture etc

Internal markers are associated with the object (a region surrounded j ( gby bright point (of higher altitude))

External markers are associated with the background (watershed lines)lines)

Segment each sub-region by some segmentation algorithm

How to find markers

• Apply filtering to get a smoothed image

• Segment the smooth image to find the internal markers. – Look for a set of point surrounded by bright pixels. How this segmentation should be

done is not well defined. Many methods can be used.

• Segment the smooth image using watershed to find the externalmarkers, with the restriction that the internal markers are theonly allowed regional minima. The resulting watershed lines arethen used as external markers.

• We now know that each region inside an external marker consists of a single object and its background. Apply a segmentation algorithm (watershed region growing thresholdsegmentation algorithm (watershed, region growing, thresholdetc. ) only inside each watershed.


W t h d d d lWatershed – advanced example

Original Opening Threshold globalOriginal Opening Threshold, global

Matlab:

Imopen bwdist imimposemin imregionalmax

Watershed of inverse Second watershed to split Distance transform- Distance from a point in a

imregionalmax watershed


of distance transform cellsDistance from a point in a

region to the border of the region

Watershed – example 3

O i i l Opening Th h ldOriginal Opening Threshold

Matlab:i• imopen

• imimposemin• bwmorph

Find internal and Watershed

• watershed


external markers from gradient image

Watershed

• Advantages– Gives connected componentsp

– A priori information can be implemented in the method using markers

• Disadvantages :• Disadvantages :– Often needs preprocessing to work well

– Over-segmentation can be a problem


Segmentation by motion

• In video surveillance one of the main tasks is to segment foreground objects from the background g g j gscene to detect moving objects.

• The background may have a lot of natural movements and changes (moving trees, snow, sun, shadows etc)shadows etc)

Two main segmentation approaches are:• Two main segmentation approaches are:– Intensity difference between two frames are thresholded

– Background subtraction


g

Background subtraction

• Approach:– Estimate the background image (per pixel) by computing the mean g g (p p ) y p g

and variance of the n previous time frames at pixel (x,y).

– Subtract background image from current frame

– Differences above a threshold are interesting pixelsDifferences above a threshold are interesting pixels

– Note that the background is updated by only considering the n previous frames. This eliminated slow trends.


Other segmentation methods

• Active contour/snakes: Adjust a curve around the object. Initialize a curve and then change it based on a cost function (INF 5300)

• PDE/Level set: Describes the image as partial differential equations. Look at segmentation as a cut in 3D plan. Initialize a curve and then change it based on a cost function.change it based on a cost function.

• Graph-cutting methods.

• Statistical models (MRF), classification

• Neural network for classification of pixels• Neural network for classification of pixels• And many others…………


Date post:	13-Apr-2018
Category:	Documents
Upload:	lynhan
View:	216 times
Download:	3 times

Sections 10 1Sections 10.1 Edge linking 10.2.7 (very briefly) … · “How did Andy Warhol get his...

Documents