Lecture 6Image Segmentation

Slides by:David A. Forsyth

Clark F. OlsonSteven M. Seitz

Linda G. Shapiro

2

Segmentation and grouping

Motivation: obtain a compact representation for an image (or set of edge pixels or motion sequence, etc.)

Should support application.

Much work has been done, but broad theory is absent at present.

3

Image segmentation

Collecting edge pixels into curves, lines, etc. is one form of segmentation.

Another form of segmentation is grouping (or clustering) image pixels into coherent regions.

• Collect together pixels that “belong together”• Segments are usually disjoint• Segments are often connected• Union of segments should cover image

4

How do we decide what pixels belong together?

Segmentation

5

What are properties of pixels (or other objects) that “belong together?”

Segmentation cues

6

Segmentation cues

7

Segmentation as thresholding

The simplest way to segment a (grey-level) image is to apply a threshold at some brightness value.

How do we know what value to use for thresholding?

8

Thresholding

A histogram can give us information about the distribution of pixels.

Bimodal distributions are best for this technique.• Modes are not always easy to determine.

9

Thresholding

Most images are not straightforward to threshold.

Where do we threshold this distribution?

10

Thresholding application

Thresholding works best in controlled situations. For example, it is sometimes useful in medical imaging.

11

Segmentation as clustering

Goal: cluster together pixels, that belong together.

Agglomerative clustering:• start with every pixel as separate

cluster• combine closest two clusters• repeat

Divisive clustering:• start with every pixel in one

cluster• split cluster along best boundary• repeat

Pixel-Pixel distances:• color• texture• proximity• etc.

Cluster-Cluster distances• single-link clustering

- minimum distance• complete-link clustering

- maximum distance• average-link clustering

- average distance

12

Segmentation as clustering

Overall methodology for determining the location and composition of clusters is not straightforward.

If we know the cluster locations, how should we assign points to clusters?

If we know the assignment of points to clusters, how should we assign locations to clusters?

13

K-means clustering

Must choose a fixed number of clusters.

Choose cluster centers and point-cluster allocations to minimize error. • Can’t do this by search, because there are too many possible allocations.

Algorithm• fix cluster centers; allocate points to closest cluster• fix allocation; compute best cluster centers

Note the distance doesn’t need to be just proximity.• be careful about scaling

x j i2

jelements of i'th cluster

iclusters

14

K-means clustering

How many clusters should you use?

K-means clustering will always converge after some number of iterations.

However, it won’t always be the global minimum of the error measure:

x j i2

jelements of i'th cluster

iclusters

15

K-means clustering using intensity alone and color alone(5 clusters, no location information included in the distance)

Image Clusters on intensity Clusters on color

K-means clustering

16

Divisive clustering

In divisive clustering, an image is represented by a graph whose nodes are pixels or small groups of pixels.

The goal is to partition the vertices into disjoint sets so that the similarity within each set is high andacross different sets is low.

17

Minimal cuts

Let G = (V,E) be a graph. Each edge (u,v) has a weight cu,v that represents the similarity between u and v.

Graph G can be broken into 2 disjoint graphs with node sets A and B by removing edges that connect these sets.

Let

One way to segment G is to find the minimal cut.- Fast algorithms for this exist.

18

Normalized cuts

Using minimal cuts favors small sets of pixels to be divided from a larger group.

“Normalized cuts” can improve this.• cuts are weighted using the “volume” of the set• volume: all weights that are connected to nodes in the set

19

Figure from “Image and video segmentation: the normalized cut framework”, by Shi and Malik, copyright IEEE, 1998

Example of divisive clustering using color, texture, and location.

Normalized cuts

20

Segmentation as contour optimization

Another approach to segmentation optimizes a contour such that it lies along a strong boundary in the image.

21

Active contoursThese methods start with a particular contour and iteratively improve it until a final state is reached.

Several names:• Active contours• Snakes• Intelligent Scissors

Example from: E. N. Mortensen and W. A. Barrett, Intelligent Scissors for Image Composition, in ACM Computer Graphics (SIGGRAPH `95), pp. 191-198, 1995

22

Intelligent scissors

The goal is to find path a between two pixels that stays on image edges.

Each pixel is given a cost depending on how “edgy” it is:• Edges have low cost• Non-edge have high cost

Now must find lowest cost path.

23

Intelligent scissors

Can be formulated as a graph algorithm:• Each pixel in the image is a node• There is a cost between each pair of adjacent pixels (rather than a

cost for each pixel)• Contour is path between two pixels that minimizes the cost

between each pair of adjacent pixel in the contour

How do we determine the “shortest” path in the graph?

Can determine all shortest paths for one starting pixel in less than 1 second for reasonable image sizes.

24

Expectation maximization

The EM (expectation maximization) algorithm is common in computer vision and is a popular variation of this method.

• Clusters are represented by Gaussian distributions• E step: assign points to clusters

- Assignment is “soft” using probability• M step: determine mean, standard deviation of Gaussian

25

Figure from “Color and Texture Based Image Segmentation Using EM and Its Application to Content Based Image Retrieval”,S.J. Belongie et al., Proc. Int. Conf. Computer Vision, 1998.

3 clusters 4 clusters

Segmentation with EM