Assignment 2Basic Image Processing Algorithms

Fall 2017

General informations● during the semester there will be 3-4 programming assignments with different

level of complexity - for different number of assignment-scores○ you have to reach at least 50 points to be able to participate to the exam

(required minimum level)○ you have to reach at least 80 points as a necessary condition to Offered Final Grade

● the deadlines for the different assignments will be published separately

2

Assignment 2: Pseudo mean-shift segmentation

● In this assignment you have to implement the ‘dummy version’ of the mean-shift segmentation algorithm (with the help of a validator script)

● the point value of this assignment: maximum 40 points● deadline: 23:59, 28 November 2017;● You have to upload the functions into the assignment2 folder on the SVN

server

3

The concept of Mean Shift SegmentationMean shift is a nonparametric iterative algorithm or a nonparametric density gradient estimation using a generalized kernel approach.

Mean shift is used for image segmentation, clustering, visual tracking, space analysis, mode seeking etc.

The algorithm:For each data point, mean shift defines a window around it and computes the mean of data points. Then shifts the center of the window to the mean and repeats the algorithm until it converges.

Topological backgroundLet F be the concatenated rgb and xy space which we call feature matrix.

In this F space every data point is a (1×1×5) size vector (feature vector) containing [x,y,r,g,b] coordinates alongside the 3rd dimension.

We assume that there is a random variable with an unknown probability density function, and each data point in F is a sample of that random variable.

Topological backgroundWe are looking for the nearest local maximum on the probability density function that we can reach from that data point.

Each data point from which the algo- rithm leads to the same maximum will belong to the same cluster. Therefore we can clusterize (segment) the image based on the different properties of the data points in the F space.

Figure source: https://www.cse.unr.edu/~bebis/CS773C/ObjectRecognition/Papers/Comaniciu02.pdf#page=9

The algorithmFor each data point...

The algorithmFor each data point, mean shift defines a window around it...

The algorithmFor each data point, mean shift defines a window around it, and computes the mean of data points.

The algorithmFor each data point, mean shift defines a window around it, and computes the mean of data points.

Then shifts the center of the window to the mean...

The algorithmFor each data point, mean shift defines a window around it, and computes the mean of data points.

Then shifts the center of the window to the mean and repeats the algorithm...

The algorithmFor each data point, mean shift defines a window around it, and computes the mean of data points.

Then shifts the center of the window to the mean and repeats the algorithm...

The algorithmFor each data point, mean shift defines a window around it, and computes the mean of data points.

Then shifts the center of the window to the mean and repeats the algorithm...

The algorithmFor each data point, mean shift defines a window around it, and computes the mean of data points.

Then shifts the center of the window to the mean and repeats the algorithm til it converges (so the length of the shift vector is less than conv_threshold).

The algorithmFor each data point, mean shift defines a window around it, and computes the mean of data points.

Then shifts the center of the window to the mean and repeats the algorithm til it converges (so the length of the shift vector is less than conv_threshold).

Then assigns the original data point to that local maximum (in an assignment vector).

What you have to doIn this assignment your task is to implement the Poor man’s Mean Shift algorithm.

We have created a validator script, available at: http://users.itk.ppke.hu/~nasma1/bipa/assignment2_meanshift_validator.m

This validator script tries to call the following functions (which you have to create):● {NEPTUN}_feature_extractor● {NEPTUN}_kernel_function● {NEPTUN}_maximum_finder● {NEPTUN}_mean_shift

Where {NEPTUN} is your NeptunID (e.g. B4TM4N_kernel_function)

The functions are described on the upcoming slides.

{NEPTUN}_feature_extractor

Create a function that:● realizes a feature extractor,● parameters of this function should be as follows:

○ input: image_matrix (N×M×3)○ output: feature_matrix (N×M×5)

● it should work with color images (check if this holds).

The feature extractor outputs an (N×M×5) size matrix where the 1st and 2nd layers along the 3rd dimension are the row and column coordinates.

{NEPTUN}_feature_extractor – explanation

image_matrix

feature_matrixN

M3

r g b

N

M5

r g bx y

Where x and y like

Create a function that:● realizes the K kernel function,● parameters of this function should be as follows:

○ input1: a (1×5 size feature vector)○ input2: lambda○ input3: bandwidth○ output: K

● it should work with row vectors

The kernel function outputs a number according to the equation:where , h is the bandwidth,and in case of Poor man’s Mean Shift:

Also there are more complex k functions, see the references for some of them, but please implement this one!

{NEPTUN}_kernel_function

{NEPTUN}_maximum_finder

Create a function that:● runs the Poor man’s Mean Shift algorithm on one data point (see slides 7–15)

● parameters of this function should be as follows:○ input1: F (N×M×5 size feature matrix)○ input2: origo (1×2 vector describing the row and column coordinates of the data point)○ input3: nbhd_size (size of the neighborhood window; e.g. for a 5×5 window nbhd_size=5)○ input4: lambda○ input5: bandwidth○ input6: conv_threshold○ output: m (1×5 size feature vector of the maximum point)

In this function's implementation you should calculate the weighted mean around the x data point as

where N(x) is the neighborhood of x. The mean shift itself is the vector m(x) – x

{NEPTUN}_maximum_finder – algorithm1.) Set x as the feature vector at the origo and convert it to a row vector2.) while not convergent…

a.) Select the neighboring feature vectors around xb.) for each neighboring feature vector xi…

i.) Calculate its difference from xii.) Use your {NEPTUN}_kernel_function to calculate K from this differenceiii.) sum the K values according to the equation of m(x) [see previous slide]

c.) Calculate m from the sum variablesd.) Calculate the shift vector from m and xe.) Let the new x be mf.) Let the origo be the round-ed first two coordinates of m (hint: origo = round(m(1:2))

g.) Check whether the maximum is found (the approaching converged), so the norm() of the shift vector is less than conv_threshold (if yes, exit while loop)

3.) Return m

{NEPTUN}_maximum_finder – comment

Please note that in the maximum finder function you are returning the calculated maximum vector which is NOT necessarily be an element of the feature matrix.

In other words:The returned m may contain non-integer values, please deal with it in the next function! (By rounding them to integers).

Do not round m in the maximum finder function because it will affect the approaching algorithm and misleads it!

{NEPTUN}_mean_shift

Create a function that:● runs Poor man’s Mean Shift on the whole image● parameters of this function should be as follows:

○ input1: image_matrix (N×M×3 size image matrix)○ input2: nbhd_size○ input3: lambda○ input4: bandwidth○ input5: conv_threshold○ output: segmented_image

● it should work on color images (check if this holds).

For each data point of F, this function calls the previously implemented functions with the appropriate parameters. The mean shift function should return a segmented image matrix where each pixel has the color of that pixel to which the maximum finding leads to. (See next slide for more detailed explanation.)

{NEPTUN}_mean_shift – explanationimage_matrix

r g b

feature_matrix

r g bx y

segmented_image

r g b

The green pixel in the I matrix has the grey fea- tures in the F matrix.

This grey feature vector converges to the purple one according to the mean shift algorithm.

The blue pixel in the S matrix has the same position as the green pixel in the I matrix. The blue pixel will belong to the cluster specified by the purple maximum vector.

Therefore the blue pixel’s color value will be the [r,g,b] coordi- nates of the purple feature vector.

Note that the purple vector is not an element of the feature_matrix but it is a maximum vector with non-integer values.

{NEPTUN}_mean_shift – explanationimage_matrix

r g b

feature_matrix

r g bx y

segmented_image

r g b

The orange pixel in the I matrix has the pink fea- tures in the F matrix.

This pink feature vector also converges to the purple one according to the mean shift algorithm.

Now there are two blue pixels, because the ‘new’ pixel belongs to the same cluster as the old one (their trajectory in the F space converges to the same maxi- mum). So those two pixels has the same color: the color coordi- nates of the purple vector.

{NEPTUN}_mean_shift – explanationimage_matrix

r g b

feature_matrix

r g bx y

segmented_image

r g b

The red pixel in the I matrix is has the brown features in the F matrix.

This brown feature vector converges to the balck one according to the mean shift algorithm.

The deep blue pixel in the S matrix has the same position as the red pixel in the I matrix. The deep blue pixel belongs to the cluster specified by the black maximum feature vector. There- fore its color will be different from the blue pixel’s color.

{NEPTUN}_mean_shift – pseudocodefunction [segmented_image] = {NEPTUN}_mean_shift(image_matrix,...) F = {NEPTUN}_feature_extractor(image_matrix);

foreach feature_vector in F origo = feature_vector(1:2); found_maximum = ...

{NEPTUN}_maximum_finder(F, origo, nbhd_size, lambda, ... bandwidth, conv_threshold);

segmented_image(origo) = found_maximum(3:5); endend

% where feature_vector and found_maximum is like [x,y,r,g,b];

Testing your functions1.) Download the validator script from here.2.) Download the test image from here.3.) Set your NeptunID in line 10, before you run this script.4.) Run the script to create the empty functions.5.) Write your function and validate it again with the script.

If you have implemented all the necessary functions, the validator script will try to segment the test image.

My results

Original image Segmented image

nbhd_size = 9lambda = 10bandwidth = 21conv_threshold = 0.1

Comments & ReferencesWe call this algorithm the Poor man’s Mean Shift because it is really dumb: this is the most informal and inaccurate implementation of the mean shift clustering; even the clustering (collecting really similar elements in one cluster) itself is missing.

References● http://en.wikipedia.org/wiki/Mean_shift● http://soe.rutgers.edu/~meer/TEACH/ADD/meanshiftslides.pdf● book of Richard Szeliski, page 292: http://szeliski.org/Book/