Lecture: k-means & mean-shift clustering - Stanford...

Lecture 11 -Stanford University

Lecture:k-means&mean-shiftclustering

JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab

26-Oct-171


Recap:ImageSegmentation

• Goal:identifygroupsofpixelsthatgotogether

26-Oct-172


Recap:GestaltTheory• Gestalt:wholeorgroup

– Wholeisgreaterthansumofitsparts– Relationshipsamongpartscanyieldnewproperties/features

• Psychologistsidentifiedseriesoffactorsthatpredisposesetofelementstobegrouped(byhumanvisualsystem)

Untersuchungen zur Lehre von der Gestalt,Psychologische Forschung, Vol. 4, pp. 301-350, 1923http://psy.ed.asu.edu/~classics/Wertheimer/Forms/forms.htm

“I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnessesand nuances of colour. Do I have "327"? No. I have sky, house, and trees.”

Max Wertheimer(1880-1943)

26-Oct-173


Recap:GestaltFactors

• Thesefactorsmakeintuitivesense,butareverydifficulttotranslateintoalgorithms.

26-Oct-174


Whatwillwelearntoday?

• K-meansclustering• Mean-shiftclustering

26-Oct-175

Reading:[FP] Chapters:14.2,14.4D.Comaniciu andP.Meer,MeanShift:ARobustApproachtowardFeatureSpaceAnalysis,PAMI2002.




26-Oct-176



ImageSegmentation:ToyExample

• Theseintensitiesdefinethethreegroups.• Wecouldlabeleverypixelintheimageaccordingtowhich

oftheseprimaryintensitiesitis.– i.e.,segmenttheimagebasedontheintensityfeature.

• Whatiftheimageisn’tquitesosimple?

intensityinput image

blackpixels graypixels

whitepixels

1 23

Slide credit: Kristen Grauman

26-Oct-177


Pixelcou

nt

Inputimage

InputimageIntensity

Pixelcou

nt

Intensity


26-Oct-178


• Nowhowtodeterminethethreemainintensitiesthatdefineourgroups?

• Weneedtocluster.

InputimageIntensity

Pixelcou

nt


26-Oct-179


• Goal:choosethree“centers”astherepresentativeintensities,andlabeleverypixelaccordingtowhichofthesecentersitisnearestto.

• BestclustercentersarethosethatminimizeSumofSquareDistance(SSD)betweenallpointsandtheirnearestclustercenterci:

Slid

e cr

edit

: Kr

iste

n G

raum

an

0 190 255

1 23

Intensity

26-Oct-1710

SSD = x − ci( )2x∈clusteri∑

clusteri∑


ClusteringforSummarization

Goal:clustertominimizevarianceindatagivenclusters– Preserveinformation

Whether is assigned to

Cluster center Data

Slide:DerekHoiem

c*, δ* = argminc, δ

1N

δiji

K

∑ ci − x j( )2

j

N

∑

x j ci

26-Oct-1711


Clustering• Withthisobjective,itisa“chickenandegg”problem:– Ifweknewtheclustercenters,wecouldallocatepointstogroupsbyassigningeachtoitsclosestcenter.

– Ifweknewthegroupmemberships,wecouldgetthecentersbycomputingthemeanpergroup.

Slid

e cr

edit

: Kr

iste

n G

raum

an

26-Oct-1712


K-meansclustering1. Initialize():clustercenters

2. Compute:assigneachpointtotheclosestcenter– denotesthesetofassignmentforeachtoclusteratiterationt

1. Computer:updateclustercentersasthemeanofthepoints

1. Update,RepeatStep2-3tillstopped

Slide:DerekHoiem

c1,...,cK

δ t = argminδ

1N

δ t−1iji

K

∑ ct−1i − x j( )2

j

N

∑

ct = argminc

1N

δ tiji

K

∑ ct−1i − x j( )2

j

N

∑

t = t +1

x j ciδ t

t = 0

δ t

ct

26-Oct-1713


K-meansclustering1. Initialize():clustercenters

2. Compute:assigneachpointtotheclosestcenter

1. Computer:updateclustercentersasthemeanofthepoints

2. Update,RepeatStep2-3tillstopped

Slide:DerekHoiem

c1,...,cKt = 0

δ t

ct

• Commonlyused:randominitialization• OrgreedilychooseKtominimizeresidual

• Typicaldistancemeasure:• Euclidean• Cosine• Others

• doesn’tchangeanymore.ct

sim(x, !x ) = xT !xsim(x, !x ) = xT !x x ⋅ !x( )

t = t +1

ct = argminc

1N

δ tiji

K

∑ ct−1i − x j( )2

j

N

∑

26-Oct-1714


K-meansclustering

Illustration Source: wikipedia

1. Initialize Cluster Centers

2. Assign Points to Clusters

3. Re-compute Means

Repeat (2) and (3)

• Javademo:http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/AppletKM.html

26-Oct-1715


• Convergestoalocalminimum solution– Initializemultipleruns

• Betterfitforsphericaldata

• NeedtopickK(#ofclusters)

26-Oct-1716

K-meansclustering


SegmentationasClustering

2clusters

26-Oct-1717

Originalimage

3clusters


K-Means++

• Canwepreventarbitrarilybadlocalminima?

1. Randomlychoosefirstcenter.2. Picknewcenterwithprob.proportionalto

– (Contributionofx tototalerror)

3. RepeatuntilK centers.

• Expectederror*optimal

Arthur & Vassilvitskii 2007

Slid

e cr

edit

: St

eve

Seit

z

26-Oct-1718

x − ci( )2

=O log K( )


FeatureSpace• Dependingonwhatwechooseasthefeaturespace,wecangrouppixelsindifferentways.

• Groupingpixelsbasedonintensitysimilarity

• Featurespace:intensityvalue(1D)Slide credit: Kristen Grauman

26-Oct-1719


FeatureSpace• Dependingonwhatwechooseasthefeaturespace,wecan

grouppixelsindifferentways.

• Groupingpixelsbasedon colorsimilarity

• Featurespace:colorvalue(3D)

R=255G=200B=250

R=245G=220B=248

R=15G=189B=2

R=3G=12B=2

R

GB


26-Oct-1720


FeatureSpace• Dependingonwhatwechooseasthefeaturespace,wecan

grouppixelsindifferentways.

• Groupingpixelsbasedon texturesimilarity

• Featurespace:filterbankresponses(e.g.,24D)

Filterbankof24filters

F24

F2

F1

…Slide credit: Kristen Grauman

26-Oct-1721


SmoothingOutClusterAssignments

• Assigningaclusterlabelperpixelmayyieldoutliers:

• Howcanweensuretheyarespatiallysmooth? 1 2

3?

Original Labeledbyclustercenter’sintensity


26-Oct-1722


SegmentationasClustering• Dependingonwhatwechooseasthefeaturespace,wecangrouppixelsindifferentways.

• Groupingpixelsbasedonintensity+position similarity

ÞWaytoencodebothsimilarity andproximity.Slide credit: Kristen Grauman

X

Intensity

Y

26-Oct-1723


K-MeansClusteringResults• K-meansclusteringbasedonintensityorcolorisessentiallyvectorquantizationoftheimageattributes– Clustersdon’thavetobespatiallycoherent

Image Intensity-basedclusters Color-basedclusters

Imag

e so

urce

: Fo

rsyt

h &

Pon

ce

26-Oct-1724


K-MeansClusteringResults

• K-meansclusteringbasedonintensityorcolorisessentiallyvectorquantizationoftheimageattributes– Clustersdon’thavetobespatiallycoherent

• Clusteringbasedon(r,g,b,x,y)valuesenforcesmorespatialcoherence

Imag

e so

urce

: Fo

rsyt

h &

Pon

ce

26-Oct-1725


Howtoevaluateclusters?

• Generative– Howwellarepointsreconstructedfromtheclusters?

• Discriminative– Howwelldotheclusterscorrespondtolabels?

• Canwecorrectlyclassifywhichpixelsbelongtothepanda?

– Note:unsupervisedclusteringdoesnotaimtobediscriminativeaswedon’thavethelabels.

Slide:DerekHoiem26-Oct-1726


Howtochoosethenumberofclusters?Trydifferentnumbersofclustersinavalidationsetandlookatperformance.

Slide:DerekHoiem26-Oct-1727


K-Meansprosandcons• Pros

• Findsclustercentersthatminimizeconditionalvariance(goodrepresentationofdata)

• Simpleandfast,Easytoimplement• Cons

• NeedtochooseK• Sensitivetooutliers• Pronetolocalminima• Allclustershavethesameparameters

(e.g.,distancemeasureisnon-adaptive)

• *Canbeslow:eachiterationisO(KNd)forNd-dimensionalpoints

• Usage• Unsupervisedclustering• Rarelyusedforpixelsegmentation

26-Oct-1728




26-Oct-1729



Mean-ShiftSegmentation

• Anadvancedandversatiletechniqueforclustering-basedsegmentation

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

D.Comaniciu andP.Meer,MeanShift:ARobustApproachtowardFeatureSpaceAnalysis,PAMI2002. Slid

e cr

edit

: Sv

etla

na L

azeb

nik

26-Oct-1730


Mean-ShiftAlgorithm

• IterativeModeSearch1. Initializerandomseed,andwindowW2. Calculatecenterofgravity(the“mean”)ofW:3. Shiftthesearchwindowtothemean4. RepeatStep2untilconvergence

Slid

e cr

edit

: St

eve

Seit

z

26-Oct-1731


Region ofinterest

Center ofmass

Mean Shiftvector

Mean-Shift

SlidebyY.Ukrainitz &B.Sarel

26-Oct-1732


Region ofinterest

Center ofmass

Mean Shiftvector

Mean-Shift


26-Oct-1733


Region ofinterest

Center ofmass

Mean Shiftvector

Mean-Shift


26-Oct-1734


Region ofinterest

Center ofmass

Mean Shiftvector

Mean-Shift


26-Oct-1735


Region ofinterest

Center ofmass

Mean Shiftvector

Mean-Shift


26-Oct-1736


Region ofinterest

Center ofmass

Mean Shiftvector

Mean-Shift


26-Oct-1737


Region ofinterest

Center ofmass

Mean-Shift


26-Oct-1738


Tessellate the space with windows Run the procedure in parallel SlidebyY.Ukrainitz&B.Sarel

RealModalityAnalysis

26-Oct-1739


The blue data points were traversed by the windows towards the mode. SlidebyY.Ukrainitz&B.Sarel

RealModalityAnalysis

26-Oct-1740


Mean-ShiftClustering

• Cluster:alldatapointsintheattractionbasinofamode

• Attractionbasin:theregionforwhichalltrajectoriesleadtothesamemode


26-Oct-1741


Mean-ShiftClustering/Segmentation• Findfeatures(color,gradients,texture,etc)• Initializewindowsatindividualpixellocations• Performmeanshiftforeachwindowuntilconvergence• Mergewindowsthatendupnearthesame“peak”ormode

Slid

e cr

edit

: Sv

etla

na L

azeb

nik

26-Oct-1742


Mean-ShiftSegmentationResults

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html Slid

e cr

edit

: Sv

etla

na L

azeb

nik

26-Oct-1743


MoreResults

Slid

e cr

edit

: Sv

etla

na L

azeb

nik

26-Oct-1744


MoreResults

26-Oct-1745


• Needtoshiftmanywindows…• Manycomputationswillberedundant.

Problem:ComputationalComplexity

Slid

e cr

edit

: Ba

stia

n Le

ibe

26-Oct-1746


Speedups:BasinofAttraction

r

Slid

e cr

edit

: Ba

stia

n Le

ibe

26-Oct-1747

1. Assignallpointswithinradiusrofendpointtothemode.


Speedups

r =c

Slid

e cr

edit

: Ba

stia

n Le

ibe

26-Oct-1748

2. Assignallpointswithinradiusr/cofthesearchpathtothemode->reducethenumberofdatapointstosearch.


TechnicalDetails

26-Oct-1749

Comaniciu &Meer,2002


TechnicalDetails

26-Oct-1750


• Term1:thisisproportionaltothedensityestimateatx(similartoequation1fromthepreviousslide).

• Term2:thisisthemean-shiftvectorthatpointstowardsthedirectionofmaximumdensity.


TechnicalDetails

26-Oct-1751


Finally,themeanshiftprocedurefromagivenpointxt is:1. Computerthemeanshirtvectorm:

2. Translatethedensitywindow:

3. Iteratesteps1and2untilconvergence.


SummaryMean-Shift• Pros

– General,application-independenttool– Model-free,doesnotassumeanypriorshape(spherical,

elliptical,etc.)ondataclusters– Justasingleparameter(windowsizeh)

• hhasaphysicalmeaning(unlikek-means)– Findsvariablenumberofmodes– Robusttooutliers

• Cons– Outputdependsonwindowsize– Windowsize(bandwidth)selectionisnottrivial– Computationally(relatively)expensive(~2s/image)– Doesnotscalewellwithdimensionoffeaturespace

Slid

e cr

edit

: Sv

etla

na L

azeb

nik

26-Oct-1752


Whatwillwehavelearnedtoday


26-Oct-1753


Date post:	14-May-2018
Category:	Documents
Upload:	vonhi
View:	227 times
Download:	1 times

Lecture: k-means & mean-shift clustering - Stanford...

Documents