Performance Comparison of K-Means and Expectation Maximization with Gaussian Mixture Models for Clustering

EE6540 Final ProjectDevin Cornell & Sushruth Sastry

Outline

● problem statement● background● experiments● results● conclusions

Problem Statement

● basic clustering● classification through distribution

modeling

Figures from [1]

Background: GMM

● equations from [2]

Background: EM

● equations, algorithm from [3]

Background: EM-GMMAlgorithm 2 reprint from [4]

Background: k-means

● special case of EM-GMM [2] with○ no component cluster covariances○ fixed priori of all covariances for K components○ no membership weights, each point just

belongs to the class with the nearest mean

k-means Algorithm● based on modifications to algorithm mentioned by [2]

Experiment 1: Separate GMM Data

Experiment 2: Intermixed GMM Data

Experiment 3: Concentric Gaussian with Large Covariance Differences

Experiment 4: Radial Poisson Distributions with Different Means

Demonstration

● see Matlab

Experiment 1: Results

Experiment 2: Results

Experiment 3: Results

Experiment 4a: Results

Experiment 4b: Results

Experiment 4c: Results

Experiment 4d: Results

Results Summary

Conclusions● EM-GMM is much slower than k-means● EM-GMM was more accurate for all experiments

performed here● These algorithms can be more flexible if run with

different values of K● With a way to map “fitted distributions” to

“generating distributions”, GMM can estimate arbitrary distributions with fewer fitted distributions

References[1] A. W. Moore, “Clustering with Gaussian Mixtures,”, School of Computer Science, Carnegie Mellon University, http://cs.cmu.edu/awm

[2] D. K. P. Murphy, Machine learning: a probabilistic perspective. MIT press, 2012.

[3] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the em algorithm,” Journal of the royal statistical society. Series B (methodological), pp. 1-38, 1977.

[4] Barber, Bayesian reasoning and machine learning. Cambridge University Press, 2012.