Machine Learning 2 - Northeastern UniversityPioter Piotr Substitution (i for e) Insertion (o)...

Machine Learning 2DS 4420 - Spring 2018

From clustering to EMByron C. Wallace

Clustering

Four Types of Clustering1. Centroid-based (K-means, K-medoids)

Notion of Clusters: Voronoi tesselation

Four Types of Clustering2. Density-based (DBSCAN, OPTICS)

Notion of Clusters: Connected regions of high density

Four Types of Clustering3. Connectivity-based (Hierarchical)

Notion of Clusters: Cut off dendrogram at some depth

Four Types of Clustering4. Distribution-based (Mixture Models)

Notion of Clusters: Distributions on features

Hierarchical Clustering

Dendrogram

Root

Internal Branch

Terminal Branch

LeafInternal Node

Root

Internal Branch

Terminal Branch

LeafInternal Node

Similarity of A and B is represented as height of lowest shared internal node

(a.k.a. a similarity tree)

Dendrogram

Similarity of A and B is represented as height of lowest shared internal node


(Bovine: 0.69395, (Spider Monkey: 0.390, (Gibbon:0.36079,(Orang: 0.33636, (Gorilla: 0.17147, (Chimp: 0.19268, Human: 0.11927): 0.08386): 0.06124): 0.15057): 0.54939);

D(A,B)

Dendrogram

Natural when measuring genetic similarity, distance to common ancestor


(Bovine: 0.69395, (Spider Monkey: 0.390, (Gibbon:0.36079,(Orang: 0.33636, (Gorilla: 0.17147, (Chimp: 0.19268, Human: 0.11927): 0.08386): 0.06124): 0.15057): 0.54939);

D(A,B)

Example: Iris data

https://en.wikipedia.org/wiki/Iris_flower_data_set

Iris Setosa

Iris versicolor

Iris virginica


Hierarchical Clustering


(Euclidian Distance)


Edit Distance Change dress color, 1 point Change earring shape, 1 point Change hair part, 1 point

D(Patty, Selma) = 3

Change dress color, 1 point Add earrings, 1 point Decrease height, 1 point Take up smoking, 1 point Lose weight, 1 point

D(Marge,Selma) = 5

Distance Patty and Selma

Distance Marge and Selma

Can be defined for any set of discrete features

Edit Distance for Strings

Peter

Piter

Pioter

Piotr

Substitution (i for e)

Insertion (o)

Deletion (e)

• Transform string Q into string C, using only Substitution, Insertion and Deletion.

• Assume that each of these operators has a cost associated with it.

• The similarity between two strings can be defined as the cost of the cheapest transformation from Q to C.

Similarity “Peter” and “Piotr”?

Substitution 1 UnitInsertion 1 UnitDeletion 1 Unit

D(Peter,Piotr) is 3

Piot

r

Pyot

r

Petro

s

Piet

ro

Pedr

o Pi

erre

Pier

o

Pete

r

Hierarchical Clustering(Edit Distance)

Pio

tr P

yotr

Pet

ros

Pie

troPedro

Pie

rre P

iero

Pet

erPe

der

Pek

a P

eada

rM

ichal

isM

ichae

lMiguel

Mick

Cristovao

Chris

toph

erCh

risto

phe

Chris

toph

Crisd

ean

Crist

obal

Crist

ofor

oKr

istof

fer

Krys

tof

Pedro (Portuguese)Petros (Greek), Peter (English), Piotr (Polish), Peadar (Irish), Pierre (French), Peder (Danish), Peka (Hawaiian), Pietro (Italian), Piero (Italian Alternative), Petr (Czech), Pyotr (Russian)

Cristovao (Portuguese)Christoph (German), Christophe (French), Cristobal (Spanish), Cristoforo (Italian), Kristoffer(Scandinavian), Krystof (Czech), Christopher (English)

Miguel (Portuguese)Michalis (Greek), Michael (English), Mick (Irish)

A Demonstration of Hierarchical Clustering using String Edit Distance Slide based on one by Eamonn Keogh

Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms

Meaningful Patterns

Pedro(Portuguese/Spanish)Petros (Greek), Peter (English), Piotr(Polish), Peadar (Irish), Pierre (French), Peder (Danish), Peka (Hawaiian), Pietro(Italian), Piero (Italian Alternative), Petr(Czech), Pyotr (Russian)

Slide from Eamonn Keogh


Edit distance yields clustering according to geography

Spurious Patterns

18

ANGUILLAAUSTRALIA St. Helena &Dependencies

South Georgia &South Sandwich Islands

U.K.Serbia & Montenegro(Yugoslavia)

FRANCE NIGER INDIA IRELAND BRAZIL

Hierarchal clustering can sometimes show patterns that are meaningless or spurious

The tight grouping of Australia, Anguilla, St. Helena etc is meaningful; all these countries are former UK colonies

However the tight grouping of Niger and India is completely spurious; there is no connection between the two

Slide based on one by Eamonn Keogh


In general clusterings will only be as meaningful as your distance metric

Spurious Patterns

18

ANGUILLAAUSTRALIA St. Helena &Dependencies

South Georgia &South Sandwich Islands

U.K.Serbia & Montenegro(Yugoslavia)

FRANCE NIGER INDIA IRELAND BRAZIL

Hierarchal clustering can sometimes show patterns that are meaningless or spurious

The tight grouping of Australia, Anguilla, St. Helena etc is meaningful; all these countries are former UK colonies

However the tight grouping of Niger and India is completely spurious; there is no connection between the two



In general clusterings will only be as meaningful as your distance metric

Former UK colonies No relation

“Correct” Number of Clusters

19

We can look at the dendrogram to determine the “correct” number of clusters.



“Correct” Number of Clusters

19

We can look at the dendrogram to determine the “correct” number of clusters.


Yijun Zhao DATA MINING TECHNIQUES Clustering AlgorithmsDetermine number of clusters by looking at distance

Detecting Outliers

20

Outlier

One potential use of a dendrogram: detecting outliersThe single isolated branch is suggestive of a data point that is very different to all others



Bottom up vs. Top downBottom-up (agglomerative): Each item starts as its own cluster; greedily merge

Bottom up vs. Top downBottom-up (agglomerative): Each item starts as its own cluster; greedily merge

Top-down (divisive): Start with one big cluster (all data); recursively split

Distance Matrix

22

0 8 8 7 7

0 2 4 4

0 3 3

0 1

0

D( , ) = 8D( , ) = 1

We begin with a distance matrix which contains the distances between every pair of objects in our database.



Bottom-up (Agglomerative Clustering)

25

…Consider all possible merges…

Choose the best

Consider all possible merges… …

Choose the best

Consider all possible merges…

Choose the best…

Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.



25


Choose the best


Choose the best


Choose the best…




25


Choose the best


Choose the best


Choose the best…




25


Choose the best


Choose the best


Choose the best…


Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms 26


Choose the best


Choose the best


Choose the best…




25


Choose the best


Choose the best


Choose the best…




Choose the best


Choose the best


Choose the best…



Can you now implement this?


25


Choose the best


Choose the best


Choose the best…




Choose the best


Choose the best


Choose the best…



Distances between examples (can calculate using metric)


25


Choose the best


Choose the best


Choose the best…




Choose the best


Choose the best


Choose the best…



How do we calculate the distance to a cluster?

Clustering CriteriaSingle link:

(Closest point)d(A, B) = min

a2A,b2Bd(a, b)

Complete link: (Furthest point)

d(A, B) = maxa2A,b2B

d(a, b)

Group average: (Average distance)

d(A, B) =1|A||B|X

a2A,b2B

d(a, b)

Centroid: (Distance of average)

d(A, B) = d(µA,µB) µX =1|X |X

x2X

x

Hierarchical Clustering Summary

+ No need to specify number of clusters+ Hierarchical structure maps nicely onto

human intuition in some domains - Scaling: Time complexity at least O(n2)

in number of examples - Heuristic search method:

Local optima are a problem - Interpretation of results is (very) subjective





















Evaluation?

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 83

Cluster Validity

O For supervised classification we have a variety of measures to evaluate how good our model is

– Accuracy, precision, recall

O For cluster analysis, the analogous question is how to evaluate the “goodness” of the resulting clusters?

O But “clusters are in the eye of the beholder”!

O Then why do we want to evaluate them?– To avoid finding patterns in noise– To compare clustering algorithms– To compare two sets of clusters– To compare two clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 84

Clusters found in Random Data

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x

y

Random Points

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x

y

K-means

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x

y

DBSCAN

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x

yComplete Link

Clustering CriteriaInternal Quality Criteria Measure compactness of clusters • Sum of Squared Error (SSE) • Scatter Criteria

External Quality Criteria• Precision-Recall Measure • Mutual Information

Clustering CriteriaInternal Quality Criteria Measure compactness of clusters • Sum of Squared Error (SSE) • Scatter Criteria

External Quality Criteria• Precision-Recall Measure • Mutual Information

From K-means to Mixture Models

From K-means to Mixture ModelsLet’s come back to K-means for a momentK-means Algorithm

Input: X = {x1, x2, . . . , xN}Number of clusters K

Initialize: K random centroids µ1, µ2, . . . , µK

Repeat Until Convergence

1 For i = 1, . . . ,K do

Ci = {x 2 X |i = arg min1jK

k x� µj k2}2 For i = 1, . . . ,K do

µi = argminz

Px2Ci

k z� x k2}

Output: C1,C2, . . . ,CK


A probabilistic view• K-means feels a bit heuristic

• What if we instead took a probabilistic view of clustering?

• Mixture models define a “generative story” for the data observed

Some slides derived from Matt Gormley and Eric Xing (CMU)

K-Means vs Gaussian Mixture Models

μ1

μ2

μ3

Idea: Learn both means μk and covariances Σk

μ3 Σ3 μ2 Σ2

μ1 Σ1

Don’t just learn where the center of the cluster is, but also how big it is, and what shape it has.

μ1

μ2

μ3

�nk = I[zn = k]<latexit sha1_base64="93Lo9JLGfOSZ3crmU0IM1QPo76Q=">AAAF7HicfZRPb9MwFMC9scIo/za4IHGp6IVDNSVj2nZBqoYm4Dam/ZOaqnKc19Sq4wTb2dZa5lNwQ9wQN/gEfBG+DU4biSaOcKTkye/3/vm9OMwYlcrz/qyt39lo3b23eb/94OGjx0+2tp9eyDQXBM5JylJxFWIJjHI4V1QxuMoE4CRkcBlO3xb6y2sQkqb8TM0yGCY45nRMCVZ2a7T1PIhxkuCR5lPTedP5MJiPuP1Oh6OtrrfjLVbHFfxS6PbRcp2Mtjd+B1FK8gS4IgxLOfC9TA01FooSBqYd5BIyTKY4hoEVOU5ADvWiBNOpaM/8oR6nXAEnFTONE5lgNXE2C1hWd8nEBgZRDVtuDnXhJQJJY161ChPTbgcRjO1xLjLTUchyMPr03ZHRXm//dc/fPTA1REBUEv6h17NPHYgFAC+Rw72ev3/oMlkuMgb/IK/AimwEcLghqe0Sj3RwDcQM7PkEwGUuoChEB2Giu74xxoGXqLVZ6NvBqvLWaB1UEgyu9a2pY7MVrCjUQjMHmjf5mjvYpyYsUBNQuCF71UzjvIHNHTZ3IeFAop4hNMaETFKWcqee8Qq9mJOxG5StMGWPC5fM/qMRdjxmk2Y8m1CHPa115tQU47JKYBEn2PY5SDMQWKWi+OluqJowmlAldak3rhXl/7ey+nqwY1MdyuIdhvrYOCQJ2WIwq2fnTigRUZUrqmzAYlHFlo1rALMaWB5wQdr7zq/fbq5wsbvjW/njXrd/VN58m+gFeoleIR8doD56j07QOSLoM/qOfqJfLd760vra+rZE19dKm2eoslo//gJWbira</latexit><latexit sha1_base64="93Lo9JLGfOSZ3crmU0IM1QPo76Q=">AAAF7HicfZRPb9MwFMC9scIo/za4IHGp6IVDNSVj2nZBqoYm4Dam/ZOaqnKc19Sq4wTb2dZa5lNwQ9wQN/gEfBG+DU4biSaOcKTkye/3/vm9OMwYlcrz/qyt39lo3b23eb/94OGjx0+2tp9eyDQXBM5JylJxFWIJjHI4V1QxuMoE4CRkcBlO3xb6y2sQkqb8TM0yGCY45nRMCVZ2a7T1PIhxkuCR5lPTedP5MJiPuP1Oh6OtrrfjLVbHFfxS6PbRcp2Mtjd+B1FK8gS4IgxLOfC9TA01FooSBqYd5BIyTKY4hoEVOU5ADvWiBNOpaM/8oR6nXAEnFTONE5lgNXE2C1hWd8nEBgZRDVtuDnXhJQJJY161ChPTbgcRjO1xLjLTUchyMPr03ZHRXm//dc/fPTA1REBUEv6h17NPHYgFAC+Rw72ev3/oMlkuMgb/IK/AimwEcLghqe0Sj3RwDcQM7PkEwGUuoChEB2Giu74xxoGXqLVZ6NvBqvLWaB1UEgyu9a2pY7MVrCjUQjMHmjf5mjvYpyYsUBNQuCF71UzjvIHNHTZ3IeFAop4hNMaETFKWcqee8Qq9mJOxG5StMGWPC5fM/qMRdjxmk2Y8m1CHPa115tQU47JKYBEn2PY5SDMQWKWi+OluqJowmlAldak3rhXl/7ey+nqwY1MdyuIdhvrYOCQJ2WIwq2fnTigRUZUrqmzAYlHFlo1rALMaWB5wQdr7zq/fbq5wsbvjW/njXrd/VN58m+gFeoleIR8doD56j07QOSLoM/qOfqJfLd760vra+rZE19dKm2eoslo//gJWbira</latexit><latexit sha1_base64="93Lo9JLGfOSZ3crmU0IM1QPo76Q=">AAAF7HicfZRPb9MwFMC9scIo/za4IHGp6IVDNSVj2nZBqoYm4Dam/ZOaqnKc19Sq4wTb2dZa5lNwQ9wQN/gEfBG+DU4biSaOcKTkye/3/vm9OMwYlcrz/qyt39lo3b23eb/94OGjx0+2tp9eyDQXBM5JylJxFWIJjHI4V1QxuMoE4CRkcBlO3xb6y2sQkqb8TM0yGCY45nRMCVZ2a7T1PIhxkuCR5lPTedP5MJiPuP1Oh6OtrrfjLVbHFfxS6PbRcp2Mtjd+B1FK8gS4IgxLOfC9TA01FooSBqYd5BIyTKY4hoEVOU5ADvWiBNOpaM/8oR6nXAEnFTONE5lgNXE2C1hWd8nEBgZRDVtuDnXhJQJJY161ChPTbgcRjO1xLjLTUchyMPr03ZHRXm//dc/fPTA1REBUEv6h17NPHYgFAC+Rw72ev3/oMlkuMgb/IK/AimwEcLghqe0Sj3RwDcQM7PkEwGUuoChEB2Giu74xxoGXqLVZ6NvBqvLWaB1UEgyu9a2pY7MVrCjUQjMHmjf5mjvYpyYsUBNQuCF71UzjvIHNHTZ3IeFAop4hNMaETFKWcqee8Qq9mJOxG5StMGWPC5fM/qMRdjxmk2Y8m1CHPa115tQU47JKYBEn2PY5SDMQWKWi+OluqJowmlAldak3rhXl/7ey+nqwY1MdyuIdhvrYOCQJ2WIwq2fnTigRUZUrqmzAYlHFlo1rALMaWB5wQdr7zq/fbq5wsbvjW/njXrd/VN58m+gFeoleIR8doD56j07QOSLoM/qOfqJfLd760vra+rZE19dKm2eoslo//gJWbira</latexit><latexit sha1_base64="NcJZo1QTOBeujZmLoiTOXO9RCk0=">AAAF7HicfZRLb9QwEIDd0oWyvFq4IHFZsRcOqyopVdsLUlVUAbdS9SVtVivHmc1a6zjBdvqyzK/ghrghbvAL+CP8G+xtJDZxhCMlI883L8/EccGoVEHwZ2n5zkrn7r3V+90HDx89frK2/vRU5qUgcEJylovzGEtglMOJoorBeSEAZzGDs3j21unPLkBImvNjdV3AKMMppxNKsLJb47XnUYqzDI81n5nem96H4c2Y2+9sNF7rBxvBfPV8IayEPqrW4Xh95XeU5KTMgCvCsJTDMCjUSGOhKGFgulEpocBkhlMYWpHjDORIz0swvZr2OBzpSc4VcFIz0ziTGVZTb9PBsr5LpjYwiHrYanOknZcEJE153SrOTLcbJTCxxznPTCcxK8Hoo3f7RgeD7deDcHPHNBABSUWEu8HAPk0gFQC8Qna3BuH2rs8UpSgY/IMCh7lsBHC4JLntEk90dAHEDO35RMBlKcAVoqM40/3QGOPBt6i1meu70aLyymgd1RKMLvSVaWLXC5gr1ELXHnTT5uvGwz61YZGagsIt2at2GpctbOmxpQ8JDxLNDKE1JhSSspx79UwW6PmcTPygbIGpeuxcMvuPJtjzWEzb8WJKPfao0Zkj48ZlkcAizbDtc5QXILDKhfvpLqmaMppRJXWlN74V5f+3svpmsANTH0r3jmN9YDySxGw+mPWz8yeUiKTOuSpbsFTUsdvGtYBFA6wO2JH2vgubt5svnG5uhFb+uNXf269uvlX0Ar1Er1CIdtAeeo8O0Qki6DP6jn6iXx3e+dL52vl2iy4vVTbPUG11fvwFBJ8qmg==</latexit>

Hard assignments to clusters

(one-hot vector)

Idea: Replace hard assignments with soft assignments

Soft assignments to clusters

(posterior probability)�nk = p(zn=k | xn)

<latexit sha1_base64="BJ/xva1AWL+z+wmBUMTzPebmzMI=">AAAF/XicfZRLb9QwEIBd6EJZXi0cuaTspZVWVVKqtpdKVVEFx1L1JdWrlePM7lrrOMFx+rIsfgK/ghvihrgg+B38G5xsJDZxhCNFI883L8/YYcpZpnz/z8K9+4udBw+XHnUfP3n67PnyyouzLMklhVOa8ERehCQDzgScKqY4XKQSSBxyOA+nbwv9+RXIjCXiRN2mMIjJWLARo0TZreHyOh6TOCZDLabG2/PStbuh8PDqHl71ph6OWeThK31jhmJ9uNzzN/xyea4QVEJvH83W0XBl8SeOEprHIBTlJMsuAz9VA02kYpSD6eI8g5TQKRnDpRUFiSEb6LIm49W0J8FAjxKhQNCamSZxFhM1cTYLOKvv0okNDLIettoc6MJLBBkbi7pVGJtuF0cwsudbZqajkOdg9PG7A6P9/vabfrC5YxqIhKgigl2/b78mMJYAokJ2t/rB9q7LpLlMOfyD/AIrspEg4Jomtm0i0vgKqLm054NBZLmEohCNw1j3AmOMA89Qa1Pqu3heeWO0xrUEy943sds5rCjUQrcOdNfm687BPrZhWE1AkZbsVTtN8hY2d9jchaQDyWaG0BoT0ozxRDj1jObock5GblA+x1Q9Llxye2kj4nhMJ+14OmEOe9zozLEpxmWeIHIcE9tnnKQgiUpkcemumZpwFjOV6UpvXCsm/m9l9c1gh6Y+lMU/DPWhcUga8nIw62fnTiiVUZ0rqmzBxrKOzRrXAqYNsDrggrTvXdB83VzhbHMjsPKHrd7+QfXyLaFX6DVaQwHaQfvoPTpCp4iiz+g7+oV+dz51vnS+dr7N0HsLlc1LVFudH38BUqsxIw==</latexit><latexit sha1_base64="BJ/xva1AWL+z+wmBUMTzPebmzMI=">AAAF/XicfZRLb9QwEIBd6EJZXi0cuaTspZVWVVKqtpdKVVEFx1L1JdWrlePM7lrrOMFx+rIsfgK/ghvihrgg+B38G5xsJDZxhCNFI883L8/YYcpZpnz/z8K9+4udBw+XHnUfP3n67PnyyouzLMklhVOa8ERehCQDzgScKqY4XKQSSBxyOA+nbwv9+RXIjCXiRN2mMIjJWLARo0TZreHyOh6TOCZDLabG2/PStbuh8PDqHl71ph6OWeThK31jhmJ9uNzzN/xyea4QVEJvH83W0XBl8SeOEprHIBTlJMsuAz9VA02kYpSD6eI8g5TQKRnDpRUFiSEb6LIm49W0J8FAjxKhQNCamSZxFhM1cTYLOKvv0okNDLIettoc6MJLBBkbi7pVGJtuF0cwsudbZqajkOdg9PG7A6P9/vabfrC5YxqIhKgigl2/b78mMJYAokJ2t/rB9q7LpLlMOfyD/AIrspEg4Jomtm0i0vgKqLm054NBZLmEohCNw1j3AmOMA89Qa1Pqu3heeWO0xrUEy943sds5rCjUQrcOdNfm687BPrZhWE1AkZbsVTtN8hY2d9jchaQDyWaG0BoT0ozxRDj1jObock5GblA+x1Q9Llxye2kj4nhMJ+14OmEOe9zozLEpxmWeIHIcE9tnnKQgiUpkcemumZpwFjOV6UpvXCsm/m9l9c1gh6Y+lMU/DPWhcUga8nIw62fnTiiVUZ0rqmzBxrKOzRrXAqYNsDrggrTvXdB83VzhbHMjsPKHrd7+QfXyLaFX6DVaQwHaQfvoPTpCp4iiz+g7+oV+dz51vnS+dr7N0HsLlc1LVFudH38BUqsxIw==</latexit><latexit sha1_base64="BJ/xva1AWL+z+wmBUMTzPebmzMI=">AAAF/XicfZRLb9QwEIBd6EJZXi0cuaTspZVWVVKqtpdKVVEFx1L1JdWrlePM7lrrOMFx+rIsfgK/ghvihrgg+B38G5xsJDZxhCNFI883L8/YYcpZpnz/z8K9+4udBw+XHnUfP3n67PnyyouzLMklhVOa8ERehCQDzgScKqY4XKQSSBxyOA+nbwv9+RXIjCXiRN2mMIjJWLARo0TZreHyOh6TOCZDLabG2/PStbuh8PDqHl71ph6OWeThK31jhmJ9uNzzN/xyea4QVEJvH83W0XBl8SeOEprHIBTlJMsuAz9VA02kYpSD6eI8g5TQKRnDpRUFiSEb6LIm49W0J8FAjxKhQNCamSZxFhM1cTYLOKvv0okNDLIettoc6MJLBBkbi7pVGJtuF0cwsudbZqajkOdg9PG7A6P9/vabfrC5YxqIhKgigl2/b78mMJYAokJ2t/rB9q7LpLlMOfyD/AIrspEg4Jomtm0i0vgKqLm054NBZLmEohCNw1j3AmOMA89Qa1Pqu3heeWO0xrUEy943sds5rCjUQrcOdNfm687BPrZhWE1AkZbsVTtN8hY2d9jchaQDyWaG0BoT0ozxRDj1jObock5GblA+x1Q9Llxye2kj4nhMJ+14OmEOe9zozLEpxmWeIHIcE9tnnKQgiUpkcemumZpwFjOV6UpvXCsm/m9l9c1gh6Y+lMU/DPWhcUga8nIw62fnTiiVUZ0rqmzBxrKOzRrXAqYNsDrggrTvXdB83VzhbHMjsPKHrd7+QfXyLaFX6DVaQwHaQfvoPTpCp4iiz+g7+oV+dz51vnS+dr7N0HsLlc1LVFudH38BUqsxIw==</latexit><latexit sha1_base64="3RCsbNdr5mqlxZKhKDsXVBb7jcw=">AAAF/XicfZRLa9wwEICVNNum21fSHntxupcElmAnIcklEFJCe0xDXhAtiyzP7oqVZVeS8xKiP6G/orfSW+mltL+j/6a2Y+jaMpXBDJpvXpqRwpQzpX3/z9z8g4XOw0eLj7tPnj57/mJp+eWZSjJJ4ZQmPJEXIVHAmYBTzTSHi1QCiUMO5+H0baE/vwKpWCJO9G0Kg5iMBRsxSnS+NVxaw2MSx2RoxNR6e166ejcUHl7Zwyve1MMxizx8ZW7sUKwNl3r+ul8uzxWCSuihah0Nlxd+4iihWQxCU06Uugz8VA8MkZpRDraLMwUpoVMyhstcFCQGNTBlTdaraU+CgRklQoOgNTNDYhUTPXE2C1jVd+kkDwyyHrbaHJjCSwSKjUXdKoxtt4sjGOXnW2ZmopBnYM3xuwNr/P72Zj/Y2LENREJUEcGu38+/JjCWAKJCdrf6wfauy6SZTDn8g/wCK7KRIOCaJnnbRGTwFVB7mZ8PBqEyCUUhBoex6QXWWge+R3ObUt/Fs8obawyuJVj2vondzmBFoTl060B3bb7uHOxjG4b1BDRpyV630yRrYTOHzVxIOpBsZgitMSFVjCfCqWc0Q5dzMnKD8hmm6nHhkueXNiKOx3TSjqcT5rDHjc4c22JcZgkixzHJ+4yTFCTRiSwu3TXTE85ippWp9Na1YuL/Vrm+GezQ1oey+IehObQOSUNeDmb97NwJpTKqc0WVLdhY1rH7xrWAaQOsDrgg8/cuaL5urnC2sR7k8oet3v5B9fItotfoDVpFAdpB++g9OkKniKLP6Dv6hX53PnW+dL52vt2j83OVzStUW50ffwEA3DDj</latexit>

μ3 Σ3 μ2 Σ2

μ1 Σ1

K-Means vs Gaussian Mixture Models

Mixture models

Gaussian Mixture Models

μ3 Σ3 μ2 Σ2

μ1 Σ1

Idea 1: Points in each cluster are sampled for a Gaussian

Idea 2: Compute probability that point belongs to each cluster

zn ⇠ Discrete(⇡1, . . . ,⇡K)

xn | zn=k ⇠ Norm(µk,⌃k)<latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="3cbWSxdmV0r9eqXo0fV9T0T9Hos=">AAAFxHicfZRPb9MwFMC9scIIA7Yzl4leOFRTMtDWIwJNcBxl3SY11eQ4r601xwm2062zLCGOXPk6fBG+DXYaiSaOcKToye/3/P74+SUFo1KF4Z+t7Uc7vcdPdp8Gz/aC5y9e7u9dyrwUBMYkZ7m4TrAERjmMFVUMrgsBOEsYXCW3H53+aglC0pxfqFUB0wzPOZ1RgpXdOr/Z74dHYbUOfSGqhT6q183Bzu84zUmZAVeEYSknUVioqcZCUcLABHEpocDkFs9hYkWOM5BTXcVpDhvai2iqZzlXwEnDTONMZlgtvE0Hy+YuWVjHIJpu682pdqekIOmcN62SzARBnMLM1qyKTKcJK8Ho0acPRoeDk7eD6PjUtBABaU1Ew3BgvzYwFwC8RobvBtHJ0GeKUhQM/kGhw1w0AjjckTzLME91vARiJrY+MXBZCnCJ6DjJdD8yxnjwGrU2lT6IN5X3Ruu4EWC81Pemja02MJeohVYe9NB11oOHfevCYrUAhTuiV900LjvY0mNLHxIeJNoRQqdPKCRlOffymW3QVZ/MfKdsg6nv2B3J7ENMsXdisejGiwX12FHrZkbGtcsmgcU8w/ae47wAgVUu3KO7o2rBaEaV1LXe+FaU/9/K6tvOzkyzKd0/SfSZ8UiSsKoxm7XzO5SItMm5LDuwuWhi64vrAIsWWBfYkXbcRe3h5guXx0eRlb+EaBe9Qq/RGxShU/QefUbnaIwIStFP9Kv3tbfqfV+Pxe2tej4eoMbq/fgLSA0fDA==</latexit><latexit sha1_base64="AoyELedfKaAu50cE2ilfZ8Dwl30=">AAAGN3icfZRPb9MwFMDTbYURBmxcuXRMQptUTclAWy9ICJhAQkJj7J9UV5XjvLZWYyfYztbN8jfjyEfgwlfghrhxwuki0cQRjpS8+P3eH/s9O8oSKlUQ/GgtLa+079xdveffX3vw8NH6xtqZTHNB4JSkSSouIiwhoRxOFVUJXGQCMIsSOI+mbwr9+SUISVN+oq4zGDA85nRECVZ2arjOb4a88wxJyjpIwUzpt1QSAQrMNsroMOx2UBKnStqv/f2w4yPU72Vq4M+sHWI07lgHaPMl2pxW3HxMBbMuWD6cWtPPdMzwcLrjD9e3gt1gPjquEJbClleOo+HGylcUpyRnwBVJsJT9MLDhNRaKkgSMj3IJGSZTPIa+FTlmIAd6vjGmU9GehAM9SrkCTipmGjPJsJo4kwUsq7NkYgODqIYtJwe68BKDpGNetYqY8X0Uw8gWaZ6ZjqMkB6OP3702OujuP++GewemhgiISyLsBV371IGxAOAl0nvRDfd7LpPlIkvgHxQUWJGNAA5XJGUM81ijSyCmb/cHAZe5gGIhGkVMb4XGGAe+Ra3NXO+jReXMaI0qCaJLPTN17HoBKxZqoWsHumnydeNgX5owpCagcEP2qpnGeQObO2zuQsKBRD1DaIwJmaRJyp31jBboeZ+M3KDJAlPWuHCZ2JMfY8djNmnGswl12ONaZY5N0S6LBBb2NNs6ozQDgVUqikN3RdUkoYwqqUu9ca0o/7+V1deDHZpqUxbvKNKHxiFJlMwbs7p3bocSEVe5YpUN2FhUsdvCNYBZDSw3uCDtfRfWbzdXONvbDa38KfBWvSfeU2/bC70D75X33jvyTj3ifff+tJZbK+1v7Z/tX7c341KrvCIfe5XR/v0XBSNH4g==</latexit><latexit sha1_base64="AoyELedfKaAu50cE2ilfZ8Dwl30=">AAAGN3icfZRPb9MwFMDTbYURBmxcuXRMQptUTclAWy9ICJhAQkJj7J9UV5XjvLZWYyfYztbN8jfjyEfgwlfghrhxwuki0cQRjpS8+P3eH/s9O8oSKlUQ/GgtLa+079xdveffX3vw8NH6xtqZTHNB4JSkSSouIiwhoRxOFVUJXGQCMIsSOI+mbwr9+SUISVN+oq4zGDA85nRECVZ2arjOb4a88wxJyjpIwUzpt1QSAQrMNsroMOx2UBKnStqv/f2w4yPU72Vq4M+sHWI07lgHaPMl2pxW3HxMBbMuWD6cWtPPdMzwcLrjD9e3gt1gPjquEJbClleOo+HGylcUpyRnwBVJsJT9MLDhNRaKkgSMj3IJGSZTPIa+FTlmIAd6vjGmU9GehAM9SrkCTipmGjPJsJo4kwUsq7NkYgODqIYtJwe68BKDpGNetYqY8X0Uw8gWaZ6ZjqMkB6OP3702OujuP++GewemhgiISyLsBV371IGxAOAl0nvRDfd7LpPlIkvgHxQUWJGNAA5XJGUM81ijSyCmb/cHAZe5gGIhGkVMb4XGGAe+Ra3NXO+jReXMaI0qCaJLPTN17HoBKxZqoWsHumnydeNgX5owpCagcEP2qpnGeQObO2zuQsKBRD1DaIwJmaRJyp31jBboeZ+M3KDJAlPWuHCZ2JMfY8djNmnGswl12ONaZY5N0S6LBBb2NNs6ozQDgVUqikN3RdUkoYwqqUu9ca0o/7+V1deDHZpqUxbvKNKHxiFJlMwbs7p3bocSEVe5YpUN2FhUsdvCNYBZDSw3uCDtfRfWbzdXONvbDa38KfBWvSfeU2/bC70D75X33jvyTj3ifff+tJZbK+1v7Z/tX7c341KrvCIfe5XR/v0XBSNH4g==</latexit><latexit sha1_base64="gmSp9yGGdLqLCSOOcgzNoK+siuI=">AAAGQnicfZTNb9MwFMBTtsIIXxscuXRUQptUTclAWy9I02ACCQmNsS+priIneW2txk6wna2b5f+MI38CF/4Fbogb4oDTRaKJIxwpefH7vQ/7PTvMEiKk531v3Vpabt++s3LXvXf/wcNHq2uPT0Wa8whOojRJ+XmIBSSEwYkkMoHzjAOmYQJn4fR1oT+7AC5Iyo7lVQZDiseMjEiEpZkKVtl1wDrPkSC0gyTMpHpDRMRBgt5AGQn8XgclcSqF+Zrf95suQoN+JofuzNghSuKOcYDWX6H1acXNh5RT44LmwdSYfiJjioPpphusdr0tbz46tuCXQtcpx2GwtvwFxWmUU2AySrAQA98z4RXmkkQJaBflAjIcTfEYBkZkmIIYqvnG6E5Fe+wP1ShlElhUMVOYCorlxJosYFGdjSYmMPBq2HJyqAovMQgyZlWrkGrXRTGMTJHmmak4THLQ6ujtvlZeb+dFz9/e1TWEQ1wSft/rmacOjDkAK5H+y56/07eZLOdZAv8gr8CKbDgwuIxSSjGLFbqASA/M/iBgIudQLEShkKqur7W24BvU2Mz1LlpUzrRSqJIgulAzXceuFrBioQa6sqDrJl/XFva5CUNyAhI3ZC+baZw3sLnF5jbELYjXM4TGmJAJkqTMWs9ogZ73ycgOmiwwZY0Ll4k5+TG2PGaTZjybEIs9qlXmSBftskhgbk6zqTNKM+BYprw4dJdEThJCiRSq1GvbirD/Wxl9PdiBrjZl8Q5DdaAtMgqTeWNW987u0IjHVa5YZQM25lXspnANYFYDyw0uSHPf+fXbzRZOt7d8I3/0unv75c234jx1njkbju/sOnvOO+fQOXEi55vzu7XUWm5/bf9o/2z/ukFvtUqbJ05ltP/8BUlrSY8=</latexit><latexit sha1_base64="iq7hPoss7PQskHtd23AE9/yVMFM=">AAAGQnicfZTNb9MwFMDTbYURvjo4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOF4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0ePW2tPzkSS8RBOwyRO+EWABcSEwakkMoaLlAOmQQznwfR1rj+/BC5Iwk7kdQoDiseMjEiIpZkattjNkLVfIEFoG0mYSfWGiJCDBL2BUjL0u20UR4kU5mt+32+6CPV7qRy4M2OHKInaxgFaf4XWpyU3HxJOjQuaDafG9BMZUzycbrrDVsfb8uajbQt+IXScYhwN11a+oCgJMwpMhjEWou97JrzCXJIwBu2iTECKwykeQ9+IDFMQAzXfGN0uaU/8gRolTAILS2YKU0GxnFiTOSzKs+HEBAZeDltMDlTuJQJBxqxsFVDtuiiCkSnSPDMVBXEGWh2/PdDK6+6+7Prbe7qCcIgKwu95XfNUgTEHYAXS2+n6uz2bSTOexvAP8nIsz4YDg6swoRSzSKFLCHXf7A8CJjIO+UIUCqjq+FprC75Fjc1c76JF5UwrhUoJoks101XsegHLF2qgawu6qfN1Y2Gf6zAkJyBxTfaynsZZDZtZbGZD3IJ4NUOojQmpIHHCrPWMFuh5n4zsoPECU9Q4dxmbkx9hy2M6qcfTCbHY40pljnXeLosE5uY0mzqjJAWOZcLzQ3dF5CQmlEihCr22rQj7v5XRV4Md6nJT5u8gUIfaIsMgnjdmee/sDg15VObyVdZgY17GbgtXA6YVsNjgnDT3nV+93WzhbHvLN/LHnc7+QXHzrTrPnOfOhuM7e86+8845ck6d0Pnm/G4sN1aaX5s/mj+bv27RpUZh89Qpjeafv0qrSZM=</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="iq7hPoss7PQskHtd23AE9/yVMFM=">AAAGQnicfZTNb9MwFMDTbYURvjo4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOF4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0ePW2tPzkSS8RBOwyRO+EWABcSEwakkMoaLlAOmQQznwfR1rj+/BC5Iwk7kdQoDiseMjEiIpZkattjNkLVfIEFoG0mYSfWGiJCDBL2BUjL0u20UR4kU5mt+32+6CPV7qRy4M2OHKInaxgFaf4XWpyU3HxJOjQuaDafG9BMZUzycbrrDVsfb8uajbQt+IXScYhwN11a+oCgJMwpMhjEWou97JrzCXJIwBu2iTECKwykeQ9+IDFMQAzXfGN0uaU/8gRolTAILS2YKU0GxnFiTOSzKs+HEBAZeDltMDlTuJQJBxqxsFVDtuiiCkSnSPDMVBXEGWh2/PdDK6+6+7Prbe7qCcIgKwu95XfNUgTEHYAXS2+n6uz2bSTOexvAP8nIsz4YDg6swoRSzSKFLCHXf7A8CJjIO+UIUCqjq+FprC75Fjc1c76JF5UwrhUoJoks101XsegHLF2qgawu6qfN1Y2Gf6zAkJyBxTfaynsZZDZtZbGZD3IJ4NUOojQmpIHHCrPWMFuh5n4zsoPECU9Q4dxmbkx9hy2M6qcfTCbHY40pljnXeLosE5uY0mzqjJAWOZcLzQ3dF5CQmlEihCr22rQj7v5XRV4Md6nJT5u8gUIfaIsMgnjdmee/sDg15VObyVdZgY17GbgtXA6YVsNjgnDT3nV+93WzhbHvLN/LHnc7+QXHzrTrPnOfOhuM7e86+8845ck6d0Pnm/G4sN1aaX5s/mj+bv27RpUZh89Qpjeafv0qrSZM=</latexit>

�nk = p(zn=k | xn)<latexit sha1_base64="s6BIwT4drCR6QCX+SZn7AuZMIgA=">AAAF/HicfZRLb9QwEIDd0oWyvFo4ctmylyKtqqRUbS+VqqIKjqXqS6pXK8eZzVrrOMF2+rLMT+BXcEPcECfgf/BvSLKR2MQRjhSNPN+8PGMHKWdKe96fhcV7S537D5Yfdh89fvL02crq8zOVZJLCKU14Ii8CooAzAaeaaQ4XqQQSBxzOg+nbQn9+BVKxRJzo2xSGMYkEGzNKdL41WlnHEYljMjJiant7vXT9biR6eG0Pr017OGZhD1+ZGzsSr0crfW/DK1fPFfxK6O+j2ToarS79xGFCsxiEppwodel7qR4aIjWjHGwXZwpSQqckgstcFCQGNTRlSbZX0574QzNOhAZBa2aGxComeuJsFrCq79JJHhhkPWy1OTSFlxAUi0TdKohtt4tDGOfHW2ZmwoBnYM3xuwNrvMH2m4G/uWMbiISwIvxdb5B/TSCSAKJCdrcG/vauy6SZTDn8g7wCK7KRIOCaJnnXRGjwFVB7mZ8PBqEyCUUhBgex6fvWWgeeoblNqe/ieeWNNQbXEix738Ru57Ci0By6daC7Nl93DvaxDcN6Apq0ZK/baZK1sJnDZi4kHUg2M4TWmJAqxhPh1DOeo8s5GbtB+RxT9bhwyfM7GxLHYzppx9MJc9jjRmeObTEu8wSRUUzyPuMkBUl0IotLd830hLOYaWUqvXWtmPi/Va5vBju09aEs/kFgDq1D0oCXg1k/O3dCqQzrXFFlCxbJOjZrXAuYNsDqgAsyf+/85uvmCmebG34uf9jq7x9UL98yeoleoXXkox20j96jI3SKKPqMvqNf6HfnU+dL52vn2wxdXKhsXqDa6vz4C3FeMPk=</latexit><latexit sha1_base64="s6BIwT4drCR6QCX+SZn7AuZMIgA=">AAAF/HicfZRLb9QwEIDd0oWyvFo4ctmylyKtqqRUbS+VqqIKjqXqS6pXK8eZzVrrOMF2+rLMT+BXcEPcECfgf/BvSLKR2MQRjhSNPN+8PGMHKWdKe96fhcV7S537D5Yfdh89fvL02crq8zOVZJLCKU14Ii8CooAzAaeaaQ4XqQQSBxzOg+nbQn9+BVKxRJzo2xSGMYkEGzNKdL41WlnHEYljMjJiant7vXT9biR6eG0Pr017OGZhD1+ZGzsSr0crfW/DK1fPFfxK6O+j2ToarS79xGFCsxiEppwodel7qR4aIjWjHGwXZwpSQqckgstcFCQGNTRlSbZX0574QzNOhAZBa2aGxComeuJsFrCq79JJHhhkPWy1OTSFlxAUi0TdKohtt4tDGOfHW2ZmwoBnYM3xuwNrvMH2m4G/uWMbiISwIvxdb5B/TSCSAKJCdrcG/vauy6SZTDn8g7wCK7KRIOCaJnnXRGjwFVB7mZ8PBqEyCUUhBgex6fvWWgeeoblNqe/ieeWNNQbXEix738Ru57Ci0By6daC7Nl93DvaxDcN6Apq0ZK/baZK1sJnDZi4kHUg2M4TWmJAqxhPh1DOeo8s5GbtB+RxT9bhwyfM7GxLHYzppx9MJc9jjRmeObTEu8wSRUUzyPuMkBUl0IotLd830hLOYaWUqvXWtmPi/Va5vBju09aEs/kFgDq1D0oCXg1k/O3dCqQzrXFFlCxbJOjZrXAuYNsDqgAsyf+/85uvmCmebG34uf9jq7x9UL98yeoleoXXkox20j96jI3SKKPqMvqNf6HfnU+dL52vn2wxdXKhsXqDa6vz4C3FeMPk=</latexit><latexit sha1_base64="s6BIwT4drCR6QCX+SZn7AuZMIgA=">AAAF/HicfZRLb9QwEIDd0oWyvFo4ctmylyKtqqRUbS+VqqIKjqXqS6pXK8eZzVrrOMF2+rLMT+BXcEPcECfgf/BvSLKR2MQRjhSNPN+8PGMHKWdKe96fhcV7S537D5Yfdh89fvL02crq8zOVZJLCKU14Ii8CooAzAaeaaQ4XqQQSBxzOg+nbQn9+BVKxRJzo2xSGMYkEGzNKdL41WlnHEYljMjJiant7vXT9biR6eG0Pr017OGZhD1+ZGzsSr0crfW/DK1fPFfxK6O+j2ToarS79xGFCsxiEppwodel7qR4aIjWjHGwXZwpSQqckgstcFCQGNTRlSbZX0574QzNOhAZBa2aGxComeuJsFrCq79JJHhhkPWy1OTSFlxAUi0TdKohtt4tDGOfHW2ZmwoBnYM3xuwNrvMH2m4G/uWMbiISwIvxdb5B/TSCSAKJCdrcG/vauy6SZTDn8g7wCK7KRIOCaJnnXRGjwFVB7mZ8PBqEyCUUhBgex6fvWWgeeoblNqe/ieeWNNQbXEix738Ru57Ci0By6daC7Nl93DvaxDcN6Apq0ZK/baZK1sJnDZi4kHUg2M4TWmJAqxhPh1DOeo8s5GbtB+RxT9bhwyfM7GxLHYzppx9MJc9jjRmeObTEu8wSRUUzyPuMkBUl0IotLd830hLOYaWUqvXWtmPi/Va5vBju09aEs/kFgDq1D0oCXg1k/O3dCqQzrXFFlCxbJOjZrXAuYNsDqgAsyf+/85uvmCmebG34uf9jq7x9UL98yeoleoXXkox20j96jI3SKKPqMvqNf6HfnU+dL52vn2wxdXKhsXqDa6vz4C3FeMPk=</latexit><latexit sha1_base64="TPTl4FzVvO4VXZrW2Hc+rC2QUOU=">AAAF/HicfZRLb9QwEIDd0oWyvLZw5JKylyKtqqRUbS+VqqIKjqXqS6pXKyeZ3bXWcYLt9GWZn8Cv4Ia4IU7A/+Df4KSR2MQRjhSNPN+8PGOHGaNS+f6fhcV7S537D5Yfdh89fvL0WW/l+alMcxHBSZSyVJyHRAKjHE4UVQzOMwEkCRmchbO3hf7sEoSkKT9WNxkMEzLhdEwjouzWqLeGJyRJyEjzmfF2vWztdsQ9vLqLV2ceTmjs4Ut9bUb89ajX99f9cnmuEFRCH1XrcLSy9BPHaZQnwFXEiJQXgZ+poSZC0YiB6eJcQkaiGZnAhRU5SUAOdVmS8Wra42CoxylXwKOamSaJTIiaOpsFLOu70dQGBlEPW20OdeElBkknvG4VJqbbxTGM7fGWmek4ZDkYffRu32h/sPVmEGxsmwYiIK6IYMcf2K8JTAQAr5CdzUGwteMyWS4yBv8gv8CKbARwuIpS2zUea3wJkbmw54OBy1xAUYjGYaL7gTHGge9Qa1Pqu3heeW20xrUEy943sZs5rCjUQjcOdNvm69bBPrZhWE1BkZbsVTtN8hY2d9jchYQDiWaG0BoTMklZyp16xnN0OSdjNyibY6oeFy6ZvbMxcTxm03Y8m1KHPWp05sgU4zJPEDFJiO0zTjMQRKWiuHRXVE0ZTaiSutIb14ry/1tZfTPYgakPZfEPQ31gHDIKWTmY9bNzJzQScZ0rqmzBJqKO3TWuBcwaYHXABWnfu6D5urnC6cZ6YOUPm/29/erlW0Yv0Su0hgK0jfbQe3SITlCEPqPv6Bf63fnU+dL52vl2hy4uVDYvUG11fvwFH48wuQ==</latexit>

KX

k=1

�nk = 1<latexit sha1_base64="kB+keM/zdRKnmv9weAjlrJQEy8Q=">AAAGGXicfZRLb9QwEIDd0oWyvFo4clmxFw6rKt4u3XCoVBVVIHEpVV/SZlk5yWw22sQJttOX5V/CT+AvcOGGuCFOwJ/ByQaxiSMcKRrNfPOwZ2w3jUIuLOvnyuqttdbtO+t32/fuP3j4aGPz8SlPMubBiZdECTt3CYcopHAiQhHBecqAxG4EZ+78VW4/uwDGw4Qei+sUxjEJaDgNPSK0arJxIJ0iyIgF7lhaWzvDF9i2e9ZW38Y2HmjBermNh33l8CyeyPkuVu/fdpyAxDGZSDpXnd0OVpONrgaL1TEFXArdPbRYh5PNtc+On3hZDFR4EeF8hK1UjCVhIvQiUG0n45ASb04CGGmRkhj4WBalqk7FeozHcppQAdSruEkS85iImaHMYV7VejOdGFg1bakcyzyKDzwMaNXLjVW77fgw1WdfVCZ9N8pAyaPX+0pavZ3tHu4PVQ1h4JcEtq2e/upAwABoidiDHt6xTSbNWBrBP8jKsbwaBhQuvUR3h/rSuQBPjfT5OEB5xiDfiHTcWHaxUsqAF6j2KextZ9l4pWQ5KX8LdC7klapj10tYvlENXRvQTVOsGwP70IQ5YgaCNFQvmmmSNbCZwWYmxAyI1SuExpyQ8jBKqLGf6RJdzMnUTBotMWWP85CRvtA+MSKms2Y8nYUGe1TrzJHKx2WZICyIie6zk6TAiEhYfukuQzGLwjgUXJZ2ZXqF9P9e2l5PdqCqQ5n/XVceKIP03KgYzOrZmRPqMb/K5btswAJWxRaNawDTGlgecE7q9w7XXzdTOO1vYS2/G3T39suXbx09Rc/Qc4TREO2hN+gQnSAPfULf0S/0u/Wx9aX1tfVtga6ulD5PUGW1fvwBmCw5yA==</latexit><latexit sha1_base64="kB+keM/zdRKnmv9weAjlrJQEy8Q=">AAAGGXicfZRLb9QwEIDd0oWyvFo4clmxFw6rKt4u3XCoVBVVIHEpVV/SZlk5yWw22sQJttOX5V/CT+AvcOGGuCFOwJ/ByQaxiSMcKRrNfPOwZ2w3jUIuLOvnyuqttdbtO+t32/fuP3j4aGPz8SlPMubBiZdECTt3CYcopHAiQhHBecqAxG4EZ+78VW4/uwDGw4Qei+sUxjEJaDgNPSK0arJxIJ0iyIgF7lhaWzvDF9i2e9ZW38Y2HmjBermNh33l8CyeyPkuVu/fdpyAxDGZSDpXnd0OVpONrgaL1TEFXArdPbRYh5PNtc+On3hZDFR4EeF8hK1UjCVhIvQiUG0n45ASb04CGGmRkhj4WBalqk7FeozHcppQAdSruEkS85iImaHMYV7VejOdGFg1bakcyzyKDzwMaNXLjVW77fgw1WdfVCZ9N8pAyaPX+0pavZ3tHu4PVQ1h4JcEtq2e/upAwABoidiDHt6xTSbNWBrBP8jKsbwaBhQuvUR3h/rSuQBPjfT5OEB5xiDfiHTcWHaxUsqAF6j2KextZ9l4pWQ5KX8LdC7klapj10tYvlENXRvQTVOsGwP70IQ5YgaCNFQvmmmSNbCZwWYmxAyI1SuExpyQ8jBKqLGf6RJdzMnUTBotMWWP85CRvtA+MSKms2Y8nYUGe1TrzJHKx2WZICyIie6zk6TAiEhYfukuQzGLwjgUXJZ2ZXqF9P9e2l5PdqCqQ5n/XVceKIP03KgYzOrZmRPqMb/K5btswAJWxRaNawDTGlgecE7q9w7XXzdTOO1vYS2/G3T39suXbx09Rc/Qc4TREO2hN+gQnSAPfULf0S/0u/Wx9aX1tfVtga6ulD5PUGW1fvwBmCw5yA==</latexit><latexit sha1_base64="kB+keM/zdRKnmv9weAjlrJQEy8Q=">AAAGGXicfZRLb9QwEIDd0oWyvFo4clmxFw6rKt4u3XCoVBVVIHEpVV/SZlk5yWw22sQJttOX5V/CT+AvcOGGuCFOwJ/ByQaxiSMcKRrNfPOwZ2w3jUIuLOvnyuqttdbtO+t32/fuP3j4aGPz8SlPMubBiZdECTt3CYcopHAiQhHBecqAxG4EZ+78VW4/uwDGw4Qei+sUxjEJaDgNPSK0arJxIJ0iyIgF7lhaWzvDF9i2e9ZW38Y2HmjBermNh33l8CyeyPkuVu/fdpyAxDGZSDpXnd0OVpONrgaL1TEFXArdPbRYh5PNtc+On3hZDFR4EeF8hK1UjCVhIvQiUG0n45ASb04CGGmRkhj4WBalqk7FeozHcppQAdSruEkS85iImaHMYV7VejOdGFg1bakcyzyKDzwMaNXLjVW77fgw1WdfVCZ9N8pAyaPX+0pavZ3tHu4PVQ1h4JcEtq2e/upAwABoidiDHt6xTSbNWBrBP8jKsbwaBhQuvUR3h/rSuQBPjfT5OEB5xiDfiHTcWHaxUsqAF6j2KextZ9l4pWQ5KX8LdC7klapj10tYvlENXRvQTVOsGwP70IQ5YgaCNFQvmmmSNbCZwWYmxAyI1SuExpyQ8jBKqLGf6RJdzMnUTBotMWWP85CRvtA+MSKms2Y8nYUGe1TrzJHKx2WZICyIie6zk6TAiEhYfukuQzGLwjgUXJZ2ZXqF9P9e2l5PdqCqQ5n/XVceKIP03KgYzOrZmRPqMb/K5btswAJWxRaNawDTGlgecE7q9w7XXzdTOO1vYS2/G3T39suXbx09Rc/Qc4TREO2hN+gQnSAPfULf0S/0u/Wx9aX1tfVtga6ulD5PUGW1fvwBmCw5yA==</latexit><latexit sha1_base64="1JyZBQ5Z7F2Urhr97RI+dmF3euU=">AAAGGXicfZRLb9QwEIDd0oWyvFo4clmxFw6rKt6WbjhUqooqkLiUqi9ps6ycZDYbreME2+nL8i/hJ/AXuHBD3BAn4M/gbIPYxBGOFI1mvnnYM7af0VhIx/m5tHxrpXX7zurd9r37Dx4+Wlt/fCLSnAdwHKQ05Wc+EUBjBscylhTOMg4k8Smc+rNXhf30HLiIU3YkrzIYJSRi8SQOiDSq8dq+8uZBhjzyR8rZ2B68wK7bczb6LnbxlhGcl5t40NeeyJOxmu1g/f5tx4tIkpCxYjPd2elgPV7rGnC+OraAS6GLynUwXl/57IVpkCfAZECJEEPsZHKkCJdxQEG3vVxARoIZiWBoREYSECM1L1V3KtYjPFKTlElgQcVNkUQkRE4tZQGLqjaYmsTAq2lL5UgVUUIQccSqXn6i220vhIk5+3llKvRpDlodvt7Tyultb/Zwf6BrCIewJLDr9MxXByIOwErE3erhbddmspxnFP5BToEV1XBgcBGkpjssVN45BHpozscDJnIOxUaU5yeqi7XWFnyDGp+5ve0tGi+1Kiflb4HeubrUdexqASs2aqArC7puinVtYR+aME9OQZKG6mUzTfIGNrfY3Ia4BfF6hdCYEzIR05RZ+5ks0PM5mdhJ6QJT9rgISc2FDokVMZs249k0ttjDWmcOdTEuiwThUUJMn700A05kyotLdxHLKY2TWApV2rXtFbP/exl7Pdm+rg5l8fd9ta8tMvDpfDCrZ2dPaMDDKlfssgGLeBW7aVwDmNXA8oAL0rx3uP662cJJfwMb+d1Wd3evfPlW0VP0DD1HGA3QLnqDDtAxCtAn9B39Qr9bH1tfWl9b327Q5aXS5wmqrNaPP0ZdOYg=</latexit>

Weights sum to 1:

Algorithm

Initialize parameters to

Repeat until convergence

1. Update cluster assignments

2. Update parameters

✓ := {µ1:K ,⌃1:K ,⇡}

“Hard EM” with Gaussians

Parameter Updates

znk := I[zn = k] Nk :=PN

n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

PNn=1 znk xn

⌃k =1Nk

PNn=1 znk (xn �µk)(xn �µk)>


n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk

Pn=1 znk (xn �µk)(xn �µk)>


n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

PNn=1 znk xn

⌃k =1Nk



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk


Assignment Update


Initialize parameters randomlywhile not converged

1. E-Step:Set the latent variables to the the values that maximizes likelihood, treating parameters as observed

2. M-Step:Set the parameters to the values that maximizes likelihood, treating latent variables as observed

Slide credit: Matt Gormley and Eric Xing (CMU)

“Hard” EM: General

Hallucinate labels






Hallucinate labels

Train (as if supervised)






Algorithm 1 Hard EM for MMs

1: procedure H EM(D = {t(i)}Ni=1)

2: Randomly initialize parameters, �, �3: while not converged do4: E-Step:

z(i) � �`;K�tz

HQ; p(t(i)|z; �) + HQ; p(z; �)

5: M-Step:

� � �`;K�t�

N�

i=1

HQ; p(z(i); �)

� � �`;K�t�

N�

i=1

HQ; p(t(i)|z; �)

6: return (�, �)





z(i) � �`;K�tz

HQ; p(t(i)|z; �) + HQ; p(z; �)

5: M-Step:

� � �`;K�t�

N�

i=1

HQ; p(z(i); �)

� � �`;K�t�

N�

i=1

HQ; p(t(i)|z; �)

6: return (�, �)

Just loop over potential assignments





z(i) � �`;K�tz

HQ; p(t(i)|z; �) + HQ; p(z; �)

5: M-Step:

� � �`;K�t�

N�

i=1

HQ; p(z(i); �)

� � �`;K�t�

N�

i=1

HQ; p(t(i)|z; �)

6: return (�, �)

Supervised learning

Just loop over potential assignments


Algorithm 1 Hard EM for GMMs


2: Randomly initialize parameters, �, µ,�3: while not converged do4: E-Step:

z(i) � �`;K�tz

HQ; p(t(i)|z; µ,�) + HQ; p(z; �)

5: M-Step:

�k � 1

N

N�

i=1

I(z(i) = k), �k

µk ��N

i=1 I(z(i) = k)t(i)

�Ni=1 I(z(i) = k)

, �k

�k ��N

i=1 I(z(i) = k)(t(i) � µk)(t(i) � µk)T

�Ni=1 I(z(i) = k)

, �k

6: return (�, µ,�)





z(i) � �`;K�tz

HQ; p(t(i)|z; µ,�) + HQ; p(z; �)

5: M-Step:

�k � 1

N

N�

i=1

I(z(i) = k), �k

µk ��N

i=1 I(z(i) = k)t(i)

�Ni=1 I(z(i) = k)

, �k

�k ��N

i=1 I(z(i) = k)(t(i) � µk)(t(i) � µk)T

�Ni=1 I(z(i) = k)

, �k

6: return (�, µ,�)





z(i) � �`;K�tz

HQ; p(t(i)|z; µ,�) + HQ; p(z; �)

5: M-Step:

�k � 1

N

N�

i=1

I(z(i) = k), �k

µk ��N

i=1 I(z(i) = k)t(i)

�Ni=1 I(z(i) = k)

, �k

�k ��N

i=1 I(z(i) = k)(t(i) � µk)(t(i) � µk)T

�Ni=1 I(z(i) = k)

, �k

6: return (�, µ,�)


Parameter Updates


n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

PNn=1 znk xn

⌃k =1Nk



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

PNn=1 znk xn

⌃k =1Nk



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk


Assignment Update

How can we deal with overlapping clusters in a better way?


Learn Soft Assignments to ClustersPosterior on Cluster Assignments (from Bayes’ Rule)

�nk = p(zn=k | xn) =p(xn | zn=k)p(zn=k)

p(xn)<latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="01dvNLe0oQZXqM25sO+shPAxN0c=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0EPVOhqtLn3GQULzGLikEcmyS89N5VARIRmNQHdxnkFK6JSEcGlETmLIhqo8E+3UrCfeUI0TLoHTmpsicRYTObGUBZzVtXRiEoOop62UQ1VECSBjIa97+bHudnEAY9OfsjIV+FEOWh2/P9DKHWy/HXibO7qBCAgqwtt1B+ZrAqEA4BWyuzXwtndtJs1FGsE95BZYUY0ADtc0MW3ngcJXQPWlOR8MPMsFFBtR2I9Vz9NaW/AMNT6lvYvnjTdaKVwrsGx+E7udw4qNGujWgu7aYt1Z2Kc2DMsJSNJSvWynSd7C5hab25CwINGsEFpzQpqxKOHWfsZzdDknYztpNMdUPS5CRubSB8SKmE7a8XTCLPa40ZljXYzLPEFEGBPTZ5ykIIhMRHHprpmcRCxmMlOVXdtejP/fy9ibyQ51fSiLv++rQ22R1I/KwayfnT2hVAR1rthlCxaKOjZrXAuYNsDqgAvSvHde83WzhbPNDc/IH7d6+wfVy7eMXqM3aB15aAftow/oCJ0iir6iX+gP+tv50vne+dH5OUMXFyqfV6i2Or//AdjXSHE=</latexit>

Likelihood Prior

Marginal LikelihoodPosterior

Learn Soft Assignments to ClustersPosterior on Cluster Assignments (from Bayes’ Rule)

�nk = p(zn=k | xn) =p(xn | zn=k)p(zn=k)

p(xn)<latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="01dvNLe0oQZXqM25sO+shPAxN0c=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0EPVOhqtLn3GQULzGLikEcmyS89N5VARIRmNQHdxnkFK6JSEcGlETmLIhqo8E+3UrCfeUI0TLoHTmpsicRYTObGUBZzVtXRiEoOop62UQ1VECSBjIa97+bHudnEAY9OfsjIV+FEOWh2/P9DKHWy/HXibO7qBCAgqwtt1B+ZrAqEA4BWyuzXwtndtJs1FGsE95BZYUY0ADtc0MW3ngcJXQPWlOR8MPMsFFBtR2I9Vz9NaW/AMNT6lvYvnjTdaKVwrsGx+E7udw4qNGujWgu7aYt1Z2Kc2DMsJSNJSvWynSd7C5hab25CwINGsEFpzQpqxKOHWfsZzdDknYztpNMdUPS5CRubSB8SKmE7a8XTCLPa40ZljXYzLPEFEGBPTZ5ykIIhMRHHprpmcRCxmMlOVXdtejP/fy9ibyQ51fSiLv++rQ22R1I/KwayfnT2hVAR1rthlCxaKOjZrXAuYNsDqgAvSvHde83WzhbPNDc/IH7d6+wfVy7eMXqM3aB15aAftow/oCJ0iir6iX+gP+tv50vne+dH5OUMXFyqfV6i2Or//AdjXSHE=</latexit>

Likelihood Prior

Marginal LikelihoodPosterior

p(zn=k) = ⇡k

p(xn | zn=k) =1p

2⇡|⌃|e�

12 (xn�µk)>⌃�1(xn�µk)

p(4vxn) =KX

k=1

p(xn | zn=k)p(zn=k)<latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="QaDSdZsdnnw46aiXjE1zxHFSMx8=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMiykoj4Vllum6qCsbext+3v7I/sxwwmOaaNLW8KhIPOd246x8eNfZIw2/5dW3iw+LD+aOmx+eTps+cvlldeniRRShE+RpEf0TMXJtgnIT5mhPn4LKYYBq6PT93Je6U/vcQ0IVF4xKYx7gdwFJIhQZDJq8Hyn3j9ZhBaYLULVq1J0zLXuhaIyWBiAtDrxKxvxuvgkl8LBQXEsySdwZOmuWZ2TTCkEHFHcJBcUMbbyti6lRbgkIwCKG6FMPE537jj2sK6c7hhKTBIxWDSPAcsiq17Q2nj/AMVMjeZ1uZM1bRUykkaDPik64jzT5ae8X19xXIHyw37rZ0dSxecXGgY+dkfrCz+Al6E0gCHDPkwSXqOLR+JQ8oI8rHMLE1wDNEEjnBPiiEMcNLnWZ+EVdAeOX0+jEKGQ1Qw4zBIAsjG2qWCk+ItGsvAmBbD5pd9rrx4OCGjsGjlBsI0gYeHcmayzLjn+ikW/ODDruB2a+tdy2lvixJCsZcTTsduyV8ZGFGMwxzpbLacrY7OxCmNfXwP2QpT2VAc4isUBQEMPQ4uMRI9+T4Ah0lKsSqEAzfgDUfIeSrDM1TaZHoTzCuvBeegkGA2G2VsOoepQiU01aCbKl83GnZRhQE2xgxWZM+qaZhWsKnGpjpENYiWM8SVMXGcED8KtXqGc3Q2J0M9qD/H5D1WLn25iDyoeYzH1Xg8Jhp7UOrMgVDjMk9AKveF7DOIYkwhi6j66K4IG/skICzhuV7oViT8v5XUl4PtieJQqn/X5XtCI5HrZ4NZfDt9QhH1ipyqsgIb0SI2a1wFGJfA/IEVKfedU95uunDSfutI+fNmY2c333xLxmvjjbFuOMa2sWN8NPaNYwPVUO1b7UftZ/2wPq1/qX+doQu13OaVUTj1738B/zV22Q==</latexit>

p(zn=k) = ⇡k

p(xn | zn=k) =1p

2⇡|⌃|e�

12 (xn�µk)>⌃�1(xn�µk)

p(4vxn) =KX

k=1


p(zn=k) = ⇡k

p(xn | zn=k) =1p

2⇡|⌃|e�

12 (xn�µk)>⌃�1(xn�µk)

p(4vxn) =KX

k=1


p(zn=k) = ⇡k

p(xn | zn=k) =1p

2⇡|⌃|e�

12 (xn�µk)>⌃�1(xn�µk)

p(4vxn) =KX

k=1


p(zn=k) = ⇡k

p(xn | zn=k) =1p

2⇡|⌃|e�

12 (xn�µk)>⌃�1(xn�µk)

p(xn) =KX

k=1

p(xn | zn=k)p(zn=k)<latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="RgVg34VZMZaqts3thXMJPnaoqpI=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FhpWf3eHS/C8QJH4aoZj7IWSs77bVEwlIOfZDpPJKGSLQn8Ix6isxhhFiA5F1SToF7YE7EKMk5ij2C2YCRiyCfGJcapgVb/2JCoxoMWx+ORDaS4AYHsdFKy+Stg0CNFITk2UmAi9MkRT7H7alaLc23rXczqYsIRQFOeF22y31KwNjilCcI931lrvRNRmSUhKie6itMZ0NRTG68JMognEgwDnyZV+9D0AxSynShQjgRaLhSjVNZfgWVTaZ3gazykspBCgkmM1GGbuawXShCroyoOsqX9cGdlaFAT5BHFZkz6tpmFawqcGmJkQNiJYzRJUxEWE4TGKjntEMnc3JyAwazjB5j7XLUK2hABoeyaQaJxNssPulzuxLPS6zBKRqW6g+g4QgCnlC9Ud3gfkkxBHmTOR6aVrh+P9WSl8OtiOLQ6n/PU/sSIP0vTAbzOLbmRPq06DI6SorsDEtYreNqwBJCcwfWJNq37nl7WYKR523rpL31htb2/nmW7BeW2+sVcu1Nq0t66O1ax1afg3WvtV+1H7W9+oX9S/1r7foXC23eWUVTv37X2OLdhs=</latexit>

p(zn=k) = ⇡k

p(xn | zn=k) =1p

2⇡|⌃|e�

12 (xn�µk)>⌃�1(xn�µk)

p(xn) =KX

k=1

p(xn | zn=k)p(zn=k)<latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="RgVg34VZMZaqts3thXMJPnaoqpI=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FhpWf3eHS/C8QJH4aoZj7IWSs77bVEwlIOfZDpPJKGSLQn8Ix6isxhhFiA5F1SToF7YE7EKMk5ij2C2YCRiyCfGJcapgVb/2JCoxoMWx+ORDaS4AYHsdFKy+Stg0CNFITk2UmAi9MkRT7H7alaLc23rXczqYsIRQFOeF22y31KwNjilCcI931lrvRNRmSUhKie6itMZ0NRTG68JMognEgwDnyZV+9D0AxSynShQjgRaLhSjVNZfgWVTaZ3gazykspBCgkmM1GGbuawXShCroyoOsqX9cGdlaFAT5BHFZkz6tpmFawqcGmJkQNiJYzRJUxEWE4TGKjntEMnc3JyAwazjB5j7XLUK2hABoeyaQaJxNssPulzuxLPS6zBKRqW6g+g4QgCnlC9Ud3gfkkxBHmTOR6aVrh+P9WSl8OtiOLQ6n/PU/sSIP0vTAbzOLbmRPq06DI6SorsDEtYreNqwBJCcwfWJNq37nl7WYKR523rpL31htb2/nmW7BeW2+sVcu1Nq0t66O1ax1afg3WvtV+1H7W9+oX9S/1r7foXC23eWUVTv37X2OLdhs=</latexit>

Prior

Likelihood

MarginalLikelihood

Learn Gaussian for Each Cluster Maximum Likelihood Estimation

µ⇤,⌃⇤,⇡⇤ = argmaxµ,⌃,⇡

log p(x1, . . . , xN | µ,⌃,⇡)<latexit sha1_base64="VGcx1UxYqdEnlbi/OsG2rnPTTNM=">AAAGZHicfZTrbtMwFIDTsZVRBmxM/AKhiAo0UDUlY9r2B2kamuAXGmM3aSmV45y21pwLtrOb5T0DD8SL8AI8Ar+x02g0cTRXqk7P+c7FPqcnzCjhwvN+t2buzc61788/6DxcePT4yeLS0yOe5gzDIU5pyk5CxIGSBA4FERROMgYoDikch2cfjf34HBgnaXIgrjLox2iUkCHBSGjVYPFncC6DOFff3/VcI34joxjd/sqIFt0PboCYVl8OZElPs7ekcm9uApqO3GxFay7VwNcmGqWCF4hWfHGDmETuXUHeDha73qpXHNcW/FLobjuTszdYmv0VRCnOY0gEpojzU9/LRF8iJgimoDpBziFD+AyN4FSLCYqB92XxdMqtWA/8vhymiYAEV9wkinmMxNhSGphXtXisEwOrpi2VfWmiRMDJKKl6hbHqdIIIhrqNRWUyCmkOSu5/2lHS62287/lrm6qGMIhKwt/yevpTB0YMICmRrfWev7FlM1nOMgr/Ic9gphoGCVzgNI5REunGA1an+n0CSHjOwFxEBmEsu75SyoInqPYp7J1g2nippAwqBRbTUceupjBzUQ1dWdB1U6xrC/vRhAViDAI1VC+aaZQ3sLnF5jbELIjVK4TGnJBxQtPEus9wii7mZGgnpVNM2WMTkurdECErYjZuxrMxsdj9Wmf2lRmXaWKyLpQM0gwYEikzf7oLIsaUxERwWdqV7UWSu720vZ5sV1WH0nyHodxVFolDWgxm9e3sCcUsqnLmlg3YiFWxSeMawKwGlg9sSL3v/Pp2s4WjtVVfy1/Xu9s75eabd547r5wVx3c2nW3ns7PnHDrY+dt60XrdejP3p73QXm4/m6AzrdJn2amc9st/8VFVCw==</latexit><latexit sha1_base64="VGcx1UxYqdEnlbi/OsG2rnPTTNM=">AAAGZHicfZTrbtMwFIDTsZVRBmxM/AKhiAo0UDUlY9r2B2kamuAXGmM3aSmV45y21pwLtrOb5T0DD8SL8AI8Ar+x02g0cTRXqk7P+c7FPqcnzCjhwvN+t2buzc61788/6DxcePT4yeLS0yOe5gzDIU5pyk5CxIGSBA4FERROMgYoDikch2cfjf34HBgnaXIgrjLox2iUkCHBSGjVYPFncC6DOFff3/VcI34joxjd/sqIFt0PboCYVl8OZElPs7ekcm9uApqO3GxFay7VwNcmGqWCF4hWfHGDmETuXUHeDha73qpXHNcW/FLobjuTszdYmv0VRCnOY0gEpojzU9/LRF8iJgimoDpBziFD+AyN4FSLCYqB92XxdMqtWA/8vhymiYAEV9wkinmMxNhSGphXtXisEwOrpi2VfWmiRMDJKKl6hbHqdIIIhrqNRWUyCmkOSu5/2lHS62287/lrm6qGMIhKwt/yevpTB0YMICmRrfWev7FlM1nOMgr/Ic9gphoGCVzgNI5REunGA1an+n0CSHjOwFxEBmEsu75SyoInqPYp7J1g2nippAwqBRbTUceupjBzUQ1dWdB1U6xrC/vRhAViDAI1VC+aaZQ3sLnF5jbELIjVK4TGnJBxQtPEus9wii7mZGgnpVNM2WMTkurdECErYjZuxrMxsdj9Wmf2lRmXaWKyLpQM0gwYEikzf7oLIsaUxERwWdqV7UWSu720vZ5sV1WH0nyHodxVFolDWgxm9e3sCcUsqnLmlg3YiFWxSeMawKwGlg9sSL3v/Pp2s4WjtVVfy1/Xu9s75eabd547r5wVx3c2nW3ns7PnHDrY+dt60XrdejP3p73QXm4/m6AzrdJn2amc9st/8VFVCw==</latexit><latexit sha1_base64="VGcx1UxYqdEnlbi/OsG2rnPTTNM=">AAAGZHicfZTrbtMwFIDTsZVRBmxM/AKhiAo0UDUlY9r2B2kamuAXGmM3aSmV45y21pwLtrOb5T0DD8SL8AI8Ar+x02g0cTRXqk7P+c7FPqcnzCjhwvN+t2buzc61788/6DxcePT4yeLS0yOe5gzDIU5pyk5CxIGSBA4FERROMgYoDikch2cfjf34HBgnaXIgrjLox2iUkCHBSGjVYPFncC6DOFff3/VcI34joxjd/sqIFt0PboCYVl8OZElPs7ekcm9uApqO3GxFay7VwNcmGqWCF4hWfHGDmETuXUHeDha73qpXHNcW/FLobjuTszdYmv0VRCnOY0gEpojzU9/LRF8iJgimoDpBziFD+AyN4FSLCYqB92XxdMqtWA/8vhymiYAEV9wkinmMxNhSGphXtXisEwOrpi2VfWmiRMDJKKl6hbHqdIIIhrqNRWUyCmkOSu5/2lHS62287/lrm6qGMIhKwt/yevpTB0YMICmRrfWev7FlM1nOMgr/Ic9gphoGCVzgNI5REunGA1an+n0CSHjOwFxEBmEsu75SyoInqPYp7J1g2nippAwqBRbTUceupjBzUQ1dWdB1U6xrC/vRhAViDAI1VC+aaZQ3sLnF5jbELIjVK4TGnJBxQtPEus9wii7mZGgnpVNM2WMTkurdECErYjZuxrMxsdj9Wmf2lRmXaWKyLpQM0gwYEikzf7oLIsaUxERwWdqV7UWSu720vZ5sV1WH0nyHodxVFolDWgxm9e3sCcUsqnLmlg3YiFWxSeMawKwGlg9sSL3v/Pp2s4WjtVVfy1/Xu9s75eabd547r5wVx3c2nW3ns7PnHDrY+dt60XrdejP3p73QXm4/m6AzrdJn2amc9st/8VFVCw==</latexit><latexit sha1_base64="Nnf+J7NPD+a2xcEGiekcvI6Ye+A=">AAAGZHicfZTdbtMwFIDTsZVRBmxMXIFQRAUaqJqSMW27QZqGJrhCY+xPmkvlJKetNScxtrM/y3sGHogX4QV4BK5x0mg0cTRXqk7P+c6PfU5PwCgR0vN+t2buzc61788/6DxcePT4yeLS0yORZjyEwzClKT8JsABKEjiURFI4YRxwHFA4Ds4+5vbjc+CCpMmBvGLQj/EoIUMSYmlUg8Wf6FyhONPf3/XcXPxGRjG+/cWIEd0PLsLcqC8HqqSn2VtSuzc3iKYjl60YzaUe+MZEo1SKAjGKLy6KSeTeFeTtYLHrrXrFcW3BL4WuU569wdLsLxSlYRZDIkOKhTj1PSb7CnNJQgq6gzIBDIdneASnRkxwDKKviqfTbsV64PfVME0kJGHFTeFYxFiOLWUOi6o2HJvEwKtpS2Vf5VEiEGSUVL2CWHc6KIKhaWNRmYoCmoFW+592tPJ6G+97/tqmriEcopLwt7ye+dSBEQdISmRrvedvbNkMyzij8B/yciyvhkMCF2EaxziJTOMh1KfmfRAkIuOQX0ShIFZdX2ttwRPU+BT2Dpo2XmqlUKXAYjrq2NUUll/UQFcWdN0U69rCfjRhSI5B4obqZTONswY2s9jMhrgF8XqF0JgTmCA0Taz7DKfoYk6GdlI6xZQ9zkNSsxsibEVk42acjYnF7tc6s6/zcZkmJutCK5Qy4FimPP/TXRA5piQmUqjSrm0vktztZez1ZLu6OpT5dxCoXW2RYUCLway+nT2hIY+qXH7LBmzEq9ikcQ0gq4HlA+ek2Xd+fbvZwtHaqm/kr+vd7Z1y8807z51XzorjO5vOtvPZ2XMOndD523rRet16M/envdBebj+boDOt0mfZqZz2y3+fglTL</latexit>

Nk =X

n

�nk<latexit sha1_base64="SDjl3WfkWXVthwbBKbP9SkNtmcg=">AAAGF3icfZRLb9NAEMe3pYESXi0cuVjkwiGK7PQRc0CqWlVwQqXqS4qjaG1PHCv22uyu+1rtB+Ej8B2QuCFuiBv9NqwdR8Rew0aKRju//8zsznjdNAoZN827ldV7a637D9Yfth89fvL02cbm8zOWZNSDUy+JEnrhYgZRSOCUhzyCi5QCjt0Izt3ZQe4/vwTKwoSc8JsURjEOSDgJPczV1njjQDhFkCEN3JEwe7uDHcu2u2avb1u2ta0M882WNejLD+OZ8dZwWBaPBZGGE+A4xsqcSTne6CyExkJoLISG1TOL1dlD83U03lz76viJl8VAuBdhxoaWmfKRwJSHXgSy7WQMUuzNcABDZRIcAxuJolBpVLwn1khMEsKBeBWZwDGLMZ9qmznMqrveVCUGWk1bbo5EHsUHFgakqnJj2W47PkzUzReVCd+NMpDi+N2+FGZ3d6tr9QeyhlDwS8Kyza761YGAApASsbe71q6tM2lG0wj+QmaO5dVQIHDlJaozxBfOJXhyqO7HAcIyCvlBhOPGomNJKTV4jipN4W87y85rKco5WRToXIprWcdulrD8oAq60aDbpli3GvapCXP4FDhuqJ430zhrYDONzXSIahCtVwiNOSFlYZQQ7TyTJbqYk4meNFpiyh7nISP1OftYi5hOm/F0Gmrsca0zxzIfl2UC0yDGqs9OkgLFPKH5R3cV8mkUxiFnovRLXRWS/6uUv57sUFaHMv93XXEoNdJzo2Iwq3enT6hH/SqXn7IBC2gVmzeuAUxrYHnBOaneu8WjZvzbOOv3LGV/3O7s7Zcv3zp6iV6h18hCA7SH3qMjdIo89AX9RL/RXetz61vre+vHHF1dKTUvUGW1fv0B0Hc52A==</latexit><latexit sha1_base64="SDjl3WfkWXVthwbBKbP9SkNtmcg=">AAAGF3icfZRLb9NAEMe3pYESXi0cuVjkwiGK7PQRc0CqWlVwQqXqS4qjaG1PHCv22uyu+1rtB+Ej8B2QuCFuiBv9NqwdR8Rew0aKRju//8zsznjdNAoZN827ldV7a637D9Yfth89fvL02cbm8zOWZNSDUy+JEnrhYgZRSOCUhzyCi5QCjt0Izt3ZQe4/vwTKwoSc8JsURjEOSDgJPczV1njjQDhFkCEN3JEwe7uDHcu2u2avb1u2ta0M882WNejLD+OZ8dZwWBaPBZGGE+A4xsqcSTne6CyExkJoLISG1TOL1dlD83U03lz76viJl8VAuBdhxoaWmfKRwJSHXgSy7WQMUuzNcABDZRIcAxuJolBpVLwn1khMEsKBeBWZwDGLMZ9qmznMqrveVCUGWk1bbo5EHsUHFgakqnJj2W47PkzUzReVCd+NMpDi+N2+FGZ3d6tr9QeyhlDwS8Kyza761YGAApASsbe71q6tM2lG0wj+QmaO5dVQIHDlJaozxBfOJXhyqO7HAcIyCvlBhOPGomNJKTV4jipN4W87y85rKco5WRToXIprWcdulrD8oAq60aDbpli3GvapCXP4FDhuqJ430zhrYDONzXSIahCtVwiNOSFlYZQQ7TyTJbqYk4meNFpiyh7nISP1OftYi5hOm/F0Gmrsca0zxzIfl2UC0yDGqs9OkgLFPKH5R3cV8mkUxiFnovRLXRWS/6uUv57sUFaHMv93XXEoNdJzo2Iwq3enT6hH/SqXn7IBC2gVmzeuAUxrYHnBOaneu8WjZvzbOOv3LGV/3O7s7Zcv3zp6iV6h18hCA7SH3qMjdIo89AX9RL/RXetz61vre+vHHF1dKTUvUGW1fv0B0Hc52A==</latexit><latexit sha1_base64="SDjl3WfkWXVthwbBKbP9SkNtmcg=">AAAGF3icfZRLb9NAEMe3pYESXi0cuVjkwiGK7PQRc0CqWlVwQqXqS4qjaG1PHCv22uyu+1rtB+Ej8B2QuCFuiBv9NqwdR8Rew0aKRju//8zsznjdNAoZN827ldV7a637D9Yfth89fvL02cbm8zOWZNSDUy+JEnrhYgZRSOCUhzyCi5QCjt0Izt3ZQe4/vwTKwoSc8JsURjEOSDgJPczV1njjQDhFkCEN3JEwe7uDHcu2u2avb1u2ta0M882WNejLD+OZ8dZwWBaPBZGGE+A4xsqcSTne6CyExkJoLISG1TOL1dlD83U03lz76viJl8VAuBdhxoaWmfKRwJSHXgSy7WQMUuzNcABDZRIcAxuJolBpVLwn1khMEsKBeBWZwDGLMZ9qmznMqrveVCUGWk1bbo5EHsUHFgakqnJj2W47PkzUzReVCd+NMpDi+N2+FGZ3d6tr9QeyhlDwS8Kyza761YGAApASsbe71q6tM2lG0wj+QmaO5dVQIHDlJaozxBfOJXhyqO7HAcIyCvlBhOPGomNJKTV4jipN4W87y85rKco5WRToXIprWcdulrD8oAq60aDbpli3GvapCXP4FDhuqJ430zhrYDONzXSIahCtVwiNOSFlYZQQ7TyTJbqYk4meNFpiyh7nISP1OftYi5hOm/F0Gmrsca0zxzIfl2UC0yDGqs9OkgLFPKH5R3cV8mkUxiFnovRLXRWS/6uUv57sUFaHMv93XXEoNdJzo2Iwq3enT6hH/SqXn7IBC2gVmzeuAUxrYHnBOaneu8WjZvzbOOv3LGV/3O7s7Zcv3zp6iV6h18hCA7SH3qMjdIo89AX9RL/RXetz61vre+vHHF1dKTUvUGW1fv0B0Hc52A==</latexit><latexit sha1_base64="uhIbF8M0t4WkHuwHZFnMrV1hqBU=">AAAGF3icfZTNbtNAEMe30EAJXy0cuVjkwiGK7LQ05oBUtarghErVLymOorU9cays12Z33a/VPgiPwDsgcUPcEDf6NqxdR8RZw0aKRju//8zsznj9jMRc2PbNyp27q61799cetB8+evzk6frGsxOe5iyA4yAlKTvzMQcSUzgWsSBwljHAiU/g1J/tFf7Tc2A8TumRuMpglOCIxpM4wEJvjdf3pFcGGbLIH0m7tz147bhu1+71Xcd1trRhv9l0Bn31YTyz3loez5OxpMryIpwkWJszpcbrnbnQmgutudByena5OqhaB+ON1a9emAZ5AlQEBHM+dOxMjCRmIg4IqLaXc8hwMMMRDLVJcQJ8JMtClVXzHjkjOUmpABrUZBInPMFiamwWMK/vBlOdGFg9bbU5kkWUEHgc0brKT1S77YUw0TdfViZDn+Sg5OG7XSXt7vZm1+kP1BLCIKwIx7W7+rcMRAyAVoi71XW2XZPJcpYR+AvZBVZUw4DCRZDqztBQeucQqKG+Hw8ozxkUB5Gen8iOo5Qy4FtUa0p/21t0XipZzcm8QO9cXqpl7GoBKw6qoSsDum6KdW1gn5owT0xB4IbqRTON8wY2N9jchJgBseUKoTEnZDwmKTXOM1mgyzmZmEnJAlP1uAhJ9OccYiNiNm3Gs2lssIdLnTlUxbgsEphFCdZ99tIMGBYpKz66i1hMSZzEgsvKr0xVTP+v0v7lZPuqPpTFv+/LfWWQgU/KwazfnTmhAQvrXHHKBixidey2cQ1gtgRWF1yQ+r2bP2rWv42Tfs/R9setzs5u9fKtoRfoJXqFHDRAO+g9OkDHKEBf0E/0G920Pre+tb63ftyid1YqzXNUW61ffwB+qDmY</latexit>

µk =1Nk

X

n

�nkxn<latexit sha1_base64="wLZYBBbHUes2ZdhmkUiMsX4ibyA=">AAAGCnicfZTNbtQwEIDd0oWy/LVw5LJiLxxWVVKqbS9IVVEFJ1Sq/knNauU4s1lrEyfYzvbH8hvwCDwFNwQnxA1egLfBzkZiE0c4UjSa+cYz4xk7zBMqpOf9WVm9s9a5e2/9fvfBw0ePn2xsPj0TWcEJnJIsyfhFiAUklMGppDKBi5wDTsMEzsPZG2s/nwMXNGMn8iaHUYpjRieUYGlU441hMFdBWujxrPe6F0w4JsrX6v14pnuBKNKxYkaIcZpiI1rlXF3rsXHse1teuXqu4FdCfx8t1tF4c+17EGWkSIFJkmAhLn0vlyOFuaQkAd0NCgE5JjMcw6URGU5BjFRZoO7VrCf+SE0yJoGRmpvCqUixnDpKC4u6lkxNYOD1sJVypOwuEQgas7pXmOpuN4hgYg67zExFYVKAVsdvD7TyBsNXA397VzcQDlFF+HvewHxNIOYArEL2dgb+cM9l8oLnCfyDPIvZbDgwuCKZaRCLVDAHoi/N+QTARMHBFqKCMFV9X2vtwAvU+JT2brBsvNZKBbUEy9Y3sZslzBZqoBsHum3b69bBPrZhgZyCxC3Zy3YaFy1s4bCFC3EH4s0MoTUm5IImGXPqmSzR5ZxM3KDJElP12G6ZmBscYWfHfNqO51PqsMeNzhxrOy7LBOZxik2fgywHjmXG7aW7onKa0JRKoSq7dr0o+7+XsTeDHer6UNp/GKpD7ZAkTMrBrJ+dO6GER3XOVtmCxbyOLRrXAuYNsDpgS5r3zm++bq5wtr3lG/nDTn//oHr51tFz9AK9RD7aRfvoHTpCp4igz+gH+oV+dz51vnS+dr4t0NWVyucZqq3Oz78GQDi+</latexit><latexit sha1_base64="wLZYBBbHUes2ZdhmkUiMsX4ibyA=">AAAGCnicfZTNbtQwEIDd0oWy/LVw5LJiLxxWVVKqbS9IVVEFJ1Sq/knNauU4s1lrEyfYzvbH8hvwCDwFNwQnxA1egLfBzkZiE0c4UjSa+cYz4xk7zBMqpOf9WVm9s9a5e2/9fvfBw0ePn2xsPj0TWcEJnJIsyfhFiAUklMGppDKBi5wDTsMEzsPZG2s/nwMXNGMn8iaHUYpjRieUYGlU441hMFdBWujxrPe6F0w4JsrX6v14pnuBKNKxYkaIcZpiI1rlXF3rsXHse1teuXqu4FdCfx8t1tF4c+17EGWkSIFJkmAhLn0vlyOFuaQkAd0NCgE5JjMcw6URGU5BjFRZoO7VrCf+SE0yJoGRmpvCqUixnDpKC4u6lkxNYOD1sJVypOwuEQgas7pXmOpuN4hgYg67zExFYVKAVsdvD7TyBsNXA397VzcQDlFF+HvewHxNIOYArEL2dgb+cM9l8oLnCfyDPIvZbDgwuCKZaRCLVDAHoi/N+QTARMHBFqKCMFV9X2vtwAvU+JT2brBsvNZKBbUEy9Y3sZslzBZqoBsHum3b69bBPrZhgZyCxC3Zy3YaFy1s4bCFC3EH4s0MoTUm5IImGXPqmSzR5ZxM3KDJElP12G6ZmBscYWfHfNqO51PqsMeNzhxrOy7LBOZxik2fgywHjmXG7aW7onKa0JRKoSq7dr0o+7+XsTeDHer6UNp/GKpD7ZAkTMrBrJ+dO6GER3XOVtmCxbyOLRrXAuYNsDpgS5r3zm++bq5wtr3lG/nDTn//oHr51tFz9AK9RD7aRfvoHTpCp4igz+gH+oV+dz51vnS+dr4t0NWVyucZqq3Oz78GQDi+</latexit><latexit sha1_base64="wLZYBBbHUes2ZdhmkUiMsX4ibyA=">AAAGCnicfZTNbtQwEIDd0oWy/LVw5LJiLxxWVVKqbS9IVVEFJ1Sq/knNauU4s1lrEyfYzvbH8hvwCDwFNwQnxA1egLfBzkZiE0c4UjSa+cYz4xk7zBMqpOf9WVm9s9a5e2/9fvfBw0ePn2xsPj0TWcEJnJIsyfhFiAUklMGppDKBi5wDTsMEzsPZG2s/nwMXNGMn8iaHUYpjRieUYGlU441hMFdBWujxrPe6F0w4JsrX6v14pnuBKNKxYkaIcZpiI1rlXF3rsXHse1teuXqu4FdCfx8t1tF4c+17EGWkSIFJkmAhLn0vlyOFuaQkAd0NCgE5JjMcw6URGU5BjFRZoO7VrCf+SE0yJoGRmpvCqUixnDpKC4u6lkxNYOD1sJVypOwuEQgas7pXmOpuN4hgYg67zExFYVKAVsdvD7TyBsNXA397VzcQDlFF+HvewHxNIOYArEL2dgb+cM9l8oLnCfyDPIvZbDgwuCKZaRCLVDAHoi/N+QTARMHBFqKCMFV9X2vtwAvU+JT2brBsvNZKBbUEy9Y3sZslzBZqoBsHum3b69bBPrZhgZyCxC3Zy3YaFy1s4bCFC3EH4s0MoTUm5IImGXPqmSzR5ZxM3KDJElP12G6ZmBscYWfHfNqO51PqsMeNzhxrOy7LBOZxik2fgywHjmXG7aW7onKa0JRKoSq7dr0o+7+XsTeDHer6UNp/GKpD7ZAkTMrBrJ+dO6GER3XOVtmCxbyOLRrXAuYNsDpgS5r3zm++bq5wtr3lG/nDTn//oHr51tFz9AK9RD7aRfvoHTpCp4igz+gH+oV+dz51vnS+dr4t0NWVyucZqq3Oz78GQDi+</latexit><latexit sha1_base64="iUdAi1B2oPRFFR3Sl7J0S6e2ggw=">AAAGCnicfZTNbtQwEIDd0oWy/LVw5LJiLxxWVVKqthekqqiCEypV/6R6tXKc2V1rEyfYzvbH8hvwCDwFNwQnxA1egLfBTiOxiSMcKRrNfOOZ8Ywd5QmTKgj+LC3fWencvbd6v/vg4aPHT9bWn57KrBAUTmiWZOI8IhISxuFEMZXAeS6ApFECZ9HsjbOfzUFIlvFjdZ3DMCUTzsaMEmVVo7VtPNc4Lcxo1nvdw2NBqA6Nfj+amR6WRTrS3AoTkqbEik4511dmZB37wUZQrp4vhJXQR9U6HK2vfMdxRosUuKIJkfIiDHI11EQoRhMwXVxIyAmdkQlcWJGTFORQlwWaXs16HA71OOMKOK25aZLKlKipp3SwrGvp1AYGUQ9bKYfa7RKDZBNe94pS0+3iGMb2sMvMdBwlBRh99Hbf6GCw/WoQbu6YBiIgrohwNxjYrwlMBACvkN2tQbi96zN5IfIE/kGBw1w2Ajhc0sw2iMcaz4GaC3s+GLgsBLhCNI5S3Q+NMR58i1qf0t7Fi8YrozWuJVi2voldL2CuUAtde9BN2143HvaxDcNqCoq0ZK/aaVK0sIXHFj4kPEg0M4TWmJBLlmTcq2e8QJdzMvaDJgtM1WO3ZWJvcEy8HfNpO55PmcceNTpzZNy4LBJETFJi+4yzHARRmXCX7pKpacJSpqSu7Mb3Yvz/XtbeDHZg6kPp/lGkD4xH0igpB7N+dv6EUhHXOVdlCzYRdey2cS1g3gCrA3akfe/C5uvmC6ebG6GVP2z19/arl28VPUcv0EsUoh20h96hQ3SCKPqMfqBf6HfnU+dL52vn2y26vFT5PEO11fn5F7RiOH4=</latexit>

⇡k =Nk

N<latexit sha1_base64="4s/F4xsMBYlMi+mf8agPCZwO+oo=">AAAF6XicfZRLbxMxEIDd0kAJrxTEiUtELhyiardUbS5IVVEFp6pUfUl1FHm9k8SK12tsb1+WfwQ3xA1xg7/AH+HfsJusRHa9wiutRjPfvOyxI8mZNkHwZ2X13lrr/oP1h+1Hj588fdbZeH6m00xROKUpT9VFRDRwJuDUMMPhQiogScThPJq9L+znV6A0S8WJuZUwTMhEsDGjxOSqUecllmw0677r4rEi1B6OZs4eulGnF2wG89X1hbAUentosY5GG2u/cZzSLAFhKCdaX4aBNENLlGGUg2vjTIMkdEYmcJmLgiSgh3Zev+tWrCfh0I5TYUDQipsliU6ImXrKAtZVLZ3miUFV05bKoS2ixKDZRFS9osS12ziGcb6X88psHPEMnD3+sO9s0N952w+3dl0NURCXRDgI+vlXByYKQJTIYLsf7gx8RmZKcvgHBQVWVKNAwDVNk4SI2OIroO4y3x8MQmcKikYsjhLbC51zHrxAc5+5vY2XjTfOWlwpEF/ZG1fHbpewotEcuvWgu6ZYdx72uQnDZgqGNFRvmmmSNbCZx2Y+pDxI1SuExpwgNeOp8PoZL9HzORn7SfkSU55xEZLnFzQmXkQ5bcbllHnsce1kjl0xLssEUZOE5OeMUwmKmFQVl+6amSlnCTPalnbnezHxf6/cXk924KpDWfyjyB44j6QRnw9mde/8CaUqrnJFlw3YRFWxxcE1gLIGlhtckPl7F9ZfN18429oMc/nTdm9vv3z51tEr9Bq9QSHaRXvoIzpCp4gii76jn+hXa9b60vra+rZAV1dKnxeoslo//gJT2CpK</latexit><latexit sha1_base64="4s/F4xsMBYlMi+mf8agPCZwO+oo=">AAAF6XicfZRLbxMxEIDd0kAJrxTEiUtELhyiardUbS5IVVEFp6pUfUl1FHm9k8SK12tsb1+WfwQ3xA1xg7/AH+HfsJusRHa9wiutRjPfvOyxI8mZNkHwZ2X13lrr/oP1h+1Hj588fdbZeH6m00xROKUpT9VFRDRwJuDUMMPhQiogScThPJq9L+znV6A0S8WJuZUwTMhEsDGjxOSqUecllmw0677r4rEi1B6OZs4eulGnF2wG89X1hbAUentosY5GG2u/cZzSLAFhKCdaX4aBNENLlGGUg2vjTIMkdEYmcJmLgiSgh3Zev+tWrCfh0I5TYUDQipsliU6ImXrKAtZVLZ3miUFV05bKoS2ixKDZRFS9osS12ziGcb6X88psHPEMnD3+sO9s0N952w+3dl0NURCXRDgI+vlXByYKQJTIYLsf7gx8RmZKcvgHBQVWVKNAwDVNk4SI2OIroO4y3x8MQmcKikYsjhLbC51zHrxAc5+5vY2XjTfOWlwpEF/ZG1fHbpewotEcuvWgu6ZYdx72uQnDZgqGNFRvmmmSNbCZx2Y+pDxI1SuExpwgNeOp8PoZL9HzORn7SfkSU55xEZLnFzQmXkQ5bcbllHnsce1kjl0xLssEUZOE5OeMUwmKmFQVl+6amSlnCTPalnbnezHxf6/cXk924KpDWfyjyB44j6QRnw9mde/8CaUqrnJFlw3YRFWxxcE1gLIGlhtckPl7F9ZfN18429oMc/nTdm9vv3z51tEr9Bq9QSHaRXvoIzpCp4gii76jn+hXa9b60vra+rZAV1dKnxeoslo//gJT2CpK</latexit><latexit sha1_base64="4s/F4xsMBYlMi+mf8agPCZwO+oo=">AAAF6XicfZRLbxMxEIDd0kAJrxTEiUtELhyiardUbS5IVVEFp6pUfUl1FHm9k8SK12tsb1+WfwQ3xA1xg7/AH+HfsJusRHa9wiutRjPfvOyxI8mZNkHwZ2X13lrr/oP1h+1Hj588fdbZeH6m00xROKUpT9VFRDRwJuDUMMPhQiogScThPJq9L+znV6A0S8WJuZUwTMhEsDGjxOSqUecllmw0677r4rEi1B6OZs4eulGnF2wG89X1hbAUentosY5GG2u/cZzSLAFhKCdaX4aBNENLlGGUg2vjTIMkdEYmcJmLgiSgh3Zev+tWrCfh0I5TYUDQipsliU6ImXrKAtZVLZ3miUFV05bKoS2ixKDZRFS9osS12ziGcb6X88psHPEMnD3+sO9s0N952w+3dl0NURCXRDgI+vlXByYKQJTIYLsf7gx8RmZKcvgHBQVWVKNAwDVNk4SI2OIroO4y3x8MQmcKikYsjhLbC51zHrxAc5+5vY2XjTfOWlwpEF/ZG1fHbpewotEcuvWgu6ZYdx72uQnDZgqGNFRvmmmSNbCZx2Y+pDxI1SuExpwgNeOp8PoZL9HzORn7SfkSU55xEZLnFzQmXkQ5bcbllHnsce1kjl0xLssEUZOE5OeMUwmKmFQVl+6amSlnCTPalnbnezHxf6/cXk924KpDWfyjyB44j6QRnw9mde/8CaUqrnJFlw3YRFWxxcE1gLIGlhtckPl7F9ZfN18429oMc/nTdm9vv3z51tEr9Bq9QSHaRXvoIzpCp4gii76jn+hXa9b60vra+rZAV1dKnxeoslo//gJT2CpK</latexit><latexit sha1_base64="/a7eQ6x7KnFFyD0d9IcX9lnNdw0=">AAAF6XicfZRPb9MwFMC9scIo/zYQJy4VvXCopmRM2y5I09AEp2lU6zZprirHeW2tOk6wnf2z/CG4IW6IG3wFvgjfhriNRBNHOFL09N7v/bOfHWWcKR0Ef1ZW76217j9Yf9h+9PjJ02cbm8/PVJpLCgOa8lReREQBZwIGmmkOF5kEkkQczqPZe2c/vwKpWCpO9W0Gw4RMBBszSnShGm28xBkbzTrvOngsCTXHo5k1x3a00Q22gvnq+EJYCl1UrpPR5tpvHKc0T0BoyolSl2GQ6aEhUjPKwbZxriAjdEYmcFmIgiSghmZev+1UrKfh0IxToUHQipshiUqInnpKB6uqlk6LxCCraUvl0LgoMSg2EVWvKLHtNo5hXOzlvDITRzwHa/ofDq0Jertve+H2nq0hEuKSCPeDXvHVgYkEECWyv9MLd/d9JstlxuEfFDjMVSNBwDVNk4SI2OAroPay2B8MQuUSXCMGR4nphtZaD16ghc/c3sbLxhtrDK4UiK/Mja1jt0uYa7SAbj3orinWnYd9bsKwnoImDdXrZprkDWzusbkPSQ+S9QqhMSdkivFUeP2Ml+j5nIz9pHyJKc/YheTFBY2JFzGbNuPZlHlsv3YyfevGZZkgcpKQ4pxxmoEkOpXu0l0zPeUsYVqZ0m59Lyb+71XY68mObHUo3T+KzJH1SBrx+WBW986fUCrjKue6bMAmsootDq4BzGpgucGOLN67sP66+cLZ9lZYyJ92ugeH5cu3jl6h1+gNCtEeOkAf0QkaIIoM+o5+ol+tWetL62vr2wJdXSl9XqDKav34CwIJKgo=</latexit>

Cluster Mean

Cluster Covariance

Fraction of pointsin each cluster

Idea: Use weights γnk = p(zn=k | xn) to compute estimates

⌃k =1Nk

X

n

�nk(xn �µk)(xn �µk)><latexit sha1_base64="m+VqJWgg9/bBz5V+ool4bizAanE=">AAAGWHicfZRLb9QwEICzpd22y2tbTsAlYi9FWlBSqrYXpKqoghMqpS9pvUSOM5u1NnGC4/Rl+cbP4o/AT+BX4GRTsYkjfIhGM9887JmMn0Y0E47zq7P0YHmlu7q23nv46PGTp/2NzfMsyTmBM5JECb/0cQYRZXAmqIjgMuWAYz+CC3/2obBfXAHPaMJOxW0K4xiHjE4owUKrvP4PicogIx76Y+m8dcozNASFriT6SsMYK29mv7fRhGMiXSU/ezNloyyPPcm0EOI4xlrUyi3tcqM8Zr+xUZx7s9em5hsSSWorrz+4z2ObglsJgwNrfo69jeWfKEhIHgMTJMJZNnKdVIwl5oKSCFQP5RmkmMxwCCMtMhxDNpblTZVds566YzlJmABGam4Sx1mMxdRQFnBW15KpTgy8nrZSjmURJYCMhqzu5ceq10MBTHTryspk4Ec5KHny8VBJZ7j7buhu76kGwiGoCHdft2foNIGQA7AK2d8Zurv7JpPmPI3gH+QUWFENBwbXJNE9ZIFEV0DUSL8PApblHIqLSOTHcuAqpQx4jmqf0t5Di8YbJatBuy+wHIUmdruAFRfV0K0B3bXFujOw720YElMQuKV60U7jvIXNDTY3IW5AvFkhtOaENKNRwoz7TBbock4mZtJogal6XISM9D4IsBExnbbj6ZQa7EmjMyeqGJdFAnO9HHSfUZICxyLhxU93TcU0ojEVmazsyvSi7P9e2t5MdqTqQ1l8fV8eKYMkflQOZv3tzAklPKhzxS1bsJDXsXnjWsC0AVYPXJB637nN7WYK59tvXS1/2RkcHFabb816ab2ytizX2rMOrE/WsXVmEetPp9953nmx8rtrdVe763N0qVP5PLNqp7v5F1y6S/g=</latexit><latexit sha1_base64="m+VqJWgg9/bBz5V+ool4bizAanE=">AAAGWHicfZRLb9QwEICzpd22y2tbTsAlYi9FWlBSqrYXpKqoghMqpS9pvUSOM5u1NnGC4/Rl+cbP4o/AT+BX4GRTsYkjfIhGM9887JmMn0Y0E47zq7P0YHmlu7q23nv46PGTp/2NzfMsyTmBM5JECb/0cQYRZXAmqIjgMuWAYz+CC3/2obBfXAHPaMJOxW0K4xiHjE4owUKrvP4PicogIx76Y+m8dcozNASFriT6SsMYK29mv7fRhGMiXSU/ezNloyyPPcm0EOI4xlrUyi3tcqM8Zr+xUZx7s9em5hsSSWorrz+4z2ObglsJgwNrfo69jeWfKEhIHgMTJMJZNnKdVIwl5oKSCFQP5RmkmMxwCCMtMhxDNpblTZVds566YzlJmABGam4Sx1mMxdRQFnBW15KpTgy8nrZSjmURJYCMhqzu5ceq10MBTHTryspk4Ec5KHny8VBJZ7j7buhu76kGwiGoCHdft2foNIGQA7AK2d8Zurv7JpPmPI3gH+QUWFENBwbXJNE9ZIFEV0DUSL8PApblHIqLSOTHcuAqpQx4jmqf0t5Di8YbJatBuy+wHIUmdruAFRfV0K0B3bXFujOw720YElMQuKV60U7jvIXNDTY3IW5AvFkhtOaENKNRwoz7TBbock4mZtJogal6XISM9D4IsBExnbbj6ZQa7EmjMyeqGJdFAnO9HHSfUZICxyLhxU93TcU0ojEVmazsyvSi7P9e2t5MdqTqQ1l8fV8eKYMkflQOZv3tzAklPKhzxS1bsJDXsXnjWsC0AVYPXJB637nN7WYK59tvXS1/2RkcHFabb816ab2ytizX2rMOrE/WsXVmEetPp9953nmx8rtrdVe763N0qVP5PLNqp7v5F1y6S/g=</latexit><latexit sha1_base64="m+VqJWgg9/bBz5V+ool4bizAanE=">AAAGWHicfZRLb9QwEICzpd22y2tbTsAlYi9FWlBSqrYXpKqoghMqpS9pvUSOM5u1NnGC4/Rl+cbP4o/AT+BX4GRTsYkjfIhGM9887JmMn0Y0E47zq7P0YHmlu7q23nv46PGTp/2NzfMsyTmBM5JECb/0cQYRZXAmqIjgMuWAYz+CC3/2obBfXAHPaMJOxW0K4xiHjE4owUKrvP4PicogIx76Y+m8dcozNASFriT6SsMYK29mv7fRhGMiXSU/ezNloyyPPcm0EOI4xlrUyi3tcqM8Zr+xUZx7s9em5hsSSWorrz+4z2ObglsJgwNrfo69jeWfKEhIHgMTJMJZNnKdVIwl5oKSCFQP5RmkmMxwCCMtMhxDNpblTZVds566YzlJmABGam4Sx1mMxdRQFnBW15KpTgy8nrZSjmURJYCMhqzu5ceq10MBTHTryspk4Ec5KHny8VBJZ7j7buhu76kGwiGoCHdft2foNIGQA7AK2d8Zurv7JpPmPI3gH+QUWFENBwbXJNE9ZIFEV0DUSL8PApblHIqLSOTHcuAqpQx4jmqf0t5Di8YbJatBuy+wHIUmdruAFRfV0K0B3bXFujOw720YElMQuKV60U7jvIXNDTY3IW5AvFkhtOaENKNRwoz7TBbock4mZtJogal6XISM9D4IsBExnbbj6ZQa7EmjMyeqGJdFAnO9HHSfUZICxyLhxU93TcU0ojEVmazsyvSi7P9e2t5MdqTqQ1l8fV8eKYMkflQOZv3tzAklPKhzxS1bsJDXsXnjWsC0AVYPXJB637nN7WYK59tvXS1/2RkcHFabb816ab2ytizX2rMOrE/WsXVmEetPp9953nmx8rtrdVe763N0qVP5PLNqp7v5F1y6S/g=</latexit><latexit sha1_base64="Rmhbw9pD1UBLs2lCJVC602qBVzs=">AAAGWHicfZRLb9QwEICzpd22y6PbcgIuEXsp0lIlpWp7QaqKKjihUvqS1kvkOLPZaBMn2E5flm/8LP4I/AR+BU6aik0c4UM0mvnmYc9k/CyOuHCcX52FR4tL3eWV1d7jJ0+frfXXN855mjMCZySNU3bpYw5xROFMRCKGy4wBTvwYLvzZh8J+cQWMRyk9FbcZjBMc0mgSESy0yuv/kKgMMmKhP5bOllOeoSEodCXR1yhMsPJm9nsbTRgm0lXyszdTNuJ54kmqhRAnCdaiVm5qlxvlUfutjZLcm70xNd+QSDNbef3BQx7bFNxKGFjVOfbWF3+iICV5AlSQGHM+cp1MjCVmIiIxqB7KOWSYzHAIIy1SnAAfy/Kmyq5ZT92xnKRUACU1N4kTnmAxNZQFzOtaMtWJgdXTVsqxLKIEwKOQ1r38RPV6KICJbl1ZmQz8OAclTz4eKukMd98N3e091UAYBBXh7uv2DJ0mEDIAWiH7O0N3d99kspxlMfyDnAIrqmFA4Zqkuoc0kOgKiBrp90FAec6guIhEfiIHrlLKgO9R7VPae2jeeKNkNWgPBZaj0MRu57Diohq6NaC7tlh3Bva9DUNiCgK3VC/aaZy3sLnB5ibEDIg1K4TWnJDxKE6pcZ/JHF3OycRMGs8xVY+LkLHeBwE2ImbTdjybRgZ70ujMiSrGZZ7ATC8H3WeUZsCwSFnx011HYhpHSSS4rOzK9Iro/720vZnsSNWHsvj6vjxSBkn8uBzM+tuZE0pYUOeKW7ZgIatj941rAbMGWD1wQep95za3mymcb2+5Wv6yMzg4rDbfivXKem1tWq61Zx1Yn6xj68wi1p9Ov/Oi83Lpd9fqLndX79GFTuXz3Kqd7sZfCutLuA==</latexit>

Parameter Updates


n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

PNn=1 znk xn

⌃k =1Nk



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

PNn=1 znk xn

⌃k =1Nk


Assignment Update



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk



n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk



Nk :=PN

n=1 �nk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

PNn=1 �nk xn

⌃k =1Nk

PNn=1 �nk (xn �µk)(xn �µk)>

Nk :=PN

n=1 �nk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

PNn=1 �nk xn

⌃k =1Nk


Nk :=PN

n=1 �nk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

PNn=1 �nk xn

⌃k =1Nk


Parameter Updates

Gaussian Mixture Models


n=1 znk

⇡ = (N1/N , . . . , NK/N)

µk =1Nk

Pn=1 znk xn

⌃k =1Nk


Soft Assignment Update


EM for Gaussian Mixtures

Credit: Andrew Moore













Consider Naive Bayes

evaluate the classifier. One good evaluation metric is accuracy, which is defined as

# correctly predicted labels in the test set# of instances in the test set

(6)

2.2 Naive Bayes model

The naive Bayes model defines a joint distribution over words and classes–that is, the prob-ability that a sequence of words belongs to some class c.

p(c, w1:N |⇡, ✓) = p(c|⇡)NY

n=1

p(wn|✓c) (7)

⇡ is the probability distribution over the classes (in this case, the probability of seeing spame-mails and of seeing ham e-mails). It’s basically a discrete probability distribution overthe classes that sums to 1. ✓c is the class conditional probability distribution–that is, givena certain class, the probability distribution of the words in your vocabulary. Each class hasa probability for each word in the vocabulary (in this case, there is a set of probabilities forthe spam class and one for the ham class).Given a new test example, we can classify it using the model by calculating the conditionalprobability of the classes given the words in the e-mail p(c | wn) and see which class ismost likely. There is an implicit independence assumption behind the model. Since we’remultiplying the p(wn | ✓c) terms together, it is assumed that the words in a document areconditionally independent given the class.

2.3 Class prediction with naive Bayes

We classify using the posterior distribution of the classes given the words, which is propor-tional to the joint distribution since P (X|Y ) = P(X, Y)/P(Y) / P(X, Y). The posteriordistribution is

p(c|w1:N , ⇡, ✓) / p(c|⇡)NY

n=1

p(wn|✓c) (8)

Note that we don’t need a normalizing constant because we only care about which proba-bility is bigger. The classifier assigns the e-mail to the class which has highest probability.

2.4 Fitting a naive Bayes model with maximum likelihood

We find the parameters in the posterior distribution used in class prediction by learningon the labeled training data (in this case, the ham and spam e-mails). We use the maxi-mum likelihood method method in finding parameters that maximize the likelihood of theobserved data set. Given data {wd,1:N , cd}D

d=1, the likelihood under a certain model withparameters (✓1:C , ⇡) is

p(D|✓1:C , ⇡) =DY

d=1

p(cd|⇡)NY

n=1

p(wn|✓cd)!

3

evaluate the classifier. One good evaluation metric is accuracy, which is defined as

# correctly predicted labels in the test set# of instances in the test set

(6)

2.2 Naive Bayes model

The naive Bayes model defines a joint distribution over words and classes–that is, the prob-ability that a sequence of words belongs to some class c.

p(c, w1:N |⇡, ✓) = p(c|⇡)NY

n=1

p(wn|✓c) (7)

⇡ is the probability distribution over the classes (in this case, the probability of seeing spame-mails and of seeing ham e-mails). It’s basically a discrete probability distribution overthe classes that sums to 1. ✓c is the class conditional probability distribution–that is, givena certain class, the probability distribution of the words in your vocabulary. Each class hasa probability for each word in the vocabulary (in this case, there is a set of probabilities forthe spam class and one for the ham class).Given a new test example, we can classify it using the model by calculating the conditionalprobability of the classes given the words in the e-mail p(c | wn) and see which class ismost likely. There is an implicit independence assumption behind the model. Since we’remultiplying the p(wn | ✓c) terms together, it is assumed that the words in a document areconditionally independent given the class.

2.3 Class prediction with naive Bayes

We classify using the posterior distribution of the classes given the words, which is propor-tional to the joint distribution since P (X|Y ) = P(X, Y)/P(Y) / P(X, Y). The posteriordistribution is

p(c|w1:N , ⇡, ✓) / p(c|⇡)NY

n=1

p(wn|✓c) (8)

Note that we don’t need a normalizing constant because we only care about which proba-bility is bigger. The classifier assigns the e-mail to the class which has highest probability.

2.4 Fitting a naive Bayes model with maximum likelihood

We find the parameters in the posterior distribution used in class prediction by learningon the labeled training data (in this case, the ham and spam e-mails). We use the maxi-mum likelihood method method in finding parameters that maximize the likelihood of theobserved data set. Given data {wd,1:N , cd}D

d=1, the likelihood under a certain model withparameters (✓1:C , ⇡) is

p(D|✓1:C , ⇡) =DY

d=1

p(cd|⇡)NY

n=1

p(wn|✓cd)!

3

The model

In-class exercise: How would we use EM here?




In-class exercise


Let’s review (on board)

Summing up• Mixture models can be used to perform probabilistic clustering

• General idea: Assume instances are generated from distinct components. Each component has its own model parameters.

• Fitting: More difficult here than in standard supervised learning because we do not observe z. (One) Solution: Expectation-Maximization.







Date post:	31-Aug-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

Machine Learning 2 - Northeastern UniversityPioter Piotr Substitution (i for e) Insertion (o)...

Documents