Lecture 1 graphical models

04/10/2023 1

Lecture 1: Probabilistic Graphical ModelsPhạm Duy Tùng

Email: [email protected]

9/9/2012Some slides copied from Pattern Recognition and Machine Learning (Bishop 2006)

Machine Learning

04/10/2023 2

Introduction

• Probabilistic graphical models • A visual presentation of probability distributions, using

diagrams, called PGMs

• Two types of PGM• Bayesian Network (Directed Graphical Model)• Markov Network (Undirected Graphical Model)

04/10/2023 3

Introduction

• Offering several useful properties:• They provide a simply way to visualize the

structure of probabilistic models and motivate new models.

• Insights into the properties of the models, including the conditional independence properties, can be obtain by inspection of the graph.

• Graph based algorithms for calculation and computation

04/10/2023 4

Topics

• Introduction• Representation

• Bayesian Networks (Directed Graphical Model)• Markov Networks (Undirected Graphical Model)• Converting Bayesian Networks to Markov Networks• Directed vs. Undirected Graphs

• Some examples• Naïve Bayes Classifier• Ising Model

• Inference• Learning

04/10/2023 5

Representation

• Random variables -> Nodes• In Conditional Independences -> Edges

(Directed or Undirected)

Probability

Models

Graphical Models

04/10/2023 6

Topics





04/10/2023 7

Representation->Bayesian Network

• Using Directed Acyclic Graph (DAG)

• Joint distribution factorizes according to graph

04/10/2023 8


04/10/2023 9


• Conditional independence:

Or equivalently:

Denoted by:

04/10/2023 10


• Conditional independence: Example 1

04/10/2023 11



04/10/2023 12



04/10/2023 13



04/10/2023 14



04/10/2023 15



04/10/2023 16

Representation->Bayesian Network• D-separation

• A, B, and C are non-intersecting subsets of nodes in a directed graph.

• A path from A to B is blocked if it contains a node such that eithera)the arrows on the path meet either head-to-tail or tail-to-tail

at the node, and the node is in the set C, orb)the arrows meet head-to-head at the node, and neither the

node, nor any of its descendants, are in the set C.• If all paths from A to B are blocked, A is said to be d-separated

from B by C. • If A is d-separated from B by C, the joint distribution over all

variables in the graph satisfies .

04/10/2023 17


• D-separation: Example

04/10/2023 18


• Markov Blanket

Factors independent of xi cancel between numerator and denominator.

04/10/2023 19

Example

• Mixture of Gaussians

04/10/2023 20

Home Work

• LDA model (David M.Blei)

04/10/2023 21

Topics





04/10/2023 22

Representation->Markov Network

• Many phenomenon in real life, we can not determine exactly the directionality to the interaction between random variables.

• We use Markov Network to modeling these phenomenon instead of Bayesian Network.

04/10/2023 23


Markov Blanket

• Conditional independence

04/10/2023 24


• Factorization properties

04/10/2023 25


Clique

Maximal Clique

• Clique

04/10/2023 26


• where is the potential over clique C and

is the normalization coefficient; note: M K-state variables KM terms in Z.

• Energies and the Boltzmann distribution

04/10/2023 27

Representation -> Converting Bayesian Networks to Markov Networks

04/10/2023 28

Representation -> Converting Bayesian Networks to Markov Networks

• Additional links

04/10/2023 29

Directed vs. Undirected Graphs

Directed vs. Undirected Graphs

04/10/2023 31

Topics• Introduction• Representation




04/10/2023 32

Naïve Bayes Classifier

• Predicting

04/10/2023 33


• How to estimate and ?• Solution: We can separately estimate two

parts of the model using Maximum Likelihood

04/10/2023 34


04/10/2023 35


• Estimating • Y is discrete

Assume that Y = Where yk {0, 1} and ∈

Then we have: p(y) = log p(y)=log

04/10/2023 36


• Likelihood Function =

=• Maximum Likelihood Function

subject to Where

04/10/2023 37


• Solution for MLE problem• Using Lagrange Multiplier• Maximizing ()

• Sum over k two sides: • Then we have:

04/10/2023 38


• Estimating • Maximizing Likelihood function

04/10/2023 39


• Solution:• Assume that is discrete and similar to Y =

Where xlm {0, 1} and ∈

• p() = where • logp() = log

04/10/2023 40


• We have =• Our problem:

subject to: for m=1, …, MWhere =

04/10/2023 41


• Using Lagrange Multiplier again

• We have: =N=

04/10/2023 42


• Summary:– We have training data set (, )– Then we can estimate (Y) and (X|Y)

=

– Have new x, we can predict value of y

04/10/2023 43


• Home works:– Estimating when Y is discrete and – Design a online estimating algorithm: =f(, )– Building a recommender system using Naïve Bayes

Classifier

04/10/2023 44

Ising Model and Image de-noising

• Markov Random Field

04/10/2023 45

Ising Model and Image de-noising• An example of Ising Model used in Image

processing.

Original Image Noisy Image

04/10/2023 46


• Binary Images: X = (, …, ), Y = (, …, ) where {-1, +1}

• X is original image and not observed.• Y is noise-image and observed.• Suppose that the noise level is low, so there

will be a strong correlation between X and Y.• And neighbouring pixels are highly correlated.

04/10/2023 47


• All of our assumptions about images are encoded as follows:

04/10/2023 48


• The joint distribution of X, Y P(X, Y) = (*)Where: =exp{-}, (**)=- , (>0)

=exp{-}, (***)=- , (>0)

04/10/2023 49


• From (*) (**) (***) we have:P(X, Y) = exp(-E(X, Y))Where energy function E(X, Y) = -

• Using Bayes Rule: P(X|Y) • We wish to find a image X that has a high

probability

04/10/2023 50


• Solving the Maximizing Problem

=

• Some algorithms:– Iterated Conditional Models (Kittler and Foglein,

1984)– Graph cuts (Greig et al., 1989; Boykov et al., 2001;

Kolmogorov and Zabih,2004)

04/10/2023 51


• Results

Restored Image (ICM)Noisy Image

04/10/2023 52


• Comparing two optimizing algorithms: Graph cuts vs ICM

Restored Image (Graph cuts)Restored Image (ICM)

04/10/2023 53


• Home works– Design and implement an image segmentation

algorithms using Ising model.

04/10/2023 54

Topics• Introduction• Representation




04/10/2023 55

Inference

04/10/2023 56

Learning

Date post:	12-Dec-2014
Category:	Technology
Upload:	duy-tung-pham
View:	215 times
Download:	2 times

Lecture 1 graphical models

Technology