Belief Propagation and its Generalizations

Shane Oldenburger

Outline

The BP algorithm

MRFs – Markov Random Fields

Gibbs free energy

Bethe approximation

Kikuchi approximation

Generalized BP

Outline

The BP algorithm

MRFs – Markov Random Fields

Gibbs free energy

Bethe approximation

Kikuchi approximation

Generalized BP

Recall from the Jointree Algorithm

We separate evidence e into:

e+: denotes evidence pertaining to ancestors

e-: denotes evidence pertaining to descendants

BEL(X) = P(X|e) = P(X|e+,e-) = P(e-|X,e+)*P(X|e+)/P(e-|e+) = P(e-|X)P(X|e+) = *(X)*(X): messages from parents: messages from children: normalization constant

Pearl’s Belief Propagation Algorithm:Initialization

Nodes with evidence (xi) = 1 where xi = ei; 0 otherwise

(xi) = 1 where xi = ei; 0 otherwise

Nodes with no parents (xi) = p(xi) //prior probabilities

Nodes with no children (xi) = 1

Pearl’s BP algorithmIterate

For each X: If all messages from parents of X have arrived, combine

into (X) If all messages from children of X have arrived, combine

into (X) If (X) has been computed and all messages other than

from Yi have arrived, calculate and send message XYi to child Yi

If (X) has been computed and all messages other than from Ui have arrived, calculate and send message XUi to parent Ui

Compute BEL(X) = *(X)*(X)

Example of data propagation in a simple tree

BP propertiesExact for Polytrees Only one path between any two nodes Each node X separates graph into two disjoint

graphs (e+, e-)

But most graphs of interest are not Polytrees – what do we do? Exact inference

Cutset conditioning Jointree method

Approximate inference Loopy BP

Loopy BP

In the simple tree example, a finite number of messages where passed

In a graph with loops, messages may be passed around indefinitely Stop when beliefs converge Stop after some number of iterations

Loopy BP tends to achieve good empirical results Low-level computer vision problems Error-correcting codes: Turbocodes, Gallager codes

Outline

The BP algorithm

MRFs – Markov Random Fields

Gibbs free energy

Bethe approximation

Kikuchi approximation

Generalized BP

Markov Random Fields

BP algorithms have been developed for many graphical modelsPairwise Markov Random Fields are used in this paper for ease of presentationAn MRF consists of “observable” nodes and “hidden” nodes Since it is pairwise, each observable node is

connected to exactly one hidden node, and each hidden is connected to at most one observable node

Markov Random Fields

Two hidden variables xi and xj are connected by a “compatibility function” ij(xi, yi)Hidden variable xi is connected to observable variable yi by “evidence function” i(xi, yi) = xi(xi)The joint probability for a pairwise MRF isp({x}) = (1/Z) ijij(xi, yi) ixi(xi)

The BP algorithm for pairwise MRFs is similar to that for Bayesian Networks

Conversion between graphical models

We can limit ourselves to considering pairwise MRFs

Any pairwise MRF or BN can be converted to an equivalent “Factor graph”

Any factor graph can be converted into an equivalent pairwise MRF or BN

An intermediary model

A factor graph is composed of “variable” nodes represented by circles “function” nodes represented by squares

Factor graphs are a generalization of Tanner graphs, where the “function” nodes are parity checks of its connected variablesA function node for a factor graph can be any arbitrary function of the variables connected to it

From pairwise MRF to BN

From BN to pairwise MRF

Outline

The BP algorithm

MRFs – Markov Random Fields

Gibbs free energy

Bethe approximation

Kikuchi approximation

Generalized BP

Gibbs Free Energy

Gibbs free energy is the difference in the energy of a system from an initial state to a final state of some process (e.g. chemical reaction)

For a chemical reaction, if the Gibbs free energy is negative then the reaction is “spontaneous”, or “allowed”

If the Gibbs free energy is non-negative, the reaction is “not allowed”

Gibbs free energy

Instead of difference in energy of a chemical process, we want to define Gibbs free energy in term of the difference between a target probability distribution p and an approximate probability distribution bDefine the “distance” between p({x}) and b({x}) as D(b({x}) || p({x})) = {x}b({x}) ln[b({x})/ p({x})] This is known as the Kullback-Liebler distance

Boltzmann’s law: p({x}) = (1/Z) e-E({x})/T

Generally assumed by statistical physicists Here we will use Boltzmann’s law as our definition of “energy” E T acts as a unit scale parameter; let T = 1

Substituting Boltzmann’s law into our distance measure D(b({x}) || p({x})) = {x}b({x})E({x}) + {x}b({x})ln[b({x})] + ln Z

Gibbs free energy

Our distance measure D(b({x}) || p({x})) = {x}b({x})E({x}) + {x}b({x})ln[b({x})] + ln Z

We see will be zero (p = b) when G(b({x})) = {x}b({x})E({x}) + {x}b({x})ln[b({x})] = U(b({x}) - S(b({x})

is minimized at F = -ln Z G: “Gibbs free energy” F: “Helmholz free energy” U: “average energy” S: “entropy”

Outline

The BP algorithm

MRFs – Markov Random Fields

Gibbs free energy

Bethe approximation

Kikuchi approximation

Generalized BP

Bethe approximation

We would like to derive Gibbs free energy in terms of one- and two-node beliefs bi and bij

Due to the pairwise nature of pairwise MRFs, bi and bij are sufficient to compute the average energy U U = - ijbij(xi,xj)lnij(xi,xj) - ibi(xi)lni(xi)

The exact marginals probabilities pi and pij yeild the same form, so this average energy is exact if the one- and two-node beliefs are exact

Bethe approximation

The entropy term is more problematic Usually must settle for an approximation

Entropy can be computed exactly if it can be explicitly expressed in terms of one- and two-node beliefs B({x}) = ij bij(xi,xj) / i bi(xi)qi-1 where qi = #neighbors of xi

Then the Bethe approximation to entropy is SBethe = ij xixjbij(xi,xj)lnbij(xi,xj) + (qi -1) xibi(xi)lnbi(xi)

For singly connected networks, this is exact and GBethe = U – SBethe corresponds to the exact marginal probabilities p

For graphs with loops, this is only an approximation (but usually a good one)

Equivalence of BP and Bethe

The Bethe approximation is exact for pairwise MRF’s when the graphs contain no loops, so the Bethe free energy is minimal for the correct marginalsBP gives correct marginals when the graph contains no loopsThus, when there are no loops, the BP beliefs are the global minima of the Bethe free energyWe can say more: a set of beliefs gives a BP fixed point in any graph iff they are local stationary points of the Bethe free energy This can be shown by adding Lagrange multipliers to

GBethe to enforce the marginalization constraints

Outline

The BP algorithm

MRFs – Markov Random Fields

Gibbs free energy

Bethe approximation

Kikuchi approximation

Generalized BP

Kikuchi approximation

Kikuchi approximation is an improvement on and generalization of Bethe

With this association between BP and the Bethe approximation to Gibbs free energy, can we use better approximation methods to craft better BP algorithms?

Cluster variational method

Free energy approximated as a sum of local free energies of sets of regions of nodes“Cluster variational method” provides a way to select the set of regions Begin with a basic set of clusters

including every interaction and node Subtract the free energies of over-

counted intersection regions Add back over-counted intersections of

intersections, etc.

Bethe is a Kikuchi approximation where the basic clusters are set to the set of all pairs of hidden nodes

Cluster variational method

Bethe regions involve one or two nodesDefine local free energy of a single nodeGi(bi(xi)) = xibi(xi)*ln(bi(xi) + Ei(xi))

Define local free energy involving two nodesGij(bi(xi,xj)=xi,xjbij(xi,xj)*ln(bij(xi,xj) + Eij(xi,xj))

Then for the regions corresponding to Bethe,GBethe = G12 + G23 + G45 + G56 + G14 + G25 +

G36 – G1 – G3 – G4 – G6 – 2G2 – 2G5

Cluster variational method

For the Kikuchi example shown below, regions involve four nodes

Extend the same logic as before

Define local free energy involving four nodese.g. G1245(b1245(x1,x2,x4,x5) = x1,x2,x4,x5

b1245(x1,x2,x4,x5)* ln(b1245(x1,x2,x4,x5) + E1245(x1,x2,x4,x5))

Then for the Kikuchi regions shown,GKikuchi = G1245 + G2356 – G25

A more general example

Now we have basic regions [1245], [2356], [4578], [5689]

Intersection regions [25], [45], [56], [58], and

Intersection of intersection region [5]

Then we have

GKikuchi = G1245 + G2356 + G4578 + G5689 - G25 - G45 - G56 - G58 + G5

Outline

The BP algorithm

MRFs – Markov Random Fields

Gibbs free energy

Bethe approximation

Kikuchi approximation

Generalized BP

Generalized BP

We show how to construct a GBP algorithm for this example

First find the intersections, intersections of intersections, etc. of the basic clusters Basic: [1245], [2356], [4578], [5689] Intersections: [25], [45], [56], [58] Intersection of intersections: [5]

Region Graph

Next, organize regions into the region graphA hierarchy of regions and their “direct” subregions

”direct” subregions are subregions not contained in another subregion

e.g. [5] is a subregion of [1245], but is also a subregion of [25]

Messages

Construct messages from all regions r to direct subregions sThese correspond to each edge of the region graphConsider the message from region [1245]

to subregion [25]A message from nodes not in the

subregion (1,4) to those in the subregion (2,5) m1425

Belief Equations

Construct belief equations for every region r

br({x}r) proportional to each compatibility matrix and evidence term completely contained in r b5 = k[5][m25m45m65m85]

b45 = k[4545][m1245m7845m25m65m85]

b1245 = k[124512142545] [m3625m7845m65m85]

Belief Equations

b5 = k[5][m25m45m65m85]

Belief Equations

b45 = k[4545][m1245m7845m25m65m85]

Belief Equationsb1245=k[124512142545][m3625m7845m65m85]

Enforcing Marginalization

Now, we need to enforce the marginalization condition relating each pair of regions that share an edge in the hierarchy

e.g. between [5] and [45]b5(x5) = x4b45(x4, x5)

Message Update

Adding the marginalization into the belief equations, we get the message update rule:

m45(x5)

k x4,x2 4(x4)45(x4,x5)m1245(x4,x5)m7825(x2,x5)

The collection of belief equations and the message update rules define out GBP algorithm

Complexity of GBP

Bad news: running time grows exponentially with the size of the basic clusters chosenGood news: if the basic clusters encompass the shortest loops in the graphical model, usually nearly all the error from BP is eliminatedThis usually requires only a small addition

amount of computation than BP