Post on 06-Mar-2019
transcript
17-18 October, 2005 Tokyo Institute of Technology 1
Probabilistic image processing and Bayesian networkKazuyuki Tanaka
Graduate School of Information Sciences,Tohoku University
kazu@smapip.is.tohoku.ac.jphttp://www.smapip.is.tohoku.ac.jp/~kazu/
ReferencesReferencesK. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, vol.35, pp.R81-R150 (2002).K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing, J. Phys. A, vol.37, pp.8675-8695 (2004).
17-18 October, 2005 Tokyo Institute of Technology 2
Bayesian Network and Belief Propagation
Probabilistic Information Processing
Probabilistic Model
Bayes Formula
Belief Propagation
J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44(1996).
Bayesian Network
17-18 October, 2005 3Tokyo Institute of Technology
Formulation Formulation of Belief Propagationof Belief Propagation
Link between Link between belief propagation belief propagation andand statistical statistical mechanics.mechanics.Y. Kabashima and D. Saad, Belief propagation vs. TAP for decodinY. Kabashima and D. Saad, Belief propagation vs. TAP for decoding g corrupted messages, corrupted messages, Europhys. Lett.Europhys. Lett. 4444 (1998). (1998). M. Opper and D. Saad (eds), M. Opper and D. Saad (eds), Advanced Mean Field Methods Advanced Mean Field Methods ------Theory andTheory andPracticePractice (MIT Press, 2001).(MIT Press, 2001).
Generalized belief propagationGeneralized belief propagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing freeJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free--energy energy approximations and generalized belief propagation algorithms, IEapproximations and generalized belief propagation algorithms, IEEE EE Transactions on Information Theory, Transactions on Information Theory, 5151 (2005).(2005).
Information geometrical interpretation Information geometrical interpretation of belief propagationof belief propagationS. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free eneS. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and rgy, and information geometry, Neural Computation, information geometry, Neural Computation, 1616 (2004).(2004).
17-18 October, 2005 4Tokyo Institute of Technology
Extension of Belief PropagationExtension of Belief Propagation
Generalized Belief PropagationGeneralized Belief PropagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing freeJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free--energy energy approximations and generalized belief propagation algorithms, IEapproximations and generalized belief propagation algorithms, IEEE EE Transactions on Information Theory, Transactions on Information Theory, 5151 (2005).(2005).
Generalized belief propagation is equivalent Generalized belief propagation is equivalent to the cluster variation method in statistical to the cluster variation method in statistical mechanicsmechanicsR. Kikuchi: A theory of cooperative phenomena, Phys. Rev., R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 8181 (1951).(1951).T. Morita: Cluster variation method of cooperative phenomena andT. Morita: Cluster variation method of cooperative phenomena and its its generalization I, J. Phys. Soc. Jpn, generalization I, J. Phys. Soc. Jpn, 1212 (1957).(1957).
17-18 October, 2005 5Tokyo Institute of Technology
Application of Belief Application of Belief PropagationPropagation
Image ProcessingImage ProcessingK. Tanaka: StatisticalK. Tanaka: Statistical--mechanical approach to image processing (Topical Review), J. mechanical approach to image processing (Topical Review), J. Phys. A, Phys. A, 3535 (2002).(2002).A. S. Willsky: Multiresolution Markov Models for Signal and ImagA. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, e Processing, Proceedings of IEEE, Proceedings of IEEE, 9090 (2002).(2002).
Low Density Parity Check CodesLow Density Parity Check CodesY. Kabashima and D. Saad: Statistical mechanics ofY. Kabashima and D. Saad: Statistical mechanics of lowlow--density paritydensity parity--check codes check codes (Topical Review), J. Phys. A, (Topical Review), J. Phys. A, 3737 (2004). (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and lowand low--density density parityparity--check codes, IEEE Transactions on Information Theory, check codes, IEEE Transactions on Information Theory, 5050 (2004).(2004).
CDMA Multiuser Detection AlgorithmCDMA Multiuser Detection AlgorithmY. Kabashima: A CDMA multiuser detection algorithm on the basis Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief of belief propagation, J. Phys. A, 36 (2003).propagation, J. Phys. A, 36 (2003).T. Tanaka and M. Okada: Approximate Belief propagation, density T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Trastatistical neurodynamics for CDMA multiuser detection, IEEE Transactions on nsactions on Information Theory, Information Theory, 5151 (2005).(2005).
SSatisfability atisfability PProblemroblemO. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics meO. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase thods and phase transitions in optimization problems, Theoretical Computer Scientransitions in optimization problems, Theoretical Computer Science, ce, 265265 (2001).(2001).M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic soluM. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random tion of random satisfability problems, Science, satisfability problems, Science, 297297 (2002).(2002).
17-18 October, 2005 Tokyo Institute of Technology 6
ContentsContents1. Introduction2. Belief Propagation3. Belief Propagation and Cluster Variation Method4. Belief Propagation for Gaussian Graphical Model5. Cluster Variation Method for Gaussian Graphical Model6. Bayesian Image Analysis and Gaussian Graphical Model7. Image Segmentation8. Concluding Remarks
17-18 October, 2005 Tokyo Institute of Technology 7
How should we treat the calculation of the summation over 2N configuration?
( )∑ ∑ ∑= = =1,0 1,0 1,0
211 2
,,,x x x
NN
xxxW LL
It is very hard to calculate exactly except some special cases.It is very hard to calculate exactly except some special cases.
Formulation for approximate algorithmAccuracy of the approximate algorithm
Belief Propagation
17-18 October, 2005 Tokyo Institute of Technology 8
Tractable Model
Factorizable
Not Factorizable
Probabilistic models with no loop are tractable.
Probabilistic models with loop are not tractable.
( )
⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛= ∑∑∑∑
∑∑∑∑
dcbadcba
dcba
),(),(),(),(
),(),(),(,
xDxCxBxA
xDxCxBxAa b c d
a
b
cd
ab
cd
( )∑∑∑∑a b c d
dcba xW ,,,,
17-18 October, 2005 Tokyo Institute of Technology 9
Probabilistic model on a graph with no loop
( )( ) ( ) ),(),(,),(,
,,,,,
12 yWyWyxWxWxWyxP
DCBA dcbadcba
≡
( ) ( )∑∑∑∑∑≡a b c d
dcbax
yxPyP ,,,,,2
ab
1
cd
2
3 4
56
Marginal probability of the node 2
17-18 October, 2005 Tokyo Institute of Technology 10
Probabilistic model on a graph with no loop
( )( )∑≡
→
aa xW
xM
A ,13
1
a3
1
ab
12
3 4
( ) ( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )∑
∑ ∑∑
∑∑∑
→→
→
=
⎟⎠
⎞⎜⎝
⎛⎟⎠
⎞⎜⎝
⎛=
≡
x
xBA
xBA
xMxMyxW
xWxWyxW
xWxWyxWyM
141312
12
1221
,
,,,
,,,
ba
a b
ba
ba
b41
( )( )∑≡
→
bb xW
xM
B ,14
ab
1
cd
2
3 4
56
17-18 October, 2005 Tokyo Institute of Technology 11
Probabilistic model on a graph with no loop( ) ( ) ( ) ),(),(,),(,,,,,, 12 yWyWyxWxWxWyxP DCBA dcbadcba ≡
ab
1
cd
2
3 4
56
( ) ( )
( )
( ) ( )
( )
( )
( ) ( ) ( )yMyMyM
xWxWyxWyWyW
yWyWxWxWyxW
,x,y,,,PyP
yM
xBA
yM
D
yM
C
xDCBA
x
212625
12
12
2
212625
),(),(,),(),(
),(),(),(),(,
→→→=
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
⎟⎠
⎞⎜⎝
⎛⎟⎠
⎞⎜⎝
⎛
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
=
⎟⎠
⎞⎜⎝
⎛⎟⎠
⎞⎜⎝
⎛⎟⎠
⎞⎜⎝
⎛⎟⎠
⎞⎜⎝
⎛=
≡
→→→
∑ ∑∑∑∑
∑ ∑∑∑∑
∑∑∑∑∑
4444444 34444444 214342143421 badc
dcba
a b c d
badc
dcba
dcba
φ=dcba III
17-18 October, 2005 Tokyo Institute of Technology 12
Probabilistic model on a graph with no loop
Message from the node 1 to the node 2 can be expressed in terms of the product of message from all the neighbouring nodes of the node 1 except one from the node 2.
( ) ( ) ( ) ( )yMyMyMyP 2126252 →→→=
Marginal probability can be expressed in terms of the product of messages from all the neighbouring nodes of node 2.
ab
1
cd
2
3 4
56
( ) ( ) ( ) ( )∑ →→→ =y
yMyMyxWxM 26251212 ,
( ) ( ) ( ) ( )∑ →→→ =x
xMxMyxWyM 14131221 ,
17-18 October, 2005 Tokyo Institute of Technology 13
Belief Propagation on a graph with no loop
{ } ( )∏−
=++==
1
111, ,1Pr
N
iiiii xxW
ZxX
( ) ( )
( ) ( ) ( ) ( )∑
∑∑ ∑ ∏
++→−→−→−
=++++→
=
⎟⎟⎠
⎞⎜⎜⎝
⎛=
k
k
xkkkkkkkkkkkkk
x x x
k
iiiiikkk
xxWxMxMxM
xxWxM
11,321
111,11
,
,1 2
L
1X2X 3X
1−kX
kX
2−kX
3−kX
1+kX
17-18 October, 2005 Tokyo Institute of Technology 14
Probabilistic Model on a Graph with Loops
( ) ( )∏∈
=Nij
jiijL xxWZ
xxxP ,1,,, 21 L
( ) ( ){ }∑=
1\2111 ,,,
xLxxxPxP
xL
Marginal Probability
( )∑∏∈
≡x Nij
jiij xxWZ ,Ω
( ) ( ){ }∑=
21,\212112 ,,,,
xxLxxxPxxP
xL
17-18 October, 2005 Tokyo Institute of Technology 15
Message Passing Rule of Belief Propagation
Fixed Point Equations for Massage
( )MMrrr
Φ=
( )( ) ( ) ( ) ( )
( ) ( ) ( ) ( )∑∑
∑
→→→
→→→
→ =
ξ ς
ς
ςςςξς
ςςςξςξ
15141312
15141312
21 ,
,
MMMW
MMMWM
1
33
44 2
5
13→M
14→M
15→M
21→M
17-18 October, 2005 Tokyo Institute of Technology 16
Approximate Representation of Marginal Probability
( ) ( )
( ) ( ) ( ) ( )( ) ( ) ( ) ( )∑
∑
→→→→
→→→→=
≡
1
1
115114113112
115114113112
\11
x
x
xMxMxMxMxMxMxMxM
PxPx
x144 2
5
13→M
14→M
15→M
12→M
33
Fixed Point Equations for Messages( )MM
rrrΦ=
17-18 October, 2005 Tokyo Institute of Technology 17
Fixed Point Equation and Iterative Method
Fixed Point Equation ( )** MMrrr
Φ=Iterative Method
( )( )( )
M
rr
rr
rr
23
12
01
MM
MM
MM
Φ←
Φ←
Φ←
0M1M
1M
0
xy =
)(xy Φ=
y
x*M
17-18 October, 2005 Tokyo Institute of Technology 18
ContentsContents1. Introduction2. Belief Propagation3. Belief Propagation and Cluster Variation Method4. Belief Propagation for Gaussian Graphical Model5. Cluster Variation Method for Gaussian Graphical Model6. Bayesian Image Analysis and Gaussian Graphical Model7. Image Segmentation8. Concluding Remarks
17-18 October, 2005 Tokyo Institute of Technology 19
Kullback-Leibler divergence and Free Energy
[ ] ( )( ) 0ln)( ≥⎟⎟
⎠
⎞⎜⎜⎝
⎛≡∑ x
xxx P
QQPQD ( ) ⎟⎠
⎞⎜⎝
⎛=≥ ∑
xxx 1)( ,0 QQQ
( ) ( )
ZQF
ZQQxxWQPQD
QF
Nijjiij
ln][
lnln)(,ln)(]|[
][
+=
++= ∑∑∑∈ 4444444 34444444 21
xxxxx
( ) ( ) [ ] 0=⇒= PQDPQ xx
( ) ZPFQQFQ
ln][1][min −==⎭⎬⎫
⎩⎨⎧
=∑x
x
( )∑∏∈
≡x Nij
jiij xxWZ ,
( ) ( )∏∈
=Nij
jiijL xxWZ
xxxP ,1,,, 21 L
Free Energy
17-18 October, 2005 Tokyo Institute of Technology 20
Free Energy and Cluster Variation Method
[ ] [ ] ( )ZQFPQD ln+=
[ ] ( ) ( )
{ }( ) ( )
( ) ( ) ( )xx
xxx
xxx
x
xx
xx
QQxxWxxQ
QQxxWQ
QQxxWQQF
Nij x xjiijjiij
Nij x xjiij
xx
Nijjiij
i j
i j ji
ln)(,ln,
ln)(,ln)(
ln)(,ln)(
,\
∑∑∑∑
∑∑∑∑ ∑
∑∑∑
+=
+⎟⎟⎠
⎞⎜⎜⎝
⎛=
+≡
∈
∈
∈
[ ] ( )( ) 0ln)( ≥⎟⎟
⎠
⎞⎜⎜⎝
⎛≡∑ x
xxx P
QQPQD
Free EnergyKL Divergence
( ) ( )∏∈
=Nij
jiij xxWZ
P ,1x
{ }∑≡
ji xx
jiij
Q
xxQ
,)(
),(
\xx
17-18 October, 2005 Tokyo Institute of Technology 21
Free Energy and Cluster Variation Method
[ ] [ ] ( )ZQFPQD ln+=
[ ] ( ) ( )
( )
( ) ( )
( ) ( )
( ) ( ) ( ) ( ) ( ) ( )∑ ∑∑∑∑
∑∑
∑∑∑
∑
∑∑∑
∈
Ω∈
∈
∈
⎟⎟⎠
⎞⎜⎜⎝
⎛−−+
+
≈
+
=
Nijjjiiijij
iii
Nijijij
Nijijij
QQQQQQ
WQ
WQQF
ξξξ ζ
ξ
ξ ζ
ξ ζ
ξξξξζξζξ
ξξ
ζξζξ
ζξζξ
lnln,ln,
ln
,ln,
ln)(
,ln,
xxx
Bethe Free Energy
Free EnergyKL Divergence( ) ( )∏
∈
=Nij
jiij xxWZ
P ,1x
{ }∑≡
ji xxjiij QxxQ
,
)(),(\x
x
∑≡ix
ii QxQ\x
x)()(
17-18 October, 2005 Tokyo Institute of Technology 22
Basic Framework of Cluster Variation Method
[ ] FPQDQQ γγ
minargminarg ≅
( ) ( )∑=ς
ςξξ ,iji QQ
[ ] { }[ ] ZQQFPQD iji ln,Bethe +≅
[ ]{ }
{ }[ ]ijiQQQ
QQFPQDiji
,minargminarg Bethe,
⇒
( ) ( ) 1, ==∑∑∑ξ ςξ
ςξξ iji QQ
{ }[ ] ( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )∑ ∑∑∑∑
∑∑∑∑∑
∈
Ω∈∈
⎟⎟⎠
⎞⎜⎜⎝
⎛−−+
+≡
Nijjjiiijij
iii
Nijijijiji
QQQQQQ
QQWQQQF
ξξξ ς
ξξ ς
ξξξξςξςξ
ξξςξςξ
lnln,ln,
ln,ln,,Bethe
17-18 October, 2005 Tokyo Institute of Technology 23
Basic Framework of Cluster Variation Method
{ }[ ] { }[ ]
( ) ( ) ( )
( ) ( )∑ ∑∑∑ ∑
∑∑∑ ∑
∈Ω∈
Ω∈ ∈
⎟⎟⎠
⎞⎜⎜⎝
⎛−−⎟⎟
⎠
⎞⎜⎜⎝
⎛−−
⎟⎟⎠
⎞⎜⎜⎝
⎛−−
≡
Nijijij
iii
i Njijiji
ijiiji
QQFQQL
i
1,1
,
,,
,
BetheBethe
ξ ζξ
ξ ς
ζξνξν
ζξξξλ
{ }{ }[ ] ( ) ( ) ( ) ( )
⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
=== ∑∑∑∑ 1, ,,,minarg Bethe, ξ ςξς
ςξξςξξ ijiijiijiQQ
QQQQQQFiji
Lagrange Multipliers to ensure the constraints
17-18 October, 2005 Tokyo Institute of Technology 24
Basic Framework of Cluster Variation Method
{ }[ ] { }[ ] ( ) ( ) ( )
( ) ( )
( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( )∑ ∑∑∑ ∑∑∑∑ ∑
∑ ∑∑∑∑
∑∑∑∑∑
∑ ∑∑∑ ∑
∑∑∑ ∑
∈Ω∈Ω∈ ∈
∈
Ω∈∈
∈Ω∈
Ω∈ ∈
⎟⎟⎠
⎞⎜⎜⎝
⎛−−⎟⎟
⎠
⎞⎜⎜⎝
⎛−−⎟⎟
⎠
⎞⎜⎜⎝
⎛−−
⎟⎟⎠
⎞⎜⎜⎝
⎛−−+
+=
⎟⎟⎠
⎞⎜⎜⎝
⎛−−⎟⎟
⎠
⎞⎜⎜⎝
⎛−−
⎟⎟⎠
⎞⎜⎜⎝
⎛−−≡
Nijijij
iii
i Njijiji
Nijjjiiijij
iii
Nijijij
Nijijij
iii
i Njijijiijiiji
QQQQ
QQQQQQ
QQWQ
QQQQFQQL
i
i
1,1,
lnln,ln,
ln,ln,
1,1
,,,
,
,BetheBethe
ξ ζξξ ζ
ξξξ ζ
ξξ ζ
ξ ζξ
ξ ς
ζξνξνζξξξλ
ξξξξζξζξ
ξξζξζξ
ζξνξν
ζξξξλ
( ) { }[ ] 0,Bethe =∂
∂iji
ii
QQLxQ
Extremum Condition
( ) { }[ ] 0,, Bethe =
∂∂
ijijiij
QQLxxQ
17-18 October, 2005 Tokyo Institute of Technology 25
Approximate Marginal Probability in Cluster Variation Method
27→M144 2
5
13→M
14→M
15→M
12→M
33
26→M
144
5
13→M
14→M
15→M
12W33
2
6
27→M
88
77
28→M
( ) ( ) ( )
( ) ( )115114
1131121
111
xMxM
xMxMZ
xQ
→→
→→
×
= ( ) ( ) ( ) ( )
( ) ( ) ( ) ( )2282272262112
11511411312
2112
,
1,
xMxMxMxxW
xMxMxMZ
xxQ
→→→
→→→
×
=
ExtremumCondition( ) { }[ ] 0,Bethe =
∂∂
ijiii
QQLxQ ( ) { }[ ] 0,
, Bethe =∂
∂iji
jiij
QQLxxQ
17-18 October, 2005 Tokyo Institute of Technology 26
Cluster Variation Method and Belief Propagation
27→M144 2
5
13→M
14→M
15→M
12→M
33
26→M
144
5
13→M
14→M
15→M
12W33
2
6
27→M
88
77
28→M
( ) ( )∑=ς
ζξξ ,121 QQ
( ) ( ) ( )
( ) ( )115114
1131121
111
xMxM
xMxMZ
xQ
→→
→→
×
= ( ) ( ) ( ) ( )
( ) ( ) ( ) ( )2282272262112
11511411312
2112
,
1,
xMxMxMxxW
xMxMxMZ
xxQ
→→→
→→→
×
=
( )( ) ( )
( ) ( )ςς
ςξς
ξ
ς
1514
1312
21
,
→→
→
→
×
∝∑MM
MWM
Message Update Rule
17-18 October, 2005 Tokyo Institute of Technology 27
ContentsContents1. Introduction2. Belief Propagation3. Belief Propagation and Cluster Variation Method4. Belief Propagation for Gaussian Graphical Model5. Cluster Variation Method for Gaussian Graphical Model6. Bayesian Image Analysis and Gaussian Graphical Model7. Image Segmentation8. Concluding Remarks
17-18 October, 2005 Tokyo Institute of Technology 28
Gaussian Graphical Model
( )
formula. integral Gauss ldimensiona-multi theusingby
calculated becan quantities lstatisticaother the
and average The xxx d∫ ρ
{ }Ω∈= ixix
( ) ( ) ( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−= ∑∑
∈Ω∈ Nijji
iii xxgx
Z22
21
21exp1 αβρ x
( )+∞∞−∈ ,ix
( ){ }
( )
( )( )⎪
⎪⎩
⎪⎪⎨
⎧
∈
=+
=
∑∈
otherwise 0 /
/1
Nij
jiNijj
ij βα
βα
H
( ) gHxxx 1−=∫ dρmatrix : Ω×ΩH
17-18 October, 2005 Tokyo Institute of Technology 29
Message Passing Rule of Belief Propagation
( )( ) ( ) ( ) ( )
( ) ( ) ( ) ( )∑∑
∑
→→→
→→→
→ =
ξ ς
ς
ςςςξς
ςςςξςξ
15141312
15141312
21 ,
,
MMMW
MMMWM
1
33
44 2
5
13→M
14→M
15→M
21→M ( ) ⎟⎠⎞
⎜⎝⎛ −−≡ →→
→
→
2
21exp
2
)(
jijiji
jiM
μςλπ
λ
ς
17-18 October, 2005 Tokyo Institute of Technology 30
Message Passing Rule of Belief Propagation
1
33
44 2
5
( )1313 , →→ λμ( )1414 , →→ λμ
( )1515 , →→ λμ( )2121 , →→ μλ
( )151413
15141321
→→→
→→→→ ++++
+++=
λλλαβλλλβαλ
151413
151514141313121
→→→
→→→→→→→ +++
+++=
λλλβλμλμλμβμ g
Fixed-Point Equations
⎟⎟⎠
⎞⎜⎜⎝
⎛Φ=⎟⎟
⎠
⎞⎜⎜⎝
⎛λμ
λμ
r
rr
r
r
Natural Iteration
17-18 October, 2005 Tokyo Institute of Technology 31
Message Passing Rule of Belief Propagation
1
33
44 2
5
( )1313 , →→ λμ( )1414 , →→ λμ
( )1515 , →→ λμ( )2121 , →→ μλ
( )151413
15141321
→→→
→→→→ ++++
+++=
λλλαβλλλβαλ
151413
151514141313121
→→→
→→→→→→→ +++
+++=
λλλβλμλμλμβμ g
( ) ⎟⎠⎞
⎜⎝⎛ −−≡ →→
→
→
21
1
21exp
2
)(
jijiji
ji
x
xM
μλπ
λ
( ) ( ) ( ) ( ) ( )( ) ( ) ( ) ( )∑ →→→→
→→→→=
1
115114113112
11511411311211
xxMxMxMxM
xMxMxMxMxP
17-18 October, 2005 Tokyo Institute of Technology 32
ContentsContents1. Introduction2. Belief Propagation3. Belief Propagation and Cluster Variation Method4. Belief Propagation for Gaussian Graphical Model5. Cluster Variation Method for Gaussian Graphical Model6. Bayesian Image Analysis and Gaussian Graphical Model7. Image Segmentation8. Concluding Remarks
17-18 October, 2005 Tokyo Institute of Technology 33
Kullback-Leibler Divergence of Gaussian Graphical Model
[ ] ( ) ( )( ) [ ] ZQFdQQQD lnln +=⎟⎟
⎠
⎞⎜⎜⎝
⎛≡ ∫
∞+
∞−z
zzz
ρρ
[ ] ( )( )
( )( ) ( ) ( )∫∑
∑
+−++−+
−−=
∈
Ω∈
zzz dQQmmVVV
gmVQF
jijijiijNij
iiiii
ln221
21
2
2
α
β
( ) zz dQzm ii ∫≡( ) ( ) zz dQmzV iii ∫ −≡ 2
( )( ) ( ) zz dQmzmzV jjiiij ∫ −−≡
Entropy Term
( ) ( ) ( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−= ∑∑
∈Ω∈ Nijji
iii xxgx
Z22
21
21exp1 αβρ x
17-18 October, 2005 Tokyo Institute of Technology 34
Cluster Variation Method
( ) ( ) ( )∫ −≡ zz dQzxxQ iiii δ
( ) ( ) ( )( ) ( )⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛= ∏∏
∈Ω∈ Nij jii
jiij
iii xQxQ
xxQxQQ
,x
Trial Function
( ) ( ) ( ) ( )∫ −−≡ zz dQzxzxxxQ jjiijiij δδ,
Tractable Form
17-18 October, 2005 Tokyo Institute of Technology 35
Cluster Variation Method for Gaussian Graphical Model
( ) ( ) ( )( ) ( )⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛= ∏∏
∈Ω∈ Nij jii
jiij
iii xQxQ
xxQxQQ
,x
( ) ( ) ( )
( ) ( )( ) ( )⎟
⎠⎞
⎜⎝⎛ −−−=
−≡
−
∫
γγγγγ
γγ
γγγγ
π
δ
mxAmxA
zzzxx
1T
21exp
det2
1
dQQ
Trial Function
Marginal Distribution of GGM is also GGM
( ) iiiV=γA ( ) ijij
V=γA( ) iim=γm
17-18 October, 2005 Tokyo Institute of Technology 36
Cluster Variation Method for Gaussian Graphical Model
[ ] { }[ ]( )( ) ( )( )
( ) ( ) ( )( )∑∑
∑∑
∈Ω∈
∈Ω∈
⎟⎠⎞
⎜⎝⎛ ++⎟
⎠⎞
⎜⎝⎛ +−
−++−+−−=
=
Nijij
ii
Nijjijiji
iiii
ijii
Vi
mmVVVgmV
VVmFQF
Adet2ln2112ln
211
221
21
,,
2
22
ππμ
αβ
( )( ) ( )
( ) ( )⎟⎠⎞
⎜⎝⎛ −−−= −
γγγγγ
γγγγ
πmxAmx
Ax 1T
21exp
det2
1Q
( ) ( ) ( )( ) ( )⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛= ∏∏
∈Ω∈ Nij jii
jiij
iii xQxQ
xxQxQQ
,x
Bethe Free Energy in GGMBethe Free Energy in GGM
17-18 October, 2005 Tokyo Institute of Technology 37
Cluster Variation Method for Gaussian Graphical Model
( ) ( ){ }
1
141−
∈
−
⎟⎟⎠
⎞⎜⎜⎝
⎛+−−= ∑
Nijjiiiji i
V Aαβμ
( )jiij VVV 241121 αα
++−=
{ }[ ] ( ) ( ){ }
00,,
=−−−⇒=∂
∂∑∈Nijj
jiiii
ijii mmgmm
VVmFαβ
{ }[ ] ( ){ }
0340,, 11 =−++⇒=
∂
∂∑∈
−−
Nijjiiiji
i
ijii AV
VVmFAαβ
{ }[ ] ( ) 00,, 1 =−−⇒=
∂
∂ −
ijijij
ijii
VVVmF
Aα
⎟⎟⎠
⎞⎜⎜⎝
⎛≡
jij
ijiij VV
VVA
17-18 October, 2005 Tokyo Institute of Technology 38
Iteration Procedure
Fixed Point Equation ( )*VΨV =*
Iteration
( )( )( )
M
)2()3(
)1()2(
)0()1(
VΨVVΨVVΨV
←
←
←
)0(M)1(V
)1(V
0
xy =
)(xy Ψ=
y
x*V
17-18 October, 2005 Tokyo Institute of Technology 39
Cluster Variation Method and TAP Free Energy
Loopy Belief Propagation ( )( )
( )0
41121
5223
2
+→
+−=
++−=
α
ααα
αα
OVVVV
VVV
jiji
jiij
{ }[ ]( )( ) ( )( )
( ) ( )∑∑
∑∑
∈Ω∈
∈Ω∈
+−⎟⎠⎞
⎜⎝⎛ +−
−+++−−=
Nijji
ii
Nijjiji
iiii
ijii
OVVV
mmVVgmV
VVmF
42
22
212ln
211
21
21
,,
ααπ
αβ
( ) iiiV=γA ( ) ijij
V=γA
TAP Free TAP Free EnergyEnergy
( ) 01 =−− −
ijijAα
Mean Field Free EnergyMean Field Free Energy
17-18 October, 2005 Tokyo Institute of Technology 40
ContentsContents1. Introduction2. Belief Propagation3. Belief Propagation and Cluster Variation Method4. Belief Propagation for Gaussian Graphical Model5. Cluster Variation Method for Gaussian Graphical Model6. Bayesian Image Analysis and Gaussian Graphical Model7. Image Segmentation8. Concluding Remarks
17-18 October, 2005 Tokyo Institute of Technology 41
Bayesian Image Analysis
Original Image Degraded Image
Transmission
Noise
{ } { } { }{ }
444 3444 21
44 844 7644444 844444 76
44444 344444 21
Likelihood Marginal
yProbabilit PrioriA Processn Degradatio
yProbabilit PosterioriA Image Degraded
Image OriginalImage OriginalImage DegradedImage DegradedImage Original
PrPrPr
Pr =
17-18 October, 2005 Tokyo Institute of Technology 42
Bayesian Image AnalysisDegradation Process ( )+∞∞−∈ ,, ii gf
( ) ( )∏Ω∈
⎟⎠⎞
⎜⎝⎛ −−=
iii gfP 2
221exp
21,
σσπσfg
( )2,0~ σNn
nfg
i
iii += Ω
Original Image Degraded Image
Transmission
Additive White Gaussian Noise
17-18 October, 2005 Tokyo Institute of Technology 43
Bayesian Image Analysis
A Priori Probability ( )+∞∞−∈ ,, ji gf
( ) ( ) ( )∏∈
⎟⎠⎞
⎜⎝⎛ −−=
Nijji ff
ZP 2
PR 21exp1 α
ααf
Standard Images
Generate
Similar?
17-18 October, 2005 Tokyo Institute of Technology 44
Bayesian Image Analysis
( )+∞∞−∈ ,, ji gf
( ) ( ) ( )( )
( )
( ) ( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−×
=
=
∑∑∈Ω∈ Nij
jii
ii ffgf
Z
PPP
P
222
POS
21
21exp
,,1
,,
,,
ασ
σα
σαασ
σα
g
gffg
gf
A Posteriori Probability
Gaussian Graphical Model
17-18 October, 2005 Tokyo Institute of Technology 45
Bayesian Image Analysis( )+∞∞−∈ ,, ji gf
( ) ( )
( ) ( )
( ) ( )∏
∑∑
∈
∈Ω∈
=
⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−×
=
Nijjiij
Nijji
iii
ffWZ
ffgf
ZP
,,,
1
21
21exp
,,1,,
POS
222
POS
σα
ασ
σασα
g
ggf
( ) ( ) ( ) ( ) ⎟⎠⎞
⎜⎝⎛ −−−−−−≡ 22
22
2 21
81
81exp, jijjiijiij ffgfgfffW α
σσ
A Posteriori Probability
Gaussian Graphical Model
17-18 October, 2005 Tokyo Institute of Technology 46
Bayesian Image Analysis
Ωy
x
{ }Ω∈= ifif { }Ω∈= igig
fg
( )αfP ( )σ,fgP gOriginal Image Degraded Image
( ) ( ) ( )( )σα
ασσα
,,
,,g
ffggf
PPP
P =
( ) ( ) iiiii dffPfdPff ∫∫+∞
∞−== σασα ,,,,ˆ gfgf
A Priori Probability
A Posteriori Probability
Degraded Image
Pixels
17-18 October, 2005 Tokyo Institute of Technology 47
Hyperparameter Determination by Maximization of Marginal Likelihood
( ) ( ) ( ) ( ) fffgfgfg dPPdPP ∫∫ == ασσασα ,,,,
( )( )
( )σασασα
,max argˆ,ˆ,
gP=
MarginalizationMarginalization
g( )σα ,gP{ }Ω∈= ifif
Original Image
Marginal Likelihood{ }Ω∈= igig
Degraded Image
Ωy
x
( )∫= fgf dPff ii σα ˆ,ˆ,ˆ
f g( )αfP ( )σ,fgP g( ) ( ) ( )ασσα ffggf PPP ,,, =
17-18 October, 2005 Tokyo Institute of Technology 48
Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm
( ) ( ) ( ) fffgg dPPP ∫= ασσα ,,Marginal Likelihood
{ }Ω∈= igigIncomplete Data
Ωy
x
( ) ( ) ( ) fgfgfg dPPQ ∫= ',',ln,,,,',' σασασασα
( ) ( ) 0,,',''
,0,,','' ','','
=⎥⎦⎤
⎢⎣⎡∂∂
=⎥⎦⎤
⎢⎣⎡∂∂
==== σσαασσαα
σασασ
σασαα
gg QQ
( ) ( ) 0, ,0, =∂∂
=∂∂ σα
ασα
σgg PP
Equivalent
Q-Function
17-18 October, 2005 Tokyo Institute of Technology 49
Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm
( ) ( ) ( ) fffgg dPPP ∫= ασσα ,,Marginal Likelihood Ω
y
x
( ) ( ) ( ) fgfgfg dPPQ ∫= ',',ln,,,,',' σασασασαQ-Function
( ) ( )( ) ( ) ( )( ) ( )( ) ( )( )
( )( ) ( )( ).,','maxarg1,1 :Step-M
.',',ln,,,',' :Step-E
','ttQtt
dPttPttQ
σασασα
σασασασα
βα←++
← ∫ fgfgf
Iterate the following EM-steps until convergence:EM Algorithm
A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).
17-18 October, 2005 50Tokyo Institute of Technology
OneOne--Dimensional SignalDimensional Signal
EM Algorithm
i
i
i
0 127 255
0 127 255
0 127 255
100
0
200
100
0
200
100
0
200
if
ig
if
Original Signal
Degraded Signal
Estimated Signal
40=σ
17-18 October, 2005 51Tokyo Institute of Technology
Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model
Original ImageOriginal Image Degraded ImageDegraded Image
MSE: 1529MSE: 1529
MSE: 1512MSE: 1512
EM Algorithm with Belief Propagation
17-18 October, 2005 Tokyo Institute of Technology 52
Exact Results of Gaussian Graphical Model
( ) ( ) ( ) ( )
∫
∑∑
⎟⎠⎞
⎜⎝⎛ −−−
⎟⎠⎞
⎜⎝⎛ −−−
=
⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−==
∈Ω∈
fCffgf
Cffgf
ggf
d
ffgfZ
PNij
jii
ii
T22
T22
222
POS
21
21exp
21
21exp
21
21exp
,,1,,
ασ
ασ
ασσα
σα
( ) ( )( ) ( ) ⎟
⎠⎞
⎜⎝⎛
+−
+=
Ωg
CICg
CICg 2
T2 2
1expdet2det,
ασα
ασπασαP
( ) gCIf12ˆ −
+= ασ
Multi-dimensional Gauss integral formula
Ωy
x
( )( )
( )σασασα
,max argˆ,ˆ,
gP=
17-18 October, 2005 53Tokyo Institute of Technology
Comparison of Belief Propagation with Comparison of Belief Propagation with Exact Results in Gaussian Graphical ModelExact Results in Gaussian Graphical Model
( )2ˆ||
1MSE ∑Ω∈
−Ω
=i
ii ff
--5.214445.2144437.91937.9190.0007590.000759315315ExactExact
--5.192015.1920136.30236.3020.0006110.000611327327Belief Belief PropagationPropagation
MSEMSE α σ ( )σα ˆ,ˆln gP
--5.175285.1752834.97534.9750.0006520.000652236236ExactExact
--5.152415.1524133.99833.9980.0005740.000574260260Belief Belief PropagationPropagation
MSEMSE α σ ( )σα ˆ,ˆln gP
40=σ
40=σ ( )( )
( )σασασα
,max argˆ,ˆ,
gP=
17-18 October, 2005 54Tokyo Institute of Technology
Image Restoration by Image Restoration by Gaussian Graphical ModelGaussian Graphical Model
Original ImageOriginal Image
MSE:315MSE:315MSE: 325MSE: 325
MSE: 545MSE: 545 MSE: 447MSE: 447MSE: 411MSE: 411
MSE: 1512MSE: 1512
Degraded ImageDegraded Image Belief PropagationBelief Propagation
LowpassLowpass FilterFilter Median FilterMedian Filter
Exact
Wiener Filter
( )2ˆ||
1MSE ∑Ω∈
−Ω
=i
ii ff
17-18 October, 2005 55Tokyo Institute of Technology
Original ImageOriginal Image
MSE236MSE236MSE: 260MSE: 260
MSE: 372MSE: 372 MSE: 244MSE: 244MSE: 224MSE: 224
MSE: 1529MSE: 1529
Degraded ImageDegraded Image Belief PropagationBelief Propagation
LowpassLowpass FilterFilter Median FilterMedian Filter
Exact
Wiener Filter
( )2ˆ||
1MSE ∑Ω∈
−Ω
=i
ii ff
Image Restoration by Gaussian Image Restoration by Gaussian Graphical ModelGraphical Model
17-18 October, 2005 56Tokyo Institute of Technology
Extension of Belief PropagationExtension of Belief Propagation
Generalized Belief PropagationGeneralized Belief PropagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing freeJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free--energy energy approximations and generalized belief propagation algorithms, IEapproximations and generalized belief propagation algorithms, IEEE EE Transactions on Information Theory, Transactions on Information Theory, 5151 (2005).(2005).
Generalized belief propagation is equivalent Generalized belief propagation is equivalent to the cluster variation method in statistical to the cluster variation method in statistical mechanicsmechanicsR. Kikuchi: A theory of cooperative phenomena, Phys. Rev., R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 8181 (1951).(1951).T. Morita: Cluster variation method of cooperative phenomena andT. Morita: Cluster variation method of cooperative phenomena and its its generalization I, J. Phys. Soc. Jpn, generalization I, J. Phys. Soc. Jpn, 1212 (1957).(1957).
17-18 October, 2005 57Tokyo Institute of Technology
Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model
( )2ˆ||
1MSE ∑Ω∈
−Ω
=i
ii ff
--5.211725.2117237.90937.9090.0007580.000758315315Generalized Generalized
Belief Belief PropagationPropagation
--5.214445.2144437.91937.9190.0007590.000759315315ExactExact
--5.192015.1920136.30236.3020.0006110.000611327327Belief Belief PropagationPropagation
MSEMSE α σ ( )σα ˆ,ˆln gP
--5.172565.1725634.97134.9710.0006520.000652236236Generalized Generalized
Belief Belief PropagationPropagation
--5.175285.1752834.97534.9750.0006520.000652236236ExactExact
--5.152415.1524133.99833.9980.0005740.000574260260Belief Belief PropagationPropagation
MSEMSE α σ ( )σα ˆ,ˆln gP
40=σ
40=σ
( )( )
( )σασασα
,max argˆ,ˆ,
gP=
17-18 October, 2005 58Tokyo Institute of Technology
Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters
( )( )
2
,,, ˆ∑
Ω∈−
Ω=
yxyxyx ff
||1MSE
548548(5x5)(5x5)
445445(5x5)(5x5)315315
Generalized Generalized Belief Belief
PropagationPropagation864864(3x3)(3x3)Wiener Wiener
FilterFilter315315ExactExact
486486(3x3)(3x3)Median Median FilterFilter
413413(5x5)(5x5)327327Belief Belief
PropagationPropagation388388(3x3)(3x3)LowpassLowpass
FilterFilter
MSEMSEMSEMSE40=σ
GBPGBP
(3x3) (3x3) LowpassLowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener
17-18 October, 2005 59Tokyo Institute of Technology
Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters
( )( )
2
,,, ˆ∑
Ω∈−
Ω=
yxyxyx ff
||1MSE
372372(5x5)(5x5)
244244(5x5)(5x5)236236
Generalized Generalized Belief Belief
PropagationPropagation703703(3x3)(3x3)Wiener Wiener
FilterFilter236236ExactExact
331331(3x3)(3x3)Median Median FilterFilter
224224(5x5)(5x5)260260Belief Belief
PropagationPropagation241241(3x3)(3x3)LowpassLowpass
FilterFilter
MSEMSEMSEMSE40=σ
GBPGBP
(5x5) (5x5) LowpassLowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener
17-18 October, 2005 Tokyo Institute of Technology 60
ContentsContents1. Introduction2. Belief Propagation3. Belief Propagation and Cluster Variation Method4. Belief Propagation for Gaussian Graphical Model5. Cluster Variation Method for Gaussian Graphical Model6. Bayesian Image Analysis and Gaussian Graphical Model7. Image Segmentation8. Concluding Remarks
17-18 October, 2005 Tokyo Institute of Technology 61
Image Segmentation by Image Segmentation by Gauss Mixture ModelGauss Mixture Model
( ) ( )∏Ω∈
=i
iaP γγa
( )( ) ( )
( )( )∏Ω∈
⎟⎟⎠
⎞⎜⎜⎝
⎛−−=
iii
ii
afaa
P 222
1exp2
1,, μσσπ
σμaf
( ) ( )( ) ( )( ) ( ) ( ) ( ) 204321
,1924 ,1923,1272 ,641
========
σσσσμμμμ
( )( ) ( )
( )( )
( )( )( )∏∑
∑
Ω∈ =⎟⎟⎠
⎞⎜⎜⎝
⎛ −−=
=
i
K
k
i
kkf
kk
PP
P
12
2
2exp
2
,,
,,
σμ
σπγ
aγaσμaf
γσμfGauss Mixture Model
( )( ) ( )
( )γσμfγaσμaf
γσμfa
,,,,
,,,
PPP
P
=
17-18 October, 2005 Tokyo Institute of Technology 62
Image Segmentation by Combining Image Segmentation by Combining Gauss Mixture Model with Potts Model Gauss Mixture Model with Potts Model
( ) ( ) ( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛= ∏∏
∈Ω∈ Bijaa
ii ji
aZ
P ,PR
exp1 δαγγ
γa
( )( ) ( )
( )( )∏Ω∈
⎟⎟⎠
⎞⎜⎜⎝
⎛−−=
iii
ii
afaa
P 222
1exp2
1,, μσσπ
σμaf
( ) ( )( ) ( )( ) ( ) ( ) ( ) 204321
,1924 ,1923,1272 ,641
========
σσσσμμμμ
( )( ) ( )
( )γσμfγaσμaf
γσμfa
,,,,
,,,
PPP
P
=
Belief PropagationBelief Propagation
{ }Ω∈= iaia
{ }Ω∈= ifif
{ }Ω∈= iaiˆaPotts Model
17-18 October, 2005 63Tokyo Institute of Technology
Image SegmentationImage Segmentation
Original Image Histogram Gauss Mixture Model
Gauss Mixture Model and
Potts Model
Belief Belief PropagationPropagation
( ) ( ) ( )( ) ( ) ( )( ) ( ) ( )( ) ( ) ( )( ) ( ) ( ) 0101.05 ,4.145 ,8.2245
3982.04 ,7.114 ,4.16843375.03 ,6.233 ,6.13030711.02 ,0.182 ,2.422
1831.01 ,7.21 ,7.121
===============
γσμγσμγσμγσμγσμ
17-18 October, 2005 64Tokyo Institute of Technology
Motion DetectionMotion Detection
SegmentationAND
Detection
ba −
cb −
Gauss Mixture Model and Potts Model with Belief PropagationGauss Mixture Model and Potts Model with Belief Propagation
Segmentation
a
b
c
17-18 October, 2005 Tokyo Institute of Technology 65
ContentsContents1. Introduction2. Belief Propagation3. Belief Propagation and Cluster Variation Method4. Belief Propagation for Gaussian Graphical Model5. Cluster Variation Method for Gaussian Graphical Model6. Bayesian Image Analysis and Gaussian Graphical Model7. Image Segmentation8. Concluding Remarks
17-18 October, 2005 66Tokyo Institute of Technology
SummarySummaryFormulation of belief propagationFormulation of belief propagationAccuracy of belief propagation in Bayesian Accuracy of belief propagation in Bayesian image analysis by means of Gaussian image analysis by means of Gaussian graphical model (Comparison between the graphical model (Comparison between the belief propagation and exact calculation)belief propagation and exact calculation)Some applications of Bayesian image Some applications of Bayesian image analysis and belief propagationanalysis and belief propagation
17-18 October, 2005 67Tokyo Institute of Technology
Related ProblemRelated Problem
( ) ( ) ( ) fgffggff ddPP∫ − ασσα ,,,ˆ 2
Statistical Performance Spin Glass Theory
H. Nishimori: Statistical Physics of Spin Glasses and Information Processing: An Introduction, Oxford University Press, Oxford, 2001.
f g( )αfP ( )σ,fgP g( ) ( ) ( )ασσα ffggf PPP ,,, =
( ) ( )∫= fgfg dPff ii σασα ,,,,ˆ