Post on 13-Feb-2016
description
transcript
RC2005 (19 July, 2005, Sendai)
1
Probabilistic image processing and Bayesian networkKazuyuki Tanaka
Graduate School of Information Sciences,Tohoku University
kazu@smapip.is.tohoku.ac.jphttp://www.smapip.is.tohoku.ac.jp/~kazu/
ReferencesReferencesK. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, vol.35, pp.R81-R150 (2002).K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing, J. Phys. A, vol.37, pp.8675-8695 (2004).
RC2005 (19 July, 2005, Sendai)
2
Bayesian Network and Belief Propagation
Probabilistic Information Processing
Probabilistic Model
Bayes Formula
Belief Propagation
J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996).
Bayesian Network
RC2005 (19 July, 2005, Sendai)RC2005 (19 July, 2005, Sendai) 33
Formulation of Belief PropagationFormulation of Belief PropagationLink between Link between belief propagation belief propagation andand statistical statistical mechanics.mechanics.Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory and Practice (MIT Press, 2001).
Generalized belief propagationGeneralized belief propagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).
Information geometrical interpretation Information geometrical interpretation of belief propagationof belief propagation
S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004).
RC2005 (19 July, 2005, Sendai)RC2005 (19 July, 2005, Sendai) 44
Application of Belief PropagationApplication of Belief PropagationImage ProcessingImage ProcessingK. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002).A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002).
Low Density Parity Check CodesLow Density Parity Check CodesY. Kabashima and D. Saad: Statistical mechanics of low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004).
CDMA Multiuser Detection AlgorithmCDMA Multiuser Detection AlgorithmY. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003).T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005).
SSatisfability atisfability PProblemroblemO. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265 (2001).M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002).
RC2005 (19 July, 2005, Sendai) 5
ContentsContents1. Introduction2. Belief Propagation3. Bayesian Image Analysis and Gaussian
Graphical Model4. Image Segmentation5. Concluding Remarks
RC2005 (19 July, 2005, Sendai)
6
How should we treat the calculation of the summation over 2N configurations.
1,0 1,0 1,0
211 2
,,,x x x
NN
xxxW
It is very hard to calculate exactly except some special cases.It is very hard to calculate exactly except some special cases.
Formulation for approximate algorithmAccuracy of the approximate algorithm
Belief Propagation
RC2005 (19 July, 2005, Sendai)
7
Tractable Model
Factorizable
Not Factorizable
Probabilistic models with no loop are tractable.
Probabilistic models with loop are not tractable.
dcbadcba
dcba
),(),(),(),(
),(),(),(,
xDxCxBxA
xDxCxBxAa b c d
a
b
cd
a
b
cd
a b c d
dcba xW ,,,,
RC2005 (19 July, 2005, Sendai)
8
Probabilistic Model on a Graph with Loops
Bij
jiijL xxWZ
xxxP ,1,,, 21
1\2111 ,,,
xLxxxPxP
x
Marginal Probability
x Bij
jiij xxWZ ,
21,\212112 ,,,,
xxLxxxPxxP
x
RC2005 (19 July, 2005, Sendai)
9
Message Passing Rule of Belief Propagation
Fixed Point Equations for Massage MM
15141312
15141312
21 ,
,
MMMW
MMMWM
1
33
44 2
5
13M
14M
15M
21M
RC2005 (19 July, 2005, Sendai)
10
Approximate Representation of Marginal Probability
1
1
115114113112
115114113112
\11
x
x
xMxMxMxMxMxMxMxM
PxPx
x
144 2
5
13M
14M
15M
12M
33
Fixed Point Equations for Messages
MM
RC2005 (19 July, 2005, Sendai)
11
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Iterative Method
23
12
01
MM
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
RC2005 (19 July, 2005, Sendai) 12
ContentsContents1. Introduction2. Belief Propagation3. Bayesian Image Analysis and Gaussian
Graphical Model4. Image Segmentation5. Concluding Remarks
RC2005 (19 July, 2005, Sendai)
13
Bayesian Image Analysis
Original Image Degraded Image
Transmission
Noise
Likelihood Marginal
yProbabilit PrioriA Processn Degradatio
yProbabilit PosterioriA Image Degraded
Image OriginalImage OriginalImage DegradedImage DegradedImage Original
PrPrPr
Pr
RC2005 (19 July, 2005, Sendai)
14
Bayesian Image AnalysisDegradation Process ,, ii gf
iii gfP 2
221exp
21,
fg
2,0~ Nn
nfg
i
iii
Original Image Degraded Image
Transmission
Additive White Gaussian Noise
RC2005 (19 July, 2005, Sendai)
15
Bayesian Image Analysis
A Priori Probability ,, ji gf
Bijji ff
ZP 2
PR 21exp1
f
Standard Images
Generate
Similar?
RC2005 (19 July, 2005, Sendai)
16
Bayesian Image Analysis ,, ji gf
Bijjiij ffW
Z
PPP
P
,,,
1
,,
,,
POS
g
gffg
gf
22
22
2 21
81
81exp, jijjiijiij ffgfgfffW
A Posteriori Probability
Gaussian Graphical Model
RC2005 (19 July, 2005, Sendai)
17
Bayesian Image Analysis
y
x
ifif igig
fg
fP ,fgP gOriginal Image Degraded Image
,,
,,g
ffggf
PPP
P
iiiii dffPfdPff
,,,,ˆ gfgf
A Priori Probability
A Posteriori Probability
Degraded Image
Pixels
RC2005 (19 July, 2005, Sendai)
18
Hyperparameter Determination by Maximization of Marginal Likelihood
fffgg dPPP ,,
,max argˆ,ˆ,
gP
MarginalizationMarginalization
g ,gP ifif
Original Image
Marginal Likelihood
igigDegraded Image
y
x
fgf dPff ii ˆ,ˆ,ˆ
f g fP ,fgP g ffggf PPP ,,,
RC2005 (19 July, 2005, Sendai)
19
Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm
fffgg dPPP ,,Marginal Likelihood
igigIncomplete Data
y
x
fgfgfg dPPQ ,,ln',',,',',
0,',',','
gQ
0,
gP
Equivalent
Q-Function
RC2005 (19 July, 2005, Sendai)
20
Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm
fffgg dPPP ,,Marginal Likelihood
y
x
fgfgfg dPPQ ,,ln',',,',',Q-Function
.,,maxarg1,1 :Step-M
.,,ln,,,, :Step-E
,ttQtt
dPttPttQ
fgfgfIterate the following EM-steps until convergence:EM Algorithm
A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).
21RC2005 (19 July, 2005, Sendai)
One-Dimensional One-Dimensional SignalSignal
EM Algorithm
i
i
i
0 127 255
0 127 255
0 127 255
100
0
200
100
0
200
100
0
200
if
ig
if
Original Signal
Degraded Signal
Estimated Signal
40
22RC2005 (19 July, 2005, Sendai)
Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model
Original ImageOriginal Image Degraded ImageDegraded Image
MSE: 1529MSE: 1529
MSE: 1512MSE: 1512
EM Algorithm with Belief Propagation
RC2005 (19 July, 2005, Sendai)
23
Exact Results of Gaussian Graphical Model
fCffgf
Cffgfgf
dP
T22
T22
21
21exp
21
21exp
,,
g
CICg
CICg 2
T2 2
1expdet2det,
P
gCIf12ˆ
Multi-dimensional Gauss integral formula
y
x
,max argˆ,ˆ,
gP
24RC2005 (19 July, 2005, Sendai)
Comparison of Belief Propagation with Comparison of Belief Propagation with Exact Results in Gaussian Graphical ModelExact Results in Gaussian Graphical Model
2ˆ||
1MSE
i
ii ff
MSEMSE
Belief Belief PropagationPropagation 327327 0.0006110.000611 36.30236.302 -5.19201-5.19201
ExactExact 315315 0.0007590.000759 37.91937.919 -5.21444-5.21444
ˆ,ˆln gP
MSEMSE
Belief Belief PropagationPropagation 260260 0.0005740.000574 33.99833.998 -5.15241-5.15241
ExactExact 236236 0.0006520.000652 34.97534.975 -5.17528-5.17528
ˆ,ˆln gP
40
40
,max argˆ,ˆ,
gP
25RC2005 (19 July, 2005, Sendai)
Image Restoration by Image Restoration by Gaussian Graphical ModelGaussian Graphical Model
Original ImageOriginal Image
MSE:315MSE:315MSE: 325MSE: 325
MSE: 545MSE: 545 MSE: 447MSE: 447MSE: 411MSE: 411
MSE: 1512MSE: 1512
Degraded ImageDegraded Image Belief PropagationBelief Propagation
Lowpass FilterLowpass Filter Median FilterMedian Filter
Exact
Wiener Filter
2ˆ||
1MSE
i
ii ff
26RC2005 (19 July, 2005, Sendai)
Original ImageOriginal Image
MSE236MSE236MSE: 260MSE: 260
MSE: 372MSE: 372 MSE: 244MSE: 244MSE: 224MSE: 224
MSE: 1529MSE: 1529
Degraded ImageDegraded Image Belief PropagationBelief Propagation
Lowpass FilterLowpass Filter Median FilterMedian Filter
Exact
Wiener Filter
2ˆ||
1MSE
i
ii ff
Image Restoration by Gaussian Image Restoration by Gaussian Graphical ModelGraphical Model
27RC2005 (19 July, 2005, Sendai)
Extension of Belief PropagationExtension of Belief Propagation
Generalized Belief PropagationGeneralized Belief PropagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, Transactions on Information Theory, 5151 (2005). (2005).
Generalized belief propagation is equivalent Generalized belief propagation is equivalent to the cluster variation method in statistical to the cluster variation method in statistical mechanicsmechanicsR. Kikuchi: A theory of cooperative phenomena, Phys. Rev., R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 8181 (1951). (1951).T. Morita: Cluster variation method of cooperative phenomena and its T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, generalization I, J. Phys. Soc. Jpn, 1212 (1957). (1957).
28RC2005 (19 July, 2005, Sendai)
Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model
2ˆ||
1MSE
i
ii ff
MSEMSE
Belief Belief PropagationPropagation 327327 0.0006110.000611 36.30236.302 -5.19201-5.19201
Generalized Generalized Belief Belief
PropagationPropagation315315 0.0007580.000758 37.90937.909 -5.21172-5.21172
ExactExact 315315 0.0007590.000759 37.91937.919 -5.21444-5.21444
ˆ,ˆln gP
MSEMSE
Belief Belief PropagationPropagation 260260 0.0005740.000574 33.99833.998 -5.15241-5.15241
Generalized Generalized Belief Belief
PropagationPropagation236236 0.0006520.000652 34.97134.971 -5.17256-5.17256
ExactExact 236236 0.0006520.000652 34.97534.975 -5.17528-5.17528
ˆ,ˆln gP
40
40
,max argˆ,ˆ,
gP
29RC2005 (19 July, 2005, Sendai)
Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters
2
,,,
ˆ
yx
yxyx ff||
1MSE
MSEMSE MSEMSE
Belief Belief PropagationPropagation 327327 Lowpass Lowpass
FilterFilter(3x3)(3x3) 388388
(5x5)(5x5) 413413
Generalized Generalized Belief Belief
PropagationPropagation315315 Median Median
FilterFilter(3x3)(3x3) 486486
(5x5)(5x5) 445445
ExactExact 315315 Wiener Wiener FilterFilter
(3x3)(3x3) 864864
(5x5)(5x5) 548548
40
GBPGBP
(3x3) Lowpass(3x3) Lowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener
30RC2005 (19 July, 2005, Sendai)
Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters
2
,,,
ˆ
yx
yxyx ff||
1MSE
MSEMSE MSEMSE
Belief Belief PropagationPropagation 260260 Lowpass Lowpass
FilterFilter(3x3)(3x3) 241241
(5x5)(5x5) 224224
Generalized Generalized Belief Belief
PropagationPropagation236236 Median Median
FilterFilter(3x3)(3x3) 331331
(5x5)(5x5) 244244
ExactExact 236236 Wiener Wiener FilterFilter
(3x3)(3x3) 703703
(5x5)(5x5) 372372
40
GBPGBP
(5x5) Lowpass(5x5) Lowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener
RC2005 (19 July, 2005, Sendai) 31
ContentsContents1. Introduction2. Belief Propagation3. Bayesian Image Analysis and Gaussian
Graphical Model4. Image Segmentation5. Concluding Remarks
RC2005 (19 July, 2005, Sendai) 32
Image Segmentation by Image Segmentation by Gauss Mixture ModelGauss Mixture Model
i
iaP γa
iii
ii
afaa
P 222
1exp2
1,,
σμaf
204321
,1924 ,1923,1272 ,641
i
K
k
i
kkf
kk
PP
P
12
2
2exp
2
,,
,,
aγaσμaf
γσμfGauss Mixture Model
γσμf
γaσμaf
γσμfa
,,,,
,,,
PPP
P
RC2005 (19 July, 2005, Sendai) 33
Image Segmentation by Combining Image Segmentation by Combining Gauss Mixture Model with Potts Model Gauss Mixture Model with Potts Model
Bijaa
ii jia
ZP ,
PRexp1
γγa
iii
ii
afaa
P 222
1exp2
1,,
σμaf
204321
,1924 ,1923,1272 ,641
γσμf
γaσμaf
γσμfa
,,,,
,,,
PPP
P
Belief PropagationBelief Propagation
iaia
ifif
iaiˆaPotts Model
34RC2005 (19 July, 2005, Sendai)
Image SegmentationImage Segmentation
Original Image Histogram Gauss Mixture Model
Gauss Mixture Model and Potts Model
Belief Belief PropagationPropagation
0101.05 ,4.145 ,8.2245
3982.04 ,7.114 ,4.16843375.03 ,6.233 ,6.13030711.02 ,0.182 ,2.422
1831.01 ,7.21 ,7.121
35RC2005 (19 July, 2005, Sendai)
Motion DetectionMotion Detection
SegmentationAND
Detection
ba
cb
Gauss Mixture Model and Potts Model with Belief PropagationGauss Mixture Model and Potts Model with Belief Propagation
Segmentation
a
b
c
RC2005 (19 July, 2005, Sendai) 36
ContentsContents1. Introduction2. Belief Propagation3. Bayesian Image Analysis and Gaussian
Graphical Model4. Image Segmentation5. Concluding Remarks
37RC2005 (19 July, 2005, Sendai)
SummarySummaryFormulation of belief propagationFormulation of belief propagationAccuracy of belief propagation in Bayesian Accuracy of belief propagation in Bayesian image analysis by means of Gaussian image analysis by means of Gaussian graphical model (Comparison between the graphical model (Comparison between the belief propagation and exact calculation)belief propagation and exact calculation)Some applications of Bayesian image Some applications of Bayesian image analysis and belief propagationanalysis and belief propagation
38RC2005 (19 July, 2005, Sendai)
Related ProblemRelated Problem
fgffggff ddPP ,,,ˆ 2
Statistical Performance Spin Glass Theory
H. Nishimori: Statistical Physics of Spin Glasses and Information Processing: An Introduction, Oxford University Press, Oxford, 2001.
f g fP ,fgP g ffggf PPP ,,,
fgfg dPff ii ,,,,ˆ
RC2005 (19 July, 2005, Sendai)RC2005 (19 July, 2005, Sendai) 3939
確率的情報処理の動向確率的情報処理の動向田中和之・樺島祥介編著 , “ ミニ特集 / ベイズ統計・統計力学と情報処理” , 計測と制御 2003 年 8 月号.田中和之,田中利幸,渡辺治 他著,“連載 / 確率的情報処理と統計力学 ~様々なアプローチとそのチュートリアル~”,数理科学 2004 年 11 月号から開始.田中和之,岡田真人,堀口剛 他著,“小特集 / 確率を手なづける秘伝の計算技法 ~古くて新しい確率・統計モデルのパラダイム~”,電子情報通信学会誌 2005 年 9 月号