Abstract We discuss the basic concepts of computer vi-sion with stochastic partial differential equations (SPDEs).In typical approaches based on partial differential equa-tions (PDEs), the end result in the best case is usually onevalue per pixel, the “expected” value. Error estimates oreven full probability density functions PDFs are usually notavailable. This paper provides a framework allowing oneto derive such PDFs, rendering computer vision approachesinto measurements fulfilling scientific standards due to fullerror propagation. We identify the image data with ran-dom fields in order to model images and image sequenceswhich carry uncertainty in their gray values, e.g. due tonoise in the acquisition process. The noisy behaviors ofgray values is modeled as stochastic processes which areapproximated with the method of generalized polynomialchaos (Wiener-Askey-Chaos). The Wiener-Askey polyno-mial chaos is combined with a standard spatial approxi-
T. Preusser (�)Center of Complex Systems and Visualization, BremenUniversity, Bremen, Germanye-mail: [email protected]
H. Scharr · K. KrajsekInstitute for Chemistry and Dynamics of the Geosphere,Institute 3: Phytosphere, Forschungszentrum Juelich GmbH,Juelich, Germany
R.M. KirbySchool of Computing and Scientific Computing and ImagingInstitute, University of Utah, Salt Lake City, UT, USAe-mail: [email protected]
mation based upon piecewise multi-linear finite elements.We present the basic building blocks needed for computervision and image processing in this stochastic setting, i.e.we discuss the computation of stochastic moments, projec-tions, gradient magnitudes, edge indicators, structure ten-sors, etc. Finally we show applications of our frameworkto derive stochastic analogs of well known PDEs for de-noising and optical flow extraction. These models are dis-cretized with the stochastic Galerkin method. Our selectionof SPDE models allows us to draw connections to the clas-sical deterministic models as well as to stochastic imageprocessing not based on PDEs. Several examples guide thereader through the presentation and show the usefulness ofthe framework.
In computer vision applications, e.g. medical or scientificimage data analysis, as well as in industrial scenarios, im-ages are used as input measurement data. Of course it isgood scientific practice that proper measurements must beequipped with error estimates (Gauss 1987; de Laplace1812). Thus, for many applications not only the measuredvalues, but also their errors should be—and more and moreare—taken into account for further processing. This errorpropagation must be done for every processing step, suchthat the final result comes with a reliable precision estimate.Unfortunately, for realistic models the computation of er-ror propagation is sometimes difficult or cumbersome, and
therefore most contributions dealing with error estimates arerestricted in one or more of the following:
• Data is presumed to be Gaussian distributed —the error isthen represented by a variance (cf. e.g. Nestares and Fleet2003; Weber and Malik 1994);
• Bounds on the error are derived only (cf. e.g. Nestares etal. 2000; Nestares and Fleet 2003);
• No true error estimates, confidence measures only are de-rived (cf. e.g. Bruhn et al. 2005; Haussecker et al. 1998).
This paper presents a framework for the treatment ofcomputer vision models on images and image sequenceswhose gray values are not assumed to be single values onlybut distributions of values. The methodology is not restrictedto Gaussian distributions, and the output is not restricted toerror bounds. In fact the distributions can be much richerthan those that are completely described by the two valuesmean and variance.
The key concept is an identification of image data withrandom fields. Thereby we identify the gray values of im-ages with random processes which are supposed to modelthe behavior of image detector elements and the influenceof noise. We approximate those stochastic processes withthe method of generalized polynomial chaos (gPC) andsupplement this approximation with a spatial discretiza-tion by Finite Element shape functions. This leads us toan ansatz-space for stochastic images and image-sequences.Computer vision models and algorithms using this con-cept of stochastic images as random fields transform theinput distributions into output distributions without loos-ing information on the precision due to approximated errorbounds.
We look at computer vision models which originate fromthe minimization of energies through the solution of Euler-Lagrange equations, or which come from other partial differ-ential equations (PDEs). It is straight forward to augment thedeterministic PDEs with the stochastic setting by replacingclassical image functions by the stochastic analog. Specialcare, however, must be taken for nonlinear operators whichlead to a coupling of the stochastic modes and moments. Forstandard tools in computer vision we discuss the stochasticgeneralization. With the help of projections and mass lump-ing in the stochastic space we are able to write down simpleequations for the computation of means, variances, and co-variances, as well as gradient magnitudes, edge indicators,and structure tensors. Those operations include the calcula-tion of stochastic integrals which can be computed in ad-vance and stored in lookup tables.
We demonstrate the usage of the formalism by exem-plary implementations of very well known and thus wellunderstood algorithms: Gaussian smoothing via isotropicdiffusion, Perona-Malik isotropic nonlinear diffusion (Per-ona and Malik 1990), variational optical flow estimation
by Horn-Schunck (1981), a version with a robust smooth-ing term (Black and Anandan 1991; Cohen 1993; Weickert1998) and optical flow computation with a regularized dataterm (Bruhn et al. 2005). These algorithms are prototypesof linear and nonlinear, energy-based computer vision ap-proaches for regularization, noise suppression, and parame-ter estimation with a wealth of practical applications. Theyare applied to very simple, thus easy to interpret, test data inorder to show the benefits and limitations of the formalismput forward here. We do not advocate that these algorithmsare the ‘best’ for certain applications (cf. e.g. Hausseckerand Spies 1999; Papenberg et al. 2006; Amiaz and Kiryati2006 for state of the art optical flow approaches): The ad-vantages and disadvantages of the exemplary algorithms arewell known. Here they are only used for didactical reasonsand as demonstrators of the formalism. This enables thereader to draw conclusions and connections to deterministicimage processing as well as stochastic image processing notbased on PDEs.
Although we present our stochastic framework in com-bination with a classic finite element approach it is possibleto combine the stochastic Galerkin method with finite dif-ference schemes as well. In the present work we have cho-sen the former approach since there exists a wide variety offunctional analytic tools for finite element methods.
Benefit of the novel approach Results of the above men-tioned algorithms for the mean (or more precisely expectedvalue) calculated with the new formalism do not signifi-cantly differ from results a standard finite element algorithmwould give. In fact, the new formalism often boils downto a standard deterministic implementation when we modelthe input distribution only by its mean. The benefit of ournew approach lies in the handling of distributions: It al-lows for precise, local and data depending error estimationsbeyond the Gaussian assumption. Under some assumptionsconcerning the smoothness of the output process, the calcu-lated output distributions converge to the same distributionsone would get when running infinitely many Monte-Carlo(MC) simulations of the applied algorithm and projectingthe resulting probability density function into our stochasticansatz space. Thus, our approach outperforms MC in termsof accuracy and computational efficiency, as the full knowl-edge about the input distribution is used to calculate the out-put distribution, not only (few) samples from the input dis-tribution.
Related work The use of PDEs in computer vision hasbeen popular during the last decades. Mostly, those PDEsare the necessary conditions (Euler-Lagrange equations) forminima of certain energy functionals. Approaches to de-noising, restoration, in-painting, segmentation, registration,optical flow estimation, etc. , and combinations of the latterare too numerous to give a short overview here.
Int J Comput Vis (2008) 80: 375–405 377
To the best of the authors’ knowledge, SPDEs have notyet been applied in computer vision to transform input dis-tributions into output distributions, but they are well es-tablished tools in other disciplines. Based on the Wiener-Hermite polynomial chaos expansion (Wiener 1938), thestochastic Galerkin method has been applied to a range ofproblems in computational mechanics (Meecham and Jeng1968; Chorin 1971; Chorin 1974; Maltz and Hitzl 1979;Deb et al. 2001; Le Maître et al. 2002; Xiu and Kar-niadakis 2002; Xiu and Karniadakis 2003; Lucor et al.2004). This technique has also recently been introduced intoother disciplines such as thermodynamics (Ghanem 1999;Narayanan and Zabaras 2004; Xiu and Karniadakis 2003)and physical chemistry (Reagan et al. 2004; Reagan et al.2005), in part because it leads to efficient solutions to sto-chastic problems of interest, i.e. not only parameter sensi-tivity and uncertainty quantification.
Our contribution shall not be confused with statisticalparameter selection methods, e.g. via Stochastic Differen-tial Equations (SDEs) (Bao and Krim 2004) or Markov-Random-Field assumptions (Scharr et al. 2003). Howeverour framework yields general extensions to the stochas-tic interpretation of energy functionals presented in Scharr(2006).
Paper organization In Sect. 2 we review some notion fromthe theory of probability and introduce the reader to the the-ory of random fields. We derive a way of identifying im-ages and image-sequences with random fields. Thereby wecombine the approximation of stochastic processes by theWiener-Askey Polynomial Chaos with the standard multi-linear interpolation/approximation schemes in space. In thefollowing Sect. 3 we discuss basic building blocks for com-puter vision with images as random fields. There we alsoanalyze the structure of the resulting block operators andtheir assembly which involves the computation of integralsin the random space. In Sect. 4 the stochastic generaliza-tion of some well known PDEs used in computer vision ispresented and discretized with help of the building blocks.We consider a linear and a nonlinear diffusion (Perona andMalik 1990) model, the optical flow extraction with theHorn and Schunck approach (Horn and Schunck 1981),a robust smoothing term for the optical flow field (Black andAnandan 1993; Cohen 1993; Weickert 1998), and finally acombined local global (CLG) approach (Bruhn et al. 2005).An investigation of the bias of the CLG model further un-derlines the usefulness of our framework. Conclusions aredrawn and an outlook is given in Sect. 6. In the Appendixwe summarize all building blocks and stochastic PDE mod-els in order to support the understanding of the material, thereimplementation of the models, and the reproduction of ourresults.
2 Stochastic Images as Random Fields
In the following sections (see Sects. 2.1–2.3) we give theexact mathematical definition of our ansatz space.
The core idea we put forward is that, in the presence ofnoise, gray values in each pixel are samples from per pixeldistributions. We model these per-pixel distributions or, tobe more precise, their Probability Density Functions (PDF)in contrast to standard image processing approaches whereonly the expected value or single samples from the distrib-utions are modeled. Roughly speaking, each pixel stores arepresentation of the random field at that point (with corre-sponding PDF) instead of a single value, i.e. in the follow-ing we consider images and image sequences with uncer-tain gray values as realizations of random fields. Therebywe model the uncertainty of the gray values with randomprocesses. The stochastic finite element space we introducein the sections below is continuous in space as is typical inFEM approaches. It is not only continuous in the spatial do-main, where it consists of a standard bi-linear interpolationscheme, represented by compact interpolation functions Pi ,where i indicates pixel position (cf. Fig. 2), but also the sto-chastic domain. In the stochastic domain per-pixel randomvariables ξi with uniform distribution are applied. The uni-form distributions of ξi are transformed into proper, contin-uous PDFs using process functions. These process functionsare approximated as weighted sums of polynomial, orthogo-nal basis functions Hα , where α indicates the degree of thepolynomial. The weights f i
α describing a function in the sto-chastic finite element space are denoted modes, e.g. the firstmode of a gray value image u is an image containing the ex-pected gray values, the second mode is an image containingweights belonging to H 2—using a special selection of H 2,it is proportional to the standard deviation, but we also useother selections—etc.
2.1 Random Fields and Wiener-Askey Polynomial Chaos
First we review some background from the theory of prob-ability, define some notions, and review the Wiener-Askeygeneralized polynomial chaos (gPC). A good overview ofthe methodology we use in the following is given in Keese(2004).
Let (�, A,μ) be a complete probability space, where� is the event space, A ⊂ 2� the σ -algebra, and μ theprobability measure. Following (Xiu and Karniadakis 2002),we can represent any general second-order random processX(ω),ω ∈ � in terms of a collection of random variablesξ = (ξ1, . . . , ξN ) with independent components. Let ρi :�i → R
+ be the PDFs of the random variables ξi(ω),ω ∈�, and its image �i ≡ ξi(�) ∈ R be intervals in R for
378 Int J Comput Vis (2008) 80: 375–405
i = 1, . . . ,N . Then
ρ(ξ) =N∏
i=1
ρi(ξi), ∀ξ ∈ �
is the joint probability density of the random vector ξ withthe support
� =N∏
i=1
�i ⊂ RN.
As commented in Xiu and Karniadakis (2002), this al-lows us to conduct numerical formulations in the finite di-mensional (N -dimensional) random space �. Let us denoteL2(�) as the probabilistic Hilbert space (Malliavin 1997) inwhich the random processes based upon the random vari-ables ξ reside. The inner product (scalar-product) of thisHilbert space is given by:
〈a, b〉 =∫
�
(a · b) dμ =∫
�
(a · b)ρ(ξ)dξ
where we have exploited independence of the random vari-ables allowing to write the measure as product of measuresin each stochastic direction. For notational simplicity wewill often omit the integral in terms of integration againstthe probability measure as a shorthand for the decomposi-tion implied above. We similarly define the expectation of arandom process X ∈ L2(�) as:
E[X(ξ)] =∫
�
X(ξ) dμ =∫
�
X(ξ)ρ(ξ )dξ .
Considering a spatial domain D ⊂ Rd we define a set of
random processes which are indexed by the spatial positionx ∈ D:
f : D × � → R.
Such a set of processes is referred to as a random field(Keese 2004) which can also be interpreted as a function-valued random variable, because for every event ω ∈ �
the realization f (·, ξ(ω)) : D → R is a function on D.For a vector-space Y the class Y ⊗ L2(�) denotes thespace of random fields whose realizations lie in Y for a.e.ξ ∈ �. Throughout this paper we will use random fieldsf ∈ L2(D) ⊗ L2(�) such that f (·, ξ) ∈ L2(D) for almostall ξ ∈ �. In this case let us define the norm |||·||| as
|||f (x, ξ)|||2 = E[‖f (x, ξ )‖2L2(D)
]
=∫
�
∫
D
(f (x, ξ )
)2dx ρ(ξ)dξ ,
that is, |||·||| denotes the expected value of the L2-norm of thefunction f .
A key to the numerical treatment of a stochastic processf i ∈ L2(�i) is the expansion
f i(ξi) =p∑
α=1
f iαHα(ξi) (1)
with stochastic ansatz functions Hα(ξ). Those can be cho-sen according to the Wiener-Askey Polynomial Chaos Ap-proach (Xiu and Karniadakis 2002) (generalized PolynomialChaos, gPC) in which the functions Hα(ξ) are orthogonalpolynomials ranging up to pth order (that is, a polynomialof degree p − 1) and p must be chosen to be large enoughso that the solutions will meet the accuracy requirementsfor the particular system of interest. In what is to follow wewill denote the coefficients f i
α as modes of the stochasticprocess f i(·). Furthermore we denote the space spanned bythe polynomials with
Pp = span{Hα | α = 1, . . . , p}.
Convergence rates of the system depend on the choice oforthogonal polynomials for underlying probability densityfunctions of a random model parameter. Each probabilitydistribution has a corresponding optimal set of orthogonalpolynomials (Xiu and Karniadakis 2002); e.g., for Gaussianrandom functions, Hermite polynomials provide the bestconvergence, whereas Legendre polynomials are best uti-lized for functions of uniform distributions, etc.
Expansions like (1) exhibit fast convergence rates whenthe stochastic response of the system is sufficiently smoothin the random space, e.g., bifurcation behavior is absent. Un-like the Monte Carlo method, which is amazingly robust be-cause it uses effectively no information about the underlyingprocess to determine the sampling or reconstruction proce-dure, the gPC methodology attempts to use as much of theinherent structure (such as smoothness within the stochasticspace) as possible to make the methodology computation-ally tractable.
The corresponding PDF of a random process is obtainedthrough the branches of the derivative of the inversion of itsprocess-function. In many cases the derivative of the inversecan be obtained with the inverse function theorem, but if theinversion is analytically not feasible a binning and histogramof the function values of the process is beneficial.
In the experiments shown throughout the paper we haveselected Legendre polynomials to span the polynomialspace Pp . They are simple to use for modeling processes anddistributions having finite support, like uniform and trun-cated Gaussian distributions. In Fig. 1 we show how distri-butions ranging from uniform to Gaussian shaped processescan be approximated with Legendre polynomials.
Int J Comput Vis (2008) 80: 375–405 379
Fig. 1 Top row: For a randomprocess (left) that has beenmodeled with a linearcombination of 2 Legendrepolynomials we show theresulting probability densityfunction (right). Bottom row:For a random process that hasbeen modeled with a linearcombination of 4 Legendrepolynomials we show theresulting PDF
2.2 Stochastic Still Images
Let us assume we are examining still images. An ac-quisition process of such digital images yields noisy im-ages due to various technical and physical reasons (cf.Forsyth and Ponce 2003, Chap. 1). In this section our in-tention is to model the stochastic process of gray-valuemeasurement (i.e. the uncertain output of the detector el-ements of the camera) with the help of the methodol-ogy presented in the last section. For the sake of sim-plicity we restrict our presentation to two-dimensional im-ages. An extension to n-dimensional images is not diffi-cult.
We assume the pixels of the image are located at a regularquadrilateral grid of dimension {1, . . . ,N1} × {1, . . . ,N2}.So the image has N := N1N2 pixels and its degrees offreedom (DOF) lie on the vertices of a regular grid Gwith (N1 − 1)(N2 − 1) quadrilateral elements Ej . We de-note the set of vertices of G with I and order the pix-els xi ∈ I lexicographically, i.e. from left to right, fromtop to bottom. Classically, we introduce a finite elementspace by using a bi-linear interpolation scheme (Preusserand Rumpf 1999). This means, we consider the finite-
dimensional space
V h2 := span{Pi |Pi ∈ C0(D),Pi(xj ) = δij ,
Pi |E is bilinear ∀E ∈ G} ⊂ H 1(D)
on the domain D = [1,N1] × [1,N2] ⊂ R2. Above, δ
is the Kronecker-δ. So the space V h2 is spanned by the
classical piecewise linear tent-functions Pi , which areequal to one at xi and vanish at every other vertex (seeFig. 2 for their support). The space H 1(D) denotes theSobolev space H 1,2(D) of functions having square inte-grable weak derivatives. Every image f ∈ V h
2 has a rep-resentation
f (x) =∑
i∈If iPi(x) (2)
where the vector of degrees of freedom is (f 1, . . . , f N) ∈R
N .Let us now assume that gray values of the pixels reveal
some uncertainty and thus have a random distribution. Inmore detail, we assume that the behavior of a pixel at loca-tion xi is determined by a separate stochastic process whichdepends on a random variable ξi . Furthermore we assume
380 Int J Comput Vis (2008) 80: 375–405
Fig. 2 We model the random behavior of the detector elements by ran-dom variables ξi each having a spatial extent which is given by finiteelement shape functions Pi . Left: For 2D images or still 3D images thebehavior of each pixel is modeled by a separate (independent) random
variable. Right: For image sequences the behavior of pixels which havethe same spatial coordinate (but may be located in different frames) ismodeled by the same random variable
that these random variables are independent and that theyare all supported on the same domain �i = �∗ ⊂ R. Theindependence is based on the physical assumption of inde-pendence of the noise of each detector element or camerapixel. Thus, for a still image with N pixels, the stochasticspace � = �N∗ is N -dimensional and ξ = (ξ1, . . . , ξN ) is avector of random variables.
As described in the previous section, the stochasticGalerkin method represents any stochastic process, X(ξ),by a weighted sum of orthogonal polynomials (Ghanem andSpanos 1991; Xiu and Karniadakis 2002; Xiu and Karni-adakis 2002; Xiu and Karniadakis 2003). These polynomialsare functions of a vector of independent random variables,ξ(ω),ω ∈ �, of known distribution. In the case of this study,the random processes of interest are the (stochastic) grayvalues attributed to each pixel. The random variables will bechosen to represent the distributions from which gray valuesare sampled.
Consequently, our ansatz space is the tensor productspace V h ⊗ Pp ⊂ H 1(D) ⊗ L2(�), i.e. we are consideringrandom fields whose realizations are functions in V h. Sincethe random variables for the pixels are independent, an im-age f ∈ V h
2 ⊗ L2(�) decomposes into
f (x, ξ) =∑
i∈If i(ξi)Pi(x). (3)
This means that the behavior of a pixel with spatial-extent(support) Pi is modeled by the stochastic process f i(ξi) (cf.Fig. 2).
Combining (1) and (3), we construct a finite-dimensionalspace Hh,p
still := V h2 ⊗ Pp containing discrete random fields
f (x, ξ) =∑
i∈I
p∑
α=1
f iα Hα(ξi)Pi(x). (4)
With notational abuse we can also write f (x, ξ ) =∑p
α=1 fα(x)Hα(ξ) where fα(x) = ∑i∈I f i
αPi(x). So thefα are the images that show the stochastic mode α of the
pixel’s processes. Altogether, the stochastic image has pN
DOF and we use the ordering
F := (F1; . . . ;Fp)
:= (f 11 , . . . , f N
1 ; . . . ;f 1p , . . . , f N
p ) ∈ RpN . (5)
Remark 1 Indeed we have constructed a proper finite di-mensional sub-space of H 1(D) ⊗ L2(�). Consequently thestochastic images (interpreted as discrete random fields)have weak spatial derivatives as typical for FEM methods.Moreover, setting p = 1 we reduce to the classical finite el-ement discretization and Hh,p
still ≡ V h2 .
2.3 Stochastic Image Sequences
Let us now consider an image sequence consisting ofNt frames. Each frame has dimension M := N1N2, thusits pixels lie on a regular quadrilateral grid of dimen-sion {1, . . . ,N1} × {1, . . . ,N2}. The complete image se-quence has N := N1N2Nt pixels and its degrees of free-dom (DOF) lie on the vertices of a regular grid G with(N1 − 1)(N2 − 1)(Nt − 1) hexahedral elements Ej . For thelater use it is convenient to introduce a special indexing ofthe vertices of this grid G , which has a spatial and a tem-poral component. Within each frame we order the verticeslexicographically as we did for still images in the previoussection. For the spatio-temporal ordering we use a multi-index i = (ix, it ) such that a pixel yi = y(ix ,it ) in the imagesequence is then referred to with its index ix ∈ {1, . . . ,M}within the frame it ∈ {1, . . . ,Nt }. In the following we de-note the multi-index set with J = {1, . . . ,M}× {1, . . . ,Nt },moreover we use the abbreviation y = (t, x).
Classically, we identify the image sequence with a three-dimensional image and a trilinear interpolation scheme(Mikula et al. 2004). This means that we use a bilin-ear interpolation within the frames and an additional lin-ear interpolation between the frames, i.e. we define the
Int J Comput Vis (2008) 80: 375–405 381
space
V h3 := span{Pi |Pi ∈ C0(D × I ),Pi(yj ) = δij ,
Pi |E is trilinear ∀E ∈ G} ⊂ H 1(D × I )
on the spatio-temporal domain D × I = [1,N1] × [1,N2] ×[1,Nt ] ⊂ R
3. In the following we will also use the no-tation R := D × I . In the definition, δij = δixjx δit jt isthe Kronecker-δ applied to the multi-indices i and j . Ofcourse, every image sequence has a representation analogto (2).
To derive a model for stochastic image sequences wemust take into account that each frame is recorded by thesame set of detector/camera elements. Thus, a sequence withNt frames and N = N1N2Nt degrees of freedom is modeledby N1N2 random variables ξi only (cf. Fig. 2). This meansthat the random variables ξi are time independent. However,it does not mean that PDFs described with them are time in-dependent. Temporal changes of the PDFs are modeled bytemporal changes of the stochastic modes f i
α . In fact, theacquisition of gray values for pixels yi and yj of the imagesequence is modeled by the same time-dependent stochasticprocess if ix = jx . Consequently we have to modify the ex-pansion (4) such that it involves Hα(ξix ) instead of Hα(ξi).
Our ansatz space Hh,pseq := V h
3 ⊗ Pp ⊂ L2(D × I ) ⊗ L2(�)
has dimension pN and the discrete random fields have theexpansion
f (y, ξ) =∑
i∈I
p∑
α=1
f iαHα(ξix )Pi(y) =
p∑
α=1
fαHα(ξ). (6)
Again by abusing the notation we can define image se-quences fα(y) := f i
αPi(y) showing the stochastic modesand thus f (y, ξ) = ∑p
α=1 fα(y)Hα(ξ). Let us finally notethat we have again constructed a proper finite dimensionalsub-space Hh,p
seq ⊂ H 1(D × I ) ⊗ L2(�).
Remark 2 The discretization defined in (6) indeed yieldstime-dependent processes as can be seen as follows: Assumewe have discretized each frame of the sequence by standard2D tent-functions Q2
j (x) such that a frame with stochas-
tic data has the representation∑M
j=1 fj (ξj )Q2j (x). The sto-
chastic processes fj (ξj ) model the random behavior of thepixel j with spatial extent spanQ2
j . Considering an imagesequence the stochastic process must be time-dependent,i.e. the image has the expansion
f (t, x, ξ) =M∑
j=1
fj (t, ξj )Q2j (x).
Now we discretize the stochastic processes by gPC andpiecewise linear expressions in time, i.e.
fj (t, ξj ) =p∑
α=1
Nt∑
k=1
(fj )kαHα(ξj )Q
1k(t),
where Q1j are 1D tent-functions. Putting these discretiza-
tions together we obtain
f (y, ξ) = f (t, x, ξ )
=M∑
j=1
p∑
α=1
Nt∑
k=1
(fj )kαHα(ξj )Q
1k(t)Q
2j (x),
which is the same as (6) if we set Pi(y) = Q2j (x)Q1k(t)
and f iα = (fj )
kα for the multi-index i = (j, k). Here we use
the fact that the standard nD tent-functions are tensor prod-ucts of the 1D tent functions. We have chosen to presentthe discretization as above since it is more consistent withthe standard discretization of, e.g. the optical flow equationswith finite elements, although the notation is slightly morecomplicated.
3 Building Blocks for Computer Vision with StochasticFinite Elements
In the section above we introduced finite dimensional ansatzspaces that model stochastic still images (Hh,p
still ) as well as
stochastic image sequences (Hh,pseq ). We now present some
tools that are needed for image processing tasks with thoseansatz spaces. Later we will use these building blocks to dis-cretize the stochastic analog of some well known models forimage denoising and optical flow computation. The presen-tation in this section is based on the ansatz space Hh,p
still forstochastic still images. We emphasize that a modification forthe ansatz space for stochastic image sequences Hh,p
seq is verysimple.
Throughout this section we are going to use the followingproperties of the polynomial basis Hα , which in fact are sat-isfied by possible choices (Hermite, Legendre, etc.) of basisfunctions for the gPC approach:
• The first basis function is constant,
H 1 ≡ 1 and such that E[H 1] = 1,
• The basis is orthogonal with respect to the measure,
i.e. for α �= β we have 〈Hα,Hβ〉 = 0.
(7)
As a simple consequence of those properties we directly getfor α > 1 that the basis functions Hα have zero mean, i.e.E[Hα] = 〈Hα,H 1〉 = 0.
382 Int J Comput Vis (2008) 80: 375–405
3.1 Mean, Variance, and Covariance
An analysis of the stochastic images involves the stochas-tic moments of the images’ distributions. From the full dis-cretization (4) we can compute the mean of the random fieldf as
E[f ](x) =∫
�
f (x, ξ) ρ(ξ)dξ
=∑
i∈I
p∑
α=1
f iαPi(x)
∫
�
Hα(ξi) ρ(ξ)dξ
=∑
i∈If i
1Pi(x) (8)
because only the stochastic integral over H 1(ξi) (the meanmode) does not vanish.
From here we can proceed using the identity Var [f ] =E[f 2] − (E[f ])2 to obtain
Var [f ](x) = E[f 2](x) − (E[f ])2(x)
=∫
�
∑
i,j∈I
p∑
α,β=1
f iαf
jβ Hα(ξi)H
β(ξj )
× Pi(x)Pj (x)ρ(ξ )dξ
−∑
i,j∈If i
1fj
1 Pi(x)Pj (x).
Because of the orthogonality of the polynomials Hα thestochastic integral 〈Hα(ξi),H
β(ξj )〉 vanishes if i = j andα �= β . For the same reason only the term for α = β = 1 re-mains if i �= j . Because the first mode is constant equal toone we have 〈H 1,H 1〉 = E[H 1] = 1 and thus
Var [f ](x) =∑
i∈I
p∑
α=1
(f iα)2〈Hα,Hα〉P 2
i (x)
+∑
i,j∈Ii �=j
f i1f
j
1 Pi(x)Pj (x)
−∑
i,j∈If i
1fj
1 Pi(x)Pj (x)
=∑
i∈I
p∑
α=1
(f iα)2〈Hα,Hα〉P 2
i (x)
−∑
i∈I(f i
1 )2P 2i (x)
=∑
i∈I
p∑
α=2
(f iα)2〈Hα,Hα〉P 2
i (x).
This expression is not an element of the physical finite ele-ment space V h
2 any longer as we have the square of our linear
finite element basis functions (recall that our finite elementspace consists of only affine functions). But we can use astandard interpolation (or nodal evaluation) to represent theterm in V h
2 . This leads us to
Var[f ](x) =∑
i∈I
p∑
α=2
(f iα)2〈Hα,Hα〉Pi(x). (9)
We see that the square at Pi(x) vanished due to this approx-imation.
Along the same line formulas for higher stochastic mo-ments like skewness, curtosis, etc. can be derived.
Let us now assume that we have two random images f
and g whose covariance we would like to compute. Usingthe identity Cov [f,g](x) = E[(f − E[f ])(g − E[g])] weget
Cov [f,g](x) =∫
�
∑
i,j∈I
p∑
α,β=2
f iαg
jβHα(ξi)H
β(ξj )
× Pi(x)Pj (x)ρ(ξ )dξ .
As above, the only non-zero terms in these sums are fori = j and α = β . Together with the nodal evaluation of theresulting spatial terms this yields
Cov [f,g](x) =∑
i∈I
p∑
α=2
f iαgi
α〈Hα,Hα〉Pi(x). (10)
Here again a term Pi(x) vanished due to nodal evaluation.As expected we get Cov [f,f ] = Var [f ] from our expres-sions.
3.2 Projections
Quite often in computer vision (nonlinear) functions of thegray values or their derivatives must be evaluated. In the fol-lowing we present a recipe for the treatment of such an eval-uation with stochastic images. So let us consider a functiong : Hh,p
still → L2(R)⊗L2(�) of a discrete stochastic image f
having an expansion as in (4). Examples for g are gradientmagnitudes and edge-indicator functions like
g(u) = ∇u · ∇u, and g(u) = (1 + |∇u|2/λ2)−1. (11)
Both functions are well known and often needed in imageprocessing, e.g. for the Perona-Malik diffusion (Perona andMalik 1990).
In general the result of the application of g on u doesnot lie in Hh,p
still any longer. For classical image processingwith finite elements, this problem arises as well. There, e.g.approximations of the gradient with finite differences or in-exact quadrature rules are used as a remedy. It seems ap-pealing to use such approximations in the stochastic case as
Int J Comput Vis (2008) 80: 375–405 383
well. However, we emphasize that for nonlinear quantitiesspecial care must be taken, since a coupling of the stochas-tic modes (cf. Fig. 4) takes place, which may be difficult tocapture with such approximations.
To obtain an expansion of g(u) of the form (4), we com-pute a L2-projection gh,p(x, ξ) of g(u) onto Hh,p
still . The pro-jection is defined by the orthogonality relation
E
[∫
D
gh,p(x, ξ)Hβ(ξj )Pj (x) dx
]
= E
[∫
D
g(u)(x, ξ )Hβ(ξj )Pj (x) dx
].
for β = 1, . . . , p and for all j ∈ I . Substituting the expan-sion (4) of gh,p into this orthogonality yields
∑
i∈I
p∑
α=1
⟨Hα(ξi),H
β(ξj )⟩ ∫
D
giαPi(x)Pj (x) dx
= E
[∫
D
g(u)(x, ξ )Hβ(ξj )Pj (x) dx
](12)
for β = 1, . . . , p and for all j ∈ I . This is a linear systemof equations for the coefficients (gi
α)i,α of the projectiongh,p . Denoting the vector of coefficients with G = (gi
α)i,αthe system can be written as
MG = R with(13)
R =(
E
[∫
D
g(u)(x, ξ )Hβ(ξj )Pj (x) dx
])
j,β
where M = ((Mα,β)ij ) is the stochastic block-mass matrix
(Mα,β)ij = ⟨Hα(ξi),H
β(ξj )⟩ ∫
D
Pi(x)Pj (x) dx (14)
whose blocks correspond to the modes of u. In Sect. 3.6 wediscuss the mass matrix and its assembly in more detail.
The desired expansion of g is given by the solution of thissystem which involves the inversion of the stochastic massmatrix: G = M−1R. This inversion of M may be computa-tionally intensive. However, using mass lumping (Thomee1984) can simplify the effort enormously, since it diagonal-izes the stochastic mass matrix. Lumping of masses yields ablock diagonal mass matrix M such that
(Mα,β)ij = δi,j δα,β
∑
k∈I
p∑
γ=1
(Mα,γ )ik
= δi,j δα,β
∑
k∈I
p∑
γ=1
⟨Hα(ξi),H
γ (ξk)⟩
×∫
D
Pi(x)Pk(x) dx.
In Sect. 3.6.1 we describe how to compute this mass matrix.For the assembly of the right hand side R we have differ-ent options. Usually one would use a stochastic quadraturerule to evaluate the expectation in (13). However, if a directexpansion of g as a product of expansions like (4) is avail-able, we can proceed differently as we shall see in the nextparagraph.
3.3 Gradient Magnitude
Let us use the projection discussed above to derive an ex-pression for the gradient magnitude of a stochastic image.We consider g(u) = |∇u|2 = ∇u · ∇u and insert this intothe system of (13). Directly using mass lumping leads us to
giα = (Mα,α)−1
ii E
[∫
D
(∇u · ∇u)(x, ξ )Hα(ξi)Pi(x) dx
].
Using the basis representation (4) of ∇u we can write ∇u ·∇u as
(∇u · ∇u)(x, ξ )
=∑
j,k∈I
p∑
β,γ=1
ujβuk
γ Hβ(ξj )Hγ (ξk)∇Pj (x) · ∇Pk(x).
If we order the DOF of u in the vector U as in (5) we canderive from this the block-system
giα = M−1
(i,α),(i,α)U · K(i,α) U, (15)
where the block-matrix K(i,α) is defined by
((K(i,α))
β,γ)j,k
= ⟨Hα(ξi)Hβ(ξj ),H
γ (ξk)⟩
×∫
D
Pi(x)∇Pj (x) · ∇Pk(x) dx.
Again, the blocks of this matrix correspond to the modesof u. Here, K(i,α) is not a block-diagonal matrix, thus thereis a coupling between the modes of U . This is the reasonwhy standard approximations on the modes (like finite dif-ferences for each mode) do not yield the correct result. InSect. 3.6.1 we discuss the assembly and the structure of thismatrix in more detail.
In Figs. 3 and 4 we show the computation of the gradientmagnitude on a test image. For the computations we haveused p = 3 and Legendre polynomials as basis functions forthe stochastic processes on �∗ = [−1,1]. From the imagesit is clearly visible how the modes are coupled through thenonlinear-operators, i.e. the spatial variation of the varianceis visible in the mean of the gradient-magnitude. Conversely,we also observe that the variance captures the gradients ofthe mean input image. The spatial resolution of the test im-age is 257 × 257, the gray values range in the interval [0,1],and we have used a value of λ = 0.01 (see (11)).
384 Int J Comput Vis (2008) 80: 375–405
Fig. 3 We show the stochasticmodes for the test evaluation ofthe gradient magnitude and theedge-indicator (cf. Fig. 4) on atest image. One realization ofthe distribution modeled in thisexample is shown in the top leftimage. On the right thestochastic modes used in theexpansion (4) are depicted.According to (8) the meancorresponds to the first mode ofthe expansion. The variancemust be computed from theremaining modes accordingto (9). We have used a colorcoding as shown in the colorramp (bottom left) to make adifferentiation between positiveand negative values possible.Note that we have scaled theimages to match the full colorrange. Thus, the colors give aqualitative impression only
Fig. 4 For a sample image (left)we show the gradient magnitude(middle) and an edge-indicator(right). The expectations areshown in the top row, whereasthe variances are shown in thebottom row. Note that we showthe gradient images with acontrast enhancement to bettervisualize their global variance
3.4 Edge-Indicator Function
The computation of the gradient magnitude was simplifiedby taking mass lumping into account. Furthermore it ben-efited from the fact that we can directly write down the ex-pansion of ∇u ·∇u as a product of sums. The setting is morecomplicated if we consider a nonlinear edge indicator func-tion, e.g.
g(v) = (1 + v/λ2)−1, (16)
where v is the representation of |∇u|2 whose coefficientshave been derived in (15). As above, we insert this functioninto the right hand side R (cf. (13)) and get for G = M−1R
giα = (Mα,α)−1
ii E
[∫
D
g(v)(x, ξ)Hα(ξi)Pi(x) dx
]
= (Mα,α)−1ii
∫
�
∫
D
g(v)(x, ξ)Hα(ξi)Pi(x) dx ρ(ξ)dξ .
To be more specific let us substitute the actual edge-indicator function g(v) = (1 + v/λ2)−1 and the expansion(4) for v. This leads us to the identity
Int J Comput Vis (2008) 80: 375–405 385
giα = (Mα,α)−1
ii
∫
�
∫
D
Hα(ξi)Pi(x)
1 + λ−2∑
j∈I∑p
β=1 Vjβ Hβ(ξj )Pj (x)
dx ρ(ξ)dξ (17)
which is only computable with a quadrature rule in thephysical and the stochastic space. This will be discussed inSect. 3.6.1.
In Figs. 3 and 4 we show the evaluation of the edge-indicator function, where we have used the quadrature rulesdescribed in the next section. Again we have used Legendrepolynomials and p = 3. It is again clearly visible from theimages how the stochastic modes are coupled through thenonlinear edge-indicator function.
3.5 Diffusion- and Structure-Tensors
The concepts presented in the last paragraph can easily begeneralized to tensor-valued functions. If we consider e.g.the structure tensor J = (∇f )T ∇f of a stochastic image f
we need to compute the stochastic representation of deriv-atives (∂mf )(∂nf ). Those quantities, however, can be ob-tained with the projection technique from Sect. 3.3. We justhave to replace the expansion of ∇u ·∇u with the expansionof the desired product of derivatives (∂mf )(∂nf ). In Fig. 5we show the three components of the structure tensor of thetest image from Fig. 4.
3.6 Stochastic Integrals
In the last sections we have defined multiple quantitieswhich involve integration over the random space �. In this
section we describe how to evaluate these high-dimensionalintegrals and how to use quadrature rules to compute coeffi-cients as in (17). Although � is an M-dimensional space,where M is the number of random variables that modelthe stochastic behavior of the image or image sequence, thecomputation of the integrals is not very complicated. Let usfirst focus on the matrices and tensors that appeared in theprevious sections. Then we discuss the numerical stochas-tic quadrature that is used to compute the coefficients of theedge-indicator.
3.6.1 Stochastic Matrices
During the last sections we have encountered the inner prod-uct on the random space multiple times, i.e. integrals overpairs or triples of stochastic basis functions. Those inte-grals are the coefficients that are multiplied with integralsin the physical space e.g. like for coefficients of the stochas-tic mass matrix (14). In general it is possible to separate thestochastic and the spatial integration such that in most casesthe computation of integrals reduces to a component-wisemultiplication of an integral over the random space with anintegral over the physical space.
The expectations of products of tuples of stochastic ba-sis functions play a central role in the concept presented inthis work. In the section on a stochastic Galerkin methodfor diffusion and optical flow models, we will need those
Fig. 5 For the test image shownin Fig. 4 we depict the stochasticmoments of the structure tensor.Note that again we haveenhanced the contrast of thegray values for the presentation
386 Int J Comput Vis (2008) 80: 375–405
expectations again. So let us focus on the integral⟨Hα(ξi),H
β(ξj )⟩
= E[Hα(ξi)H
β(ξj )] =
∫
�
Hα(ξi)Hβ(ξj ) ρ(ξ)dξ
=∫
�1
· · ·∫
�M
Hα(ξi)Hβ(ξj )ρ(ξ1) dξ1 · · ·ρ(ξM)dξM.
Already for images of moderate size M a computation ofthis high dimensional integral seems not feasible. But for-tunately, we have to integrate over a small number (here 2)stochastic coordinates (random variables) only. Moreover,the values of the integrals do not depend on the actual phys-ical location of the corresponding pixel (here the values of i
and j ). Only we have to decide whether the locations coin-cide, if the pixels are neighbors, or if their spatial extent doesnot overlap. So without loss of generality, we can assumethat the spatial locations of the random variables lie within areference-element (as is standard in finite element methods(Thomee 1984)) and thus attain the values {1, . . . ,4}.
Having this simplification in mind, we can easily com-pute the expectation of products of pairs, triples and quadru-ples of stochastic basis functions, which are the quanti-ties that we need for the concept presented in this work.For every choice of polynomial basis (Legendre, Hermite,Laguerre, etc. ) we can store the values in a lookup ta-ble.
For the later use we define the tensors
Ai,j,k,lα,β,γ,δ =
⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨
⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩
∫
�1
Hα(ξ1)Hβ(ξ1)Hγ (ξ1)Hδ(ξ1) ρ(ξ1)dξ1, i = j = k = l,
∫
�1
∫
�2
Hα(ξ1)Hβ(ξ1)Hγ (ξ1)Hδ(ξ2) ρ(ξ1)dξ1ρ(ξ2)dξ2, i = j = k �= l,
...permutations of threeequal latin indices,
∫
�1
∫
�2
∫
�3
Hα(ξ1)Hβ(ξ2)Hγ (ξ3)Hδ(ξ3) ρ(ξ1)dξ1 · · ·ρ(ξ3)dξ3, i, j �= k = l, i �= j,
...permutations of twoequal latin indices,
∫
�1
∫
�2
∫
�3
∫
�4
Hα(ξ1)Hβ(ξ2)Hγ (ξ3)Hδ(ξ4) ρ(ξ1)dξ1 · · ·ρ(ξ4)dξ4all latin indicesare different,
Bi,j,kα,β,γ =
⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨
⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩
∫
�1
Hα(ξ1)Hβ(ξ1)Hγ (ξ1) ρ(ξ1)dξ1, i = j = k,
∫
�1
∫
�2
Hα(ξ1)Hβ(ξ1)Hγ (ξ2) ρ(ξ1)dξ1ρ(ξ2)dξ2, i = j �= k,
... permutations of twoequal latin indices,
∫
�1
∫
�2
∫
�3
Hα(ξ1)Hβ(ξ2)Hγ (ξ3) ρ(ξ1)dξ1 · · ·ρ(ξ3)dξ3, i �= j, i �= k, j �= k,
Ci,jα,β :=
⎧⎪⎪⎪⎨
⎪⎪⎪⎩
∫
�1
Hα(ξ1)Hβ(ξ1) ρ(ξ1)dξ1, i = j,
∫
�1
∫
�2
Hα(ξ1)Hβ(ξ2) ρ(ξ1)dξ1ρ(ξ2)dξ2, i �= j,
(18)
which give us the desired lookup-tables for all possiblecombinations of tuples of basis functions and their loca-tions within the reference-element. We have used the factthat |�i | = ∫
�iρ(ξ)dξ i = ∫
�iρi(ξi)dξi = 1 for any ran-
dom variable ξi . Using (7) we can easily derive someproperties of the tensors A, B , and C, e.g. we can check
whether an entry is zero due to reasons of orthogonal-ity.
Int J Comput Vis (2008) 80: 375–405 387
Let us finally note that the storage of the integrals overtuples of spatial basis functions or their derivatives in lookuptables is a standard approach in the numerical treatment offinite element methods (cf. e.g. Preusser and Rumpf 1999).In our setting many of the matrix entries and integrals canbe computed by pairwise multiplication of an entry from thestochastic lookup table with an entry from the spatial lookuptable.
3.6.2 Stochastic Quadrature
To concretize the quadrature announced in Sect. 3.4 (17),for the computation of the edge-indicator function we usean M-dimensional quadrature rule for the stochastic inte-gral. This quadrature shall be the tensor product of a d-pointone-dimensional quadrature rule. Thereby we use the fact(cf. Sect. 2.2) that all random processes are defined on thesame domain �i = �∗. Let us denote the weights of theone-dimensional quadrature by {κ1, . . . , κd} ∈ R
d and thequadrature points with {z1, . . . , zd} ∈ �d∗ . Thus, we have tocompute
giα ≈ (Mα,α)−1
ii
d∑
l1,...,lM=1
κl1 · · ·κlM Hα((zl)i )
×∫
D
g(v)(x,zl)Pi(x) dy,
where we set zl = (zl1 , . . . , zlM ) ∈ � such that (zl)i = zli .Fortunately, the spatial basis function Pi has compact sup-port and the random variables are coupled to the support ofspatial basis functions. Thus we must integrate over thosestochastic variables only, whose supporting spatial basisfunction intersects the support of Pi . Splitting the integralover D into a sum of integrals over the elements E of thegrid G yields
giα ≈ (Mα,α)−1
ii
∑
E∈G
d∑
lj =1j=1,...,M
xj ∈E
(∏
j
κlj
)Hα((zl )i)
×∫
E
g(v)(x,zl )Pi(x) dy.
So on each element E ∈ G the M-dimensional stochastic in-tegral reduces to a four-dimensional stochastic integral be-cause every element E has four vertices.
For the particular edge-indicator function from Sect. 3.4this means
giα ≈ (Mα,α)−1
ii
∑
E∈G
d∑
lj =1j=1,...,M
xj ∈E
(∏
j
κlj
)Hα(zli )
×∫
E
Pi(x)
1 + λ−2∑
lk=1k=1,...,M
xk∈E
∑p
β=1 V kβ Hβ(zlk )Pk(x)
dy.
Although it appears complicated, this integral can easily becomputed with a traversal of the grid and local operationson the elements E.
4 Stochastic Galerkin Method for Diffusion Filtering
In this section we show the usage of the stochastic finite ele-ment discretization and the building blocks presented in thepreceding sections. To do so, we apply them to stochasticversions of well known and simple partial differential equa-tions frequently used in computer vision. We have chosena selection of very basic and simple diffusion equations forwhich the computer vision community has gained a wideunderstanding and insight.
Namely, first we present linear diffusion of an image u
via ∂tu = �u, with initial condition u(t = 0) = f and where� denotes the spatial Laplacian. This describes the temporalevolution of u, creating the so-called Gaussian Scale-Space(Iijima 1962, 1963; Witkin 1983). And second we presenta nonlinear diffusion of u via ∂tu = div(g(|∇u|)∇u), alsocalled Perona-Malik diffusion (Perona and Malik 1990).
Our intention is to enable the reader to gain understand-ing of computer vision with SPDEs and to see the connec-tions to classical/deterministic approaches with PDEs on thebasis of those simple models. We are neither going to ad-dress issues of existence and uniqueness of solutions of thecontinuous SPDEs, nor are we going to discuss the spacesin which solutions reside. Instead, we point the interestedreader to the ongoing research in the area of Galerkin FEMfor SPDEs (cf. e.g. Deb et al. 2001; Keese 2004; Ghanemand Spanos 1991; Xiu and Karniadakis 2002).
Discretization of SPDEs is completely analogous to clas-sical FEM discretizations of PDEs, when having the sto-chastic ansatz spaces from Sect. 2 at hand:
1. temporal derivatives are exchanged by an Euler scheme,2. the resulting equations are converted into their respective
weak formulation1 by projection onto test functions z andintegrating by parts, and
3. solution images u are represented by their expansions.
This yields a system of equations which can be written inblock form. The system is then solved for the coefficientsof the expansion of u. We will now examine each of theaforementioned diffusion problems in detail.
1The weak formulation of a PDE means that it does not have to holdpoint-wise everywhere (strong form), but only when integrated againsta test function. This relaxes the problem: instead of deriving an exactsolution everywhere, we derive a solution satisfying the strong form onaverage over the test-function domain.
388 Int J Comput Vis (2008) 80: 375–405
4.1 Linear Diffusion Filtering
We begin with the SPDE formulation of the most promi-nent linear filter, the heat equation which yields the GaussianScale-Space (Iijima 1962, 1963; Witkin 1983). We definethe classical diffusion PDE
∂tu = �u (19)
as an evolution equation acting on the image gray values u.Instead of the usual intensity image u(t, x), we directly usethe gray-value functions u(t, x, ξ) with the random fieldsintroduced in Sect. 2. Thus, the stochastic version of the heatequation reads:
Given initial data f (x, ξ) find a family {u(t, x, ξ)}t∈R+of filtered versions of this image such that
∂tu(t, x, ξ ) − �u(t, x, ξ ) = 0 a.e. in R+ × D × �,
∂νu(t, x, ξ ) = 0 a.e. on R+ × ∂D × �,
u(0, x, ξ) = f (x, ξ) a.e. in D × �.
(20)
We will now discretize this equation step-by-step. Firstlet us apply the temporal backwards Euler scheme. Thismeans we replace the temporal derivative by a back-ward difference quotient ∂tu(t, x, ξ ) ≈ (u(t, x, ξ ) − u(t −τ, x, ξ))/τ , introducing the time-step τ . Using the notationun(x, ξ) := u(nτ, x, ξ) we obtain:
For n = 1,2,3, . . . find un : D × � → R such that
un(x, ξ) − τ�un(x, ξ) = un−1(x, ξ) a.e. in D × �,
∂νun(x, ξ) = 0 a.e. on R
+ × ∂D × �,
(21)
where u0(x, ξ) = f (x, ξ) a.e. in D × �.
Consequently we have transferred the parabolic SPDE (20)into a sequence of elliptic SPDEs (21). Each of the equa-tions of this sequence must be interpreted in a weak sense:Following the stochastic Galerkin Method (Deb et al. 2001),we first need to project each equation to a set of test func-tions z in the same way as done for a standard GalerkinMethod (Thomee 1984). Second, we integrate the expres-sion by parts. The difference is that in the stochastic case allz reside in the space H 1(D)⊗L2(�) introduced in Sect. 2.2.The projection is done as follows: We multiply each equa-tion by a test-function z ∈ H 1(D) ⊗ L2(�), we integrateover the physical domain R and consider the expectation ofthe resulting integrals. This yields
E
[∫
D
un(x, ξ) z(x, ξ ) dx − τ
∫
D
�un(x, ξ) z(x, ξ ) dx
]
= E
[∫
D
un−1(x, ξ) z(x, ξ ) dx
]
for all test-functions z ∈ H 1(D) ⊗ L2(�). Integrating byparts in D leads to the weak form of the SPDE (21)
E
[∫
D
un(x, ξ) z(x, ξ ) dx − τ
∫
D
∇un(x, ξ) · ∇z(x, ξ ) dx
]
= E
[∫
D
un−1(x, ξ) z(x, ξ ) dx
]. (22)
Having derived the weak form, we need to represent the so-lution un in a finite-dimensional sub-space, i.e. we considerun ∈ Hp,h
still . To do so, we substitute the expansion (4) intothis weak form. In addition to this we plug in the basis func-tions Hβ(ξj )Pj (x) as test-functions z. Using the linearity ofthe expectation to pull the coefficients (un)iα and (un−1)iα infront of the integrals we get
∑
i∈I
p∑
α=1
(un)iα
(E
[∫
D
Hα(ξi)Pi(x)Hβ(ξj )Pj (x) dx
]
− τE
[∫
D
(Hα(ξi)∇Pi(x)
) · (Hβ(ξj )∇Pj (x))dx
])
=∑
i∈I
p∑
α=1
(un−1)iαE
[∫
D
Hα(ξi)Pi(x)Hβ(ξj )Pj (x) dx
]
for all j ∈ I and β = 1, . . . , p. Remember, the coefficients(un)iα are the stochastic modes of u at pixel positions i, i.e.images containing weights belonging to polynomials Hα oforder α.
This is a system of equations which can be written in ablock form
p∑
α=1
(Mα,β + τLα,β)(Un)α =p∑
α=1
Mα,β(Un−1)α
for β = 1, . . . , p (23)
ordering the unknowns of un as in (5) to get Un. The ma-trices Mα,β and Lα,β are stochastic mass- and stiffness-matrices, respectively. They have the entries
(Mα,β)ij = E
[∫
D
Hα(ξi)Pi(x)Hβ(ξj )Pj (x) dx
]
=(∫
�
Hα(ξi)Hβ(ξj ) ρ(ξ)dξ
)
×(∫
D
Pi(x)Pj (x) dx
)
= Ci,jα,β
(∫
D
Pi(x)Pj (x) dx
), (24)
Int J Comput Vis (2008) 80: 375–405 389
Fig. 6 Structure of the block system of the stochastic heat equa-tion (23)
(Lα,β)ij = E
[∫
D
(Hα(ξi)∇Pi(x)
) · (Hβ(ξj )Pj (x))dx
]
=(∫
�
Hα(ξi)Hβ(ξj )ρ(ξ)dξ
)
×(∫
D
∇Pi(x) · ∇Pj (x) dx
)
= Ci,jα,β
(∫
D
∇Pi(x) · ∇Pj (x) dx
)(25)
where we have used the tensor C from (18). The matri-ces result from the classical mass- and stiffness-matrices byan entry-wise multiplication with the expectation of pair-products of stochastic basis functions. The coefficient C
i,jα,β
is zero for α �= β due to the orthogonality of the basisfunctions Hα . Consequently the resulting system is block-diagonal (cf. Fig. 6) where each diagonal block correspondsto the smoothing of one stochastic mode of the image.
Remark 3 The fact that the stochastic heat equation leads toa block diagonal system is due to the linearity of the heatequation. Already in Sect. 3.3 we have seen that a nonlinearoperator couples the stochastic modes and thus results in adense system (in the stochastic space).
Due to the block structure we can implement the solutionof the stochastic heat equation very efficiently, because wecan use existing deterministic FEM code on each of the sto-chastic modes separately, provided we have multiplied thedeterministic system matrix component-wise with the ten-sor C
ijα,β . We are going to discuss results of the stochastic
linear diffusion in Sect. 4.4.
4.2 Perona Malik Diffusion
We will now implement a stochastic version of the wellknown nonlinear Perona Malik diffusion (Perona and Malik
1990) in order to demonstrate the influence of its non-linearbehavior. Proceeding analogously to Sect. 4.1 we formulatethis problem as:
Given initial data f (x, ξ) find a family {u(t, x, ξ)}t∈R+of filtered versions of this image such that
Here g is the edge indicator function we have alreadyworked with in Sect. 3.4.
Remark 4 It is known that the Perona Malik model suf-fers from an ill-posedness which leads to non-existence ofsolutions (Kichenassamy 1997). Since numerical schemesunavoidably introduce regularizations, the ill-posedness ingeneral does not lead to problems in practice. However, weemphasize that in our framework it is straight-forward to im-plement the regularized Perona-Malik model as introducedby Catté et al. (1992), which uses a smoothed version uρ
of the image u to compute the diffusion tensor g(|∇uρ |).In the FEM context this smoothing can be obtained by onescale step of the (stochastic) heat equation (cf. Sect. 5.5).
Again we apply the backward Euler approximation of thetemporal derivative and evaluate the non-linearity at the oldtime-step leading to a semi-implicit scheme. Interpreting theequation in a weak sense yields
E
[∫
D
un(x, ξ) z(x, ξ ) dx
−τ
∫
D
g(|∇un−1(t, x, ξ )|)∇un(x, ξ) · ∇z(x, ξ ) dx
]
= E
[∫
D
un−1(x, ξ)z(x, ξ ) dx
]
for all test functions z ∈ H 1(D) ⊗ L2(�). We observe thatonly the second term differs from its linear analog (22), andonly by the factor g(|∇un−1(t, x, ξ )|), i.e. the nonlinearitywhich is treated explicitly here. Considering un ∈ Hh,p
still asabove leads to a system of equations
∑
i∈I
p∑
α=1
(un)iα
(E
[∫
D
Hα(ξi)Pi(x)Hβ(ξj )Pj (x) dx
]
− τE
[∫
D
(p∑
γ=1
∑
k∈I(Gn)kγ Hγ (ξk)Pk(x)
)
× (Hα(ξi)∇Pi(x)
) · (Hβ(ξj )∇Pj (x))dx
])
390 Int J Comput Vis (2008) 80: 375–405
=∑
i∈I
p∑
α=1
(un−1)iαE
[∫
D
Hα(ξi)Pi(x)Hβ(ξj )Pj (x) dx
]
for each β = 1, . . . , p and j ∈ I . Here we have substitutedthe expansion (4) for the edge indicator g(|∇un|) which hasbeen derived in Sect. 3.4, i.e. (Gn)kγ denotes the stochasticmodes of g.
Remark 5 For the classical deterministic diffusion equationwe have to assert that the diffusion tensor is positive def-inite such that the resulting equation is elliptic and the re-sulting bilinear form coercive. An analogous condition musthold for the stochastic diffusion equation (Deb et al. 2001),and we have to take care of the positivity of the stochas-tic process of the diffusion tensor g. In case the positiv-ity is violated, the stochastic process describing the distri-bution of the diffusion tensor must be truncated such thatit stays away from zero. If f (ξ, x) describes the diffusiontensor at a fixed location x ∈ D, a truncated process wouldbe f (ξ, x) = max{c, f (ξ, x)} for some constant 0 < c � 1.Consequently, all integrations in the weak form of the equa-tion must use the truncated process f . However, in our nu-merical experiments we did not encounter any problemswith the projection and quadrature method described inSects. 3.4 and 3.6.2 and the resulting g was always positive.
The above system of equations is again a block-systemwhich can be written asp∑
α=1
(Mα,β + τ(Ln)α,β
)(Un)α =
p∑
α=1
Mα,β(Un−1)α
for β = 1, . . . , p
if we order the unknowns of un as in (5) to get Un. Themass-matrices Mα,β are as before in (24) and (Ln)α,β is now((Ln)α,β
)ij
= E
[∫
D
(p∑
γ=1
∑
k∈I(Gn)kγ Hγ (ξk)Pk(x)
)
× (Hα(ξi)∇Pi(x)
) · (Hβ(ξj )Pj (x))dx
]
=p∑
γ=1
∑
k∈I(Gn)kγ
(∫
�
Hα(ξi)Hβ(ξj )Hγ (ξk)ρ(ξ)dξ
)
×(∫
D
∇Pi(x) · ∇Pj (x)Pk(x) dx
)
=p∑
γ=1
∑
k∈I(Gn)kγ B
i,j,kα,β,γ
(∫
D
∇Pi(x) · ∇Pj (x)Pk(x) dx
).
(26)
This block system is not block-diagonal any more becauseof the stochastic diffusion tensor g which leads to an entry-
wise multiplication and summation with the tensor B in theblock system.
Integration of the entries of (Ln)α,β can be simplified byusing an inexact integration scheme. If E denotes one hexa-hedral element of the grid, we can use
p∑
γ=1
∑
k∈I(Gn)kγ B
i,j,kα,β,γ
(∫
E
∇Pi(x) · ∇Pj (x)Pk(x) dx
)
≈ Gi,jα,β
∫
E
∇Pi(x) · ∇Pj (x) dx
where
Gi,jα,β = 1
4
p∑
γ=1
∑
k∈I∩E
(Gn)kγ Bi,j,kα,β,γ
is the evaluation of the diffusion coefficient at the center-point of the element E. Using this approximation simplifiesimplementation of the matrix assembly, since existing FEMcode for the Perona-Malik model can be reused.
4.3 Numerical Aspects and Efficiency
In (24), (25), and (26) we have seen that for standard diffu-sion models the local stochastic mass- and stiffness-matricesresult from an entry-wise multiplication of the classicalmass and stiffness matrices with the coefficients of the ten-sors A, B , and C from (18). This means that in the standardcase the additional effort for the assembly of the local sto-chastic matrices is just one multiplication with the tensorcoefficient per local matrix entry. The tensors A, B , and C
as well as local standard mass- and stiffness-matrices can becomputed in advance and stored in lookup tables.
For the stochastic heat equation and the stochastic Per-ona Malik equation the resulting linear system of equationshas p × p blocks (cf. Fig. 10), where p is the number ofstochastic modes used in the expansion (4). Each of thoseblocks has the size of the corresponding classical determin-istic problem, i.e. N × N , where N is the number of pixelsin the image. Consequently the dimension of the stochasticlinear system is (pN) × (pN). Since the systems are sym-metric and positive definite they can be solved by standarditerative linear solvers like e.g. the conjugate gradient (CG)method (Avriel 2003).
As a consequence of the local support of the basis func-tions Pi the resulting systems are sparse. Therefore, matrixvector multiplications in the iterative CG solver need an ef-fort of O(p2N) in contrast to O(N) for the deterministicmatrices, since the number of bands is multiplied by p andthe number of unknowns is multiplied by p as well. Sincea CG solver converges after M steps if the number of un-knowns is M , we conclude that the effort of the stochastic
Int J Comput Vis (2008) 80: 375–405 391
Fig. 7 For a stochastic inputimage (mean and variancedepicted in the top left frame)we show several scale stepsfrom the linear diffusion (topframe) and the nonlinearPerona-Malik diffusion (bottomframe). In each frame the toprow shows the mean and thebottom row shows the variance.For the linear diffusion thevariance of the image drops bymore than one order ofmagnitude per scale step. Thusthe variance-images appearblack. In contrast to that, thecoupling of the image gradientonto the variance is nicelyvisible from the Perona-Malikimages. One realization of therandom field (i.e. a noisy image)is shown in the bottom leftimage
variant is equal to the deterministic effort multiplied by afactor of p2, i.e. O(p2N) instead of O(N).
The experiments shown in Fig. 8 below show that ourframework outperforms a naive approach like Monte-Carlo(MC) sampling. In this MC approach samples are drawnfrom the input distribution, i.e. different noisy images. Tothese images the standard, deterministic algorithm is ap-plied. Doing so, the output distribution is sampled. In fact,for two stochastic modes p = 2 the effort of our algorithmcorresponds to p2 = 4 Monte-Carlo samples for which agood approximation of the distribution in general is not ob-tained (cf. Fig. 8).
4.4 Results
In Fig. 7 we present results of the linear and nonlinear dif-fusion applied to a stochastic test-image. We have chosen towork with a stochastic basis containing Legendre polynomi-als up to order p = 2. The image is of resolution 129 × 129and the gray values range in the interval [0,1]. For the sakeof simplicity, the variance of the input data is defined by set-
ting the second mode to
f2(x) = δ |x|, δ = 0.3
129√
2
and letting all higher modes be zero. This means that wemodel a spatially varying uniform distribution of the inputgray values (cf. Fig. 1, top). In Sect. 5.7 we will modelGaussian-shaped distributions as well.
Let us note that for uniform distributions over an inter-val [c − d, c + d] the mean is given by c and the varianceis σ 2 = d2/3. Conversely, given the variance σ 2 the intervalhalf width is d = √
3σ . Using two modes for the descrip-tion of the stochastic process and setting the second modef2(x) = a means that we have a variance of
Var (f ) = a2〈H 2,H 2〉 = a2∫ 1
−1ξ2i ρi(ξi)dξi
= a2∫ 1
−1
ξ2i
2dξi = 1
3a2.
This results from the fact that the second Legendre basisfunction is H 2(ξi) = ξi and the PDF is ρi(ξi) = 1/2. Wesee that the second mode f2 = a directly represents the halfwith of the interval of a uniform distribution.
392 Int J Comput Vis (2008) 80: 375–405
Fig. 8 A comparison of thestochastic Perona-Malik modelwith Monte-Carlo experimentsis shown for the scale step(n = 6). As before, the top rowshows the mean and the bottomrow the variance. Left: Inputdistribution of a test image (cf.Fig. 7). Middle left: Result fromthe stochastic Perona-Malikmodel discretized with thestochastic Galerkin (SG)method. Middle right: MonteCarlo experiment with 4samples. Right: Monte CarloExperiment with 50 samples
In our example the variance of the input data ranges fromzero to
σ = maxx f2√3
= maxx δ|x|√3
= 0.3√3.
This means in our example we model an uncertainty of thegray values varying up to ±0.3 around the mean gray value.For the computations we have chosen a time-step of τ = 1and we use λ = 0.02 (see (11)).
Looking at the top frames in Fig. 7 we see, that no struc-ture from the expectation images (top) can be seen in thevariances (second row) and vice versa. We see that as ex-pected from theory the heat equation damps the modes sep-arately without coupling them. In fact the damping is veryfast, and the maximum value of the variance drops by morethan one order of magnitude per scale-step. Thus the imagesdepicted in the corresponding row of Fig. 7 are black. If weadjusted the contrast, the variances of the smoothed imagewould have the same structure as the input variance.
In contrast to that, the variance of the images fromPerona-Malik diffusion show the structure of the gradi-ent, i.e. we see the coupling of the modes. There is alsoa smoothing of the variance present with increasing scalestep n. In the limit n → ∞ the results of the linear and non-linear diffusion models will be the same, i.e. an image ofconstant gray value and zero variance.
In Fig. 8 we show a comparison of the stochastic Perona-Malik diffusion results with a Monte-Carlo experiment. Tothis end we have created samples of the input image distribu-tion shown in Fig. 7, applied our deterministic Perona-Malikdiffusion solver, and computed the mean and the varianceof the resulting images through the standard formulas fromstatistics. This experiment can be seen as a validation ofthe Galerkin approach for the stochastic Perona Malik equa-tion. Indeed the Monte-Carlo experiment seems to convergeto the result from our new approach. However, we empha-size that the new approach is much more efficient than the
Monte-Carlo method. To compute the mean and the varianceof the output the stochastic Galerkin method needs the effortof about 4 deterministic runs. From the figures we see thatwith just 4 samples the results from the Monte-Carlo methodis quite bad. This underlines the power of our framework.
Finally, in Fig. 9 we investigate the edge enhancing char-acter of the stochastic Perona Malik equation. Starting theevolution with a blurred version of the test-pattern alreadyused before we indeed get an enhancement of edges, whichhowever depends on the variance of the input data. Fromthe figure we see that the edge enhancement occurs first inregions with small variance (i.e. top left corner of the testimage). For later time steps the enhancement also affects ar-eas with larger variance. This is visible in the figures by anenhancement which progresses from the top left corner ofthe image to the lower right. In this example the couplingof the image gradients on the variance of the output is verymild. In fact, the structure of the image can be seen just veryweakly in the variance images. We use a time-step τ = 5 andan edge-indicator parameter λ = 0.002.
5 Optical Flow Computations
In the following section the stochastic finite element methodis applied to slightly more complex models for the wellknown optical flow problem. Intentionally we have chosenvery simple models such as:
• the Horn and Schunck (HS) model (Horn and Schunck1981),
• a discontinuity preserving optical flow model (Black andAnandan 1991; Cohen 1993; Weickert 1998), and
• the combined local global (CLG) model (Bruhn et al.2002; Bruhn et al. 2005).
Int J Comput Vis (2008) 80: 375–405 393
Fig. 9 The edge enhancingproperty of the stochasticPerona Malik model isinvestigated experimentally.From left to right: Input imagedistribution and three scale stepsof the stochastic Perona Malikmodel. Note the edgeenhancement, which occurs inregions of low variance, andwhich is progressing from thetop left corner of the image withincreasing scale
5.1 A Stochastic Optic Flow Equation
Let us first derive an optical flow equation for stochas-tic images. We consider a given noisy image sequence f :D × I → R on a spatial domain D ⊂ R
2 and a time intervalI := [0, T ] for T > 0. To distinguish spatial from spatio-temporal derivatives we introduce the notation ∇x,t for thespace-time gradient and ∇x for the purely spatial gradient.The partial derivatives are denoted with ∂1, ∂2, and ∂t and,finally, we write R := D × I .
We search for a vector field (optical flow field) w : R ×� → R
2 such that w = (u, v) describes the motion of struc-tures inside the image sequence f . Classically, gray valuesare preserved along trajectories x(t) of objects through theimage sequence. Here gray values are preserved along sto-chastic trajectories x(t, ξ), i.e.
f (x(t, ξ), t, ξ ) = const.
The stochastic trajectories yield a stochastic optical flowfield w(x, t, ξ) = x(t, ξ ). As for the classical optical flowequation, we differentiate the brightness constancy with re-spect to time t :
The temporal behavior of the image sequence is modeled bythe stochastic process f , but the vector of random variablesξ is time independent (cf. Sect. 2.3). This means that ξ doesnot change over time t (i.e. ξ = 0), thus we can omit thelast term. So we can formulate our stochastic optical flowconstraint as
This equation is completely analog to the classical opticalflow equation, and as such we observe the ill-posedness alsoknown as the aperture problem (see e.g. Haussecker andSpies 1999).
5.2 Stochastic Horn and Schunck Model
The standard HS model overcomes the aperture problem byselecting as the solution the flow field with minimal over-all gradient. By this we mean that the classical HS solutionminimizes the energy
EHS(w) = ||w · ∇xf + ∂tf ||2 + 1
2κ||∇x,tw||2
=∫
R
(w(y) · ∇xf (y) + ∂tf (y))2 dy
+ 1
2κ
∫
R
∇x,tw(y) · ∇x,tw(y) dy (28)
where the first term usually is called a data term and thesecond a smoothness or regularization term. Here we use aregularization in space and time which usually yields bet-ter results (Weickert and Schnörr 2001). Also the stochas-tic analog of the HS model regularizes the ill-posed opticflow equation by requiring the gradient of the flow field∇x,tw(x, t, ξ) to be small. However, the peculiarity of thestochastic approach is that we take the |||·||| norm as it hasbeen introduced in Sect. 2 instead of the L2-norm in (28).Thus, our energy involves the expectation
E(w) = |||w · ∇xf + ∂tf |||2 + 1
2κ|||∇x,tw|||2
= E
[∫
R
(w(y, ξ) · ∇xf (y, ξ) + ∂tf (y, ξ))2 dy
+ 1
2κ
∫
R
∇x,tw(y, ξ) · ∇x,tw(y, ξ) dy
](29)
394 Int J Comput Vis (2008) 80: 375–405
which in fact is the expectation of the classical HS energyEHS from (28) applied to stochastic image sequences, i.e.
E(w) = E[EHS(w(y, ξ))]. (30)
Remark 6 Equation (30) can be interpreted as follows: Ifwe insert a stochastic optical flow field into the classical HSenergy EHS we obtain a stochastic process that yields thedistribution of EHS with respect to ξ . Taking the expecta-tion within (30) means that we consider the expected energyfrom this distribution and minimize it.
As in the classical approach, we derive the Euler equa-tions as a necessary condition for a minimum of this en-ergy. As there, we vary the components u and v of theflow field w independently. To do so, we select a functionz : D × I × � → R
2 and compute a variation of the energyin direction zek for k = 1,2, where e1,2 are the Euclideanbasis vectors of R
2. This means k = 1 and 2 belong to u andv, respectively. Using the observation (30) we get
0 = d
dεE(w + εzek)
∣∣∣∣ε=0
= d
dεE[EHS(w + εzek)]
∣∣∣∣ε=0
= E
[d
dεEHS(w + εzek)
∣∣∣∣ε=0
]
were we have assumed the energies involved to be finite,such that we can interchange differentiation and integration.The above equation means that the Euler equations for thestochastic energy (29) are just the expectation of the Eulerequations of the classical energy EHS and read
0 = E
[∫
R
z(y, ξ ) ∂1f (y, ξ)
× (w(y, ξ) · ∇xf (y, ξ) + ∂tf (y, ξ)
)dy
+ κ
∫
R
∇x,tu(y, ξ) · ∇x,t z(y, ξ ) dy
],
0 = E
[∫
R
z(y, ξ ) ∂2f (y, ξ)
× (w(y, ξ) · ∇xf (y, ξ) + ∂tf (y, ξ)
)dy
+ κ
∫
R
∇x,t v(y, ξ) · ∇x,t z(y, ξ ) dy
].
(31)
Thus, the stochastic optical flow w is a solution of theSPDE-system
in the weak sense defined by (31). We note that this SPDEsystem is completely analog to the classical system whichresults from a minimization of the HS Energy.
5.3 Discretization
Let us derive the linear systems of equations which resultfrom the discretization of the Euler equations of the opti-cal flow energies (32). In the following we consider onlyone equation of the system, since the derivation for the otherequation is completely analog.
We start by substituting f (y, ξ), u(y, ξ), and v(y, ξ) withtheir respective Galerkin expansions (6) for stochastic im-age sequences into the weak form (cf. (31))
0 = E
[∫
R
z ∂1f (w · ∇f + ∂tf ) dy + κ
∫
R
∇u · ∇z dy
]
=: E[I ] + κE[II ].Together with the test function z = Hβ(ξjx )Pj (y) we get
E[I ] = E
[Hβ(ξjx )Pj (y)
∑
k∈J
p∑
γ=1
f kγ Hγ (ξkx )∂1Pk(y)
×(∑
i∈J∑p
α=1 uiαHα(ξix )Pi(x)
∑i∈J
∑p
α=1 viαHα(ξix )Pi(x)
)
×∑
l∈J
p∑
δ=1
f lδH
δ(ξlx )∇Pl(y)
]
+ E
[Hβ(ξjx )Pj (y)
∑
k∈J
p∑
γ=1
f kγ Hγ (ξkx )∂1Pk(y)
×∑
l∈J
p∑
δ=1
f lδH
δ(ξlx )∂tPl(y)
]
which can be collapsed to
E[I ] =∑
i∈J
p∑
α=1
(∑
k,l∈J
p∑
γ,δ=1
f kγ f l
δ
×∫
�
Hα(ξix )Hβ(ξjx )H
γ (ξkx )Hδ(ξlx ) ρ(ξ)dξ
×∫
R
Pi(y)Pj (y)∂1Pk(y)
× (∂1Pl(y)ui
α + ∂2Pl(y)viα
)dy
)
+∑
k,l∈J
p∑
γ,δ=1
f kγ f l
δ
×∫
�
Hβ(ξjx )Hγ (ξkx )H
δ(ξlx ) ρ(ξ)dξ
×∫
R
Pj (y)∂1Pk(y)∂tPl(y) dy. (33)
Int J Comput Vis (2008) 80: 375–405 395
Fig. 10 Left: Structure andsub-structure of the blocksystem (37). Right: Non-zeropattern of a block matrix for twostochastic moments (i.e. p = 2),a spatial size of 4 × 4 pixels and4 frames
The second term becomes
E[II ] =∑
i∈J
p∑
α=1
uiα
∫
�
Hα(ξix )Hβ(ξjx ) ρ(ξ)dξ
×∫
R
∇Pi(y) · ∇Pj (y) dy. (34)
This identity leads to the stiffness matrix Lα,β which we al-ready defined for the heat equation in (25). Moreover, usingthe tensors A and B from Sect. 3.6.2 we can define the ma-trices
(Sα,βmn )ij =
N∑
k,l∈J
p∑
γ,δ=1
f kγ f l
δAix,jx ,kx ,lxα,β,γ,δ
×∫
R
Pi(y)Pj (y)∂mPk(y)∂nPl(y) dy, (35)
as well as the vector
(Rβm)j =
∑
k,l∈J
p∑
γ,δ=1
f kγ f l
δBjx,kx ,lxβ,γ,δ
×∫
R
Pj (y)∂mPk(y)∂tPl(y) dy, (36)
for m,n = 1,2, i, j = 1, . . . ,N and α,β = 1, . . . , p. Now,we can write the discretized Euler equations as
p∑
α=1
((S
α,β
11 Sα,β
12
Sα,β
21 Sα,β
22
)+ κ
(Lα,β 0
0 Lα,β
))(Uα
V α
)
= −(
Rβ
1
Rβ
2
), for β = 1, . . . , p. (37)
This equation describes a block system of dimension 2pN ×2pN and consists of p blocks corresponding to the stochas-tic modes. The blocks themselves contain 2 × 2 sub-blockscorresponding to the two coordinate directions. Each blockis similar to the deterministic system and each sub-block isa N × N matrix. In Fig. 10 we show the structure of this
block-system. Again it is dense in the stochastic space (i.e.all blocks are non-zero), since the integrals involve multipli-cations with more than two factors. However, the stiffness-matrix Lα,β (corresponding to the smoothness term) appearson the block-diagonals only as we have already seen for theheat equation.
As seen above, the HS energy EHS from (28) consists ofa data term and a regularization term. The regularizationterm consists of a linear diffusion (cf. Sect. 4.1) on the ve-locity components u and v. Discontinuity-preserving opti-cal flow (Black and Anandan 1993; Cohen 1993; Weickert1998) contains the same data term as in EHS, but it uses anonlinear regularization term instead of a linear one. Thisnonlinear regularization boils down to Perona Malik non-linear diffusion as in Sect. 4.2. Having a discretization fornonlinear diffusion at hand, it is now straight forward togeneralize the discontinuity-preserving optical flow modelto a stochastic setting as well. This yields the SPDE sys-tem
∂1f (y, ξ)(w(y, ξ ) · ∇f (y, ξ) + ∂tf (y, ξ))
− κ div(g(|∇f (y, ξ)|)∇u(y, ξ )) = 0,
(38)∂2f (y, ξ)(w(y, ξ ) · ∇f (y, ξ) + ∂tf (y, ξ))
− κ div(g(|∇f (y, ξ)|)∇v(y, ξ )) = 0,
in R and a.e. �, which is interpreted in a sense analog to(31). To discretize the model we only have to replace thehomogeneous stiffness matrices Lα,β from (37) by the in-homogeneous one defined in (26). As for the HS energy weget a system of equations like (37) which has a structure asshown in Fig. 10.
Note that for this model, one can also use a regularizedversion fρ of the input image sequence inside the edge indi-cator function, i.e. g(|∇fρ(y, ξ)|). We emphasize that as forthe Perona-Malik model from Sect. 4.2 this extension can
396 Int J Comput Vis (2008) 80: 375–405
straight forwardly be incorporated within our framework.Here, in our FEM setting, the smoothing can be obtainedby e.g. one small scale step of length ρ2/2 the stochasticheat equation, which takes the image sequence f as initialdata.
5.5 Combined Local Global (CLG) Method
In Sect. 5.4, above, we exchanged the linear regulariza-tion term in the HS energy EHS by a nonlinear one. Herewe keep the linear regularization term, but exchange thedata term. Let us denote the homogeneous motion vectorwith w := (u, v,1). The classical HS energy from (28) thenreads
EHS(w) = ‖w · J w‖ + 1
2κ‖∇x,tw‖2 (39)
where we used the notation J = (∇x,tf )T (∇x,t f ) for theouter product of the spatio-temporal gradient ∇x,tf ofthe image sequence f . Smoothing this 3 × 3 tensor J
component-wise yields the so-called structure-tensor Jρ
(Jähne 1993). In a finite difference setting ρ indicates thevariance of a Gaussian kernel used to smooth J . Againwe consider Jρ as the solution of the heat equation (asin Sect. 4.1) for a small single time step of length ρ2/2with initial data J . Such a diffusion has to be appliedindependently to each structure tensor component Jmn,m,n ∈ {1,2, t}. Using Jρ instead of J in the HS en-ergy EHS yields the stochastic version of the CLG en-ergy
E(w) := |||w · Jρw||| + 1
2κ|||∇x,tw|||2.
As above Euler-Lagrange equations are given by the expec-tation of the Euler-Lagrange equations of the deterministicmodel, thus the solution obeys
0 = E
[∫
R
z(y, ξ )(J 11
ρ (y, ξ)u(y, ξ ) + J 12ρ (y, ξ)v(y, ξ)
+ J 1tρ (y, ξ)
)dx
+ κ
∫
R
∇x,tu(y, ξ) · ∇x,t z(y, ξ ) dx
],
0 = E
[∫
R
z(y, ξ )(J 21
ρ (y, ξ)u(y, ξ ) + J 22ρ (y, ξ)v(y, ξ)
+ J 2tρ (y, ξ)
)dx
+ κ
∫
R
∇x,t v(y, ξ) · ∇x,t z(y, ξ ) dx
].
The discretization is completely analog to the discretizationof the stochastic HS model from Sect. 5.2. To obtain matri-ces similar to (35) let us denote the coefficients of the ex-pansion (6) of the smoothed structure tensor with (Jmn
ρ )kγfor m,n ∈ {1,2, t}, pixel position k ∈ J and expansion or-der γ = 1, . . . , p and define
(Sα,βmn )ij =
N∑
k∈J
p∑
γ=1
(Jmnρ )kγ B
ix,jx ,kx
α,β,γ
×∫
R
Pi(y)Pj (y)Pk(y) dy, (40)
and a right hand side
(Rβm)j =
∑
k∈J
p∑
γ=1
(Jm,tρ )kγ C
jx,kx
β,γ
∫
R
Pj (y)Pk(y) dy.
The block system is the same as (37) where S and R havebeen replaced with the ones from above.
Using an inexact quadrature rule as in Sect. 4.2 we cansimplify the computation of the entries of S
p∑
γ=1
∑
k∈J(Jmn
ρ )kγ Bix,jx ,kx
α,β,γ
(∫
E
Pi(x)Pj (x)Pk(x) dx
)
≈ (J mnρ )
i,jα,β
∫
E
Pi(x)Pj (x) dx
where
(J mnρ )
i,jα,β = 1
8
p∑
γ=1
∑
k∈J ∩E
(Jmnρ )kγ B
ix,jx ,kx
α,β,γ
and similarly for the right-hand-side R.
5.6 Numerical Aspects and Efficiency
As for the standard diffusion models the local stochastic ma-trices result from entry-wise multiplications with the coeffi-cients of the tensors A, B and C from (18). However, in thepresent case of nonlinearities in (35), (36), and (40) there isalso a summation over the stochastic modes γ, δ = 1, . . . , p
used in the expansion (4). As for the diffusion models, manycomponents of the deterministic local matrices can be com-puted in advance and stored in lookup tables. Due to thesummation over the stochastic modes the assembly effortis multiplied by a factor of p2 for the stochastic Horn andSchunck model (35), and by a factor of p for the CLG ap-proach (40). However in the case of the CLG model we needadditional effort for the computation and the smoothing ofthe structure tensor.
Again the resulting linear systems of equations aresparse, symmetric, and positive definite. Thus we can use
Int J Comput Vis (2008) 80: 375–405 397
Fig. 11 We consider atest-sequence of a texturedsquare moving to the right(frame 5 shown in the top leftpicture). The gray values areuniformly distributed and fromtop to bottom we increase thevariance of the input data andshow the color coded opticalflow field. It is clearly visiblehow the mean and the varianceof the flow field capture thegradients of the input data.Moreover we see that withincreasing smoothness of theflow field the variancedecreases. The color wheel onthe lower left indicates the colorcoding of the flow directions
a CG method for the solution of the systems. Since weare dealing with vector valued problems, the dimension ofthe stochastic system is (2pN) × (2pN). As above for thediffusion models, the effort for the solution is multipliedby a factor of p2 in comparison to the deterministic sys-tem.
In the following section we will present an experimentthat shows how our framework outperforms a Monte-Carlosampling approach. For the computation of mean and vari-ance of the output we would need p = 2 modes. Thusthe effort of our framework is approximately equal to 4Monte-Carlo samples for which a convergence can not beexpected.
5.7 Results
Let us start with the computation of the optical flow field oftwo test-sequences. First we consider a disc which is filledwith a sin(cxx) sin(cyy) pattern and which moves to theright in front of a background that has a slight gradient inx-direction. For this first example we consider only two sto-chastic modes, thus approximating the distributions of theinput gray-values with a uniform distribution. In Sect. 3.1we have seen how the variance can be computed from themodes of the polynomial expansion.
In Fig. 11 we show results of our computations with thestochastic HS model. The mean gray values of the imagesequence are in the range [0,1]. In our experiment we have
398 Int J Comput Vis (2008) 80: 375–405
Fig. 12 From a sequenceshowing a moving texturedsquare (spatial resolution65 × 65, 11 frames, mean shownin (a), for the variance see thetext) we extract the optical flowwith the HS model (c), with thediscontinuity preservingmodel (d) and with the CLGapproach (e). The mean of theedge-indicator function isdepicted in (b). For the flowfields we show the maximumcomponent of the covariancematrix in the bottom row
considered variances of the input data f to be 2−6, 2−8 and2−10. This corresponds to errors in the input gray values of21%, 11%, and 5%, respectively. Moreover we have studiedthe influence of the smoothing parameter κ .
We see that in general there is a high variance of theflow field in the vicinity of edges which are orthogonal tothe direction of motion. This behavior can be interpreted asthe uncertainty of the location of edges of the moving ob-ject.
Fixing the smoothness parameter κ , we see from the im-ages that with increasing variance of the input data f themean of the flow field becomes more inhomogeneous in-side the moving square. The same behavior can be observedfor the variance of the flow field. In fact the mean and vari-ance capture the structure of the texture, because the modelis non-linear in the derivatives of f . In Sect. 3.3 and Fig. 4we have already seen that a nonlinear function couples thestochastic modes, thus an output variance is sensitive to gra-dients of the mean of the input data. For fixed variance ofthe input data and increasing smoothness κ the influence ofthe gradients in the input mean is weakened and thus theamplitude of the variance of the flow field is damped. Thisbehavior we have already seen from the heat equation inSect. 4.1 (cf. also Sect. 4.4). Still there is uncertainty aboutthe edges of the moving objects, thus the variance remainshigh in those regions.
In the second numerical test we consider the texturedsquare shown in Fig. 12 which moves to the right with unitspeed w = (u, v) = (1,0) in front of a textured background.The spatial resolution is of size 65 × 65 and the sequencehas 11 frames. We consider 4 stochastic modes (i.e. p = 4)and set the image shown in Fig. 12 to be the mean. The grayvalues of the mean image range from 0 to 1. Furthermorewe set
f2(y) ≡ 2.96281 · 10−2, f3(y) ≡ 0,
f4(y) ≡ 9.87604 · 10−2,
such that we model stochastic processes as the one shown inthe bottom row of Fig. 1.
In Fig. 12 we show the result of the optical flow compu-tation with the HS model, with the discontinuity preservingmodel and the CLG approach. We have used a smoothingparameter κ = 0.03, set λ = 1/300 and ρ = 0.0075. Ob-viously, for the edge preserving model the covariance ishigh in the vicinity of the edges, whereas the HS modelonly yields high variances for edges which are perpen-dicular to the motion direction. The CLG model shows asimilar behavior although the covariance is smaller sincethe smoothing is larger with the chosen set of parame-ters.
Again, we emphasize the benefit of our approach: Incontrast to existing work our Ansatz does not only yieldbounds for errors or just confidence measures. Indeed wetransform distributions of the input data into distributions ofthe output data. Such distributions of the velocity are de-picted in Fig. 13 for two different locations in the movingsquare sequence from the previous experiment. For a pixelinside the moving object (32/32/5) and for a pixel at itsupper border (32/14/5) the figure shows the resulting ran-dom processes of the u and v velocity component and theirPDFs.
Let us use our framework to examine the bias of the CLGapproach. The bias B is defined by the difference betweenthe true optical flow components w0 = (u0, v0) and expec-tation value E[w] of the estimated optical flow components
B = w0 − E[w].The bias occurring in optical flow estimators based on leastsquares estimation leading to an underestimation of the op-tical flow components u,v has been extensively examined(cf. e.g. Van Huffel and Vandewalle 1991; Kearney et al.1987; Fermüller et al. 2001). For local optical flow estima-tors, e.g. of Lucas-Kanade type (Lucas and Kanade 1981),the functional relationship between the estimated opticalflow components u0, v0 for noise free input signals (vari-ance σ 2 = 0) and the estimated optical flow components u,v
Int J Comput Vis (2008) 80: 375–405 399
Fig. 13 For two different locations in the moving square image se-quence we plot the stochastic processes of the x- and y-component ofthe velocity as well as the corresponding PDFs. In the top row we con-sider a location in the center frame of the sequence and in the centerof the moving object. In the bottom row a location in the center frame
and at the boundary of the moving square is considered. The processis approximated using polynomials of degree three (p = 4). As suchpolynomials are antisymmetric around their inflexion point, the PDF isapproximated to be symmetric around its maximum
Fig. 14 Left: We plot (red curve) the value of the x-component u ofthe extracted velocity versus the standard deviation σ of the gray valuesto investigate the bias of the CLG approach. The theoretical devolutionof the bias (black curve) depending on standard deviation matches ourresult very well. Right: The sample mean for two Monte Carlo exper-
iments, one with N = 4 (denoted as MC4, blue curve) and the otherwith N = 50 (denoted as MC50, red curve) is depicted. Again, thetheoretical devolution of the bias depending on standard deviation isshown (black curve). Note that the values on the horizontal axis aregiven in percent
400 Int J Comput Vis (2008) 80: 375–405
Fig. 15 The estimation ofoptical flow from the “streetsequence” is shown. Left: Frame110 of the sequence. Middle left:Optical flow of the deterministicsequence. Right: Optical flowfor the sequence, which hasbeen supplied with two differentvariances of the input grayvalues. Again, the mean of theoptical flow is depicted in thetop row, and the maximum entryof the covariance matrix isdepicted in the bottom row
for noisy input signals (σ 2 > 0) has been derived in Süh-ling (2006). This relationship can be extended to the CLG-approach leading to a function of the expectation value de-pending on the variance of the input signal
E[w] =(
τu
τu + σ2u0,
τv
τv + σ2v0
),
B =(
σ2
τu + σ2u0,
σ2
τv + σ2v0
).
Here, τu, τv denote real valued parameters that depend onthe input signal. Our framework allows to easily examinethe dependency of the bias on the variance of the input sig-nal. In Fig. 14 (left) we plot the estimate of the expecta-tion value E[u] of the x component of the optical flow, us-ing our stochastic Galerkin (SG) method, (the true under-lying flow vector is w = (u, v) = (1,0)) versus the stan-dard deviation σ (in percent) of the input data. In order todemonstrate the quantitative performance of our framework,we plot also the theoretical relationships for the expectationvalue of the estimated optical flow component. The theoret-ical curve plotted in both graphs is the same, thus enablingfor a comparison of the SG results with the MC experi-ments.
In Fig. 14 (right) the sample mean u = 1N
∑Nj=1 u(j) is
depicted resulting from two Monte Carlo experiments, onewith N = 4 samples (denoted as MC4) and the other withN = 50 samples (denoted as MC50). We see from the fig-ures that the optical flow estimate decreases for increas-ing noise in the input image. Consequently, the bias in-creases for increasing noise in the input image. We can ob-serve that the results from our approach correspond verywell to the theoretical curve. In our framework it sufficesto use two stochastic modes p = 2. In that case the ef-fort of our approach is about p2 = 4 times the effort ofa deterministic CLG approach (cf. Sect. 5.6). For the
Monte Carlo approach (N = 4) with the same computa-tional complexity, the sample mean shows significant fluc-tuations and cannot cope with the precision of our ap-proach.
As soon as we significantly increase the number of sam-ples (N = 50), we obtain a comparable result also for theMonte Carlo approach. But the price we have to pay for alikewise performance is an enormous increase of the num-ber of estimates from 84 (21 different noise levels times eneffort equivalent to 4 estimates per noise level) to 1050 (21different noise levels times 50 estimates per noise level) es-timates. With our framework the system of (37) with entries(40) must be solved only once per noise level. Thus, this ap-proach outperforms the naive MC simulation by a very largefactor. We conclude that our framework involves the poten-tial to analyze the bias of estimators for which it is not yetknow and for which the analytic derivation is cumbersomeor even unfeasible.
Our final numerical experiment deals with the estimationof optical flow from the “street sequence”. Since for this ar-tificially generated sequence we do not have any measure-ment errors, we set the variance of the input data homoge-neously to three different values. In Fig. 15 we show theresulting optical flow field as well as the maximum entriesof the covariance matrices of the distributions. The sequenceis of resolution 200 × 200 and its gray values are scaled tothe range [0,1]. We use two stochastic modes p = 2 and setthe parameters to κ = 0.1, ρ = 0.1. For a better comparisonwe depict the estimated optical flow from the deterministicnoise free sequence (Var ≡ 0) as well. As in our syntheticalexample we see a high variance of the optic flow in regionsof high gradient of the input image. Again, this is due tothe nonlinear dependence of the model on the input data (cf.Sects. 4.2 and 4.4). Moreover, we see two additional effectsfrom the images: First, due to the large smoothness coeffi-cient κ the variance is damped in regions of nearly homoge-neous gray value. Second, because of the above mentioned
Int J Comput Vis (2008) 80: 375–405 401
bias of the optical flow estimation, the magnitude of the op-tical flow field decreases in particular in regions with lowgradients.
6 Conclusions and Future Work
We have presented a model for the interpretation of imagesand image sequences with uncertain gray-values as randomfields. The distribution of gray values for pixels is modeledby random processes for which we use an approximationaccording to the Wiener-Askey polynomial chaos approach.Moreover, we have presented algorithmic building blocksfor image processing based on the notion of random fields.These building blocks and the stochastic Galerkin finite ele-ment method are the key ingredients for a treatment of sto-chastic energies and stochastic partial differential equationsin computer vision and image processing. We have guidedthe reader through the derivation and discretization of sto-chastic analogs of well known partial differential equationsfrequently used in computer vision. The resulting discretiza-tional schemes are simple, since existing deterministic FEMcode can be reused. Moreover the extended models reduceto the deterministic ones if all stochastic modes except themean mode vanish.
Our framework allows for the efficient study of errorpropagation through computer vision models. In contrastto existing research, our approach does neither a priori as-sume Gaussian distributions nor does it deliver error boundsor confidence measures only. In fact the input can be ap-proximations of arbitrary random processes and distribu-tions which by our framework are transformed into outputprocesses and distributions. Previously such results in gen-eral could be obtained by computationally very expensiveMonte-Carlo simulations only.
We have demonstrated the usefulness of the frameworkwith various numerical experiments. For linear models thereis no coupling between the modes. For the heat equationthe mean mode is smoothed and the variance mode isdamped. However, for nonlinear operators (gradient mag-nitude, edge indicator, Perona-Malik diffusion) we haveshown how the stochastic modes are coupled. Thus, vari-ances of the output images are influenced by gradients ofthe mean of the input data. For the Perona-Malik diffu-sion we have successfully validated our new frameworkagainst a naive Monte-Carlo simulation. Moreover, we couldshow experimentally that the stochastic Perona-Malik equa-tion also has an edge enhancement property. However,this edge enhancement occurs in regions of low varianceonly.
As more complicated demonstrator applications we haveconsidered various models for the estimation of optical flow.On several test sequences we have demonstrated the per-formance of our framework. These numerical tests give
some interesting insights and show how the various build-ing blocks act together within the flow estimators. Moreover,we have considered the computation of the bias of the CLGmethod. Our framework was able to reproduce the theoreti-cally predicted curve. In addition to that we have comparedour results with a Monte-Carlo simulation for the bias esti-mation.
The application of the building blocks for stochastic im-age processing to existing well known PDE models offersome very interesting insights and raise many new ques-tions about the modeling and the propagation of errors. Forthe modeling of images as random fields we have assumedindependence of the random variables that steer the behav-ior of different pixels. In the future we will investigate howthis assumption affects the various existing models in com-puter vision. We plan to combine our approach with statis-tic/stochastic data analysis, performing Karhunen Loewe ex-pansions of the input data. This is going to yield the minimalset of independent random variables describing the uncer-tain behavior of the input data. We thus expect to improveour ansatz space for random images and to provide enhancedmodels for computer vision tasks on noisy images.
Future research directions also include a closer analysisof the bias computation and a correction for optical flow es-timation, potentially leading to a higher precision of the es-timation result. For optical flow estimation, most of the cur-rent estimation schemes are formulated in finite differenceschemes rather than finite element schemes, profiting fromwell adapted convolution filters. The implementation of ourframework in a stochastic Galerkin/finite difference schemeis already under investigation. This work is going to makeour approach compatible with state-of-the-art finite differ-ence discretizations.
Appendix 1: Summary of Building Blocks andStochastic Models
We summarize all ansatz spaces, building blocks and sto-chastic variants of classical PDE models in computer vision.Our goal is to provide the reader with a dense and compactsummary of the key ingredients of our framework in orderto support an implementation and a reproduction of our re-sults.
In the first part, we list the key notation for the ansatzspaces for static images and image sequences. Moreover, wetabulate the main formulas for the building blocks. In thesecond part, we give all formulas that are relevant for theimplementation of the stochastic analogs of the computervision models we have discussed. There, we summarize permodel, which building blocks are used, and how the result-ing linear systems of equations are constituted.
A.1 Building Blocks
402 Int J Comput Vis (2008) 80: 375–405
Ansatz Spaces
Static imagesSect. 2.2
• Spatial domain D = [1,N1] × [1,N2]• Nodes/pixels of static image xi , i ∈ I = {1, . . . ,M}• Physical basis functions Pi ∈ C0(D) with Pi(xj ) = δij , for all i, j ∈ I and such that Pi |E is bilinear on each grid cell• Stochastic domain �∗ = [−1,1]• Random variables ξ = (ξ1, . . . , ξM). PDFs ρi(ξi )= 1
2• Indexing of stochastic modes α = 1, . . . ,p
• Stochastic basis functions Hα : [−1,1] → R, e.g. Legendre Polynomials on [−1,1]• Random image f (x, ξ ) =
∑
i∈I
p∑
α=1
f iα Hα(ξi)Pi(x)
Image sequencesSect. 2.3
• Spatio-temporal domain R := D × I := ([1,N1] × [1,N2]) × [1,Nt ]• Nodes/pixels of image sequence yi , i = (ix , it ) ∈ J = {1, . . . ,M} × {1, . . . ,Nt }• Physical basis functions Pi ∈ C0(R) with Pi(yj ) = δij = δixjx δit jt , for all i, j ∈ J and such that Pi |E is trilinear on
Required building blocks • Random Image Sequence Ansatz Space• Tensor components A
ix,jx ,kx ,lxα,β,γ,δ and B
ix ,jx ,kx
α,β,γ
• Structure Tensor Components Jm,t
• Stochastic Heat Equation for smoothing of structure tensor Jm,tρ
Input • p modes of the distribution of the image sequence f (y, ξ )
Linear system of equationsα,β = 1, . . . , p
Linear System as for Horn and Schunck model
(Sα,βmn )ij =
N∑
k∈J
p∑
γ=1
(Jmnρ )kγ B
ix ,jx ,kx
α,β,γ
∫
R
Pi(y)Pj (y)Pk(y) dy,
(Rβm)j =
∑
k∈J
p∑
γ=1
(Jm,tρ )kγ C
jx ,kx
β,γ
∫
R
Pj (y)Pk(y) dy
Lα,β as for stochastic heat equation
Output • p modes of the distribution of the x- and y-component of the optical flow field w(y, ξ )
References
Amiaz, T., & Kiryati, N. (2006). Piecewise-smooth dense optical flowvia level sets. International Journal of Computer Vision, 68(2),111–124.
Avriel, M. (2003). Nonlinear programming: Analysis and methods.New York: Dover.
Bao, Y., & Krim, H. (2004). Smart nonlinear diffusion: A probabilisticapproach. Pattern Analysis and Machine Intelligence, 26(1), 63–72.
Black, M. J., & Anandan, P. (1991). Robust dynamic motion estima-tion over time. In Proc. computer vision and pattern recognition,CVPR-91 (pp. 296–302), June 1991.
Black, M. J., & Anandan, P. (1993). A framework for the robust esti-mation of optical flow. In Proc. ICCV93 (pp. 231–236).
Bruhn, A., Weickert, J., & Schnörr, C. (2002). Combining the advan-tages of local and global optic flow methods. In Proc. DAGM(pp. 454–462).
Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade meetsHorn/Schunck: combining local and global optic flow methods.International Journal of Computer Vision, 61(3), 211–231.
Catté, F., Lions, P.-L., Morel, J.-M., & Coll, T. (1992). Image selec-tive smoothing and edge detection by nonlinear diffusion. SIAMJournal on Numerical Analysis, 29(1), 182–193.
Chorin, A. J. (1971). Hermite expansions in Monte Carlo computation.Journal of Computational Physics, 8, 471–482.
Chorin, A. J. (1974). Gaussian fields and random flow. Journal of FluidMechanics, 63, 21–32.
Cohen, I. (1993). Nonlinear variational method for optical flow com-putation. In SCIA93 (pp. 523–530).
de Laplace, P. S. (1812). Théorie analytique des probabilites. Paris:Courcier Imprimeur.
Deb, M. K., Babuška, I. M., & Oden, J. T. (2001). Solutions of sto-chastic partial differential equations using Galerkin finite elementtechniques. Computer Methods in Applied Mechanics Engineer-ing, 190, 6359–6372.
Fermüller, C., Shulman, D., & Aloimonos, Y. (2001). The statistics ofoptical flow. Computer Vision and Image Understanding, 82(1),1–32.
Forsyth, D. A., & Ponce, J. (2003). Computer vision: a modern ap-proach. Englewood Cliffs: Prentice Hall.
Ghanem, R. G. (1999). Higher order sensitivity of heat conductionproblems to random data using the spectral stochastic finite el-ement method. ASME Journal of Heat Transfer, 121, 290–299.
Ghanem, R. G., & Spanos, P. (1991). Stochastic finite elements: a spec-tral approach. New York: Springer.
Haussecker, H., & Spies, H. (1999). Motion. In B. Jähne,H. Haußecker, & P. Geißler (Eds.), Handbook of computer visionand applications (pp. 309–396). San Diego: Academic Press.
Haussecker, H., Spies, H., & Jähne, B. (1998). Tensor-based im-age sequence processing techniques for the study of dynamicalprocesses. In Proceedings of the international symposium on real-time imaging and dynamic analysis, ISPRS, commission V, work-ing group IC V/III, Hakodate, Japan, June 1998.
Horn, B. K. P., & Schunck, B. (1981). Determining optical flow. Arti-ficial Intelligence, 17, 185–204.
Van Huffel, S., & Vandewalle, J. (1991). Frontiers in applied mathe-matics: Vol. 9. The total least squares problem: Computationalaspects and analysis. Philadelphia: SIAM.
Iijima, T. (1962). Basic theory on normalization of pattern (in case oftypical one-dimensional pattern). Bulletin of the ElectrotechnicalLaboratory, 26, 368–388 (in Japanese).
Iijima, T. (1963). Theory of pattern recognition. Electronics and Com-munications in Japan (pp. 123–134).
Jähne, B. (1993). Spatio-temporal image processing: Theory and sci-entific applications. Lecture notes in computer science. Berlin:Springer.
Kearney, J. K., Thompson, W. B., & Boley, D. L. (1987). Optical flowestimation: An error analysis of gradient-based methods with lo-cal optimization. PAMI, 9(2), 229–244.
Keese, A. (2004). Numerical solution of systems with stochastic un-certainties: A general purpose framework for stochastic finite el-ements. Ph.D. thesis, Technical University Braunschweig.
Kichenassamy, S. (1997). The Perona-Malik paradox. SIAM Journalon Applied Mathematics, 57(5), 1328–1342.
Lucas, B., & Kanade, T. (1981). An iterative image registration tech-nique with an application to stereo vision. In DARPA image un-derstanding workshop (pp. 121–130).
Lucor, D., Su, C.-H., & Karniadakis, G. E. (2004). Generalized poly-nomial chaos and random oscillators. International Journal forNumerical Methods in Engineering, 60, 571–596.
Le Maître, O. P., Reagan, M., Najm, H. N., Ghanem, R. G., & Knio,O. M. (2002). A stochastic projection method for fluid flow
Int J Comput Vis (2008) 80: 375–405 405
II: random process. Journal of Computational Physics, 181(1),9–44.
Malliavin, P. (1997). Stochastic analysis. New York: Springer.Maltz, F. H., & Hitzl, D. L. (1979). Variance reduction in Monte
Carlo computations using multi-dimensional Hermite polynomi-als. Journal of Computational Physics, 32, 345–376.
Meecham, W. C., & Jeng, D. T. (1968). Use of Wiener-Hermite expan-sion for nearly normal turbulence. Journal of Fluid Mechanics,32, 225–249.
Mikula, K., Preusser, T., & Rumpf, M. (2004). Morphological imagesequence processing. Computing and Visualization in Science,6(4), 197–209.
Narayanan, V. A., & Zabaras, N. (2004). Stochastic inverse heat con-duction using a spectral approach. International Journal for Nu-merical Methods in Engineering, 60, 1569–1593.
Nestares, O., & Fleet, D. J. (2003). Error-in-variables likelihood func-tions for motion estimation. In IEEE international conference onimage processing (ICIP) (Vol. III, pp. 77–80). Barcelona.
Nestares, O., Fleet, D. J., & Heeger, D. (2000). Likelihood func-tions and confidence bounds for total-least-squares problems. InCVPR’00 (Vol. 1).
Papenberg, N., Bruhn, A., Brox, T., Didas, S., & Weickert, J. (2006).Highly accurate optic flow computation with theoretically justi-fied warping. International Journal of Computer Vision, 67(2),141–158.
Perona, P., & Malik, J. (1990). Scale space and edge detection usinganisotropic diffusion. IEEE Transactions on Pattern Analysis andMachine Intelligence, 12, 629–639.
Preusser, T., & Rumpf, M. (1999). An adaptive finite element methodfor large scale image processing. In Proceedings scale-space ’99,scale space theories in computer vision, second international con-ference (pp. 223–234).
Reagan, M. T., Najm, H. N., Debusschere, B. J., Le Maître, O. P., Knio,O. M., & Ghanem, R. G. (2004). Spectral stochastic uncertaintyquantification in chemical systems. Combustion Theory and Mod-elling, 8, 607–632.
Reagan, M. T., Najm, H. N., Pebay, P. P., Knio, O. M., & Ghanem,R. G. (2005). Quantifying uncertainty in chemical systems mod-eling. International Journal of Chemical Kinetics, 37, 386–382.
Scharr, H. (2006). Diffusion-like reconstruction schemes from lineardata models. In Lecture notes in computer science: Vol. 4174. Pat-tern recognition 2006 (pp. 51–60). Berlin: Springer.
Scharr, H., Black, M. J., & Haussecker, H. W. (2003). Image statisticsand anisotropic diffusion. In Int. conf. on computer vision, ICCV2003 (pp. 840–847), Nice, France.
Sühling, M. (2006). Myocardial motion and deformation analysis fromechocardiograms. Ph.D. thesis, Swiss Federal Institute of Tech-nology Lausanne (EPFL), July 2006.
Thomee, V. (1984). Galerkin—finite element methods for parabolicproblems. New York: Springer.
Gauss, C. F. (1987). Theory of the combination of observations leastsubject to errors, part one and part two. Supplement (Classics inApplied Mathematics 11) (trans: Stewart, G. W.). Society for In-dustrial Mathematics, Facsimile edition. English version in 1987.Original version in Latin in 1820s.
Weber, J., & Malik, J. (1994). Robust computation of optical flow in amulti-scale differential framework. International Journal of Com-puter Vision, 14(1), 5–19.
Weickert, J. (1998). On discontinuity-preserving optic flow. In Proc.computer vision and mobile robotics workshop (pp. 115–122).
Weickert, J., & Schnörr, C. (2001). Variational optic flow computationwith a spatio-temporal smoothness constraint. Journal of Mathe-matical Imaging and Vision, 14(3), 245–255.
Wiener, N. (1938). The homogeneous chaos. American Journal ofMathematics, 60(4), 897–936.
Witkin, A. P. (1983). Scale-space filtering. In Proc. eighth int. jointconf. on artificial intelligence (IJCAI) (Vol. 2, pp. 1019–1022).
Xiu, D. B., & Karniadakis, G. E. (2002a). Modeling uncertainty insteady state diffusion problems via generalized polynomial chaos.Computational Methods in Applied Mechanics and Engineering,191, 4927–4948.
Xiu, D. B., & Karniadakis, G. E. (2002b). The Wiener-Askey polyno-mial chaos for stochastic differential equations. SIAM Journal onScientific Computing, 24, 619–644.
Xiu, D. B., & Karniadakis, G. E. (2003a). Modeling uncertainty in flowsimulations via generalized polynomial chaos. Journal of Compu-tational Physics, 187, 137–167.
Xiu, D. B., & Karniadakis, G. E. (2003b). A new stochastic approachto transient heat conduction modeling with uncertainty. Interna-tional Journal of Heat and Mass Transfer, 46, 4681–4693.
