ACCELERATION TECHNIQUES FOR APPROXIMATING THEMATRIX EXPONENTIAL OPERATOR∗
M. POPOLIZIO† AND V. SIMONCINI‡
Abstract. In this paper we investigate some well-established and more recent methods that aimat approximating the vector exp(A)v when A is a large symmetric negative semidefinite matrix, byefficiently combining subspace projections and spectral transformations. We show that some recentlydeveloped acceleration procedures may be restated as preconditioning techniques for the partialfraction expansion form of an approximating rational function. These new results allow us to devisea priori strategies to select the associated acceleration parameters; theoretical and numerical resultsare shown to justify these choices. Moreover, we provide a performance evaluation among severalnumerical approaches to approximate the action of the exponential of large matrices. Our numericalexperiments provide a new, and in some cases, unexpected picture of the actual behavior of thediscussed methods.
1. Introduction. In this paper we are interested in the numerical approximationof the action of the matrix exponential to a vector, namely
y = exp(A)v,
when the real n × n matrix A is large and symmetric negative semidefinite. In thefollowing we assume that ‖v‖ = 1, where ‖ · ‖ denotes the Euclidean norm. We inves-tigate some well-established and more recent methods that aim at approximating thevector y by efficiently combining subspace projections and spectral transformations.We refer the reader to [29] for a more complete recent survey.
Two apparently distinct classes of approaches have been discussed in literaturewhen A is large and sparse. In the first type of strategy, the matrix is projected ontoa possibly much smaller space, the exponential is then applied to the reduced matrix,and finally the approximation is projected back to the original large space. If H and edenote the projected and restricted versions of A and v, respectively, then this processcan be summarized as follows:
y ≈ V exp(H)e,
where the columns of V form a basis of the projection space; see, e.g., [14], [15],[16], [23], [31], [32], [33], [34], [41]. In particular, van den Eshof and Hochbruck haverecently devised an acceleration method based on a spectral transformation, whichappears to significantly reduce the dimension of the approximation space withoutsacrificing accuracy and efficiency [41].
∗Received by the editors October 20, 2006; accepted for publication (in revised form) by A. From-mer February 11, 2008; published electronically June 13, 2008.
http://www.siam.org/journals/simax/30-2/67285.html†Dipartimento di Matematica, Universita di Bari, Via E. Orabona 4, I-70125 Bari, Italy
([email protected]).‡Dipartimento di Matematica, Universita di Bologna, Piazza di Porta S. Donato, 5, I-40127
In the second class of methods, the exponential function is first approximatedby an appropriate simpler function, and then the action of this matrix function isevaluated; see, e.g., [4], [8], [10], [27], [29], [31]. To this end, a special role is played byrational function approximation to the exponential; see, e.g., [3], [9], [11], [40]. Let Rν
be such a rational function, so that Rν(A)v ≈ exp(A)v, and assume that Rν admitsthe following partial fraction expansion:
exp(A)v ≈ Rν(A)v = ω0v +
ν∑j=1
ωj(A− ξjI)−1v.(1.1)
In this approach, an approximation to y can be obtained by first solving the shiftedlinear systems appearing in the sum, and then by collecting the sum terms; see, e.g.,[4], [20]. The computation of the system solutions can be either carried out by a directmethod, or, if memory requirements become unacceptable, by iterative methods suchas Krylov subspace solvers [35]. In particular, some of these methods can fully exploitthe recent developments in iterative linear system solvers; see, e.g., [39].
In this paper we argue that the distinction between the two aforementioned cat-egories is in fact very vague, and that increased understanding can be gained byexploiting both viewpoints.
The aim of this paper is twofold. On the one hand, we show that the accelerationmethod by van den Eshof and Hochbruck cited above may be restated as a precondi-tioning technique of the rational function approximation in (1.1) when the exponentialis replaced by an approximating rational function. In addition, we show that anotherrecently proposed method (see [1]) may also be viewed as a preconditioning techniquefor appropriately solving the shifted systems in (1.1). These new results allow us todevise a priori strategies to select the associated acceleration parameters; our com-pletely algebraic analysis complements proposed selections based in some cases (cf.[41]) on the numerical approximation of the solution to analytic problems; numericalresults are shown to justify our choices.
Available comparisons of different schemes in the two categories above are verylimited; see, e.g., [37]. Our second aim is to provide a performance evaluation amongseveral numerical approaches to approximate the exponential, therefore filling a gapin the recent literature. Our numerical experiments show that the ranking of themethods changes significantly depending on whether linear systems can be solved bya direct method. In particular, our numerical findings highlight the competitiveness ofthe simple partial fraction expansion form in (1.1) over newly developed accelerationprocedures when appropriate iterative methods are used. On the other hand, in thecase when direct methods are applicable, ad hoc and acceleration techniques, such as(shift-invert) Lanczos, are superior to the partial fraction expansion method.
This paper is organized as follows. Section 2 reviews the role of rational functionsin the matrix exponential context and recalls the notation and basic facts associatedwith the approximation in the Krylov subspace. Section 3 discusses an accelerationmethod based on the shift-invert Lanczos (SI), while section 3.1 provides a theoret-ical justification of the parameter selection in the shift-invert step. Some theoreticaland computational guidelines for the method are reported in sections 3.2 and 3.3,respectively. Section 4 discusses a second acceleration method, and a cheaper strategyis proposed to deal with the acceleration matrix. The associated parameter is ana-lyzed in section 4.1, where a theoretical justification for its selection is provided; someimplementation improvements are discussed in section 4.2. Section 5 and its subsec-tions report on our numerical experience with all analyzed methods. Finally, section
ACCELERATION METHODS FOR THE MATRIX EXPONENTIAL 659
6 presents further experiments with enhanced implementations of the discussed ac-celeration procedures, while section 7 summarizes our final conclusions.
Throughout this paper we assume that the spectrum of A, spec(A), is containedin the interval [α, 0], for some α < 0. This is not a restrictive assumption. Indeed,if spec(A) ⊂ [α, β], with β < 0, then the spectrum of A1 = A − βI is contained in[α − β, 0] and exp(A) = exp(A1) exp(β). Therefore the behavior of exp(A) can berecovered from that of exp(A1). As we shall see, standard procedures are particularlyslow for large values of ‖A‖ = |α|, and thus in the context of acceleration procedures,our main interest will be in large ‖A‖. Throughout our analysis we assume to workin exact arithmetic and we refer to the paper by Druskin, Greenbaum, and Knizh-nerman [12] for an analysis of Krylov subspace methods for matrix functions in finiteprecision arithmetic.
2. Rational function approximation and Krylov subspaces. Let Rν(z) =Nν(z)/Dν(z) be a rational function approximating the exponential function, withNν ,Dν polynomials of degree ν. When Rν is the rational Chebyshev function it holds(see [9] and references therein) that
supλ≥0
| exp(−λ) −Rν(λ)| ≈ 10−ν ,
which implies a similar estimate for ‖ exp(A)v −Rν(−A)‖ when A is symmetric andnegative semidefinite. Due to this nice approximation property, Chebyshev rationalfunctions are commonly employed to approximate exp(A)v when A has a wide spec-trum.
Let (1.1) be the partial fraction expansion of Rν ; note that Chebyshev rationalfunctions have distinct poles, so that (1.1) can be correctly employed in this case.Since A is real, the poles in (1.1) come in complex conjugates; therefore we can write1
Rν(A)v = ω0v +
ν−1∑j=1
j odd
2�(ωj(A− ξjI)
−1v)
+ ων(A− ξνI)−1v,(2.1)
where ξν denotes the real pole if ν is odd.
When dealing with large dimension problems, the shifted systems can be solvedby means of iterative methods. The simplified quasi-minimal residual (QMR) method[18] can be used to obtain an approximation to x(j) = (A−ξjI)
−1v separately for eachj. The method is an appropriately refined version of the non-Hermitian Lanczos algo-rithm, which exploits the (shifted) complex symmetric form of the coefficient matrixto devise a single short-term recurrence. Preconditioning can be successfully appliedto this variant as long as the preconditioner is also complex symmetric. We refer tothis approach as PFE+QMR in our numerical experiments of section 5.2. An alter-native choice is to use the single Krylov subspace Kk(A, v) = span{v,Av, . . . , Ak−1v}as the approximation space. Assuming a zero initial solution approximation, for each
j the kth iterate x(j)k belongs to Kk(A − ξjI, v) = Kk(A, v), where the last equality
1The use of Chebyshev functions implies a change of sign of the coefficient matrix in Rν(−A)and in its partial fraction expansion. If ζ denotes a Chebyshev pole, then the system to be solvedis (−A − ζI)x = v, which is equivalent to (A − ξI)x = −v, with ξ = −ζ. In the following we omitspecifying this change, and in practice, it is sufficient to change each pole’s sign when Chebyshevapproximation is used.
is due to the shift invariance of Krylov subspaces. Then the linear combination
(2.2) xk = ω0v +
ν∑j=1
ωjx(j)k ∈ Kk(A, v)
is an approximation to Rν(A)v. To speed up convergence without loosing the shiftedform, it is possible to precondition all systems with the same matrix, say (A− τI)−1
with τ > 0, namely
(A− τI)−1(A− ξI)x = (A− τI)−1v,(2.3)
for an appropriate selection of a single τ for all poles. The matrix (A−τI)−1(A−ξI) isknown as a Cayley transformation in the eigenvalue context; see, e.g., [2]. Interestinglyenough, this preconditioning approach has not appeared to have been investigatedexplicitly in this context, possibly because of the requirement to solve systems withthe (real) matrix A− τI. We show in section 3 that this is precisely what the recentlyproposed method in [33], [41] performs when the exponential is replaced by a rationalfunction. In section 4 we also show that the method in [1] amounts to solving (2.3) byfirst resorting to the normal equation and then applying a conjugate gradient (CG)method.
A seemingly different approach consists of approximating the operation exp(A)vby projecting the problem onto a subspace of a possibly much smaller dimension.Krylov subspaces have been extensively explored to this purpose, due to their favorablecomputational and approximation properties; see, e.g., [14], [15], [16], [20], [24], [25],[43]. Let Vk = [v1, . . . , vk] be an n × k matrix whose orthonormal columns spanKk(A, v). The vectors vi, i = 1, . . . , k, can be generated by means of the followingLanczos recurrence:
where ei is the ith column of the identity matrix of a given dimension, eTk is thetranspose of ek, and Hk = V T
k AVk, Hk = (hij) is a symmetric tridiagonal matrix. Anapproximation to x = exp(A)v may be obtained as
(2.5) xk = Vk exp(Hk)e1.
We shall refer to this approximation as the “standard Lanczos” method. For k � n,the procedure projects the matrix problem onto the much smaller subspace Kk(A, v),so that exp(Hk) can be cheaply evaluated with techniques such as scaling and squaringPade [22]. The quality of the approximation strongly depends on the spectral prop-erties of A and on the ability of Kk(A, v) to capture them. Typically, convergencestarts taking place after a number of iterations at least as large as ‖A‖1/2 [24]. Afirst characterization of the approximation was given by Saad in [34, Theorem 3.3],where it is shown that Vk exp(Hk)e1 represents a polynomial approximation p(A)v toexp(A)v, in which the polynomial p of degree k − 1 interpolates the exponential inthe Hermite sense on the set of eigenvalues of Hk. Other polynomial approximationshave been explored; see, e.g., [13], [33].
It is important to realize that the partial fraction expansion and the Krylovsubspace approximation may be related in a natural way whenever the exponential isreplaced by a rational function. In such a situation, the two approaches may coincide
If Rν accurately approximates the exponential, then the two approaches that employxk and VkRν(Hk)e1 evolve similarly as the Krylov subspace dimension increases; see,e.g., the discussion in [19]. The behavior just described could justify the use of thepartial fraction expansion in place of the standard method, especially if accelerationprocedures can be determined to efficiently approximate each system solution. In fact,this is the unifying argument of the results in this paper.
3. The shift-invert Lanczos method. In [32] and independently in [41], theauthors have proposed a technique for accelerating the standard Lanczos approxi-mation to functions such as the exponential. The method is closely related to theshift-invert technique for eigenvalue problems and consists of first applying the Lanc-zos recurrence to the matrix (I − σA)−1, for some σ ∈ R, σ > 0, and starting vectorv1 = v, ‖v‖ is the 2-norm of v. giving
(I − σA)−1Vm = VmTm + βmvm+1eTm.(3.1)
An approximation to y = exp(A)v is then obtained as
ySI := Vm exp((I − T−1m )/σ)e1.(3.2)
The procedure in [32], [41] was tailored to general analytic functions f , and thusis perfectly applicable to the case of rational functions. For f = Rν , in the nextproposition we describe the shift-invert algorithmic procedure by means of a partialfraction expansion of Rν . This allows us to analyze the selection of the shift parameterin [32], [41] for f = Rν , that is, in terms of rational functions.
Proposition 3.1. Let Rν be a rational function with distinct poles and partialfraction expansion Rν(z) = ω0 +
∑νj=1 ωj/(z− ξj). For a chosen σ > 0, let ySI be the
approximation to y = Rν(A)v as in (3.2) when the exponential is replaced by Rν .
Let yprec = ω0v +∑ν
j=1 ωjx(j)m , where for each j, x
(j)m is the Galerkin approxima-
tion to x(j) = (A− ξjI)−1v in Km((A− 1
σ I)−1(A− ξjI), v). Then ySI = yprec.
Proof. When the exponential is replaced by Rν in (3.2), ySI can be written as
ySI = Vm
⎛⎝ω0e1 +
ν∑j=1
ωj
(− 1
σT−1m +
(1
σ− ξj
)I
)−1
e1
⎞⎠ .(3.3)
On the other hand, yprec is obtained as yprec = ω0v +∑ν
j=1 ωjx(j)m , where each
x(j)m approximates x(j) = (A− ξjI)
−1v. For j = 1, . . . , ν, we multiply by (A− 1σ I)
−1
the system (A− ξjI)x(j) = v from the left, that is
(3.4)
(A− 1
σI
)−1
(A− ξjI)x(j) =
(A− 1
σI
)−1
v,
so that (I +( 1σ − ξj)(A− 1
σ I)−1)x(j) = (A− 1
σ I)−1v. We then determine x
(j)m by using
a Galerkin procedure in Km((A− 1σ I)
−1(A− ξjI), v). Note that this space would notbe the “natural” space for a standard Galerkin procedure, which would instead use
−1v), for left preconditioning. Due to the shiftinvariance of Krylov subspaces, it holds that
Km
((A− 1
σI
)−1
(A− ξjI), v
)= Km
((A− 1
σI
)−1
, v
).
Moreover, relation (3.1) can be scaled as(A− 1
σI
)−1
Vm = −VmσTm − σβmvm+1eTm.(3.5)
Therefore, let x(j) ≈ x(j)m ∈ Km((A − 1
σ I)−1, v) with x
(j)m = Vmz
(j)m . Imposing the
Galerkin condition on the residual vector yields
V Tm
(I +
(1
σ− ξj
)(A− 1
σI
)−1)Vmz(j)
m = V Tm
(A− 1
σI
)−1
Vme1.
Taking into account (3.5), we obtain(I −
(1
σ− ξj
)σTm
)z(j)m = −σTme1,
or, equivalently, (− 1σT
−1m + ( 1
σ − ξj)I)z(j)m = e1. We have thus shown that
yprec = Vm
⎛⎝ω0e1 +
ν∑j=1
ωj
(− 1
σT−1m +
(1
σ− ξj
)I
)−1
e1
⎞⎠ ,
which is the same as (3.3).The previous proposition shows that when applied to a rational function, the shift-
invert procedure is mathematically equivalent to a Galerkin procedure for the shiftedsystems involving the poles, appropriately preconditioned with the same matrix(A − 1
σ I). We will use this insightful relation to derive an automatic selectionof the acceleration parameter σ.
3.1. Selecting the acceleration parameter. The effectiveness of the describedscheme strongly depends on the choice of the acceleration parameter. In [41], an anal-ysis is performed to select an optimal parameter at each iteration m, and the actualvalues are tabulated (cf. Table 3.1) by numerically evaluating the quantity
Em−1m−1(σ) := inf
r∈Πm−1m−1
supt≥0
|r(t) − exp(−t)|,
where Πji = {p(t)(1 + σt)−i | p ∈ Πj} and Πj is the space of polynomials of degree j
or less. We stress that the inf-sup problem above depends on m, the iteration index.Our fully algebraic analysis aims to overcome this difficulty by resorting to rationalfunctions in place of exp. Practical guidelines on how to use the tabulated valueswithout varying the parameter at each iteration are also given in [41]. In [30], theauthor essentially conforms to this strategy. In both cases the employed argumentsare tied to the theoretical analysis performed in [36].
The result of Proposition 3.1 leads us to analyze the influence of the parameter σwith a completely different strategy, namely by studying its role in the preconditionedsystem (2.3), that is,(
I +
(1
σ− ξj
)(A− 1
σI
)−1)x(j) =
(A− 1
σI
)−1
v.(3.6)
In the rest of this section we omit the dependence of ξj and x(j) on j. Moreover,without loss of generality (cf. (2.1)), we consider only the complex poles with positiveimaginary part.
We start by observing that the eigenvalues of the coefficient matrix are given byλ = 1 + ( 1
σ − ξ)/(λ − 1σ ), where λ is an eigenvalue of A; this means that the λ’s lie
on a line of the complex plane. Assuming that 1σ − ξ = 0 and dividing by ( 1
σ − ξ), weobtain ((
A− 1
σI
)−1
− χI
)x = v, with χ =
1
ξ − 1σ
,(3.7)
and v defined accordingly. The eigenvalues of the coefficient matrix lie on the hori-zontal line (x, y0) with
y0 :=(ξ)
| 1σ − ξ|2, and x ∈
[− 1
1σ
+1σ −�(ξ)
| 1σ − ξ|2,
1
α− 1σ
+1σ −�(ξ)
| 1σ − ξ|2
].
The assumption 1σ − ξ = 0 is not restrictive: If 1
σ = ξ for some (real) ξ, then from(3.6) it follows that the system solution associated with that pole is readily obtained,and the analysis need not be performed.
The coefficient matrix in (3.7) is given by a real negative definite symmetricmatrix shifted by a complex multiple of the identity. It was shown in [17], [27] thatin this case the performance of Krylov subspace methods may be fully characterizedby using spectral information of the coefficient matrix. Therefore, estimates for theoptimal parameter σ may be obtained by analyzing the spectrum of ((A− 1
σ I)−1−χI)
as σ changes. To this end, we recall here the following bound for the linear systemerror in our notation.
Proposition 3.2 (see [27, Lemma 5.2]). Given the linear system (A− χI)x = v
with A symmetric and semidefinite and χ ∈ C, let xm be the Galerkin approximatesolution to x in Km(A, v). Let λmax, λmin be the largest and smallest eigenvalues of
A−�(χ)I in absolute value, respectively. Then the error satisfies
where g is a function of the spectrum of A, v and of χ only, while ρ = γ +√
γ2 − 1and
γ =|λmin − i(χ)| + |λmax − i(χ)|
|λmin − λmax|.
The proposition above shows that the larger γ, the faster the convergence interms of the subspace dimension m. We recall that spec(A) ⊂ [α, 0] with α < 0.In our context, we can apply the result above both to the original partial fractionexpansion approximation, as well as to the preconditioned system (3.6). In the former
case, setting A = A and χ = ξ, we obtain
γ(ξ) =|α− ξ| + |ξ|
−α.(3.8)
In the preconditioned case, setting A = (A− 1σ I)
−1 and χ = 1/(ξ − 1σ ), after simple
algebraic manipulations, we get
γprec(ξ, σ) =( 1σ − α)|ξ| + 1
σ |α− ξ|−α| 1σ − ξ|
.(3.9)
The expression in (3.8) shows that for |α| � |ξ|, the error bound predicts veryslow convergence of the linear system, as in this case γ ≈ 1. It is desirable that awell-chosen σ make γprec(ξ, σ) much larger than γ(ξ), so as to significantly improvethe convergence rate. An ideal value of σ would satisfy something like
minξ
γprec(ξ, σ) ≥ maxξ
γ(ξ),
to ensure faster convergence for all poles. However, this inequality turned out to behard to analyze. Nonetheless, it is possible to relate the two convergence coefficients.To simplify the notation, in the rest of this section we use
τ :=1
σ,
and, with some abuse of notation, we use γprec(ξ, τ). We have
γprec(ξ, τ) = F (α, ξ, τ)γ(ξ),(3.10)
where
F (α, ξ, τ) =τ
|τ − ξ| −α|ξ|
(|α− ξ| + |ξ|)|τ − ξ| =τ − c
|τ − ξ| ,(3.11)
with c = α|ξ|/(|α− ξ| + |ξ|).The following proposition shows that, for each pole, it is possible to determine
the least value of the parameter that improves convergence, and also the one thatmaximizes the ratio between the two convergence rates. Unfortunately, the resultingparameter depends on the given pole, and thus it may not be optimal for other poles.
Proposition 3.3. Given α and ξ, let F (τ) = F (α, ξ, τ) be defined in (3.11), andassume that �(ξ) > α/2 and (ξ) = 0. Then
ACCELERATION METHODS FOR THE MATRIX EXPONENTIAL 665
(ii) F (τmax) ≥ F (τ), for every τ , where
τmax = τmax(ξ) =�(ξ)c− |ξ|2c−�(ξ)
and F (τmax) =|c− ξ||(ξ)| ≥ 1;(3.12)
(iii) γprec(ξ, τ0) = γ(ξ) and limτ→∞
γprec(ξ, τ) = γ(ξ).
Proof. Let ξ = ξR + ıξI . We first show that ξR > c. Since c < 0, then clearlyξR > c when ξR ≥ 0. For ξR < 0, using α < 2ξR we obtain α|ξ| < 2ξR|ξ| ≤ ξR|ξ| ≤ξR(|ξ| + |α− ξ|), from which
ξR >α|ξ|
|ξ| + |α− ξ| = c.
To prove (i), we observe that
F (τ) ≥ 1 ⇔ 2(ξR − c)τ ≥ |ξ|2 − c2.(3.13)
Using ξR > c, the previous requirement corresponds to imposing τ ≥ τ0.To prove (ii) we explicitly write
F ′(τ) = − (τ − ξR)
|τ − ξ|3 (τ − c) +1
|τ − ξ| = 0 ⇔ −(τ − ξR)(τ − c) + |τ − ξ|2 = 0,
from which the expression for τmax follows. Moreover, F is an increasing function forτ ≤ τmax and a decreasing one otherwise, so that F (τmax) is a maximum.
To prove that F (τmax) ≥ 1 we notice that F (τmax)2 = 1 + (c − ξR)2/ξ2
I , from
which we obtain that (F (τmax) − 1)(F (τmax) + 1) = (c−ξR)2
ξ2I
. The result follows by
taking into account that
F (τmax) + 1 =|ξI | + |c− ξ|
|ξI |.
Finally, the first equality in (iii) follows from F (τ0) = 1 in (3.13), while it can bereadily verified that limτ→∞ F (τ) = 1.
In light of Proposition 3.3(i), one could restrict the choice of the parameter τto the interval [τ0,∞[. However, (iii) indicates that values of the parameter that aretoo close to the extremes of this interval do not accelerate convergence; see similarconclusions in [30]. The hypothesis that �(ξ) > α/2 is crucial; otherwise F (τ) ≥ 1only for τ < 0. The only (unlikely) exception is ξ = α/2 ∈ R, in which case F (τ) ≥ 1for any nonnegative τ . On the other hand, for the values of α of interest, |α| � |ξ|,and thus the hypothesis �(ξ) > α/2 is clearly verified. It is also important to noticethat for |α| � |ξ| it follows that τmax ≈ |ξ|, indicating the obvious fact that, for eachpole ξ, the best real parameter is related to the pole itself.
Although quite sharp, the results above still depend on the spectrum of A throughα and do not provide a simple way to select a good single parameter for all poles.To complete our understanding, we thus look for a quantity that well represents thebehavior of γprec, especially for large |α|. To this end, we observe that −c ≤ |ξ|, so
that F (ξ, τ) ≤ τ+|ξ||τ−ξ| . The bound is sharp for |α| � |ξ|, that is, γ(ξ) ≈ 1, in which case
c ≈ −|ξ|. The quantity H(τ, ξ) := τ+|ξ||τ−ξ| ≥ 1 also appears explicitly in the following
and this estimate is again sharp when γ(ξ) ≈ 1. We next analyze the behavior ofH, which does not depend on the spectrum of A, but only on the poles and on theparameter. We have
H(τ, ξ)2 = 1 + 2(|ξ| + �(ξ))τ
|τ − ξ|2 .
Note that |ξ| + �(ξ) ≥ 0 for any ξ, and for a given nonreal pole ξ it holds thatτ
|τ−ξ|2 ≤ |ξ|||ξ|−ξ|2 , where the right-hand side is attained for τ = |ξ|. Therefore, for each
pole ξi,
H(|ξi|, ξi) = maxτ>0
H(τ, ξi).
To take τi as a priori parameter for all systems, we need to make sure that thisvalue of τ is also effective for a different pole ξj . Let the poles be sorted with decreasing(positive) imaginary parts. Setting τi = |ξi|, we state the following discrete problem:2
maxτ1,...,τν
minξ1,...,ξν
H(τi, ξj),(3.15)
which can be solved, once and for all, for a given class of rational functions and foreach selected degree. As an example, Table 3.2 reports the values of H(τi, ξj) as τiand ξj vary, for Chebyshev rational functions and ν = 14 (poles are computed fromthe coefficients as listed in [9]). In the table, the optimal value of τ for problem (3.15)with ν = 14 is given by τ1 = 18.8616, ensuring that γprec(ξ, τ) ≥ H(τ1, ξ1) = 1.1657.Note that, for all degrees, the best value of τ turns out to always be associated withξ1. Therefore, we propose to use the parameter
τopt := |ξ1| ⇔ σopt =1
|ξ1|.(3.16)
The corresponding values associated with Chebyshev rational poles are listed inTable 3.3 for ν ≤ 20. The entries in the table can be used as follows: If a finaltolerance tol on the approximation of exp(A)v is requested, then the shift-invert ap-proach may be used with a shift value corresponding to ν ≥ − log10(tol) (e.g., tol =10−8 yields ν ≥ 8 so that σ = 0.1062 or a smaller value in the table may be used).
Our derivation suggests a parameter selection somehow similar to that given in[41] (cf. Table 3.1), although our justification is completely different, and it does notdepend on m. This similarity may be viewed as an additional motivation for thereliability of the approach.
2A continuous problem in τ and ξ could also be formulated, but the unnecessary added difficultyis beyond the scope of this analysis.
3.2. Asymptotic behavior. Our parameter selection is based on asymptoticarguments, that is, on information of the matrix spectral interval and not on the actualeigenvalue distribution. In particular, we recall that the convergence of (symmetric)linear systems often exhibits superlinear behavior, in the sense that the rate of conver-gence may increase as convergence takes place; see, e.g., the discussion in [39]. Such im-portant characterization is not captured by an asymptotic analysis. Therefore, in somecases other values of the parameter may lead to better convergence than that obtainedwith our analytically selected choice. As an example, we consider the matrix A of sizen = 3375 of Example 5.1 in section 5, whose extreme eigenvalues are λmin ≈ −2329.4and λmax ≈ −22.597, and we define the singular matrix A = A−λmaxI. We study theperformance of the accelerated process with the optimal parameter σopt = 0.053 andwith another possible candidate, σmin = 1/maxj |�(ξj)| = 0.1124, taken for ν = 14poles. The vector v is taken as a normalized vector of all ones. Figure 3.1 shows theconvergence curves of the SI procedure with A and the two parameters (lower solidand dashed curves), showing a slightly better performance of σmin over σopt; this isnot predicted by our theory. However, our arguments better describe the behavior ofthe n×n diagonal matrix D (middle solid and dashed curves), whose nonzero entriesare uniformly distributed values in the same spectral interval as A. The vector v isunchanged. In this case, the convergence slope is steeper when using σopt than withσmin. The upper curves show the convergence rate predicted by the asymptotic quan-tity H(1/σ, ξ1)
j , j = 1, . . . ,m, for σ = σopt (filled squares) and σ = σmin (circles).Note that both curves well represent the initial convergence phase of the shift-invert
0 5 10 15 20 25 30 35 40 45
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
SI(A,σopt
)
number of iterations
est
ima
ted
err
or
SI(A,σmin
)
SI(D,σmin
)
SI(D,σopt
)
asymptotics
10−4
10−3
10−2
10−1
100
101
102
102
value of the parameter
nu
mb
er
of
itera
tion
s
Fig. 3.1. Left: Convergence history of SI for a matrix stemming from a shifted Laplace operatorand for a diagonal matrix D with uniformly distributed eigenvalues in spec(A). Here σopt = 0.053and σmin = 0.1124. Also reported are the asymptotic values H(1/σ, ξ1)j , j = 1, . . . ,m, for σmin
(circles) and σopt (filled squares). Right: Number of iterations of SI applied to the diagonal matrixD versus value of the shift σ; the symbol “*” refers to the choice σ = σopt.
procedure with D, with a slightly better performance for H(1/σopt, ξ1). To fully ap-preciate the performance of the choice σ = σopt with the matrix D, in the right plot ofFigure 3.1 we display the number of iterations of SI to achieve the required stoppingthreshold, as the value of the parameter varies in [10−4, 102]; the symbol “*” refers tothe choice σ = σopt. Note that the performance of the method is not overly sensitiveto overshooting values of σ (and this conforms to the tabulated values in Table 3.1),but it may considerably degrade if σ is chosen too small. In particular, the plot showsthat a typical practical value suggested in [41, section 6], namely σ = 0.01, wouldforce the method to perform more iterations on this matrix.
3.3. Implementation details. The algorithmic aspects of the shift-invert pro-cedure were described in [41]. A possible implementation generates the matrix Vm
one vector at a time by means of the Lanczos algorithm (see, e.g., [21]) and the cor-responding elements of the tridiagonal matrix Tm in (3.1). It is important to observethat convergence at high accuracy is often obtained for a small approximation space,so that little memory is required to store Vm. The difficulties associated with theapproximate solutions with I − τA are also treated in [41].
A crucial part in the overall procedure is how to monitor convergence, since theerror norm is not available. Although the analysis of stopping criteria is beyond thescope of this paper, we need to face this problem to avoid premature termination; werefer to [19] for a recent analysis and an accurate estimation of the error norm forthe approximation of various rational matrix operators. With the notation of (3.5), aclassical stopping criterion is given by the quantity
tm+1,m|eTm exp((I − T−1m )/σ)e1|,(3.17)
which is cheaply available during the computation; in the case of standard Lanczos,using (2.5) the criterion above reduces to hk+1,k|eTk exp(Hk)e1|. It is known that form very small, this quantity may highly underestimate the true error; cf. the (red)dash-dotted curve of Figure 3.2. In our experiments of sections 5 and 6, for the firstfew iterations we replace the absolute estimate of the error with a relative quantity,until this falls below a safeguard parameter set to 10−2. In the case of (3.17), this
0 20 40 60 80 100 120 140 160 18010
−14
10−12
10−10
10−8
10−6
10−4
10−2
100
102
number of iterations
norm
of err
or
and e
stim
ate
s
formula (3.17)formula (3.18)formula (3.19)true error norm
Fig. 3.2. Convergence history of standard Lanczos and different error estimates. The safeguardparameter for (3.18) is equal to 0.1 (see text).
In practice, this somewhat conservative safeguard procedure is maintained until thecomponents of the approximation vector exp((I −T−1
m )/σ)e1 take the expected expo-nential pattern; see [26]. The reported values in the figure refer to the approximationof exp(0.1A)v, where A is the 4900×4900 matrix stemming from the two-dimensional(2D) Laplace operator with Dirichlet homogeneous boundary conditions, v is the nor-malized vector of all ones, and the safeguard parameter is equal to 0.1. In Figure 3.2we also report the behavior of the higher-order estimate (for standard Lanczos)
tk+1,k|eTk ϕ1(Hk)e1|, ϕ1(z) =exp(z) − 1
z,(3.19)
which was also proposed in [34]. We observe that this estimate seems to cure the prob-lem given by the lower-order estimate in (3.17). In practice, whenever eTk exp(Hk)e1 istoo small, so that (3.17) is unreliable, it holds that |eTk ϕ1(Hk)e1| ≈ |eTkH−1
k e1|. FromFigure 3.2 it is clear that both the safeguard strategy and the higher-order criterionallow one to safely continue the iteration until true convergence takes place. In ourexperiments we used (3.18) because it is in general cheaper to compute than (3.19).We refer to [12], [19], [27], [34] for further considerations and for higher-order stoppingcriteria.
4. Application of a real-valued method for solving linear systems. Inthis section we increase our understanding of a method recently proposed by Axelssonand Kucherov in [1], for solving complex symmetric systems by means of a formu-lation that only uses real arithmetic computation. The method can be used in ourcontext when a partial fraction expansion of a rational function approximation is em-ployed, as described in section 2. We show that the method can be derived using ourpreconditioning framework. Moreover, we propose a variant that makes the overallprocedure significantly more efficient.
We first briefly recall the main steps of the approach. Given the complex system
(R + ıS)u = b,(4.1)
with u = uR + ı uI and b = bR + ı bI , the proposed procedure uses the real form(R− ηS
√1 + η2S√
1 + η2S −R− ηS
)(uR
z
)=
(bR
(bI − ηbR)/(√
1 + η2)
),
where η > 0 is a real parameter and z = (ηuR−uI)/√
1 + η2. The Schur complementreduction provides the following linear system for uR
CuR = w(4.2)
with C = R − ηS + (1 + η2)S(R + ηS)−1S and w = bR + S(R + ηS)−1(bI − ηbR).The imaginary part, uI , may be computed by using the relation RuR −S uI = bR. Itis shown in [1] that under certain hypotheses on S and R, it is possible to derive anoptimal choice of η so that the matrix B = R + ηS is an effective preconditioner forthe system (4.2).
In our context, the complex symmetric system to be solved is (A− ξI)u = v for afixed pole ξ = ξR + ıξI . Therefore, we have R = A− ξRI and S = −ξII. Substitutingin the coefficient matrix of the system (4.2) we obtain
C = −B + 2η ξI I − (1 + η2)ξ2IB
−1,
where the preconditioner becomes B = −(R + ηS) = (ξR + η ξI)I − A, and thepreconditioned system reads
MuR = B−1w, with M = B−1C = −I + 2ηξIB−1 − ξ2
I (1 + η2)B−2,(4.3)
w = v − η ξIB−1v.
Moreover, uI = 1ξI
(−A+ξRI)uR+ 1ξIv. Therefore, for each pole ξ, the original complex
symmetric system is transformed into the real (preconditioned) system (4.3), whichneeds to be solved by an iterative method. We next show in Proposition 4.1 that thematrix −M is symmetric positive definite for any choice of η > 0 and for all poles,and thus the conjugate gradient method can be used. Moreover, we show that thesystem (4.3) resulting from the procedure outlined above is nothing but the real partof the normal equation of (2.3) for a special choice of the acceleration parameter.
Proposition 4.1. Let uR be the solution to MuR = B−1w (cf. (4.3)). For τ =ξR + η ξI , η > 0, consider the (preconditioned) linear system
(τI −A)−1(A− ξI)u = (τI −A)−1v,
and set K = (τI − A)−1(A − ξI). Then M = −K∗K ∈ Rn×n. Moreover, uR is the
real part of the solution of K∗Ku = K∗(τI −A)−1v.Proof. Let R = A− ξRI and S = −ξII, and note that R and S commute so that
Therefore, K∗K is real symmetric and M is negative definite. Analogously, we canwrite K∗(τI −A)−1v = (R+ ηS)−2(R− iS)v whose real part is given by �(K∗(τI −A)−1v) = (R + ηS)−2Rv = (R + ηS)−1w. Therefore, the real part of the equationK∗Ku = K∗(τI − A)−1v is given by −M�(u) = −B−1w, from which it follows thatuR = �(u).
The original implementation in [1] provided an optimal parameter τ for eachshifted system to be solved, yielding a different acceleration matrix (τI − A)−1 foreach pole. The authors suggested using τ = τ(ξj) = |ξj |, for A − ξjI having definitesymmetric real part. This condition is not satisfied in our case, since �(ξj) may beeither positive or negative. In the next section we show how to select a single τ for allsystems, so as to lower the computational costs.
A completely different preconditioning strategy could also be adopted. The use ofan optimal preconditioner (τI−A) with τ = τ(ξ) would be feasible if it were possibleto update the factorization for different shifts without recomputing the factors fromscratch; see the results in [6] in this direction for linear system preconditioning.
4.1. Selecting the acceleration parameter. In this section we derive a singlequasi-optimal positive parameter τ , from which, according to the relation τ = ξR+ηξI ,a different η follows for each system in (4.3). Therefore, while M differs for each shiftedsystem, the matrix B = τI −A is the same for all shifts.
ACCELERATION METHODS FOR THE MATRIX EXPONENTIAL 671
Proposition 4.1 shows that M is symmetric and negative definite for any η > 0.Similar conclusions were derived in [1, Remark 1]. The next proposition provides sharpbounds for the condition number of M with no further hypotheses on A− ξI.
Proposition 4.2. Assume that the hypotheses of Proposition 4.1 hold and thatτ > max{0,�(ξ)}. Then
(4.4) cond(M) ≤ max
{|ξ|2τ2
,|α− ξ|2(α− τ)2
}|τ − ξ|2
ξ2I
.
Moreover, if it also holds that τ ≤ |ξ|, then |α−ξ|2(α−τ)2 ≤ |ξ|2
For λ ∈ [α, 0], let μ ∈ spec(−M), μ = g(λ) = λ2−2λξR+|ξ|2(λ−τ)2 . We have
g′(λ) = 2λ(ξR − τ) + τξR − |ξ|2
(λ− τ)3= 0 ⇔ λ :=
τξR − |ξ|2τ − ξR
.
Since τ > ξR, it holds that g′(λ) > 0 only for λ > λ; hence
g(λ) =ξ2I
|τ − ξ|2 ≤ μ ∀μ ∈ spec(−M).(4.6)
To derive an upper bound, we notice that, since λ is the only critical point and it isassociated with a minimum, the maximum of g in [α, 0] is given by max{g(α), g(0)}.Collecting this bound and (4.6), the bound (4.4) for cond(M) follows.
We next assume that τ ≤ |ξ| holds for all poles ξ. We write
g(α) − g(0) =α2(τ2 − |ξ|2) − 2ατ(ξRτ − |ξ|2)
τ2(τ − α)2.
For τ ≤ |ξ| the first addend in the numerator of the last expression is negative. Forthe second addend, we separately treat the cases of positive and negative pole’s realpart. If ξR < 0, then the second addend gives −2ατ(ξRτ − |ξ|2) ≤ 0. If ξR > 0,then we can get −2ατ(ξRτ − |ξ|2) ≤ −2ατ(τ2 − |ξ|2) ≤ 0. We have thus shown thatg(α) − g(0) ≤ 0, which completes the proof.
The bound in (4.4) may be rather sharp. Its sharpness depends on whether theextremes of the function g defined in the proof are attained. Table 4.1 reports thebound in (4.4) for the 125 × 125 matrix obtained by the discretization of the three-dimensional (3D) Laplacian with homogeneous boundary conditions, shifted so as tohave zero largest eigenvalue. The poles correspond to the Chebyshev rational approx-imation of degree ν = 14. We used τopt = minj=1,...,ν |ξj | = 5.7485; see below for anexplanation of this choice.
To derive a single parameter τ for all poles ξ, we study the bound in (4.5),which does not depend on the spectrum of A. We will see that for the Chebyshevapproximation it is possible to derive a single parameter that satisfies τ ≤ |ξ|.
Let Wξ(τ) = |τ−ξ|2τ2 be the part of the upper bound in (4.5) that depends on τ .
It can be verified that Wξ(τ)′ = − 2τ3 (−τ�(ξ) + |ξ|2), so that
Wξ(τ)′ = 0 ⇔ τ∗(ξ) = �(ξ) +(ξ)2
�(ξ)=
|ξ|2�(ξ)
.
If �(ξ) < 0, then Wξ(τ∗) is a maximum and τ∗(ξ) is negative. We thus restrict ourattention to the poles with positive real parts.3 Moreover, we observe that, for τ > τ∗and �(ξ) < 0, the function Wξ is decreasing, so that the larger τ , the smaller thebound for �(ξ) < 0. We then recall that for (4.5) to hold, the selected parameter τmust satisfy
�(ξ) ≤ τ ≤ |ξ| ∀ξ.
Let the poles be sorted as �(ξ1) ≤ · · · ≤ �(ξν). Then τ∗(ξν) ≥ �(ξj) for j ≤ ν, andwe define
(4.7) τopt := min
{min
j=1,...,ν|ξj |, τ∗(ξν)
}.
For the Chebyshev poles it holds that minj=1,...,ν |ξj | = |ξν | so that
τopt = |ξν |.
In Figure 4.1 we report the total number of conjugate gradient iterations requiredby the method to solve all systems MuR = w (see Algorithm AK), for different valuesof the parameter τ ∈ [0, 7]. The data are as in Example 5.1 and ν = 8. The symbol “*”indicates the number of iterations for the choice τ = τopt, showing the high quality ofthe a priori selected parameter.
The analysis above conforms with the multiple choice in [1], although in ourcase extremely fast convergence cannot be achieved for all shifted systems. It is alsointeresting that, as opposed to the shift-invert procedure, the pole with the smallestmodulo is selected as the optimal parameter.
4.2. Implementation details. The real-valued method for approximating y =exp(A)v can be summarized as follows. For simplicity and without loss of generality,we take here a rational function of even degree. For odd degree rational approximation,the real shifted system corresponding to the real pole can be solved explicitly withoutresorting to the method discussed previously.
3We recall that in the case of Chebyshev approximation, poles are used with the opposite sign.
ACCELERATION METHODS FOR THE MATRIX EXPONENTIAL 673
0 1 2 3 4 5 6 7
25
30
35
40
45
50
τopt
value of the parameter τ
tota
l n
um
be
r o
f ite
ratio
ns
Fig. 4.1. Total number of iterations for the variant of the Axelsson–Kucherov method, as afunction of the parameter τ . The symbol “*” refers to the number of iterations for τ = τopt.
Algorithm AK.
Given A, v, ξ1, . . . , ξν , ω1, . . . , ων
(i) Choose a parameter τ > 0.
(ii) Set B = τI −A; w1 = B−1v;w2 = B−1w1.
(iii) For each pole ξj = ξR + ıξI , j = 1, 3, 5, . . . , ν − 1:
– Solve MuR = w with M = −I + 2(τ − ξR)B−1 − |τ − ξ|2B−2
and w = +w1 − (τ − ξR)w2.
– Compute uI = 1ξI
(−AuR + ξRuR + v).
– Set xj = uR + ıuI .
(iv) Compute yAK = ω0v + 2
ν−1∑j=1
j odd
�(ωjxj).
As already mentioned, the solution of MuR = w is performed iteratively, as Mshould not be explicitly computed but only applied in products such as y = Mx, asis the case in conjugate gradient methods. Each matrix-vector multiplication with Mrequires solving two systems with B = τI −A, and this is related to the fact that Mis the coefficient matrix of the normal equation.
The final attainable accuracy of the overall computation depends primarily on therational function used, but also on the accuracy with which the linear systems of step(iii) are solved. This requires the stopping tolerance to be smaller than the accuracyrequested; in our experiments we noticed that tol = 10−ν−2 delivered a sufficientlyaccurate final solution to the exponential. No further study was attempted to refinethis value.
We consider solving systems with B with both a direct and an iterative method.In the former case, the cost of factorizing the single matrix B is performed once forall systems. This provides significant computational savings over the original methodin [1], a sample of which is reported in Table 4.2. In the table we compare the originalmethod, where an optimal B = B(τ) is determined and factorized for each pole, withAlgorithm AK, where a single suboptimal B is computed and factorized at step (ii)of the algorithm. The numbers show that the new strategy improves performance,
Problems of section 5. CPU time of Algorithm AK when different iterative schemes are usedto solve with B = τI −A.
n tol AK+Variant AK+CG AK+PCGExample 5.1
10−5 0.02 0.04 0.05125 10−8 0.04 0.07 0.08
10−11 0.08 0.15 0.1710−14 0.15 0.29 0.32
10−5 0.42 0.65 1.223375 10−8 0.77 1.75 2.91
10−11 1.73 3.88 6.0710−14 2.81 6.69 11.11
10−5 3.20 4.57 8.6115625 10−8 5.88 13.31 21.21
10−11 13.42 28.07 44.5110−14 22.10 52.51 83.22
Example 5.2
10−5 0.68 1.38 1.102500 10−8 1.69 4.02 3.01
10−11 3.43 8.32 8.4610−14 5.86 15.70 12.58
10−5 3.67 9.38 7.6910000 10−8 8.60 28.54 22.34
10−11 17.50 61.85 47.1210−14 29.54 122.99 89.27
especially on large problems, while the total number of iterations does not signifi-cantly grow, compared to the optimal shift selection in [1]. The results in the tablewere obtained by using a direct method for solving the involved inner linear systems.Additional numerical experiments, not reported here, with iterative solves confirmedthe superiority of our new approach even in this inner-outer setting.
In the case an iterative solver is used, one is faced with the problem of efficientlysolving two systems with B at each iteration of the solver with M . By exploitingthe positive definiteness of B, we consider the following alternatives: (a) two calls tothe conjugate gradients in sequence; (b) two calls to preconditioned CGs in sequence;(c) one call to a variant of the CG method proposed by van der Vorst in [43] to
ACCELERATION METHODS FOR THE MATRIX EXPONENTIAL 675
simultaneously solve for B and B2 with a single recurrence. The CPU times obtainedfor the problems of Examples 5.1 and 5.2 are reported in Table 4.3.
The numbers show that the variant that simultaneously approximates the systemswith B and B2 is faster than both the standard CG method and its preconditionedversion. It is important to notice that in the approach proposed in [43] precondi-tioning is not applicable; nonetheless, its performance is superior to that of standardpreconditioned CG applied twice. We should mention that when using the approachin [43], one could employ a different (optimal) B for each shifted system at no addi-tional cost. We decided to maintain Algorithm AK for consistency with the case ofthe direct solves.
5. Numerical experiments. In this section we report on our numerical expe-rience with the discussed methods, which we summarize as follows:
• Partial fraction expansion (PFE). Computation of (2.1) by explicitly solvingeach complex symmetric system. Systems corresponding to conjugate pairsare coupled.
• Standard Lanczos. Classical Lanczos approach described in section 2; see,e.g., [34].
• Axelsson–Kucherov (AK). Variant of method by Axelsson and Kucherov des-cribed in section 4.
• Shift-invert Lanczos (SI). Acceleration procedure described in section 3 [41].When using methods that explicitly rely on the partial fraction expansion, namely
PFE and AK, the final accuracy influences the number of terms in the expansion, andthus the number of shifted systems to be solved. In our implementation of the shift-invert procedure, the parameter selection is also guided by the required accuracy; cf.Table 3.3. In general, this is not strictly necessary, and one could choose σ as theoptimal parameter associated with an approximation of large degree, say ν = 14.
Since the error norm cannot be explicitly monitored, stopping criteria were intro-duced as discussed in previous sections. In the small examples, however, we computedthe actual error and verified that a satisfactory tolerance was reached, achieving therequired order of magnitude for the absolute error norm. It should be mentioned thatthe solution norm influences the stopping criterion. Indeed, depending on the spec-trum of A, the vector exp(A)v may have a very small norm, which makes a loosestopping tolerance completely useless, yielding an approximate solution with no digitaccuracy. In all problems considered, the vector exp(tA)v with t = 0.1 had a normnot smaller than 10−4.
All methods except the standard Lanczos procedure require solving real or com-plex shifted systems. In all of these cases, such a step employs over 95% of the totalcomputational efforts, so that it is really the only bottleneck of these methods. In thenext two subsections we report results when solving these systems by either a director an iterative method, yielding in this latter case an inner-outer procedure.
All numerical experiments of this section were performed in MATLAB [28], ver-sion 7.0.1 (R14-SP1), and CPU timings were obtained with the function cputime.We like to mention that different CPU time performance was observed when usingdifferent MATLAB versions or releases, which in some cases significantly affected thecomparison among the methods.
5.1. Using direct methods. In this section, we report our experiments when adirect method is used to solve with (A−ξjI) or (τI−A). When dealing with the sym-metric and positive definite matrix (τI−A), the Cholesky factorization is performed,after a reordering of the matrix entries (MATLAB function symamd). Permutation sig-
nificantly improves the overall cost of solving with the shifted matrix (both the factor-ization and the solution phases). In the following, the matrix is always reordered, andthe reported timings include the factorization cost. The LU decomposition withoutpivoting the complex symmetric matrix (A− ξjI) yields a (symmetric) LDLT factor-ization. After reordering, the number of (now complex) nonzero entries is the same asfor the real factor. We emphasize, however, that in the case of the PFE, the complexsystem solutions were carried out by means of the MATLAB backslash operation “\”,which is significantly faster than the two-step procedure of first factorizing the matrixand then solving with the factors.
In all tables, the number of iterations for standard Lanczos and for SI coincideswith the dimension of the generated Krylov subspace. For AK, the number in paren-theses is the global number of iterations performed to solve all shifted systems withM = M(ξj). The stopping tolerance tol = 10−ν is fixed for all codes. Methods employ-ing the rational function approximation thus use the corresponding function degreeν.
Example 5.1. We consider the n×n matrix stemming from the finite difference dis-cretization of the 3D Laplace operator on the unit cube and Dirichlet homogeneousboundary conditions, with eigenvalues in [−179.14,−12.862] for n = 125. Differentdiscretization refinements are considered. These data represent a typical benchmarkfor the approximate evaluation of the matrix exponential in PDE contexts. We ap-proximate the vector exp(tA)v, with t = 0.1 and v a normalized normally distributedrandom vector. The elapsed time and the number of iterations (in parentheses) forvarious problem dimensions and final tolerances are reported in Table 5.1.
Table 5.1
Example 5.1. CPU time (and number of iterations in parentheses when appropriate) for allmethods when systems with shifted matrices are solved with a direct method. Different dimensionproblems and various stopping tolerances are reported.
Several comments are in order. First, we observe that explicitly dealing with thePFE by means of direct solvers becomes significantly more expensive, especially forthe large size matrix. Moreover, all methods behave quite consistently as the problemdimension increases, and the performance ranking is already clear for n = 3375.
Second, on this problem, standard Lanczos is the most efficient approach, as far asCPU time is concerned; the shift-invert procedure shows the second best performance.Memory requirements for standard Lanczos, however, become increasingly high as thenumber of iterations increases, since the whole basis needs to be stored. This problemmay be overcome by resorting to a two-pass strategy. In the first pass, the Lanczos
ACCELERATION METHODS FOR THE MATRIX EXPONENTIAL 677
basis is not stored, but only the projected solution is; in the second pass, the Lanczosbasis vectors are recomputed one at a time to form the final solution; see [19] formore details. This approach drastically reduces memory needs, but requires almosttwice the time to complete the computation, making the performance of the methodmore comparable to that of SI. For the latter method, we observe that the number ofiterations does not grow (in fact, it decreases) as the problem dimension increases forthe same required tolerance; see, e.g., [30], [41] for a discussion.
Third, the AK approach does not perform satisfactorily, as compared to theLanczos methods, although its memory requirements are roughly limited to theCholesky factor and to a few CG vectors. The number of iterations does not growsignificantly as the problem dimension increases for a fixed final tolerance. We shouldmention that AK improves the performance of the original PFE method on the largematrix. This seems to indicate that AK may be advantageous for approximating theaction of other matrix rational functions for which the standard Lanczos proceduredoes not show superlinear convergence. Moreover, the method’s limitations are lessapparent when a loose final accuracy is required, which is precisely the context sug-gested in the original paper [1].
In summary, this example shows that for moderately large spectra, the standardLanczos approach is still competitive, and the analyzed acceleration procedures donot seem to significantly improve its performance. The next example faces a moreextreme case for which using an acceleration procedure is mandatory.
Example 5.2. In this example we approximate exp(tA)v, t = 0.1, where the n×nmatrix A stems from the finite difference discretization of the 2D operator
on the unit square, with Dirichlet homogeneous boundary conditions [41]. Two gridrefinements have been considered. The spectrum is contained in the interval [−35424,−25.256] for the smaller problem. The vector v is taken as in the previous example.The CPU time and number of iterations, when appropriate, are reported in Table 5.2.This example has special features that make it very different compared to the previousone. In particular, ‖A‖ and the spectral range are significantly large, penalizing thestandard Lanczos method. Moreover, the finite difference discretization of the 2Doperator generates a sparser matrix than in Example 5.1, allowing cheaper systemsolves. We can thus predict especially good performance of all acceleration techniques,including PFE, compared to standard Lanczos. The results in Table 5.2 fully confirmthese considerations.
5.2. Using iterative methods. The use of iterative methods for solving thelarge linear systems provides a significantly different picture from what is shown inthe previous section. In the case of AK and SI, the resulting algorithm is an inner-outer procedure. We next compare the standard Lanczos method with the followingiterative procedures:
• PFE+QMR. Partial fraction expansion where each complex shifted system issolved by a preconditioned simplified QMR method [18]. The preconditioneris a complex symmetric LDLT incomplete factorization of the shifted matrix,obtained by a simple modification of the factors computed with the MAT-LAB luinc factorization with dropping tolerance equal to 10−2. The systemstopping threshold is 10−ν .
• SI+PCG. Shift-invert Lanczos where systems with I − σA are solved withpreconditioned conjugate gradients. The MATLAB cholinc function with
Example 5.2. CPU time (and the number of iterations in parentheses when appropriate) for allmethods when systems with shifted matrices are solved with a direct method. Different dimensionproblems and various stopping tolerances are reported.
dropping tolerance 10−2 is used to generate the preconditioner. The innersystem stopping threshold is 10−ν .
• AK+Variant. We report the results of Table 4.3 of the variant of the Axelsson–Kucherov method, which solves systems with B = τI−A and B2 with a singleiterative method. If occurring, the system with the real pole is solved withpreconditioned conjugate gradients as in SI+PCG. The inner system stoppingthreshold is 10−ν−3.
In SI+PCG and AK+Variant, the shifted matrix was reordered with a Cuthill–McKee permutation (MATLAB function symrcm) before building the preconditioner,whereas minimum degree reordering was used for PFE+QMR; see [5] for a comprehen-sive discussion of various permutations related to preconditioning. We should mentionthat, in SI, it is not necessary to solve the inner system at high accuracy but that,on the contrary, the accuracy can be relaxed as convergence takes place [41], [38]. Wepostpone the exploration of this alternative to the next section, where enhancementstrategies for the PFE+QMR algorithm are also devised.
The CPU times for the two test problems are reported in Table 5.3 (the resultsfor n = 125 are omitted). For SI+PCG and AK+Variant, the total number of outeriterations and the average number of inner iterations are shown. For PFE+QMR, theaverage number of iterations is also shown in parentheses. For ease of comparison, wealso reproduce the CPU time of standard Lanczos from Tables 5.1–5.2.
Compared to the previous results that used a direct solver for the shifted systems,we can see that the overall costs have significantly decreased for all methods except SI.In the case of the 3D Laplace operator (Example 5.1), the standard Lanczos methodremains the method of choice even after a two-step procedure, although the differ-ences are far less prominent. For the 2D operator the solution of the shifted systems in(1.1) with an appropriately preconditioned iterative solver yields the most competitiveapproach, even for the small size problem. It appears that, for these examples, thetwo preconditioners (A − τI), corresponding to SI and the incomplete LDLT factor-ization for PFE, show comparable performance in approximating the PFE, in spiteof stemming from rather different approximation strategies.
6. Further tests. In this section we explore performance enhancements for thetwo methods SI+PCG and PFE+QMR. All experiments in this section were carriedout on one processor of a Sun Fire V40z with 2390.895 MHz and 16 GB RAM, runningMATLAB 7.4. We first discuss some natural implementation improvements for bothalgorithms and then analyze their performance on a time-stepping problem, so as to
provide a more realistic framework.As already mentioned, the inner-outer version of SI may be implemented so as to
relax the accuracy with which the inner system is solved at each Lanczos iteration.Given a fixed tolerance ε > 0, in [41, (5.4)] the following stopping tolerance ηj for theinner system was proposed:
ηj =ε
‖ej−1‖ + ε,
where ej−1 is the error in the approximation of the exponential operator at the previ-ous iteration, j − 1. In practice, ‖ej−1‖ is replaced by an estimate; in our implemen-tation we used the estimate associated with (4.9) in [41], and we fixed ε to be equalto the initial inner tolerance. Clearly, ηj increases towards one as ‖ej−1‖ → 0, so thatthe inner solver may be stopped earlier as the outer iteration converges, thus hope-fully decreasing the overall computational costs. We refer to [38], [42] for a generaldiscussion on relaxation strategies.
A straightforward enhancement for the partial fraction evaluation (hereafterPFE+QMR+mono) is to compute a single preconditioner, and then to apply it to allsystems. This strategy makes the SI+PCG and the PFE+QMR methods even closerto each other, since we have shown that SI+PCG may be viewed as a special way ofpreconditioning the PFE systems with a single, parameter dependent matrix. Here,we take as a single preconditioner of PFE+QMR+mono the factor of the incompleteCholesky factorization (MATLAB 7.4 function cholinc) with dropping tolerance 10−2
of the shifted complex symmetric matrix A−ξ1I, where ξ1 is the pole with the largestimaginary part, which provided the best performance. Reordering with symamd ofthe matrix A (and of I) was performed before calling cholinc. The results obtainedfor the problem of Example 5.2 are reported in Table 6.1. The original methodsPFE+QMR and SI+PCG, together with their enhanced versions, PFE+QMR+monoand SI+PCG+relax, are displayed. For SI+PCG+relax, the initial inner tolerance was
Parabolic problem (cf. (6.1)). CPU times of standard Lanczos and of enhanced acceleratedmethods to approximate the solution at T = 0.1, for different time step lengths δt and differentnumber of nodes nx, ny in the discretization of the domain (0, 1)2.
Grid Final Standard SI+PCG PFE+QMR(nx, ny) accuracy δt Lanczos relax mono
equal to the outer tolerance. The improvement over the corresponding original methodis significant, reaching almost 50% for SI+PCG+relax in some instances. Note that,on this problem, the single preconditioned PFE+QMR+mono is very effective for allsystems, allowing for the same average number of iterations as of the original method.In bold are the best timings, which show that the two different enhanced precondi-tioned techniques, PFE+QMR+mono and SI+PCG+relax, behave quite similarly.In general, timings are so close, the difference being within the MATLAB timingsfluctuation that it is difficult to depict a clear winner.
ACCELERATION METHODS FOR THE MATRIX EXPONENTIAL 681
We next consider the discretization of the following parabolic equation in twospatial dimensions [41]:
∂
∂tu = L(u), (x, y) ∈ (0, 1)2, 0 ≤ t ≤ T,(6.1)
where the solution u = u(t, x, y) is subject to the initial condition u(0, x, y) = u0(x, y)and to mixed boundary conditions (b.c.): homogeneous Dirichlet b.c. on the westernand eastern boundaries, and homogeneous Neumann b.c. on the northern and south-ern boundaries of the domain. The operator L is as in Example 5.2. After standardcentered finite difference space discretization, the solution at t = T is approximatedby a sequence of T/δt applications of exp(δtA) as exp(δtA) · · · exp(δtA)u0, where δtis the time step length and u0 is the initial vector. We considered different possiblespace and time discretizations so as to approximate the exact solution at T = 0.1.The results of our experiments for δt = 0.005, 0.01, 0.05, 0.1 are reported in Table6.2; u0 is the normalized vector of all ones. The standard Lanczos and the enhancedversions of the SI and PFE methods are considered. Different final accuracies werealso used, which are of interest in the context of evolution problems. All inner andouter stopping thresholds were tuned so as to reach the requested final accuracy.
The acceleration procedures allow the discretization process to take much largertime steps than with standard Lanczos, to the point that in all examples a singletime step (δt = 0.1) is faster than the best Lanczos timing. This is clearly a welcomeevent and is one of the main reasons for using acceleration procedures in the context ofparabolic equations. In addition, we explicitly observe that as the number of time stepsdecreases, so does the cost of the acceleration procedures, whereas that of standardLanczos becomes unacceptably high due to the increasing value of ‖δtA‖.
In PFE, the common preconditioner is computed once and for all, whereas eachshifted system is solved separately. This is the major remaining drawback of theenhanced PFE+QMR method when a few time steps are performed, since manysystems need to be solved. On the other hand, SI precisely avoids this step, sinceit constructs a single preconditioner that, in the case of a rational function, stillallows one to keep the shifted form of the systems, so that all systems can be solvedsimultaneously with a single SI iteration as in (3.6); see also [19]. Albeit limited,our numerical experiments confirm that the relaxed SI method is able to efficientlysolve the parabolic system when few time steps are taken, compared to standardLanczos. As already noticed in the previous example, a single time step makes thegeneric enhanced PFE procedure more competitive. Due to these favorable resultsof the PFE+QMR method, it would be interesting to further explore enhancementtechniques for this approach, such as those in [7].
7. Conclusions. In this paper we have presented a common framework for somerecently developed acceleration techniques for approximating the action of the matrixexponential to a vector. This framework is based on the rational function approxi-mation to the exponential, which allows one to transform the approximation probleminto that of solving several algebraic linear systems. It is thus natural to compare theperformance of the acceleration techniques with that of methods such as PFE thatexplicitly solve these systems. We can summarize our theoretical and experimentalfindings as follows:
(i) Whenever the exponential is replaced by its rational function approximation,we have shown that the analyzed methods SI and AK are simply different ways ofpreconditioning the given linear systems.
(ii) Our framework allowed us to derive an a priori fully algebraic and iteration-free selection of the involved single parameter in both methods.
(iii) We have performed a numerical comparison among various acceleration meth-ods, including their enhanced versions, thus filling a gap in the current literature. Inour opinion, these experiments provide a new perspective and new insights on whenand which acceleration procedures should be preferred. The experiment on a parabolic2D problem shows the effectiveness of the enhanced SI and PFE+QMR processes,allowing the time discretization for truly large iteration steps.
Our findings are in fact quite general. It would be interesting to see whethersimilar conclusions can be generalized to other functions for which a rational functionapproximation is available; see, e.g., [19]. The case of nonsymmetric A is also verychallenging, since the rational Chebyshev approximation is not optimal in this case.
Acknowledgments. The authors would like to thank Marlis Hochbruck andIgor Moret for insightful conversations, and the two anonymous referees for theirconstructive criticism, which led to the addition of section 6. Finally, we are gratefulto Andreas Frommer for his handling of the manuscript.
REFERENCES
[1] O. Axelsson and A. Kucherov, Real valued iterative methods for solving complex symmetriclinear systems, Numer. Linear Algebra Appl., 7 (2000), pp. 197–218.
[2] Z. Bai, J. Demmel, J. Dongarra, A. Ruhe, and H. A. van der Vorst, eds., Templates for theSolution of Algebraic Eigenvalue Problems: A Practical Guide, SIAM, Philadelphia, 2000.
[3] G. A. Baker and P. Graves-Morris, Pade Approximants, Encyclopedia Math. Appl. 59,2nd ed., Cambridge University Press, Cambridge, UK, 1996.
[4] C. Baldwin, R. Freund, and E. Gallopoulos, A parallel iterative method for exponentialpropagation, in Proceedings of the Seventh SIAM Conference on Parallel Processing forScientific Computing, D. Bailey et al., eds., SIAM, Philadelphia, 1995, pp. 534–539.
[5] M. Benzi, Preconditioning techniques for large linear systems: A survey, J. Comput. Phys.,182 (2002), pp. 418–477.
[6] M. Benzi and D. Bertaccini, Approximate inverse preconditioning for shifted linear systems,BIT, 43 (2003), pp. 231–244.
[7] M. Benzi and D. Bertaccini, Block Preconditioning of Real-valued Iterative Algorithms forComplex Linear Systems, to appear in IMA J. Numer. Anal.
[8] L. Bergamaschi and M. Vianello, Efficient computation of the exponential operator forlarge, sparse, symmetric matrices, Numer. Linear Algebra Appl., 7 (2000), pp. 27–45.
[9] A. J. Carpenter, A. Ruttan, and R. S. Varga, Extended numerical computations onthe 1/9 conjecture in rational approximation theory, in Rational Approximation andInterpolation, Lecture Notes in Math. 1105, P. R. Graves-Morris, E. B. Saff, and R. S.Varga, eds., Springer-Verlag, Berlin, 1984, pp. 383–411.
[10] P. Castillo and Y. Saad, Preconditioning the matrix exponential operator with applications,J. Sci. Comput., 13 (1999), pp. 275–302.
[11] W. J. Cody, G. Meinardus, and R. S. Varga, Chebyshev rational approximations to e−x
in [0,+∞) and applications to heat-conduction problems, J. Approx. Theory, 2 (1969),pp. 50–65.
[12] V. Druskin, A. Greenbaum, and L. Knizhnerman, Using nonorthogonal Lanczos vectors inthe computation of matrix functions, SIAM J. Sci. Comput., 19 (1998), pp. 38–54.
[13] V. Druskin and L. Knizhnerman, Two polynomial methods of calculating functions ofsymmetric matrices, USSR Comput. Math. Math. Phys., 29 (1989), pp. 112–121.
[14] V. Druskin and L. Knizhnerman, Krylov subspace approximation of eigenpairs and matrixfunctions in exact and computer arithmetic, Numer. Linear Algebra Appl., 2 (1995),pp. 205–217.
[15] V. Druskin and L. Knizhnerman, Extended Krylov subspaces: Approximation of the matrixsquare root and related functions, SIAM J. Matrix Anal. Appl., 19 (1998), pp. 755–771.
[16] M. Eiermann and O. Ernst, A restarted Krylov subspace method for the evaluation of matrixfunctions, SIAM J. Numer. Anal., 44 (2006), pp. 2481–2504.
ACCELERATION METHODS FOR THE MATRIX EXPONENTIAL 683
[17] R. W. Freund, On conjugate gradient type methods and polynomial preconditioners for aclass of complex non-Hermitian matrices, Numer. Math., 57 (1990), pp. 285–312.
[18] R. W. Freund and N. M. Nachtigal, Software for simplified Lanczos and QMR algorithms,Appl. Numer. Math., 19 (1995), pp. 319–341.
[19] A. Frommer and V. Simoncini, Stopping criteria for rational matrix functions of Hermitianand symmetric matrices, SIAM J. Sci. Comput., 30 (2008), pp. 1387–1412
[20] E. Gallopoulos and Y. Saad, Efficient solution of parabolic equations by Krylov approxi-mation methods, SIAM J. Sci. Statist. Comput., 13 (1992), pp. 1236–1264.
[21] G. Golub and C. F. Van Loan, Matrix Computations, 3rd ed., The Johns Hopkins UniversityPress, Baltimore, 1996.
[22] N. J. Higham, The scaling and squaring method for the matrix exponential revisited, SIAMJ. Matrix Anal. Appl., 26 (2005), pp. 1179–1193.
[23] M. Hochbruck and M. E. Hochstenbach, Subspace Extraction for Matrix Functions,preprint, 2005.
[24] M. Hochbruck and C. Lubich, On Krylov subspace approximations to the matrix exponentialoperator, SIAM J. Numer. Anal., 34 (1997), pp. 1911–1925.
[25] M. Hochbruck and C. Lubich, Exponential integrators for quantum-classical moleculardynamics, BIT, 39 (1999), pp. 620–645.
[26] A. Iserles, How large is the exponential of a banded matrix?, New Zealand J. Math., 29(2000), pp. 177–192.
[27] L. Lopez and V. Simoncini, Analysis of projection methods for rational function approxima-tion to the matrix exponential, SIAM J. Numer. Anal., 44 (2006), pp. 613–635.
[28] The MathWorks, Inc., MATLAB 7, 2004.[29] C. Moler and C. Van Loan, Nineteen dubious ways to compute the exponential of a matrix,
twenty-five years later, SIAM Rev., 45 (2003), pp. 3–49.[30] I. Moret, On RD-rational Krylov approximations to the core-functions of exponential
integrators, Numer. Linear Algebra Appl., 14 (2007), pp. 445–457.[31] I. Moret and P. Novati, An interpolatory approximation of the matrix exponential based on
Faber polynomials, J. Comput. Appl. Math., 131 (2001), pp. 361–380.[32] I. Moret and P. Novati, RD-rational approximations of the matrix exponential, BIT, 44
(2004), pp. 595–615.[33] I. Moret and P. Novati, Interpolating functions of matrices on zeros of quasi-kernel
polynomials, Numer. Linear Algebra Appl., 11 (2005), pp. 337–353.[34] Y. Saad, Analysis of some Krylov subspace approximations to the matrix exponential operator,
SIAM J. Numer. Anal., 29 (1992), pp. 209–228.[35] Y. Saad, Iterative Methods for Sparse Linear Systems, 2nd ed., SIAM, Philadelphia, 2003.[36] E. B. Saff, A. Schonhage, and R. S. Varga, Geometric convergence to e−z by rational
functions with real poles, Numer. Math., 25 (1976), pp. 307–322.[37] R. B. Sidje, Expokit: A Software Package for Computing Matrix Exponentials, ACM Trans.
Math. Softw., 24 (1998), pp. 130–156.[38] V. Simoncini and D. B. Szyld, Theory of inexact Krylov subspace methods and applications
to scientific computing, SIAM J. Sci. Comput., 25 (2003), pp. 454–477.[39] V. Simoncini and D. B. Szyld, Recent computational developments in Krylov subspace
methods for linear systems, Numer. Linear Algebra Appl., 14 (2007), pp. 1–59.[40] L. N. Trefethen, J. A. C. Weideman, and T. Schmelzer, Talbot quadratures and rational
approximations, BIT, 46 (2006), pp. 653–670.[41] J. van den Eshof and M. Hochbruck, Preconditioning Lanczos approximations to the
matrix exponential, SIAM J. Sci. Comput., 27 (2006), pp. 1438–1457.[42] J. van den Eshof and G. L. G. Sleijpen, Inexact Krylov subspace methods for linear
systems, SIAM J. Matrix Anal. Appl., 26 (2004), pp. 125–153.[43] H. A. van der Vorst, An iterative solution method for solving f(A)x = b using Krylov
subspace information obtained for the symmetric positive definite matrix A, J. Comput.Appl. Math., 18 (1987), pp. 249–263.
