Numerical behavior of inexact saddle point solvers
Pavel Jiranek1,2, Miroslav Rozloznık1,2
Faculty of Mechatronics and Interdisciplinary Engineering Studies,Technical University of Liberec,
Czech Republic1
and
Institute of Computer Science,Czech Academy of Sciences, Prague,
Czech Republic2
9th IMACS International Symposiumon Iterative Methods in Scientific Computings
March 17–20, 2008, Lille, France
1 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Saddle point problems
We consider a saddle point problem with the symmetric 2× 2 block form„A BBT 0
«„xy
«=
„f0
«.
A is a square n× n nonsingular (symmetric positive definite) matrix,
B is a rectangular n×m matrix of (full column) rank m.
Applications: mixed finite element approximations, weighted least squares,constrained optimization etc. [Benzi, Golub, and Liesen, 2005].
2 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
inexact solutions of inner systems + rounding errors→ inexact saddle point solver
3 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Schur complement reduction method
Compute y as a solution of the Schur complement system
BTA−1By = BTA−1f,
compute x as a solution of
Ax = f −By.
Systems with A are solved inexactly, the computed solution u of Au = b isinterpreted an exact solution of a perturbed system
(A+ ∆A)u = b+ ∆b, ‖∆A‖ ≤ τ‖A‖, ‖∆b‖ ≤ τ‖b‖, τκ(A)� 1.
4 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Iterative solution of the Schur complement system
choose y0, solve Ax0 = f −By0
compute αk and p(y)k
yk+1 = yk + αkp(y)k˛
˛˛˛
solve Ap(x)k = −Bp(y)
k
back-substitution:
A: xk+1 = xk + αkp(x)k ,
B: solve Axk+1 = f −Byk+1,
C: solve Auk = f −Axk −Byk+1,
xk+1 = xk + uk.
9>>>>>>>>>=>>>>>>>>>;inneriteration
r(y)k+1 = r
(y)k − αkB
T p(x)k
9>>>>>>>>>>>>>>>>>>>>>=>>>>>>>>>>>>>>>>>>>>>;
outeriteration
5 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Measure of the limiting accuracy
The limiting (maximum attainable) accuracy is measured by the ultimate(asymptotic) values of:
1 the Schur complement residual: BTA−1f −BTA−1Byk;
2 the residuals in the saddle point system: f −Axk −Byk and −BTxk;
3 the forward errors: x− xk and y − yk.
Numerical example:
A = tridiag(1, 4, 1) ∈ R100×100, B = rand(100, 20), f = rand(100, 1),
κ(A) = ‖A‖ · ‖A−1‖ = 7.1695 · 0.4603 ≈ 3.3001,
κ(B) = ‖B‖ · ‖B†‖ = 5.9990 · 0.4998 ≈ 2.9983.
6 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Accuracy in the outer iteration process
BT (A+ ∆A)−1By = BT (A+ ∆A)−1f,
‖BTA−1f −BTA−1By‖ ≤ τκ(A)
1− τκ(A)‖A−1‖‖B‖2‖y‖.
‖ −BTA−1f +BTA−1Byk − r(y)k ‖ ≤
O(τ)κ(A)
1− τκ(A)‖A−1‖‖B‖(‖f‖+ ‖B‖Yk).
0 50 100 150 200 250 30010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration number k
rela
tive
resi
dual
nor
ms
||BTA
−1 f−
BTA
−1 B
y k||/||B
TA
−1 f−
BTA
−1 B
y 0||, ||
r(y)
k||/
||r(y
)0
||
7 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Accuracy in the saddle point system
−BTA−1f +BTA−1Byk = −BTxk −BTA−1(f −Axk −Byk)
‖f −Axk −Byk‖ ≤O(α1)κ(A)
1− τκ(A)(‖f‖+ ‖B‖Yk),
‖ −BTxk − r(y)k ‖ ≤
O(α2)κ(A)
1− τκ(A)‖A−1‖‖B‖(‖f‖+ ‖B‖Yk),
Yk ≡ max{‖yi‖ | i = 0, 1, . . . , k}.
Back-substitution scheme α1 α2
A: Generic update
xk+1 = xk + αkp(x)k
τ u
B: Direct substitutionxk+1 = A−1(f −Byk+1)
τ τ
C: Corrected dir. subst.xk+1 = xk +A−1(f −Axk −Byk+1)
u τ
}additionalsystem with A
8 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Generic update: xk+1 = xk + αkp(x)k
0 50 100 150 200 250 30010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration number k
resi
dual
nor
m ||
f−A
x k−B
y k||
0 50 100 150 200 250 30010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u), τ = 10−10, τ=10−6, τ=10−2
iteration number k
rela
tive
resi
dual
nor
ms
||−B
Tx k||/
||−B
Tx 0||,
||r k(y
) ||/||r
0(y) ||
9 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Direct substitution: xk+1 = A−1(f −Byk+1)
0 50 100 150 200 250 30010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration number k
resi
dual
nor
m ||
f−A
x k−B
y k||
0 50 100 150 200 250 30010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration number k
rela
tive
resi
dual
nor
ms
||−B
Tx k||/
||−B
Tx 0||,
||r k(y
) ||/||r
0(y) ||
10 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Corrected direct substitution: xk+1 = xk +A−1(f −Axk −Byk+1)
0 50 100 150 200 250 30010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u), τ = 10−10, τ=10−6, τ=10−2
iteration number k
resi
dual
nor
m ||
f−A
x k−B
y k||
0 50 100 150 200 250 30010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration number k
rela
tive
resi
dual
nor
ms
||−B
Tx k||/
||−B
Tx 0||,
||r k(y
) ||/||r
0(y) ||
11 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Forward error of computed approximate solution
‖x− xk‖ ≤ γ1‖f −Axk −Byk‖+ γ2‖ −BTxk‖,
‖y − yk‖ ≤ γ2‖f −Axk −Byk‖+ γ3‖ −BTxk‖,
γ1 = σ−1min(A), γ2 = σ−1
min(B), γ3 = σ−1min(BTA−1B).
0 50 100 150 200 250 30010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration number k
rela
tive
erro
r no
rms
||x−
x k|| A/||
x−x 0|| A
, ||y
−y k|| B
TA
−1 B
/||y−
y 0|| BTA
−1 B
12 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Conclusions
All bounds of the limiting accuracy depend on the maximum norm ofcomputed iterates, cf. [Greenbaum, 1997].
The accuracy measured by the residuals of the saddle point problemdepends on the choice of the back-substitution scheme [J, R, 2008].
Care must be taken when solving nonsymmetric systems [J, R, 2008b].
0 50 100 150 200 250 300 350 400 450
10−15
10−10
10−5
100
105
iteration number k
rela
tive
resi
dual
nor
ms
||BTA
−1 f−
BTA
−1 B
y k||/||B
TA
−1 f||
and
||r(y
)k
||/||B
TA
−1 f||
The residuals in the outer iteration process and the forward errors ofcomputed approximations are proportional to the backward error insolution of inner systems.
13 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Thank you for your attention.
http://www.cs.cas.cz/∼miro
P. Jiranek and M. Rozloznık. Maximum attainable accuracy of inexact saddlepoint solvers. SIAM J. Matrix Anal. Appl., 29(4):1297–1321, 2008.
P. Jiranek and M. Rozloznık. Limiting accuracy of segregated solution methodsfor nonsymmetric saddle point problems. J. Comput. Appl. Math., 215:28–37,2008.
14 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Null-space projection method
compute x ∈ N(BT ) as a solution of the projected system
(I −Π)A(I −Π)x = (I −Π)f,
compute y as a solution of the least squares problem
By ≈ f −Ax,
Π is the orthogonal projector onto R(B).
The least squares with B are solved inexactly, i.e. the computed solution v ofBv ≈ c is an exact solution of a perturbed least squares problem
(B + ∆B)v ≈ c+ ∆c, ‖∆B‖ ≤ τ‖B‖, ‖∆c‖ ≤ τ‖c‖, τκ(B)� 1.
15 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Iterative solution of the null-space projected system
choose x0, solve By0 ≈ f −Ax0
compute αk and p(x)k ∈ N(BT )
xk+1 = xk + αkp(x)k˛
˛˛˛
solve Bp(y)k ≈ r(x)
k − αkAp(x)k
back-substitution:
A: yk+1 = yk + p(y)k ,
B: solve Byk+1 ≈ f −Axk+1,
C: solve Bvk ≈ f −Axk+1 −Byk,
yk+1 = yk + vk.
9>>>>>>>>>=>>>>>>>>>;inneriteration
r(x)k+1 = r
(x)k − αkAp
(x)k −Bp(y)
k
9>>>>>>>>>>>>>>>>>>>>>=>>>>>>>>>>>>>>>>>>>>>;
outeriteration
16 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Accuracy in the saddle point system
‖f −Axk −Byk − r(x)k ‖ ≤
O(α3)κ(B)
1− τκ(B)(‖f‖+ ‖A‖Xk),
‖ −BTxk‖ ≤O(τ)κ(B)
1− τκ(B)‖B‖Xk,
Xk ≡ max{‖xi‖ | i = 0, 1, . . . , k}.
Back-substitution scheme α3
A: Generic update
yk+1 = yk + p(y)k
u
B: Direct substitutionyk+1 = B†(f −Axk+1)
τ
C: Corrected dir. subst.yk+1 = yk +B†(f −Axk+1 −Byk)
u
}additional leastsquare with B
17 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Generic update: yk+1 = yk + p(y)k
0 10 20 30 40 50 60 70 80 90 10010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u), τ = 10−10, τ=10−6, τ=10−2
iteration number
rela
tive
resi
dual
nor
ms
||f−
Ax k−
By k||/
||f−
Ax 0−
By 0||,
||r(x
)k
||/||r
(x)
0||
0 10 20 30 40 50 60 70 80 90 10010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration numberre
sidu
al n
orm
||−
BTx k||
18 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Direct substitution: yk+1 = B†(f −Axk+1)
0 10 20 30 40 50 60 70 80 90 10010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration number
rela
tive
resi
dual
nor
ms
||f−
Ax k−
By k||/
||f−
Ax 0−
By 0||,
||r(x
)k
||/||r
(x)
0||
0 10 20 30 40 50 60 70 80 90 10010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration numberre
sidu
al n
orm
||−
BTx k||
19 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
Corrected direct substitution: yk+1 = yk +B†(f −Axk+1 −Byk)
0 10 20 30 40 50 60 70 80 90 10010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u), τ = 10−10, τ=10−6, τ=10−2
iteration number
rela
tive
resi
dual
nor
ms
||f−
Ax k−
By k||/
||f−
Ax 0−
By 0||,
||r(x
)k
||/||r
(x)
0||
0 10 20 30 40 50 60 70 80 90 10010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
τ = O(u)
τ = 10−2
τ = 10−6
τ = 10−10
iteration numberre
sidu
al n
orm
||−
BTx k||
20 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers
References
M. Benzi, G. H. Golub, and J. Liesen. Numerical solution of saddle pointproblems. Acta Numer., 14:1–137, 2005.
A. Greenbaum. Estimating the attainable accuracy of recursively computedresidual methods. SIAM J. Matrix Anal. Appl., 18(3):535–551, 1997.
P. Jiranek and M. Rozloznık. Maximum attainable accuracy of inexact saddlepoint solvers. SIAM J. Matrix Anal. Appl., 29(4):1297–1321, 2008a.
P. Jiranek and M. Rozloznık. Limiting accuracy of segregated solution methodsfor nonsymmetric saddle point problems. J. Comput. Appl. Math., 215:28–37, 2008b.
21 Pavel Jiranek, Miroslav Rozloznık Numerical behavior of inexact saddle point solvers