Iterative methods for nonlinear systems of equations:
an introduction
Laboratori de Càlcul Numèric (LaCàN)Dep. de Matemàtica Aplicada III
Universitat Politècnica de Catalunyawww-lacan.upc.es
2
Contents
§ Problem statement§ Motivation§ Functional iteration§ Method of direct iteration§ Picard’s method§ Newton’s method§ Quasi-Newton methods
3
Problem statement
§ Starting point
§ General form of nonlinear system
(1 nonlinear equation with 1 unknown)
(linear system of order n)
f transforms vectors into vectors
α is a zero of f if
4
Problem statement
In more detail:
In general, all components of f are nonlinear w.r.t. all components of x
5
Problem statement
§ Particular form of nonlinear system
with and
Similar structure to linear system
Can be (trivially) transformed into the general form:
6
Motivation
Mathematical models in engineeringLinear models§ Response of physical system proportional to external actions§ Simple models§ A first approximation to the real behaviour
Examples: linear systems of equations; linear PDEs
¬ Nonlinear models§ No proportionality between actions and response§ More complex models§ More realistic description of real beaviourExamples: nonlinear systems of equations; nonlinear PDEs
7
Functional iteration
§ Analogy with root finding in 1-D:1-D problem n-D problem
§ Consistency: function φ must verify
(zeros of f) (fixed points of φ)
Nonlinear equation(s)
Initial approximation
Iterative scheme
8
Functional iteration
§ Convergence: contractive mapping theorem
Let φ: D D, D a closed subset of R . If there exists λ ∈ [0,1) such that
then:(a) there exists a unique fixed point α of φ in D.
(b) for any initial approximation x in D, the sequence {x } generated by x = φ(x )remains in D and converges linearly to α with constant λ.
n
0k kk+1
9
Method of direct iteration(or successive approximations)
Advantages§ Very simple technique (evaluate f once per iteration)Drawbacks§ Contractivity of φ not guaranteed§ Convergence is typically linear (if it converges!)
Problem in general form
Iteration function
Iteration scheme
10
Method of direct iteration(or successive approximations)
11
Picard’s method(or secant matrix method)
Problem in particular form
If matrix is inversible,
define the iteration function
so the iteration scheme is
Attention: do not invert matrix!
12
Picard’s method(or secant matrix method)
§ Solve one linear system per iteration
§ Matrix A(x) and vector b(x) one iteration behind
If matrix is inversible,
Practical algorithm
13
Picard’s method(or secant matrix method)
Matrix A(x) is a secant matrix
14
Picard’s method(or secant matrix method)
Advantages§ If A(x) has a special structure (e.g. banded SPD),
it can be exploited when solving the linear systems
Drawbacks§ Matrix A(x) may be singular for some x
§ Convergence is typically linear (if it converges!)
§ Computational cost: matrix A(x) and vector b(x) change at every iteration
15
Newton’s method(or Newton-Raphson’s method)
Problem in general form
Correction of non-converged approximation:
First-order Taylor’s series expansion:
16
Newton’s method(or Newton-Raphson’s method)
Jacobian matrix
Involves the computation of derivatives
17
Newton’s method(or Newton-Raphson’s method)
Practical algorithm
Iteration function is
Jacobian matrix not inverted in practice!
18
Newton’s method(or Newton-Raphson’s method)
19
Newton’s method(or Newton-Raphson’s method)
For problem in particular form, the iterative scheme is
similar to iterative schemes for linear systems
The Jacobian matrix does not retain structure of A(x)
20
Newton’s method(or Newton-Raphson’s method)
Advantages§ Convergence is quadratic (for J(α) not singular)
Drawbacks§ Matrix J(x) may be singular for some x
§ Computational cost: at every iteration, (1) compute matrix J(x) and vector f(x) and (2) solve linear system
§ If A(x) has a special structure (e.g. banded SPD), it is lost when computing J(x)
21
Quasi-Newton methods§ Secant method in 1-D
Similar to Newton’s method
with tangent approximated
by secant defined by the last two iterations k-1 and k
22
Quasi-Newton methods
23
Quasi-Newton methods§ Extension to nonlinear systems
Jacobian matrix is approximated by a secant matrix
defined by the last two iterations k-1 and k
n unknowns and only n equations additional conditions on matrix S required
2
