Home >Documents >Lecture 2: Randomized Iterative Methods for Linear · PDF file Randomized Iterative Methods...

Lecture 2: Randomized Iterative Methods for Linear · PDF file Randomized Iterative Methods...

Date post:30-May-2020
Category:
View:3 times
Download:0 times
Share this document with a friend
Transcript:
  • Lecture 2: Randomized Iterative Methods for Linear Systems

    February 21 - 28, 2020

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 1 / 43

  • 1. Pseudoinverse solutions of linear systems

    Consider a linear system of equations iℓ

    Ax = b, A ∈ Rm×n, b ∈ Rm.

    The system is called consistent if b ∈ range(A), otherwise, inconsistent.

    We are interested in the pseudoinverse solution A†b, where A†

    denotes the Moore-Penrose pseudoinverse of A.

    Ax = b rank(A) A†b consistent = n unique solution consistent < n unique minimum ℓ2-norm solution inconsistent = n unique least-squares (LS) solution inconsistent < n unique minimum ℓ2-norm LS solution

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 2 / 43

  • Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 3 / 43

  • 2. Notation and preliminaries

    For any random variable ξ, let E 󰀅 ξ 󰀆 denote its expectation.

    For an integer m ≥ 1, let

    [m] := {1, 2, 3, . . . ,m}.

    For any vector u ∈ Rm, we use uT and 󰀂u󰀂2 to denote the transpose and the Euclidean norm (ℓ2-norm) of u, respectively.

    I: the identity matrix whose order is clear from the context.

    For any matrix A ∈ Rm×n, we use AT, A†, 󰀂A󰀂F, range(A),

    σ1(A) ≥ σ2(A) ≥ · · · ≥ σr(A) > 0

    to denote the transpose, the Moore-Penrose pseudoinverse, the Frobenius norm, the column space, and all the nonzero singular values of A, respectively. Obviously, r is the rank of A.

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 4 / 43

  • For index sets I ⊆ [m] and J ⊆ [n], let AI,:, A:,J , and AI,J denote the row submatrix indexed by I, the column submatrix indexed by J , and the submatrix that lies in the rows indexed by I and the columns indexed by J , respectively. Let {I1, I2, . . . , Is} denote a partition of [m], that is,

    Ii ∩ Ij = ∅, ∪si=1Ii = [m].

    Let {J1,J2, . . . ,Jt} denote a partition of [n]. Let

    P = {I1, I2, . . . , Is}× {J1,J2, . . . ,Jt}

    Lemma 1

    For any vector u ∈ Rm and any matrix A ∈ Rm×n, it holds

    uTAATu ≤ 󰀂A󰀂2FuTu.

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 5 / 43

  • Lemma 2

    For any matrix A ∈ Rm×n with rank r and any vector u ∈ range(A), it holds

    uTAATu ≥ σ2r (A)󰀂u󰀂22.

    Lemma 3

    Let α > 0 and A be any nonzero real matrix. For every u ∈ range(A), it holds

    󰀐󰀐󰀐󰀐󰀐

    󰀕 I− αAA

    T

    󰀂A󰀂2F

    󰀖k u

    󰀐󰀐󰀐󰀐󰀐 2

    ≤ 󰀕 max 1≤i≤r

    󰀏󰀏󰀏󰀏1− ασ2i (A)

    󰀂A󰀂2F

    󰀏󰀏󰀏󰀏

    󰀖k 󰀂u󰀂2.

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 6 / 43

  • 3. A doubly stochastic block Gauss-Seidel algorithm [2]

    Algorithm 1: Doubly stochastic block Gauss-Seidel (DSBGS)

    Let α > 0. Initialize x0 ∈ Rn for k = 1, 2, . . . , do

    Pick (I,J ) ∈ P with probability 󰀂AI,J 󰀂 2 F

    󰀂A󰀂2F Set xk = xk−1 − αI:,J (AI,J )

    T(I:,I)T

    󰀂AI,J 󰀂2F (Axk−1 − b)

    Landweber [3] (s = 1 and t = 1)

    xk = xk−1 − α A T

    󰀂A󰀂2F (Axk−1 − b).

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 7 / 43

  • Randomized Kaczmarz (RK) [7] (s = m, t = 1, α = 1)

    At step k, RK projects xk−1 onto the hyperplane {x | Ai,:x = bi},

    xk = xk−1 − Ai,:x k−1 − bi

    󰀂Ai,:󰀂22 (Ai,:)

    T,

    where Ai,: is the ith row of A and bi is the ith component of b.

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 8 / 43

  • Randomized Gauss-Seidel [4][5] (s = 1, t = n, α = 1)

    xk = xk−1 − (A:,j) T(Axk−1 − b) 󰀂A:,j󰀂22

    I:,j ,

    where A:,j is the jth column of A and I:,j is the jth column of the n× n identity matrix I.

    Doubly stochastic Gauss-Seidel [6] (s = m, t = n)

    xk = xk−1 − αAi,j(Ai,:x k−1 − bi)

    |Ai,j |2 I:,j ,

    where Ai,j is the (i, j) entry of A.

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 9 / 43

  • 3.1 Convergence of the norms of the expectations

    Theorem 4

    Let xk denote the kth iterate of DSBGS applied to the consistent linear system

    Ax = b

    with arbitrary x0 ∈ Rn. In exact arithmetic, it holds

    󰀂E[xk − x0󰂏]󰀂2 ≤ 󰀕 max 1≤i≤r

    󰀏󰀏󰀏󰀏1− ασ2i (A)

    󰀂A󰀂2F

    󰀏󰀏󰀏󰀏

    󰀖k 󰀂x0 − x0󰂏󰀂2,

    where x0󰂏 = (I−A†A)x0 +A†b

    i.e., the projection of x0 onto the solution set

    {x ∈ Rn | Ax = b}.

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 10 / 43

  • Proof of Theorem 4:

    Note that the conditioned expectation on xk−1

    E[xk |xk−1]

    = xk−1 − αE 󰀗 I:,J (AI,J )T(I:,I)T

    󰀂AI,J 󰀂2F

    󰀘 (Axk−1 − b)

    = xk−1 − α

    󰀳

    󰁃 󰁛

    (I,J )∈P

    I:,J (AI,J )T(I:,I)T

    󰀂AI,J 󰀂2F 󰀂AI,J 󰀂2F 󰀂A󰀂2F

    󰀴

    󰁄 (Axk−1 − b)

    = xk−1 − α A T

    󰀂A󰀂2F (Axk−1 − b).

    Then the conditioned expectation E[xk − x0󰂏 |xk−1] is given by

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 11 / 43

  • E[xk − x0󰂏 |xk−1] = E[xk |xk−1]− x0󰂏

    = xk−1 − α A T

    󰀂A󰀂2F (Axk−1 − b)− x0󰂏

    = xk−1 − α A T

    󰀂A󰀂2F (Axk−1 −Ax0󰂏)− x0󰂏

    =

    󰀕 I− αA

    TA

    󰀂A󰀂2F

    󰀖 (xk−1 − x0󰂏).

    Taking expectation gives

    E[xk − x0󰂏] = E[E[xk − x0󰂏 |xk−1]] = 󰀕 I− αA

    TA

    󰀂A󰀂2F

    󰀖 E[xk−1 − x0󰂏]

    =

    󰀕 I− αA

    TA

    󰀂A󰀂2F

    󰀖k (x0 − x0󰂏).

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 12 / 43

  • Applying the norms to both sides we obtain

    󰀂E[xk − x0󰂏]󰀂2 ≤ 󰀕 max 1≤i≤r

    󰀏󰀏󰀏󰀏1− ασ2i (A)

    󰀂A󰀂2F

    󰀏󰀏󰀏󰀏

    󰀖k 󰀂x0 − x0󰂏󰀂2.

    Here the inequality follows from the fact that

    x0 − x0󰂏 = A†Ax 0 −A†b ∈ range(AT)

    and Lemma 3.

    Remark 1

    If x0 ∈ range(AT), then x0󰂏 = A†b. To ensure convergence of the expected iterate, it suffices to have

    max 1≤i≤r

    󰀏󰀏󰀏󰀏1− ασ2i (A)

    󰀂A󰀂2F

    󰀏󰀏󰀏󰀏 < 1 i.e., 0 < α < 2󰀂A󰀂2F σ21(A)

    .

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 13 / 43

  • Theorem 5

    Let xk denote the kth iterate of DSBGS applied to the consistent or inconsistent linear system

    Ax = b

    with arbitrary x0 ∈ Rn. In exact arithmetic, it holds

    󰀂E[Axk −Ax󰂏]󰀂2 ≤ 󰀕 max 1≤i≤r

    󰀏󰀏󰀏󰀏1− ασ2i (A)

    󰀂A󰀂2F

    󰀏󰀏󰀏󰀏

    󰀖k 󰀂Ax0 −Ax󰂏󰀂2,

    where x󰂏 is any solution of

    ATAx = ATb.

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 14 / 43

  • Proof of Theorem 5:

    Note that the conditioned expectation on xk−1

    E[Axk −Ax󰂏 |xk−1] = A(E[xk |xk−1]− x󰂏)

    = A

    󰀕 xk−1 − α A

    T

    󰀂A󰀂2F (Axk−1 − b)− x󰂏

    󰀖

    = A

    󰀕 xk−1 − αA

    T

    󰀂A󰀂2F (Axk−1 −Ax󰂏)− x󰂏

    󰀖

    (by ATb = ATAx󰂏)

    = Axk−1 −Ax󰂏 − αAAT

    󰀂A󰀂2F (Axk−1 −Ax󰂏)

    =

    󰀕 I− αAA

    T

    󰀂A󰀂2F

    󰀖 (Axk−1 −Ax󰂏).

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 15 / 43

  • Taking expectation gives

    E[Axk −Ax󰂏] = E[E[Axk −Ax󰂏 |xk−1]]

    =

    󰀕 I− αAA

    T

    󰀂A󰀂2F

    󰀖 E[Axk−1 −Ax󰂏]

    =

    󰀕 I− αAA

    T

    󰀂A󰀂2F

    󰀖k (Ax0 −Ax󰂏).

    Applying the norms to both sides we obtain

    󰀂E[Axk −Ax󰂏]󰀂2 ≤ 󰀕 max 1≤i≤r

    󰀏󰀏󰀏󰀏1− ασ2i (A)

    󰀂A󰀂2F

    󰀏󰀏󰀏󰀏

    󰀖k 󰀂Ax0 −Ax󰂏󰀂2.

    Here the inequality follows from the fact that

    Ax0 −Ax󰂏 ∈ range(A)

    and Lemma 3.

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 16 / 43

  • 3.2 Convergence of the expected norms

    Theorem 6

    Let xk denote the kth iterate of DSBGS applied to the full column rank consistent linear system

    Ax = b

    with arbitrary x0 ∈ Rn. Assume

    0 < α < 2/t.

    In exact arithmetic, it holds

    E[󰀂xk −A†b󰀂22] ≤ 󰀕 1− (2α− tα

    2)σ2n(A)

    󰀂A󰀂2F

    󰀖k 󰀂x0 −A†b󰀂22.

    Randomized Iterative Methods Lecture 2 February 21 - 28, 2020 17 / 43

  • Proof of Theorem 6:

    󰀂xk −A†b󰀂22

    =

    󰀐󰀐󰀐󰀐x k−1 − α

    󰀕 I:,J (AI,J )

    T(I:,I) T

    󰀂AI,J 󰀂2F

    󰀖 (Axk−1 − b)−A†b

    󰀐󰀐󰀐󰀐 2

    2

    =

    󰀐󰀐󰀐󰀐x k−1

Click here to load reader

Reader Image
Embed Size (px)
Recommended