3.VI.1. Orthogonal Projection Into a Line3.VI.2. Gram-Schmidt Orthogonalization3.VI.3. Projection Into a Subspace

3.VI. Projection

3.VI.1. & 2. deal only with inner product spaces.

3.VI.3 is applicable to any direct sum space.

3.VI.1. Orthogonal Projection Into a Line

Definition 1.1: Orthogonal ProjectionThe orthogonal projection of v into the line spanned by a nonzero s is the vector.

ˆ ˆproj s v v s s

v ss

s s

ˆ

s ss

s s s

Example 1.3: Orthogonal projection of the vector ( 2 3 )T into the line y = 2x.

2 11 1

2 63 25 5

proj

s

2

x

x

s 2 24 5x x x s11

ˆ25

s

18

25

Example 1.4: Orthogonal projection of a general vector in 3 into the y-axis

2

0

1

0

e2

0 0

1 1

0 0

x x

proj y y

z z

e

0

1

0

y

0

0

y

Example 1.5: Project = Discard orthogonal componentsA railroad car left on an east-west track without its brake is pushed by a wind blowing toward the northeast at fifteen miles per hour; what speed will the car reach?

1proj ev w

115

02

115

12

w

15

2speed v

Example 1.6: Nearest PointA submarine is tracking a ship moving along the line y = 3x + 2.Torpedo range is one-half mile. Can the sub stay where it is, at the origin on the chart, or must it move to reach a place where the ship will pass within range?

Ship’s path is parallel to vector11

ˆ310

s

Point p of closest approach is

0 0

2 2proj

sp

0 16 1

2 310 10

0 13

2 35

31

15

110

5p 0.632 (Out of range)

Exercises 3.VI.1.

1. Consider the function mapping a plane to itself that takes a vector to itsprojection into the line y = x. (a) Produce a matrix that describes the function’s action.(b) Show also that this map can be obtained by first rotating everything in the plane π/4 radians clockwise, then projecting into the x-axis, and then rotating π/4 radians counterclockwise.

3.VI.2. Gram-Schmidt Orthogonalization

Given a vector s, any vector v in an inner product space can be decomposed as

proj proj s sv v v v // v v // 0 v vwhere

Definition 2.1: Mutually Orthogonal Vectors

Vectors v1, …, vk n are mutually orthogonal if

vi · vj = 0 i j

Theorem 2.2:A set of mutually orthogonal non-zero vectors is linearly independent.

Proof:i i

i

c v 0 0j i i j j ji

c c v v v v→

→ cj = 0 j

Corollary 2.3:A set of k mutually orthogonal nonzero vectors in V k is a basis for the space.

Definition 2.5: Orthogonal Basis An orthogonal basis for a vector space is a basis of mutually orthogonal vectors.

Example 2.6: Turn1 0 1

1 , 2 , 0

1 0 3

into an orthogonal basis for 3.

1

1

1

1

κ

12

0 0

2 2

0 0

proj

κκ

0 12

2 13

0 1

1 23

1 1 1

0 0 0

3 3 3

proj proj

κ κκ

1

0 1 0 11

2 1 2 13

0 1 0 1

proj

κ1 1 3 κ κ

12

13

1

12

23

1

2 2

24 8

9 3 κ κ

1

1 1 1 11

0 1 0 13

3 1 3 1

proj

κ

14

13

1

2

1 1 1 13 2 2

0 2 0 28 3 3

3 1 3 1

proj

κ

12

23

1

1 1 14 2

0 1 23 3

3 1 1

1

0

1

1 1 1

21 , 2 , 0

31 1 1

K

Theorem 2.7: Gram-Schmidt OrthogonalizationIf β1 , …, βk is a basis for a subspace of n then the κj s calculated from the following scheme is an orthogonal basis for the same subspace.

1 1κ β

12 2 2proj κκ β β 2 12 1

1 1

β κ

β κκ κ

1 23 3 3 3proj proj κ κκ β β β 3 1 3 23 1 2

1 1 2 2

β κ β κ

β κ κκ κ κ κ

1

1j

k

k k kj

proj

κκ β β1

1

kk j

k jj j j

β κ

β κκ κ

1 12

1 1

κ κI β

κ κ

1 1 2 22

1 1 2 2

κ κ κ κI β

κ κ κ κ

1

1

kj j

kj j j

κ κI β

κ κ

Proof: For m 2 , Let β1 , …, βm be mutually orthogonal, and

11 1

1

mm j

m m jj j j

β κ

κ β κκ κ

11 1

1

mm j

i m i m i jj j j

β κ

κ κ κ β κ κκ κ

Then 1 1i m m i κ β β κ 0 1, ,i m QED

If each κj is furthermore normalized, the basis is called orthonormal.

The Gram-Schmidt scheme simplifies to:

1 1κ β

12 2 2proj κκ β β 2 2 1 1 β β e e

1 23 3 3 3proj proj κ κκ β β β 3 3 1 1 3 2 2 β β e e β e e

1

1j

k

k k kj

proj

κκ β β 1

1

k

k k j jj

β β e e

1 1 2 I e e β

1 1 2 2 2 I e e e e β

1

1

k

j j kj

I e e β

11

1

κ

eκ

22

2

κ

eκ

33

3

κ

eκ

kk

k

κ

eκ

Exercises 3.VI.2.

1. Perform the Gram-Schmidt process on this basis for 3,2 1 0

2 , 0 , 3

2 1 1

2. Show that the columns of an nn matrix form an orthonormal set if and only if the inverse of the matrix is its transpose. Produce such a matrix.

3.VI.3. Projection Into a Subspace

Definition 3.1:

For any direct sum V = M N and any v V such that

v = m + n with m M and n N

The projection of v into M along N is defined as

projM, N (v) = m

Reminder:

• M & N need not be orthogonal.

• There need not even be an inner product defined.

Example 3.2: The space 22 of 22 matrices is the direct sum of

,0 0

a bM a b

R0 0

,N c dc d

R

Task: Find projM , N (A), where 3 1

0 4

A

Solution:

Let the bases for M & N be1 0 0 1

,0 0 0 0M

B

→

0 0 0 0,

1 0 0 1N

B

M MB B Bº1 0 0 1 0 0 0 0

, , ,0 0 0 0 1 0 0 1

is a basis for 22 .

1 0 0 1 0 0 0 03 1 0 4

0 0 0 0 1 0 0 1

A∴

,

1 0 0 13 1

0 0 0 0M Nproj

A3 1

0 0

Example 3.3: Both subscripts on projM , N (v) are significant.

Consider 2 0

x

M y y z

z

with basis1 0

0 , 2

0 1M

B

0

0

1

N k k

R &and subspaces0

1

2

L k k

R

It’s straightforward to verify 3 M N M L R

where

2

2

5

vTask: Find projM , N (v) and projM , L (v)

Solution:

For1 0 0

0 , 2 , 0

0 1 1M N M N

B B Bº

2

1

4M N

v

B

→ ,

2

1

0M N

M Nproj

v

B

2

2

1

For1 0 0

0 , 2 , 1

0 1 2M L M L

B B Bº

2

2

5

v→

,

2

9 / 5

0M N

M Lproj

v

B

2

18 / 5

9 / 5

2 2

2 5

a b

a b

2

9 / 5

8 / 5M L

B

Note: ML is orthogonal but MN is not.

Definition 3.4: Orthogonal Complement

The orthogonal complement of a subspace M of n is

M = { v n | v is perpendicular to all vectors in M } ( read “M perp” ).

The orthogonal projection projM (v ) of a vector is its projection into M along M

.

Example 3.5:

In 3, find the orthogonal complement of the plane 3 2 0

x

P y x y z

z

Solution: Natural basis for P is 1 0

0 , 1

3 2

B ( parameter = z)

→1 1

2 2

3 3

1 0 3 0

0 1 2 0

v v

v v

v v

3

2

1

k k

R 1 20, 0P v v β v β

Lemma 3.7:

Let M be a subspace of n. Then M is also a subspace and n = M M .

Hence, v n, v projM (v) is perpendicular to all vectors in M.

Proof: Construct bases using G-S orthogonalization.

Theorem 3.8:

Let v be a vector in n and let M be a subspace of n with basis β1 , …, βk .

If A is the matrix whose columns are the β’s then

projM (v ) = c1β1 + …+ ck βk

where the coefficients ci are the entries of the vector (AT A) AT v. That is,

projM (v ) = A (AT A)1 AT v.

Proof: Mproj Mv

T TA A c A v

where c is a column vector

T 0 A v Ac

→ Mproj v A c

By lemma 3.7,

1T Tc A A A v→ →

Example 3.9:

To orthogonally project1

1

1

v 0

x

P y x z

z

1 0

0 1

1 0

A

into subspace

From1 0

0 1 ,

1 0

P x y x y

R we get

→

11 0

1/ 2 0 1 0 10 1

0 1 0 1 01 0

T T

A A A A

1 01 0 1

0 10 1 0

1 0

T

A A2 0

0 1

1 1/ 2 0

0 1T

A A

1 01/ 2 0 1/ 2

0 10 1 0

1 0

1/ 2 0 1/ 2

0 1 0

1/ 2 0 1/ 2

1/ 2 0 1/ 2 1

0 1 0 1

1/ 2 0 1/ 2 1Pproj

v

0

1

0

Exercises 3.VI.3.

1. Project v into M along N, where

3

0

1

v 0

x

M y x y

z

1

0N c c

R

2. Find M for 3 0

x

M y x y z

z

3. Define a projection to be a linear transformation t : V → V with the property

that repeating the projection does nothing more than does the projection alone:

( t t )(v) = t (v) for all v V.

(a) Show that for any such t there is a basis = β1 , …, βn for V such that

1, 2, ,

for1, 2, ,

ii

i rt

i r r n

ββ

0

where r is the rank of t.

(b) Conclude that every projection has a block partial-identity representation:

Rep t

I 0

0 0B B