Projection

Md. Sahidul Islam RiponDepartment of statistics

Rajshahi UniversityEmail: ripon.ru.statistics@gmail.com

Content

Orthogonal vectorOrthonormal vectorProjectionGram-Schmidt orthogonalization

Orthogonal vector

In mathematics, two vectors are orthogonal if they are perpendicular, i.e., they form a right angle. The relation is clearly symmetric; that is, if x is orthogonal to y then then and so y is orthogonal to x.

Two vectors, x and y, in an inner product space, V, are orthogonal if their inner product is zero.

0, yx

Fig : The line segments AB and CD are orthogonal to each other

Orthogonal vector

x

y

x + y

Pythagoras,

This always not true. This only true when 0yx

222yxyx

0

02

0

)()(

yx

yx

xyyx

xyyxyyxxyyxx

yxyxyyxx

ExampleAre the vector (1,2,2)T and (2,3,-4)T are orthogonal ?

0

2

5

3

;

4

3

2

,

2

2

1

yx

yxyx

38yx

;29y;9x

Are the vector (4,2,3)T and (7,3,-4)T are orthogonal ?

22

12628

1

5

11

;

4

3

7

,

3

2

4

yx

yxyx

222yxyx

147yx

;74y;29x

Theorem 1: An orthogonal set of non zero vectors in a vector space is linearly independent.

Subspace S is orthogonal to subspace T

How???

row space is orthogonal to null space

Orthogonal Subspace

Orthonormal vector Defination: Two vector are said to be orthonormal if it is orthogonal and it has unit length.

Example: Two vector are said to be orthogonal

6

16

26

1

,

3

13

13

1

BA

Orthonormal vector

Theorem: Let { u1, …, un} be an orthonormal basis for a vector space V. Let v be a vector in V. v can be written as a linear combination of these vectors as follows.

nn uuvuuvuuvv ).(...).().( 2211

Proof: Since { u1, …, un} is a basis there exist scalars c1,…,cn such that

v= c1u1+…+cn un

We shall show that, c1 =v1 .u1,…,cn =vn .un

Projection

In linear algebra and functional analysis, a projection is a linear

transformation P from a vector space to itself such that P2 = P.

It leaves its image unchanged. Though abstract, this definition

of "projection" formalizes and generalizes the idea of graphical

projection. One can also consider the effect of a projection on a

geometrical object by examining the effect of the projection on

points in the object.

Projection

Orthogonal Projection

Oblique projection

Orthogonal projection

Let V be any inner-product space and let u V be any vector Let be defined by

The vector u is called the orthogonal projection of v onto u.if the vector u is unit. i,e.

Vvuu

uuvvPu ,

,

,:)(

VVPu :

1u

VvuvvPu ,:)(

x

y

x

y

x x

y

x

Projection Graph

Orthogonal projection

xxx

yx

x

y

)cos(Lx

The projection (or shadow) of a vector x on a vector y is=

Projection of x on y

yL

1

L

yxy

yy

yx

yy

cosθL

L

yx

LL

1

L

yx

x

y

yyy

Why projection?

Beause Ax=b may have no solution. That is when the system of equations are Inconsistent.

Example

Solution: Let be the projection vector. The magnitude of is,

With direction given by the unit vector

Then,

So that can be espressed as,

38.677

56.

X

YXY

Y

)46.0,68.0,57.0(

)4,6,5(78.8

1

X

X

73.077

56

.

.

)94.2,34.4,64.3(

)46.0,68.0,57.0(ˆ

XX

YXand

Y

Find the orthogonal projection of Y=(2,7,1) on to the vector X=(5,6,4)

Y

XY 73.0ˆ

Projection on to a plan

1x

Y

2x

Find the orthogonal projection of Y=(7,7,8) on to the plane spanned by vector X1=(5,6,4) and X2=(9,5,1).

)1,2,5(x1

(7,7,8)Y

)2,5,9(x2

,0.99)(7.59,7.34Y

Solution:

Since must lie in the plane spanned by X1 and X2

2211ˆ XXY

And forming inner products with X and X, we have the equations

and 2211 eXXY

eXXXXXXY

eXXXXXXY

.).().(.

.).().(.

22222112

11221111

339.0

107106106

106170127

1

21

21

and

)0.7,34.0,59.0(

)99.0,34.7,59.7(-(7,7,8)Y-Y e ,

)99.0,34.7,59.7(

0.665) 3.275, (5.895,0.339) 4.068, (1.695,

1) 3, (9, 0.6551) 12, (5, 0.339

ˆ221 1

Where

XXY

Application

1. Gram –schmidt orthogonalization 2. Curve fitting by ordinary least square method. 3. The area of a parallelogram

Gram Schmit Orthogonalization

The Gram-Schmidt orthogonalization process allows us to turn any set of linearly independent vectors into an orthonormal set of the same cardinality. In particular, this holds for a basis of an inner product space. If you feed the machine any basis for the space, the process cranks out an orthonormal basis.

Jorgen Pedersen Gram(1850 - 1916)

Erhard Schmidt(1876 - 1959)

Gram Schmit Orthogonalization

Let be a basis for vector space V. The set of vector defined as follows is orthogonal. To obtain a orthogonal basis for V, normalized each of the vector

},...,,{ 21 nvvv},...,,{ 21 nuuu

uuu

uvv

,

, Proj u

1

1iii

3333

222

11

vvu

vv v u

v v u

v u

21

1

i

ju

uu

u

jproj

projproj

proj

i

ii u

uq

u

uq

u

uq

u

uq

3

33

2

22

1

11

Fig: First two steps of Gram schmidt orthogonalization

Geometric Interpretation

Consider the following set of vectors in R2 (with the conventional inner product)

Now, perform Gram–Schmidt, to obtain an orthogonal set of vectors:

We check that the vectors u1 and u2 are indeed orthogonal:

noting that if the dot product of two vectors is 0 then they are orthogonal.We can then normalize the vectors by dividing out their sizes as shown above:

;

Example

ExampleThe set {(1,2,0,3), (4,0,5,8), (8,1,5,6)} is linearly independent in R4. The vectors form a basis for a three-dimensional subspace V of R4. Construct an orthonormal basis for V. Solution:Let v1=(1, 2, 0, 3), v2=(4, 0, 5, 8), v3=(8, 1, 5, 6). We now use the Gram- Schmidt process to consturect an orthogonal st {u, u, u} from these vectors.

222

231

11

1333333

111

122222

11

u .uu

.uvu

.uu

.uv -vvv v u Let,

u .uu

.uv -vv v u Let,

3) 0, 2, (1, v u Let,

21

1

uu

u

projproj

proj

When this process is implemented on a computer, the vectors uk are often

not quite orthogonal, due to rounding errors. For the Gram–Schmidt process as described above (sometimes referred to as "classical Gram–Schmidt") this loss of orthogonality is particularly bad; therefore, it is said that the (classical) Gram–Schmidt process is numerically unstable.The Gram–Schmidt process can be stabilized by a small modification; this version is sometimes referred to as modified Gram-Schmidt or MGS. This approach gives the same result as the original formula in exact arithmetic and introduces smaller errors in finite-precision arithmetic. Instead of computing the vector uk as it is computed as Each step finds a

vector orthogonal to . Thus is also orthogonalized against any errors introduced in computation of .

Modified Gram-Schmidt orthogonalization

Modified Gram-Schmidt orthogonalization

Compare classical and modified G-S for the vectorsTTT vvv )0,0,,1( ,)0,0,,1( ,)0,0,,1( 321

Making approximation

),0,,0(

)0,,0,1()0,,0,1).(0,,0,1(

)0,,0,1).(,0,0,1()0,0,,1(

)0,0,,1).(0,0,,1(

)0,0,,1).(,0,0,1(),0,0,1(

)0,,,0()0,0,,1()0,0,,1).(0,0,,1(

)0,0,,1).(0,,0,1()0,,0,1(

)0,0,,1(

3

2

1

u

u

u

21)/ 0, (0,1,

2/)0,1,1,0(

)0,0,,1(

3

33

2

22

1

11

u

uq

u

uq

u

uq

21

),0,,0(

)0,,0,1()0,,0,1).(0,,0,1(

)0,,0,1).(,0,0,1()0,0,,1(

)0,0,,1).(0,0,,1(

)0,0,,1).(,0,0,1(),0,0,1(

)0,,,0()0,0,,1()0,0,,1).(0,0,,1(

)0,0,,1).(0,,0,1()0,,0,1(

)0,0,,1(

3

2

1

u

u

u

21)/ 0, (0,1,

2/)0,1,1,0(

)0,0,,1(

3

33

2

22

1

11

u

uq

u

uq

u

uq

Classical Vs Modified

),0,,0(

)0,,0,1()0,,0,1).(0,,0,1(

)0,,0,1).(,0,0,1()0,0,,1(

)0,0,,1).(0,0,,1(

)0,0,,1).(,0,0,1(),0,0,1(

)0,,,0()0,0,,1()0,0,,1).(0,0,,1(

)0,0,,1).(0,,0,1()0,,0,1(

)0,0,,1(

3

2

1

k

k

k

u

u

u

61)/ 0, (0,1,

2/)0,1,1,0(

)0,0,,1(

3

33

2

22

1

11

u

uq

u

uq

u

uq

k

k

k

Classical Vs Modified

Classical Vs Modified

To check the orthogonality

0122)/ 1,- (0,-1,).0,1,1,0(

1/21)/2 0, (0,1,).0,1,1,0(

32

32

k

T

k

T

qq

qq

Curve fitting by ordinary least square method

One of the most widely used methods of curve fitting a straight line to data points is that of ordinary least squares, which makes use of orthogonal projections

eX

eYY

ˆ

).().(.

).().(.

XXXlXY

lXlllY

2

1

2

1

xnx

yxnyx

xy

n

ii

i

n

ii

Refference1. Applied linear algebra in the statistical sciences.

-by Alexander Basilevsky.2. Lecture on linear Algebra. (Gilbert Strange) Institute of MIT.3. Applied Multivariate analysis. -by R. A. Johnson, D. W. Wichern.4. Linear Algebra Theory and Application (2003) -by Ward Cheney and David Kincaid5. Linear Algebra with Application (2007)

- by Granth Williams

Thank You