50
Con
juga
te G
radi
ent
(CG
)
Maj
id L
esan
iA
lirez
aM
asou
m
Ove
rvie
w
Bac
kpro
paga
tion
Gra
dien
t Des
cent
Qua
drat
ic F
orm
sG
radi
ent D
esce
nt in
Qua
drat
ic F
orm
sE
igen
vect
ors
and
valu
esG
radi
ent D
esce
nt C
onve
rgen
ce
Con
juga
te G
radi
ent
Bac
kPro
paga
tion
Abs
tract
ion
Gen
eral
izat
ion
prob
lem
•H
euris
tic fe
atur
es•
Sm
all n
etw
orks
•E
arly
sto
ppin
g•
Reg
ular
izat
ion
Sea
rch
Con
verg
ence
pro
blem
Or S
teep
est D
esce
ntG
radi
ent D
esce
nt
xy
xf ∂∂
),
(yy
xf ∂∂
),
(
Fast
er T
rain
ing
Gra
dien
t Des
cent
mod
ifica
tion
Gra
dien
t Des
cent
BP
with
Mom
entu
mV
aria
ble
Lear
ning
Rat
e B
Pnu
mer
ical
opt
imiz
atio
n te
chni
ques
Con
juga
te G
radi
ent B
PQ
uasi
-New
ton
BP
Gra
dien
t Des
cent
The
prob
lem
is c
hoos
ing
the
step
siz
e
Gra
dien
t Des
cent
Cho
osin
g B
est S
tep
Siz
e
Cho
ose
Whe
re
is m
inim
um
(By
chai
n ru
le)i
α)
(1+ix
f
0)
(1=
∂∂
+ iixfα
0).
()
(1
=∇
=∂+
∂⇒
+i
ii
ii
ir
xf
rx
fαα
01=
⇒+i
T ir
r
Gra
dien
t Des
cent
Cho
osin
g B
est S
tep
Siz
e
Qua
drat
ic fo
rms
Our
dis
cuss
ion
is to
min
imiz
e th
e qu
adra
tic
func
tion:
cx
bAx
xx
fT
T+
−=21
)(
Pos
itive
def
inite
(for
eve
ry v
ecto
r v,
)0
>Av
vT
Qua
drat
ic F
orm
s
A S
ymm
etric
Pos
itive
-Def
inite
Mat
rix h
ave
a gl
obal
min
imum
whe
re g
radi
ent i
s ze
ro
Sol
ving
equ
atio
n A
x =
b eq
uals
to m
inim
ize
f
cx
bAx
xx
fT
T+
−=21
)(
bAx
xf
−=
∇=
)(
0
Gra
dien
t Des
cent
for Q
uadr
atic
For
ms
stee
pest
des
cent
for q
uadr
atic
form
is
Eig
enV
ecto
rs a
nd E
igen
Val
ues
An
eige
nvec
tor o
f a m
atrix
A is
a n
onze
ro v
ecto
r tha
t do
es n
ot ro
tate
whe
n A
is a
pplie
d to
it. O
nly
scal
e by
co
nsta
nt
Eve
ry s
ymm
etric
mat
rix h
ave
n or
thog
onal
eig
enve
ctor
with
it’s
rela
ted
eige
nva
lue
Usi
ng E
igen
Vec
tors
thin
k of
a v
ecto
r as
a su
m o
f oth
er
vect
ors
who
se b
ehav
ior i
s un
ders
tood
Usi
ng E
igen
Vec
tors
Pos
itive
def
inite
mat
rix is
a m
atrix
that
al
l its
eig
enva
lues
are
pos
itive
Eig
enve
ctor
s ar
e ax
is o
f our
rota
ted
ellip
se a
nd e
ach
radi
us re
late
to
corre
spon
ding
eig
enva
lue
Gen
eral
Con
verg
ence
of
Ste
epes
t Des
cent
Rel
atio
n be
twee
n ei
gen
valu
es o
f AE
igen
vect
or c
ompo
nent
s of
erro
r
Fast
Con
verg
ence
Sam
e ei
gen
valu
es h
ave
fast
co
nver
genc
e
Poo
r Con
verg
ence
Diff
eren
t Eig
enve
ctor
s an
d er
ror c
ompo
nent
in
dire
ctio
n of
eig
enve
ctor
s of
sm
alle
r eig
enva
lues
Con
juga
te G
radi
ent O
verv
iew
Orth
ogon
al D
irect
ions
Con
juga
te v
ecto
rsC
onju
gate
Dire
ctio
nsG
ram
-Sch
mid
t alg
orith
mG
radi
ent a
nd e
rror o
ptim
ality
Con
juga
te G
radi
ent
Orth
ogon
al D
irect
ions
Ste
epes
t des
cent
go
in o
ne d
irect
ion
man
y tim
esif
we
have
n o
rthog
onal
sea
rch
dire
ctio
ns
and
choo
se b
est s
tep
ever
y tim
e
Afte
r n s
teps
we
are
at th
e go
al!
Orth
ogon
al D
irect
ions
We
need
eve
ry ti
me
erro
r be
orth
ogon
al to
pre
viou
s di
rect
ion
Con
juga
te v
ecto
rs
Con
juga
te v
ecto
rs
Two
vect
ors
and
are
A
-orth
ogon
al (
or c
onju
gate
) if
Bei
ng C
onju
gate
in s
cale
d sp
ace
mea
ns o
rthog
onal
in u
nsca
led
spac
e
Con
juga
te D
irect
ions
If w
e ha
ve n
con
juga
te s
earc
h di
rect
ions
an
d lik
e or
thog
onal
dire
ctio
ns c
hoos
e be
st s
tep
ever
y tim
e
Afte
r n s
teps
we
are
at th
e go
al!
Con
juga
te D
irect
ions
Orth
ogon
al D
irect
ions
Con
juga
te D
irect
ions
We
need
eve
ry ti
me
erro
r be
A-o
rthog
onal
to p
revi
ous
dire
ctio
n
Con
juga
te D
irect
ions i
ii
i
ii
rb
AxAx
AxAe
xx
e−
=−
=−
=−
=
Gra
m-S
chm
idt a
lgor
ithm
So,
onl
y re
mai
ns to
find
n c
onju
gate
di
rect
ions
Gra
m-S
chm
idt a
lgor
ithm
do
itha
ve n
inde
pend
ent
Giv
es n
con
juga
te d
irect
ions
Gra
m-S
chm
idt a
lgor
ithm
Gra
m-S
chm
idt a
lgor
ithm
Con
juga
te D
irect
ions
So
Alg
orith
m is
com
plet
ebu
t it’s
!
We
had
Gau
ssia
n el
imin
atio
n al
gorit
hm b
efor
e
Con
juga
te D
irect
ions
with
axi
al u
nit v
ecto
rs
Gra
dien
t and
err
or o
ptim
ality
For e
very
We
have
It
mea
ns
Con
juga
te G
radi
ent
Use
fo
r M
akes
equ
atio
ns v
ery
sim
ple
Com
plex
ity fr
om O
(n^2
) per
iter
atio
n re
duce
to O
(m),
m is
num
ber o
f non
zero
ent
ries
of A
Line
Sea
rch
Find
ing
step
size
com
pute
bes
t step-size
)(
min
arg
0i
ii
dx
f⋅
+∈
≥α
αα
End Th
anks
for y
our p
atie
nce!
/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputCondition () /PDFXRegistryName (http://www.color.org) /PDFXTrapped /Unknown
/Description >>> setdistillerparams> setpagedevice
