Root FindingRoot Finding

COS 323COS 323

1-D Root Finding1-D Root Finding

• Given some function, find location Given some function, find location where where ff((xx)=0)=0

• Need:Need:– Starting position Starting position xx00, hopefully close to , hopefully close to

solutionsolution

– Ideally, points that Ideally, points that bracketbracket the root the rootff((xx++) > 0) > 0

ff((xx––) < 0) < 0

1-D Root Finding1-D Root Finding

• Given some function, find location Given some function, find location where where ff((xx)=0)=0

• Need:Need:– Starting position Starting position xx00, hopefully close to , hopefully close to

solutionsolution

– Ideally, points that Ideally, points that bracketbracket the root the root

– Well-behaved functionWell-behaved function

What Goes Wrong?What Goes Wrong?

Tangent point:Tangent point:very difficultvery difficult

to findto find

Singularity:Singularity:brackets don’tbrackets don’tsurround rootsurround root

Pathological case:Pathological case:infinite number ofinfinite number of

roots – e.g. sin(1/x)roots – e.g. sin(1/x)

Bisection MethodBisection Method

• Given points Given points xx++ and and xx–– that bracket a root, that bracket a root,

findfindxxhalfhalf = ½ ( = ½ (xx+++ + xx––))

and evaluate and evaluate ff((xxhalfhalf))

• If positive, If positive, xx+ + xxhalfhalf else else xx– – xxhalfhalf

• Stop when Stop when xx+ + and and xx–– close enough close enough

• If function is continuous, this If function is continuous, this willwill succeed succeedin finding in finding somesome root root

BisectionBisection

• Very robust methodVery robust method

• Convergence rate:Convergence rate:– Error bounded by size of [Error bounded by size of [xx++… … xx––] interval] interval

– Interval shrinks in half at each iterationInterval shrinks in half at each iteration

– Therefore, error cut in half at each iteration:Therefore, error cut in half at each iteration:||nn+1+1|| = = ½½ ||nn||

– This is called “linear convergence”This is called “linear convergence”

– One extra bit of accuracy in One extra bit of accuracy in xx at each at each iterationiteration

Faster Root-FindingFaster Root-Finding

• Fancier methods get super-linear Fancier methods get super-linear convergenceconvergence– Typical approach: model function locally by Typical approach: model function locally by

something whose root you can find exactlysomething whose root you can find exactly

– Model didn’t match function exactly, so Model didn’t match function exactly, so iterateiterate

– In many cases, these are less safe than In many cases, these are less safe than bisectionbisection

Secant MethodSecant Method

• Simple extension to bisection: Simple extension to bisection: interpolate or extrapolate through two interpolate or extrapolate through two most recent pointsmost recent points

11

2233

44

Secant MethodSecant Method

• Faster than bisection:Faster than bisection:||nn+1+1|| = = const.const. ||nn||1.61.6

• Faster than linear: number of correct bits Faster than linear: number of correct bits multiplied by 1.6multiplied by 1.6

• Drawback: the above only true if Drawback: the above only true if sufficiently closesufficiently close to a root of a to a root of a sufficiently sufficiently smoothsmooth function function– Does not guarantee that root remains Does not guarantee that root remains

bracketedbracketed

False Position MethodFalse Position Method

• Similar to secant, but guarantee Similar to secant, but guarantee bracketingbracketing

• Stable, but linear in bad casesStable, but linear in bad cases

11

2233

44

Other Interpolation StrategiesOther Interpolation Strategies

• Ridders’s method: fit exponential toRidders’s method: fit exponential toff((xx++), ), ff((xx––), and ), and ff((xxhalfhalf))

• Van Wijngaarden-Dekker-Brent method:Van Wijngaarden-Dekker-Brent method:inverse quadratic fit to 3 most recent inverse quadratic fit to 3 most recent pointspointsif within bracket, else bisectionif within bracket, else bisection

• Both of these Both of these safesafe if function is nasty, but if function is nasty, butfastfast (super-linear) if function is nice (super-linear) if function is nice

Newton-RaphsonNewton-Raphson

• Best-known algorithm for getting Best-known algorithm for getting quadraticquadraticconvergence when derivative is easy to convergence when derivative is easy to evaluateevaluate

• Another variant on the extrapolation Another variant on the extrapolation themetheme

)(

)(1

n

nnn xf

xfxx

)(

)(1

n

nnn xf

xfxx

1122

33

44

Slope = derivative at 1Slope = derivative at 1

Newton-RaphsonNewton-Raphson

• Begin with Taylor seriesBegin with Taylor series

• Divide by derivative (can’t be zero!)Divide by derivative (can’t be zero!)

0...2

)()()()( 2

wantn

nnn

xfxfxfxf

0...

2

)()()()( 2

wantn

nnn

xfxfxfxf

21

2

2

2

~)(2

)(

0)(2

)(

0)(2

)(

)(

)(

nnn

nNewton

n

nNewton

n

n

n

n

xf

xf

xf

xf

xf

xf

xf

xf

21

2

2

2

~)(2

)(

0)(2

)(

0)(2

)(

)(

)(

nnn

nNewton

n

nNewton

n

n

n

n

xf

xf

xf

xf

xf

xf

xf

xf

Newton-RaphsonNewton-Raphson

• Method fragile: can easily get confusedMethod fragile: can easily get confused

• Good starting point criticalGood starting point critical– Newton popular for “polishing off” a root Newton popular for “polishing off” a root

found approximately using a more robust found approximately using a more robust methodmethod

Newton-Raphson ConvergenceNewton-Raphson Convergence

• Can talk about “basin of convergence”:Can talk about “basin of convergence”:range of range of xx00 for which method finds a for which method finds a

rootroot

• Can be extremely complex:Can be extremely complex:here’s an examplehere’s an examplein 2-D with 4 rootsin 2-D with 4 roots

Popular Example of Newton: Popular Example of Newton: Square RootSquare Root

• Let Let ff((xx) = ) = xx22 – – aa: zero of this is square : zero of this is square root of root of aa

• ff’(’(xx) = 2) = 2xx, so Newton iteration is, so Newton iteration is

• ““Divide and average” methodDivide and average” method

nxa

nn

nnn x

x

axxx

2

1

2

1 2

nxa

nn

nnn x

x

axxx

2

1

2

1 2

Reciprocal via NewtonReciprocal via Newton

• Division is slowest of basic operationsDivision is slowest of basic operations

• On some computers, hardware divide On some computers, hardware divide not available (!): simulate in softwarenot available (!): simulate in software

• Need only subtract and multiplyNeed only subtract and multiply

nn

x

xnn

x

x

bba

bxxb

xx

xf

bxf

a

2

)(

0)(

*

2

2

1

1

1

1

1

1

nn

x

xnn

x

x

bba

bxxb

xx

xf

bxf

a

2

)(

0)(

*

2

2

1

1

1

1

1

1

Rootfinding in >1DRootfinding in >1D

• Behavior can be complex: e.g. in 2DBehavior can be complex: e.g. in 2D

0),(want

yxg 0),(want

yxg

0),(want

yxf 0),(want

yxf

Rootfinding in >1DRootfinding in >1D

• Can’t bracket and bisectCan’t bracket and bisect

• Result: few general methodsResult: few general methods

Newton in Higher DimensionsNewton in Higher Dimensions

• Start withStart with

• Write as vector-valued functionWrite as vector-valued function

0),(

0),(want

want

yxg

yxf

0),(

0),(want

want

yxg

yxf

),(

),()(

yxg

yxfnxf

),(

),()(

yxg

yxfnxf

Newton in Higher DimensionsNewton in Higher Dimensions

• Expand in terms of Taylor seriesExpand in terms of Taylor series

• ff’’ is a Jacobian is a Jacobian

0...)()()(want

nnn δxfxfδxf 0...)()()(want

nnn δxfxfδxf

yxn

ffJxf )( yxn

ffJxf )(

Newton in Higher DimensionsNewton in Higher Dimensions

• Solve for Solve for

• Requires matrix inversion (we’ll see this Requires matrix inversion (we’ll see this later)later)

• Often fragile, must be carefulOften fragile, must be careful– Keep track of whether error decreasesKeep track of whether error decreases

– If not, try a smaller step in direction If not, try a smaller step in direction

)()(1nn xfxJδ )()(1nn xfxJδ