Slide 1

Local search algorithmsMost local search algorithms are based on derivatives to guide the search.For differentiable function it has been shown that even if you need to calculate derivatives by finite differences, derivative-based algorithms are better than ones that do not use derivatives.However, we will start with Nelder-Mead sequential simplex algorithm that can deal with non-differentiable functions, and even with some discontinuities.

Later in the semester we will describe algorithms for unconstrained optimization that are based on derivatives. For continuously differentiable functions these algorithms are very effective for finding local minima.

In this lecture we will introduce instead an algorithm that does not need differentiability or even continuity, which is the Nelder-Mead sequential simplex algorithm.1Nelder Mead Sequential Simplex AlgorithmAn old local-search algorithm that contains the ingredients of modern search techniques:No derivativesPopulation basedSimple idea that does not require much mathematicsBasis for Matlabs fminsearch testimonial to its robustness.Simplex is the simplest body in n dimensionsHas n+1 vertices (triangle in 2D and tetrahedron in 3D)

The algorithm was actually invented by Spendley, Hext and Himsworth: (Spendley, W., Hext, G. R., and Himsworth, F. R., Sequential Application ofSimplex Designs in Optimisation and Evolutionary Operation, Technometrics,4 (4), pp. 441461, 1962.)

I was then refined by British Mathermaticians Nelder and Mead: (Nelder, J. A. and Mead, R., A Simplex Method for Function Minimization, Computer J., 7, pp. 308313, 1965.)

Matlab refers instead to a paper on its convergence properties in one or two dimensionsLagarias, J.C., J. A. Reeds, M. H. Wright, and P. E. Wright, "Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions,"SIAM Journal of Optimization, Vol. 9 Number 1, pp. 112-147, 1998.

So it is an old algorithm, but it embodies three properties that are characteristic of modern search techniques that may be responsible fo its longevity: (i) it does not use derivatives; (ii) it is population based; (iii) it uses a simple idea and does not require much mathematics.

The basic unit is a simplex, which is the simplest body in n-dimensional space. It is bounded by n+1 vertices, where we evaluate the objective functions. In two dimensions a simplex is a triangle, and in three dimensions it is a tetrahedron.2Sequential Simplex Method (Mostly from Wikipedia and Matlab)In n dimensional space start with n+1 vertices of a selected simplex, evaluate the function there and order points by function value

Calculate x0, the center of gravity of all the points except xn+1Reflect xn+1 about x0

If new point is better than 2nd worst, but not best, use to replace worst and go back to step 1.

Reflect worst point about c.g.

Read about expansion and contraction

The basic idea of the Nelder-Mead algorithm is movement away from the worst point in the directions of the other points. Specifically, one moves in the direction of the center of gravity of the other points. The move is called reflection, because in its simplest form one goes to the point that is the mirror image of the worst point on the other side (=1).

There are three possibilities for the value of the function at the reflection point xr. Most of the time it will be an improvement on the worst point, but not on the best point. In that case, we will replace xn+1 by xr..

The second possibility is that the new point is better than all the points in the current simplex. This means that the reflection direction is very promising and we should go further along in that direction, something called expansion.

The third possibility is that the new point is worst than all the existing points. This means that we have possibly gone too far, and we should retract in a process called contraction.

Expansion and contraction are discussed in the next slide.3Expansion and contractionExpansion (new point is best)

Contraction (new point is worst than 2nd worst)

If new point is the best, then we try to move further. In Matlab, we double the reflection distance from the center of gravity to find the expansion point xe If the new point is better than the reflection point we replace the worst point by the expansion point. If not, we replace it by the reflection point.

Conversely, we may find that the reflection point is not good. If it is worse than the second worst point, we assume that reflection was too drastic and we try a smaller move in the same direction. In Matlab we halve the move distance from the center of gravity to get xc. If is better than the worst point, then we use it to replace the worst point.

Finally, if even the contraction point is not better than the worst point, we conclude that the simplex is too big and shrink it, as described on the next slide.4Reduction (shrinkage)If the contracted point is not better than worst, then contract all points about the best one

The reduction (Wikipedia term) or shrinkage (Matlab) is very simple. We move each point closer to the best point. In Matlab we move it half the distance.5Problems Nelder Mead operatorsFor a 2D problem, the current simplex has f(0,1)=3, f(0,0)=1, f(1,0)=2. Where will you evaluate f next? SolutionIf the next two points gave us function values of 4 and 5, respectively, where will you evaluate the function next? SolutionIf instead, the next point gave us a function value of 0, where will you evaluate the function next? Solution6Rosenbrock Banana functionVarious versionsThis is from Wikeipedia (minimum at (1,1))

The Rosenbrock Banana function was invented by Howard Rosenbrock (British statistician, 1920-2010). It is popular for testing optimization algorithms, and here we will use a version of the function from Wikipedia. 7Generating first 20 points for Banana functionfunction [y]=banana(x)global z1; global z2; global ygglobal county=100*(x(2)-x(1)^2)^2+(1-x(1))^2;z1(count)=x(1); z2(count)=x(2);yg(count)=y;count=count+1;global z2; global yg; global z1global countcount =1;options=optimset('MaxFunEvals',20)[x,fval] = fminsearch(@banana,[-1.2, 1],options) mat=[z1;z2;yg]mat =

Columns 1 through 8

-1.200 -1.260 -1.200 -1.140 -1.080 -1.080 -1.020 -0.960 1.000 1.000 1.050 1.050 1.075 1.125 1.1875 1.150 24.20 39.64 20.05 10.81 5.16 4.498 6.244 9.058

Columns 9 through 16

-1.020 -1.020 -1.065 -1.125 -1.046 -1.031 -1.007 -1.013 1.125 1.175 1.100 1.100 1.119 1.094 1.078 1.113 4.796 5.892 4.381 7.259 4.245 4.218 4.441 4.813

The banana function shown on the left, does not only codes the Banana function, but it also stores the coordinates and the function values every time it is called. That allows us to use these on the next slide in order to visualize the progress of the algorithm. The matlab segment on the right shows how we use fminsearch with a limit on the number of function evaluation set at 20, and the bottom frame shows the first 16 points.

The first three are the initial simplex. The first point is specified in the calling sequence, and fminsearch generates the other two points by 5% perturbation to the coordinate of the first point.8Reflection and expansion.

-1.26-1.24-1.22-1.2-1.18-1.16-1.14-1.12-1.1-1.08-1.0611.011.021.031.041.051.061.071.081.0939.624.2

20.0510.815.16

Here the first reflection is very successful, in that the value of the function, at 10.81, is better than any of the previous points, so we double the distance and get an even better result of 5.169Next iteration

24.220.055.16

The second iteration again starts with reflection of the worst point (24.2) about the middle of the line connecting the other two points. This gives us an improvement to 4.49, so we try expansion again. However, this time we do not get an improvement, so the simplex for the next iteration will include the 20.05, 5.16, and 4.49 points.10Complete search examples from Wikipediahttp://en.wikipedia.org/wiki/Nelder-Mead_methodShows progress for the Banana function as well as for Himmelblaus function.11Problem fminsearchTrack and plot the first few iterations of fminsearch on Himmelblaus function starting from (1,1).Solution