MATH2070/2970 Optimisation
Introduction
Semester 2, 2012Lecturer: I.W. Guo
Lecture slides courtesy of J.R. Wishart
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Course Information
I Lecture InformationI Optimisation: Weeks 1 – 7
I Contact InformationI Email: [email protected] Office: 807, Extension: 41274
I ConsultationI Wednesday 11am in Room 707A.
I AssessmentI Assignment 10%I Quiz 10%I Project (Fin Maths) 10%I Final Exam 70%
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Course Readings
I Lecture Slides.I Split into relevant topicsI Available at the course website. Website
I Lecture NotesI Notes for both Optimisation & Financial Mathematics.I Available at Kopystop Website
Shop 3 / 55 Mountain St
Broadway NSW 2007
I Google Maps Link Map
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Optimisation Outline
I Introduction and Motivation
I Linear Programming
I Non-linear Optimisation without constraints
I Non-linear Optimisation with constraints
I Dynamic Programming (At the end of the course)
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Review of Introductory Material
Motivation
Optimising Differentiable Functions of One Variable
Optimising Differentiable Functions of Several Variables
System of Equations
Pivot Operations and variable simplification
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Motivation
Optimising Differentiable Functions of One Variable
Optimising Differentiable Functions of Several Variables
System of Equations
Pivot Operations and variable simplification
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
How are optimisation techniques used?
Motivation, In most basic format either,
I Minimise costs/time.
I Maximise output.
Examples:
I Timetabling (Cityrail, Sydney Buses)
I HR: Staff allocations.
I Manufacturing: Blending of raw materials
I Retail: Determine optimal prices.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Terminology
Definition (Parameters)
A set of values (fixed or variable) that are used to describe therelationship between quantities. More
Definition (Objective Function)
A function f(x) of a set of parameters x that one wishes tomaximise/minimise will be termed the ‘objective’ function. More
Definition (Constraints)
A set of conditions that the parameters need to be satisfied duringthe optimisation. More
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Water tank design example
Wish to minimise heat loss through surface area from an openrectangular water storage tank that has a fixed volume, V .
Dimensions: x, y and z.
Formulate Problem
Minimise,S = 2xy + 2yz + xz
such that,
V = xyz ≡ constant and x, y, z > 0.
Terminology
Parameters : x, y and z.Constraints : V = xyz and x, y, z > 0.Objective function : S(x, y, z) = 2xy + 2xz + xz
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Evaluation
Simplification obtained by eliminating z variable using theconstraint.
Note, V = xyz ⇒ z = Vxy which gives,
Simplified Problem
Minimise,
S(x, y) = 2xy +2V
x+
V
y
such that,x, y > 0 and V ≡ constant.
How do we optimise such a function?
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Motivation
Optimising Differentiable Functions of One Variable
Optimising Differentiable Functions of Several Variables
System of Equations
Pivot Operations and variable simplification
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Review of Univariate optimisation
Single Variable Calculus
Univariate = Single variable x ∈ [a, b] ⊂ R.
Objective function f : [a, b]−→R.
Location of Extrema
At which values of x do the extrema (maxima or minima) occur forthe function f(x)?
Two types of Extrema
Local Extrema
Global Extrema
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Formal definition of the two types of Extrema
Let x ∈ [a, b] and f : [a, b]−→R.
Definition (Local maximum/minimum)
A value x0 is said to be a local maximum (minimum) iff(x0) ≥ f(x) (f(x0) ≤ f(x)) for all x in a neighbourhood of x0
xx∗x0
f(x)
Figure 1: Plot of a function that has one maximum and two minima.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Formal definition of the two types of Extrema
Let x ∈ [a, b] and f : [a, b]−→R.
Definition (Global maximum/minimum)
A value x∗ is said to be a global maximum (minimum) iff(x∗) ≥ f(x) (f(x∗) ≤ f(x)) for all x ∈ [a, b]
xx∗x0
f(x)
Figure 2: Plot of a function that has global minimum at x∗.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
x−1 1
f(x)
Figure 3: Plot of a function that has global extrema at the endpoints.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
x−1 1x∗x∗
f(x)
Figure 4: Plot of a function that has global extrema inside the domain.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Linear vs Non-linear Extrema
Linear functions
Extremum always attained at constraint boundaries
A local extremum is also a global extremum
Non-linear functions
Extrema may be in the interior as well as at boundaries.
A local extremum is not necessarily a global extremum.
Linear Nonlinear
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Finding the local extrema
If f is differentiable on (a, b). That is, f ′(x) exists and is welldefined. Previous calculus methods can be used.
Find stationary points by checking the first derivative
Definition
A point x0 ∈ (a, b) is said to be a stationary point of f if
f ′(x0) = 0
�� ��A stationary point is also referred to as a critical point.
The behaviour of the stationary point can be determined by the..
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Finding the local extrema (cont.)
Basic second derivative test
If f ′(x0) = 0 for some x0 ∈ (a, b) and
I If f ′′(x0) < 0, then x0 is the location of a local maximum.
I If f ′′(x0) > 0, then x0 is the location of a local minimum.
I If f ′′(x0) = 0, then test fails, x0 is possibly a point ofinflection.
f(x)
t
f ′′(t) > 0
g(x)
s
g′′(s) < 0
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Finding the local extrema (cont.)
Generalised higher derivative test
Let m be a positive integer and assume that there exists anx0 ∈ (a, b) such that
f (1)(x0) = f (2)(x0) = . . . = f (2m−1)(x0) = 0.
Then the following holds
I If f (2m)(x0) < 0, then x0 is the location of a local maximum.
I If f (2m)(x0) > 0, then x0 is the location of a local minimum.
I If f (2m)(x0) = 0, then test fails, x0 is possibly a point ofinflection?
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Motivation
Optimising Differentiable Functions of One Variable
Optimising Differentiable Functions of Several Variables
System of Equations
Pivot Operations and variable simplification
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Review of Multivariate optimisation
Multivariate Calculus
Multivariate = Many variables
x = (x1, x2, . . . , xn) ∈ D ⊂ Rn.
f = f(x1, x2, . . . , xn) = Objective function of those n variables
Necessary condition for stationary points
∂f∂x1
= ∂f∂x2
= · · · = ∂f∂xn
= 0 .
Sufficient conditions for maxima/minima of multivariate functionsconsidered later in the course.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Non-linear functions
Extrema may be in the interior as well as at boundaries.
A local extremum is not necessarily a global extremum.
Definition (Local maximum/minimum)
A point x0 = (x1, x2, . . . , xd) ∈ D is said to be a local maximum(minimum) if f(x0) ≥ f(x) (f(x0) ≤ f(x)) for all x in aneighbourhood of x0
Definition (Global maximum/minimum)
A value x∗ ∈ D is said to be a global maximum (minimum) iff(x∗) ≥ f(x) (f(x∗) ≤ f(x)) for all x ∈ D
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Multivariate example
02
46 0
2
4
6−1
0
1
Figure 5: 3-dimensional function with stationary points.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Other type of multivariate Optimisation
Linear problems
Linear programming problem
Maximise/Minimise
f(x1, x2, . . . , xn) = c1x1 + c2x2 + · · ·+ cnxn,
such that,a1x1 + a2x2 + . . .+ anxn ≤ C,
where ci, ai and C are constants,
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Linear Programming Solution?
In a similar vein to the univariate case,
Optimal solution for Linear problems
Extremum are always attained at constraint corner points
To see this note that if partial derivatives are set to zero:
∂f
∂x1= 0⇒ c1 = 0
......
...
∂f
∂xn= 0⇒ cn = 0.
Thus need to consider the boundaries for optimal solution
Solutions can be difficult in high dimensional problems.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Example : Manufacturing problem
A company manufactures two types of drugs by using threedifferent resources. The resources have limited supply. Thecompany wishes to maximise its profit.
I Each unit of drug earns the following profit:I Drug 1 earns a profit of $3,000.I Drug 2 earns a profit of $5,000.
I Each unit of drug uses the following resources:I Drug 1 uses 1 gm of Resource 1 and 3 gm of Resource 2.I Drug 2 uses 2 gm of Resource 2 and 2 gm of Resource 3.
I Each resource has the following supply limit.I Only 4 gm of Resource 1 are available.I Only 18 gm of Resource 2 are available.I Only 12 gm of Resource 3 are available.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Model formulation.
Let x1, x2 represent the number of units of Drug 1 and Drug 2produced.
Write problem mathematically with:
Maximize: Z = 3x1 + 5x2
subject to: x1 ≤ 43x1 + 2x2 ≤ 18
2x2 ≤ 12
with: x1 ≥ 0 , x2 ≥ 0 .
Problem formulation
Very important to be able to formulate problem mathematically.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Motivation
Optimising Differentiable Functions of One Variable
Optimising Differentiable Functions of Several Variables
System of Equations
Pivot Operations and variable simplification
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Elementary Row Operations
Consider the following system of linear equations:
x1 −x2 x3 = −22x1 +x2 −x3 = −5−x1 +2x2 3x3 = 0
(1)
Written in augmented matrix notation, 1 −1 1 −22 1 −1 −5−1 2 3 0
Solution vector x = (x∗1, x
∗2, x∗3) exists that satisfies (1).
Method uses Elementary Row Operations.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Elementary Row Operations
The three elementary row operations used are
1. Interchange two rows (denoted Ri ↔ Rj);
2. Multiply (or divide) any row by a non-zero constant (denotedRi → aRi);
3. Add a (non-zero) multiple of one row to any other row(denoted Ri → Ri + aRj).
Interchange of rows
Interchange of rows is not used in linear programming methods.
Solution is unchanged
Elementary row operations do not change the solution (x∗1, x∗2, x
∗3)
Return to example.
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Return to the previous matrix:
T1 =
x1 x2 x3 RHS1 −1 1 −2 ← R1
2 1 −1 −5 ← R2
−1 2 3 0 ← R3
then the row-operations: R3 → R3 +R1 and R2 → R2 − 2R1
transform T1 to the equivalent tableau:
T2 =
x1 x2 x3 RHS
1 −1 1 −20 3 −3 −10 1 4 −2
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Motivation
Optimising Differentiable Functions of One Variable
Optimising Differentiable Functions of Several Variables
System of Equations
Pivot Operations and variable simplification
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Pivot operations
Let aij be any non-zero element (called the pivot element) of thecoefficient matrix A of a given system of linear equations: i.e. aijis the element in row i and column j of the corresponding tableau.Then to pivot on aij 6= 0, denoted Pij ,
1. Divide row i by aij ;
2. Transform to zero all elements akj , k 6= i (i.e. the elements inthe same column j as aij except row i) by adding suitablemultiples of row i.
Result, a11 a12 · · · a1na21 a22 · · · a2n
...... aij
...an1 an2 · · · ann
Pij−→
b11 b12 0 b1nb21 b22 0 b2n
...... 1
...bn1 bn2 0 bnn
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Pivot Example
Recall example tableau
x1 x2 x3 RHS1 −1 1 −22 1 −1 −5−1 2 3 0
.
Pivot the tableau T1 on the element a21 = 2 (row 2, column 1).
T2 = P21T1 =
x1 x2 x3 RHS0 −3/2 3/2 −9/21 1/2 −1/2 −5/20 5/2 5/2 −5/2
T2 is also equivalent to T1 since the pivot operation does notchange the solution set (x1, x2, x3) (Only uses ERO’s).
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Example: Pivot solution
Pivot on the example matrix to obtain identity.
x1 x2 x3 RHS1 −1 1 −22 1 −1 −5−1 2 3 0
1 −1 1 −2P11 : 0 3 −3 −1
0 1 4 −21 0 0 −7/3
P22 : 0 1 −1 −1/30 0 5 −5/31 0 0 −7/6
P33 : 0 1 0 −2/30 0 1 −1/3
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
More variables than equations
Consider now the following system of three equations in fivevariables,
x1 −x2 +x3 −x4 = −22x1 +x2 −x3 +x5 = 5−x1 +2x2 +3x3 +x4 +2x5 = 0
I Express (x1, x2, x3) in terms of (x4, x5).
I The variables we solve for (x1, x2, x3), are called the basicvariables.
I They must be linearly independent.
I The remaining variables (x4, x5) are called non-basic variables:
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Method of solving new system
Basic–Variables Non–Basic
x1 x2 x3 x4 x5 RHS
1 −1 1 −1 0 −22 1 −1 0 1 5−1 2 3 1 2 0
1 −1 1 −1 0 −20 3 −3 2 1 90 1 4 0 2 −21 0 0 −1/3 1/3 10 1 −1 2/3 1/3 30 0 5 −2/3 5/3 −51 0 0 −1/3 1/3 10 1 0 8/15 2/3 20 0 1 −2/15 1/3 −1
Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination
Solution for new equations
Basic–Variables Non–Basic
x1 x2 x3 x4 x5 RHS
1 0 0 −1/3 1/3 10 1 0 8/15 2/3 20 0 1 −2/15 1/3 −1
Solution given by:
x1 = 1 + 13x4 − 1
3x5
x2 = 2− 815x4 − 2
3x5
x3 = −1 + 215x4 − 1
3x5