Introduction to Linear Programming

Linear programming is a widely used technique in optimisation.Here is a link to a short history. What I find interesting is that,while the technique only became widely used after the SecondWorld War, the ideas go back much further. It seems thatoptimisation, as an end in itself, only became popular in recenttimes. It’s likely that the only reason that linear programming wasnot fully developed earlier was lack of interest; not lack ofmathematical sophistication.

This brings us to the first key point. Linear programming is arelatively recent technique, but it is not particularly difficult.

The best way to introduce linear programming is via examples.

The Diet ProblemConsider a community which produces two types of grains — G1and G2 — and whose citizens require three types of nutrients —starch, proteins and vitamins.

Starch Proteins Vitamins Price $/kilo

G1 5 4 2 0.6

G2 7 2 1 0.35

Nutrient content and cost of food

The requirement per person per day of starch, proteins andvitamins is 8, 15 and 3 respectively. The problem is to find howmuch of each food to consume per day so as to get the requiredamount per day of each nutrient at minimal cost.

This is a problem that can be formulated as a linear program. Thefirst step is to decide what are the decision variables. In this casewe choose

I x1: number of units of grain G1 to be consumed per day, and

I x2: number of units of grain G1 to be consumed per day.

Objective Function

The next step is to find the objective function. This is the functionto be maximised or minimised. In this case we want to minimisecost. This is given by

z = 0.6x1 + 0.35x2.

ConstraintsFinally we need to identify the constraints. Some constraints areobvious. We cannot produce a negative amount of grain, so wemust have

x1 ≥ 0, x2 ≥ 0.

These are the non-negativity constraints.

We also have constraints that arise from the need to meet dietaryrequirements. From the table, we see that the amount of starch isgiven by 5x1 + 7x2. We need this to be at least 8. Hence we havethe constraint

5x1 + 7x2 ≥ 8.

And similarly we have

4x1 + 2x2 ≥ 15 and

2x1 + x2 ≥ 3.

All up, we can formulate the problem as a linear program asfollows.

Minimise z = 0.6x1 + 0.35x2subject to

5x1 + 7x2 ≥ 84x1 + 2x2 ≥ 15

2x1 + x2 ≥ 3x1 ≥ 0, x2 ≥ 0.

Almost all optimisation problems can be expressed as ones oftrying to maximise or minimise something subject to some sorts ofconstraints. What distinguishes a linear programming problem isthat the objective function is a linear function and the constraintsare either linear equalities or, more usually, linear inequalities.

While the above problem was artificially simple, it’s easy to seethat there is a whole family of related problems dealing withsupplying adequate nutrition at minimum cost that can beexpressed in a similar way.

The Transportation Problem

Suppose a company manufacturing widgets has two factorieslocated at cities F1 and F2 and three retail centres located at C1,C2 and C3. The monthly demand at the retail centres are (inthousands of widgets) 8, 5 and 2 respectively while the monthlysupply at the factories are 6 and 9 respectively. Notice that thetotal supply equals the total demand. We are also given the cost oftransportation of 1 widget between any factory and any retailcenter.

C1 C2 C3

F1 5 5 3

F2 6 4 1

Cost of transportation (in 0.01$/widget).

In the transportation problem, the goal is to determine thequantity to be transported from each factory to each retail centerso as to meet the demand at minimum total shipping cost. Inorder to formulate this problem as a linear program, we first choosethe decision variables. Let xij (i = 1, 2 and j = 1, 2, 3) be thenumber of widgets (in thousands) transported from factory Fi tocity Cj . Given these xij ’s, we can express the total shipping cost,i.e. the objective function to be minimised, by

5x11 + 5x12 + 3x13 + 6x21 + 4x22 + x23.

AsideNote that we already have significantly more variables. In practice,it is not unusual for linear programming problems to have manythousands of variables.

In the transportation problem, the goal is to determine thequantity to be transported from each factory to each retail centerso as to meet the demand at minimum total shipping cost. Inorder to formulate this problem as a linear program, we first choosethe decision variables. Let xij (i = 1, 2 and j = 1, 2, 3) be thenumber of widgets (in thousands) transported from factory Fi tocity Cj . Given these xij ’s, we can express the total shipping cost,i.e. the objective function to be minimised, by

5x11 + 5x12 + 3x13 + 6x21 + 4x22 + x23.

AsideNote that we already have significantly more variables. In practice,it is not unusual for linear programming problems to have manythousands of variables.

Back to the Transportation Problem

We now need to write down the constraints. First, we have thenonnegativity constraints saying that xij ≥ 0 for i = 1, 2 andj = 1, 2, 3. Moreover, we have that the demand at each retailcentre must be met. This gives rise to the following constraints:

x11 + x21 = 8,

x12 + x22 = 5,

x13 + x23 = 2.

Finally, each factory cannot ship more than its supply, resulting inthe following constraints:

x11 + x12 + x13 ≤ 6

x21 + x22 + x23 ≤ 9.

These inequalities can be replaced by equalities since the totalsupply is equal to the total demand. A linear programmingformulation of this transportation problem is therefore given by:

Minimise z = 5x11 + 5x12 + 3x13 + 6x21 + 4x22 + x23subject to

x11 + x21 = 8x12 + x22 = 5x13 + x23 = 2

x11 + x12 + x13 = 6x21 + x22 + x23 = 9

x11 ≥ 0, x21 ≥ 0, x31 ≥ 0, x12 ≥ 0, x22 ≥ 0, x32 ≥ 0.

Notice that one of the constraints is redundant in that it is impliedby the others; but that’s ok. Also, here, the non negativityconstraints are all equalities, which is a little unusual.

The two problems described above are minimisation problems, butwe just as often have maximisation problems. A typicalminimisation problem is one of minimising costs and a typicalmaximisation problem is one of maximising profit.

Graphical Solution of 2-variable LP problems

While problems with two variables are artificially small, they areuseful for gaining insight into how we solve more general problems.

Example 1

Find the maximum value of

z = 3x + 2y

subject to

x + 2y ≤ 4

x − y ≤ 1

x ≥ 0, y ≥ 0.

Half PlanesFirst observe that each constraint is a 2-variable linear inequality.The set of solutions to such an inequality is a half plane, which ison one side of the line obtained by replacing the inequality by anequality.

Feasible Region

The intersection of all of these half planes is the feasible region.Our solution must come from a point in the feasible region.Observe that the feasible region is a convex polygon. Note thatconvex polygons have extreme points or vertices.

Level CurvesNow consider the objective function. This is the equation of aplane. We want the point on that plane that achieves themaximum value of z for points (x , y) in the feasible region.Rewrite the objective function as

y = −3

2x +

z

2.

Consider a fixed value of z , say z = 4. Then we obtain a levelcurve of the plane. Indeed we obtain

y = −3

2x + 2.

Now, say z = 6. Then we obtain

y = −3

2x + 3.

Note that the two level curves are parallel lines. In general the levelcurves form a family of parallel lines.

Hitting the Maximum

Now think about what happens as the lines sweep through thefeasible region with increasing values of z . The maximum occursat the last point of the region that meets such a line. It’s now nothard to make the following fundamental observation.

The maximum of the objective function must occur at an extremepoint of the feasible region.

Hmmm. That’s almost but not quite true. Can you think of why?

That’s radical. Instead of having to check the infinite number ofpossible points we need only check values of the objective functionat extreme points. We get the following

(x , y) z

(0, 0) 0(1, 0) 3(0, 2) 6(2, 1) 8

The maximum is z = 8 which occurs when x = 2 and y = 1.

Example 2

Of course we can use exactly the same strategy to solve aminimisation problem.

Minimisez = 5x + 7y

subject to x ≥ 0, y ≥ 0, and

2x + 3y ≥ 6

3x − y ≤ 15

−x + 4 ≤ 4

2x + 5y ≤ 27

Graphing, we obtain the following feasible region.

Checking the extreme points gives

(x , y) z

(0, 2) 14(0, 4) 28(1, 5) 40(6, 3) 51(5, 0) 25(3, 0) 15

The minimum is 14 obtained at (x , y) = (0, 2). Of course, if wewanted, we could just as easily have found the maximum.

Example 3

Consider an LP problem, where the goal is to maximise

z = 2x + 2y

with the following feasible region.

Many Optimal Solutions

Doing the usual calculation we see that the maximum of 12 occursat both (2, 4) and (5, 1). We have more than one optimal solution.In this case it is easy to check that the maximum also occurs atany point on the line segment joining (2, 4) and (5, 1).

We’ve now exposed the little white lie from earlier on. While it isalways true that optimal solutions occur at extreme points, if thereis more than one optimal solution, then there will be optimalsolutions at other points as well.

Example 4

All of the examples we have seen so far have had bounded feasibleregions, that is, regions with a finite area, we can also haveunbounded feasible regions.

Find the maximum value of

z = 4x + 2y

subject to

x + 2y ≥ 4

3x + y ≤ 1

x + 2y ≤ 7

x ≥ 0, y ≥ 0.

In this case the feasible region is

The values of z at the extreme points (4, 0), (2, 1) and (1, 4) arerespectively 16, 10 and 12, giving a maximum of 16. But this isnot the correct solution as it’s easy to see that we can make theobjective function arbitrarily large!

This is an example of an LP that has no optimal solution. Inpractice such LPs are rare; and, if you found one, you wouldprobably be happy. If your problem is to maximise profit, I thinkyou would be pretty excited if you found out that you could makean infinitely large profit!

On the other hand, problems with unbounded feasible regions canhave perfectly good solutions. If we wanted to minimise z , thenthe solution would be 10 at the point (2, 1).

Inconsistent ConstraintsIt is also possible that the constraints are such that the feasibleregion is empty, that is, there are no points that satisfy theconstraints. It’s easy to construct examples. While this is easy torecognise in 2-variable examples, it’s not always so obvious in LPswith many variables.

Real-Life Examples

The examples we have looked at so far are toy examples. Real lifeLPs will typically involve many more variables and would beimpossible to graph. Nonetheless, we have learnt quite a lot fromthe toy examples. In general, the following are true for any LP.

I Instead of being a convex polygon, the feasible region of ageneral LP is a convex polyhedron.

I The feasible region may be bounded, unbounded, or empty.

I Optimal solutions always occur at extreme points.

I If the feasible region is bounded, there will always be anoptimal solution.

I If the feasible region is unbounded there may or may not bean optimal solution.

I If there is more than one optimal solution, then there will alsobe solutions at points other than extreme points.

Those bullet points are all very well, but at this stage they are justunsubstantiated claims! Let’s say we believe them. Then all wehave to do is find the extreme points of the feasible region andcalculate the corresponding values of the objective function —with a decent computer, how hard can that be?

Oh DearConsider a problem with 100 variables. It’s quite possible that thefeasible region for this problem could be the 100-dimensionalhypercube. In this case you can check that the feasible region has2100 extreme points! As we know, that’s up there with the numberof atoms in the universe. Combinatorial explosion has hit again.We need a better way than just checking all the extreme points.