Post on 07-Apr-2022
transcript
CS6501: Topics in Learning and Game Theory(Fall 2019)
Linear Programming Duality
Instructor: Haifeng Xu
Slides of this lecture is adapted from Shaddin Dughmi athttps://www-bcf.usc.edu/~shaddin/cs675sp18/index.html
2
ΓRecap and Weak Duality
ΓStrong Duality and Its Proof
ΓConsequence of Strong Duality
Outline
3
Linear Program (LP)
minimize (or maximize) π" β π₯subject to π& β π₯ β€ π& βπ β πΆ-
π& β π₯ β₯ π& βπ β πΆ/π& β π₯ = π& βπ β πΆ1
General form:
maximize π" β π₯subject to π& β π₯ β€ π& βπ = 1,β― ,π
π₯6 β₯ 0 βπ = 1,β― , π
Standard form:
4
Application: Optimal Production
Γ π products, π raw materials
ΓEvery unit of product π uses π&6 units of raw material π
ΓThere are π& units of material π availableΓProduct π yields profit π6 per unit
ΓFactory wants to maximize profit subject to available raw materials
Can be formulated as an LP in standard form
max π" β π₯s.t. β6;-< π&6 π₯6 β€ π&, βπ β [π]
π₯6 β₯ 0, βπ β [π]
5
Primal and Dual Linear Program
max π" β π₯s.t. β6;-< π&6 π₯6 β€ π&, βπ β [π]
π₯6 β₯ 0, βπ β [π]
Primal LP Dual LP
min π" β π¦s.t. β&;-@ π&6 π¦& β₯ π6, βπ β [π]
π¦& β₯ 0, βπ β [π]
Dual LP corresponds to the buyerβs optimization problem, as follows:ΓBuyer wants to directly buy the raw material
ΓDual variable π¦& is buyerβs proposed price per unit of raw material πΓDual price vector is feasible if factory is incentivized to sell materials
ΓBuyer wants to spend as little as possible to buy raw materials
Economic Interpretation:
6
Primal and Dual Linear Program
max π" β π₯s.t. β6;-< π&6 π₯6 β€ π&, βπ β [π]
π₯6 β₯ 0, βπ β [π]
Primal LP Dual LP
min π" β π¦s.t. β&;-@ π&6 π¦& β₯ π6, βπ β [π]
π¦& β₯ 0, βπ β [π]
Upperbound Interpretation:
Dual LP can be interpreted as finding best upperbound for the primalΓ Multiplying each row π of primal by π¦& and summing the constraints
Γ Goal: find the best such π¦ to get the smallest upper bound
7
Γ So far, mainly writing the Dual based on syntactic rules
Γ Next, will show Primal and Dual are inherently related
8
Weak Duality
max πA β π₯s.t. π΄π₯ β€ π
π₯ β₯ 0
Primal LPmin πA β π¦s.t. π΄Aπ¦ β₯ π
π¦ β₯ 0
Dual LP
Theorem [Weak Duality]: For any primal feasible π₯ and dualfeasible π¦, we have π" β π₯ β€ π" β π¦
Corollary:Γ If primal is unbounded, dual is infeasibleΓ If dual is unbounded, primal is infeasibleΓ If primal and dual are both feasible, then
OPT(primal) β€ OPT(dual)
obj value of dual
obj value of primal
9
Weak Duality
max πA β π₯s.t. π΄π₯ β€ π
π₯ β₯ 0
Primal LPmin πA β π¦s.t. π΄Aπ¦ β₯ π
π¦ β₯ 0
Dual LP
Theorem [Weak Duality]: For any primal feasible π₯ and dualfeasible π¦, we have π" β π₯ β€ π" β π¦
Corollary: If π₯ is primal feasible and π¦ is dualfeasible, and π" β π₯ = π" β π¦, then both are optimal.
obj value of dual
obj value of primal
10
Interpretation of Weak Duality
Economic Interpretation: If prices of raw materials are set such that there is incentive to sell raw materials directly, then factoryβs total revenue from sale of raw materials would exceed its profit from any production.
Upperbound Interpretation: The method of rescaling and summing rows of the Primal indeed givens an upper bound of the Primalβs objective value (well, self-evidentβ¦).
11
Proof of Weak Duality
max πA β π₯s.t. π΄π₯ β€ π
π₯ β₯ 0
Primal LPmin πA β π¦s.t. π΄Aπ¦ β₯ π
π¦ β₯ 0
Dual LP
π¦" β π β₯ π¦" β π΄π₯ = π₯" β π΄"π¦ β₯ π₯" β π
12
ΓRecap and Weak Duality
ΓStrong Duality and Its Proof
ΓConsequence of Strong Duality
Outline
13
Strong Duality
Theorem [Strong Duality]: If either the primal or dual is feasibleand bounded, then so is the other and OPT(primal) = OPT(dual).
obj value of primal
obj value of dual
John von Neumann
β¦ I thought there was nothing worth publishing until the Minimax Theorem was proved.
14
Interpretation of Strong Duality
Economic Interpretation: There exist raw material prices such that the factory is indifferent between selling raw materials or products.
Upperbound Interpretation: The method of scaling and summing constraints yields a tight upperbound for the primal objective value.
15
Proof of Strong Duality
16
Projection Lemma
Weierstrassβ Theorem: Let π be a compact set, and let π(π§) be acontinuous function on π§. Then min{ π(π§) βΆ π§ β π } exists.
π§
π(π§)
17
Projection Lemma
Weierstrassβ Theorem: Let π be a compact set, and let π(π§) be acontinuous function on π§. Then min{ π(π§) βΆ π§ β π } exists.
Projection Lemma: Let π β β@ be a nonempty closed convex setand let π¦ β π. Then there exists π§β β π with minimum π/ distancefrom π¦. Moreover, β π§ β π we have π¦ β π§β "(π§ β π§β) β€ 0.
π¦ π§β
π§Proof: homework exercise
π
18
Separating Hyperplane Theorem
Theorem: Let π β β@ be a nonempty closed convex set and letπ¦ β π. Then there exists a hyperplane πΌ" β π§ = π½ that strictlyseparates π¦ from π. That is, πΌ" β π§ β₯ π½, β π§ β π and πΌ" β π¦ < π½.
π¦ π§β
π§
Proof: choose πΌ = π§β β π¦ and π½ = πΌ β π§β and use projection lemmaΓ Homework exercise
πΌ" β π§ = π½
ππΌ
19
Farkasβ LemmaFarkasβ Lemma: Let π΄ β β@Γ< and π β β@, then exactly one ofthe following two statements holds:a) There exists π₯ β β< such that π΄π₯ = π and π₯ β₯ 0b) There exists y β β@ such that π΄"π¦ β₯ 0 and π"π¦ < 0
Case a):
20
Farkasβ LemmaFarkasβ Lemma: Let π΄ β β@Γ< and π β β@, then exactly one ofthe following two statements holds:a) There exists π₯ β β< such that π΄π₯ = π and π₯ β₯ 0b) There exists y β β@ such that π΄"π¦ β₯ 0 and π"π¦ < 0
Case a):
21
Farkasβ LemmaFarkasβ Lemma: Let π΄ β β@Γ< and π β β@, then exactly one ofthe following two statements holds:a) There exists π₯ β β< such that π΄π₯ = π and π₯ β₯ 0b) There exists y β β@ such that π΄"π¦ β₯ 0 and π"π¦ < 0
Case b):
22
Farkasβ Lemma
Geometric interpretation:
Farkasβ Lemma: Let π΄ β β@Γ< and π β β@, then exactly one ofthe following two statements holds:a) There exists π₯ β β< such that π΄π₯ = π and π₯ β₯ 0b) There exists y β β@ such that π΄"π¦ β₯ 0 and π"π¦ < 0
Zπ-
Zπ/
Zπ6 is πβth column of π΄π
a) π is in the cone
23
Farkasβ Lemma
Geometric interpretation:
Farkasβ Lemma: Let π΄ β β@Γ< and π β β@, then exactly one ofthe following two statements holds:a) There exists π₯ β β< such that π΄π₯ = π and π₯ β₯ 0b) There exists y β β@ such that π΄"π¦ β₯ 0 and π"π¦ < 0
Zπ-
Zπ/
Zπ6 is πβth column of π΄
πa) π is in the coneb) π is not in the cone, and there exists a hyperplane with direction π¦
that separates π from the cone
π¦
24
Farkasβ Lemma
Proof: Γ Cannot both hold; Otherwise, yields contradiction as follows:
Γ Next, we prove if (a) does not hold, then (b) must holdβ’ This implies the lemma
Farkasβ Lemma: Let π΄ β β@Γ< and π β β@, then exactly one ofthe following two statements holds:a) There exists π₯ β β< such that π΄π₯ = π and π₯ β₯ 0b) There exists y β β@ such that π΄"π¦ β₯ 0 and π"π¦ < 0
= π¦" β π΄π₯ = π¦" β π < 0.0 β€ (π΄"π¦)" β π₯
25
Farkasβ Lemma
ΓConsider Z = {π΄π₯: π₯ β₯ 0} so that π is closed and convexΓ(a) does not hold β π β πΓBy separating hyperplane theorem, there exists hyperplane πΌ β π§ = π½ such that πΌ" β π§ β₯ π½ for all π§ β π and πΌ" β π < π½
ΓNote 0 β π, therefore π½ β€ πΌ" β 0 = 0 and thus πΌ" β π < 0ΓπΌ"π΄π₯ β₯ π½ for any π₯ β₯ 0 implies πΌ"π΄ β₯ 0 since π₯ can be arbitrary
largeΓLetting πΌ be our π¦ yields the lemma
Farkasβ Lemma: Let π΄ β β@Γ< and π β β@, then exactly one ofthe following two statements holds:a) There exists π₯ β β< such that π΄π₯ = π and π₯ β₯ 0b) There exists y β β@ such that π΄"π¦ β₯ 0 and π"π¦ < 0
Claim: if (a) does not hold, then (b) must hold.
26
An Alternative of Farkasβ LemmaFollowing corollary of Farkasβ lemma is more convenient for our proof
Corollary: Exactly one of the following systems holds:
β π₯ β β<, s.t.π΄ β π₯ β€ ππ₯ β₯ 0
β π¦ β β@, s.t.π΄A β π¦ β₯ 0πA β π¦ < 0π¦ β₯ 0
Compare to the original version
β π₯ β β<, s.t.π΄ β π₯ = ππ₯ β₯ 0
β π¦ β β@, s.t.π΄A β π¦ β₯ 0πA β π¦ < 0
27
An Alternative of Farkasβ Lemma
Corollary: Exactly one of the following systems holds:
β π₯ β β<, s.t.π΄ β π₯ β€ ππ₯ β₯ 0
β π¦ β β@, s.t.π΄A β π¦ β₯ 0πA β π¦ < 0π¦ β₯ 0
Proof: Apply Fakasβ lemma to the following linear systems
β π₯ β β<, s.t.π΄ β π₯ + πΌ β π = ππ₯, π β₯ 0
β π¦ β β@, s.t.π΄A β π¦ β₯ 0πΌ β π¦ β₯ 0πA β π¦ < 0
Following corollary of Farkasβ lemma is more convenient for our proof
28
Proof of Strong Duality
ProofΓDual of the dual is primal; so w.l.o.g assume primal is feasible and
bounded
ΓWeak duality yields OPT(primal) β€ OPT(dual) ΓNext we prove the converse, i.e., OPT(primal) β₯ OPT(dual)
max πA β π₯s.t. π΄π₯ β€ π
π₯ β₯ 0
Primal LPmin πA β π¦s.t. π΄Aπ¦ β₯ π
π¦ β₯ 0
Dual LP
Theorem [Strong Duality]: If either the primal or dual is feasibleand bounded, then so is the other and OPT(primal) = OPT(dual).
29
Proof of Strong Duality
ΓWe prove if OPT(primal)< π½ for some π½, then OPT(dual)< π½ΓApply Farkasβ lemma to the following linear system
max πA β π₯s.t. π΄π₯ β€ π
π₯ β₯ 0
Primal LPmin πA β π¦s.t. π΄Aπ¦ β₯ π
π¦ β₯ 0
Dual LP
βπ₯ β β< such thatπ΄π₯ β€ πβπA β π₯ β€ βπ½π₯ β₯ 0
βπ¦ β β@ and π§ β βπ΄Aπ¦ β ππ§ β₯ 0π"π¦ β π½π§ < 0π¦, π§ β₯ 0
ΓBy assumption, the first system is infeasible, so the second must holdβ’ If π§ > 0, can rescale (π¦, π§) to make π§ = 1, yielding OPT(dual)< π½β’ If π§ = 0, then system π΄Aπ¦ β₯ 0, π"π¦ < 0, π¦ β₯ 0 feasible. Farkasβ lemma implies
that system π΄π₯ β€ π, π₯ β₯ 0 is infeasible, contradicting theorem assumption.
30
ΓRecap and Weak Duality
ΓStrong Duality and Its Proof
ΓConsequence of Strong Duality
Outline
31
Complementary Slackness
max πA β π₯s.t. π΄π₯ β€ π
π₯ β₯ 0
Primal LPmin πA β π¦s.t. π΄Aπ¦ β₯ π
π¦ β₯ 0
Dual LP
Γ π & = π β π΄π₯ & is the πβth primal slack variableΓ π‘6 = π΄"π¦ β π 6 is the πβth dual slack variable
Complementary Slackness:π₯ and π¦ are optimal if and only if they are feasible andΓ π₯6π‘6 = 0 for all j = 1,β― ,πΓ π¦&π & = 0 for all π = 1,β― , π
Remark: can be used to recover optimal solution of the primal from optimal solution of the dual (very useful in optimization).
32
Economic Interpretation of Complementary Slackness: Given the optimal production and optimal raw material pricesΓ It only produces products for which profit equals raw material
costΓ A raw material is priced greater than 0 only if it is used up in
the optimal production
max π" β π₯s.t. β6;-< π&6 π₯6 β€ π&, βπ β [π]
π₯6 β₯ 0, βπ β [π]
Primal LP Dual LP
min π" β π¦s.t. β&;-@ π&6 π¦& β₯ π6, βπ β [π]
π¦& β₯ 0, βπ β [π]
33
Proof of Complementary Slackness
max πA β π₯s.t. π΄π₯ β€ π
π₯ β₯ 0
Primal LPmin πA β π¦s.t. π΄Aπ¦ β₯ π
π¦ β₯ 0
Dual LP
34
Proof of Complementary Slackness
Γ Add slack variables into both LPs
max πA β π₯s.t. π΄π₯ + π = π
π₯, π β₯ 0
Primal LPmin πA β π¦s.t. π΄Aπ¦ β π‘ = π
π¦, π‘ β₯ 0
Dual LP
π¦"π β π₯"π = π¦" π΄π₯ + π β π₯" π΄"π¦ β π‘ = π¦"π + π₯"π‘
35
Proof of Complementary Slackness
Γ Add slack variables into both LPs
Γ For any feasible π₯, π¦, the gap between primal and dual objectivevalue is precisely the βaggregated slacknessβ π¦"π + π₯"π‘
Γ Strong duality implies π¦"π + π₯"π‘ = 0 for the optimal π₯, π¦.
Γ Since π₯, π , π¦, π‘ β₯ 0, we have π₯6π‘6 = 0 for all j and π¦&π & = 0 for all π.
max πA β π₯s.t. π΄π₯ + π = π
π₯, π β₯ 0
Primal LPmin πA β π¦s.t. π΄Aπ¦ β π‘ = π
π¦, π‘ β₯ 0
Dual LP
π¦"π β π₯"π = π¦" π΄π₯ + π β π₯" π΄"π¦ β π‘ = π¦"π + π₯"π‘