QUADRATIC AND PARAMETRIC QUADRATIC OPTIMIZATIONoplab/thesis/romankoMS.pdf · term generally refers...

QUADRATIC AND PARAMETRIC QUADRATIC

OPTIMIZATION

AN INTERIOR POINT APPROACH TO QUADRATIC AND

PARAMETRIC QUADRATIC OPTIMIZATION

By

Oleksandr Romanko, B.Sc., M.A.

A Thesis

Submitted to the School of Graduate Studies

in Partial Fulfilment of the Requirements

for the Degree

Master of Science

McMaster University

c© Copyright by Oleksandr Romanko, August 2004

MASTER OF SCIENCE (2004) McMaster University(Computing and Software) Hamilton, Ontario

TITLE: An Interior Point Approach to Quadratic andParametric Quadratic Optimization

AUTHOR: Oleksandr Romanko, B.Sc., M.A.

SUPERVISOR: Dr. Tamas Terlaky

NUMBER OF PAGES: x, 123

ii

Abstract

In this thesis sensitivity analysis for quadratic optimization problems is stud-

ied. In sensitivity analysis, which is often referred to as parametric optimization

or parametric programming, a perturbation parameter is introduced into the op-

timization problem, which means that the coefficients in the objective function

of the problem and in the right-hand-side of the constraints are perturbed. First,

we describe quadratic programming problems and their parametric versions. Sec-

ond, the theory for finding solutions of the parametric problems is developed. We

also present an algorithm for solving such problems.

In the implementation part, the implementation of the quadratic optimiza-

tion solver is made. For that purpose, we extend the linear interior point package

McIPM to solve quadratic problems. The quadratic solver is tested on the prob-

lems from the Maros and Meszaros test set.

Finally, we implement the algorithm for parametric quadratic optimization.

It utilizes the quadratic solver to solve auxiliary problems. We present numerical

results produced by our parametric optimization package.

iii

Acknowledgments

The thesis was written under the guidance and with the help of my super-

visor, Prof. Tamas Terlaky. His valuable advices and extended knowledge of the

area helped me to do my best while working on the thesis. I am very grateful to

Mr. Alireza Ghaffari Hadigheh and Ms. Xiaohang Zhu for their contribution to

my thesis. My special thanks are to the members of the examination committee:

Dr. Ryszard Janicki (Chair), Dr. Antoine Deza, Dr. Jiming Peng and Dr. Tamas

Terlaky.

It would not be possible to complete this thesis without support and help of

all members of the Advanced Optimization Laboratory and the Department of

Computing and Software. I would like to acknowledge McMaster University for

the Ashbaugh Graduate Scholarship. I am thankful to the Canadian Operational

Research Society (CORS) for the 2004 Student Paper Competition Prize that

encouraged me in my work on the thesis.

Finally, I appreciate the support of my friends and parents and thankful to

them for their patience and understanding.

iv

Contents

List of Figures ix

List of Tables x

1 Introduction 1

1.1 Linear and Quadratic Optimization Problems . . . . . . . . . . . 1

1.2 Parametric Optimization . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Origins of Parametric Optimization: Portfolio Example . . . . . . 4

1.4 Parametric Optimization: DSL Example . . . . . . . . . . . . . . 7

1.5 Outline of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 Interior Point Methods for Quadratic Optimization Problems 11

2.1 Quadratic Optimization Problems . . . . . . . . . . . . . . . . . . 11

2.2 Primal-Dual IPMs for QO . . . . . . . . . . . . . . . . . . . . . . 13

2.2.1 The Central Path . . . . . . . . . . . . . . . . . . . . . . . 14

2.2.2 Computing the Newton Step . . . . . . . . . . . . . . . . . 14

2.2.3 Step Length . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.2.4 A Prototype IPM Algorithm . . . . . . . . . . . . . . . . . 17

2.3 Homogeneous Embedding Model . . . . . . . . . . . . . . . . . . . 18

v

2.3.1 Description of the Homogeneous Embedding Model . . . . 19

2.3.2 Finding Optimal Solution . . . . . . . . . . . . . . . . . . 20

2.4 Computational Practice . . . . . . . . . . . . . . . . . . . . . . . 22

2.4.1 Solving Homogeneous Embedding Model with Upper Bound

Constrains . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.4.2 Step Length . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.4.3 Recovering Optimal Solution and Detecting Infeasibility . 29

2.5 Solving the Newton System of Equations . . . . . . . . . . . . . . 30

2.5.1 Solving the Augmented System: Shermann-Morrison For-

mula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.5.2 Augmented System vs. Normal Equations . . . . . . . . . 31

3 Implementation of Interior Point Methods for Quadratic Opti-

mization Problems 33

3.1 General Interface of the McIPM Package . . . . . . . . . . . . . . 34

3.1.1 Reading QO Data Files . . . . . . . . . . . . . . . . . . . . 35

3.1.2 Preprocessing and Postprocessing in Quadratic Optimization 38

3.2 Structure of the McIPM Package . . . . . . . . . . . . . . . . . . 41

3.3 Solving the Newton System . . . . . . . . . . . . . . . . . . . . . 43

3.3.1 Sparse Linear Algebra Package McSML . . . . . . . . . . . 43

3.3.2 Sparse Linear Algebra Package LDL . . . . . . . . . . . . 46

3.4 Computational Algorithm for QO . . . . . . . . . . . . . . . . . . 47

3.4.1 Predictor-Corrector Strategy . . . . . . . . . . . . . . . . . 47

3.4.2 Self-Regular Functions and Search Directions . . . . . . . . 48

3.4.3 Stopping Criteria . . . . . . . . . . . . . . . . . . . . . . . 50

vi

3.5 Numerical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4 Parametric Quadratic Optimization 59

4.1 Origins of Quadratic and Linear Parametric Optimization and the

Existing Literature . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.2 Properties of Convex QO Problems . . . . . . . . . . . . . . . . . 62

4.3 The Optimal Value Function in Simultaneous Perturbation Sensi-

tivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.4 Properties of the Optimal Value Function . . . . . . . . . . . . . . 65

4.4.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . 66

4.4.2 Derivatives, Invariancy Intervals, and Transition Points . . 74

4.4.3 Computational Algorithm . . . . . . . . . . . . . . . . . . 83

4.5 Simultaneous Perturbation in Linear Optimization . . . . . . . . . 85

5 Implementation of Parametric Quadratic Optimization 89

5.1 Illustrative Example . . . . . . . . . . . . . . . . . . . . . . . . . 89

5.2 Implementing the Parametric Algorithm for QO . . . . . . . . . . 92

5.2.1 Implementation Algorithm . . . . . . . . . . . . . . . . . . 92

5.2.2 Determining Optimal Partitions and Support Sets . . . . . 95

5.3 Structure of the McParam Package . . . . . . . . . . . . . . . . . 97

5.3.1 McParam Arguments and Output . . . . . . . . . . . . . . 97

5.3.2 McParam Flow Chart . . . . . . . . . . . . . . . . . . . . 100

5.4 Computational Results . . . . . . . . . . . . . . . . . . . . . . . . 100

5.5 Analysis of Results . . . . . . . . . . . . . . . . . . . . . . . . . . 107

6 Conclusions and Further Work 109

vii

A McIPM and McParam Options 113

B Maros and Meszaros Test Set 115

Bibliography 119

viii

List of Figures

1.1 Mean-Variance Efficient Portfolio Frontier . . . . . . . . . . . . . 6

3.1 General Structure of the McIPM Package . . . . . . . . . . . . . . 41

3.2 General Structure of the McIPM Quadratic Solver . . . . . . . . . 42

4.1 The Invariancy Intervals and Transition Points . . . . . . . . . . . 71

4.2 The Optimal Value Function on Invariancy Interval . . . . . . . . 73

4.3 Neighboring Invariancy Intervals . . . . . . . . . . . . . . . . . . . 81

5.1 The Optimal Value Function for Illustrative Problem . . . . . . . 91

5.2 Optimal Partition Determination Counterexample . . . . . . . . . 96

5.3 Flow Chart of the McParam Package . . . . . . . . . . . . . . . . 101

5.4 Optimal Value Function for Perturbed lotschd QO Problem . . . 103

5.5 Optimal Value Function for Perturbed lotschd LO Problem . . . 103

5.6 Optimal Value Function for Perturbed qsc205 QO Problem . . . 104

5.7 Optimal Value Function for Perturbed qsc205 LO Problem . . . . 105

5.8 Optimal Value Function for Perturbed aug3dc QO Problem . . . 106

ix

List of Tables

3.1 McIPM Performance on QO Test Set (I) . . . . . . . . . . . . . . 55

3.2 McIPM Performance on QO Test Set (II) . . . . . . . . . . . . . . 56

3.3 McIPM Performance on QO Test Set (III) . . . . . . . . . . . . . 57

3.4 McIPM Performance on Difficult QO Problems . . . . . . . . . . 58

5.1 Transition Points, Invariancy Intervals, and Optimal Partitions

for the Illustrative Problem . . . . . . . . . . . . . . . . . . . . . 90

5.2 McParam Output for Perturbed lotschd Problem (QO Formula-

tion) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

5.3 McParam Output for Perturbed lotschd Problem (LO Formulation)102

5.4 McParam Output for Perturbed qsc205 Problem (QO Formulation)104

5.5 McParam Output for Perturbed qsc205 Problem (LO Formulation)105

5.6 McParam Output for Perturbed aug3dc Problem (QO Formulation)106

B.1 The Maros and Meszaros QO Test Set (I) . . . . . . . . . . . . . 116

B.2 The Maros and Meszaros QO Test Set (II) . . . . . . . . . . . . . 117

B.3 The Maros and Meszaros QO Test Set (III) . . . . . . . . . . . . 118

x

Chapter 1

Introduction

Optimization is a technique used for searching extremum of a function. This

term generally refers to mathematical problems where the goal is to minimize

(maximize) an objective function subject to some constraints. Depending on the

nature and the form of the objective function and the constraints, continuous

optimization problems are classified to linear, quadratic, conic and general non-

linear. Correspondingly, we distinguish the research areas of linear optimization,

quadratic optimization, etc.

1.1 Linear and Quadratic Optimization

Problems

Linear optimization, where the objective function and the constraints are linear,

is the most extensively studied branch in the optimization field. History of linear

optimization was started in the 40’s of the 20th century and has gained wide

attention in the scientific community after the development of Simplex Method

by George Dantzig in the 50’s. The Simplex algorithm and its extensions were

extensively studied since then and did not have practical competitors until the

discovery of Interior Point Methods (IPMs) in the middle 80’s. The milestone

1

M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software

work of Karmarkar in 1984 started the era of IPMs, the major rivals of Simplex

Methods, that usually outperform Simplex algorithms when solving large-scale

problems.

The main conceptual feature that differentiates IPMs from the Simplex

Method is how they search for an optimal solution. In Simplex Methods the

optimal solution is searched by moving from one vertex of the feasible region

to another until an optimal (minimum or maximum) solution is found, while in

IPMs the problem is solved by following a path inside the feasible region of the

problem that leads to optimality. IPMs allow solving large sparse optimization

problems efficiently in polynomial time. Please consult [24] for the comprehensive

description of Simplex methods and [26] for the theoretical and algorithmic back-

ground on IPMs for linear optimization. We will explore many nice properties

of IPMs that allow us to derive strong theoretical results.

In this thesis we will study quadratic optimization problems. Quadratic

Optimization (QO) problems, where the objective function is convex quadratic

while the constrains remain linear, are widely used by scientists for more than

a half-century. This class of optimization problems gained much attention in

the middle of the 50’s. A series of developments followed with the appearance

of complementarity theory and the formulation of the linear complementarity

problem.

The class of quadratic optimization problems gained its importance among

business community after Markowitz [17] used it for conducting mean-variance

analysis of investment portfolios. In the following sections we would discuss the

Markowitz model [17] in more details. Other well-known applications of QO

are coming from engineering, see e.g., [31] and other recent papers on optimal

multi-user spectrum management for Digital Subscriber Lines (DSL). Quadratic

2


optimization problems appear naturally when we want to minimize a variation

(or variance) of some quantity. In general, QO is a natural extension of linear

optimization (LO) and most of the solution methods developed for LO were

extended to QO as well.

1.2 Parametric Optimization

In many practical applications we are interested not only in the value of the

optimal solution of an optimization problem, but also in its sensitivity. In other

words, it is necessary to know how sensitive the solution is to data perturbations.

Knowing sensitivity of the solution allows to adjust the constraints (such as

resource constraints or budget constraints) or the coefficients of the objective

function (such as individual preferences or parameters of a production process)

to meet the modelling objectives in a better way and to get a ”better” solution

to the problem.

Let us look at a couple of definitions of sensitivity analysis in order to

understand its meaning better. The first definition from the Financial Glos-

sary [9] grasps the general picture and actually applies to any project or model:

”Investigation of how projected performance varies along with changes in the

key assumptions on which the quantitative estimate of future economic or finan-

cial performance are based” (in other words how the performance of the model

varies with changes in the values of important variables). The second definition

borrowed from the Mathematical Programming Glossary [11] is mathematically

more precise: ”The concern with how the solution changes if some changes are

made in either the data or in some of the solution values.”

If we ”project” both definitions to the optimization problem, we can see

that sensitivity analysis is a technique to determine how the optimal objective

3


function value varies with the change in one or more coefficients of the objective

function or right-hand side values of the problem constraints. When changes

occur in many coefficients this type of analysis is also referred to as sensitivity

analysis in many cases, but it is usually known as parametric analysis. According

to [11] ”parametric analysis is concerned with larger changes in parameter values

that affect the data in the mathematical program, such as a cost coefficient or

resource limit.”

Generally speaking, in parametric analysis a parameter λ is introduced into

the original optimization problem transforming it to the parametric one:

min f(x) φ(λ) = min f(x, λ)s.t. gi(x) ≤ 0, ∀ i =⇒ s.t. gi(x, λ) ≤ 0, ∀ i.

The goal of this thesis is to use the Interior Point Methods framework for

developing a methodology that allows finding an optimal solution vector x∗(λ)

and the optimal value function φ(λ) without discretization of the parameter space

λ and without solving the (quadratic) optimization problem at each discretiza-

tion point. This type of sensitivity analysis is often referred to as parametric

programming or parametric optimization. For the reader not to be confused,

we will use the terms sensitivity analysis, parametric analysis and parametric

optimization interchangeably throughout the thesis.

1.3 Origins of Parametric Optimization:

Portfolio Example

Mean-variance portfolio models, which are based on investor’s utility maximiza-

tion, can be formulated as optimization problems and, more precisely, as para-

metric quadratic optimization problems. Consider a universe of n assets. In such

problems the historical data known is c = (c1, ..., cn)T and Q = [σij], where ci is

4


the expected return for asset i and σij is the covariance of returns for assets i and

j. So, c is the vector of expected returns and Q is the n× n variance-covariance

matrix of asset returns (Q is a symmetric positive semidefinite matrix which fol-

lows from the properties of variance-covariance matrices). Let x = (x1, ..., xn)T

denote the vector of asset holdings. In this case the expected return of the

portfolio x is cT x and its variance is σ2 = xT Qx.

Markowitz [17] defined a portfolio to be efficient if for some fixed level of

expected return no other portfolio gives smaller variance (risk). Equivalently,

an efficient portfolio can be defined as the one for which at some fixed level of

variance (risk) no other portfolio gives larger expected return. The determina-

tion of the efficient portfolio frontier in the Markowitz mean-variance model is

equivalent to solving the following parametric QO problem due to Farrar [8]

min −λcT x + 12xT Qx

s.t. Ax = bx ≥ 0.

(1.3.1)

Here, λ > 0 is an investor’s risk aversion parameter. The linear constraints

Ax = b can represent budget constraints, bounds on asset holdings, etc. Non-

negativity constraints x ≥ 0 are short-sale constraints (non-negative asset hold-

ings). If λ is allowed to vary, (1.3.1) becomes a parametric optimization problem.

Furthermore, in this case solutions of the optimization problem for different val-

ues of λ trace the so-called efficient frontier in the mean-variance space. When

λ is large, indicating high tolerance to risk, the solution of (1.3.1) is a portfolio

with the highest expected return. When λ becomes smaller, the solution of the

optimization problems will emphasize the minimization of the portfolio variance

and put little weight on the maximization of the expected portfolio return.

If we plot the solutions of a particular instance of problem (1.3.1) for dif-

ferent values of λ in the expected return – standard deviation coordinates, they

5


3.5 4 4.5 5 5.5 6 6.5 7 7.5 84

5

6

7

8

9

10

11

12

Standard Deviation (percent per year)

Exp

ecte

d P

ortfo

lio R

etur

n (p

erce

nt p

er y

ear)

Efficient Portfolio FrontierCorner PortfoliosIndividual Stocks

λ=0

λ=∞

Figure 1.1: Mean-Variance Efficient Portfolio Frontier

trace the mean-variance efficient frontier (Figure 1.1). The mean-variance effi-

cient frontier is known to be the graphical depiction of the Markowitz efficient

set of portfolios and represents the boundary of the set of feasible portfolios that

have the maximum return for a given level of risk. Portfolios above the frontier

cannot be achieved. It was noticed that there exist some corner portfolios on the

frontier, and in between this corner portfolios the frontier is piecewise quadratic.

Figure 1.1 shows the efficient frontier in the mean-standard deviation space in

order to be consistent with the existing literature. Note that, the efficient frontier

is a piecewise quadratic function in the mean-variance space.

From the observations it seems likely that we do not need to find a solution

6


of the parametric problem for every value of λ, but instead it is necessary to

determine the corner portfolios only and ”restore” the efficient frontier between

them by calculating the quadratic function.

1.4 Parametric Optimization: DSL Example

One of the recent examples of QO problems is a model of optimal multi-user

spectrum management for Digital Subscriber Lines (DSL). Considering the be-

havior of this model under perturbations, we get a parametric quadratic problem.

Moreover, the DSL model can have simultaneous perturbation of the coefficients

in the objective function and in the right-hand side of the constraints.

Let us consider a situation when M users are connected to one service

provider via telephone line (DSL), where M cables are bundled together into the

single one. The total bandwidth of the channel is divided into N subcarriers

(frequency tones) that are shared by all users. Each user i tries to allocate his

total transmission power P imax to subcarriers to maximize his data transfer rate

N∑

k=1

pik = P i

max.

The bundling causes interference between the user lines at each subcarrier k =

1, . . . , N , that is represented by the matrix Ak of cross-talk coefficients. In addi-

tion, there is a background noise σk at frequency tone k.

Current DSL systems use fixed power levels. In contrast, allocating each

users’ total transmission power among the subcarriers ”intelligently” may result

in higher overall achievable data rates. In noncooperative environment user i

allocates his total power P imax selfishly across the frequency tones to maximize

his own rate. The DSL power allocation problem can be modelled as a mul-

tiuser noncooperative game. Nash equilibrium points of the noncooperative rate

7


maximization game correspond to optimal solutions of the following quadratic

minimization problem:

minN

∑

k=1

σkeT pk +

1

2

N∑

k=1

pTk Akpk

s.t.N

∑

k=1

pik = P i

max, i = 1, . . . ,M

pk ≥ 0, k = 1, . . . , N,

where pk = (p1k, . . . , p

Mk )T .

Engineers always look at the behavior of such models under different con-

ditions. One of the parameters that influences the model is noisiness of the

environment where the telephone cable is laid. The noisiness depends on the

material the cable is made of, on the type of insulation used, etc. This noisiness,

in turn, determines the background noise to the line σk. Users of telephone lines

residing in noisy environments may get their total transmission power Pmax in-

creased (i.e., get a more expensive modem) to improve the signal to noise ratio

of the line. Such setup results in the parametric model:

minN

∑

k=1

(σk + λ△σk)eT pk +

1

2

N∑

k=1

pTk Akpk

s.t.N

∑

k=1

pik = P i

max + λ△P imax, i = 1, . . . ,M

pk ≥ 0, k = 1, . . . , N

(1.4.1)

Parametric QO problem (1.4.1) represents a model with the noisiness pa-

rameter λ. The same parameter λ appears in the objective function and in the

right-hand side of the constraints. The parametric model allows to look at the

equilibria when the background noise and the total transmission power changes

as λ varies. This formulation, for instance, can help answering such questions

as: what happens if the background noise to the line increases two times faster

than the total transmission power available to users.

8


1.5 Outline of the Thesis

The thesis describes the theoretical background of both quadratic optimization

and parametric quadratic optimization as well as implementations of solution

techniques for them into software packages. This predetermines the following

organization of the thesis.

In the current Chapter 1, we outline the history and the use of quadratic

optimization techniques. In addition, we introduce the concept of parametric

quadratic optimization and provide an example of portfolio problem which is

formulated as parametric quadratic optimization problem. We also consider an

engineering model that is formulated as simultaneous perturbation parametric

optimization problem. Finally, the outline of the thesis is provided.

Chapter 2 contains the background of using interior point methods for

quadratic optimization. We use the homogenous embedding model and self-

regular proximity functions. Chapter 2 also contains all theoretical results nec-

essary for the implementation.

Chapter 3 is devoted to the implementation of the interior point method

algorithm outlined in Chapter 2. We describe the algorithm itself, problem

specification formats, preprocessing and postprocessing, as well as the core of the

methodology – sparse linear system solvers. Finally, we provide computational

results to benchmark our software with existing quadratic solvers.

In Chapter 4 we make a link from the quadratic optimization to its para-

metric counterpart. We provide the necessary background, prove some proper-

ties of such problems and suggest an algorithm for solving parametric quadratic

optimization problems when the perturbation occurs simultaneously at the right-

hand side of the constraints and in the objective function. In addition, we spe-

cialize our results to parametric linear optimization.

9


Chapter 5 describes the implementation of the parametric programming

algorithm in the MATLAB environment and provide an illustrative example of

solving parametric problem. Chapter 5 also presents our computational results.

Finally, Chapter 6 contains concluding remarks and suggestions for future

work.

10

Chapter 2

Interior Point Methods for

Quadratic Optimization

Problems

In this chapter we extend our introductory knowledge about quadratic optimiza-

tion (QO) problems, describe their properties and solution techniques. As we

already know, QO problem consists of minimizing a convex quadratic objective

function subject to linear constraints. In addition to showing problem formu-

lations, we review the duality theory for QO problems. Finally, Interior Point

Methods (IPMs) for solving these problems are described. Results presented in

this chapter are mainly based on [2], [35] and [36].

2.1 Quadratic Optimization Problems

A primal convex QO problem is defined as:

min cT x + 12xT Qx

(QP ) s.t. Ax = bx ≥ 0,

(2.1.1)

where Q ∈ IRn×n is a symmetric positive semidefinite matrix, A ∈ IRm×n,

rank(A) = m, c ∈ IRn, b ∈ IRm are fixed data and x ∈ IRn is an unknown

11


vector.

The Wolfe Dual of (QP ) is given by

max bT y − 12uT Qu

(QD) s.t. AT y + z − Qu = cz ≥ 0,

(2.1.2)

where z, u ∈ IRn and y ∈ IRm are unknown vectors. Note that when Q = 0 we

get a Linear Optimization (LO) problem.

The feasible regions of (QP ) and (QD) are denoted by

QP = {x : Ax = b, x ≥ 0},

QD = {(u, y, z) : AT y + z − Qu = c, z, u ≥ 0},

and their associated optimal solution sets are QP∗ and QD∗, respectively. It is

known that for any optimal solution of (QP ) and (QD) we have Qx = Qu and

xT z = 0, see e.g., Dorn [7]. It is also known from [7] that there are optimal

solutions with x = u. Since we are only interested in the solutions where x = u,

therefore, u will be replaced by x in the dual problem. The duality gap cT x +

xT Qx − bT y = xT z being zero is equivalent to xizi = 0 for all i ∈ {1, 2, . . . , n}.This property of the nonnegative variables x and z is called the complementarity

property.

Solving primal problem (QP ) or dual problem (QD) is equivalent to solving

the following system, which represents the Karush-Kuhn-Tucker (KKT) optimal-

ity conditions [33]:

Ax − b = 0, x ≥ 0,AT y + z − Qx − c = 0, z ≥ 0,

xT z = 0,(2.1.3)

where the first line is the primal feasibility, the second line is the dual feasibility,

and the last line is the complementarity condition. The complementarity condi-

12


tion can be rewritten as xz = 0, where xz denotes the componentwise product

of the vectors x and z. System (2.1.3) is referred to as the optimality conditions.

Let X = diag(x1, . . . , xn) and Z = diag(z1, . . . , zn) be the diagonal matrices

with vectors x and z forming the diagonals, respectively. For LO the Goldman-

Tucker Theorem states that there exists a strictly complementary optimal solution

(x, y, z) if both the primal and dual problems are feasible. For LO, the feasible

primal-dual pair (x, y, z) is strictly complementary if xizi = 0 and xi + zi > 0

for all i = 1, . . . , n. Equivalently, strict complementarity can be characterized by

xz = 0 and rank(X) + rank(Z) = n.

Unlike in LO, where strictly complementary optimal solution always exists,

for QO the existence of such solution is not ensured. Instead, a maximally

complementary solution can be found. A pair of optimal solutions (x, y, z) for

the QO problem is maximally complementary if it maximizes rank(X)+rank(Z)

over all optimal solution pairs. As we see in Chapter 4, this leads to tri-partition

of the optimal solution set.

2.2 Primal-Dual IPMs for QO

Primal-dual IPMs are iterative algorithms that aim to find a solution satis-

fying the optimality conditions (2.1.3). IPMs generate a sequence of iterates

(xk, yk, zk), k = 0, 1, 2, . . . that satisfy the strict positivity (interior point) condi-

tion xk > 0 and zk > 0, but feasibility (for infeasible IPMs) and optimality are

reached as k goes to infinity. In this thesis we are concerned about feasible IPM

methods which produce a sequence of iterates where the following interior point

condition (IPC) holds for every iterate (x, y, z)

Ax = b, x > 0,AT y + z − Qx = c, z > 0.

(2.2.1)

13


2.2.1 The Central Path

We perturb the complementarity condition in the optimality conditions (2.1.3)

asAx = b, x > 0,

AT y + z − Qx = c, z > 0,Xz = µe,

(2.2.2)

where µ > 0 and e = (1, . . . , 1)T . It is obvious that the last nonlinear equation

in (2.2.2) becomes the complementarity condition for µ = 0.

A desired property of system (2.2.2) is the uniqueness of its solution for

each µ > 0. The following theorem [12] shows such conditions.

Theorem 2.2.1 System (2.2.2) has a unique solution for each µ > 0 if and only

if rank(A) = m and the IPC holds for some point.

When µ is running through all positive numbers, the set of unique solutions

(x(µ), y(µ), z(µ)) of (2.2.2) define the so-called primal-dual central path. The sets

{x(µ) |µ > 0} and {(y(µ), z(µ)) |µ > 0} are called primal central path and dual

central path respectively.

One iteration of primal-dual IPMs consists of taking a Newton step applied

to the central path equations (2.2.2) for a given µ. The central path stays in the

interior of the feasible region and the algorithm approximately follows it towards

optimality. For µ → 0 the set of points (x(µ), y(µ), z(µ)) gives us a maximally

complementary optimal solution of (QP) and (QD).

2.2.2 Computing the Newton Step

Newton’s method is used to solve the system (2.2.2) iteratively. At each step

we need to compute the direction (△x,△y,△z). A new point in the computed

14


direction (x + △x, y + △y, z + △z) should satisfy

A(x + △x) = b,AT (y + △y) + (z + △z) − Q(x + △x) = c,

(x + △x)(z + △z) = µe.

For a given strictly feasible primal-dual pair (x, y, z) we can write this system

with the variables (△x,△y,△z) as

A△x = 0,AT△y + △z − Q△x = 0,

x△z + z△x + △x△z = µe − xz.(2.2.3)

System (2.2.3) is non-linear. Consequently, the Newton step is obtained by

dropping the non-linear term that gives the linearized Newton system

A△x = 0,AT△y + △z − Q△x = 0,

x△z + z△x = µe − xz.(2.2.4)

The linear system (2.2.4) is referred to as the primal-dual Newton system. It has

2n + m equations and 2n + m unknowns. The system has a unique solution if

rank(A) = m.

In matrix form the Newton system (2.2.4) can be written as

A 0 0−Q AT IZ 0 X

△x△y△z

=

00

µe − Xz

. (2.2.5)

Solving system (2.2.5) for △z gives

△z = X−1(µe − Xz − Z△x),

and substituting △z into system (2.2.5) we get(

−Q − D AT

A 0

)(

△x△y

)

=

(

µX−1e − z0

)

, (2.2.6)

where D = X−1Z is a diagonal matrix with Dii = zi

xifor i = 1, . . . , n. System

(2.2.6) is called the augmented system.

15


From the first equation of (2.2.6) we can express △x as

△x = (Q + D)−1(AT△y − µX−1e + z).

Then the augmented system reduces to:

A(Q + D)−1AT△y = A(Q + D)−1(µX−1e − z), (2.2.7)

that is often called the normal equation form.

2.2.3 Step Length

In this section we describe how to determine the next iteration point. After solv-

ing the Newton system (2.2.5) we are getting the search direction (∆x, ∆y, ∆z).

This Newton search direction is computed assuming that the step length α is

equal to one. But taking such step can lead to loosing strict feasibility of the

solution as (x+∆x, y+∆y, z+∆z) might be infeasible. Our goal is to keep strict

feasibility, therefore we want to find such an α that the next iteration point is

strictly feasible, i.e.,

(xk+1, yk+1, zk+1) = (xk, yk, zk) + α(∆xk, ∆yk, ∆zk),

with xk+1 > 0 and zk+1 > 0. This can be done in two steps:

• find the maximum possible step size αmax such that

αmax = arg maxα>0

{(

xz

)

+ α

(

△x△z

)

≥ 0

}

,

• as strict feasibility is not warranted by the previous step, we need to use a

damping factor ρ ∈ (0, 1) to choose such α that

α = min{ραmax, 1}.

16


Finally, we can compute αmax in the following way:

αmaxP = min

∆xj<0{−xj

∆xj

, j = 1, . . . , n},

αmaxD = min

∆zj<0{−zj

∆zj

, j = 1, . . . , n},αmax = min{αmax

P , αmaxD }.

2.2.4 A Prototype IPM Algorithm

Here we present a prototype Primal-Dual Path-Following IPM algorithm for QO

problems in standard form. We suppose that the IPC is satisfied. As we see in

Section 2.3, by using the homogeneous embedding technique we can always con-

struct a QO problem in a way that the IPC holds. After that, we apply Newton’s

method to the central path equations (2.2.2) to get the search direction. The step

length is determined as described in Section 2.2.3 that ensures that the iterate

remains in the interior of the feasible set. IPMs stop when the complementarity

gap is reduced below some predetermined tolerance level or when infeasibility is

detected.

One issue that we have not discussed so far is a strategy for reducing the

parameter µ. We want to follow the central path approximately (as Newton

system is solved disregarding the nonlinear terms) and so we define a proximity

function Ψ(xz, µ) to measure the distance of the current point from the central

path. A proximity parameter δ > 0 defines the bound for the proximity function

which takes the value of zero if the point is on the central path and approaches

infinity if the point approaches the boundary of the nonnegative orthant. There

are many proximity measures defined in the literature, we will describe the fam-

ily of so-called self-regular proximity functions in Section 3.4.2.

17


A Prototype IPM Algorithm

input:

a proximity parameter δ > 0;

an accuracy parameter ǫ > 0;

an update parameter 0 < θ < 1;

µ0 = 1, k = 0;

(x0, y0, z0) satisfying x0 > 0, z0 > 0 and Ψ(x0z0, µ0) ≤ δ;

begin

while (xk)T zk ≥ ǫ do

begin

µk = (1 − θ)( (xk)T zk

n);

while Ψ(xkzk, µk) ≥ δ do

begin

solve the Newton system (2.2.5) to find (△xk,△yk,△zk);

determine the step size α;

xk = xk + α△xk, yk = yk + α△yk, zk = zk + α△zk;

end

xk+1 = xk, yk+1 = yk, zk+1 = zk, k = k + 1;

end

end

2.3 Homogeneous Embedding Model

In this section we present a homogeneous algorithm to solve the QO problem.

The homogeneous embedding model is one of the ways to formulate the system

of linear equations associated with the QO problem. Such model for the LO

has been developed by Ye, Todd and Mizuno [32]. Later on it was extended by

18


Anderson and Ye [2] to monotone complementarity problems (MCP). The QO

problem is a special case of MCP and so, the algorithm applies for QO as well.

2.3.1 Description of the Homogeneous Embedding Model

From the Weak Duality Theorem it follows that system (2.1.3) is equivalent to

Ax − b = 0, x ≥ 0,AT y + z − Qx − c = 0, s ≥ 0,bT y − cT x − xT Qx ≥ 0.

(2.3.1)

A way to solve system (2.3.1) is to introduce a slack variable κ for the last

inequality in the system and a homogenization variable τ . The following system

is a homogeneous reformulation of the QO problem:

Ax − bτ = 0, x ≥ 0, τ ≥ 0,AT y + z − Qx − cτ = 0, z ≥ 0, y free,

bT y − cT x − xT Qx

τ− κ = 0, κ ≥ 0.

(2.3.2)

System (2.3.2) has some attractive properties. First, with τ = 1 and κ = 0

the solution of the system (2.3.2) gives an optimal solution of the QO problem.

Second, it has zero as its trivial solution and can be considered as an MCP

with zero right-hand side vector. Third, considered as an MCP, system (2.3.2)

does not satisfy the IPC. The modification of problem (2.3.2) to a homogeneous

problem having an interior point is (Anderson and Ye [2])

min xT z + τκs.t. Ax − bτ − rP ν = 0,

−AT y + Qx + cτ − rDν − z = 0,

bT y − cT x − xT Qx

τ− rGν − κ = 0,

rTP y + rT

Dx + rGτ = −β,y, ν free, x ≥ 0, τ ≥ 0, z ≥ 0, κ ≥ 0,

(2.3.3)

where ν is an artificial variable added in order to satisfy the IPC, the coefficients

rP , rD and rG represent the infeasibility of the primal and dual initial interior

19


points and the duality gap, respectively. These coefficients for a given initial

point (x0 > 0, y0, z0 > 0, τ 0 > 0, κ0 > 0, ν0 > 0) are defined as follows:

rP = (Ax0 − bτ 0)/ν0,rD = (−AT y0 + Qx0 + cτ 0 − z0)/ν0,

rG = (bT y0 − cT x0 − x0T

Qx0

τ0 − κ0)/ν0,β = −rT

P y0 − rTDx0 − rGτ 0.

The homogeneous embedding model (2.3.3) has several advantages. First,

the algorithm based on it does not need to use a ”big-M” penalty parameter [2].

Second, by utilizing the homogeneous embedding model we can avoid using the

two-phase method, where we need to find a feasible interior initial point to start

with, that even might not exist for many problems. It is not difficult to note

that for properly chosen rP , rD and rG the point (x0 = e, y0 = 0, z0 = e, τ 0 = 1,

κ0 = 1, ν0 = 1) is feasible for the embedding model. Third, the homogeneous

embedding model generates a solution sequence converging towards an optimal

solution of the original problem, or it produces an infeasibility certificate for

either (QP ), or (QD), or for both. We can apply the IPM algorithm outlined

in Section 2.2.4 for solving the homogeneous embedding model. The size of the

Newton system for this model is not significantly larger than the size of the

original system. Finally, a small update IPM for solving the homogeneous model

has the iteration bound O(√

n log nǫ).

2.3.2 Finding Optimal Solution

Let us consider a strictly complementary solution (y∗, x∗, τ ∗, ν∗ = 0, z∗, κ∗) of the

homogeneous problem (2.3.3). Our goal is to recover the information about solu-

tions of the original primal (QP ) and dual (QD) QO problems. We distinguish

three cases:

1. If τ ∗ > 0 and κ∗ = 0, then (x∗

τ∗, z∗

τ∗, y∗

τ∗) is a strictly complementary optimal

20


solution for (QP ) and (QD).

2. If τ ∗ = 0 and κ∗ > 0, the solution of the embedding model provides a

Farkas certificate for the infeasibility of the dual and/or primal problems

and then:

• if cT x∗ < 0, then the dual problem (QD) is infeasible;

• if −bT y∗ < 0, then the primal problem (QP ) is infeasible;

• if cT x∗ < 0 and −bT y∗ < 0, then both primal (QP ) and dual (QD)

problems are infeasible.

3. If τ ∗ = 0 and κ∗ = 0 then neither a finite solution nor a certificate proving

infeasibility exists. This cannot happen for convex QO problems, because

if no optimal solutions exist, then an infeasibility certificate always exist

[33].

Let us prove the conclusion of case 2. For τ ∗ = 0, κ∗ > 0 and ν∗ = 0 we

haveAx∗ = 0,

AT y∗ − Qx∗ + z∗ = 0.

In addition, the third constraint of (2.3.3) imply that

bT y∗ − cT x∗ − x∗T Qx∗

τ ∗− κ∗ = 0,

and as κ∗ > 0 and x∗T Qx∗

τ∗≥ 0, thus

bT y∗ − cT x∗ > 0,

i.e.,

cT x∗ − bT y∗ < 0.

21


The last inequality means that at least one of the components of the left-hand

side, namely cT x∗ or −bT y∗, is strictly less than zero. Let us consider these three

cases separately:

• Consider the case when cT x∗ < 0. Let us assume to the contrary that

a feasible solution (x∗, y, z) for the dual problem such that z ≥ 0 and

AT y − Qx∗ + z = c exists. Then

0 > cT x∗

= (AT y − Qx∗ + z)T x∗

= yT (Ax∗) − x∗T Qx∗ + zT x∗

= zT x∗

≥ 0,

that is a contradiction. Consequently, the dual problem is infeasible.

• Consider the case when −bT y∗ < 0. Let us assume to the contrary that

a feasible solution x for the primal problem such that x ≥ 0 and Ax = b

exists. Then0 > −bT y∗

= (−Ax)T y∗

= xT (−AT y∗)= xT z∗

≥ 0,

that is a contradiction. Consequently, the primal problem is infeasible.

• If both cT x∗ < 0 and −bT y∗ < 0, then by the same reasoning we get both

the primal and dual problems to be infeasible.

2.4 Computational Practice

The majority of QO problems are not given in the standard form (2.1.1). Instead,

problem formulations may include inequality constraints, free variables as well as

lower and upper bounds on the variables. In this section we derive the augmented

22


system of the type (2.2.6) and the normal equations of the type (2.2.7) for such

formulations using the homogeneous embedding model. The first step in this

process is the preprocessing stage when problem transformations are applied to

bring a general form problem to the standard form (2.1.1), while upper bounds

for the variables are allowed and treated separately. These transformations are

referred to as preprocessing.

If a QO problem has inequality constraints Ax ≤ b and non-negativity

constraints x ≥ 0, non-negative slack variables x are added to the constraints to

transform the problem to the standard form

(A I)

(

xx

)

= b,

(

xx

)

≥ 0,

where I is the identity matrix of appropriate dimension.

If a problem contains variables xj not restricted to be non-negative (free

variables), then these variables are split to two non-negative variables xj = x+j −

x−j , x+

j ≥ 0, x−j ≥ 0. So, after taking care of inequality constraints and free

variables we get a larger problem conform with the standard form (2.1.1).

Problems including variables that have upper and lower bounds require

more attention. If a variable has a lower bound xi ≥ li, then we shift the

lower bound to zero by substituting the variable xi by xi − li ≥ 0. Appropriate

changes should be made in the vectors c and b: c is substituted by c + Q l and b

by b−A l. If the variable had an upper bound xi ≤ ui, then this upper bound is

shifted as well ui = ui − li. After such shift we have a problem of the same size

with nonnegative variables and possible upper bounds on some variables. The

appropriate back transformation of variables should be made after solving the

QO problems at the postprocessing stage.

For the variables having upper bounds xi ≤ ui, extra slack variables are

added to transform the inequality constraints to equality constraints. This ob-

23


viously leads to increase in the number of variables and constraints. So, the last

two sets of constraints of the primal QO problem

min cT x + 12xT Qx

s.t. Ax = b,0 ≤ xi ≤ ui, i ∈ I,0 ≤ xj, j ∈ J ,

(2.4.1)

where A ∈ IRm×n, c, x ∈ IRn, b ∈ IRm, and I and J are disjoint partition of the

index set {1, . . . , n}, can be rewritten as

Fx + s = u, x ≥ 0, s ≥ 0, F ∈ IRmf×n,

where mf = |I| and s ∈ IRmf is the slack vector. Matrix F consists of the

unit vectors associated with the index set I as its rows. Consequently, the QO

problem becomesmin cT x + 1

2xT Qx

s.t. Ax = bFx + s = ux ≥ 0, s ≥ 0.

(2.4.2)

Its dual ismin bT y − uT w − 1

2xT Qx

s.t. AT y − F T w + z − Qx = c,w ≥ 0, z ≥ 0.

(2.4.3)

where y ∈ IRm, w ∈ IRmf and z ∈ IRn. The complementarity gap is

gap = cT x + 12xT Qx − (bT y − uT w − 1

2xT Qx)

= (AT y − F T w + z − Qx)T x − yT (Ax) + wT (Fx + s) + xT Qx= xT z + sT w.

Finally, the optimality conditions for (2.4.2) and (2.4.3) can be written as

Ax = b,Fx + s = u,

AT y − F T w + z − Qx = c,Xz = 0,Sw = 0.

(2.4.4)

24


2.4.1 Solving Homogeneous Embedding Model

with Upper Bound Constrains

Addition of the primal equality constraint Fx+s = u to the problem formulation

results in the following homogeneous embedding model that is a straightforward

generalization of (2.3.3):

min xT z + sT w + τκs.t. Ax − bτ − rP1ν = 0,

− Fx + uτ − rP2ν − s = 0,−AT y + F T w + Qx + cτ − rDν − z = 0,

bT y − uT w − cT x − xT Qx

τ− rGν − κ = 0,

rTP1y + rT

P2w + rTDx + rGτ = −β,

y, ν free, w ≥ 0, x ≥ 0, τ ≥ 0, z ≥ 0, s ≥ 0, κ ≥ 0,

(2.4.5)

whererP1 = (Ax0 − bτ 0)/ν0,rP2 = (−Fx0 + uτ 0 − s0)/ν0,rD = (−AT y0 + F T w0 + Qx0 + cτ 0 − z0)/ν0,

rG = (bT y0 − uT w0 − cT x0 − x0T

Qx0

τ0 − κ0)/ν0,β = −rT

P1y0 − rT

P2w0 − rT

Dx0 − rGτ 0.

The objective function of problem (2.4.5) can be also expressed as follows.

Multiplying the first, second, third, forth and fifth equality constraints of (2.4.5)

by yT , wT , xT , τ and ν correspondingly, and summing them up, we get

xT z + sT w + τκ = νβ.

Designing the IPM algorithm for the homogeneous embedding model we

can follow the same reasoning as in Section 2.2.1 and define the central path for

problem (2.4.5). The central path is the set of solutions of the following system

25


for all µ > 0:

Ax − bτ − rP1ν = 0,− Fx + uτ − rP2ν − s = 0,

−AT y + F T w + Qx + cτ − rDν − z = 0,

bT y − uT w − cT x − xT Qx

τ− rGν − κ = 0,

rTP1y + rT

P2w + rTDx + rGτ = −β,

Xz = µe,Sw = µe,τκ = µ.

(2.4.6)

The Newton system for (2.4.6) is

0 0 A −b −rP1 0 0 00 0 −F u −rP2 −I 0 0

−AT F T Q c −rD 0 −I 0

bT −uT −cT − 2xT Q

τ

xT Qx

τ2 −rG 0 0 −1rTP1 rT

P2 rTD rT

G 0 0 0 00 S 0 0 0 W 0 00 0 Z 0 0 0 X 00 0 0 κ 0 0 0 τ

△y△w△x△τ△ν△s△z△κ

=

00000

rsw

rxz

rτκ

, (2.4.7)

wherersw = µe − Sw,rxz = µe − Xz,rτκ = µe − τκ.

From the last three lines of (2.4.7) we get

△s = W−1(rsw − S△w),△z = X−1(rxz − Z△x),△κ = τ−1(rτκ − κ△τ).

(2.4.8)

We also have

△ν =rTswe + rT

xze + rτκ

β.

Consequently, system (2.4.7) reduces to

0 0 A −b0 W−1S −F u

−AT F T Q + X−1Z c

bT −uT −cT − 2xT Q

τ

xT Qx

τ2 + κτ

△y△w△x△τ

=

r′

r′swr′xz

r′τκ

, (2.4.9)

26


wherer′ = rP1△ν,

r′sw = rP2△ν + W−1rsw,r′xz = rD△ν + X−1rxz,r′τκ = rG△ν + τ−1rτκ.

From the second equation of (2.4.9) we get

△w = WS−1r′sw + WS−1F△x − WS−1u△τ, (2.4.10)

that allows us to reduce the system to

Q+X−1Z+F T WS−1F −AT c − F T WS−1uA 0 −b

−cT − 2xT Q

τ−uT WS−1F bT xT Qx

τ2 + κτ+uT WS−1u

△y△x△τ

=

r′′xz

r′′

r′′τκ

, (2.4.11)

whereD−1 = Q + X−1Z + F T WS−1F,

r′′ = r′,r′′xz = r′xz − F T WS−1r′sw,r′′τκ = r′τκ + uT WS−1r′sw.

Equation (2.4.11) is the augmented system. From the second equation of (2.4.11),

we have

△x = Dr′′xz + DAT△y − D(c − F T WS−1u)△τ.

Consequently, system (2.4.11) reduces to(

ADAT a1

−(a2)T a3

)(

△y△τ

)

=

(

r′′′

r′′′τκ

)

, (2.4.12)

where

a1 = −b − AD(c − F T WS−1u),

−(a2)T = bT − (cT + 2xT Q

τ+ uT WS−1F )DAT ,

a3 = xT Qx

τ2 + κτ

+ uT WS−1u + (cT + 2xT Q

τ+ uT WS−1F )D(c − F T WS−1u),

r′′′ = r′′ − ADr′′xz,

r′′′τκ = r′′τκ + (cT + 2xT Q

τ+ uT WS−1F )Dr′′xz.

From the last equation of (2.4.12), we get

△τ =1

a3(r′′′τκ + (a2)T△y),

27


and then system (2.4.12) reduces even further to

[ADAT + aaT ]△y = ξ, (2.4.13)

where a = a1

a3 , a = a2 and ξ = r′′′ − r′′′τκ

a3 a1.

At this point we have two choices: to solve the normal equations (2.4.13)

or to solve the augmented system (2.4.11). First, let us reduce the augmented

system (2.4.11) by rewriting it in the following form:

−Q − X−1Z − F T WS−1F AT −a′

A 0 −b−(a′′)T bT a′′′

△y△x△τ

=

−r′′xz

r′′

r′′τκ

, (2.4.14)

wherea′ = c − F T WS−1u,

a′′ = c + 2Qx

τ+ F T WS−1u,

a′′′ = xT Qx

τ2 + κτ

+ uT WS−1u.

From the last equation of (2.4.14), we get:

△τ =1

a′′′

[

r′′τκ −(

−a′′

b

)T (

△y△x

)

]

, (2.4.15)

and then system (2.4.14) reduces to:

[(

−Q−X−1Z−F T WS−1F AT

A 0

)

+ aaT

](

△y△x

)

=

(

−r′′xz + a′ r′′

τκ

a′′′

r′′ + b r′′τκ

a′′′

)

, (2.4.16)

where

a =

(

a′

b

)

,

aT = 1a′′′

(

−a′′

b

)T

.

Now, we have got two linear systems: the normal equations and the aug-

mented system. We can solve either of them to find the solution of the Newton

system for the homogeneous model and get the search direction for IPMs QO

algorithms. Efficient solution of those systems is the subject of Section 2.5.

28


2.4.2 Step Length

We can follow the same reasoning as in Section 2.2.3 to derive analogous results

about the step length for the homogeneous embedding model. After solving the

augmented system (2.4.16), we obtain ∆y and ∆x. Doing back substitutions to

the equations (2.4.15), (2.4.10) and (2.4.8) we get ∆τ , ∆w, ∆z, ∆s and ∆κ. The

maximum acceptable step length α for getting strictly interior point in the next

iteration is determined as follows:

αmaxx = min

∆xj<0{−xj

∆xj

, j = 1, . . . , n},

αmaxz = min

∆zj<0{−zj

∆zj

, j = 1, . . . , n},

αmaxw = min

∆wj<0{−wj

∆wj

, j = 1, . . . , n},

αmaxs = min

∆sj<0{−sj

∆sj

, j = 1, . . . , n},

αmaxτκ = min

∆τ<0, ∆κ<0{−τ

∆τ,−κ

∆κ, j = 1, . . . , n},

αmax = min{αmaxx , αmax

z , αmaxw , αmax

s , αmaxτκ , },

α = min{ραmax, 1}.

(2.4.17)

The damping parameter ρ is usually close to 0.99, in our implementation it has

been chosen equal to 0.995.

2.4.3 Recovering Optimal Solution and Detecting

Infeasibility

This subsection generalizes the results of Section 2.3.2 for the problems with

upper bounds. After getting a strictly complementary solution (y∗, w∗, x∗, τ ∗,

ν∗ = 0, z∗, s∗, κ∗) of the homogenous embedding model (2.4.5) we can either

recover the optimal solution of the original primal (2.4.2) and dual (2.4.3) QO

problems or get an infeasibility certificate. We present the generalized results

below:

1. If τ ∗ > 0 and κ∗ = 0, then (x∗

τ∗, z∗

τ∗, y∗

τ∗, w∗

τ∗, s∗

τ∗) is a strictly complementary

optimal solution for (2.4.2) and (2.4.3).

29


2. If τ ∗ = 0 and κ∗ > 0, the solution of the embedding model provides a

primal and/or dual infeasibility certificate for problems (2.4.2) and (2.4.3):

• if cT x∗ < 0, then the dual problem is infeasible;

• if −bT y∗ + uT w∗ < 0, then the primal problem is infeasible;

• if cT x∗ < 0 and −bT y∗ + uT w∗ < 0, then both primal and dual prob-

lems are infeasible.

2.5 Solving the Newton System of Equations

2.5.1 Solving the Augmented System:

Shermann-Morrison Formula

The Sherman-Morrison formula [33] is a formula that allows to compute the

inverse of a modified matrix P + UV T for a given matrix P and matrices U and

V . Here, P is n×n nonsingular matrix and the dimensions of U and V are n×k,

then the Sherman-Morrison formula is given as

(P + UV T )−1 = P−1 − P−1U(1 + V T P−1U)−1V T P−1. (2.5.1)

We can use the Sherman-Morrison formula (2.5.1) to solve the augmented

system (2.4.16) or the normal equations (2.4.13) efficiently. First, we show how

to use the Sherman-Morrison formula for the augmented system. The reasons for

choosing the augmented system over the normal equations approach will become

clear in the next subsection. Let us denote the right-hand side of the system

(2.4.16) by ξ. Furthermore, let U = a, V = a and

P =

(

−Q − X−1Z − F T WS−1F AT

A 0

)

.

Consequently, the solution of the system (2.4.16) is given by(

△y△x

)

= P−1ξ − P−1a(1 + aT P−1a)−1aT P−1ξ. (2.5.2)

30


It is known to be inefficient and numerically difficult to compute inverse matrices,

so instead of inverting P to compute P−1ξ and P−1a, we solve the following two

systems of linear equations:P v = ξ,P v = a.

(2.5.3)

Combining the solutions of the two systems above with formula (2.5.2), we get

(

△y△x

)

= v − vaT v

1 + aT v.

To solve the systems of equations (2.5.3) involving the coefficient matrix

P , we need to compute the LDLT factorization of P , as it is explained in the

next section.

2.5.2 Augmented System vs. Normal Equations

Solving the system of linear equations (augmented system or normal equations)

is the most time-consuming part of IPMs. As computational practice shows, the

number of iterations in IPMs is nearly independent of the problem size. So, the

speed of computing one iteration is extremely important and this speed is mainly

related to the costs of solving the linear systems (2.5.3).

As we know from the previous section, we have two choices: to solve the

normal equations system (2.4.13) or to solve the augmented system (2.4.16). In

the first case the effort is related to the costs of the Cholesky factorization of the

symmetric positive definite matrix ADAT [21]. In the second case the symmetric

indefinite matrix(


A 0

)

(2.5.4)

needs to be factored by using LDLT factorization [21].

We can use Cholesky decomposition in the normal equations approach as

matrix ADAT is positive definite. We can first perform the Cholesky decompo-

sition LLT = D−1 = −Q−X−1Z −F T WS−1F, to get the system A(LLT )−1AT ,

31


which is factored by the second Cholesky decomposition LLT = A(LLT )−1AT .

In practice the permutation and symbolic factorization of the matrices D and

A(LLT )−1AT is performed independently that allows to reduce the fill-in dur-

ing Cholesky decompositions. The attractive property of the normal equations

approach is that it is possible to use Cholesky decomposition, which is usually

faster than the symmetric one, for solving the linear system. Still, the price we

pay is the need to perform two decompositions.

The second approach includes performing the symmetric LDLT factoriza-

tion of the matrix given by (2.5.4). There are again two possibilities. First, we

can use Bunch-Parlett strategy to perform the LDLT factorization, where L is

a lower triangular matrix and D is a non-singular block diagonal matrix with

2× 2 diagonal blocks. Another approach to performing the symmetric factoriza-

tion follows from the observation that matrix (2.5.4) is quasi-definite. For such

matrices there exists an LDLT factorization that D is a diagonal matrix, see

e.g., Vanderbei [29]. In our implementation, we use a factorization based on the

Vanderbei’s technique. You can find more details about that in Chapter 3.

It was shown by Meszaros [21] that factoring the augmented system allows

more freedom in the pivot sequence and is advantageous in the majority of the

cases. Because of that we are using the augmented system approach for our

implementation.

In the linear optimization case usage of the augmented system has no sig-

nificant advantages over the normal equations approach. As the matrix D in the

LO case is diagonal with positive elements, both the augmented system and the

normal equations require factoring only one symmetric nonsingular (quasidefi-

nite or definite, respectively) system. The only difference is that the augmented

system approach, even in the linear case handle dense columns automatically,

while with normal equations dense columns need to be separated in advance.

32

Chapter 3

Implementation of Interior Point

Methods for Quadratic

Optimization Problems

This chapter is devoted to implementation of interior point methods for solving

QO problems. Our implementation is based on and extends implementation of

the McIPM software package for linear optimization by X. Zhu [35, 36]. As QO

is a generalization of LO, many implementation features are very similar. The

crucial difference is in the system of Newton equations which is solved at each

IPM iteration. As we have seen in the previous chapter, it is more beneficial to

solve the augmented system instead of normal equations in QO for most cases.

The complication is that the augmented system is indefinite and, consequently,

we cannot use Cholesky factorization. Instead, we employ LDLT factorization

of the system using a sparse numerical linear algebra package. We describe the

complete sequence of steps for solving a QO problem and provide the algorithm

pointing out the differences of the QO case from the linear one.

We start Section 3.1 by describing the general interface of the McIPM pack-

age, which is now capable to solve both linear and quadratic problems. Next,

we describe the input formats for the solver. The standard input format for

33


quadratic problems is the so-called QPS or extended MPS format. We modified

the MPS file reader developed for linear problems by X. Huang [15] and added

to it the ability to read QPS files. We devote a section to basic preprocessing

and postprocessing techniques for QO and emphasize their differences from the

LO case. Section 3.2 describes the general structure of the McIPM package.

Section 3.3 is a brief description of sparse linear system solvers and contains

the information how the Newton system is solved by using those packages. Sec-

tion 3.4 describes the implemented algorithm including the predictor-corrector

strategy, self-regular search directions and stopping criteria. Finally, in Sec-

tion 3.5 we present numerical results and comparison of our quadratic solver

with the existing solvers such as BPMPD, MOSEK and CPLEX.

3.1 General Interface of the McIPM Package

The McIPM package, developed by X. Zhu [35], is a Matlab based software for

solving LO problems by using IPMs. We augmented it by adding the ability to

solve QO problems as well as extended its interface for user convenience. The

package is implemented in Matlab with C subroutines for reading input files and

solving the sparse linear systems. The input format for McIPM has the following

form:min cT x + 1

2xT Qx

s.t. Ax = b,l ≤ x ≤ u,

(3.1.1)

According to Matlab conventions a user can call the package from Matlab

by the following command

[x,y,z,objp,exitflag] = mcipm(A,b,c,Q,l,u,BIG,NAME,options)

where all input arguments except for A, b and c are optional. If matrix Q is

34


not provided, the problem is treated as linear. If no lower bounds l and/or

upper bounds u are provided, then the lower bound is set to zero for all variables

and the upper bound is set to +∞. Optional parameter BIG represents a large

number that is used as the upper bound for the variables without upper bounds,

the default value is BIG=1e32. NAME is optional problem name. The description

of all options is given in Appendix A. The output of the package is a primal

optimal solution vector x, a dual optimal solution (y,z), the optimal function

value objp and the indicator exitflag that gives us the solution status of the

problem (optimal, infeasible, etc., the reader may consult Appendix A for the

list of possible solution statuses).

Alternatively, the package may be called with only one input parameter

and options as follows

[x,y,z,objp,exitflag] = mcipm(pname, options)

Here it is assumed that the problem inputs are contained in the input file pname.

The McIPM package accepts input files in two formats:

• Matlab MAT file;

• MPS/QPS file.

McIPM automatically searches for the pname file in the matdir and mpsdir

directories inside the McIPM directory or in a user-specified directory.

3.1.1 Reading QO Data Files

The standard input format for QO problems is QPS. It stands for quadratic

programming system and is an extension of the MPS format of LO problems.

The MPS file reader is already implemented in C programming language by

X. Huang [15]. For the purpose of this thesis, we extend the MPS reader to

35


the QPS reader and add several useful features to it. We start by presenting an

example of the QPS file and briefly describing the MPS file format and continue

with pointing out the extension of the format to QPS.

The sample QPS file below represents the following QO problem:

min 4 + 1.5x1 + 2x2 + 12(8x2

1 + 2x1x2 + 2x2x1 + 10x22)

s.t. 2x1 + x2 ≥ 2−x1 + 2x2 ≤ 60 ≤ x1 ≤ 20, x2 ≥ 0.

(3.1.2)

A Sample QPS File

NAME QP example

ROWS

N obj

G r1

L r2

COLUMNS

c1 r1 2.0 r2 -1.0

c1 obj 1.5

c2 r1 1.0 r2 2.0

c2 obj 2.0

RHS

rhs1 obj -4.0

rhs1 r1 2.0 r2 6.0

BOUNDS

UP bnd1 c1 20.0

QUADOBJ

c1 c1 8.0

c1 c2 2.0

c2 c2 10.0

ENDATA

36


The MPS file is a text file with the following sections: NAME, ROWS,

COLUMNS, RHS, RANGES, BOUNDS, ENDATA. The input records inside

each section are divided into fields with fixed positions and have the following

format:

Field 1 Field 2 Field 3 Field 4 Field 5 Field 6Columns 2-3 5-12 15-22 25-36 40-47 50-61Contents Indicator Name Name Value Name Value

Indicator is the record type (e.g., indicator UB in the BOUNDS section means

”upper bound” on the variable). Name in fields 2, 3 and 5 represents row and

column names with the corresponding values in fields 4 and 6. The reader may

consult [15] and [23] for more comprehensive description of the records and fields.

The NAME section specifies the problem name and the ENDATA section

ends the file. Brief descriptions of the all other sections are provided below:

• NAME

• ROWS: the ROWS section lists the row names of matrix A, one per line, with

the name preceded by its type. Type L means ≤, E means =, a G means

≥, and an N means not constrained (e.g., the objective row);

• COLUMNS: the COLUMNS section lists nonzero elements of the matrix A,

preceded by the column name and row name in which it appears. Column

names correspond to variables x;

• RHS: the RHS section lists nonzero right-hand side elements of the matrix

A and the objective function preceded by the name of the row in which it

appears. One must also give a column name to the right-hand side vector;

• RANGES: an optional RANGES section allows one to supply the ranges of

rows – upper and lower limits on constraints. One must give a column

name for the range vector;

• BOUNDS: an optional BOUNDS section allows one to supply upper and lower

bounds on variables. One must give a column name for the bounds vector.

• ENDATA

37


In the QPS files, the new section QUADOBJ is added after the BOUNDS

section. The QUADOBJ section specifies the symmetric matrix Q, and, conse-

quently, only its lower triangular part is given. The elements of the matrix Q

appear in columnwise order, the same way as elements of the matrix A. The

0.5 multiplier of the quadratic part is implied and so the double of the diagonal

elements must be entered. The relative order of columns in QUADOBJ must be

the same as in the COLUMNS section.

Our QPS reader is able to read the QUADOBJ section of files. In addition,

it returns the constant in the objective function (equals to −4 in the example

above), which is given as an RHS value. Furthermore, the QPS reader converts

the problem to the standard form by introducing slack variables, splitting free

variables to two nonnegative variables and shifting the lower bounds to zero.

An alternative way to input the QO problem is to read it from a Matlab

MAT file. The MAT file should contain all the problem data and resembles

McIPM command line arguments: A,b,c,Q,l,u,BIG,NAME. Problem name NAME

and BIG constant are optional as well as Q, l and u. The constant in the objective

function CONSTOBJ can be optionally included. The problem in the MAT file as

well as in McIPM command line needs to be specified in the form (3.1.1). Data

from a QPS file is transformed to the form (3.1.1) by the MPS/QPS file reader.

3.1.2 Preprocessing and Postprocessing in Quadratic

Optimization

In this subsection we describe basic preprocessing and postprocessing techniques

for QO problems as well as their differences from the LO case (see [14] for com-

plete description of basic and advanced QO preprocessing techniques and [15]

for implementation of LO preprocessing). As LO preprocessing techniques deal

38


primarily with the problem constraints, most of them can be easily extended to

QO with the only difference that appropriate changes in the matrix Q should be

made. Postprocessing allows to get the solution vectors in terms of the original

problem. We proceed with the description of the basic preprocessing techniques

used in our implementation.

• Comparing the lower and the upper bounds of variables: If li > ui for some

variable xi, infeasibility is reported.

• Deleting fixed variables: Fixed variable li ≤ xi ≤ ui is a variable for which

li = ui. These variables are deleted from the problem formulation. Conse-

quently, vectors c and b are adjusted to

c = c + Q·jxj,

b = b − A·jxj,

where j is an index set of the fixed variables. Afterwards, columns j are

deleted from the matrix A, rows and columns j are deleted from Q, and

the j-th elements are deleted from the vectors c, b, l and u.

• Deleting zero rows: Rows of A containing exclusively zero entries are

deleted. When for such row the corresponding element of vector b is non-

zero, problem infeasibility is reported.

• Deleting zero columns: If in a LO problem a zero column appears in the i-

th position of the matrix A, the value of the corresponding variable xi is set

to li or ui depending on the sign of ci. This technique cannot be applied

for QO problems in general. It can be applied only if the i-th columns

(rows) of the matrix Q are zero rows or contain diagonal elements only. In

39


addition, if for some zero column i, ui = +∞ and ci < 0, the LO problem

is unbounded. Similarly, the QO problem is unbounded if in addition i-th

column (and row) of Q is zero.

• Solve singleton rows: Variables corresponding to singleton rows in A are

solved. Infeasibility can be detected if more than one singleton exist in

some column of A. Appropriate changes in A, Q, c, b, l and u are made to

accommodate for solved variables.

• Shift nonzero lower bounds: Non-zero lower bounds are shifted to zero by

transformation xi = xi − li. Vectors b, c and u are adjusted to b = b − Al,

c = c + Ql, and u = u − l.

In addition to preprocessing techniques described above, we scale the prob-

lem for better numerical properties. Geometric scaling [22] of the coefficients of

the constraint matrix A in order to reduce their spread improves the numeri-

cal properties of the problem to be solved in 95% of cases. Scaling is repeated

twice on the matrix A – two vectors handle the resulting row and column scaling

factors. The purpose of the scaling is to replace the original matrix A by the

matrix A = RAC, where R and C are row and column diagonal scaling matrices

respectively, in hope for better numerical properties of the matrix A.

Scaling is performed for badly scaled matrices only, which means that the

measure of the spread – ratio of the maximum to the minimum of the absolute

values of matrix elements – is large (greater than 104 in our implementation).

Scaling factors for the columns of A are the reciprocals of the square root of the

max norm (maximum of the absolute values) of each column. After the column

scaling for the constraint matrix has been applied, we apply row scaling similarly.

40


McIPM

LO QO SOCO

McSML

solvers/command/ solvers/ipm/

LDL

Ng & Peyton Fortran linear

solver

Sparse Linear Solvers

solvers/lib/

mpsdir/

matdir/

Input Data

InterfaceRead problem,

preprocess, postprocess

Qua

drat

ic op

t.(a

ugm

ente

d sy

stem

)

Line

ar o

pt.

(nor

mal

eq u

atio

ns)

IPM solver subroutines

MPS/QPS files

MAT files

mcipm arguments

Figure 3.1: General Structure of the McIPM Package

Postprocessing, just as in the case of LO, for QO consists of making back-

ward changes in order to recover the original vector x. The operations are per-

formed backwards from the ones described above. After that the original value

for the objective function is computed.

3.2 Structure of the McIPM Package

The McIPM package is a software package developed by a team of people at

Advanced Optimization Laboratory of McMaster University. Currently it allows

solving linear, quadratic and second-order conic optimization problems. Our

contribution to it is extending the code to QO problems.

The general structure of the McIPM package is depicted in Figure 3.1.

The general structure of the quadratic solver is shown in Figure 3.2.

41


Figure 3.2: General Structure of the McIPM Quadratic Solver

42


3.3 Solving the Newton System

In this section, we describe how the linear systems given in (2.5.3) are solved.

Solving such systems is the most time consuming process at each iteration. To

solve a linear system efficiently we use sparse linear algebra packages. Princi-

pally, any sparse linear algebra package can be hooked into McIPM. The speed

and solution precision are crucial. It is essential that such sparse linear alge-

bra packages are implemented in an efficient way. That is why all sparse linear

algebra packages described below are implemented in C language.

The requirements to sparse linear algebra packages from the IPM point of

view are the following:

• efficient ordering for reducing fill-in during factorization. Ordering should

be performed once and with taking sparsity and structure of the augmented

system into account;

• symbolic factorization of a matrix, that is done once at the beginning;

• numeric matrix factorization is done once at each iteration and it uses

previously completed symbolic factorization and ordering;

• solving the system of equations with the given factorization (backsolve).

3.3.1 Sparse Linear Algebra Package McSML

The McSML (McMaster Sparse Matrix Library for IPMs) package (current ver-

sion is 1.2) is a sparse numerical linear algebra package developed by T. Seol.

McSML can solve normal equations systems (NE) and augmented systems (AS)

43


frequently arising in IPMs of the following general forms

(NE) AEAT v = rhs,

(AS)

(

−E AT

A 0

)

v = rhs.

Every McSML function is coded in C language and wrapped to a MEX-file for

interfacing Matlab. All the functions can be classified into two groups:

1. Normal equations system solver:

• sf mmd ne() – minimum degree ordering for normal equations system;

• sf scc ne() – symbolic factorization and building supernode struc-

tures;

• nf aat() – normal equations builder;

• nf scc ne() – numerical Cholesky factorization;

• nf sub ne() – solving equations for provided numerical factorization

and refinement;

• nf error ne() – checking solution error.

2. Augmented system solver:

• nf mmd as() – minimum degree ordering for augmented system re-

garding dense columns in A;

• nf scc as() – symbolic factorization and building supernode struc-

tures;

• nf aug() – augmented system builder;

• nf scc as() – numeric LDLT factorization with a dynamic regular-

ization;

• nf sub as() – backsolve and refinement;

• nf error as() – checking solution error.

44


The McSML package has a number of features specific to solving augmented

systems. It uses minimum (external) degree ordering with multiple elimination

in order to reduce fill-in. To make the (AS) matrix quasi-definite and apply Van-

derbei’s technique [29] the dynamic regularization is implemented. A symmetric

matrix is quasi-definite if it has the form:(

−E AT

A K

)

,

where E and K are positive definite matrices and A has full row rank. As the

matrix (2.5.4) is not quasi-definite, the following regularization is applied(


A 0

)

+

(

−RpI 00 RdI

)

, (3.3.1)

which makes both parts more positive definite. The regularization parameters

Rp and Rd are kept small (10−8 − 10−9) to get the reliable solution for non-

regularized matrix. Dynamic regularization applies only to very small diagonal

elements. In addition to transforming the matrix to quasi-definite form, regular-

ization improves stability of the linear algebra.

Iterative refinement of the solution is automatically performed by McSML

when the solution error is greater than 10−9. A preconditioned Conjugate Gra-

dient method is applied to refine the solution. McSML assumes that every input

matrix is sparse and every input vector is full.

Below we show the sequence of McSML function calls for solving an aug-

mented system.

info_out=1; % output level

AT=A’; % McSML requires matrix AT

(P,Pinv)=sf_mmd_as(A,E,info_out); line 3

(L,LNODE)=sf_scc_as(A,AT,E,P,info_out); line 4

D=nf_aug(A,AT,E,P,L,LNODE,0,0,info_out); line 5

45


R=nf_scc_as(L,LNODE,D,P,Rp,Rd,info_out); line 6

x=nf_sub_as(A,AT,E,L,LNODE,D,P,rhs,R,info_out); line 7

Ordering is performed in line 3, the permutation vector P and its inverse

P inv are returned. Symbolic factorization is shown in line 4. Ordering and

symbolic factorization are performed only once at the beginning of an IPM algo-

rithm. Numeric factorization, which is done at every IPM iteration, is performed

in lines 5-6, here Rp and Rd are the regularization parameters. Finally, line 7

shows how to get the solution x of the linear system with the right-hand side

vector rhs. This step is performed at least twice at each iteration.

3.3.2 Sparse Linear Algebra Package LDL

The LDL software package written by T. Davis [6] is a set of concise routines

for performing factorization of symmetric matrices. It is applicable to positive

definite and indefinite sparse matrices. The package is written in C and has

Matlab interface which has been modified for our needs.

In contrast to the McSML package, which was designed to be used in IPMs,

LDL does not provide all subroutines used for solving linear systems. That is why

we use the external approximate minimum degree ordering package (AMD) to

reduce the fill-in while solving a linear system. The sequence of steps for solving

augmented systems, i.e., ordering, symbolic factorization, numeric factorization

and backsolve is the same as for the McSML package. The input matrix should

be made quasi-definite prior to factorization by adding the regularization terms

as in (3.3.1).

The sequence of commands for solving the system with LDL is:

Permut = amd([-E A’; A speye(m,m)]); line 1

[Lp,Lnz,Parent,Pinv] = ldlsymb([-E A’; A speye(m,m)], Permut); line 2

[L,D] = ldlnum([-E-regeps*speye(m) A’; A regeps*speye(m))],...

Permut, Lp, Parent, Lnz, Pinv); line 3

46


x = ldlsolve(L, D, Permut, rhs); line 4

3.4 Computational Algorithm for QO

3.4.1 Predictor-Corrector Strategy

Mehrotra’s predictor-corrector approach is known to be the most efficient way to

implement IPMs as it allows to reduce the total number of iterations by perform-

ing a second-order correction to the pure Newton direction. This actually means

that a number of steps are made within each iteration. Making several steps

within one iteration is much cheaper as we are solving the system of equations

with the same left-hand side and just changing the right-hand side vector. So,

the same factorization is used and a number of backsolves is performed.

The first step of the approach is to make a predictor or so-called affine-

scaling step. The predictor direction is found by solving the Newton system

(2.4.7) for µ = 0. As a result, the direction (∆xa, ∆ya, ∆za, ∆wa, ∆sa, ∆τa, ∆κa,

∆νa) is produced. The complementarity gap is reduced by the factor (α − 1) at

the predictor step, where α is the maximum acceptable step length from (2.4.17).

The second step is computing the corrector direction for the chosen cen-

tering parameter µ suggested by Mehrotra:

µ =

(

ga

g

)2ga

n + mf + 1,

where g is the actual complementarity gap and ga is the predicted complemen-

tarity gap

ga = (x + αa∆xa)T (z + αa∆za) + (s + αa∆sa)

T (w + αa∆wa)

+(τ + αa∆τa)T (κ + αa∆κa) = (1 − αa)(x

T z + sT w + τκ).

In the corrector step instead of neglecting the second-order terms ∆x∆z, ∆w∆s

and ∆τ∆κ in the Newton system, they are replaced by their approximate values

∆xa∆za, ∆wa∆sa and ∆τa∆κa computed at the predictor step. Consequently,

47


the left-hand side of the Newton system (2.4.7) solved at the corrector step is

unchanged and the coefficients on the right-hand side become

rsw = µe − Sw − ∆Sa∆wa,rxz = µe − Xz − ∆Xa∆za,rτκ = µe − τκ − ∆τa∆κa.

As a result, we solve two linear systems instead of one at each IPM iteration.

However, the matrix is factored only once that leads to much lower computational

costs.

3.4.2 Self-Regular Functions and Search Directions

Significant efficiency and complexity improvement in our implementation of IPM

comes from using self-regular proximity functions and search directions. We have

already seen a proximity function Ψ(xz, µ) in the prototype IPM algorithm in

Chapter 2. To make a quick tour into a new method for finding search direc-

tions in IPM based on self-regular proximity functions introduced in [25], let

us consider a function ψ(t) ∈ C2 : (0,∞) → IR+. Observing that the last

set of equations defining the central path (2.2.2) can be written equivalently as

ψ(xz) = ψ(µe) or ψ(xzµ

) = ψ(e) and applying Newton’s method to such trans-

formed central path equations, we get

A△x = 0,AT△y + △z − Q△x = 0,

xµψ′(xz

µ)△z + z

µψ′(xz

µ)△x = ψ(e) − ψ(xz

µ).

(3.4.1)

Introducing a new variable v =√

xzµ

and the notations dx = v△x

xand dz = v△z

z

we get

µv(dx + ds) = x△z + z△x

and the linear system (3.4.1) can be written as

Adx = 0,AT△y + dz − Qdx = 0,

dx + dz = dv,(3.4.2)

48


where A = A diag(xv), Q = Q diag(x

v) and

dv =ψ(e) − ψ(v2)

vψ′(v2).

If ψ(t) = t, we have dv = v−1 − v that corresponds to the standard primal-

dual IPM. We can observe that Newton steps increase the components vi < 1 as

dvi≥ 0 and decrease the components vi ≥ 1 as dvi

≤ 0 with iterations.

The idea behind the new search direction is that we can increase more the

small components of v and decrease more the large components of v by making

dv = v−q − vp instead of using the standard direction dv = v−1 − v. The class

of self-regular functions ψ(t), that are strongly convex functions, have global

minimum zero at the point t = 1 (i.e., ψ(1) = ψ′(1) = 0) and satisfy some other

convexity conditions is described in [25]. The following subclass of self-regular

functions gives the value dv = v−q − vp:

Γp,q(t) =tp+1 − 1

p + 1+

t1−q − 1

q − 1, p ≥ 1, q > 1. (3.4.3)

Note that for ψ(t) = Γp,q(t) the derivative is

ψ′(t) = tp − t−q.

In IRn we define a self-regular proximity function as

Ψ(v) =n

∑

i=1

ψ(vi).

Taking the gradient of Ψ(v) we get

∇Ψ(v) = ∇(

n∑

i=1

ψ(vi)

)

= ψ′(v) = (ψ′(v1), ψ′(v2), . . . , ψ

′(vn))T .

For the self-regular function Ψ(v) =∑n

i=1 Γp,q(vi), we also have

∇Ψ(v) = ∇(

n∑

i=1

Γp,q(vi)

)

= −(v−q − vp).

49


Consequently, we can rewrite system (3.4.1) as

Adx = 0,AT△y + dz − Qdx = 0,

dx + dz = −∇Ψ(v),(3.4.4)

and the corresponding Newton system becomes

A△x = 0,AT△y + △z − Q△x = 0,

x△z + z△x = −µv∇Ψ(v).(3.4.5)

The use of such self-regular search direction allows improving the iteration com-

plexity of large-update IPMs to O(√

n log n log nǫ) [25].

Putting self-regular search directions into the framework of the homoge-

neous embedding model, we can compute the corrector step by solving the New-

ton system (2.4.7) with the following coefficients at the right-hand side

rsw = −µvsw∇Ψ(vsw) − ∆Sa∆wa,rxz = −µvxz∇Ψ(vxz) − ∆Xa∆za,rτκ = −µvτκ∇Ψ(vτκ) − ∆τa∆κa,

where

vsw =

√

sw

µ, vxz =

√

xz

µ, vτκ =

√

τκ

µ.

In the implementation we use the self-regular proximity function Γp,q(v)

with p = 1 and initial value of q = 1. The value of q is increased dynamically to

3 if at some iteration the iterate gets too close to the boundary of the feasible

region. The reader may consult [35, 36] or [25] for more details about self-regular

proximity functions.

3.4.3 Stopping Criteria

In this section we describe the stopping criteria of the algorithm as well as crite-

ria for reporting the solution status (optimal, infeasible, numerical difficulties).

In the description below we use the modifications of the criteria developed by

X. Zhu [35].

50


The following measures are defined as the parameters of McIPM:

tol emb – duality gap tolerance for the embedding problem;

tol ori – duality gap tolerance for the original problem;

tol kt – accuracy parameter for τ/κ; in the solution, one of the values, either τ

or κ should be zero and the other one should be positive.

tol end – the tolerance for the parameter µ.

Define the relative errors for the k-th iteration of the algorithm

fea p = ||Axk−τkb||+||Fxk+sk−uτk||τk+||xk||

,

fea d = ||AT y+zk+Qxk−cτk−F T wk||τk+||zk||

,

gap ori =||cT xk−bT yk+uT wk+xkT

Qxk

τk ||

τk+||bT yk−uT wk− 1

2

xkTQxk

τk ||,

and

error = max{fea p, fea p, gap ori}. (3.4.6)

The algorithm terminates if:

• errork < tol ori and κk < τ k tol kt. The solution

(x∗, z∗, y∗, w∗, s∗) =

(

xk

τ k,zk

τ k,yk

τ k,wk

τ k,sk

τ k

)

is feasible and firm optimal in this case;

• µk < tol end. The solution

(x∗, z∗, y∗, w∗, s∗) =

(

xk

τ k,zk

τ k,yk

τ k,wk

τ k,sk

τ k

)

is feasible and is reported to be optimal up to a certain precision;

• µk < tol emb and τ k < κk tol kt the problem is infeasible (see Sec-

tion 2.3.2 for the description of infeasibility detection);

• µk < tol emb, τ k > κk, but τ k > κk tol kt the problem is non-firm

infeasible and we move to the next iteration k + 1. If the solution is still

51


non-firm infeasible after a couple of iterations as for numerical reasons no

more precise solution can be reached, infeasibility is reported up to a certain

precision.

3.5 Numerical Results

In this section our computational results for the McIPM-QO solver are presented.

All computations are performed on an IBM RS/6000 workstation with four pro-

cessors and 8Gb RAM. The operating system is IBM AIX 4.3.

The test problems used for our numerical testing are coming from the Maros

and Meszaros QP test set [18] which consist of 138 problems. The sizes of

problems vary from a couple of variables to around 300, 000 variables. You can

find the description of the problems in the testset, including original problem

dimensions and problem dimensions after McIPM preprocessing, in Appendix B.

The McIPM parameter settings used for testing are the following:

% PARAMETERS - Algorithmic parameters for MCIPM

p = 1;

q = 1;

regeps = 1.e-9; % regularization

MAXITER = 150; % Maximum number of iteration

tol emb = 5.0e-17; % an accuracy parameter for mu

tol ori = 5.0e-09; % an accuracy parameter for gap ori

tol end = 1.e-15; % an accuracy parameter for nu

tol bad = 1.e-15;

tol kt = 1.e-3; % an accuracy parameter for tau/ka

damp = 0.995; % a variable damping parameter

bad step = 0.4;

worst step= 0.1;

MAXQ = 3;

The testing results are shown in Tables 3.1, 3.2 and 3.3. As there are no

52


standard confirmed solutions for the test set, we have solved all the problems with

CPLEX and MOSEK and also used the BPMPD solution reported in [22] for

comparison. In columns ’Diff.’ of Tables 3.1–3.3 we report the difference of the

optimal solutions found by McIPM with the solutions found by CPLEX, MOSEK

and BPMPD (ObjCPLEX − ObjMcIPM , ObjMOSEK − ObjMcIPM and

ObjBPMPD − ObjMcIPM). Furthermore, column ’Sign. digits’ shows

the number of significant digits in the McIPM solutions as compared with the

solutions found by CPLEX, MOSEK and BPMPD. We also report the number of

iterations ’Iter’, optimal function value ’Objective’ and CPU time. Column

’Resid.’ is McIPM error computed according to (3.4.6).

All the test were done using the LDL package for solving the augmented

system. As we see from the results, McIPM produces reliable solutions with

six or more significant digits for the majority of the problems. The problematic

problems are depicted in bold font. The imprecise solution on this set of problems

is due to precision of solving the Newton system with the default regularization

parameter 10−9. The two sparse linear packages – LDL and McSML – are not

always able to produce precise solutions due to numerical difficulties, that result

in less precise solutions produced by McIPM. We plan to use iterative refinement

of solution of the liner system produced by LDL to improve the current results.

It is worth mentioning that using different regularization, solving the linear

system without ordering and regularization with the LDL package (it works in

most cases as zero elements does not appear in the diagonal in the upper left

block E of the matrix) or employing McSML as sparse linear system solver

allow to get the solution precision of seven or more significant digits for all

the problems in the test set except for the largest problem boyd2 which has

four significant digits in the solution. Table 3.4 shows the solutions to bold-

font problematic problems in Tables 3.1–3.3 computed using McSML or LDL

without ordering and regularization of the augmented system. In general, almost

all packages experience troubles on the number of problems shown in Table 3.4.

One can notice that McIPM has lower number of iterations than CPLEX or

53


MOSEK on the majority of difficult problems. Solution precision on this set of

difficult problems is not very good especially for commercial solvers, furthermore,

MOSEK and BPMPD produce incorrect solutions for one problem each.

Comparison of the number of iterations used by CPLEX, MOSEK and

McIPM does not show a clear trend, but in the majority of the cases McIPM

beats the commercial packages confirming that the usage of self-regular proximity

functions and search directions can reduce the number of IPM iterations. In

the column ’#iter q>1’ we show the number of iterations when self-regular

parameter q was dynamically increased from 1 to 3. The solution time of McIPM

in most cases is more than the one of CPLEX and MOSEK. This is partially

explained by the fact that CPLEX and MOSEK are pure C packages while

McIPM is partially coded in Matlab. The second reason for the difference in

solution time is that all packages – CPLEX, MOSEK and BPMPD are utilizing

more advanced preprocessing techniques than McIPM. For example, BPMPD

reduces problem size for the problem qcapri from 271 × 373 to 148 × 214 while

McIPM only to 267 × 353. Similarly, MOSEK reduces the size of the largest

problem in the test set boyd2 from 186531 × 279785 to 119721 × 150580 while

McIPM leaves it unchanged.

We do not show all the results obtained when the McSML package is em-

ployed for solving the augmented systems as it generally exhibit slightly poorer

performance than LDL. Still, McIPM with the McSML sparse linear solver is

able to solve more than 90% of the test set problems.

In general, we may conclude that McIPM is competitive even with the

commercial packages. One of the sources for improving competitiveness is the

possibility of using more advanced sparse linear algebra packages, e.g., TAUCS,

which does not have separate symbolic and numeric LDLT factorization at the

moment but plans to have it in the future. Another possibility is to get Mc-

SML to the state of the art by implementing Bunch-Parlett or Bunch-Kaufman

techniques. The other option is porting McIPM to C language for improving its

speed.

54


Table 3.1: McIPM Performance on QO Test Set (I).

bor

PD

PM

PB

evit

cej

bO

retI

em

aN

.c

es

UP

Ce

vitc

ejb

Or

etI.

ce

s U

PC

evit

cej

bO

.di

se

Rr

etIe

vitc

ejb

OU

PC

.c

es

reti

#

1>

q.r

effiD

.n

giS

stigi

d.r

effiD

.n

giS

stigi

d.r

effiD

.n

giS

stigi

d

820

-E2

8.4

01

40-

E00

.5

01

30-

E0

1.

10

03

.5

16

0+E

815

711

478

6.

10

1-

E54

.6

860+

E8

114

786

.1

01

.4

60

+E3

257

114

786

.1

51

5.0

60

+E9

257

114

786

.1

1d

2gu

a

82

0-E

94.

311

00

+E0

0.0

01

40

-E

00

.5

045

.41

60+

E15

60

86

381

8.1

01-

E4

2.7

86

0+E

18

63

818

.1

09

.3

60

+E

156

086

381

8.1

59

4.

06

0+E

656

0863

81

8.

11

cd

2g

ua

92

0-

E79.

511

40

-E

00.3

01

40-E

00.

8-

36

8.4

160

+E3

047

431

89

4.

60

1-E

24.

761

60

+E8

431

89

4.

600

.5

60

+E6

047

431

894

.6

31

72

.3

60

+E5

937

431

894

.6

71

pqc

d2g

ua

820-

E19

.6

01

30-

E03

.1-

01

30

-E

03

.5-

48

7.4

160

+E

90

302

107

32.

69

0-E

08.

17

16

0+

E12

107

32.

604

.5

60

+E6

920

210

732

.6

51

38

.3

60

+E6

52

02

107

32

.6

02

pq

d2

gua

96

0-

E03.

401

70

-E

03

.3

-0

180-

E00

.9

06

6.1

20+

E07

527

760

45.

50

1-E

82

.4

62

0+E

377

604

5.5

99

.0

20+

E73

52

77

604

5.5

430

.0

20

+E9

752

776

045

.5

1d

3gu

a

960-

E63

.1

01

70

-E08

.3-

11

80

-E

00

.6

01

7.

120

+E4

683

426

21

7.

70

1-E

76.

36

20+

E44

262

17.

70

0.

12

0+

E62

834

262

17.

74

70

.0

20

+E

078

342

621

7.7

1cd

3gu

a

96

0-

E80

.4

01

70-E

01.

90

17

0-E0

1.6

004

.1

20

+E

295

412

633

9.9

90

-E

52

.4

01

20

+E

512

633

9.9

59

.0

20+

E3

86

412

6339

.9

11

92

.0

20

+E3

564

126

339

.9

31

pqc

d3g

ua

96

0-E

65.

1-

970

-E0

7.7

01

70

-E0

8.2

-0

95

.1

20

+E

65

17

673

257

.6

90-

E95

.1

21

20

+E7

673

257

.6

79

.0

20

+E

332

767

325

7.

62

113

.0

20

+E8

217

673

257

.6

61

pq

d3g

ua

E9

5.

3-

01

20-

E0

4.

21

100

+E0

0.0

22

1.

701

70+

E14

691

25

37

1.6

-90

-E1

6.2

42

70+

E0

22

537

1.6

-24

.31

70

+E

716

912

537

1.6

-7

21

9.8

270

+E1

469

125

371

.6-

04

1d

yob

91

0-

E6

9.6-

10

0+E

69.

6-

10

0+E

69.

6-

43

17

1.9

522

10

+E

155

29

02

128

.2

50-

E69

.8

05

11

0+

E7

676

521.

23

9.

56

10+

E15

966

765

21.

204

76.

98

10+

E76

276

765

21.

23

52

dy

ob

100

+

E09

.3

11

01

-E0

0.4

-1

10

1-E

00.4

-1

94

.3

00+

E93

090

583

65.

4-

90

-E

05.

11

10

0+

E90

583

65.

4-

07

.1

00

+E

340

905

836

5.

4-

11

26

.0

00

+E3

409

058

365

.4-

21

050

-tn

oc

01

90-

.3

-11

01-

E00

.1-

11

01

-E0

0.1

-0

82

.91

00

+E7

868

793

446

.4-

01-

E74

.5

21

00

+E9

793

446

.4-

00

.1

100

+E8

868

793

446

.4-

21

66

.6

00

+E8

868

793

446

.4-

41

001

-tn

oc

980-

E31

90-E

23.

511

11-E

00.

2-

11

11-

E00

.2-

258

.02

10-

E86

42

37

255

9.1

90-

E6

8.1

31

10-

E3

37

255

9.1

01.

910

-E6

642

372

559

.1

21

45

.5

10

-E

664

237

255

9.1

21

10

1-t

noc

9

21

101

-E0

0.1-

11

01-

E00

.1

-0

48.

831

00+

E30

295

784

86.

4-

90

-E8

7.2

21

00+

E95

784

86

.4

-00.

08

00+

E40

295

784

86.

4-

21

81.

74

00

+E

40

29

578

486

.4

-3

10

02

-t

no

c9

80-

E30

.

-E5

0.

3-

11

01

-E

08.

1-

990

-E9

3.4

15

2.4

41

10

-E5

037

338

429

.1

90-

E19

.4

21

10

-E7

338

429.

10

0.

47

10-

E78

273

384

29.

121

70

.6

41

0-

E44

773

384

29.

12

11

02-

tno

c9

90

E9

3.3

11

01

-E

06.

1-

880

-E4

9.5

30

2.0

37

10

-E7

068

221

519

.1

01-

E80

.4

61

10

-E2

321

519

.1

00.

072

10

-E1

958

221

519

.1

31

86.

751

10

-E1

554

321

519

.1

61

003

-tn

oc

880-

2.

26

20

+E5

2.2

620

+E4

2.2

21

18.

894

28

0+E

545

754

078

0.1

50

-E2

3.3

95

80

+E

084

078

0.1

00

.00

61

80+E

110

084

078

0.1

11

70

.1

192

80

+E0

799

740

78

0.

191

l_1

pqx

vc

620

+E5

820-

E81

.3

93

0-

E04

.2

01

40

-E0

0.

10

60

.2

60

+E

286

511

578

0.1

90-

E1

5.1

11

60

+E

611

578

0.1

09.

260

+E6

075

115

78

0.

10

10

7.8

60

+E

386

511

578

0.1

11

m_

1pq

xvc

840

-E0

2.1

-11

60

-E

00

.1

11

60

-E

00.

1-

062

.0

40

+E

021

817

095

1.1

11

-E8

8.5

940

+E8

170

95

1.1

26

.0

40

+E

121

817

0951

.1

84

0.

040

+E

91

181

709

51.

11

1s

_1p

qxv

c

90

.3

-9

20

-E

09.

801

20-E

00.

3-

02

9.

644

70

+E

903

854

248

1.8

90

-E1

9.1

21

70

+E

854

248

1.8

00

.04

27

0+E

8938

54

24

81

.8

11

87

.80

67

0+

E97

285

424

81.

82

1l

_2p

qxv

c9

10

-E

93

0-E

60.

1-

01

40

-E

06.

311

50-

E00

.4-

07

5.1

50+

E60

134

55

10

2.8

01

-E4

2.2

11

50+

E34

551

02.

803

.1

50+

E24

134

551

02.

801

44

.3

50+

E20

134

551

02

.8

9m

_2

pqx

vc

95

0-

E1

2.2

950

-E3

1.2

11

70

-E0

0.6

-0

62

.0

30

+E9

774

049

021

.8

90-

E77

.1

930

+E5

049

021

.8

26

.0

30

+E2

994

049

021

.8

92

0.0

30

+E3

774

04

90

21.

811

s_2

pq

xv

c

74.

4-

710

+E0

0.2

910

-E0

0.4

031

.93

780

+E7

440

111

75

1.

16

0-E

72.

161

80

+E0

111

751

.1

00

.00

84

80

+E

054

211

175

1.1

01

74.

899

28

0+E

784

011

175

1.1

31

l_

3pq

xvc

800

+E

82

0-

E61

.4-

01

40

-E0

0.4

82

0-E

45

.5-

05

7.2

60+

E61

478

282

63.

10

1-E

71.

341

60+

E78

282

63.

106

.7

60

+E

02

47

828

26

3.

111

15.

51

60+

E26

868

282

63.

10

1m

_3p

qxv

c

84

0-E

22.

2-

950

-E0

9.1

-9

50

-E0

0.2

-0

52.

040

+E2

222

343

491

.1

90-E

43.

19

40+

E23

434

91.

146

.0

40

+E

30

223

434

91.

101

50

.0

40+E

20

223

434

91.

12

1s

_3p

qxv

c

01

90

-E9

8.2

11

01

-E

00.3

-21

00

+E

00.

00

81

.0

10

-E

117

126

900

7.3

11-

E9

6.

36

10

-E

226

900

7.3

36

.0

10

-E

186

126

900

7.3

41

0.

01

0-E

11

71

269

007

.3

51

olk

pd

960

-E5

0.1

-11

80-

E00

.2

-9

70

-E0

2.5

-0

54

.6

20

+E

501

842

625

3.2

11-

E8

4.4

72

0+E

842

625

3.2

01

.3

20

+E

301

842

625

3.2

50

3.

020

+E

35

084

262

53.

21

3c

ot

d

11

01

-E5

2.1

990

-E

59

.3

11

01

-E2

4.1

-0

94.

020

-E5

785

692

105

.3

01-

E99

.4

41

20

-E6

692

105

.3

36

.0

20

-E

428

96

92

105

.3

31

60

.0

20

-E3

375

692

105

.3

61

1l

aud

11

01-

E87

.1-

11

11

-E0

4.5

-11

11-

E05

.5-

074

.0

20-

E87

167

633

73.

39

0-E

24.

22

120

-E

67

633

73

.3

56.

020

-E4

216

763

373

.3

21

60

.0

20

-E

321

676

337

3.3

41

2l

au

d

99

0-E

48.

20

101

-E0

6.2

01

01

-E

09

.2

-0

16

.0

10

-E

617

38

55

753

.1

01-

E11

.8

31

10

-E

485

575

3.1

86.

01

0-E

247

385

575

3.1

31

11.

01

0-E

786

38

55

753

.1

71

3l

au

d

01

90-

E01

.2-

11

01-

E04

.5

11

01-

E00

.3-

09

3.

010

-E

01

248

090

64

.7

90-

E43

.3

21

10-

E48

090

64.

766

.0

10

-E

462

480

90

64

.7

21

40.

01

0-E

08

148

090

64.

74

14

la

ud

95

0-

E40

.3-

01

70

-E0

0.9

-0

17

0-E

00.9

-5

36

.0

30

+E

403

805

255

1.6

90-

E9

7.4

42

30+

E80

525

51.

64

6.0

30+

E59

280

525

51.

681

00

.0

30

+E

59

28

052

551

.6

61

1cla

ud

96

0-E

03.

711

70

-E0

0.1

11

00

+E0

0.0

40

5.

030

+E7

296

70

315

5.

30

1-

E09

.5

91

30

+E7

703

155

.3

26

.0

30

+E

829

67

03

155

.3

51

00

.0

30

+E7

296

703

155

.3

51

2cl

aud

960

-E0

2.3

01

80-

E00

.4

11

80

-E0

0.2-

062.

020

+E0

862

32

32

72.

49

0-E

39.

39

20+

E33

23

27

2.4

26.

020

+E4

862

323

272

.4

90

0.0

20

+E

876

232

327

2.4

21

5c

lau

d

94

0-

E76

.1

11

00

+E

00.

011

00

+E0

0.0

237

.0

40

+E

338

853

90

38

.1

90-

E15

.6

02

40

+E

953

903

8.1

66.

04

0+E

338

853

903

8.1

51

00.

04

0+E

338

85

39

038

.1

31

8c

lau

d

64

0-

E08

.4-

64

0-E

28.

4-

07

9.

102

20

+E

640

592

481

4.1

-90-

E9

9.1

31

20+

E34

348

14.

1-

00

.26

20

+E

312

343

481

4.1

-5

1r

orr

e d

aer

SP

Ma

tad

xe

01

90-

E96

.3-

21

11-

E00

.6

21

11-

E00

.8

060

.0

10

-E9

639

637

17

2.

91

1-E

40.

45

10-

E96

371

72

.9

55

.0

10-

E57

396

371

72.

94

00

.0

10

-E7

739

637

172

.9

482

shn

eg

01

01

-E

20.

411

11

-E

55

.2

01

01

-E5

3.4

-0

08

.0

40-

E70

839

472

48.

19

0-

E1

7.4

21

40

-E

435

724

8.1

05.

14

0-

E06

36

94

724

8.1

51

80.

04

0-E

633

054

724

8.1

02

2p

qdl

uog

88

0-E

97.

26

50

-E5

9.1

880

-E9

7.

20

96

.0

00

+E1

279

387

260

.2

90-

E24

.3

01

00

+E0

487

26

0.

201

.1

00

+E

185

430

826

0.2

87

0.

00

0+

E00

004

872

60.

21

13

pqd

luo

g

01

70

-E

01.

401

70

-E

00.

40

170

-E0

1.4

061

.0

20

+E9

594

402

846

.6

90-

E27

.4

01

20

+E

540

284

6.6

67

.0

20

+E

999

440

284

6.6

01

00.

02

0+E0

00

540

284

6.

64

18

11s

h

86

0-

E22

.1

-8

60

-E

22.1

-8

60

-E2

2.1

-0

61

.0

10

+E

187

899

959

9.9

-90-

E6

4.2

21

10+

E00

006

99.

9-

75

.0

10

+E

000

000

06

99

.9-

01

10.

010

+E

000

000

06

99.

9-

22

12

sh

01

70

-E

67.

501

50

-E6

8.1

990

-E7

6.2

07

1.

09

0-E

232

441

937

6.2

-80-

E9

8.2

81

70-

E50

70

13

7.5

95

.0

50-

E13

305

511

68.

141

00

.0

00

+E0

000

000

000

.0

81

86

2sh

880

-E2

8.1

-8

80-

E66

.1-

88

0-

E1

7.1

-0

60

.0

10-

E91

82

11

111

1.1

90-

E7

4.2

51

0-E

11

11

111

.1

85

.0

10-

E95

111

111

11.

15

00

.0

10

-E0

111

111

111

.1

22

53

sh

99

0-

E55

.2-

99

0-E

73.5

99

0-E5

5.2

-0

01

.0

10

-E

552

00

00

005

.2

90-

E06

.3

01

10

-E

000

000

5.2

95.

01

0-E

297

000

000

5.2

01

00.

01

0-E

000

000

000

5.2

62

dom

53s

h

61

41-

E96

.1

51

41-

E59

.1

41

41-

E06

.1

06

0.

04

1-

E55

511

278

95.

1-

01-E

06.

55

61

-E

248

718

8.8

75.

051

-E88

763

172

55.

34

00.0

00

+E

000

0000

00

0.0

01

15s

h

980

-E5

5.3

11

00+

E00

.0

11

11

-E0

0.3

-0

50.

00

0+E

546

574

66

23

.5

01-

E4

0.

25

00

+E6

74

66

23.

57

5.

000

+E5

465

746

623

.5

30

0.0

00

+E

546

574

662

3.5

325

sh

DP

MP

B .p

mo

CK

ES

OM

XE

LP

CK

ES

OM .

pm

oC

XE

LP

C .p

mo

CM

PIc

M

55


Table 3.2: McIPM Performance on QO Test Set (II).

bor

PD

PM

PB

evit

cej

bO

retI

em

aN

.c

es

UP

Ce

vitc

ejb

Or

etI.

ce

s U

PC

evit

cej

bO

.di

se

Rr

etIe

vitc

ejb

OU

PC

.c

es

reti

#

1>

q.r

effiD

.n

giS

stigi

d.r

effiD

.n

giS

stigi

d.r

effiD

.n

giS

stigi

d

88

0-

E2

4.

41

10

0+E0

0.0

11

00

+E

00

.0

011

.0

00

+E8

552

320

390

.4

90

-E

67.

18

00+

E33

203

90.

49

5.

000

+E

85

523

203

90.

46

00

.0

00

+E

855

232

039

0.4

02

35s

h

980

-E

48.1

-0

19

0-E0

1.4

11

01-E

00.

2-

070

.0

00+

E61

818

181

86.

4-

01-

E7

0.1

60

0+

E28

181

86.

4-

75

.0

00

+E5

771

818

186

.4-

51

0.

00

0+E

818

181

818

6.4

-7

167

sh

62

0+E

62.

201

20

-E0

8.1

11

00+

E0

0.

00

98.

370

+E3

783

644

28

4.

39

0-E

98.

251

70

+E0

964

28

4.

369

.0

70

+E1

983

644

284

.3

21

81

.0

70

+E3

783

644

284

.3

21

dom

-se

uh

66

0+E

62.

20

12

0+E

03.

1-

01

20+

E05

.2-

05

4.

311

+E

89

836

442

84.

37

0-E

62.

13

11

1+

E09

642

84.

389

.0

11

+E5

883

644

284

.3

21

52

.0

11

+E3

78

36

442

84

.3

81

si

ts

euh

01

90-

E70

.1-

21

11

-E00

.4

11

01-

E07

.1

45

5.2

10-

E70

149

797

57.

59

0-E

39.

27

11

0-

E49

797

57.

50

4.

110

-E1

114

979

757.

591

53.

71

10

-E4

214

979

757

.5

61

pis

k

82

0-

E01

.5

81

0-E

12.

19

30

-E

04.

3-

010

.2

60

+E

094

310

690

4.2

90

-E6

1.

21

16

0+E

410

690

4.2

01

.1

60+

E2

07

410

690

4.2

21

81.

06

0+E

654

310

690

4.2

31

re

sa

l

11

0+E

10.

11

10

+E

10.

11

10

+E1

0.1

00

0.5

11

0+

E43

770

820

06.

26

0-E

08.

25

11

0+

E20

422

16.

30

9.

610

+E3

999

142

216

.3

22

19

.1

10+E

00

12

04

221

6.3

32

1t

ew

sil

110+

E93

.2

11

0+E

93.

21

10

+E9

3.

20

42

.5

110

+E5

348

199

755

.2

60-

E59

.3

51

10

+E5

875

849

.4

01

.8

10

+E

360

108

584

9.4

72

43.

31

0+

E00

748

758

49.

41

40

1te

wsi

l

11

0+E

86.

11

10

+E8

6.1

110

+E8

6.1

090

.51

10+

E14

761

250

72.

35

0-E

61

.3

51

10+

E75

932

59.

40

9.

910

+E

69

508

932

59.

463

83

.2

10+

E00

175

932

59.

49

21

1te

wsi

l

130

+E3

5.1

130+

E35

.1

13

0+E

35.

10

60.

51

20

+E

593

69

05

960

.2

40-

E61

.8

51

30

+E

47

29

637.

10

0.8

30

+E

708

58

296

37.

17

201

.5

30

+E1

624

729

637

.1

46

21t

ews

il

20-

E09

.1-

42

0-E

09.

1-

42

0-E0

9.1

-0

25

.6

31

10

+E3

476

407

105

.2

50-

E91

.2

05

110

+E6

708

994

.2

09

.6

10

+E

35

497

08

99

4.2

22

51

.1

10

+E0

016

708

994

.2

31

2te

wsi

l4

10

+E3

7.7

-1

10

+E3

7.7

-1

10+E

37

.7-

663

.64

12

0+

E98

105

303

20.

11

0-E

05.

10

51

10+

E02

210

05.

20

8.

710

+E9

932

221

005

.2

32

58

.1

10

+E

000

022

100

5.2

22

3t

ew

sil

1

50+

E99

.1-

45

0+

E9

9.

1-

45

0+E

99.

1-

31

1.

15

15

0+E

319

999

878

9.1

00+

E9

9.1

05

11

0+E

211

000

5.2

00

.7

10+

E8

69

611

000

5.2

32

67.

11

0+E

090

211

000

5.2

12

4te

wsi

l4

80+E

02.

2-

780

+E0

2.2

-7

80

+E02

.2-

109

.45

180

+E12

71

738

89

1.

20

0+E

00.

205

110

+E3

52

43

05.

203

.7

10

+E

748

092

430

5.2

12

87.

11

0+

E00

035

243

05.

21

25

te

wsi

l7

10

+E3

5.3

-1

10+

E35

.3-

11

0+E

35.

3-

231.

831

10+

E32

17

14

262

0.6

20

-E9

2.3

051

10

+E

847

599

4.2

06.

61

0+

E72

365

759

94.

212

29

.1

10+

E0

06

747

599

4.

23

26t

ews

il

1 12

0+

E95.

41

20

+E

95.

41

20

+E9

5.4

05

0.5

11

0+

E06

239

108

00.

43

0-E

80.

15

12

0+

E98

048

89.

40

0.1

12

0+

E40

64

17

488

9.4

83

85.

120

+E0

809

804

889

.4

81

7te

ws

il

13

0+E

80.

71

20

+E3

5.6

120

+E3

5.6

084

.51

10+

E93

327

242

71.

63

0-E

79.

15

13

0+

E60

074

41.

70

5.

820

+E3

237

207

441

.7

92

65

.4

20

+E

009

500

744

1.7

85

8t

ew

sil

130

+E3

4.1

13

0+

E34

.1

13

0+

E34

.1

04

1.

51

20

+E

225

418

799

2.5

30-

E41

.3

51

30

+E

315

236

9.1

06.

93

0+E

649

415

236

9.1

23

17.

43

0+E

906

215

236

9.1

06

9te

wsi

l

960-

E04

.4

96

0-

E0

6.

3-

96

0-E

02.

4-

011

.0

30

+E

659

851

489

3.2

90

-E3

8.1

11

30

+E

951

489

3.2

16.

030

+E

02

985

14

893

.2

900

.0

30

+E

419

851

489

3.2

31

dhc

sto

l

960

-E3

0.3

960

-E

51.

21

10

0+

E00.

00

46

.0

20

+E3

034

457

825

.9-

11-

E42

.4

01

20

+E4

457

825

.9-

89

.0

20+

E88

044

578

25.

9-

96

3.

02

0+

E30

344

578

25.

9-

61

1p

qr

aso

m

85

0-E

17.

19

60

-E0

3.3

01

70-E

00.

4-

025

.0

30+

E17

112

847

95.

1-

90

-E7

2.1

93

0+

E1

284

79

5.

1-

88

.0

30

+E

83

11

284

795

.1-

981

.0

30

+E5

711

284

795

.1-

51

2pq

ras

om

91

0+

E03

.8-

920+

E17

.2-

01

4.

11

01

+E

380

385

980

2.

580-

E04

.2

11

01

+E

385

980

2.5

ro

rr

e l

ata

f29

.2

01

+E2

182

859

802

.5

73

02

le

wo

p

1-

E7

9.

60

10

1-

E85

.4

11

11-

E06

.3-

00

5.

02

0-E

796

569

21

05

.3-

11-

E24

.9

01

20

-E

569

210

5.3

-0

1.

120

-E9

325

692

105

.3

-0

17

0.0

20

-E

33

756

921

05

.3

-81

1l

am

irp

01

0 11-

E00

.1

11

11

-E0

6.7

-1

10

1-E

31.

1-

046

.0

20-

E01

067

633

73.

3-

01

-E

41.

18

20-

E67

633

73.

3-

03

.1

20

-E

680

676

337

3.3

-8

80

.0

20

-E

321

676

337

3.3

-7

12

la

mir

p21

990

-E4

3.4

-9

90-

E73

.3

01

90

-E0

3.1

-0

23.

11

0-E

665

385

575

3.1

-90-

E36

.1

91

0-

E48

557

53

.1

-08

.1

10

-E9

22

38

557

53.

1-

91

2.

01

0-

E69

638

557

53.

1-

81

3la

mir

p

90-

E28

.9

01

01

-E

08.

801

90-

E89

.1

-0

83

.1

10

-E2

893

809

064

.7-

90

-E

81.

19

10

-E3

809

064

.7-

07

.1

10-

E49

838

090

64.

7-

01

01

.0

10-

E08

148

090

64

.7

-2

14l

ami

rp

9

E4

6.

20

16

0-E

09.

1-

01

60-E

01

.3-

25

4.

03

0+E

462

80

52

551

.6-

01-

E39

.5

02

30

+E

805

255

1.6

-9

9.

030

+E3

828

052

551

.6-

02

10

.0

30

+E5

92

80

525

51.

6-

61

1c

la

mi

rp

95

0- -

E02

.2-

95

0-

E64.

1-

95

0-E

74.

1-

09

2.0

30+

E08

767

031

55.

3-

90-

E90

.4

61

30+

E77

031

55.

3-

00

.1

30

+E

629

670

315

5.3

-9

110

.0

30

+E

729

670

315

5.3

-8

12c

lam

irp

95

0

85

.3

-01

70-

E03.

3-

01

70

-E0

6.3

-0

22.

02

0+E

246

232

327

2.4

-01

-E3

8.8

11

20+

E33

232

72.

4-

98

.0

20

+E

57

62

323

27

2.

4-

01

10.

020

+E

87

623

232

72.

4-

71

5c

la

mir

p9

60

-E E

42.

2-

01

50

-E

02.

101

50-

E02

.1-

04

3.0

40

+E6

779

249

038

.1-

01-

E5

9.

63

140

+E0

349

038

.1-

88

.0

40+

E46

792

490

38.

1-

11

30

.0

40+

E88

792

490

38.

1-

41

8c

la

mir

p8

40

-

91

0-

E40

.1

92

0-

E04

.1

11

30-

E00

.1-

089

.11

70

+E

698

744

447

3.1

01-

E6

3.4

82

70+

E84

444

73.

10

5.

97

0+

E01

974

444

73.

162

36

.6

70+E

598

744

44

73

.1

91

74

vf5

2q

93

0-

E74

.1

94

0-

E07

.8-

11

50-

E00

.1

09

1.0

50+

E35

858

813

08.

41

1-E

52.

42

150

+E6

881

308

.4

19.

050

+E6

675

881

308

.4

11

10

.0

50+

E45

85

88

130

8.4

21

ltt

ild

aq

0-

E32

.1-

01

90

-E

02.1

-9

90

-E0

2.6

-0

51.

00

0+E

778

71

870

95.

1-

90-

E4

2.4

31

00

+E

818

709

5.1

-3

9.

000

+E9

887

187

095

.1-

11

00

.0

00

+E9

397

187

095

.1-

81

ori

faq

98

95

0-

E05

.3-

01

60-

E00

.8

01

60

-E0

0.

21

55

.0

40+

E53

024

325

36.

19

0-E

16.

41

24

0+

E24

325

36

.1

00.

140

+E3

402

432

536

.1

91

60

.0

40

+E

730

243

253

6.1

62

md

nab

q

40-

E07

.2-

01

50

-E0

0.6

-01

40-

E0

2.1

-0

63

.0

50+

E72

060

21

74

6.1

90-

E5

6.1

51

50+

E60

217

46.

159

.0

50+

E12

060

217

46.

131

10

.0

50+E

51

060

21

74

6.1

41

fn

oc

aeb

q9 0

16

0-

E0

8.2

-01

60

-E

01.

20

16

0-E0

0.1

-1

45

.0

30

+E

820

800

200

1.3

01-

E2

4.2

91

30+

E80

020

01.

31

9.

03

0+

E94

080

020

01.

321

20

.0

30

+E

810

80

02

001

.3

81

d3

ero

bq

94

0-E

83.

101

60-

E00

.5

-0

160

-E0

0.5

-0

73.

04

0+E

268

411

573

8.2

90-

E5

3.

17

140

+E5

115

738

.2

69

.0

40

+E

758

411

57

38

.2

71

40.

040

+E7

584

115

738

.2

12

yd

na

rbq

53

0+E

77.

35

30

+E

77.

35

30

+E77

.3

081

.1

70

+E8

003

259

876

.6

90-

E03

.2

92

70

+E

392

397

6.6

01

.1

70

+E

462

392

397

6.6

82

61.

070

+E6

62

392

39

76.

60

4ir

pac

q

860

-E1

0.3

-0

170

-E0

6.1

01

70

-E0

4.1

-0

15

.0

20

+E

10

33

435

621

.2

90-

E34

.1

71

20

+E

343

562

1.2

99.

02

0+E

713

343

562

1.2

61

90.

02

0+E

782

343

562

1.2

42

62

2e

q

94

0-E

68.

301

50

-E

00.

70

150

-E0

2.1

92

9.

14

0+E

416

963

067

6.8

90-

E6

5.2

03

40

+E0

730

676

.8

04

.1

40+

E48

696

306

76.

82

344

.0

40

+E6

269

630

676

.8

23

rca

mat

eq

83

0-

E93

.9

93

0-

E14

.3

01

50-

E00

.9-

06

5.

15

0+

E16

064

741

37.

89

0-E

29.

27

250

+E7

474

137

.8

03

.1

50

+E2

046

474

137

.8

42

04

.0

50

+E2

506

474

13

7.8

23

08f

fff

fq

910

+E2

9.3

91

0+

E65

.1

15

1.

190

+E8

06

41

366

54.

71

0+E

78.

313

90

+E5

136

654

.7

01

.1

90

+E4

674

136

654

.7

52

ro

rre

da

er

SPM

na

lpr

ofq

83

0+

E3

1.

59

20

+E

06.

31

50

.2

11+

E78

485

097

00.

10

1-E

54.

13

31

1+E

950

970

0.1

01.

111

+E3

258

50

97

00.

15

2ro

rre

da

er

SPM

np

xd

rfg

q

10

-E0

5.4

91

0-

E03

.1

01

20

-E0

0.2

-0

53.

18

0+E

540

463

961

0.1

-90-

E1

0.1

12

80+

E4

63

961

0.

1-

04

.1

80+

E23

046

396

10.

1-

02

11.

08

0+

E74

046

396

10.

1-

91

51w

org

q9

00+

E54

.3

91

0-E

00.

401

20-

E00

.2-

00

1.

28

0+E

54

35

982

69

4.

1-

01

-E

20.

642

80

+E5

982

694

.1-

07

.1

80+

E50

359

826

94.

1-

32

91

.0

80+

E74

359

826

94.

1-

32

22

wor

gq

8

-E

46

.1

-0

120

-E0

5.1

-0

12

0-E

07

.3-

03

8.0

70+

E63

831

789

72.

4-

90

-E

09.

41

270

+E4

178

972

.4-

01

.1

70+

E15

83

17

897

2.4

-12

50

.0

70+

E37

83

178

972

.4-

81

7wo

rgq

91

0

MPI

cM

KE

SO

MD

PM

PB .

pm

oC

KE

SO

M .p

mo

CX

EL

PC .

pm

oC

XE

LP

C

56


Table 3.3: McIPM Performance on QO Test Set (III).

bor

PD

PM

PB

evit

cej

bO

retI

em

aN

.c

es

UP

Ce

vitc

ejb

Or

etI.

ce

s U

PC

evit

cej

bO

.di

se

Rr

etIe

vitc

ejb

OU

PC

.c

es

reti

#

1>

q.r

effiD

.n

giS

stigi

d.r

effiD

.n

giS

stigi

d.r

effiD

.n

giS

stigi

d

91

0-E5

0.2

520

+E09

.3

01

30

-E0

0.6

-2

47

.0

70

+E59

77

387

435

.2

01-

E54

.6

13

70

+E8

387

435.

201

.1

70

+E

014

722

84

35

.2

42

01.

07

0+

E98

773

874

35.

25

2l

ea

rsi

q

E8

6.8

11

01

-E0

1.7-

01

90-E

13.

1-

042

.0

30-

E08

671

452

48.

7-

90

-E

31.

36

13

0-

E90

452

48.

7-

89

.0

30

-E

587

424

52

48

.7-

81

10.

030

-E2

470

345

248

.7-

81

dne

lbc

pq

01

01-

920

-E0

6.1

-9

20-

E0

7.

10

130

-E00

.6-

26

2.

170

+E6

104

193

051

.1

90

-E

29.

132

70

+E4

193

05

1.

100

.1

70

+E3

304

193

051

.1

02

90

.0

70

+E0

104

193

051

.1

82

1ie

obc

pq

920

-E2

6.4

93

0-

E07

.8-

93

0-E

05.

9-

01

7.0

60+

E83

52

26

917

1.8

90-

E9

2.4

32

60+

E3

26

917

1.8

01.

16

0+

E15

422

691

71.

85

25

0.0

60

+E

344

226

917

1.8

54

2i

eob

cpq

920

-E2

2.2

01

30

-E0

5.2

01

30

-E0

7.1

-0

78

.0

60

+E8

774

783

402

.6

01-

E04

.7

32

60

+E5

783

402.

60

2.1

60

+E

308

478

3402

.6

32

91.

06

0+

E1

674

78

34

02.

66

2r

ia

ts

cp

q

01

30-

E04

.1

92

0-

E97

.1-

11

40-

E00

.4

29

7.4

60+

E68

986

858

27.

40

1-E

36.

31

36

0+

E96

858

27.

40

3.

360

+E7

088

685

82

7.

415

49

.0

60+

E09

986

858

27.

47

3o

nto

lip

q

80

0+

E00

.0

11

01-E

00

.3

11

00

+E0

0.0

001

.0

00

+E

000

05

78

173

.4

11-

E86

.5

700

+E0

578

17

3.

468

.0

00

+E3

000

578

173

.4

60

0.

00

0+

E00

005

781

73.

40

2t

set

pq

0-E

06.

2-

970

-E0

3.5

01

70

-E

06.

2-

044

.0

20+

E47

999

516

66.

2-

90

-E6

5.2

71

20+

E00

616

66.

2-

99

.0

20

+E1

299

951

666

.2-

71

20

.0

20

+E0

000

061

666

.2-

32

ep

ice

rq

01

7

0-

E95

.1

11

11-

E43

.9-

11

11

-E1

4.9

-0

23

.0

30-

E24

933

59

31

8.5

-90-

E0

0.3

71

30-

E8

15

931

8.5

-79

.0

30

-E6

784

359

318

.5-

71

20

.0

30

-E3

884

359

318

.5-

91

502

csq

01

9

90

0+

E26.

111

20-

E00

.1-

11

20-

E00

.1-

03

6.

080

+E8

383

973

710

.2

01-

E43

.1

22

80

+E4

973

710.

20

0.

18

0+E

738

397

3710

.2

02

50.

080

+E7

38

39

737

10.

29

15

2rg

acs

q

81

0-

E69.

301

20-

E0

1.

1-

01

20

-E05

.1-

06

3.

070

+E40

68

495

686

.2

90-

E01

.2

22

70

+E9

495

686.

201

.1

70

+E

395

849

568

6.2

81

20.

07

0+

E98

584

956

86.

27

17

rg

acs

q

81

0-

E3

6.3

01

30-

E00.

401

30

-E

00.

21

78

.0

70+

E73

619

628

86

.1

90

-E2

0.2

82

70+

E29

628

86.1

00.

17

0+

E14

619

628

86.

142

21

.0

70+

E93

619

628

86.

18

21m

xfc

sq

81

0-

E32.

401

30-E

00.

311

30

-E0

0.2

306

.1

70

+E

77

51

616

777

.2

01-

E24

.2

23

70

+E2

616

777

.2

03

.1

70+

E08

516

167

77.

20

372

.0

70

+E9

751

616

777

.2

73

2m

xf

cs

q

81

0-E9

3.5

01

30

-E0

0.8

01

20-

E00

.1

49

3.

27

0+

E16

445

361

80.

39

0-E

00.

14

37

0+

E55

361

80.

305

.1

70

+E9

644

536

180

.3

23

63

.0

70+E

17

44

53

618

0.3

53

3m

xf

csq

85

0-E

17.

401

70

-E0

0.2

11

70

-E0

0.1

07

3.

030

+E9

255

905

088

.1

11-

E10

.6

51

30

+E6

905

088

.1

49

.0

30

+E

135

590

508

8.1

21

20.

030

+E

03

559

050

88.

15

1o

ipr

ocs

q

960-

E22

.7-

96

0-E

77.

19

60-E

73.

3-

761

.1

20

+E

227

100

654

0.9

90

-E1

8.

44

22

0+E

100

654

0.9

01.

12

0+E

998

100

654

0.9

32

80.

02

0+E

583

100

654

0.9

62

8s

rcs

q

540

-E04

.2

54

0-

E0

4.

25

40

-E0

4.

293

18

9.

600

+E4

626

624

666

.8

50-

E71

.2

05

100

+E7

666

666

.8

19

.0

00

+E

647

66

66

666

.8

01

50

.0

00

+E3

476

666

666

.8

51

1ds

csq

98

0-

E06

.8

-9

70

-E

97.1

-9

70

-E7

8.1

-0

07

.0

10

+E6

804

128

080

.5

90-

E82

.4

41

10

+E4

128

080

.5

99

.0

10+

E70

931

28

08

0.5

41

11

.0

10

+E9

983

128

080

.5

12

6ds

csq

960

-E1

4.

4-

01

70

-E

03.

4-

01

70-

E00

.2-

03

1.1

20+

E14

475

367

04.

90

1-E

02.

62

12

0+

E75

367

04.

90

2.

120

+E8

937

536

704

.9

41

42

.0

20+

E1

24

753

670

4.9

61

8d

scs

q

950

-E2

1.1

-8

50-

E10

.2

11

70

-E00

.1-

12

5.

030

+E21

11

168

514

.1

01-

E63

.1

71

30

+E1

16

85

14.

199

.0

30

+E

313

116

851

4.1

71

50.

03

0+

E11

111

685

14.

11

21

pa

tcs

q

960

-E0

3.2

85

0-

E85

.5

11

00

+E

00.

00

91

.1

30+

E77

946

205

37

.1

11

-E1

6.3

31

30+

E56

205

37.

103

.1

30+

E53

556

205

37.

151

32

.0

30+

E77

946

205

37.

19

12p

atc

sq

85

0-

E08

.1

01

70-

E00

.9-

01

60

-E0

1.1

-0

15

.1

30

+E

02

86

457

834

.1

01-

E45

.9

31

30

+E7

457

834

.1

04

.1

30+

E11

864

578

34.

15

123

.0

30

+E9

086

457

834

.1

02

3p

atc

sq

91

0-E

25.

69

10

-E

62

.2

01

30

-E0

0.9

21

8.

170

+E8

430

081

841

.8

90-

E03

.1

62

70

+E1

081

841

.8

00

.1

70+

E47

50

08

184

1.

86

14

0.

070

+E7

530

081

841

.8

12

ab

esq

93

0-E

90.

101

40

-E0

1.7

940

-E0

6.7

-0

64.

050

+E1

981

387

002

.7

90-

E64

.1

62

50

+E

238

700

2.7

00

.1

50

+E

269

138

700

2.7

52

40.

050

+E

51

813

870

02.

79

2b

1er

ahs

q

840

-E0

9.2

95

0-

E02

.1

01

50-

E01

.1

08

2.0

40+

E01

719

630

71.

19

0-E

32.

371

40+

E29

630

71.

179

.0

40

+E2

27

19

630

71.

171

20

.0

40+E

12

719

630

71.

17

1b

2er

ahs

q

840

+E1

3.4

-9

30+

E02

.7

11

20

+E00

.1-

65

0.

321

+E1

348

63

62

75.

10

1-E

40.

183

21

+E8

63

62

75.

103

.1

21

+E

305

863

627

5.1

33

43.

02

1+E

03

486

362

75.

193

ll

eh

sq

82

0-E

14.

3-

01

40-

E00.

3-

11

00+

E00

.0

07

0.

160

+E1

43

55

100

24.

20

1-E

33.

151

60

+E5

510

024

.2

01

.1

60

+E8

335

510

024

.2

41

70

.0

60

+E1

43

55

100

24.

251

l4

0p

ihs

q

82

0-

E07

.2

11

00+

E00

.0

11

00

+E0

0.

00

47

.0

60

+E

037

639

942

4.2

01-E

30.

15

16

0+E

739

942

4.2

99.

060

+E0

376

39

94

24.

24

15

0.

06

0+E

037

639

942

4.2

51

s4

0pi

hsq

92

0-

E08

.1-

01

30

-E

00

.2

01

30

-E0

4.1

-0

11.

660

+E0

816

040

673

.2

90-

E85

.1

41

60

+E

604

067

3.

201

.2

60

+E

002

604

067

3.2

31

65.

16

0+

E66

160

406

73.

25

1l

80p

ihs

q

82

0-

E17

.4

93

0-E

09.8

01

30-

E09

.1-

07

7.1

60+

E92

588

275

83.

20

1-E

34

.8

51

60+

E98

275

83.

203

.1

60

+E

81

688

275

83.

231

24

.0

60+E

01

588

275

83.

27

1s

80p

ihs

q

92

0-

E38

.1

01

30-

E08

.2-

93

0-E

07

.4

-1

87.

11

60

+E

718

56

78

81

0.3

90-

E9

8.1

81

60

+E

66

78

810

.3

00

.3

60+

E98

756

788

10.

37

136

.2

60

+E0

775

678

810

.3

91

l21

pih

sq

82

0-

E30

.5

93

0-

E05

.4

01

40-

E00

.4-

19

2.

26

0+

E79

422

696

50.

30

1-E

77.

19

160

+E3

269

650

.3

04

.1

60

+E2

452

269

650

.3

61

45

.0

60

+E

39

422

696

50

.3

81

s21

pih

sq

91

0-

E20

.2-

01

20

-E

00.2

-01

20-

E03

.2-

01

8.3

70+

E20

285

405

73.

20

1-E

08.

31

270

+E8

540

573

.2

03.

170

+E2

818

540

573

.2

81

12

.0

70

+E

971

854

057

3.2

91

ar

re

isq

92

0-

E33

.4

70

0+E

55.1

11

40-

E00

.4-

098

.0

60+

E76

572

545

89.

79

0-E

69.

11

260

+E8

25

45

89.

70

3.

160

+E3

013

454

589

.7

42

31

.0

60

+E

365

725

458

9.7

81

ri

ats

q

01

70

-E0

0.8

-9

60-

E08

.9

95

0-

E91

.1

-1

99

.0

30+

E8

00

48

381

14.

69

0-

E1

6.2

71

30+

E48

381

14.

669

.0

30+

E60

148

381

14.

631

60

.0

30+E

98

838

381

14.

68

1t

adn

ats

q

97

0-E

25.

59

50

-E

68.

18

80

-E4

1.2

-0

81

.0

80

-E1

311

794

341

.2

80-

E08

.1

81

70

-E

507

013

7.5

49

.0

50

-E

133

055

116

8.1

41

10.

000

+E0

00

00

000

00.

08

186

2s

13

0-E

00.

67

00

+E

04.6

920

-E0

9.3

-1

86

.6

70+

E60

046

862

58.

2-

60

-E

61.

26

17

0+E

468

625

8.2

-02

.2

70

+E4

067

586

258

.2-

41

53

.0

70

+E5

404

686

258

.2-

21

1t

ada

ts

0

70

-E7

2.1

-5

30

-E0

1.1

11

00+

E00

.0

14

1.

810

+E3

784

666

262

.3-

90-

E52

.5

02

10

+E

566

626

2.3

-0

2.

210

+E0

22

86

552

62.

3-

61

75

.0

10+

E3

78

466

626

2.3

-22

2t

ad

ats

9

E0

9.

4-

11

90-

E0

0.

111

90-

E00

.2-

03

5.

41

10

+E1

592

549

775

.3-

11

-E

96.

981

10+

E35

497

75.

3-

06

.4

10

+E

059

254

977

5.3

-0

22

5.

11

0+E

359

254

977

5.3

-1

23

ta

dat

s9

80-

23

0+

E02

.2

23

0+E

02.

22

30

+E0

2.2

019

.1

50

+E

876

922

492

5.1

90-

E0

3.1

01

50+

E6

53

415

5.1

00

.1

50+

E77

455

341

55.

18

54

.0

50

+E0

745

534

155

.1

11

1pq

cts

830

-E9

0.2

-8

30

-E1

8.1

-8

30-

E28

.1-

021

.4

40

+E1

905

137

232

.2

90-

E27

.4

84

0+

E31

372

32.

20

1.1

40+

E67

231

372

32.

29

87

.61

40+

E27

231

37

232

.2

11

2pqc

ts

80

0+

E00

.0

11

00

+E

00.

011

00

+E0

0.0

040

.0

00+

E0

00

000

00

00

.0

90-

E90

.4

400

+E0

000

00

0.

058

.0

00

+E

000

000

000

0.0

30

0.

00

0+

E00

000

000

00.

08

em

at

66

0-E

19.

8-

660

-E2

9.8

-1

10-

E88

.4

31

80

.3

60

0+

E23

17

90

061

1.1

90-

E6

5.8

44

00+

E80

006

11.

103

.8

00+

E86

970

006

11.

104

44.

40

0+

E39

717

883

06.

19

1hb

u

880

-E1

4.

40

10

1-

E00

.5

09

5.

000

+E1

441

126

693

.1-

01-

E46

.5

41

00

+E

112

669

3.1

-3

9.

000

+E6

34

11

266

93

.1

-3

1ro

rre

da

er

SPM

se

ula

v

12

0+

E10

.1

20

0+

E84

.9

12

0+E

10.

11

17

.3

10+

E19

19

01

766

6.9

30-

E7

1.1

91

20+

E62

407

79.

105

.1

20+

E64

213

941

60.

131

52

.0

20+

E49

552

407

79.

15

2o

ay

800

+E

00

.0

11

00+

E00

.0

11

00

+E0

0.0

00

1.0

00

+E0

000

000

521

.4-

11-

E12

.4

70

0+

E00

005

21.

4-

58

.0

00

+E

000

000

052

1.4

-5

00

.0

00

+E

000

000

052

1.4

-7

12c

ive

cez

KE

SO

MX

EL

PC

DP

MP

B .p

mo

CK

ES

OM .

pm

oC

XE

LP

C .p

mo

CM

PIc

M

57


Table 3.4: McIPM Performance on Difficult QO Problems

DP

MP

B.

bor

P

evit

cej

bO

retI

em

aN

UP

C

.c

es

evit

cej

bO

re tI

UP

C

.c

es

evit

cej

bO

.di

se

Rr

e t Ie

vitc

e jb

OU

PC

.c

es

reti

#

1>

q.r

effiD

.n

giS

stigi

d.r

effiD

.n

giS

stigi

d.r

effiD

.n

giS

stigi

d

E26

3.2

-4

30-

E2

63

.2-

43

0-

E1

63.

2-

34

5.76

34

10

+E1

358

21

95

21

.2

80

-E

46

.4

49

10

+E

76

76

52

1.

23

9.

56

10+

E15

966

7652

1.2

04

76

.9

81

0+

E762

767

652

1.2

35

2d

yo

b4

30

-

86

0-E

433

.1

75

0-

E3

39.

18

60

-E

434

.1

02

7.

22

10

+E6

660

04

22

16.

39

0-E

44.

10

21

0+

E20

422

16.

309

.6

10

+E3

999

14

221

6.3

22

19

.1

10+

E001

204

221

6.3

32

1tew

sil

0-

E5

19.

1-

750

-E

51

4.1

86

0-

E51

2.2

-3

74.

33

10

+E

51

96

87

58

49

.4

01

-E

68

.8

42

10

+E

58

75

849

.4

01

.8

10

+E3

601

08

58

49.

472

43.3

10+

E00

748

758

49

.4

14

01t

ews

il

86

-E2

89.

2-

76

0-

E3

22

.6

-7

50

-E2

79.

2-

055.

03

10

+E9

186

89

32

59

.4

80

-E

12

.1

32

10

+E

75

93

25

9.

409

.9

10+

E6

95

089

325

9.4

63

83

.2

10+

E00

1759

325

9.4

92

11te

wsi

l7

50

50-

E0

58.

2-

73

0-E

251

.1

96

0-

E0

93

.2

-2

91

.33

30

+E

58

24

72

96

37.

19

0-E

64.

21

33

0+

E47

2963

7.

10

0.

83

0+

E70

858

29

63

7.1

72

1.

53

0+E

162

472

963

7.1

46

21te

wsi

l8 8

70

-E0

84

.6

-7

60

-E

508

.2

87

0-

E084

.5

-0

1.8

11

0+E

846

670

899

4.2

01

-E1

4.8

51

10

+E

67

08

994

.2

09

.6

10

+E

354

97

08

994

.2

22

51.

110

+E0

016

708

994.

23

12te

wsi

l

60

-E9

82

.2

-9

70

-E

001

.1

86

0-

E9

82

.2

-2

49

.32

10

+E

98

22

22

10

05.

29

0-E

46.

20

21

0+

E02

2100

5.

20

8.

71

0+

E99

322

2100

5.2

32

58.1

10+

E00

00

221

005

.2

22

3te

wsi

l8

60

-E3

16

.1-

76

0-

E55

3.3

86

0-

E32

5.1

-1

26.1

21

0+

E3

16

31

10

00

5.

29

0-

E7

4.

29

11

0+

E2

110

005

.2

00.

710

+E8

696

11

00

05.

232

67.1

10+

E09

021

10

005

.2

12

4t

ews

il

8

60-

E7

20

.6-

65

0-

E2

81.

37

60

-E

720

.6

-0

19.

32

10

+E

720

952

430

5.2

01

-E9

0.4

12

10

+E

35

24

305

.2

03

.7

10

+E

748

092

4305

.2

12

87.1

10+

E00

035

243

05.

212

5t

ews

il

7

60

-E0

31

.5

-7

60

-E

791

.3

76

0-

E0

35

.5

-2

47

.12

10

+E

03

13

57

59

94.

29

0-E

02.

39

11

0+

E84

75

99

4.2

06

.6

10+

E72

36

57

599

4.2

12

29

.1

10+E

006

7475

994

.2

32

6te

wsi

l7 8

60

-E0

40

.5

53

0-

E16

2.6

86

0-

E0

48

.5

06

4.

22

20

+E

694

880

488

9.4

90

-E2

5.8

52

20

+E

980

488

9.4

00

.1

120

+E4

064

174

889

.4

83

85.

120

+E0

809

804

889

.4

81

7t

ews

il

13

0+E

034

.6

74

0-

E2

31.

29

70

-E

009

.9

-2

80.

13

20

+E

999

500

744

1.7

01

-E5

7.4

13

30

+E6

007

441

.7

05

.8

20

+E

32

37

207

441

.7

92

65.

420

+E0

0950

074

41.

78

58te

wsi

l

850

-E

00

0.4

74

0-E

643

.2

01

70

-E

00

9.8

063

.72

30

+E

00

62

15

23

69.

19

0-E

45.

44

33

0+

E31

523

69.

10

6.

93

0+

E64

941

5236

9.1

23

17.4

30+

E90

62

152

369

.1

06

9te

wsi

l

91

0-E

065

.2

-01

30

-E

00

0.

801

20

-E

000

.1

06

8.

17

0+E

652

392

397

6.

601

-E8

2.

20

37

0+E

392

397

6.6

01

.1

70

+E

462

392

3976

.6

82

61.0

70+

E66

239

239

76.

604

irpa

cq

76

0-

E38

.1

-7

60

-E

58

.1

-7

60

-E

58.

1-

18

1.7

00

+E

87

25

86

66

66

.8

70

-E

50.

99

00

+E

766

666

6.8

19

.0

00

+E

647

666

6666

.8

01

50.

000

+E3

476

666

666.

85

11

dsc

sq

830

-E5

98

.4

94

0-

E8

43

.3-

940

-E8

40.

4-

02

5.

472

50

+E

01

555

341

55

.1

90

-E9

3.4

950

+E6

534

155

.1

00

.1

50

+E7

745

534

155

.1

854

.0

50+E

074

553

415

5.1

11

1p

qct

s

86

0-

E8

80

.4

11

0+

E5

51

.9-

01

80-

E01

8.2

14

4.

320

+E

19

55

240

779

.1

60-

E78

.5

02

20

+E

624

07

79

.1

05

.1

20

+E

642

139

416

0.1

31

52

.0

20+E

495

524

077

9.1

52

oa

y

MPI

cM

DP

MP

B .p

mo

CK

ES

OM .

pm

oC

KE

SO

MX

EL

PC

XE

LP

C .p

mo

C

58

Chapter 4

Parametric Quadratic

Optimization

In this chapter we present an IPM and optimal partition based technique and

provide a polynomial time algorithm for conducting parametric analysis of convex

Quadratic Optimization problems. The novelty of our results is that we allow

simultaneous variation in the coefficient vector of the linear term of the objective

function and in the right-hand side vector of the constraints. The resulting

problem we intend to solve is:

φ(λ) = min f(x, λ)s.t. Ax = b + λ△b

x ≥ 0,

where f(x, λ) is linear or quadratic function of x, and λ is the perturbation

parameter.

A method described in this chapter performs division of the parameter

space into so-called invariancy intervals and provides complete description of

the behavior of φ(λ) and x∗(λ) on each interval. The resulting algorithm allows

solving parametric quadratic (as well as parametric linear) optimization problems

efficiently in polynomial time.

59


4.1 Origins of Quadratic and Linear Parametric

Optimization and the Existing Literature

As you already know, research on parametric optimization was triggered when

a variant of parametric convex QO problems was considered by Markowitz [17].

He developed the critical line method to determine the optimal value function

of his parametric problem and applied it to mean-variance portfolio analysis.

The basic result for parametric quadratic programming obtained by Markowitz

is that the optimal value function (efficient frontier in financial terminology) is

piecewise quadratic and can be obtained by computing successive corner portfo-

lios while in between these corner portfolios the optimal solutions vary linearly.

Non-degeneracy was assumed and a variant of the simplex method was used for

computations.

Difficulties that may occur in parametric analysis when the problem is de-

generate are studied extensively in the LO literature. In case of degeneracy, the

optimal basis does not need to be unique and multiple optimal solutions may

exist. While simplex methods were used to perform the computations in earlier

studies (see e.g., Murty [24] for a comprehensive survey), recently research on

parametric analysis was revisited from the point of view of IPMs. For degener-

ate LO problems, the availability of strictly complementary solutions produced

by IPMs and use of optimal partitions associated with such solutions allow one

to overcome many difficulties associated with the use of bases. Alder and Mon-

teiro [1] pioneered the use of IPMs in parametric analysis for LO (see also Jansen

et al. [16]). Berkelaar, Roos and Terlaky [4] emphasized shortcomings of using

optimal bases in parametric LO showing by an example that different optimal

bases computed by different LO packages give different optimality intervals.

Naturally, results obtained for parametric LO were extended to the convex

QO. Berkelaar et al. [3] showed that the optimal partition approach can be gen-

eralized to the quadratic case by introducing tripartition of variables instead of

bipartition. They performed the sensitivity analysis for the cases when perturba-

60


tion occurs either in the coefficient vector of the linear term of the objective value

function or in the right-hand side of the constraints. We show that the results

obtained in Berkelaar, Roos and Terlaky [4] and Berkelaar et al. [3] can be gener-

alized further to accommodate simultaneous perturbation of the data even in the

presence of degeneracy. The theoretical results allow us to present a universal

computational algorithm for the parametric analysis of LO/QO problems.

It is worthwhile to mention that some encouraging results already exist

for parametric convex Conic Optimization (CO). For convex conic problems a

feasible point is restricted to be in a closed, convex, solid and pointed cone.

Yildirim [34] extended the concept of the optimal partition to such conic prob-

lems. He proved that the optimal value function is quadratic and presented

auxiliary conic problems for computing derivatives and boundaries of invariancy

intervals.

The chapter is organized as follows. In Section 4.2 some elementary con-

cepts related to convex QO problems are reviewed. Simple properties of the

optimal value function are summarized in Section 4.3. Section 4.4 is devoted to

deriving properties of the optimal value function in order to formulate a com-

putational algorithm. It is shown that the optimal value function is continuous

and piecewise quadratic, and an explicit formula is presented to identify it on

the subintervals. Criteria for convexity, concavity or linearity of the optimal

value function on these subintervals are derived. We investigate the first and

second order derivatives of the optimal value function as well. Auxiliary LO

problems can be used to compute the left and right derivatives. It is shown that

the optimal partition on the neighboring intervals can be identified by solving

an auxiliary self-dual QO problem. The results are summarized in a computa-

tional algorithm. Specialization of the method to LO problems is described in

Section 4.5.

61


4.2 Properties of Convex QO Problems

In this section we review the concepts and properties of convex QO problems

that are necessary for development of the parametric analysis. Primal (QP) and

dual (QD) QO problems are already defined by (2.1.1) and (2.1.2), respectively.

The first property we would like to look at directly follows from the com-

plementarity condition xizi = 0 for all i ∈ {1, 2, . . . , n} (zero duality gap).

Lemma 4.2.1 For two optimal solutions (x∗, y∗, z∗) and (x, y, z) of (QP ) and

(QD) it holds that Qx∗ = Qx , cT x∗ = cT x and bT y∗ = bT y and consequently,

xT z∗ = zT x∗ = 0. (4.2.1)

Proof. See e.g., Berkelaar et al. [3] and Dorn [7]. ¤

The concept of optimal partition is used extensively in this chapter. It is

very important to understand its meaning precisely. The optimal partition of the

index set {1, 2, . . . , n} is defined as

B = { i : xi > 0 for an optimal solution x ∈ QP∗},N = { i : zi > 0 for an optimal solution (x, y, z) ∈ QD∗},T = {1, 2, . . . , n} \(B ∪ N ),

and denoted by π = (B,N , T ). Berkelaar et al. [3] and Berkelaar, Roos and

Terlaky [4] showed that this partition is unique. Another concept related very

closely to optimal partition is the support set of a vector v which is defined

as σ(v) = {i : vi > 0}. We can establish a link between the optimal partition

and a maximally complementary solution that was defined in Section 2.1. An

optimal solution (x, y, z) is maximally complementary if it possesses the following

properties:xi > 0 if and only if i ∈ B,zi > 0 if and only if i ∈ N .

For any maximally complementary solution (x, y, z) the relations σ(x) = B and

σ(s) = N hold. The existence of a maximally complementary solution is a

62


direct consequence of the convexity of the optimal sets QP∗ and QD∗. It is

known that IPMs find a maximally complementary solution in the limit, see e.g.,

McLinden [19] and Guler and Ye [12].

We are going to consider the parametric problems with only one parameter

in this thesis. But let us look at the general multiparametric convex QO problem

first to understand better what has been done in the area before:

min (c + λc△c)T x + 12xT Qx

(QPλb,λc) s.t. Ax = b + λb△b,

x ≥ 0.

Here △b ∈ Rm and △c ∈ R

n are nonzero perturbation vectors, and λb and λc

are real parameters. The optimal value function φ(λb, λc) denotes the optimal

value of (QPλb,λc) as the function of the parameters λb and λc. As we already

mentioned, Berkelaar, Roos and Terlaky [4] and Berkelaar et al. [3] were the

first to analyze parametric convex QO by using the optimal partition approach

when variation occurs either in the right-hand side or in the linear term of the

objective function data, i.e., either △c or △b is zero. In these cases the domain

of the optimal value function φ(λb, 0) (or φ(0, λc)) is a closed interval of the real

line and the function is piecewise convex (concave) quadratic on its domain. The

authors presented an explicit formula for the optimal value function on these

subintervals and introduced the concept of transition points that separate them.

They proved that the optimal partition is invariant on the subintervals which

are characterized by consecutive transition points. The authors also studied the

behavior of the first and second order derivatives of the optimal value function

and proved that the transition points coincide with the points where the first or

second order derivatives do not exist. It was proven that by solving auxiliary self-

dual QO problems, one can identify the optimal partitions on the neighboring

subintervals.

63


4.3 The Optimal Value Function in Simultane-

ous Perturbation Sensitivity Analysis

In this section, we introduce explicitly the perturbed convex QO problem when

perturbation occurs simultaneously in the right-hand side data and in the lin-

ear term of the objective function of (QP ). In the problem (QPλb,λc) that

was mentioned in the pervious section, λb and λc are independent parame-

ters. In this thesis we are only concerned with the case when they coincide,

i.e., λb = λc = λ. Consequently, the perturbation takes the form λh, where

h = (△bT ,△cT )T ∈ Rm+n is a nonzero perturbing direction and λ ∈ R is a

parameter. Thus, we define the following primal and dual perturbed problems

corresponding to (QP ) and (QD), respectively:

min (c + λ△c)T x + 12xT Qx

(QPλ) s.t. Ax = b + λ△b,x ≥ 0,

(4.3.1)

min (b + λ△b)T y − 12xT Qx

(QDλ) s.t. AT y + z − Qx = c + λ△c,z ≥ 0,

(4.3.2)

Let QPλ and QDλ denote the feasible sets of the problems (QPλ) and

(QDλ), respectively. Their optimal solution sets are analogously denoted by

QP∗λ and QD∗

λ. The optimal value function of (QPλ) and (QDλ) is

φ(λ) = (c + λ△c)T x∗(λ) +1

2x∗(λ)T Qx∗(λ) = (b + λ△b)T y∗(λ)− 1

2x∗(λ)T Qx∗(λ),

where x∗(λ) ∈ QP∗λ and (x∗(λ), y∗(λ), z∗(λ)) ∈ QD∗

λ. Further, we define

φ(λ) = +∞ if QPλ = ∅,φ(λ) = −∞ if QPλ 6= ∅ and (QPλ) is unbounded.

Let us denote the domain of φ(λ) by

Λ = {λ : QPλ 6= ∅ and QDλ 6= ∅}.

Since it is assumed that (QP ) and (QD) have optimal solutions, it follows that

Λ 6= ∅. Proving the following property of Λ is the first in the sequence of steps

for subdividing the space of λ into subintervals.

64


Lemma 4.3.1 Λ ⊆ R is a closed interval.

Proof. Let λ 6∈ Λ. There are two cases: the primal problem (QPλ) is feasible

but unbounded or it is infeasible. We only prove the second case, the first one

can be proved analogously. If the primal problem (QPλ) is infeasible then by the

Farkas Lemma (see e.g., Murty [24] or Roos, Terlaky and Vial [26]) there is a

vector y such that AT y ≤ 0 and (b+λ△b)Ty > 0. Fixing y and considering λ as a

variable, the set S(y) ={

λ : (b + λ△b)T y > 0}

is an open half-line in λ, thus the

given vector y is a certificate of infeasibility of (QPλ) for an open interval. Hence,

the union⋃

y S(y), where y is a Farkas certificate for the infeasibility of (QPλ)

for some λ ∈ R, is open. Consequently, the domain of the optimal value function

is closed. We also need to show that this closed set is connected. Let λ1, λ2 ∈ Λ

be two arbitrary numbers. Let (x(λ1), y(λ1), z(λ1)) ∈ QP∗λ1

×QD∗λ1

. Similarly,

(x(λ2), y(λ2), z(λ2)) ∈ QP∗λ2

× QD∗λ2

. For any λ ∈ (λ1, λ2) and θ = λ2−λλ2−λ1

we

have

λ = θλ1 + (1 − θ)λ2.

Let us define

x(λ) = θx(λ1) + (1 − θ)x(λ2),

y(λ) = θy(λ1) + (1 − θ)y(λ2),

z(λ) = θz(λ1) + (1 − θ)z(λ2).

It is easy to check that (x(λ), y(λ), z(λ)) ∈ QP∗λ × QD∗

λ. This implies that the

set Λ is connected and thus Λ is a closed interval. ¤

4.4 Properties of the Optimal Value Function

In this section we investigate the properties of the optimal value function. These

are generalizations of the corresponding properties that have been proven in

Berkelaar et al. [3] for the case when △c = 0 or △b = 0.

65


4.4.1 Basic Properties

For λ∗ ∈ Λ, let π = π(λ∗) denote the optimal partition and let (x∗, y∗, z∗) be

a maximally complementary solution at λ∗. We use the following notation that

generalizes the notation introduced in Berkelaar et al. [3]:

O(π) = {λ ∈ Λ : π(λ) = π} ;

Sλ(π) = {(x, y, z) : x ∈ QPλ, (x, y, z) ∈ QDλ,

xB > 0, xN∪T = 0, zN > 0, zB∪T = 0};Sλ(π) = {(x, y, z) : x ∈ QPλ, (x, y, z) ∈ QDλ,

xB ≥ 0, xN∪T = 0, zN ≥ 0, zB∪T = 0};Λ(π) = {λ ∈ Λ : Sλ(π) 6= ∅} ;

Λ(π) ={

λ ∈ Λ : Sλ(π) 6= ∅}

;

Dπ = {(△x,△y,△z) : A△x = △b, AT△y + △z − Q△x = △c,

△xN∪T = 0,△zB∪T = 0}.

The following theorem resembles Theorem 3.1 from Berkelaar et al. [3] and

presents the basic relations between the open interval, where the optimal parti-

tion is invariant, and its closure.

Theorem 4.4.1 Let π = π(λ∗) = (B,N , T ) for some λ∗ denote the optimal

partition and (x∗, y∗, z∗) denote an associated maximally complementary solution

at λ∗. Then,

(i) Λ(π) = {λ∗} if and only if Dπ = ∅;(ii) Λ(π) is an open interval if and only if Dπ 6= ∅;(iii) O(π) = Λ(π) and clO(π) = cl Λ(π) = Λ(π);

(iv) Sλ(π) = {(x, y, z) : x ∈ QP∗λ, (x, y, z) ∈ QD∗

λ} for all λ ∈ Λ(π).

Proof. First let us recall the characteristics of a maximally complementary

solution. Any maximally complementary solution (x∗, y∗, z∗) associated with a

given λ∗ satisfies Ax∗ = b+λ∗△b, AT y∗+z∗−Qx∗ = c+λ∗△c, x∗B > 0, x∗

N∪T = 0,

66


z∗N > 0 and z∗B∪T = 0. Let (△x,△y,△z) ∈ Dπ, and define

x = x∗ + (λ − λ∗)△x, (4.4.1)

y = y∗ + (λ − λ∗)△y, (4.4.2)

z = z∗ + (λ − λ∗)△z. (4.4.3)

If λ is in an ǫ-neighborhood of λ∗ for enough small ǫ, then

Ax = b + λ△b,

AT y + z − Qx = c + λ△c,xN∪T = 0,zB∪T = 0,

xB > 0, zN > 0.

(4.4.4)

(i) [⇒] : Let Λ(π) = {λ∗}, and assume to the contrary that Dπ is not

empty. Then, there exists (△x,△y,△z) such that A△x = △b and AT△y+△z−Q△x = △c with △xN∪T = 0 and △zB∪T = 0. Let (x∗, y∗, z∗) be a maximally

complementary solution associated with λ∗, i.e., Ax∗ = b + λ∗△b, AT y∗ + z∗ −Qx∗ = c + λ∗△c, x∗

N∪T = 0, z∗B∪T = 0, x∗B > 0 and z∗N > 0. Let (x, y, z)

be defined by (4.4.1)–(4.4.3). From (4.4.4), one can conclude λ ∈ Λ(π) that

contradicts to the assumption Λ(π) = {λ∗}.(i) [⇐] : Let Dπ = ∅, and suppose to the contrary that λ, λ∗ ∈ Λ(π), with

λ 6= λ∗ and (x, y, z) is a maximally complementary solution at λ. Thus, from

(4.4.1)–(4.4.3) we can compute (△x,△y,△z) and conclude that (△x,△y,△z) ∈Dπ. This contradicts to the fact that Dπ = ∅ and thus Λ(π) = {λ∗}.

(ii) [⇒] : Let λ∗ ∈ Λ(π). Then, there is a maximally complementary

solution (x∗, y∗, z∗) at λ∗. Moreover, since Λ(π) is an open interval, there exists

a λ in an ǫ-neighborhood of λ∗ with λ 6= λ∗ and λ ∈ Λ(π). Let (x, y, z) denote a

maximally complementary solution at λ. From (4.4.1)–(4.4.3), we can compute

(△x,△y,△z) and conclude that (△x,△y,△z) ∈ Dπ 6= ∅.(ii) [⇐] : Suppose that Dπ is non-empty. Then, there exists (△x,△y,△z)

such that A△x = △b, AT△y + △z − Q△x = △c, △xN∪T = 0 and △zB∪T = 0.

On the other hand, a maximally complementary solution (x∗, y∗, z∗) at λ∗ exists

67


such that Ax∗ = b + λ∗△b, AT y∗ + z∗ − Qx∗ = c + λ∗△c, x∗N∪T = 0, z∗B∪T = 0,

x∗B > 0 and z∗N > 0. Consider (x, y, z) as defined in (4.4.1)–(4.4.3). For any

λ ∈ R, (x, y, z) satisfies

Ax = b + λ△b, AT y + z − Qx = c + λ△c,

and

xT z = (λ − λ∗)(△xT z∗ + △zT x∗).

From the definitions of π and Dπ, one can conclude that xT z = 0. Thus (x, y, z) is

a pair of primal-dual optimal solutions of (QPλ) and (QDλ) as long as x ≥ 0 and

z ≥ 0, that gives a closed interval around λ∗. Furthermore, for an open interval

Λ, xB > 0 and zN > 0. Let λ′ < λ∗ < λ, where λ′, λ ∈ Λ. If (x′, y′, z′) and (x, y, z)

are defined by (4.4.1)–(4.4.3), then x′B, xB > 0, x′

B∪T = xB∪T = 0, z′N , zN > 0,

z′N∪T = zN∪T = 0. To prove that λ ∈ Λ(π), we need to show that (x, y, z) is not

only optimal for (QPλ) and (QDλ), but also maximally complementary.

Let us assume that the optimal partition π = (B,N , T ) at λ is not identical

to π, i.e., there is a solution (x(λ), y(λ), z(λ)) such that

xB(λ) > 0, zN (λ) > 0, and xT (λ) + zT (λ) 6= 0. (4.4.5)

Let us define

x =λ − λ∗

λ − λ′x(λ) +

λ∗ − λ′

λ − λ′x′,

y =λ − λ∗

λ − λ′y(λ) +

λ∗ − λ′

λ − λ′y′,

z =λ − λ∗

λ − λ′z(λ) +

λ∗ − λ′

λ − λ′z′.

By definition (x, y, z) is optimal for λ∗, while by (4.4.5) it has a positive xi + zi

coordinate in T , contradicting to the definition of the optimal partition π at λ*.

We still need to show that Λ(π) is a connected interval. The proof follows

the same reasoning as the proof of Lemma 4.3.1 and is omitted.

(iii) Let λ ∈ O(π), then by definition π(λ) = π, and hence for λ ∈ Λ there

is a maximally complementary solution (x, y, z) which satisfies Ax = b + λ△b,

68


AT y + z − Qx = c + λ△c, xN∪T = 0, zB∪T = 0, xB > 0 and zN > 0, from which

we conclude that λ ∈ Λ(π). Analogously, one can prove that if λ ∈ Λ(π) then

λ ∈ O(π). Consequently, O(π) = Λ(π). Because Λ(π) is a polyhedral set, it

follows that clSλ(π) = Sλ(π). Since Λ(π) is an open interval, we conclude that

clO(π) = cl Λ(π) = Λ(π).

(iv) This result follows from Lemma 2.3 of Berkelaar et al. [3]. ¤

The following two corollaries are direct consequences of Theorem 4.4.1.

Corollary 4.4.2 Let λ2 > λ1 be such that π(λ1) = π(λ2). Then, π(λ) is constant

for all λ ∈ [λ1, λ2].

Corollary 4.4.3 Let (x(1), y(1), z(1)) and (x(2), y(2), z(2)) be maximally comple-

mentary solutions of (QP λ1), (QDλ1

) and (QP λ2), (QDλ2

), respectively. Then,

for any λ ∈ [λ1, λ2]

x(λ) =λ2 − λ

λ2 − λ1

x(1) +λ − λ1

λ2 − λ1

x(2),

y(λ) =λ2 − λ

λ2 − λ1

y(1) +λ − λ1

λ2 − λ1

y(2),

z(λ) =λ2 − λ

λ2 − λ1

z(1) +λ − λ1

λ2 − λ1

z(2)

is a maximally complementary solution of (QP λ) and (QDλ) if and only if

λ1, λ2 ∈ Λ(π).

The importance of Corollaries 4.4.2 and 4.4.3 is that on the intervals where

the optimal partition is constant we can determine a maximally complementary

optimal solution at any point inside such intervals by taking the convex combi-

nation of maximally complementary solutions at any two points of the interval.

The next theorem shows how to determine the endpoints of the interval

Λ(π). It is a direct consequence of Theorem 4.4.1 as well.

Theorem 4.4.4 Let λ∗ ∈ Λ and let (x∗, y∗, z∗) be a maximally complementary

solution of (QPλ∗) and (QDλ∗) with optimal partition π = (B,N , T ). Then the

69


left and right extreme points of the closed interval Λ(π) = [λℓ, λu] that contains

λ∗ can be obtained by minimizing and maximizing λ over Sλ(π), respectively, i.e.,

by solving

λℓ = minλ,x,y,z

{λ : Ax − λ△b = b, xB ≥ 0, xN∪T = 0, (4.4.6)

AT y + z − Qx − λ△c = c, zN ≥ 0, zB∪T = 0 },

and

λu = maxλ,x,y,z

{λ : Ax − λ△b = b, xB ≥ 0, xN∪T = 0, (4.4.7)

AT y + z − Qx − λ△c = c, zN ≥ 0, zB∪T = 0 }.

Proof. We will prove the theorem for λu only. The proof for λℓ goes analo-

gously. Problem (4.4.7) is feasible since problems (QPλ) and (QDλ) are feasible

for the given λ = λ∗. We continue by considering two cases:

(i) Problem (4.4.7) is unbounded. Then for every λ ≥ λ∗ there exists a

feasible solution (x, y, z) for (QPλ) and (QDλ). Further, from the complemen-

tarity property xT z = 0 we conclude that (x, y, z) is also optimal for (QPλ) and

(QDλ). Theorem 4.4.1 imply that λ and λ∗ belong to the same interval Λ(π).

Since this holds for any λ ≥ λ∗, the right boundary of Λ(π) is +∞.

(ii) Problem (4.4.7) has an optimal solution (λ, x, y, z). Similarly to (i),

(x, y, z) is feasible for (QPλ) and (QDλ). From the complementarity property

xT z = 0 we conclude that (x, y, z) is optimal for (QPλ) and (QDλ), and then

Theorem 4.4.1 imply that λ and λ∗ belong to the interval Λ(π) and so λ ≤ λu.

Since for every λ ∈ Λ(π) problem (4.4.7) has a feasible solution and for any

λ ≥ λu the optimal partition is different, the proof is completed. ¤

The open interval Λ(π) is referred to as invariancy interval because the

optimal partition is invariant on it. The points λℓ and λu, that separate neigh-

boring invariancy intervals, are called transition points (see Figure 4.1). The

following theorem shows that the optimal value function is quadratic on an in-

variancy interval (λℓ, λu), where λℓ and λu are obtained by solving (4.4.6) and

70


Figure 4.1: The Invariancy Intervals and Transition Points

(4.4.7). It presents an explicit formula for the optimal value function as well

as simple criteria to determine convexity, concavity or linearity of the optimal

value function on a specific invariancy interval. Figure 4.2 provides graphical

interpretation of Theorem 4.4.5.

Theorem 4.4.5 Let λℓ and λu be obtained by solving (4.4.6) and (4.4.7), re-

spectively. The optimal value function φ(λ) is quadratic on O(π) = (λℓ, λu).

Proof. If λℓ = λu the statement is trivial, so we may assume that λℓ < λu.

Let λℓ < λ1 < λ < λ2 < λu are given and let (x(1), y(1), z(1)), (x(2), y(2), z(2)) and

(x(λ), y(λ), z(λ)) be pairs of primal-dual optimal solutions corresponding to λ1,

λ2 and λ, respectively. Thus, there is a θ = λ−λ1

λ2−λ1∈ (0, 1) such that

λ = λ1 + θ△λ,

x(λ) = x(1) + θ△x = x(1) +λ − λ1

λ2 − λ1

△x,

y(λ) = y(1) + θ△y = y(1) +λ − λ1

λ2 − λ1

△y,

z(λ) = z(1) + θ△z = z(1) +λ − λ1

λ2 − λ1

△z,

where △λ = λ2 − λ1,△x = x(2) − x(1),△y = y(2) − y(1) and △z = z(2) − z(1). We

also have

A△x = △λ△b, (4.4.8)

AT△y + △z − Q△x = △λ△c. (4.4.9)

71


The optimal value function at λ is given by

φ(λ) = (b + λ△b)T y(λ) − 1

2x(λ)T Qx(λ)

= (b + (λ1 + θ△λ)△b)T (y(1) + θ△y) − 1

2(x(1) + θ△x)T Q(x(1) + θ△x)

= (b + λ1△b)T y(1) + θ(△λ△bT y(1) + (b + λ1△b)T△y) + θ2△λ△bT△y

− 1

2x(1)T

Qx(1) − θx(1)T

Q△x − 1

2θ2△xT Q△x. (4.4.10)

From equations (4.4.8) and (4.4.9), one gets

△xT Q△x = △λ(△bT△y −△cT△x), (4.4.11)

x(1)T

Q△x = (b + λ1△b)T△y −△λ△cT x(1). (4.4.12)

Substituting (4.4.11) and (4.4.12) into (4.4.10) we obtain

φ(λ) = φ(λ1 + θ△λ) = φ(λ1) + θ△λ(△bT y(1) + △cT x(1))

+1

2θ2△λ(△cT△x + △bT△y). (4.4.13)

Using the notation

γ1 = △bT y(1) + △cT x(1), (4.4.14)

γ2 = △bT y(2) + △cT x(2), (4.4.15)

γ =γ2 − γ1

λ2 − λ1

=△cT△x + △bT△y

λ2 − λ1

, (4.4.16)

one can rewrite (4.4.13) as

φ(λ) = (φ(λ1) − λ1γ1 +1

2λ2

1γ) + (γ1 − λ1γ)λ +1

2γλ2. (4.4.17)

Since λ1 and λ2 are two arbitrary elements from the interval (λℓ, λu), the claim

of the theorem follows directly from (4.4.17). ¤

It should be mentioned that the sign of △cT△x + △bT△y in (4.4.13) is

independent of λ1 and λ2, because both λ1 and λ2 are two arbitrary numbers in

(λℓ, λu). The following corollary is a straightforward consequence of (4.4.17).

72


Figure 4.2: The Optimal Value Function on Invariancy Interval

Corollary 4.4.6 For two arbitrary λ1 < λ2 ∈ (λℓ, λu), let (x(1), y(1), z(1)) and

(x(2), y(2), z(2)) be pairs of primal-dual optimal solutions corresponding to λ1 and

λ2, respectively. Moreover, let △x = x(2) − x(1) and △y = y(2) − y(1). Then, the

optimal value function φ(λ) is quadratic on O(π) = (λℓ, λu) and it is

(i) strictly convex if △cT△x + △bT△y > 0;

(ii) linear if △cT△x + △bT△y = 0;

(iii) strictly concave if △cT△x + △bT△y < 0.

Remark 4.4.7 For △c = 0 equation (4.4.17) reduces to the one presented in

Theorem 3.5 in Berkelaar et al. [3].

Remark 4.4.8 Note that π represents either an optimal partition at a transition

point, when λℓ = λu, or on the interval between two consequent transition points

λℓ and λu. Thus Λ =⋃

π Λ(π) =⋃

π Λ(π), where π runs throughout all possible

partitions.

Corollary 4.4.9 The optimal value function φ(λ) is continuous and piecewise

quadratic on Λ.

Proof. The fact that the optimal value function is piecewise quadratic follows

directly from Theorem 4.4.5. We need only to prove the continuity of φ(λ).

73


Continuity at interior points of any invariancy interval follows from (4.4.17). Let

λ∗ be the left transition point of the given invariancy interval (i.e., λ∗ = λℓ) and

(x(λ∗), y(λ∗), z(λ∗)) be a pair of primal-dual optimal solutions at λ∗. We need

to prove that limλ↓λ∗

φ(λ) = φ(λ∗). For any λ close enough to λ∗, there is a λ such

that for λ∗ < λ < λ with π(λ) = π(λ). Let (x(λ), y(λ), z(λ)) be a maximally

complementary optimal solution at λ and θ = λ∗−λ

λ∗−λ∈ (0, 1). We define

x(λ) = θx(λ) + (1 − θ)x(λ∗),

y(λ) = θy(λ) + (1 − θ)y(λ∗),

z(λ) = θz(λ) + (1 − θ)z(λ∗),

that shows the convergence of subsequence (x(λ), y(λ), z(λ)) to (x(λ∗), y(λ∗),

z(λ∗)) when θ goes to zero. As in our case λ∗ = λℓ, thus it follows from (4.4.6)

that x(λ)T z(λ) = θ(1 − θ)(x(λ)T z(λ∗) + z(λ)T x(λ∗)) = 0. It means that the

subsequence (x(λ), y(λ), z(λ)) is complementary.

We know that x(λ) = x(λ∗) + θ(x(λ) − x(λ∗)) and

φ(λ) = (c + λ△c)T x(λ) +1

2x(λ)T Qx(λ) = (c + λ△c)Tx(λ∗) +

1

2x(λ∗)T Qx(λ∗)

+ θ(c + λ△c)T (x(λ) − x(λ∗)) + θ(x(λ) − x(λ∗))T Qx(λ∗)

+1

2θ2(x(λ) − x(λ∗))T Q(x(λ) − x(λ∗)), (4.4.18)

with θ = λ∗−λ

λ∗−λ∈ (0, 1). When λ ↓ λ∗ (i.e., θ ↓ 0), we have φ(λ) → φ(λ∗) that

proves left continuity of the optimal value function φ(λ) at λ∗ = λℓ. Analogously,

one can prove right continuity of the optimal value function considering λ∗ = λu

by using problem (4.4.7) that completes the proof. ¤

4.4.2 Derivatives, Invariancy Intervals, and Transition

Points

In this subsection, the first and second order derivatives of the optimal value

function are studied. We also investigate the relationship between the invariancy

intervals and neighboring transition points where these derivatives may not exist.

74


Two auxiliary LO problems were presented in Theorem 4.4.4 to identify

transition points and consequently to determine invariancy intervals. The fol-

lowing theorem allows us to compute the left and right first order derivatives of

the optimal value function by solving another two auxiliary LO problems.

Theorem 4.4.10 For a given λ ∈ Λ, let (x∗, y∗, z∗) be a pair of primal-dual

optimal solutions of (QPλ) and (QDλ). Then, the left and right derivatives of

the optimal value function φ(λ) at λ satisfy

φ′−(λ) = min

x,y,z{△bT y : (x, y, z) ∈ QD∗

λ} + maxx

{△cT x : x ∈ QP∗λ}, (4.4.19)

φ′+(λ) = max

x,y,z{△bT y : (x, y, z) ∈ QD∗

λ} + minx

{△cT x : x ∈ QP∗λ}. (4.4.20)

Proof. Let ε be a sufficiently small real number. For any optimal solution

x(λ) ∈ QP∗λ and (x(λ), y(λ), z(λ)) ∈ QD∗

λ we have

φ(λ + ε) = (b + (λ + ε)△b)T y(λ + ε) − 1

2x(λ + ε)T Qx(λ + ε)

= (b + λ△b)T y(λ) − 1

2x(λ)T Qx(λ) + ε(△bT y(λ) + △cT x(λ + ε))

+ x(λ + ε)T s(λ) +1

2(x(λ + ε) − x(λ))T Q(x(λ + ε) − x(λ))

≥ (b + λ△b)T y(λ) − 1

2x(λ)T Qx(λ) + ε(△bT y(λ) + △cT x(λ + ε))

= φ(λ) + ε(△bT y(λ) + △cT x(λ + ε)), (4.4.21)

where the constraints of (QDλ) and (QPλ+ε) were used. Analogously, the con-

straints of (QPλ) and (QDλ+ε) imply

φ(λ + ε) ≤ φ(λ) + ε(△bT y(λ + ε) + △cT x(λ)). (4.4.22)

We prove only (4.4.20). The proof of (4.4.19) goes analogously. Using (4.4.21)

and (4.4.22) for a positive ε we derive the following inequality

△bT y(λ)+△cTx(λ+ ε) ≤ φ(λ + ε) − φ(λ)

ε≤ △bT y(λ+ ε)+△cTx(λ). (4.4.23)

75


For any ε small enough, there is a λ such that λ < λ + ε < λ and λ + ε and λ

are in the same invariancy interval. Let (x(λ), y(λ), z(λ)) be an optimal solution

at λ and for θ = λ−λ

λ+ε−λ> 1, we define

x(θ) = θx(λ + ε) + (1 − θ)x(λ),

y(θ) = θy(λ + ε) + (1 − θ)y(λ),

z(θ) = θz(λ + ε) + (1 − θ)z(λ),

which shows that when ε ↓ 0 (θ ↓ 1) the subsequence (x(λ+ε), y(λ+ε), z(λ+ε))

converges to (x(1), y(1), z(1)) = (x, y, z). Since λ + ε and λ are in the same

invariancy interval, thus xT z = θ(1 − θ)(x(λ + ε)T z(λ) + z(λ + ε)T x(λ)) = 0

shows that x is a primal optimal solution of (QPλ) and (x, y, z) is a dual optimal

solution of (QDλ). Letting ε ↓ 0 we get

limε↓0

△cT x(λ + ε) = △cT x and limε↓0

△bT y(λ + ε) = △bT y. (4.4.24)

From (4.4.23) and (4.4.24) one can easily obtain the inequality

△bT y(λ) + minx(λ)∈QP∗

λ

△cT x(λ) ≤ φ′+(λ) ≤ max

y(λ)∈QD∗

λ

△bT y(λ) + △cT x(λ). (4.4.25)

Since x(λ) is any optimal solution of (QPλ) and (x(λ), y(λ), z(λ)) is any optimal

solution of (QDλ), from (4.4.25) we conclude that

maxy(λ)∈QD∗

λ


λ

△cT x(λ) ≤ φ′+(λ)

and

φ′+(λ) ≤ max

y(λ)∈QD∗

λ


λ

△cT x(λ).

Then

φ′+(λ) = max

y(λ)∈QD∗

λ


λ

△cT x(λ),

completing the proof. ¤

76


Remark 4.4.11 It is worthwhile to make some remarks about Theorem 4.4.10.

We consider problem (4.4.20) only. Similar results hold for problem (4.4.19).

Let

QPD = {(x, y, z) : Ax = b + λ△b, x ≥ 0, xT z∗ = 0,

AT y + z − Qx = c + λ△c, z ≥ 0, zT x∗ = 0}.

First, in the definition of the set QPD the constraints x ≥ 0, xT z∗ = 0 and

z ≥ 0, zT x∗ = 0 are equivalent to xB ≥ 0, xN∪T = 0 and zN ≥ 0, zB∪T = 0,

where (B,N , T ) is the optimal partition in the transition point λ. Second, let

us consider the first and the second subproblems of (4.4.20). Observe that the

optimal solutions produced by each subproblem are both optimal for (QPλ) and

(QDλ) and so the vector Qx, appearing in the constraints, is always identical

for both subproblems (see, e.g., Dorn [7]). This means that we can maximize the

first subproblem over the dual optimal set QD∗λ only and minimize the second

subproblem over the primal optimal set QP∗λ only. In other words, instead of

solving two subproblems in (4.4.20) separately, we can solve the problem

minx,y,z

{△cT x −△bT y : (x, y, z) ∈ QPD} (4.4.26)

that produces the same optimal solution (x, y, z) as problem (4.4.20). There-

fore, the right derivative φ′+(λ) can be computed by using the values (x, y, z)

as φ′+(λ) = △bT y + △cT x. Consequently, we refer to the optimal solutions of

problems (4.4.20) and (4.4.26) interchangeably.

The following theorem shows that if λ is not a transition point, then the

optimal value function is differentiable and the derivative can be given explicitly.

Theorem 4.4.12 If λ is not a transition point, then the optimal value function

at λ is a differentiable quadratic function and its first order derivative is

φ′(λ) = △bT y(λ) + △cT x(λ).

77


Proof. The result can be established directly by differentiating the optimal

value function given by (4.4.17). ¤

If we have non-degenerate primal and dual optimal solutions at transition

point λ, the left and right first order derivatives coincide and can be computed by

using the formula given in Theorem 4.4.12 as well. It follows from the fact that

in such situation, the optimal solution of both the primal and the dual problems

(QPλ) and (QDλ) are unique.

The next lemma shows an important property of strictly complementary

solutions of (4.4.19) and (4.4.20) and will be used later on in this chapter.

Lemma 4.4.13 Let λ∗ be a transition point of the optimal value function. Fur-

ther, assume that the (open) invariancy interval to the right of λ∗ contains λ

with the optimal partition π = (B,N , T ). Let (x, y, z) be an optimal solution of

(4.4.20) with λ = λ∗. Then, σ(x) ⊆ B and σ(z) ⊆ N .

Proof. Let (x, y, z) be a maximally complementary solution at λ and let

(λ∗, x, y, z) be an optimal solution of (4.4.6) where the optimal partition is π.

First, we want to prove that

△cT x = △cT x and △bT y = △bT y, (4.4.27)

cT x = cT x and bT y = bT y. (4.4.28)

For this purpose we use equation (4.4.13). In (4.4.13) let λ2 = λ, x(2) = x,

y(2) = y. The continuity of the optimal value function, proved in Corollary 4.4.9,

allows us to establish that equation (4.4.13) holds not only on invariancy in-

tervals, but also at their endpoints, i.e., at the transition points. Thus, we are

allowed to consider the case when λ1 = λ∗ and (x(1), y(1), z(1)) is any optimal

solution at the transition point λ∗.

78


Computing φ(λ) at the point λ (where θ = λ−λ1

λ2−λ1= λ−λ∗

λ−λ∗= 1) by (4.4.13)

gives us

φ(λ) = φ(λ∗) + (λ − λ∗)(△bT y(1) + △cT x(1))

+1

2(λ − λ∗)[△cT (x − x(1)) + △bT (y − y(1))] (4.4.29)

= φ(λ∗) +1

2(λ − λ∗)[△cT (x + x(1)) + △bT (y + y(1))].

One can rearrange (4.4.29) as

φ(λ) − φ(λ∗)

λ − λ∗= △cT

(

x + x(1)

2

)

+ △bT

(

y + y(1)

2

)

.

Let λ ↓ λ∗, then we have

φ′+(λ∗) = lim

λ↓λ∗

φ(λ) − φ(λ∗)

λ − λ∗= △cT

(

x + x(1)

2

)

+ △bT

(

y + y(1)

2

)

. (4.4.30)

Since (x(1), y(1), z(1)) is an arbitrary optimal solution at λ∗ and φ′+(λ∗) is in-

dependent of the optimal solution choice at λ∗, one may choose (x(1), y(1), z(1)) =

(x, y, z) and (x(1), y(1), z(1)) = (x, y, z). From (4.4.30) we get

φ′+(λ∗) = △cT

(

x + x

2

)

+ △bT

(

y + y

2

)

= △cT

(

x + x

2

)

+ △bT

(

y + y

2

)

. (4.4.31)

Equation (4.4.31) reduces to △cT (x+x

2) = △cT x from which it follows that

△cT x = △cT x. Furthermore, let us consider (x(1), y(1), z(1)) = (x, y, z) and

(x(1), y(1), z(1)) = (x, y, z). From (4.4.30) we obtain △bT y = △bT y.

Now, since both (x, y, z) and (x, y, z) are optimal solutions in QP∗λ∗×QD∗

λ∗ ,

it holds that (c + λ∗△c)T x = (c + λ∗△c)T x and (b + λ∗△b)T y = (b + λ∗△b)T y

(see e.g., Dorn [7]). Consequently, it follows from (4.4.27) that cT x = cT x and

bT y = bT y.

79


As a result we can establish that

xT z = xT (c + λ△c + Qx − AT y) = cT x + λ△cT x + xT Qx − (b + λ∗△b)T y

= cT x + λ△cT x + xT Qx − (Ax)T y = xT (c + λ△c + Qx − AT y)

= xT z = 0, (4.4.32)

and

xT z = xT (c + λ∗△c + Qx − AT y) = xT (c + λ∗△c + Qx) − bT y − λ△bT y

= xT (c + λ∗△c + Qx) − bT y − λ△bT y = xT (c + λ∗△c + Qx − AT y)

= xT z = 0. (4.4.33)

For θ ∈ (0, 1) and λ = (1 − θ)λ∗ + θλ, let us consider

x = (1 − θ)x + θx,

y = (1 − θ)y + θy, (4.4.34)

z = (1 − θ)z + θz.

Utilizing equations (4.4.34) and the complementarity properties (4.4.32) and

(4.4.33), we obtain that x and (x, y, z) are feasible and complementary, and

thus optimal solutions of (QPλ) and (QDλ), respectively. Noting that (B,N , T )

is the optimal partition at (x, y, z), it follows from (4.4.34) that xB ≥ 0, xN = 0,

xT = 0 and zB = 0, zN ≥ 0, zT = 0. Then we can conclude that σ(x) ⊆ B and

σ(z) ⊆ N . ¤

Utilizing two auxiliary linear optimization problems we can also calculate

the left and right second order derivatives of φ(λ) [13].

The following theorem, that summarizes the results we got up to now, is

a direct consequence of Theorem 4.4.1 (equivalence of (i) and (ii)), the defi-

nition of a transition point (equivalence of (ii) and (iii)), and Corollary 4.4.3

80


Figure 4.3: Neighboring Invariancy Intervals

and Lemma 4.4.13 (equivalence of (iii) and (iv)). The proof is identical to the

proof of Theorem 3.10 in Berkelaar et al. [3] and it also shows that in adjacent

subintervals φ(λ) is defined by different quadratic functions.

Theorem 4.4.14 The following statements are equivalent:

(i) Dπ = ∅;(ii) Λ(π) = {λ∗};(iii) λ∗ is a transition point;

(iv) φ′ or φ′′ is discontinuous at λ∗.

By solving an auxiliary self-dual quadratic optimization problem one can obtain

the optimal partition in the neighboring invariancy interval. The result is given

by the next theorem. Figure 4.3 provides graphical interpretation of this result.

Theorem 4.4.15 Let λ∗ be a transition point of the optimal value function.

Let (x∗, z∗) be an optimal solution of (4.4.20) for λ∗. Let us assume that the

(open) invariancy interval to the right of λ∗ contains λ with optimal partition

π = (B,N , T ), and (x, y, z) is a maximally complementary solution at λ. Define

T = σ(x∗, z∗) = {1, 2, . . . , n} \ (σ(x∗) ∪ σ(z∗)). Consider the following self-dual

quadratic problem

minξ,ρ,η

{−△bT η + △cT ξ + ξT Qξ : Aξ = △b, AT η + ρ − Qξ = △c,

ξσ(z∗) = 0, ρσ(x∗) = 0, ξσ(x∗,z∗) ≥ 0, ρσ(x∗,z∗) ≥ 0},(4.4.35)

81


and let (ξ∗, η∗, ρ∗) be a maximally complementary solution of (4.4.35). Then,

B = σ(x∗) ∪ σ(ξ∗), N = σ(z∗) ∪ σ(ρ∗) and T = {1, . . . , n} \ (B ∪N ).

Proof. For any feasible solution of (4.4.35) we have

−△bT η + △cT ξ + ξT Qξ = ξT (Qξ − AT η + △c) = ξT ρ = ξTT ρT ≥ 0.

The dual of (4.4.35) is

maxδ,ξ,γ,ζ

{ △bT δ −△cT ζ − ξT Qξ : Aζ = △b, AT δ + γ + Qζ − 2Qξ = △c,

γσ(z∗) = 0, ζσ(x∗) = 0, γT ≥ 0, ζT ≥ 0}.

For a feasible solution it holds

△bT δ −△cT ζ − ξT Qξ = ξT Aζ −△cT ζ − ξT Qξ = −ζT γ − (ζ − ξ)T Q(ζ − ξ) ≤ 0.

So, the optimal value of (4.4.35) is zero. Let us assign

ξ = ζ = x − x∗, ρ = γ = z − z∗, η = δ = y − y∗, (4.4.36)

that satisfy the first two linear constraints of (4.4.35). Then, problem (4.4.35) is

feasible and self-dual.

Using the fact that by Lemma 4.4.13 σ(x∗) ⊆ B and σ(z∗) ⊆ N , it follows

that

ξσ(z∗) = xσ(z∗) − x∗σ(z∗) = 0, ξT = xT − x∗

T = xT ≥ 0

and

ρσ(x∗) = zσ(x∗) − z∗σ(x∗) = 0, ρT = zT − z∗T = zT ≥ 0.

From the proof of Lemma 4.4.13 we have xT z∗ = zT x∗ = 0, implying that (4.4.36)

is an optimal solution. The fact that (x, y, z) is maximally complementary shows

that (4.4.36) is maximally complementary solution in (4.4.35) as well. For x =

x∗ + ξ, we need to consider four cases to determine B:

82


1. x∗i > 0 and ξi > 0;

2. x∗i > 0 and ξi = 0;

3. x∗i > 0 and |ξi| < x∗

i ;

4. x∗i = 0 and ξi > 0.

One can easily check that B = σ(x∗) ∪ σ(ξ∗). Analogous arguments hold

for N , that completes the proof. ¤

4.4.3 Computational Algorithm

In this subsection we summarize the results in a computational algorithm. This

algorithm is capable of finding the transition points; the right first order deriva-

tives of the optimal value function at transition points; and optimal partitions at

all transition points and invariancy intervals. Note that the algorithm computes

all these quantities to the right from the given initial value λ∗. One can easily

outline an analogous algorithm for the transition points to the left from λ∗. It

is worthwhile to mention that all the subproblems used in this algorithm can be

solved in polynomial time by IPMs.

Algorithm: Transition Points, First-Order Derivatives

of the Optimal Value Function and Optimal Partitions

at All Subintervals for Convex QO

input:

a nonzero direction of perturbation: r = (△b,△c);

a maximally complementary solution (x∗, y∗, z∗) of (QPλ) and (QDλ) for λ∗;

π0=(B0,N 0, T 0), where B0=σ(x∗), N 0=σ(z∗), T 0={1, . . . , n}\ (B0∪N 0);

k := 0; x0 := x∗; y0 := y∗; z0 := z∗;

83


ready:= false;

begin

while not ready do

begin

solve λk = maxλ,x,y,z{λ : Ax − λ△b = b, xBk ≥ 0, xN k∪T k = 0,

AT y + z−Qx−λ△c = c, zN k ≥ 0, zBk∪T k = 0};if this problem is unbounded, ready:=true

else let (λk, xk, yk, zk) be an optimal solution;

begin

let x∗ := xk and z∗ := zk;

solve minx,y,z{△cT x −△bT y : (x, y, z) ∈ QPDk}if this problem is unbounded, ready:= true;

else let (xk, yk, zk) be an optimal solution;

begin

let x∗ := xk and z∗ := zk;

solve minξ,ρ,η{−△bT η + △cT ξ + ξT Qξ : Aξ = △b,

AT η + ρ − Qξ = △c, ξσ(z∗) = 0,

ρσ(x∗) = 0, ξσ(x∗,z∗) ≥ 0, ρσ(x∗,z∗) ≥ 0};Bk+1 = σ(x∗) ∪ σ(ξ∗), N k+1 = σ(z∗) ∪ σ(ρ∗),

T k+1 = {1, . . . , n} \ (Bk+1 ∪N k+1);

k := k + 1;

end

end

end

end

84


4.5 Simultaneous Perturbation in Linear

Optimization

The case, when perturbation occurs in the objective function vector c or the right-

hand side vector b of an LO problem was extensively studied. A comprehensive

survey can be found in the book of Roos, Terlaky and Vial [26]. Greenberg [10]

has investigated simultaneous perturbation of the objective and right-hand side

vectors when the primal and dual LO problems are formulated in canonical form.

He proved some properties of the optimal value function in that case and showed

that the optimal value function is piecewise quadratic.

Theorems 4.5.1–4.5.3 resemble Greenberg’s findings, but they are presented

in the manner more suitable for implementation.

We start this section by emphasizing the differences in the optimal parti-

tions of the optimal value function in LO and QO problems and then proceed to

specialize our results to the LO case. Let us define the simultaneous perturbation

of a LO problem as

(LPλ) min { (c + λ△c)T x : Ax = b + λ△b, x ≥ 0 } .

Its dual is

(LDλ) max { (b + λ△b)T y : AT y + z = c + λ△b, z ≥ 0 } .

The LO problem can be derived from the convex QO problem by substitut-

ing zero matrix for Q. As a result, vector x does not appear in the constraints

of the dual problem, and the set T in the optimal partition is always empty.

The following theorem shows that to identify an invariancy interval, we

don’t need to solve problems (4.4.6) and (4.4.7) as they are formulated for the

85


QO case. Its proof is based on the fact that the constraints in these problems

separate when Q = 0 and is omitted.

Theorem 4.5.1 Let λ∗ ∈ Λ be given and let (x∗, y∗, z∗) be a strictly com-

plementary optimal solution of (LPλ∗) and (LDλ∗) with the optimal partition

π = (B,N ). Then, the left and right extreme points of the interval Λ(π) = [λℓ, λu]

that contains λ∗ are λℓ = max {λPℓ, λDℓ

} and λu = min {λPu, λDu

}, where

λPℓ= min

λ,x{λ : Ax − λ△b = b, xB ≥ 0, xN = 0},

λPu= max

λ,x{λ : Ax − λ△b = b, xB ≥ 0, xN = 0},

λDℓ= min

λ,y,z{λ : AT y + z − λ△c = c, zN ≥ 0, zB = 0},

λDu= max

λ,y,z{λ : AT y + z − λ△c = c, zN ≥ 0, zB = 0}.

We also state the following lemma that does not hold for QO problems but only

for LO case.

Lemma 4.5.2 Let λℓ and λu be obtained from Theorem 4.5.1 and (x(ℓ), y(ℓ), z(ℓ))

and (x(u), y(u), z(u)) be any strictly complementary solutions of (LPλ) and (LDλ)

corresponding to λℓ and λu, respectively. Then it holds that

△bT△y = △cT△x,

where △y = y(u) − y(ℓ) and △x = x(u) − x(ℓ).

Proof. Subtracting the constraints of (LPλℓ) from (LPλu

) and the constraints

of (LDλℓ) from (LDλu

) results in

A△x = △λ△b, (4.5.1)

AT△y + △z = △λ△c, (4.5.2)

86


where △λ = λu − λℓ and △z = z(u) − z(ℓ). Premultiplying (4.5.1) by △yT and

(4.5.2) by △xT , the result follows from the fact that △xT△z = 0, completing

the proof. ¤

Utilizing Lemma 4.5.2 and using the same notation as in (4.4.14)–(4.4.16),

we can state the following theorem that gives explicit expressions for computing

the objective value function. The theorem also gives the criteria to determine

convexity, concavity and linearity of the objective value function on its subinter-

vals.

Theorem 4.5.3 Let λ1 < λ2 and π(λ1) = π(λ2) = π, let (x(1), y(1), z(1)) and

(x(2), y(2), z(2)) be strictly complementary optimal solutions of problems (LPλ)

and (LDλ) at λ1 and λ2, respectively. The following statements hold:

(i) The optimal partition is invariant on (λ1, λ2).

(ii) The optimal value function is quadratic on this interval and is given by

φ(λ) = (φ(λ1) − λ1γ1 +1

2λ2

1γ) + (γ1 − λ1γ)λ +1

2γλ2

= φ(λ1) + θ△λ(△bT y(1) + △cT x(1)) + θ2△λ△cT△x

= φ(λ1) + θ△λ(△bT y(1) + △cT x(1)) + θ2△λ△bT△y

(iii) On any subinterval, the objective value function is

• strictly convex if △cT△x = △bT△y > 0,

• linear if △cT△x = △bT△y = 0,

• strictly concave if △cT△x = △bT△y < 0.

Computation of derivatives can be done by solving smaller LO problems

than the problems introduced in Theorem 4.4.10. The following theorem sum-

marizes these results.

87


Theorem 4.5.4 For a given λ ∈ Λ, let (x∗, y∗, z∗) be a pair of primal-dual opti-

mal solutions of (LPλ) and (LDλ). Then, the left and right first order derivatives

of the optimal value function φ(λ) at λ are

φ′−(λ) = min

y,z{△bT y : AT y + z = c + λ△c, z ≥ 0, zT x∗ = 0}

+ maxx

{△cT x : Ax = b + λ△b, x ≥ 0, xT z∗ = 0},

φ′+(λ) = max

y,z{△bT y : AT y + z = c + λ△c, z ≥ 0, zT x∗ = 0}

+ minx

{△cT x : Ax = b + λ△b, x ≥ 0, xT z∗ = 0}.

88

Chapter 5

Implementation of Parametric

Quadratic Optimization

Implementation details of the parametric LO/QO package McParam are de-

scribed in this chapter. Section 5.1 illustrates the desired output of the paramet-

ric solver on a simple parametric QO problem. Implementation of the algorithm

and optimal partition determination are subjects of Secton 5.2. The structure of

the McParam package is considered in Section 5.3. Sections 5.4 and 5.5 contain

computational results and their analysis.

5.1 Illustrative Example

Here we present an illustrative numerical example that shows the desired output

of a parametric solver based on the algorithm outlined in Section 4.4.3. Com-

putations related to finding optimal solutions of auxiliary subproblems can be

performed by using any IPM solver for LO and convex QO problems. Let us

consider the following convex QO problem with x, c ∈ R5, b ∈ R

3, Q ∈ R5×5

being a positive semidefinite symmetric matrix, A ∈ R3×5 with rank(A) = 3.

89


Table 5.1: Transition Points, Invariancy Intervals, and Optimal Partitions forthe Illustrative Problem

Inv. Intervals and Tr. Points B N T φ(λ)λ = −8.0 {3,5} {1,4} {2}

−8.0 < λ < −5.0 {2,3,5} {1,4} ∅ 68.0λ + 8.5λ2

λ = −5.0 {2} {1,3,4,5} ∅−5.0 < λ < 0.0 {1,2} {3,4,5} ∅ −50.0 + 35.5λ + 4λ2

λ = 0.0 {1,2} ∅ {3,4,5}0.0 < λ < 1.739 {1,2,3,4,5} ∅ ∅ −50.0 + 35.5λ − 6.9λ2

λ = 1.739 {2,3,4,5} ∅ {1}1.739 < λ < 3.333 {2,3,4,5} {1} ∅ −40.0 + 24.0λ − 3.6λ2

λ = 3.333 {3,4,5} {1} {2}3.333 < λ < +∞ {3,4,5} {1,2} ∅ 0

The problem data are

Q =

4 2 0 0 02 5 0 0 00 0 0 0 00 0 0 0 00 0 0 0 0

, c =

−16−20

000

, △c =

76000

,

A =

2 2 1 0 02 1 0 1 02 5 0 0 1

, b =

118

20

, △b =

111

.

With this data the perturbed convex QO problem (QPλ) is

min (−16 + 7λ)x1 + (−20 + 6λ)x2 + 2x21 + 2x1x2 + 5

2x2

2

s.t. 2x1 + 2x2 + x3 = 11 + λ2x1 + x2 + x4 = 8 + λ2x1 + 5x2 + x5 = 20 + λx1, x2, x3, x4, x5 ≥ 0.

(5.1.1)

The computational results we are interested in are presented in Table 5.1.

The set Λ for the optimal value function φ(λ) is [−8, +∞). Figure 5.1 depicts the

graph of φ(λ). Transition points and the optimal partitions at each transition

point and on the invariancy intervals are identified by solving the problems in

90


−8 −6 −4 −2 0 2 4 6−140

−120

−100

−80

−60

−40

−20

0

20

λ

φ(λ)

Figure 5.1: The Optimal Value Function for Illustrative Problem

Theorem 4.4.4, Remark 4.4.11 and Theorem 4.4.15 according to the algorithm

from Section 4.4.3. The optimal value function on the invariancy intervals is

computed by using formula (4.4.17). Convexity, concavity or linearity of the

optimal value function can be determined by the sign of the quadratic term of

the optimal value function (see Table 5.1). As shown in Figure 5.1, the optimal

value function is convex on the first two invariancy intervals, concave on the

third and fourth and linear on the last one. The first order derivative does not

exists at transition point λ = −5.

91


5.2 Implementing the Parametric Algorithm

for QO

Theoretical developments presented in Chapter 4 allowed us to develop an ef-

ficient algorithm for solving parametric QO problems. As we have seen, the

algorithm has polynomial time complexity as all the subproblems can be effi-

ciently solved with an IPM.

5.2.1 Implementation Algorithm

In this section one can find the algorithm which represents an extended version of

that in Section 4.4.3 but adapted for implementation purposes. This algorithm

defines the structure of the McParam package.

Implementation Algorithm for Parametric QO

input:

a quadratic problem (QP ) in the standard form: A, Q, c and b;

a nonzero direction of perturbation: r = (△b,△c);

initial value λ0 for the parameter λ (optional);

begin

solve the parametric problem for λ = λ0 to find a maximally complementary

solution (x0, y0, z0) of (QPλ) and (QDλ);

recover the optimal partition π0 = (B0,N 0, T 0) at λ = λ0, where B0 = σ(x0),

N 0 = σ(z0) and T 0 = {1, . . . , n} \ (B0 ∪N 0);

Solve parametric QO for λ > λ0:

k := 0;

stop:= false;

while not stop do

begin

92


solve problem (PQO1):

λk = maxλ,x,y,z{λ : Ax − λ△b = b, xBk ≥ 0, xN k∪T k = 0,

AT y + z − Qx − λ△c = c, zN k ≥ 0, zBk∪T k = 0};if (PQO1) is unbounded then stop:=true

else let (λk, x, y, z) be an optimal solution of (PQO1);

begin


minx,y,z{△cT x −△bT y :

Ax = b + λk△b, xσ(x)∪σ(x,z) ≥ 0, xσ(z) = 0,

AT y + z−Qx = c+λk△c, zσ(z)∪σ(x,z) ≥ 0, zσ(x) = 0};if (PQO2) is infeasible then stop:= true;

else let (x∗, y∗, z∗) be an optimal solution of (PQO2);

begin


minξ,ρ,η{−△bT η + △cT ξ + ξT Qξ : Aξ = △b,

AT η + ρ − Qξ = △c, ξσ(s∗) = 0,

ρσ(x∗) = 0, ξσ(x∗,z∗) ≥ 0, ρσ(x∗,z∗) ≥ 0};let (ξ∗, ρ∗, η∗) be an optimal solution of (PQO3);

Bk+1 = σ(x∗) ∪ σ(ξ∗), N k+1 = σ(z∗) ∪ σ(ρ∗),

T k+1 = {1, . . . , n} \ (Bk+1 ∪N k+1);

k := k + 1;

end

end

Solve parametric QO for λ < λ0:

k := 0;

stop:= false;

Proceed analogously as for λ > λ0;

end

end

93


As the implementation algorithm for the parametric QO problem requires

some more explanations, we provide them in the next paragraphs. McParam

chooses an initial value for λ if it is not specified by a user. Usually λ0 = 0

is good enough as we already know that the non-parametric quadratic problem

(λ = 0) has an optimal solution. In the situations when the quadratic solver

produces imprecise solution at λ0 = 0 and it is difficult to recover the optimal

partition based on this solution, we choose another value of λ0. It is wise to try

values of λ0 being close to zero as the problem can become infeasible if we move

far from it. Such a situation is discussed in Section 5.2.2.

For determining the invariancy intervals and transition points, we first find

the optimal partition π0 = (B0,N 0, T 0) at the initial point. After that, we solve

the parametric problem for the values of λ > λ0 until the parametric problem

becomes unbounded or infeasible. Analogously, we repeat the same procedure

for the values of λ to the left from the initial point λ0.

Problem (PQO1) does not require much explanation. It allows us to get

the next transition point λk and an optimal solution (x, y, z) in that point. We

need this optimal solution to get the support sets σ(x), σ(z) and σ(x, z) =

{1, . . . , n} \ (σ(x) ∪ σ(z)) which are used as inputs for the problem (PQO2).

The derivative subproblem (PQO2) can be more challenging than it seems.

The difficulties here are caused by the fact that we want to solve the derivative

subproblem without knowing the optimal partition at the current transition point

λk. This is actually the reason why we have nonnegativity constraints xσ(x,z) ≥ 0

and zσ(x,z) ≥ 0 in the problem (PQO2). Presence of these constraints reflects

the fact that we do not actually know to which tri-partition Bk, N k or T k the

indices σ(x, z) will belong. It is the consequence of not having a maximally com-

plementary solution at the current transition point λk. This imply that we need

94


to somehow enforce the hidden constraint (xσ(x,z))j (zσ(x,z))j = 0 ∀ j ∈ σ(x, z)

for the problem (PQO2). In contrast, if we know maximally complementary so-

lution of the parametric problem for λk and consequently the optimal partition

(Bk, N k, T k) at the transition point λk, the problem (PQO2) becomes:

minx,y,z

△cT x −△bT y

(PQO′2) s.t. Ax = b + λk△b, xBk ≥ 0, xN k∪T k = 0,

AT y + z − Qx = c + λk△c, zN k ≥ 0, zBk∪T k = 0.

Our computational experience shows that if (xσ(x,z))j > 0 and (zσ(x,z))j > 0 for

some j, then B = xσ(x) and N = zσ(z) in that transition point and we just use

this partition while solving (PQO2).

Finally, if we obtain an optimal solution for the problem (PQO2), we can

use the union of its support set with the support set for the problem (PQO3) to

find the optimal partition on the invariancy interval to the right from the current

one. The steps for the points to the left from λ0 are derived analogously.

5.2.2 Determining Optimal Partitions and Support Sets

Determination of the optimal partition for a given maximally complementary

optimal solution or determination of the support set for a given optimal solution

is a challenging task because of numerical reasons. From Chapter 4 we know

that for a given maximally complementary solution (x∗, y∗, z∗):

i ∈ B if z∗i = 0 and x∗i > 0,

i ∈ N if x∗i = 0 and z∗i > 0,

i ∈ T if x∗i = 0 and z∗i = 0.

Unfortunately, numerical solution produced by a LO/QO solver may not

allow to determine the optimal partition or support set in 100% of cases. So, we

introduce a zero tolerance parameter tol zero (the default value is 10−4, which

95


−40 −20 0 20 40 60 80 100 120 140 1602.684

2.686

2.688

2.69

2.692

2.694

2.696

2.698

2.7x 10

7

λ

φ(λ)

Figure 5.2: Optimal Partition Determination Counterexample

performs quite well in practice), and compare the entries of the vectors x∗ and

z∗ to it. As a result, we adopt the following strategy for determining the optimal

partition (support set):

if xi ≤ tol zero and zi ≤ tol zero then i ∈ T

elseif xi > tol zero and zi < tol zero then i ∈ B

elseif xi < tol zero and zi > tol zero then i ∈ N

elseif xi ≥ zi then i ∈ B

else i ∈ N

96


The methodology described above does not give the desired results in all

cases. Even in the linear case when the partition consists of two sets B and

N only, the task is not easy. In the quadratic case the tri-partition introduces

even more complications as there are 3 sets and the differences between the

entries are smaller. We do not provide any probabilistic analysis of the partition

determination in this thesis, but it can be performed.

Below we show an example of the case when determination of the optimal

partition in a correct way is difficult for a range of starting points. Consider the

problem qscagr7 from the Maros and Meszaros test set and apply the following

perturbation vectors △c = (1, 1, 0, 1,−1, 0, . . . , 0)T and △b = (1,−1, 0, . . . , 0)T .

For that perturbation vectors this problem has one invariancy interval (see Fig-

ure 5.2). Only one coordinate (i = 152) causes problems on the parameter in-

terval λ ∈ (−0.14, 3.98). On this interval the variable x152 becomes numerically

slightly larger than z152 = 0.0016667 and that causes i = 152 to be moved from

the set N to B for the mentioned above interval. This, in turn, causes the op-

timal partition to be determined wrongly and, consequently, leads to computing

incorrect parametric interval, if any.

5.3 Structure of the McParam Package

5.3.1 McParam Arguments and Output

The McParam package for solving parametric quadratic optimization problems

is implemented in Matlab and can be called from it by the following command

[output,exitflag] = mcparam(A,b,db,c,dc,Q,l,u,du,lambda0,options),

where all input arguments except for A, b, c and dc or db are optional. If matrix

Q is not provided, the problem is treated as linear. If no lower bounds l and/or

97


upper bounds u are provided, then the lower bound is set to zero for all variables

and the upper bound is set to +∞. You can notice that upper bound u can be also

perturbed by du. In that case we have one more pair of primal and dual variables

s and w and the size of the index set becomes (n + mf ). Consequently, the

partition and support sets are determined based on this enlarged set consisting

of the variables (xT , sT )T and (zT , wT )T . The argument lambda0 is the initial

value for the parameter λ. The description of options parameter, that allows

us to select the LO/QO solver employed for solving auxiliary subproblems and

specify the output file for saving results, is given in Appendix A. The output of

the package contains the indicator exitflag that gives us the solution status of

the parametric problem (0 corresponds to infeasible problem for initial λ and 1 to

successfully terminated algorithm). The structure output contains the following

fields:

output(i).type type of the invariancy interval i (’invariancyinterval’ or ’transition point’);

output(i).B optimal partition set B;output(i).N optimal partition set N ;output(i).T optimal partition set T ;output(i).lambdaleft left end of the invariancy interval i;output(i).lambdaright right end of the invariancy interval i;output(i).lambda value of λ in the transition point i;output(i).obj 3D vector representing the optimal value function

φ(λ) = aλ2 + bλ + c on the invariancy interval i;output(i).lderiv value of the left derivative in the transition point i

(if available);output(i).rderiv value of the right derivative in the transition point i

(if available);output(i).x optimal solution vector x in the transition point i;output(i).y optimal solution vector y in the transition point i;output(i).z optimal solution vector z in the transition point i.

Optimal value function output(i).obj is computed based on the objec-

tive function values at the endpoints of the invariancy interval and left (right)

98


derivative in one of the endpoints.

It is worthwhile to make the following notes about the output produced by

the McParam package:

• The problem is preprocessed using the same preprocessing techniques as

described in Section 3.1.2. All the subproblems solved later on are utilizing

the results of preprocessing. The optimal partitions are computed by the

package in terms of the preprocessed problem.

• Note that the optimal partition reported in the transition points may not

be complete. The determination of the optimal partition in a transition

point is based on the union of the support sets obtained from solving prob-

lems (PQO1) and (PQO2), which does not necessarily gives the optimal

partition, but provides its best ”approximation” based on the available

information. This allows avoiding solving an additional parametric QO

problem in the transition point to compute the optimal partition. But

even though such problem is solved, the optimal partition can be deter-

mined only subject to numerical values of the optimal solution and may

not be complete as well.

• Optimal partition reported for parametric LO problems in the transition

points may include set T . As we know for LO problems the optimal par-

tition π = (B,N ) consists of two sets only. So, all the indices appearing

in the set T are the indices which are not determined to be in either B or

N as we do not want to solve the additional optimization problem in the

transition point.

99


5.3.2 McParam Flow Chart

Taking into account all above information we can summarize the actions taken

by McParam in the flow chart on Figure 5.3. As we see from Figure 5.3, the

precautions are taken against numerical difficulties which can arise while solving

LO/QO auxiliary subproblems. First, the difficulties can occur while solving

problem (PQO1) using incorrectly determined optimal partition (Section 5.2.2).

In that case we choose a random value of λ close to the initial λ0 and try to resolve

problem (QPλ) in attempt to recover the optimal partition correctly. Second, if

numerical difficulties occur while solving problems (PQO2) or (PQO3) we follow

the same strategy: choose a value of λ close to the current one, for instance,

λk+1 = λk + ǫ (where ǫ > 0 is a small number), if we are moving to the right

from λ0, and solve problem (QPλk+1) to determine the optimal partition in order

to proceed further.

5.4 Computational Results

In this section we show the performance of our McParam package on a set of

parametric convex QO problems. As there are no other solvers that can perform

analogous analysis, we cannot compare our software with another packages. That

is why we only present the computational results produced by McParam on

selected QO/LO problems for the illustration purposes. All computations are

performed on a Windows PC with Pentium IV 3.0 GHz processor and 1Gb RAM.

Variable t in the McParam printouts plays the role of λ, phi(t) gives the the

optimal value function, type is the interval type (invariancy interval or transition

point).

Tables 5.2 and 5.3 contain the McParam output for the perturbed quadratic

100


Solve (QP ) for

Error computing optimal partition

Choose another

Find optimal partition (Bk,Nk,Tk)

Solve (PQO1)

(PQO1) is primal infeasible

Parametric interval is

unbounded

No

Yes

(PQO1) is dual infeasible

Yes

No (PQO1) is optimal

Solve (PQO2)

Yes

(PQO2) is optimal

Yes

Solve (PQO3)

No(PQO2) is infeasible

End of parametric

interval

Yes No

(PQO3) is optimal

Yes

No(PQO3) is infeasibleYes

No

End of parametric interval. Stop

End of parametric

interval

Numerical difficulties. Stop

Input datak = 0

No

k = k + 1

Recover from numerical difficulties:

, k = k + 1

0l

kll

ell +=+ kk 1

Figure 5.3: Flow Chart of the McParam Package

101


Table 5.2: McParam Output for Perturbed lotschd Problem (QO Formulation)

type tl tu B N T phi(t)

------------------------------------------------------------------------------------------------

tr. point -19.38488 -19.38488 1 3 5 7 9 11 2 6 8 10 12 4 3076.87621

inv. interv -19.38488 -9.11811 1 3 4 5 7 9 11 2 6 8 10 12 2.5t2+27.9t+2671.6

tr. point -9.11811 -9.11811 1 3 4 5 7 9 11 2 8 10 12 6 2626.26944

inv. interv -9.11811 -7.31937 1 3 4 5 6 7 9 11 2 8 10 12 -3.6t2-84.4t+2159.5

tr. point -7.31937 -7.31937 1 3 5 6 7 9 11 2 8 10 12 4 2582.09702

inv. interv -7.31937 +0.30244 1 3 5 6 7 9 11 2 4 8 10 12 0.8t2-19.1t+2398.4

tr. point +0.30244 +0.30244 1 3 5 6 7 9 11 2 4 8 10 12 2392.71830

inv. interv +0.30244 +11.00000 1 3 5 6 7 9 11 12 2 4 8 10 0.3t2-18.8t+2398.4

tr. point +11.00000 +11.00000 3 5 6 7 9 11 12 1 2 4 8 10 2223.41981

inv. interv +11.00000 +18.06373 2 3 5 6 7 9 11 12 1 4 8 10 1.6t2-11.6t+2157.9

tr. point +18.06373 +18.06373 2 3 5 6 7 9 11 1 4 8 10 12 2469.53014

inv. interv +18.06373 +44.49132 2 3 5 6 7 9 11 1 4 8 10 12 2.9t2-58.2t+2578.8

tr. point +44.49132 +44.49132 2 3 5 7 9 11 1 4 8 10 12 6 5706.58820

Table 5.3: McParam Output for Perturbed lotschd Problem (LO Formulation)

type tl tu B N T phi(t)

------------------------------------------------------------------------------------------------

tr. point -19.38488 -19.38488 1 3 5 7 9 11 2 6 8 10 12 4 75.45527

inv. interv -19.38488 -0.00000 1 3 4 5 7 9 11 2 6 8 10 12 0.3t2+0.9t+0.0

tr. point -0.00000 -0.00000 1 3 4 5 7 9 11 12 2 6 8 10 0.00000

inv. interv -0.00000 +11.00000 1 3 5 6 7 9 11 2 4 8 10 12 -0.9t2-21.3t-0.0

tr. point +11.00000 +11.00000 3 5 6 7 9 11 1 2 4 8 10 12 -343.02008

inv. interv +11.00000 +44.49132 2 3 5 6 7 9 11 1 4 8 10 12 1.0t2-42.2t+0.0

tr. point +44.49132 +44.49132 2 3 5 7 9 11 1 4 8 10 12 6 99.96501

and linear formulations of lotschd problem from the Maros and Meszaros test

set. Optimal value functions graphed by McParam are shown at Fugures 5.4

and 5.5. Problem lotschd is a small-size problem, with 12 variables and 7 con-

straints and that is why McParam prints optimal partition information. For

larger problems, only intervals and corresponding optimal value functions are

shown.

Tables 5.4 and 5.5 provide the printout of McParam output for the per-

turbed quadratic and linear formulations of medium-size qsc205 problem

(317 variables and 205 constarints). Fugures 5.6 and 5.7 graph the optimal

value functions for it.

Small and medium-size problem are handled pretty well by McParam. For

102


−20 −10 0 10 20 30 402000

2500

3000

3500

4000

4500

5000

5500

6000

λ

φ(λ)

Figure 5.4: Optimal Value Function for Perturbed lotschd QO Problem

−20 −10 0 10 20 30 40−500

−400

−300

−200

−100

0

100

λ

φ(λ)

Figure 5.5: Optimal Value Function for Perturbed lotschd LO Problem

103


Table 5.4: McParam Output for Perturbed qsc205 Problem (QO Formulation)

type tl tu phi(t)

--------------------------------------------------------------------

tr. point -200.00000 -200.00000 -0.00000

inv. interval -200.00000 -198.05825 27.0t2+10600.0t+1040000.0

tr. point -198.05825 -198.05825 -286.54916

inv. interval -198.05825 -189.47368 5.95t2+2261.9t+214285.7

tr. point -189.47368 -189.47368 -664.81994

inv. interval -189.47368 -13.50000 -0.02t2-0.0t-0.0

tr. point -13.50000 -13.50000 -3.37500

inv. interval -13.50000 -0.02326 -0.02t2-0.0t-0.01

tr. point -0.02326 -0.02326 -0.00581

inv. interval -0.02326 +0.30814 -0.0t2+0.0t-0.01

tr. point +0.30814 +0.30814 -0.00581

inv. interval +0.30814 +0.49057 -0.1t2+0.1t-0.0

tr. point +0.49057 +0.49057 -0.00926

inv. interval +0.49057 +94.54545 -0.1t2+0.0t-0.0

tr. point +94.54545 +94.54545 -343.80165

inv. interval +94.54545 +100.00000 12.9t2-2444.4t+115555.6

tr. point +100.00000 +100.00000 -0.00000

−200 −150 −100 −50 0 50 100−700

−600

−500

−400

−300

−200

−100

0

λ

φ(λ)

Figure 5.6: Optimal Value Function for Perturbed qsc205 QO Problem

104


Table 5.5: McParam Output for Perturbed qsc205 Problem (LO Formulation)

type tl tu phi(t)

------------------------------------------------------------

tr. point -200.00000 -200.00000 -0.00000

inv. interval -200.00000 -100.00000 1.0t2+200.0t+0.0

tr. point -100.00000 -100.00000 -10000.00000

inv. interval -100.00000 -1.00000 0.0t2+100.0t-0.0

tr. point -1.00000 -1.00000 -100.00000

inv. interval -1.00000 -0.66667 2.0t2+102.0t-0.0

tr. point -0.66667 -0.66667 -67.11111

inv. interval -0.66667 -0.26282 -1.0t2+28.3t-47.8

tr. point -0.26282 -0.26282 -55.30182

inv. interval -0.26282 -0.25537 -0.7t2+25.3t-48.6

tr. point -0.25537 -0.25537 -55.11111

inv. interval -0.25537 -0.24484 -0.6t2+17.7t-50.5

tr. point -0.24484 -0.24484 -54.92099

inv. interval -0.24484 +0.03057 -0.4t2+11.0t-52.2

tr. point +0.03057 +0.03057 -51.86571

inv. interval +0.03057 +0.50000 1.1t2-31.4t-50.9

tr. point +0.50000 +0.50000 -66.33333

inv. interval +0.50000 +100.00000 1.3t2-133.3t-0.0

tr. point +100.00000 +100.00000 0.00000

−200 −150 −100 −50 0 50 100

−10000

−9000

−8000

−7000

−6000

−5000

−4000

−3000

−2000

−1000

0

λ

φ(λ)

Figure 5.7: Optimal Value Function for Perturbed qsc205 LO Problem

105


Table 5.6: McParam Output for Perturbed aug3dc Problem (QO Formulation)

type tl tu phi(t)

---------------------------------------------------------

tr. point +0.00000 +0.00000 -1165.23756

inv. interval +0.00000 +1.10980 -0.3t2-0.3t-1165.2

tr. point +1.10980 +1.10980 -1165.91114

inv. interval +1.10980 Inf 0.1t2-1.1t-1164.8

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5−1169

−1168.5

−1168

−1167.5

−1167

−1166.5

−1166

−1165.5

−1165

λ

φ(λ)

Figure 5.8: Optimal Value Function for Perturbed aug3dc QO Problem

106


the large-size parametric problems (number of variables n ≥ 1000), numerical

difficulties occur often, especially while solving quadratic self-dual subproblem

(PQO3) due to problems with determination of optimal partitions and support

sets. The example of such case is shown in Table 5.6 and Fugure 5.8. For the

large-size perturbed QO problem aug3dc (7746 variables and 1000 constarints)

the quadratic self-dual subproblem (PQO3) is not solved in the transition point

λ = 1.1098 because of numerical difficulties and so, the strategy for recovering

the optimal partition described in Section 5.3.2 is used.

5.5 Analysis of Results

Summarizing shortly the computational experience of the McParam package the

following conclusions can be drawn.

• Performance of the parametric quadratic solver depends on the number of

variables. For small- and medium-size problems it performs well, but large-

size problems represent significant challenge. It mostly happens because

of numerical troubles occurring when solving auxiliary subproblems or due

to difficulties with determination of the optimal partition or support sets.

From our computational experience, it especially applies to the self-dual

quadratic problem (PQO3). Note that problem (PQO3) seems to be twice

as large as the original QO problem, but because of its self-duality property

it contains only two more rows than the original problem.

• Robustness of the McParam package is an important issue. It was designed

to handle unexpected situations. The package tries to find the invariancy

intervals, optimal value function, etc. on the whole set Λ. McParam makes

attempts to recover when some of the auxiliary subproblems are not solved

107


by a QO/LO solver. McParam mostly fails only if QO/LO solver fails.

• Difficulties in determining the optimal partition can be overcome by the

strategy described in Section 5.3. In addition, we can use other strategies

for determining the optimal partition, such as doing heuristic analysis of

the problematic indices or solving the parametric problem of the reduced

size by eliminating primal and dual variables, which are determined to be

in either set B or N .

• Both QO/LO solvers, MOSEK and McIPM, currently used for solving aux-

iliary subproblems, perform comparably well. Still, MOSEK seems to pro-

vide better solution precision for some problems and is faster on many

problems.

108

Chapter 6

Conclusions and Further Work

In this thesis we considered techniques for solving quadratic optimization prob-

lems and their parametric counterparts. We modified the linear optimization

software package McIPM to be able to solve quadratic problems. Consequently,

McIPM became the tool for solving auxiliary problems in our parametric algo-

rithm, but any other IPMs solvers can be used for that purpose as well. Testing

results show that our implementation is competitive with non-commercial and

even commercial quadratic optimization software.

We extended the existing theory of parametric quadratic optimization to

simultaneous perturbation sensitivity analysis when the variation occurs in both

the right-hand side vector of the constraints and the coefficient vector of the

objective function linear term. In our analysis the rate of variation, represented

by the parameter λ, was identical for both perturbation vectors △b and △c.

One of the main results is that the invariancy intervals and transition points can

be determined by solving auxiliary linear or quadratic problems. This means

that we should not worry about ”missing” short-length invariancy intervals. As

we already mentioned, all auxiliary problems can be solved in polynomial time.

Finally, we developed and implemented the algorithm that represents a sequence

109


of linear and quadratic auxiliary problems to identify all invariancy intervals and

graph the optimal value function.

The implementation of the algorithm for parametric optimization and test-

ing results suggest that our implementation is well suitable for solving small- and

medium-size parametric quadratic problems and can be used for large-scale prob-

lems as well with some caution connected to the precision of solutions produced

by a quadratic solver. As none of the existing software packages can perform

parametric analysis for quadratic optimization, we illustrated our implementa-

tion on our own set of parametric quadratic problems.

As it was mentioned in the introduction, the most famous application of

the convex QO sensitivity analysis is the mean-variance portfolio optimization

problem introduced by Markowitz [17]. Our method allows us to analyze not

only the original Markowitz model, but also some of its extensions. One possible

extension is when the investors’ risk aversion parameter λ influences not only

risk-return preferences, but also budget constraints. One more practical problem,

where our methodology can be used, is coming from engineering. It consists in

optimizing power distribution between users of digital subscriber lines (DSL).

The variation of λ represents variation of noisiness.

There are many possible extensions and future work in both theoretical

and implementation directions. I just want to mention some of them.

(i) As we know the number of transition points for the parametric quadratic

problem is finite, but can grow exponentially in the number of variables.

The finite number of transition points is determined by the finite number

of partitions of the index set. It would be interesting to establish an upper

bound on the number of transition points which can possibly depend on

the problem dimension in primal and dual spaces.

110


(ii) There are some results recently developed for parametric Convex Conic Op-

timization, where the variables are restricted to belong to cones. Yildirim

[34] extended the concept of the optimal partition to conic problems. Con-

sequently, one of the further research directions can include generalizing

the analysis of this thesis to Second-Order Cone Optimization (SOCO) and

Semidefinite Optimization (SDO) problems. The first priority here would

be to generalize the methodology and develop an algorithm for simultane-

ous perturbation sensitivity analysis for Second-Order Cone Optimization

problems. It is also worthwhile exploring parametric SOCO applications

to financial models.

(iii) All the analysis in this thesis is done for single parameter perturbation. It

may be also interesting to look at the problem (QPλb,λc) where the param-

eter λ is different for the objective function and for the constraints. This

case can be generalized further to multiparametric quadratic optimization.

(iv) A number of studies exist for multiparametric quadratic optimization where

λ is a vector (see, e.g., [28]). Multiparametric QO is widely applied in

optimal control. No doubt that the analysis is becoming more complicated

when we work in multidimensional parameter space λ. We would like to

look at the possibility of extending our results and the algorithm to n-

dimensional parametric optimization.

From the practical or implementation side we want to suggest the following

future directions:

(iv) Make the implementation of McParam more robust and improve the solu-

tion precision of the linear system in McIPM using iterative refinement.

(v) Experiment with warm-start strategy.

111


(vi) Extend the implementation to SOCO case. We already have a SOCO

package (McSOCO) developed in the Advanced Optimization Laboratory

by B. Wang [30] that can become the base solver for such implementation.

112

Appendix A

McIPM and McParam Options

This appendix contains the description of the command-line options of the

McIPM and McParam packages. Options are specified according to Matlab

conventions. Furthermore, McIPM output parameter exitflag values are also

described.

113


McIPM Options

Display Level of display. ’off’ displays no output; ’iter’ (de-fault) displays output at each iteration; ’final’ displaysthe final output only.

Diagnostics Print diagnostic information such as MPS/QPS readerstatistics, preprocessing information, ordering time, etc.:’No’ or ’Yes’ (default).

MaxIter Maximum number of iterations allowed (default is 150).

McParam Options

Solver LO/QO solver to be used for solving auxiliary subprob-lems. Currently two solvers are available, ’MOSEK’ and’McIPM’, the first one is default.

Outfile Filename for saving computational results.

McIPM exitflag Values

2 NONFIRM OPTIMAL Non-firm optimal solution found.1 FIRM OPTIMAL Optimal solution found.0 MAXITER UNKNOWN Maximum number of iterations ex-

ceeded or status unknown.-1 PRIMAL INFEAS Primal infeasibility detected.-2 DUAL INFEAS Dual infeasibility detected.-3 PRIMDUAL INFEAS Primal-dual infeasibility detected.-4 NONFIRM PRIMAL INFEAS Non-firm primal infeasibility detected.-5 NONFIRM DUAL INFEAS Non-firm dual infeasibility detected.-6 NONFIRM PRIMDUAL INFEAS Non-firm primal-dual infeasibility de-

tected.

114

Appendix B

Maros and Meszaros Test Set

This appendix contains the description of the Maros and Meszaros [18] test set

of convex quadratic optimization problems. The set currently contains 138 prob-

lems. In the tables the following notation is used:

Name – problem name;m – number of rows in matrix A;n – number of variables;NZ A – number of nonzeros in A;NZ Q diag – number of diagonal entries in Q;NZ Q off-diag – number of off-diagonal entries in the lower

triangular part of Q;Prepr. Rows – number of rows in matrix A after McIPM

preprocessing;Prepr. Cols – number of variables after McIPM preprocessing.

115


Table B.1: The Maros and Meszaros QO Test Set (I)

Problem Statistics McIPM Preprocessing

Name m n NZ A NZ Q diag NZ Q off-diag Prepr. Rows Prepr. Colsaug2d 10000 20200 40000 19800 0 10000 40400aug2dc 10000 20200 40000 20200 0 10000 40400aug2dcqp 10000 20200 40000 20200 0 10000 20200aug2dqp 10000 20200 40000 19800 0 10000 20200aug3d 1000 3873 6546 2673 0 1000 7746aug3dc 1000 3873 6546 3873 0 1000 7746aug3dcqp 1000 3873 6546 3873 0 1000 3873aug3dqp 1000 3873 6546 2673 0 1000 3873boyd1 18 93261 558985 93261 0 18 93261boyd2 186531 279785 423784 2 0 186531 279785cont-050 2401 2597 12005 2597 0 2401 2597cont-100 9801 10197 49005 10197 0 9801 10197cont-101 10098 10197 49599 2700 0 10098 10197cont-200 39601 40397 198005 40397 0 39601 40397cont-201 40198 40397 199199 10400 0 40198 40397cont-300 90298 90597 448799 23100 0 90298 90597cvxqp1 l 5000 10000 14998 10000 29984 5000 10000cvxqp1 m 500 1000 1498 1000 2984 500 1000cvxqp1 s 50 100 148 100 286 50 100cvxqp2 l 2500 10000 7499 10000 29984 2500 10000cvxqp2 m 250 1000 749 1000 2984 250 1000cvxqp2 s 25 100 74 100 286 25 100cvxqp3 l 7500 10000 22497 10000 29984 7500 10000cvxqp3 m 750 1000 2247 1000 2984 750 1000cvxqp3 s 75 100 222 100 286 750 100dpklo1 77 133 1575 77 0 77 266dtoc3 9998 14999 34993 14997 0 9998 29994dual1 1 85 85 85 3473 1 85dual2 1 96 96 96 4412 1 96dual3 1 111 111 111 5997 1 111dual4 1 75 75 75 2724 1 75dualc1 215 9 1935 9 36 215 223dualc2 229 7 1603 7 21 229 235dualc5 278 8 2224 8 28 278 285dualc8 503 8 4024 8 28 503 510exdata 3001 3000 7500 1500 1124250 3001 6000genhs28 8 10 24 10 9 8 20gouldqp2 349 699 1047 349 348 349 699gouldqp3 349 699 1047 698 697 349 699hs118 17 15 39 15 0 17 32hs21 1 2 2 2 0 1 3hs268 5 5 25 5 10 5 15hs35 1 3 3 3 2 1 4hs35mod 1 3 3 3 2 1 3hs51 3 5 7 5 2 3 10hs52 3 5 7 5 2 3 10

116


Table B.2: The Maros and Meszaros QO Test Set (II)


Name m n NZ A NZ Q diag NZ Q off-diag Prepr. Rows Prepr. Colshs53 3 5 7 5 2 3 5hs76 3 4 10 4 2 3 7hues-mod 2 10000 19899 10000 0 2 10000huestis 2 10000 19899 10000 0 2 10000ksip 1001 20 18411 20 0 1001 1041laser 1000 1002 3000 1002 3000 1000 3004liswet1 10000 10002 30000 10002 0 10000 30004liswet10 10000 10002 30000 10002 0 10000 30004liswet11 10000 10002 30000 10002 0 10000 30004liswet12 10000 10002 30000 10002 0 10000 30004liswet2 10000 10002 30000 10002 0 10000 30004liswet3 10000 10002 30000 10002 0 10000 30004liswet4 10000 10002 30000 10002 0 10000 30004liswet5 10000 10002 30000 10002 0 10000 30004liswet6 10000 10002 30000 10002 0 10000 30004liswet7 10000 10002 30000 10002 0 10000 30004liswet8 10000 10002 30000 10002 0 10000 30004liswet9 10000 10002 30000 10002 0 10000 30004lotschd 7 12 54 6 0 7 12mosarqp1 700 2500 3422 2500 45 700 1457mosarqp2 600 900 2930 900 45 600 1233powel20 10000 10000 20000 10000 0 10000 30000primal1 85 325 5815 324 0 85 734primal2 96 649 8042 648 0 96 1393primal3 111 745 21547 744 0 111 1600primal4 75 1489 16031 1488 0 75 3052primalc1 9 230 2070 229 0 9 254primalc2 7 231 1617 230 0 7 240primalc5 8 287 2296 286 0 8 304primalc8 8 520 4160 519 0 8 545q25fv47 820 1571 10400 446 59053 798 1854qadlittl 56 97 383 17 70 55 137qafiro 27 32 83 3 3 27 51qbandm 305 472 2494 25 16 269 436qbeaconf 173 262 3375 18 9 148 270qbore3d 233 315 1429 28 50 199 300qbrandy 220 249 2148 16 49 149 259qcapri 271 353 1767 56 838 267 476qe226 223 282 2578 67 897 220 469qetamacr 400 688 2409 378 4069 357 692qfffff80 524 854 6227 278 1638 501 1005qforplan 161 421 4563 36 546 135 463qgfrdxpn 616 1092 2377 54 108 600 1144qgrow15 300 645 5620 38 462 300 645qgrow22 440 946 8252 65 787 440 946qgrow7 140 301 2612 30 327 140 301

117


Table B.3: The Maros and Meszaros QO Test Set (III)


Name m n NZ A NZ Q diag NZ Q off-diag Prepr. Rows Prepr. Colsqisrael 174 142 2269 42 656 174 316qpcblend 74 83 491 83 0 74 114qpcboei1 351 384 3485 384 0 347 722qpcboei2 166 143 1196 143 0 140 279qpcstair 356 467 3856 467 0 356 538qpilotno 975 2172 13057 94 391 951 2242qptest 2 2 4 2 1 2 4qrecipe 91 180 663 20 30 85 177qsc205 205 203 551 11 10 205 317qscagr25 471 500 1554 28 100 471 671qscagr7 129 140 420 8 17 129 185qscfxm1 330 457 2589 56 677 322 592qscfxm2 660 914 5183 74 1057 644 1184qscfxm3 990 1371 7777 89 1132 966 1776qscorpio 388 358 1426 22 18 388 358qscrs8 490 1169 3182 33 88 485 1270qscsd1 77 760 2388 54 691 77 760qscsd6 147 1350 4316 96 1308 147 1350qscsd8 397 2750 8584 140 2370 397 2750qsctap1 300 480 1692 36 117 300 660qsctap2 1090 1880 6714 141 636 1090 2500qsctap3 1480 2480 8874 186 861 1480 3340qseba 515 1028 4352 96 550 515 1036qshare1b 117 225 1151 18 21 112 248qshare2b 96 79 694 10 45 96 162qshell 536 1775 3556 405 34385 496 1487qship04l 402 2118 6332 14 42 356 2162qship04s 402 1458 4352 14 42 268 1414qship08l 778 4283 12802 940 34025 688 4339qship08s 778 2387 7114 538 11139 416 2171qship12l 1151 5427 16170 2023 60205 838 5329qship12s 1151 2763 8178 1042 16361 466 2293qsierra 1227 2036 7302 122 61 1222 2715qstair 356 467 3856 66 952 356 538qstandat 359 1075 3031 138 666 359 1258s268 5 5 25 5 10 5 15stadat1 3999 2001 9997 2000 0 3999 8001stadat2 3999 2001 9997 2000 0 3999 8001stadat3 7999 4001 19997 4000 0 7999 16001stcqp1 2052 4097 13338 4097 22506 770 1368stcqp2 2052 4097 13338 4097 22506 1881 2644tame 1 2 2 2 1 1 2ubh1 12000 18009 48000 6003 0 12000 29991values 1 202 202 202 3620 1 202yao 2000 2002 6000 2002 0 2000 5999zecevic2 2 2 4 1 0 2 4

118

Bibliography

[1] I. Adler and R. Monteiro. A geometric view of parametric linear program-

ming. Algorithmica, 8:161–176, 1992.

[2] E. D. Anderson and Y. Ye. On a homogeneous algorithm for the monotone

complementarity problem. Mathematical Programming, 84:375–399, 1999.

[3] A. B. Berkelaar, B. Jansen, C. Roos, and T. Terlaky. An interior point

approach to parametric convex quadratic programming. Working paper,

Erasmus University Rotterdam, Rotterdam, The Netherlands, 1997.

[4] A. B. Berkelaar, C. Roos, and T. Terlaky. The optimal set and opti-

mal partition approach to linear and quadratic programming. Chapter 6 in

Advances in Sensitivity Analysis and Parametric Programming, T. Gal and

H. J. Greenberg, eds., Kluwer Academic Publishers, Boston, USA, 1997.

[5] J. C. G. Boot. Quadratic Programming: Algorithms, Anomalies and Appli-

cations. North-Holland Publishing Company, Amsterdam, The Netherlands,

1964.

[6] T. Davis. Algorithm 8xx: a concise sparse Cholesky factorization packages.

Submitted to ACM Trans. on Math. Software, 2004.

119


[7] W. S. Dorn. Duality in quadratic programming. Quarterly of Applied Math-

ematics, 18:155–162, 1960.

[8] D. E. Farrar. The Investment Decision Under Uncertainty. Prentice-Hall,

New York, 1965.

[9] Financial Glossary ; http://www.investorwords.com, accessed 26 July

2004.

[10] H. J. Greenberg. Simultaneous primal-dual right-hand-side sensitivity

analysis from a strictly complementary solution of a linear program. SIAM

Journal of Optimization, 10:427–442, 2000.

[11] H. J. Greenberg. Mathematical Programming Glossary, 1996-2004, s.v. ”Sen-

sitivity analysis”; http://carbon.cudenver.edu/~hgreenbe/glossary/,

accessed 26 July 2004.

[12] O. Guler and Y. Ye. Convergence behavior of interior-point algorithms.

Mathematical Programming, 60:215–228, 1993.

[13] A. Ghaffari Hadigheh, O. Romanko, and T. Terlaky. Sensitivity Analysis

in Convex Quadratic Optimization: Simultaneous Perturbation of the Ob-

jective and Right-Hand-Side Vectors. Technical Report #2003/6, Advanced

Optimization Laboratory, McMaster University, Hamilton, Canada, 2003.

Submitted to Optimization.

[14] N. Gould and P. Toint. Preprocessing for quadratic programming. Mathe-

matical Programming, 100:95–132, 2004.

120


[15] X. Huang. Preprocessing and Postprocessing in LO. M. Sc. Thesis, Advanced

Optimization Laboratory, Department of Computing and Software,

McMaster University, Hamilton, Canada, 2004.

[16] B. Jansen, J. J. de Jong, C. Roos, and T. Terlaky. Sensitivity analysis

in linear programming: just be careful! European Journal of Operational

Research, 101:15–28, 1997.

[17] H. M. Markowitz. The optimization of a quadratic function subject to linear

constraints. Naval Research Logistics Quarterly, 3:111–133, 1956.

[18] I. Maros and C. Meszaros. A repository of convex quadratic optimiza-

tion problems. Departmental Technical Report DOC 97/6, Department of

Computing, Imperial College, London, UK, 1997.

[19] L. McLinden. An analogue of Moreau’s proximation theorem, with appli-

cation to the nonlinear complementary problem. Pacific Journal of Mathe-

matics, 88:101–161, 1980.

[20] S. Mehrotra. On the implementation of a primal-dual interior point method,

SIAM Journal on Optimization, 2:575-601, 1992.

[21] C. Meszaros. The separable and non-separable formulations of con-

vex quadratic problems in interior-point methods. Working paper 98-3,

Hungarian Academy of Sciences, Budapest, Hungary, 1998.

[22] C. Meszaros. The BPMPD interior point solver for convex quadratic prob-

lems. Working paper 98-8, Hungarian Academy of Sciences, Budapest,

Hungary, 1998.

121


[23] B. Murtagh. Advanced Linear Programming: Computation and Practice.

McGraw-Hill, New York, USA, 1981.

[24] K. G. Murty. Linear Programming. John Wiley & Sons, New York, USA,

1983.

[25] J. Peng, C. Roos, and T. Terlaky. Self-Regularity: A New Paradigm for

Primal-Dual Interior-Point Methods. Princeton University Press, 2002.

[26] C. Roos, T. Terlaky, and J.-Ph. Vial. Theory and Algorithms for Linear

Optimization: An Interior Point Approach. John Wiley & Sons, Chichester,

UK, 1997.

[27] T. Seol. Solving Linear Equations in Interior-Point Methods. Intermedi-

ate Technical Report, Advanced Optimization Laboratory, Department of

Computing and Software, McMaster University, Hamilton, Canada, 2004.

http://www.cas.mcmaster.ca/~oplab/seminar/students/slides/seol.pdf

[28] P. Tondel, T. A. Johansen, and A. Bemporad. Further results on multipara-

metric quadratic programming. Proc. 42th IEEE Conf. on Decision and

Control, Maui, Hawaii, USA, 3173-3178, 2003.

[29] R. J. Vanderbei. Symmetric quasi-definite matrices. SIAM Journal on

Optimization, 5(1):100113, 1995.

[30] B. Wang. Implementation of Interior-Point Methods for Second-Order

Conic Optimization. M. Sc. Thesis, Advanced Optimization Laboratory,

Department of Computing and Software, McMaster University, Hamilton,

Canada, 2003.

122


[31] N. Yamashita and Z.-Q. Luo. A Nonlinear Complementarity Approach

to Multiuser Power Control for Digital Subscriber Lines. To appear in

Optimization Methods and Software, 2004.

[32] Y. Ye, M. J. Todd, and S. Mizuno. An O(√

nL) iteration homogeneous

and self-dual linear programming algorithm. Mathematics of Operations

Research, 19:52–67, 1994.

[33] Y. Ye. Interior Point Algorithms, Theory and Analysis. John Wiley,

Chichester, UK, 1997.

[34] E. A. Yildirim. A unifying optimal partition approach to sensitivity analysis

in conic optimization. To appear in Journal of Optimization Theory and

Applications, 2004.

[35] X. Zhu. Implementing the New Self-Regular Proximity Based IPMs. M. Sc.

Thesis, Advanced Optimization Laboratory, Department of Computing and

Software, McMaster University, Hamilton, Canada, 2002.

[36] X. Zhu, J. Peng, T. Terlaky, and G. Zhang. On Implementing Self-Regular

Proximity Based Feasible IPMs. Technical Report #2004/3, Advanced

Optimization Laboratory, McMaster University, Hamilton, Canada, 2004.

Submitted to INFORMS Journal on Computing.

123

Date post:	20-Jul-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

QUADRATIC AND PARAMETRIC QUADRATIC OPTIMIZATIONoplab/thesis/romankoMS.pdf · term generally refers...

Documents