March 7, 2002 1
Using Pattern Recognition Using Pattern Recognition Techniques to Derive a Techniques to Derive a Formal Analysis of Why Formal Analysis of Why
Heuristic Functions WorkHeuristic Functions Work
B. John OommenA Joint Work with Luis G. Rueda
School of Computer ScienceCarleton University
March 7, 2002 2
Optimization Problems
Any arbitrary optimization problem:
• Instances, drawn from a finite set, X,
• An Objective function
• Some feasibility functions
The aim:
• Find an (hopefully the unique) instance of X,
• which leads to a maximum (or minimum)
• subject to the feasibility constraints.
March 7, 2002 3
An Example
The Traveling Salesman Problem (TSP)• Consider the cities numbered from 1 to n,• The salesman starts from city 1,• visits every city once, and • Returns to city 1.
An instance of X is a permutation of cities:For example, 1 4 3 2 5, if five cities considered
The objective function:• The sum of the inter-city distances:• 1 4, 4 3, 3 2, 2 5, 5 1
March 7, 2002 4
Heuristic FunctionsA Heuristic algorithm is an algorithm
which attempts to find a certain
instance X
that maximizes the objective function
It iteratively invokes a Heuristic function.
The heuristic function estimates (or
measures) the cost of the solution.
The heuristic itself is a method that
performs one or more changes to the
current instance.
March 7, 2002 5
An Open Problem
Consider a Heuristic algorithm that invokes any of
Two Heuristic Functions : H1 and H2
used in estimating the solution to an
Optimization problem
If Estimation accuracy of H1 >
Estimation accuracy of H2
Does it imply that
H1 has higher probability of leading to the
optimal QEP?
March 7, 2002 6
Pattern Recogniton Modeling
Two heuristic functions : H1 and H2
Probability of choosing a cost value of a Solution:two independent random variables: X1 and X2
Distribution -- doubly exponential:
c
where ,
and
cX ][E
2
2][Var
X
||
2
1)( cxexf
X
March 7, 2002 7
Pattern Recogniton Modeling
Our model:
Error function is doubly exponential.
Typical in reliability analysis and failure models.
How reliable is a Solution when only estimate known?
Assumptions:Mean cost of Optimal Solution: , then
shift the origin by E[X] = 0Variances:
Estimate X1 better than Estimate of X2
][Var22
][Var 222
21
1 XX
March 7, 2002 8
Main Result (Exponential) H1 and H2, two heuristic functions.
X1 and X2, two r.v. optimal solution obtained by H1 and
H2
X1’ and X2
’, other two r.v. for sub-optimal solution
cXEXXEX ]'[]'[E][][E0 2121
Let p1 and p2 the prob. that H1 and H2 respectively make the wrong decision.
Shown that:
]'[Var][Var22
]'[Var][Var if 2222
21
11 XXXX
then : 21 pp
March 7, 2002 9
Proof (Graphical Sketch)For a particular x, the prob. that x leads to wrong decision by H1 is given by:
I e dux
u c11
1
2 11
( )
X1(opt) X1
(subopt)
X2(subopt)X2
(opt)
March 7, 2002 10
Proof (Cont’d)
or
X1(opt) X1
(subopt)
X2(subopt)X2
(opt)
I e du e duc
u c
c
x
u c1 2
1
2
1
211
11
( ) ( )
if x < c
March 7, 2002 11
Proof (Cont’d)The total probability that H1 makes the wrong decision for all values of x is:
p I e I e I ex
c
x
c
x1 11
0
1 110
1 1 2 1
1
21 1 1
Similarly, the prob. that H2 makes the wrong decision for all values of x is:
p I e I e I exc
x
c
x2 11
0
2 110
2 1 2 2
1
22 2 2
March 7, 2002 12
Proof (Cont’d)
Solving integrals and making p1 p2, we have:
02
1ln2
1ln),( 11111
kkkF
which, using ln x x - 1, implies that p1 p2 QED
where 1 = 1 c and 2 = 2 c
Also 2 substituted for k 1
March 7, 2002 13
Second Theorem
F(1,k) can also be written in terms of 1 and k as:
02
1
2
1),( 1111
111 eeekekG kk
Suppose that 1 0 and 0 k 1,then G(1,k) 0, and
there are two solutions for G(1,k) = 0
}1,1{ 1 k },0{ 1 kand
Proof:Taking partial derivatives and solving:
and 0),( 1
k
kG 0),(
1
1
kG
March 7, 2002 14
Graphical Analysis (Histograms)
),( 1 kG
1k
R-ACM / Eq-width R-ACM / Eq-depthT-ACM / Eq-widthT-ACM / Eq-depth
G >>> 0, orp1 <<< p2
R-ACM / T-ACM Eq-width / Eq-depthG 0, or p1 p2
Minimum in 1 = 0 and 0 k 1
March 7, 2002 15
Analysis : Normal Distn’s)(
1suboptX
)(
2
suboptX
)(1
optX
)(
2
optX
No integration possible for the normal pdf
Shown numerically that p1 p2
March 7, 2002 16
Plot of the Function G
March 7, 2002 17
Estimation for Histograms
is estimated as where N is the # of samples
N
i ix
N
1||
March 7, 2002 18
Similarities of R-ACM and d-Exp
Estimated for RACM
True d-Exp
March 7, 2002 19
Simulations Details
Simulations performed in Query Optimization:
4 independent runs per simulation.
100 random Databases per run 400 per simulation.
6 Relations,
6 Attributes per relation,
100 tuples per relation.
Four independent runs on 100 databases:
R-ACM vs. Traditional using:
11 bins, 50 values
March 7, 2002 20
Empirical Results
Run R > W W > R R > D D > R
1 26 12 35 12
2 24 15 42 15
3 35 11 46 8
4 29 15 46 8
Total 114 53 169 41
# of times in which R-ACM
yields better QEP
# of times in which Eq-width
yields better QEP
# of times in which Eq-depth
yields better QEP
March 7, 2002 21
Conclusions
Applied PR Techniques to solve problem of relating Heuristic Function Accuracy and Solution Optimality
Used a reasonable model of accuracy (doubly exponential distribution).
Shown analytically how the high accuracy of heuristic function leads to a superior solutions.
Numerically shown the results for normal distributions
Shown that R-ACM yield better QEPs in a larger number of times than Equi-width and Equi-depth.
Empirical results on randomly generated databases also shown the superiority of R-ACM.
Graphically demonstrated the validity of our model.