Uncertainty in Expert Systems

CPS 4801

Uncertainty• Uncertainty is the lack of exact knowledge

that would enable us to reach a fully reliable solution.oClassical logic assumes perfect

knowledge exists:IF A is trueTHEN B is true

• Describing uncertainty:o If A is true, then B is true with

probability P

Sources of uncertainty• Weak implications: Want to be able to

capture associations and correlations, not just cause and effect.

• Imprecise language: o How often is “sometimes”?o Can we quantify “often,” “sometimes,” “always?”

• Unknown data: In real problems, data is often incomplete or missing.

• Differing experts: Experts often disagree, or have different reasons for agreeing.o Solution: attach weight to each expert

Two approaches• Bayesian reasoning

o Bayesian rule (Bayes’ rule) by Thomas Bayeso Bayesian network (Bayes network)

• Certainty factors

Probability Theory

• The probability of an event is the proportion of cases in which the event occurso Numerically ranges from zero to unity (an

absolute certainty) (i.e. 0 to 1)

P(success) + P(failure) = 1

the number of possible outcomes

the number of successessuccess)P(

the number of possible outcomesfailure)P(

the number of failures

Example• Flip a coin• P(head) = ½ P(tail) = ?• P(head) = ¼ P(tail) = ?

• Throw a dice• P(getting a 6) = ?• P(not getting a 6) = ?

• P(A) = p P(¬A) = 1-p

Example• P(head) = ½ P(head head head) = ?

• Xi = result of i-th coin flip Xi = {head, tail}

• P(X1 = X2 = X3 = X4) = ?

• Until now, events are independent and mutually exclusive.

• P(X,Y) = P(X)P(Y) (P(X,Y) is joint probability.)

Example• P( {X1 X2 X3 X4} contains >= 3 head ) = ?

Conditional Probability

• Suppose events A and B are not mutually exclusive, but occur conditionally on the occurrence of the othero The probability that event A will occur if

event B occurs is called the conditional probability

occurcanBtimesofnumbertheoccurcanBandAtimesofnumberthe

BAp

probability of A given B

Conditional Probability

• The probability that both A and B occur is called the joint probability of A and B, written p(A ∩ B)

BpBAp

BAp

ApABp

ABp

occurcanBtimesofnumbertheoccurcanBandAtimesofnumberthe

BAp

Conditional Probability

• Similarly, the conditional probability that event B will occur if event A occurs can be written as:

BpBAp

BAp

ApABp

ABp Bp

BApBAp

ApABp

ABp

Conditional Probability

BpBAp

BAp

ApABp

ABp

ApABpABp

ApABpBAp

BpBAp

BAp

ApABpABp

ApABpBAp

BpBAp

BAp

ApABpABp

ApABpBAp

BpBAp

BAp

• The Bayesian rule (named after Thomas Bayes, an 18th-century British mathematician):

The Bayesian Rule

Bp

ApABpBAp

Applying Bayes’ rule

• A = disease, B = symptom• P(disease|symptom) = P(symptom|

disease) * P(disease) / P(symptom)

Bp

ApABpBAp

Applying Bayes’ rule• A doctor knows that the disease meningitis

causes the patient to have a stiff neck for 70% of the time.

• The probability that a patient has meningitis is 1/50,000.

• The probability that any patient has a stiff neck is 1%.

• P(s|m) = 0.7• P(m) = 1/50000• P(s) = 0.01

Applying Bayes’ rule• P(s|m) = 0.7• P(m) = 1/50000• P(s) = 0.01• P(m|s) = P(s|m) * P(m) / P(s) • = 0.7 * 1/50000 / 0.01 • = 0.0014• = around 1/714• Conclusion: Less than 1 in 700 patients

with a stiff neck have meningitis.

Example: Coin Flip• P(X1 = H) = ½

1) X1 is H: P(X2 = H | X1 = H) = 0.9

2) X1 is T: P(X2 = T | X1 = T ) = 0.8

P(X2 = H) = ?

What we learned from the example?

• If event A depends on exactly two mutually exclusive events, B and ¬B, we obtain:

• P(¬X|Y) = 1 – P(X|Y)• P(X|¬Y) = 1 – P(X|Y)?

p(A) =p(AB) p(B) +p(AB) p(B)

p(B) =p(BA) p(A) +p(BA) p(A)

ApABpApABp

ApABpBAp

• If event A depends on exactly two mutually exclusive events, B and ¬B, we obtain:

• Similarly, if event B depends on exactly two mutually exclusive events, A and ¬A, we obtain:

Conditional probability

p(A) =p(AB) p(B) +p(AB) p(B)

p(B) =p(BA) p(A) +p(BA) p(A)

ApABpApABp

ApABpBAp

p(A)=p(AB) p(B)+p(AB) p(B)

p(B) =p(BA) p(A) +p(BA) p(A)

ApABpApABp

ApABpBAp

• Substituting p(B) into the Bayesian rule yields:

The Bayesian Rule

p(A)=p(AB) p(B)+p(AB) p(B)

p(B) =p(BA) p(A) +p(BA) p(A)

ApABpApABp

ApABpBAp

Bp

ApABpBAp

p(A)=p(AB) p(B)+p(AB) p(B)

p(B) =p(BA) p(A) +p(BA) p(A)

ApABpApABp

ApABpBAp

• Instead of A and B, consider H (a hypothesis) and E (evidence for that hypothesis).

• Expert systems use the Bayesian rule to rank potentially true hypotheses based on evidences

Bayesian reasoning

HpHEpHpHEp

HpHEpEHp

p(A)=p(AB) p(B)+p(AB) p(B)

p(B) =p(BA) p(A) +p(BA) p(A)

ApABpApABp

ApABpBAp

• If event E occurs, then the probability thatevent H will occur is p(H|E)

IF E (evidence) is trueTHEN H (hypothesis) is true with

probability p

Bayesian reasoning

HpHEpHpHEp

HpHEpEHp

Bayesian reasoning Example: Cancer and

Test • P(C) = 0.01 P(¬C) = 0.99• P(+|C) = 0.9 P(-|C) = 0.1• P(+|¬C) = 0.2 P(-|¬C) = 0.8

• P(C|+) = ?

HpHEpHpHEp

HpHEpEHp

Simple Bayes Network from Example

• Expert identifies prior probabilities forhypotheses p(H) and p(¬H)

• Expert identifies conditional probabilities for:o p(E|H): Observing evidence E if hypothesis

H is trueo p(E|¬H): Observing evidence E if

hypothesis H is false

Bayesian reasoning

HpHEpHpHEp

HpHEpEHp

• Experts provide p(H), p(¬H), p(E|H), and p(E|¬H)

• Users describe observed evidence Eo Expert system calculates p(H|E) using

Bayesian ruleo p(H|E) is the posterior probability that

hypothesis H occurs upon observing evidence E

• What about multiple hypotheses and evidences?

Bayesian reasoning

Bayesian reasoning with multiple hypotheses

i

n

ii

n

ii BpBApBAp

11

AB4

B3

B1

B2

p(A)

p(A) =p(AB) p(B) +p(AB) p(B)

p(B) =p(BA) p(A) +p(BA) p(A)

ApABpApABp

ApABpBAp

Bayesian reasoning with multiple hypotheses

• Expand the Bayesian rule to work with multiple hypotheses (H1...Hm)

HpHEpHpHEp

HpHEpEHp

Bayesian reasoning with multiple hypotheses and

evidences• Expand the Bayesian rule to work with

multiple hypotheses (H1...Hm) and evidences (E1...En)

• Expand the Bayesian rule to work with multiple hypotheses (H1...Hm) and evidences (E1...En)

Assuming conditional independence among evidences E1...En

Bayesian reasoning with multiple hypotheses and

evidences

m

kkknkk

iiniini

HpHEp...HEpHEp

HpHEpHEpHEpE...EEHp

121

2121

...

Summary

m

kkknkk

iiniini

HpHEp...HEpHEp

HpHEpHEpHEpE...EEHp

121

2121

...

• Expert is given three conditionally independent evidences E1, E2, and E3

o Expert creates three mutually exclusive and exhaustive hypotheses H1, H2, and H3

o Expert provides prior probabilities p(H1), p(H2), p(H3)

o Expert identifies conditional probabilities for observing each evidence Ei for all possible hypotheses Hk

Bayesian reasoning Example

• Expert data:

Bayesian reasoning Example

H ypothesi sProbability

1i 2i 3i

0.40

0.9

0.6

0.3

0.35

0.0

0.7

0.8

0.25

0.7

0.9

0.5

iHp

iHEp 1

iHEp 2

iHEp 3

• user observes E3 H ypothesi sProbability

1i 2i 3i

0.40

0.9

0.6

0.3

0.35

0.0

0.7

0.8

0.25

0.7

0.9

0.5

iHp

iHEp 1

iHEp 2

iHEp 3

Bayesian reasoning Example

32,1,=,3

13

33 i

HpHEp

HpHEpEHp

kkk

iii

0.3425.0.90+35.07.0+0.400.6

0.400.631

EHp

0.3425.0.90+35.07.0+0.400.6

35.07.032

EHp

0.3225.0.90+35.07.0+0.400.6

25.09.033

EHp

32,1,=,3

13

33 i

HpHEp

HpHEpEHp

kkk

iii

0.3425.0.90+35.07.0+0.400.6

0.400.631

EHp

0.3425.0.90+35.07.0+0.400.6

35.07.032

EHp

0.3225.0.90+35.07.0+0.400.6

25.09.033

EHp

32,1,=,3

13

33 i

HpHEp

HpHEpEHp

kkk

iii

0.3425.0.90+35.07.0+0.400.6

0.400.631

EHp

0.3425.0.90+35.07.0+0.400.6

35.07.032

EHp

0.3225.0.90+35.07.0+0.400.6

25.09.033

EHp

32,1,=,3

13

33 i

HpHEp

HpHEpEHp

kkk

iii

0.3425.0.90+35.07.0+0.400.6

0.400.631

EHp

0.3425.0.90+35.07.0+0.400.6

35.07.032

EHp

0.3225.0.90+35.07.0+0.400.6

25.09.033

EHp

expert system computes

posterior probabilities

user observes E3

H ypothesi sProbability

1i 2i 3i

0.40

0.9

0.6

0.3

0.35

0.0

0.7

0.8

0.25

0.7

0.9

0.5

iHp

iHEp 1

iHEp 2

iHEp 3

• user observes E3 E1 H ypothesi sProbability

1i 2i 3i

0.40

0.9

0.6

0.3

0.35

0.0

0.7

0.8

0.25

0.7

0.9

0.5

iHp

iHEp 1

iHEp 2

iHEp 3

32,1,=,3

131

3131 i

HpHEpHEp

HpHEpHEpEEHp

kkkk

iiii

0.1925.00.5+35.07.00.8+0.400.60.3

0.400.60.3311

EEHp

0.5225.00.5+35.07.00.8+0.400.60.3

35.07.00.8312

EEHp

0.2925.00.5+35.07.00.8+0.400.60.3

25.09.00.5313

EEHp

Bayesian reasoning Example

user observes E1

32,1,=,3

131

3131 i

HpHEpHEp

HpHEpHEpEEHp

kkkk

iiii

0.1925.00.5+35.07.00.8+0.400.60.3

0.400.60.3311

EEHp

0.5225.00.5+35.07.00.8+0.400.60.3

35.07.00.8312

EEHp

0.2925.00.5+35.07.00.8+0.400.60.3

25.09.00.5313

EEHp

expert system computes

posterior probabilities

H ypothesi sProbability

1i 2i 3i

0.40

0.9

0.6

0.3

0.35

0.0

0.7

0.8

0.25

0.7

0.9

0.5

iHp

iHEp 1

iHEp 2

iHEp 3

m

kkknkk

iiniini

HpHEp...HEpHEp

HpHEpHEpHEpE...EEHp

121

2121

...

• user observes E3 E1 E2 H ypothesi sProbability

1i 2i 3i

0.40

0.9

0.6

0.3

0.35

0.0

0.7

0.8

0.25

0.7

0.9

0.5

iHp

iHEp 1

iHEp 2

iHEp 3

32,1,=,3

1321

321321 i

HpHEpHEpHEp

HpHEpHEpHEpEEEHp

kkkkk

iiiii

0.4525.09.00.50.7

0.7

0.7

0.5

+.3507.00.00.8+0.400.60.90.30.400.60.90.3

3211

EEEHp

025.09.0+.3507.00.00.8+0.400.60.90.3

35.07.00.00.83212

EEEHp

0.5525.09.00.5+.3507.00.00.8+0.400.60.90.3

25.09.00.70.53213

EEEHp

32,1,=,3

1321

321321 i

HpHEpHEpHEp

HpHEpHEpHEpEEEHp

kkkkk

iiiii

0.4525.09.00.50.7

0.7

0.7

0.5

+.3507.00.00.8+0.400.60.90.30.400.60.90.3

3211

EEEHp

025.09.0+.3507.00.00.8+0.400.60.90.3

35.07.00.00.83212

EEEHp

0.5525.09.00.5+.3507.00.00.8+0.400.60.90.3

25.09.00.70.53213

EEEHp

Bayesian reasoning Example

expert system computesposterior probabilitiesuser observes E2

H ypothesi sProbability

1i 2i 3i

0.40

0.9

0.6

0.3

0.35

0.0

0.7

0.8

0.25

0.7

0.9

0.5

iHp

iHEp 1

iHEp 2

iHEp 3

m

kkknkk

iiniini

HpHEp...HEpHEp

HpHEpHEpHEpE...EEHp

121

2121

...

Bayesian reasoning Example

• Initial expert-based ranking:o p(H1) = 0.40; p(H2) = 0.35; p(H3) = 0.25

• Expert system ranking after observing E1, E2, E3:o p(H1) = 0.45; p(H2) = 0.0; p(H3) = 0.55

H ypothesi sProbability

1i 2i 3i

0.40

0.9

0.6

0.3

0.35

0.0

0.7

0.8

0.25

0.7

0.9

0.5

iHp

iHEp 1

iHEp 2

iHEp 3

Problems with the Bayesianapproach

• Humans are not very good at estimating probability!o In particular, we tend to make different

assumptions when calculating prior and conditional probabilities

• Reliable and complete statistical information often not available.

• Bayesian approach requires evidences to be conditionally independent – often not the case.

• One solution: certainty factors