1

Simulations: Basic ProcedureSimulations: Basic Procedure

1. Generate a true probability distribution, and a set 1. Generate a true probability distribution, and a set of of base conditionals base conditionals (premises), with associated (premises), with associated lower probability bounds.lower probability bounds.

2. Determine which 2. Determine which derived conditionals derived conditionals (conclusions) each system is able to derive from (conclusions) each system is able to derive from the given base conditionals (including lower the given base conditionals (including lower probability bounds for these conclusions).probability bounds for these conclusions).

3. Assign scores to each system, by comparing the 3. Assign scores to each system, by comparing the lower probability bounds for the derived lower probability bounds for the derived conditionals with the true probabilities.conditionals with the true probabilities.

2

Language RestrictionLanguage RestrictionWe adopted a simple language with four binary We adopted a simple language with four binary variables: a, b, c, and d variables: a, b, c, and d

We only considered conditionals with conjuctive We only considered conditionals with conjuctive antecedents and consequents (and no repetition antecedents and consequents (and no repetition of atoms), yielding of atoms), yielding 464 conditionals:464 conditionals:

64 of the form 64 of the form xxyyzzw w 96 of the form 96 of the form xxyyz z 48 of the form 48 of the form xxy y 96 of the form 96 of the form xxyyzzw w 96 of the form 96 of the form xxyyz z 64 of the form 64 of the form xxyyzzww

3

The probability The probability distributions…distributions…

are fixed by randomly setting the following are fixed by randomly setting the following sixteen values:sixteen values:

(a), (a),

(b|a), (b|a), (b|(b|a), a),

(c|a(c|ab), b), (c|a(c|ab), b), (c|(c|aab), b), (c|(c|aab), b), (d|(d|aabbc), c), (d|a(d|abbc), c), (d|a(d|abbc), c), (d|a(d|abbc), c), (d|(d|aabbc), c), (d|(d|aabbc), c), (d|(d|aabbc), c), (d|(d|aabbc), c), and and (d|(d|aabbc).c).

4

ErrorsErrors

We say that a system has made an We say that a system has made an error when the lower probability bound error when the lower probability bound for a derived conditional exceeds the for a derived conditional exceeds the true probability.true probability.

5

Scoring IScoring I

TheThe advantage-compared-to-guessing advantage-compared-to-guessing score for derived conditionals:score for derived conditionals:

ScoreScoreAvGAvG( C( C r r D, D, ) = 1/3 ) = 1/3 |r |r (D|C)|.(D|C)|.

6

Scoring IIScoring II

TheThe price-is-right price-is-right scorescore for derived for derived conditionals:conditionals:

ScoreScorePIRPIR( C( C r r D, D, ) = r, if r ) = r, if r (D|C), (D|C),

= 0, otherwise.= 0, otherwise.

7

Scoring IIIScoring III

TheThe subtle-price-is-right subtle-price-is-right scorescore for for derived conditionals:derived conditionals:

ScoreScoresPIRsPIR( C( C r r D, D, ) = r, if r ) = r, if r (D|C), (D|C),

= = (D|C) (D|C) r, r, otherwise.otherwise.

8

Minimum Probability for Minimum Probability for Base ConditionalsBase Conditionals

For each simulation, we required that For each simulation, we required that the probability for each base the probability for each base conditional (premise) exceeded some conditional (premise) exceeded some fixed value, s.fixed value, s.

9

Simulations with Randomly Simulations with Randomly Selected Base ConditionalsSelected Base Conditionals

► was determined at random, for each run. was determined at random, for each run.

►We varied the number, n, of base conditionals. (n = We varied the number, n, of base conditionals. (n = 2, 3, 4, 5, or 6)2, 3, 4, 5, or 6)

►We varied, s, the minimal probability of base We varied, s, the minimal probability of base conditionals. (s = 0.5, 0.6, 0.7, 0.8, 0.9, or 0.9999)conditionals. (s = 0.5, 0.6, 0.7, 0.8, 0.9, or 0.9999)

► The base conditionals, for each run, were selected at The base conditionals, for each run, were selected at random, from among those conditionals whose random, from among those conditionals whose associated probability was at least s.associated probability was at least s.

►We ran each combination of s and n one thousand We ran each combination of s and n one thousand times.times.

10

Simulations with Simulations with Predetermined Base Predetermined Base

ConditionalsConditionals

► was determined at random, for each was determined at random, for each run.run.

►For each set of predetermined base For each set of predetermined base conditionals, we ran each combination of conditionals, we ran each combination of s one thousand times. s one thousand times. (s = 0.5, 0.6, 0.7, 0.8, 0.9, and 0.9999)(s = 0.5, 0.6, 0.7, 0.8, 0.9, and 0.9999)

11

The DataThe Data

12

Observations (Table 1)Observations (Table 1)

1. Systems 1. Systems O O is very conservative, and is very conservative, and licences very few inferences. licences very few inferences. [YELLOW] [YELLOW]

13

Observations (Table 2)Observations (Table 2)

3. System 3. System ZZ almost always outperformed almost always outperformed the other systems by the AvG measure the other systems by the AvG measure (even though the (even though the AvG measure AvG measure sometimes sometimes punishes non-errors). punishes non-errors). [YELLOW][YELLOW]

4. 4. System System QCQC outscored all of the systems outscored all of the systems by the PIR measure (save where s = by the PIR measure (save where s = 0.9999), 0.9999), though though system system Z Z also scores also scores wellwell. . [BLUE][BLUE]

14

Observations (Table 2)Observations (Table 2)

5. 5. System System ZZ generally achieved positive generally achieved positive sPIR scores, and outscored all of the sPIR scores, and outscored all of the systems by this measure (save in the systems by this measure (save in the case where s = 0.9999). case where s = 0.9999). [GREEN][GREEN]

6. 6. QCQC usually obtained negative sPIR usually obtained negative sPIR scores, due to frequent errors.scores, due to frequent errors.

15

Observations (Table 3)Observations (Table 3)

7. Where s is fixed, increasing the 7. Where s is fixed, increasing the number of base conditionals tends to number of base conditionals tends to increase the scores (for all measures) increase the scores (for all measures) for systems for systems OO, , PP, and , and Z Z (but not for (but not for QC QC [YELLOW][YELLOW]).).

16

Observations (Table 4)Observations (Table 4)

8. For systems 8. For systems OO, , PP, and , and ZZ, the score , the score earned per inference earned per inference tends tends to increase to increase for higher values of s (exceptions in for higher values of s (exceptions in [YELLOW][YELLOW]).).

17

ConclusionsConclusions

1. System 1. System PP offers the same security as offers the same security as system system OO, and licences more inferences., and licences more inferences.

2. System 2. System Z Z licences far more inferences licences far more inferences than system than system PP, and generally infers , and generally infers lower probability bounds that are close lower probability bounds that are close to the true probability values (based on to the true probability values (based on AvG scoring).AvG scoring).

18

ConclusionsConclusions

3. 3. QCQC inferences are too risky, and very inferences are too risky, and very few of the inferences sanctioned by few of the inferences sanctioned by QCQC, and not by , and not by ZZ, should be made. , should be made. (Consider the (Consider the QCQCZZ inferences, esp. inferences, esp. sPIR scores.)sPIR scores.)

4. Of the four systems, system 4. Of the four systems, system ZZ offers offers the best balance of safety versus the best balance of safety versus inferential power.inferential power.