Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller...

transcript

Components of judgmental skill

Thomas R. Stewart, Ph.D.Center for Policy Research

Rockefeller College of Public Affairs and PolicyUniversity at Albany

State University of New YorkT.STEWART@ALBANY.EDU

Public Administration and PolicyPAD634 Judgment and Decision Making Behavior

components-of-skill.ppt 2

Fundamental tenet of correspondence research

"Human competence in making judgments and decisions under uncertainty is impressive. Sometimes performance is not. Why? Because sometimes task conditions degrade the accuracy of judgment."

Hammond, K. R. (1996). Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. New York, Oxford University Press (p. 282).

Brunswik's lens model

Distal variable Judgment

TrueDescriptors

Subjective

Judgment

CuesCues

Expanded lens model

Distal variable

Environmental predictability

Fidelity of the information system

Match between environment and judge

Reliability of information acquisition

Reliability of information processing

TrueDescriptors

Subjective

Judgment

CuesCues

Components of skill and the lens model

Distal variable

Area under ROC curve

Correlation (rYO)

O is the observation, or “gold standard”

Y is the judgment

Mean square error:

Skill score:

Alternative measures of performance(correspondence approach)

O - Yn1

= MSE 2iiY

MSEMSE

- 1 = SS 2iB )O O(

Decomposing accuracy--the skill score

Murphy (1988):

Skill = Correlation - Conditional bias - Unconditional bias

score (regression bias) (base rate bias)

YOYO sO Y

- r - r = SS

Decomposing accuracy: The Lens Model Equation (Tucker 1964)

Judgment

"Truth"

O.Xn2O.X

E + X , ... ,X ,XM = O1

Y.Xn2Y.X

E + X , ... ,X ,XM = Y1

O.XX.YO.XYOR - 1 R - 1C + R G R = r

Decomposition of Skill Score

- ( / ) ][ - [ ( Y - O ) / ]sY

Squared correlation

Conditional Unconditional

bias bias

G RY.X

rYO- - ( / ) ][ - [ ( Y - O ) / ]s

Murphy

Tucker

VT.X G R

Expandedlens

Components of skill:

1. Environmental predictability

2. Fidelity of the information system

3. Match between environment and judge

4. Reliability of information acquisition

5. Reliability of information processing

6. Conditional/regression bias

7. Unconditional/base rate bias

Step 1:

Step 2:

Step 3:

(regression) (base rate)

rYO - ( / ) ][ - [ ( Y - O ) / ]s

(1988)

(1964)

1 2 3 4 765

SS = Skill Score = 1 - ( MSE Y MSE B/ )

SS ~= VU.X

Decomposition of skill score

processing

ninformatio

ofy reliabilit

nacquisitio

ninformatio

ofy reliabilit

forecaster and

tenvironmen

between match

system

ninformatio

the offidelity

litypredictabi

talenvironmen

nalunconditio

lconditiona

Skill score =

Components of skill addressed by selected methods for improving judgments

Component of Skill* Method for improving judgments 1 2 3 4 5 6 7 Identify new descriptors through research X Develop better measures of true descriptors X Develop clear definitions of cues X Training to improve cue judgments X Improve information displays X Bootstrapping--replace judge with model X Require justification of judgments X X Combine several judgments X Decompose judgment task X Mechanical combination of cues X Train judge about environmental system X Experience with problem X X XCognitive feedback X Train judge to ignore non-predictive cues X Statistical training X XFeedback about nature of biases in judgment X XSearch for discrepant information X Statistical correction for bias X X

*1. Environmental predictability 2. Fidelity of the information system 3. Reliability of information acquisition 4. Reliability of information processing 5. Match between environment and judge 6. Conditional/regression bias 7. Unconditional/base rate bias

1. Environmental predictability

• Environmental predictability is conditional on current knowledge and information. It can be improved through research that results in improved information and improved understanding of environmental processes.

• Environmental predictability determines an upper bound on performance and therefore indicates how much improvement is possible through attention to other components.

Components of skill

Environmental predictability limits accuracy of judgment

2. Fidelity of information system

• Judgmental skill may be degraded if the information system that brings data to the judge does not accurately represent actual conditions, i.e., if the cues do not accurately measure the true descriptors. Fidelity of the information system refers to the quality, not the quantity, of information about the cues that are currently being used.

• Fidelity is improved by developing better measures, e.g., though improved instrumentation or increased density in space or time.

Components of skill

3. Match between environment and judge

• The match between the model of the judge and the environmental model is an estimate of the potential skill that the judge's current strategy could achieve if the environment were perfectly predictable (given the cues) and the judgments were unbiased and perfectly reliable.

• This component might be called “knowledge.” It is addressed by training and experience. If the judge learns to rely on the most relevant information and ignore irrelevant information, this component will generally be good.

Components of skill

Reliability

• Reliability is high if identical conditions produce identical judgments.

• Humans are rarely perfectly reliable.

• There are two sources of unreliability:– Reliability of information acquisition

– Reliability of information processing

Components of skill

Reliability

• Reliability decreases as amount of information increases.

Components of skill

Theoretical relation between amount of information and

accuracy of judgment

Reliability decreases as environmental predictability decreases

Components of skill

4. Reliability of information acquisition

• Reliability of information acquisition is the extent to which the judge can reliably interpret the objective cues.

• It is improved by organizing and presenting information in a form that clearly emphasizes relevant information.

Components of skill

Visual display of information

Components of skill

Books by Edward R. Tufte

The Visual Display of Quantitative Information.

Envisioning Information

Visual Explanations

Beautiful Evidence

The Cognitive Style of Powerpoint: Pitching Out Corrupts Within

Cheshire, Connecticut: Graphics Press

Components of skill

Image from the videotape “Study of a Numerically modeled severe storm,” National Center for Supercomputing applications, University of Illinois (Tufte, Visual Explanations, p. 20)

Tufte’s questions:How big is the cloud?What direction is it moving?What are the dimensions of the grid?

Components of skill

Redesign by Tufte and Bushell (Tufte, Visual Explanations, p. 21)

Restored quantitative informationDirectional arrowsGrid size given (and reduce the dominance of the grid pattern)Time scale, with small clouds depicting storm history

Components of skill

Chart prepared by Morton Thiokol for Challenger commission. Tufte, Visual Explanations, p. 47

Visual display problems

Disappearing legend.The O-ring damage legend was on another slide.

ChartjunkObscures cause and effect

Temperatures turned sidewaysO-ring anomalies depicted by scattered little marks

Wrong orderDate of launch rather than temperature

Visual display of information: Scatterplot of temperature vs. O-ring damage

Components of skill

Tufte, Visual Explanations, p. 45

Visual display of information influences mode of thought

Components of skill

Use of images tends to induce intuition

Use of numbers tends to induce analysis

5. Reliability of information processing

Decreases with increasing information and with increasing environmental uncertainty

Methods for improving reliability of information processing: Limit the amount of information used in making

judgments. Use a small number of very important cues. Use mechanical methods to process information. Combine several judgments. Require justification of judgments.

Components of skill

Amount of Information

Actual accuracy

Theoretical limit of accuracy

Perfectaccuracy

Noaccuracy

Effect of limited

information and

environmental uncertainty

Effect of

limitations in

information processing

Theoretical relation between amount of information and accuracy

The relation between information and accuracy depends on environmental

uncertainty

Low predictability task

Theoretical Limit of Accuracy

Actual Accuracy

Effect of limited information and environmental

uncertainty

Effect of limitations in information processing

No accuracy

Perfect accuracy

High predictability task

Theoretical Limit of Accuracy

Actual Accuracy

Effect of limited information and environmental

uncertainty Effect of limitations in information processing

No accuracy

Perfect accuracy

- - - - - Theoretical limit of accuracy——— Actual accuracy

6 and 7. Bias -- Conditional (regression bias) and unconditional (base rate bias)

Together, the two bias terms measure judgment "calibration.”

Reducing bias: Experience Statistical training Feedback about nature of biases in judgment Search for discrepant information Statistical correction for bias

Components of skill

Calibration of judgments depends on the task

Calibration data for precipitation forecasts (Murphy and Winkler, 1974) Heideman (1989)

Conclusion

• Problem: Improving judgmental accuracy– Understanding and improving judgment requires

understanding the task and the environment.– Decomposing skill can aid in identifying the factors

that limit judgmental accuracy.

Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller...

Documents