Date post: | 13-Jan-2016 |
Category: |
Documents |
Upload: | emory-crawford |
View: | 218 times |
Download: | 0 times |
Components of judgmental skill
Thomas R. Stewart, Ph.D.Center for Policy Research
Rockefeller College of Public Affairs and PolicyUniversity at Albany
State University of New [email protected]
Public Administration and PolicyPAD634 Judgment and Decision Making Behavior
components-of-skill.ppt 2
Fundamental tenet of correspondence research
"Human competence in making judgments and decisions under uncertainty is impressive. Sometimes performance is not. Why? Because sometimes task conditions degrade the accuracy of judgment."
Hammond, K. R. (1996). Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. New York, Oxford University Press (p. 282).
components-of-skill.ppt 3
Brunswik's lens model
Distal variable Judgment
X
Cues
Ye Ys
components-of-skill.ppt 4
TrueDescriptors
Subjective
Judgment
CuesCues
Expanded lens model
Distal variable
Environmental predictability
Fidelity of the information system
Match between environment and judge
Reliability of information acquisition
Reliability of information processing
TrueDescriptors
Subjective
Judgment
CuesCues
components-of-skill.ppt 5
Components of skill and the lens model
Distal variable
components-of-skill.ppt 6
Area under ROC curve
Correlation (rYO)
O is the observation, or “gold standard”
Y is the judgment
Mean square error:
Skill score:
Alternative measures of performance(correspondence approach)
O - Yn1
= MSE 2iiY
B
Y
MSEMSE
- 1 = SS 2iB )O O(
n1
= MSE
components-of-skill.ppt 7
Decomposing accuracy--the skill score
Murphy (1988):
Skill = Correlation - Conditional bias - Unconditional bias
score (regression bias) (base rate bias)
22
2
OO
Y
YOYO sO Y
- ss
- r - r = SS
components-of-skill.ppt 8
Decomposing accuracy: The Lens Model Equation (Tucker 1964)
Judgment
Cues
"Truth"
O.Xn2O.X
E + X , ... ,X ,XM = O1
Y.Xn2Y.X
E + X , ... ,X ,XM = Y1
2
Y.X
2
O.XX.YO.XYOR - 1 R - 1C + R G R = r
Decomposition of Skill Score
rYO
( )2
-
rYO
- ( / ) ][ - [ ( Y - O ) / ]sY
_
sO
2 2s
O
Squared correlation
Conditional Unconditional
bias bias
SS =
RO.X
G RY.X
GSS
rYO- - ( / ) ][ - [ ( Y - O ) / ]s
Ys
O
2 2s
O
( )2
RO.T
_
Murphy
Tucker
VT.X G R
Y.U
Expandedlens
model
Components of skill:
1. Environmental predictability
2. Fidelity of the information system
3. Match between environment and judge
4. Reliability of information acquisition
5. Reliability of information processing
6. Conditional/regression bias
7. Unconditional/base rate bias
Step 1:
Step 2:
Step 3:
(regression) (base rate)
rYO - ( / ) ][ - [ ( Y - O ) / ]s
Ys
O
2 2s
O
_-
(1988)
(1964)
1 2 3 4 765
_
_
_[ ]
2
SS = Skill Score = 1 - ( MSE Y MSE B/ )
~=
SS ~= VU.X
components-of-skill.ppt 10
Decomposition of skill score
processing
ninformatio
ofy reliabilit
nacquisitio
ninformatio
ofy reliabilit
forecaster and
tenvironmen
between match
system
ninformatio
the offidelity
litypredictabi
talenvironmen
bias
nalunconditio
bias
lconditiona
Skill score =
Components of skill addressed by selected methods for improving judgments
Component of Skill* Method for improving judgments 1 2 3 4 5 6 7 Identify new descriptors through research X Develop better measures of true descriptors X Develop clear definitions of cues X Training to improve cue judgments X Improve information displays X Bootstrapping--replace judge with model X Require justification of judgments X X Combine several judgments X Decompose judgment task X Mechanical combination of cues X Train judge about environmental system X Experience with problem X X XCognitive feedback X Train judge to ignore non-predictive cues X Statistical training X XFeedback about nature of biases in judgment X XSearch for discrepant information X Statistical correction for bias X X
*1. Environmental predictability 2. Fidelity of the information system 3. Reliability of information acquisition 4. Reliability of information processing 5. Match between environment and judge 6. Conditional/regression bias 7. Unconditional/base rate bias
components-of-skill.ppt 11
components-of-skill.ppt 12
1. Environmental predictability
• Environmental predictability is conditional on current knowledge and information. It can be improved through research that results in improved information and improved understanding of environmental processes.
• Environmental predictability determines an upper bound on performance and therefore indicates how much improvement is possible through attention to other components.
Components of skill
components-of-skill.ppt 1313
Environmental predictability limits accuracy of judgment
components-of-skill.ppt 14
2. Fidelity of information system
• Judgmental skill may be degraded if the information system that brings data to the judge does not accurately represent actual conditions, i.e., if the cues do not accurately measure the true descriptors. Fidelity of the information system refers to the quality, not the quantity, of information about the cues that are currently being used.
• Fidelity is improved by developing better measures, e.g., though improved instrumentation or increased density in space or time.
Components of skill
components-of-skill.ppt 15
3. Match between environment and judge
• The match between the model of the judge and the environmental model is an estimate of the potential skill that the judge's current strategy could achieve if the environment were perfectly predictable (given the cues) and the judgments were unbiased and perfectly reliable.
• This component might be called “knowledge.” It is addressed by training and experience. If the judge learns to rely on the most relevant information and ignore irrelevant information, this component will generally be good.
Components of skill
components-of-skill.ppt 16
Reliability
• Reliability is high if identical conditions produce identical judgments.
• Humans are rarely perfectly reliable.
• There are two sources of unreliability:– Reliability of information acquisition
– Reliability of information processing
Components of skill
components-of-skill.ppt 17
Reliability
• Reliability decreases as amount of information increases.
Components of skill
Theoretical relation between amount of information and
accuracy of judgment
components-of-skill.ppt 18
Reliability decreases as environmental predictability decreases
Components of skill
18
components-of-skill.ppt 19
4. Reliability of information acquisition
• Reliability of information acquisition is the extent to which the judge can reliably interpret the objective cues.
• It is improved by organizing and presenting information in a form that clearly emphasizes relevant information.
Components of skill
components-of-skill.ppt 20
Visual display of information
Components of skill
Books by Edward R. Tufte
The Visual Display of Quantitative Information.
Envisioning Information
Visual Explanations
Beautiful Evidence
The Cognitive Style of Powerpoint: Pitching Out Corrupts Within
Cheshire, Connecticut: Graphics Press
components-of-skill.ppt 21
Visual display of information
Components of skill
Image from the videotape “Study of a Numerically modeled severe storm,” National Center for Supercomputing applications, University of Illinois (Tufte, Visual Explanations, p. 20)
Tufte’s questions:How big is the cloud?What direction is it moving?What are the dimensions of the grid?
components-of-skill.ppt 22
Visual display of information
Components of skill
Redesign by Tufte and Bushell (Tufte, Visual Explanations, p. 21)
Restored quantitative informationDirectional arrowsGrid size given (and reduce the dominance of the grid pattern)Time scale, with small clouds depicting storm history
components-of-skill.ppt 23
Visual display of information
Components of skill
Chart prepared by Morton Thiokol for Challenger commission. Tufte, Visual Explanations, p. 47
Visual display problems
Disappearing legend.The O-ring damage legend was on another slide.
ChartjunkObscures cause and effect
Temperatures turned sidewaysO-ring anomalies depicted by scattered little marks
Wrong orderDate of launch rather than temperature
components-of-skill.ppt 24
Visual display of information: Scatterplot of temperature vs. O-ring damage
Components of skill
Tufte, Visual Explanations, p. 45
components-of-skill.ppt 25
Visual display of information influences mode of thought
Components of skill
Use of images tends to induce intuition
Use of numbers tends to induce analysis
components-of-skill.ppt 26
5. Reliability of information processing
Decreases with increasing information and with increasing environmental uncertainty
Methods for improving reliability of information processing: Limit the amount of information used in making
judgments. Use a small number of very important cues. Use mechanical methods to process information. Combine several judgments. Require justification of judgments.
Components of skill
Acc
urac
y
Amount of Information
Actual accuracy
Theoretical limit of accuracy
Perfectaccuracy
Noaccuracy
Effect of limited
information and
environmental uncertainty
Effect of
limitations in
information processing
components-of-skill.ppt 27
Theoretical relation between amount of information and accuracy
components-of-skill.ppt 28
The relation between information and accuracy depends on environmental
uncertainty
Low predictability task
Amount of Information
Acc
ura
cy
Theoretical Limit of Accuracy
Actual Accuracy
Effect of limited information and environmental
uncertainty
Effect of limitations in information processing
No accuracy
Perfect accuracy
High predictability task
Amount of Information
Acc
ura
cy
Theoretical Limit of Accuracy
Actual Accuracy
Effect of limited information and environmental
uncertainty Effect of limitations in information processing
No accuracy
Perfect accuracy
- - - - - Theoretical limit of accuracy——— Actual accuracy
- - - - - Theoretical limit of accuracy——— Actual accuracy
components-of-skill.ppt 29
6 and 7. Bias -- Conditional (regression bias) and unconditional (base rate bias)
Together, the two bias terms measure judgment "calibration.”
Reducing bias: Experience Statistical training Feedback about nature of biases in judgment Search for discrepant information Statistical correction for bias
Components of skill
components-of-skill.ppt 30
Calibration of judgments depends on the task
Calibration data for precipitation forecasts (Murphy and Winkler, 1974) Heideman (1989)
components-of-skill.ppt 31
Conclusion
• Problem: Improving judgmental accuracy– Understanding and improving judgment requires
understanding the task and the environment.– Decomposing skill can aid in identifying the factors
that limit judgmental accuracy.