Personality Retest Effects: Guilt as a Mechanism for
Managing Response Distortion
Jill E. Ellingson, Eric D. Heggestad, and Erin E. Coyne
October 13 – 14, 2006ETS Technical Advisory Group Meeting
Current Retesting Policy
• Applicants allowed to voluntarily retake assessment after period of time if displeased with outcome– Applicant elects to retake the assessment
• Common in organizations which use assessment tools for hiring
• Most often used in conjunction with cognitively-loaded assessments
Personality Assessment Retesting• Organization directs certain applicants whose
responses are likely distorted to retake the personality assessment
• Responses deemed distorted on basis of embedded intentional distortion scale– Flags extreme response profiles
• Applicants informed that responses were flagged as suspect
• Hiring decisions made using retested scores
Key Questions
• Does retesting flagged applicants lower previously inflated personality scale scores?
• What psychological mechanism operates within applicants to help explain why they would adjust their responses?
Hypothesis 1: Retesting flagged individuals will result indecreased personality scale scores in the secondassessment relative to the first assessment.
Scale Score Changes
• Flagged applicants have positively biased score profiles• Retest effect evident in degree to which second
assessment scores are lower• Preliminary research suggests that scores may be
lowered up to 0.7 standard deviation units (Ellingson & Heggestad, 2003)
Role of Guilt: Appraisal Theory
EventEvaluation Factors
Relevance?
Congruence?
Associated values?
Accountability?
Coping potential?
Behavior
Emotion
Role of Guilt: Applicant Appraisal
Told toretest
Evaluation Factors•Personally relevant
•Incongruent•Violates personal standards
•Personally accountable
•Coping potential?
Guilt
BehaviorHypothesis 2: Retesting flagged individuals willresult in increased feelings of guilt in the secondassessment relative to the first assessment.
Role of Guilt: Applicant Appraisal
Told toretest
Evaluation Factors•Personally relevant
•Incongruent
•Violates personal standards
•Personally accountable
•Make reparation
Guilt
Hypothesis 3: The level of guilt reported by flagged individuals in the second assessment will moderate the degree to which personality scale scores change.
Respondhonestly
Sample and Measures
• 288 undergraduate students• Measures:
– NEO-Five Factor Inventory (NEO-FFI)– Balanced Inventory of Desirable Responding-
Impression Management scale (BIDR-IM)– Personal Feelings Questionnaire-2 Guilt scale
(PFQ2-G)– Positive and Negative Affect Schedule-Expanded
(PANAS-X)– Test of Self-Conscious Affect-3 (TOSCA3)
ProcedureAll participants:1. Completed the TOSCA32. Completed NEO-FFI and BIDR-IM
under motivating instructions3. Completed the PFQ2-G and PANAS-X
regarding feelings had while takingthe personality measure
Sorted participantsinto 3 groupsbased on
BIDR-IM score
Low Control Group•Low BIDR-IM score
High Control Group•High BIDR-IM score
Flagged Retest Group•High BIDR-IM score
Retested for neutral reason Told responses weresuspect and unusable
Asked to retest
Tim
e 1
Tim
e 2
Effect Sizes
NEO-FFI Low Control High Control Flagged Retest
Extraversion 0.36 0.29 0.45
Conscientiousness 0.04 0.22 0.59
Agreeableness 0.12 0.26 0.56
Openness 0.04 0.10 0.28
Emotional Stability -0.10 0.05 0.27
Average d 0.09 0.18 0.43
Impression Management -0.03 -0.13 0.54
State Guilt 0.20 0.33 0.07
Positive values indicate that Time 1 score was larger than Time 2 score.
Repeated-measures MANCOVA:Personality Scales
Source Pillai’s Trace F df p η2
Within-subjects effects
Time 0.018 1.045 (5, 280) 0.392 0.018
Time x Trait Guilt 0.010 .577 (5, 280) 0.718 0.010
Time x Condition .170 5.232 (10, 562) 0.000 0.085
Between-subjects effects
Trait Guilt 0.093 5.739 (5, 280) 0.000 0.093
Condition 0.255 8.229 (10, 562) 0.000 0.128
Within-subjects factor (Time): Time 1, Time 2Between-subjects factor (Condition): Low Control, High Control, Flagged Retest
Covariate: Trait Guilt
Repeated-measures ANCOVA:Personality Scales
Within-subjects effects Between-subjects effects
Time Time x Guilt Time x Condition Trait Guilt Condition
NEO-FFI F η2 F η2 F η2 F η2 F η2
Extraversion 2.32 0.008 0.06 0.000 2.20 0.015 1.10 0.004 0.75 0.005
Conscientiousness 0.95 0.003 0.01 0.000 16.78* 0.106 4.02* 0.014 17.25* 0.108
Agreeableness 1.30 0.005 0.01 0.000 7.84* 0.052 1.24 0.004 33.07* 0.189
Openness 2.62 0.009 1.01 0.004 6.61* 0.044 8.21* 0.028 0.38 0.003
Emotional Stability
2.45 0.009 1.55 0.005 12.59* 0.081 6.26* 0.022 7.94* 0.053
Within-subjects factor (Time): Time 1, Time 2Between-subjects factor (Condition): Low Control, High Control, Flagged Retest
Covariate: Trait Guilt
* p < .05
Repeated-measures ANCOVA:Agreeableness Interaction
48
50
52
54
56
58
60
62
64
66
68
Tim e 1 Tim e 2
Ag
reea
ble
nes
s sc
ale
mea
ns
High Control
Flagged Retest
Low Control
Repeated-measures ANCOVA: State Guilt
Source Pillai’s Trace F df p η2
Within-subjects effects
Time 0.004 1.274 (1, 283) 0.260 0.004
Time x Trait Guilt 0.002 .466 (1, 283) 0.495 0.002
Time x Condition 0.004 .537 (2, 283) 0.585 0.004
Between-subjects effects
Trait Guilt 81.353 4.500 (1, 283) 0.035 0.016
Condition 244.106 6.752 (2, 283) 0.001 0.046
Within-subjects factor (Time): Time 1, Time 2Between-subjects factor (Condition): Low Control, High Control, Flagged Retest
Covariate: Trait Guilt
Moderated Regressions:Understanding Score Change
Full Model Predicting Time 2 Personality Scale ScoresStandardized Beta Coefficients and Variance Explained
Predictors Extraversion Conscientiousness Agreeableness OpennessEmotional
Stability
Step 1 Time 1 Score .838* .888* .829* .910* .850*
Step 2 Time 2 State Guilt -.083 -.195* -.291* -.138* .141*
Low Control Condition .097 .176* .064 .106* -.163*
High Control Condition .087 .143* .101* .085* -.114*
Step 3 Guilt x Low Control .074 .104* .225* .135* -.072
Guilt x High Control .077 .123* .098* .058† -.077†
R-squared .688 .730 .726 .826 .722
Δ R-squared .004 .010* .018* .007* .004
* p < .05† p < .10
Moderated Regressions:Conscientiousness Interaction
Moderating Effect of State Guilt onTime 2 Conscientiousness Scores
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
Low State Guilt High State Guilt
Tim
e 2
Co
ns
cie
nti
ou
sn
es
s
Sc
ore
Low Control
High Control
Flagged Retes t
Conclusion
• Retesting flagged applicants will result in a set of personality scale scores that are less positively inflated
• The appraisal profile of guilt helps explain this effect– Flagged applicants who feel guilty as a result
of being retested decrease their scores in response.
