Date post: | 19-Dec-2015 |
Category: |
Documents |
View: | 218 times |
Download: | 5 times |
BEYOND SKIN DEEP: INVESTIGATING THE “WHO” OF THE SENSITIVITY REVIEW
James GrandJuliya Golubovich
Ann Marie RyanNeal Schmitt
Society for Industrial & Organizational PsychologyAtlanta, GA
April 8, 2010
2
• Both here and in available sources, we have...– Examined why reviews are important– Investigated what current practices are– Discussed how to develop/approach reviews– Considered when to conduct reviews
• Answers to two important questions are missing:– Who should conduct reviews?– Do reviews make a difference on test performance?
What’s Left to Know About Sensitivity Reviews?
<< Main Focus
<< In Progress
3
Current Considerations of the “Who”
• Most commonly observed procedures/suggestions– Formally train reviewers
» Survey of current practices suggests this is rare (33% of reviewers)
– Minority review strategy (Camilli, 1993; Office for Minority Education, 1980)
» Review panel representative of diverse ethnic, gender, and geographic backgrounds (e.g., ACT, 2006; ETS, 2009)
» “Individuals sensitive to differences in cultural groups” (Reckase, 1996)
4
• Assumptions– Primary:
» Demographic representativeness increases likelihood of identifying material biased or offensive towards members of one’s subgroup
– Secondary: » Minorities better calibrated to identify problematic item content as a result of life
experiences» Sociopolitical and legal importance of diverse review panel
A Closer Look atMinority Review Strategies
“Minority review panels...serve important political and social justice goals. For these purposes, it is more appropriate that members of the [sensitivity review] panel represent constituencies pertinent to the use of the test rather than experts on the trait measured by the test.” (cited in Hood & Parker, 1989)
>
5
Evidence of Effectiveness of the Minority Review?
• Currently no evidence exists to directly support whether the demographics of reviewers influence sensitivity judgments– Engelhard, Hansche & Rutledge (1990)
» Neither Black nor White reviewers able to correctly identify which items in a test produce DIF at better than chance levels
» Furthermore, Black reviewers did not identify more biased items nor were they more confident in their judgments than White reviewers
• Individual differences likely affect judgments made by sensitivity reviewers, but has not been examined (Engelhard et al., 1990, Engelhard, Davis, & Hansche, 1999; Plake, 1980; Ramsey, 1993)
6
Research Questions
1. Do demographics influence sensitivity review judgments?
2. What individual differences influence sensitivity review judgments?
7
Hypothesis 6
Individuals who are more socially dominant and more strongly endorse status legitimacy beliefs will be less likely to detect insensitivity
Hypothesis 5
Individuals with greater cultural intelligence, better perspective taking ability, and greater empathic concern will be more likely to detect insensitivity
Hypothesis 4
The effects of gender and ethnicity on the detection of insensitivity will be moderated by gender and ethnic identification
Hypothesis 3
Stigma consciousness, perceived attributions to prejudice, and past experience with discrimination will mediate the relationship between reviewer demographics and detection of insensitivity
Hypothesis 2
Individuals who are more conscious of stigmas, have a greater tendency to make attributions to prejudice, and have had more experiences with discrimination will be more likely to detect insensitivity
Hypothesis 1
Females and non-Whites will be more likely to detect insensitivity than males and Whites
Research Model and Hypotheses
Stereotype-related Characteristics• Gender stigma consciousness• Ethnic stigma consciousness• Perceived attributions to prejudice• Past experience with discrimination
Psychological Characteristics
• Cultural intelligence• Perspective taking• Empathic concern
H1
Gender/Ethnicity
Detection of Insensitivity
H2H3
H4
H5
H6
Gender/Ethnic Identification
• Social dominance orientation• Status legitimacy beliefs
Stereotype-related Characteristics• Gender stigma consciousness• Ethnic stigma consciousness• Perceived attributions to prejudice• Past experience with discrimination
Psychological Characteristics
• Cultural intelligence• Perspective taking• Empathic concern
H1
Gender/Ethnicity
Detection of Insensitivity
H2H3
H4
H5
H6
Gender/Ethnic Identification
• Social dominance orientation• Status legitimacy beliefs
Demographics
Individual Difference Measures
8
Methods• n = 303 student reviewers (26% males, 15% non-White)
recruited to conduct a sensitivity review on a test of general knowledge– Demographics & individual differences collected online prior to
session– Provided brief instructional period on sensitivity reviews prior to
review task– Ratings collected on a 108-item test
9
Review Task: Test Breakdown
Category # Example Item Stem or Response Options
Non-problematic 54 The coach worked long and hard into the night for preparing the team’s strategy for the next game.
Offensive content 7 a. Some whites believing it’s fashionable to be Indian, stretch the truth about their ancestry, claiming, ‘My grandmother was a Cherokee princess.’
Offensive language 9 Some religious officials claim that the ancient Egyptian’s history of brutal violence, ritual sacrifices, and worship of non-Christian deities has contributed to the ________ of bloody genocide ravaging Eastern Africa.
Emotionally provocative content
11 a. Many single women with children choose not to apply for welfareb. According to a national survey, single childless women choose not to have
children because they lack monetary resources.
Portrayal of gender/ racial stereotypes
7 Grace Hopper should be an inspiration to female workers everywhere; not only did she prove that a woman could be highly successful in a field dominated by men, and she was able to do so without special treatment or excessive pleas for equality.
Unequal referrals to men and women
7 The temperaments of both architects were markedly different; Kevin was reserved and courteous, Joe was ________ and boastful.
Vocabulary unfamiliar to a group
7 In order to _________ a mortgage, an individual should periodically pay his or her lender principal and interest.
Content unfamiliar to a group
6 In India, seeing an elephant when one is leaving for a journey is considered ______ because an elephant represents Lord Ganesha, the Indian God who _______ obstacles.
Prob
lem
atic
Item
Cat
egor
ies
(54
item
s)
10
3.7
2.3
Calculating Detection of Insensitivity
Highly Insensitive Moderately Insensitive Possibly Insensitive Not Problematic
1 2 3 4
14Writing a good twenty-page research paper is more difficult than when you have to write two good ten-page papers.
03Those students who sit through her lectures day after day, having been numbed into thinking that history could never be even remotely interesting.
14The coach worked long and hard into the night for preparing the team’s strategy for the next game.
Non
-Pro
blem
atic
11Some religious officials claim that the ancient Egyptian’s history of brutal violence, ritual sacrifices, and worship of non-Christian deities has contributed to the ________ of bloody genocide ravaging Eastern Africa.
04The temperaments of both architects were markedly different; Kevin was reserved and courteous, Joe was ________ and boastful.
4
12a. Some whites believing it’s fashionable to be Indian, stretch the truth about their ancestry, claiming, ‘My grandmother was a Cherokee princess.’
Prob
lem
atic
Accuracy ScoreCorrect?Mean Item
Type RatingRatingItemItem Type
Correctly identified if rating = 1-3
Incorrectly identified if rating = 4
Correctly identified if rating = 4
Incorrectly identified if rating = 1-3
Manipulation Check:Significant difference (d = 1.49, p < .001) between problematic item ratings (M = 3.23) & non-problematic item ratings (M = 3.71)
11
• Hypothesis 1:– No main effect for Gender in ratings or accuracy– Significant main effects for ethnicity, but...
» Non-Whites tended to rate non-problematic items as slightly more insensitive than Whites (d = .19, p < .05) and were less accurate in their judgments overall (d = .36, p < .05)
– Significant interaction (p < .05) between Gender & Ethnicity on problematic item ratings
Results
Detection of Insensitivity
Gender
Ethnicity
ns
*
Non-white females = White males(Inconsistent with hypothesis)
12
Results• Hypothesis 2:– Individuals more aware of ethnic stigmas tended to rate
problematic items as more insensitive (β = -.19, p < .01)...– ...however, these individuals also tended to rate non-
problematic items as more insensitive (β = -.21, p < .01)
Stereotype-related Characteristics• Gender stigma consciousness• Ethnic stigma consciousness• Perceived attributions to prejudice• Past experience with discrimination
Detection of Insensitivity*
13
• Hypothesis 3:
– Non-Whites tended to exhibit greater awareness of ethnic stigmas, which resulted in a tendency to view both problematic and non-problematic items as more insensitive
Results
Ethnicity Ethnic Stigma Consciousness
Problematic item ratings
Non-problematic item ratings
β = -.24β = -.19
β = -.21***
14
Results• Hypothesis 4:
– Gender/ethnic identification did not moderate the relationship between demographics and sensitivity judgments
Gender/Ethnicity
Detection of Insensitivity
Gender/Ethnic Identification
ns
15
Results• Hypothesis 5:• Hypothesis 6:
– More socially dominant (β = .14, p < .05) and status legitimizing (β = .11, p = .06) individuals tended to rate problematic items as less insensitive
– More socially dominant individuals also tended to produce less accurate sensitivity judgments (β = -.12, p < .05)
Psychological Characteristics• Cultural intelligence• Perspective taking• Empathic concern Detection of
Insensitivity
H5
H6• Social dominance orientation• Status legitimacy beliefs
ns
*
16
Summary of Findings• No support obtained for minority review strategy
– Non-Whites appeared to perceive all items as generally more problematic
• Stereotype-related and psychological individual difference variables explained meaningful variance in sensitivity ratings– Part of the reason non-Whites tended to be slightly over-sensitive to
item content was attributable to their greater awareness of ethnic stigmas
– Individuals who believed in the legitimacy of social hierarchies and the need to earn one’s status in society were less sensitive to problematic content
17
Implications• Practical Implications
– Minority review strategy could potentially increase the rate of false alarms» Problematic as it lengthens development process and time-to-market
estimates– Diversity in the sensitivity review panel can still be beneficial to the
process (Hood & Parker, 1989), but it should not be the end goal
• Research Implications– Some individual difference predictors of sensitivity judgments were
identified, but many more failed– Continued investigation of psychological characteristics, dispositions,
and traits that influence the sensitivity review process