María José Rendón, B.A.
Stephanie Donnelly, M.Ed.
Billie Schwartz, M.A. Presenters
Guerda Nicolas, Ph.D. Chair
Janet Helms, Ph.D. Discussant
SYMPOSIUM OVERVIEW
• Studies of culturally and developmentally diverse groups have been critiqued for (mis)use of measures.
• This symposium examines measurement considerations in the assessment of culturally diverse children.
1. Equivalence and Bias Issues in Measurement
2. Developmental Considerations in Self-Report Data with Children
3. Adaptation of the Children’s Racial and Ethnic Socialization Scale (CRESS)
Using Measurement Instruments with Diverse Populations: A Review of Equivalence and Bias Issues in Measurement María José Rendón, B.A. University of Miami
OVERVIEW
• Equivalence, Bias, & Culturally Competent Assessment: Why Researchers Should Care
• Equivalence
Construct, Structure, Metric, Scalar
• Bias
Construct, Method, Item
• Minimizing Bias: Adaptation of Measures
• Testing for Bias: Psychometric Tools
• Measurement tools: Materializing the intangible…
• “Error” in measurement and our quest to minimize it.
EQUIVALENCE & BIAS: WHY BOTHER?
Construct
Instrument’s score
Error
Random error
Systematic error
•Measures of sociocultural identity cater to a specific population. •Disregarding equivalence and bias considerations threaten study credibility
AND is a disservice to your population of interest.
EQUIVALENCE & BIAS: WHY BOTHER?
• Cross-cultural psychology
– “Universal” vs. “Indigenous” perspectives
• Research with traditionally underrepresented groups in the U.S.
– “Universal” constructs imposed on ethnic / racial groups experiences
• Equivalence and bias relevant to minimize confounding factors in between-group differences.
Leong, Leung & Cheung (2010)
EQUIVALENCE
CONSTRUCT EQUIVALENCE •Does the construct exist in the population of interest? How is this construct operationalized?
STRUCTURAL EQUIVALENCE •Does the construct hold its relationships to related constructs? Does its factor structure hold?
METRIC EQUIVALENCE •Identical interval or ratio between units of measurement.
SCALAR EQUIVALENCE •Full scale equivalence: metric equivalence + origin equivalence.
Would racial socialization be culture-specific,
universal? Or partially equivalent?
Do the hypothesized dimensions for racial
socialization hold for the pop. of interest?
~For group comparison studies
~For group comparison studies
DO SCORES HAVE THE SAME MEANING ACROSS GROUPS?
(Van de Vijver & Leung, 2010; van de Vijver & Tanzer, 2004; Ægisdóttir et al., 2008)
CONSTRUCT BIAS •Incomplete overlap of construct operationalization across groups (though the idea is the same, the way it is expressed is culturally different).
•METHOD BIAS
•Sampling •Administration •Instrument
•ITEM BIAS •People with the same level of ability on the target trait have different scores
Does the expectation that racial socialization involves “promotion of cross-racial
relationships” hold for Caribbean immigrant
families? Is it relevant? Would lower scores
indicate this is low in the sample, or indicate that
the dimension is not relevant?
A consideration on construct bias
BIAS NUISANCE FACTORS INCREASING SYSTEMATIC ERROR
(Van de Vijver & Leung, 2010; van de Vijver & Tanzer, 2004)
SAMPLE BIAS •Differences in background variables (e.g., education, social desirability) beyond the grouping variable of interest (ethnicity). ADMINISTRATION BIAS •Differences in administration conditions (interview vs. survey method) •But consider: some groups will need interview method to be comparable in quality of responses to samples in which survey method was administered. INSTRUMENT BIAS •Different familiarity with response procedures.
Access to museums and resources in community
and school (beyond parental socialization practices at home).
Children needed interview aides to answer survey
format and explain concepts.
Scale modified for better understanding of
“magnitude”
BIAS NUISANCE FACTORS INCREASING SYSTEMATIC ERROR
ITEM BIAS •DIF: Differential item functioning •When people with the same level of ability on the target trait have different scores •Item does not capture the trait equally across groups.
Causes: •Poor item wording (ambiguity, low familiarity of item content in your sample, faulty translation) Consequences: •Respondents might misunderstand item or give responses that are not representative of their actual trait level
In measuring socialization into one’s group’s heritage, would this item be biased for children of Caribbean
immigrant families?
“In my family I am encouraged to watch
documentaries or movies on Black history”
BIAS NUISANCE FACTORS INCREASING SYSTEMATIC ERROR
MINIMIZING BIAS: ADAPTATION OF MEASURES
Concept Improving agreement of concepts across groups. E.g., Find the equivalent phrases to express the construct of racial identity across groups. Consult your local gatekeepers and your lit review.
Culture Adaptations to fit cultural norms and practices (e.g., behaviors unique to Caribbean groups in socializing children).
Language Applicable to translations (see van de Vijver & Leung, 2010), but also points to culture-specific conventions on the use of language (e.g., politeness)
Measurement Familiarity of group with assessment procedure (method bias).
A variety of adaptation models exist. We will point to Bernal & Roselló, 1995
(Van Vivjer & Leung, 2010)
TESTING FOR BIAS: PSYCHOMETRIC TOOLS
Testing for Construct & Structural Equivalence
Exploratory Factor Analyses Confirmatory Factor Analyses Path modeling (SEM)
Testing for Method Bias Include confounding variables into the design of the study. Conduct covariance or hierarchical regression analysis.
Testing for Item Bias Item Response Theory Mantel-Haenszel statistic SEM
(Sireci, Patsula & Hambleton, 2005; Corral & Landrine, 2010)
REFERENCES
• Ægisdóttir, S., Gerstein, L. H., & Çinarbaş, D. (2008). Methodological issues in cross-cultural counseling research: Equivalence, bias, and translations. The Counseling Psychologist, 36(2), 188-219. doi:10.1177/0011000007305384
• Brown, T. L., & Krishnakumar, A. (2007). Development and validation of the Adolescent Racial and Ethnic Socialization Scale (ARESS) in African American families. Journal of Youth and Adolescence, 36(8), 1072-1085. doi:10.1007/s10964-007-9197-z
• Leong, F. T. L., Leung, K., & Cheung, F. M. (2010). Integrating cross-cultural psychology research methods into ethnic minority psychology. Cultural Diversity and Ethnic Minority Psychology, 16 (4), 590–597.
• Sireci, S. G., Patsula, L. & Hambleton, R. K. (2005). Statistical methods for identifying flaws in the test adaptation process. In R. K. Hambleton, P. F. Merenda & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 93-115). Lawrence Erlbaum Associates, Mahwah, NJ
• Sue, S. (1996). Measurement, testing, and ethnic bias: Can solutions be found? In G. Sodowsky & J.C. Impara (Eds.), Multicultural assessment in counseling and clinical psychology (pp7-37). Lincoln, NB: Buros Institute of Mental Measurements.
• van de Vijver, F. R., & Leung, K. (2011). Equivalence and bias: A review of concepts, models, and data analytic procedures. In D. Matsumoto, F. R. van de Vijver, D. Matsumoto, F. R. van de Vijver (Eds.) , Cross-cultural research methods in psychology (pp. 17-45). New York, NY US: Cambridge University Press.
• van de Vijver, F., & Tanzer, N. K. (2004). Bias and equivalence in cross-cultural assessment: An overview. European Review of Applied Psychology/Revue Européenne de Psychologie Appliquée, 54(2), 119-135. doi:10.1016/j.erap.2003.12.004
LOGO
Obtaining Self-Report Data from Children: Developmental and Measurement Considerations
Stephanie Donnelly, M.Ed.
University of Miami
Children in Self-Report Research
What do we want to
know?
Experiences
Perceptions of Self
Perceptions of Others
Preferences
Etc.
Presentation Overview
Trends in Self-Report Research with Children
Developmental Considerations
Theories of Self-Concept
Developmental Differences
Measurement Considerations
Format
Scaling
Administration
Initial research focused developmental difference between ages
Later, use of theories of cognitive development to understand
the structural organization of self
Focus on individual differences in self-concept limited due to
measurement difficulties
Recent efforts have focused on understanding identity
development but this is largely limited to adolescents
Trends in Self-Report Research with Children
Looking Glass Self (Cooley, 1902)
Social construction of self
As children age make more comparisons and more influenced by
evaluation of others
Global Self-Perception (Coopersmith, 1967)
Ex. Self-esteem
Multidimensional Self (Harter, 1996; March 1987)
Self evaluations in different areas of life
More complex, dimensional with age
Hierarchical Models (Epstein, 1973)
Organization of self concept
Overall self-concept influenced by underlying multidimensional evaluations
Developmental Considerations: Theories of Self-Concept
Very Early Childhood (3-4)
• Concrete, observable characteristics
• Lack of coherence
• Overly positive
Early Childhood (5-7)
• Focus on competencies
• All or none thinking
• Compares self with others
Late Childhood (8-11)
• Global evaluations
• Greater accuracy
• Importance of other’s opinions
Developmental Considerations: Developmental Differences (Harter, 1999, pg 36)
Four-year-old
“I live in a big house with my mother and father and sister, Lisa. I
have a kitty that is orange and a television in my own room. I
know all my A, B, C’s listen. . . . I can run faster than anyone! I
like pizza and have a nice teacher. I can count to 100.”
Eight-year-old
“I’m in the third grade this year and pretty popular at least with
the girls. That’s because I’m nice, helpful, and can keep
secrets.. . . . .I’m feeling pretty smart in certain subjects,
Language Arts and Social Studies. I got A’s in these subjects on
my last report card and was really proud of my myself. But I’m
feeling pretty dumb in Arithmetic and Science, particularly when
I see how well the other kids are doing.”
(Harter, 1996, pg 208)
Developmental Considerations: Examples
Measurement Considerations: Scaling
Picture Weights Line Marking
Draw a line on the
thermometer to
show how angry
you get when
another kid takes
your toy.
Measurement Considerations: Administration
Varies depending on age
Probe for understanding
Children often need
assistance to maintain
attention
Adaptations for a variety
of reading levels
Use Response Cards
Definition list
Use of advanced technology to collect data from children
Further efforts to validate data collection methodologies and
specific scales is needed
More consistency in use of strong measures
Comparison of self-report to other sources of information
Adaptation of measures for use with
ethnically diverse populations
Future Directions
Summary
Self-report research with children is possible!!
Consider developmental abilities when determining concepts to
measure and measurement methods.
Guide research choices- questions, measures, etc., on theory of
development.
Different methods may be needed to collect data from children
of different ages (even in the same sample).
Methods should keep children’s attention, test their
understanding, and be fun.
Cooley, C. H. (1902). Human nature and the social order. New York: Scribner’s.
Coopersmith, S. (1967). The antecedents of self-esteem. San Francisco, CA: Freeman.
Epstein, S. (1973). The self-concept revisited or a theory of a theory. American Psychologist,
28, 405-416.
Harter, S. (1996). Historical roots of contemporary issues involving self-concept. In B. A.
Bracken, B. A. Bracken (Eds.) , Handbook of self-concept: Developmental, social, and
clinical considerations (pp. 1-37). Oxford England: John Wiley & Sons.
Harter, S. (1996). Developmental changes in self-understanding across the 5 to 7 shift. In A.
J. Sameroff, M. M. Haith, A. J. Sameroff, M. M. Haith (Eds.) , The five to seven year shift: The
age of reason and responsibility (pp. 207-236). Chicago, IL US: University of Chicago Press.
Harter, S. (1998). The development of self-representations. In N. Eisenberg, W. Damon, N.
Eisenberg (Eds.) , Handbook of child psychology, 5th ed.: Vol 3. Social, emotional, and
personality development (pp. 553-617). Hoboken, NJ US: John Wiley & Sons Inc.
Harter, S. (1999). The construction of the self: A developmental perspective. New York, NY:
The Guilford Press.
Marsh, H. W. (1987). The hierarchical structure of self-concept and the application of
hierarchical confirmatory factor analysis. Journal of Educational Measurement, 24, 17-19.
References
The Children’s Racial and Ethnic Socialization Scale (CRESS): The Cultural and Developmental Adaptation of a Socialization Measure for Children Billie Schwartz, M.A.
University of Miami
Why study racial & ethnic socialization?
• Research has shown that Black children will face racism and discrimination that will impact their development
• The concept of identity in young children is currently debated—can identity be measured this young?
• Racial socialization is defined as messages taught or communicated regarding race and ethnicity
• The family is considered the primary socializing factor for kids
• Positive messages can lead to greater academic achievement and self-esteem and sense of identity
• Few researchers have attempted to combine these factors into a measure for children
Adolescent Racial and Ethnic Socialization Scale (ARESS) (Brown & Krishnakumar, 2007)
• Racial Socialization (RS) & Ethnic Socialization (ES)
• RS: 16items (i.e., racial barrier awareness, coping with racism and discrimination, the promotion of cross racial relationships)
• ES: 22 items (i.e., cultural values, cultural embeddedness, history, promotion of ethnic pride)
• Youth reported on a Likert scale of 1–4
• Questions asked about messages they’ve received from maternal/paternal caregivers
• To adapt the ARESS for use with for elementary-aged children
• Children’s Racial and Ethnic Socialization Scale (CRESS)
• By creating a scale for younger children:
• Identify problems at an earlier age
• Be better informed on children’s beliefs and attitudes regarding the self
• Help coping with and navigating racism and discrimination
• Create an effective assessment tool to better understand the messages that children receive about their racial and ethnic heritage
Purpose of the Study
Adaptation Procedures
• First group of young children given the ARESS without changes
• Children reside in the Miami-Dade County area.
• 2009-2010 academic school year
• Group served as a focus group providing feedback at four week intervals
• Preliminary analysis showed difficulties three distinct areas requiring further adaptation
• words, language and content
• the concept of a Likert scale
• administration method
Adaptation: Language & Content
•The Ecological Validity and Culturally Sensitive Framework - Bernal and RossellÓ (1995)
•Vocabulary level not developmentally appropriate for use with children
•Definition list for administrator
•3 Qualitative questions :
•Better understand who is talking to the kids
•The length was reduced: 35-items to 19-items (quantitative)
•Subscale of Cultural Embeddedness removed for content with young children
Bernal et al., 1995
Elements Culturally Sensitive Language Culturally appropriate, culturally
syntonic language Metaphors Symbols and concepts shared with
the populations Content Cultural knowledge: values,
traditions; uniqueness of groups Concepts Treatment concepts consonant
with culture and context Goals Support of adaptive values from
culture of origin Methods Development and cultural
adaptation of treatment methods Context Consideration of changing
contexts: acculturative stress, phase of migration, social supports and relationship to country of origin, etc.
Original Items ARESS Adapted Items CRESS
My maternal/paternal caregiver encourages me to have Black friends
How much are you told to have friends that have the same skin color as you?
My maternal/paternal caregiver teaches me the importance of family loyalty
How much does someone talk to you about the importance of sticking together as a family?
My maternal/paternal caregiver teaches me that my skin color is beautiful
How much are you told that your skin color is beautiful?
Examples of Language Changes
Adaptation: Scaling & Administration
Scaling
•Participants had difficulties with the Likert format and marked only the extremes
•Reduction from 4 options to 3
•Introduction of picture scale
Administration
•Changed from a self-report administration to a semi-structured interview
•Depending on reading level, either one-to-one or small group of 2-4 to one adult
Summary
• The CRESS is an assessment of racial and ethnic socialization messages received by elementary-aged Black children
• Changes in:
• wording and content
• scaling
• administration style
• CRESS helps to understand the messages kids receive about their racial and ethnic identity
• Not a measure of racial and/or ethnic identity
• Future studies will examine the psychometric properties
Bernal, G., Bonilla, J., & Bellido, C. (1995). Ecological validity and cultural sensitivity for outcome research: Issues for the cultural adaptation and
development of psychosocial treatments with Hispanics. Journal of Abnormal Child Psychology, 23, 67–87.
Brown, T. L., & Krishnakumar, A. (2007). Development and validation of the Adolescent Racial and Ethnic Socialization Scale (ARESS) in African American
families. Journal of Youth and Adolescence, 36(8), 1072–1085.
Brown, T., Linver, M., Evans, M. & DeGennaro, D. (2009). African-American parents’ racial and ethnic socialization and adolescent academic grades:
teasing out the role of gender. Journal of Youth Adolescence, 38, 214-227.
Clark, K. and Clark, M. (1939). The development of consciousness of self and the emergence of racial identification in Negro preschool children. Journal of
Social Psychology, 10, 591-599.
Clark, K. B. and Clark, M. K. (1947). Racial identification and preference. Readings in Social Psychology, 16-178.
Coard, S., Foy-Watson, S., Zimmer, C. & Wallace, A. (2007). Considering culturally relevant parenting practices in intervention development and
adaptation: a randomized controlled trial of the Black Parenting Strengths and Strategies (BPSS) Program. The Counseling Psychologist, 35, 797-820.
Coard, S. I., & Sellers, R. M. (2005). African American families as a context for racial socialization. In V. C. McLoyd, N. E. Hill, & K. A. Dodge (Eds.), African
American family life: Ecological and cultural diversity (pp. 264-284). New York: Guilford.
Coard, S. I., Wallace, S. A., Stevenson, H. C., & Brotman, L. M. (2004). Towards culturally relevant preventative interventions: The consideration of racial
socialization in parent training with African American families. Journal of Child and Family Studies, 13(3), 277–293.
Hughes, D. & Johnson, D. (2001). Correlates in children's experiences of parents' racial socialization behaviors. Journal of Marriage and Family, 63(4), 981-
995.
Hughes, D., Smith, E. P., Stevenson, H. C., Rodriquez, J., Johnson, D. J., & Spicer, P. (2006). Parents’ ethnic and racial socialization practices: A review of
research and directions for future study. Developmental Psychology,42(5), 747–770.
Lesane, C. L. (2002). Race socialization in Black families: A selective review of the literature. African American Research Perspectives, 8(1), 27–34.
Marshall, S. (1995). Ethnic socialization of African American children: Implications for parenting, identity development, and academic achievement. Journal
of Youth and Adolescence, 24(4), 377–396.
Phillips Smith, E., Walker, K., Fields, L., Brookins, C. & Seay, R. (1999). Ethnic identity and its relationship to self-esteem, perceived efficacy and prosocial
attitudes in early adolescence. Journal of Adolescence, 22, 867-880.
Phinney, J. S. (1990). Ethnic identity in adolescents and adults: Review of research. Psychological Bulletin, 108, 499-514.
Stevenson, H., Reed, J., Bodison, P., & Bishop A. (1997). Racism stress management: racial socialization beliefs and the experience of depression and
anger in African American youth. Youth Society, 29(2), 197-222.
Thomas, A. J., & Speight, S. L. (1999). Racial identity and racial socialization attitudes of African American parents. Journal of Black Psychology, 25(2), 152–
170.
References