Date post: | 03-Jan-2016 |
Category: |
Documents |
Upload: | keegan-rice |
View: | 19 times |
Download: | 0 times |
SSRC Eurasia Quantitative Methods WebinarCultural Context and Measurement Validity in Comparative Survey Research
Professor Jane Zavisca University of Arizona
April 12, 2013 [email protected]
Core dilemmas In comparing groups, want to be sure observed
differences (or similarities) are substantive, not artifactual
Simultaneous need for: Identities: universal measures with comparable
meanings across contexts Equivalents: Particular measures that capture
same concepts across contexts In cross-national surveys, often have identities –
but are they equivalent?Bollen, Kenneth A., Barbara Entwisle, and Arthur S. Alderson. 1993. “Macrocomparative Research Methods.” Annual Review of Sociology 19 (January 1): 321–351. doi:10.2307/2083391.
Przeworski, Adam, and Henry Teune. 1966. “Equivalence in Cross-National Research.” Public Opinion Quarterly 30 (4) (December 21): 551–568. doi:10.1086/267455.
Heath, Anthony, Stephen Fisher, and Shawna Smith. 2005. “The Globalization of Public Opinion Research.” Annual Review of Political Science 8 (1): 297–333. doi:10.1146/annurev.polisci.8.090203.103000.
Measurement equivalence Functional equivalence: concordance of
meaning, of constructs as well as questions Measurement invariance: formal statistical
equality of parameters in measurement models
Sources of non-equivalence Errors of observation
Substantive differences in meaning: linguistic, contextual
Systematic differences in response styles: acquiescence bias, extreme response bias, social acceptability bias
Errors of non-observation non-response bias Sampling approach
Heath, Anthony, Jean Martin, and Thees Spreckelsen. 2009. “Cross-national Comparability of Survey Attitude Measures.” International Journal of Public Opinion Research 21(3): 293–315. doi:10.1093/ijpor/edp034.
Language example: happiness Poor linguistic equivalence
English: “Are you a happy person?” Russian: ”Вы счастливый человек? / Вы
счастливчик?» Could be interpreted as “Are you a lucky person?” О счастливчик! = name of “Who Wants to Be a
Millionaire” reality show in Russian Better linguistic equivalence
English: “Are you: very happy, pretty happy, not too happy, not happy at all.”
Russian: Вы: очень счастливы, довольно счастливы, не очень счастливы, очень несчастны.»
See: RUSSET Panel Survey: www.vanderveld.nlWorld Database of Happiness: http://www1.eur.nl/fsw/happiness/index.html
Relationship between happiness & life satisfaction “Varies with the cultural and linguistic
environment in which it is studied.” Constructs vs. questions: differences in meaning
even with linguistic equivalence In Russian surveys
Low correlations between general happiness and life satisfaction
High correlation between general happiness and satisfaction with personal relationships;
high correlation between life satisfaction and satisfaction with finances.
Saris, Willem E., and Anna Andreenkova. 2001. “Following Changes in Living Conditions and Happiness in Post Communist Russia: The RUSSET Panel.” Journal of Happiness Studies 2: 95-109.
Contextual difference: homeownership Conventional definition of homeowner:
resident of “owner-occcupied household” Russia has highest rate of young
homeownership (ages 21-35) in Western & Eastern Europe according to this definition (about 85%).
But most are not living autonomously
“Homeownership” Rates Ages 21-35, with and without extended family
DETECTING MEASUREMENT NON-EQUIVALENCE
USING MULTIPLE INDICATORS AND
LATENT VARIABLE MODELS
Limitations of single measures Impossible to statistically test whether observed
difference (or similarity) is meaningful Example: consumer ethnocentrism
Russians are more likely than Canadians to agree with the statement: “There should be very little trading or purchasing of goods from other countries unless out of necessity.”
Possible sources of non-equivalence Translation Different interpretation, measuring different constructs
Saffu, Kojo, and John Hugh Walker. 2005. “An Assessment of the Consumer Ethnocentric Scale (CETSCALE) in an Advanced and Transitional Country: The Case of Canada and Russia.” International Journal of Management 22 (4): 556-571.
Latent variable approach
Consumer ethnocentris
m
should always
buy domesti
c
Foreign products only out
of necessity
Real (national
) buys domestic
Curbs should be put on all
imports
Foreigners should not be allowed
to sell
Buy domestic
Keep country working
Purchasing
foreign is un-
patriotic
(National) products, first, last, foremost
Foreign should
be heavily taxed
In latent variable model, oval represents latent construct, square represents manifest indicator.
Finding for Russia
Consumer ethnocentri
sm
should always
buy domesti
c
Foreign products only out
of necessity
Real (national
) buys domestic
Curbs should be put on all
imports
Foreigners should not be allowed
to sell
Buy domestic
Keep country working
Purchasing
foreign is un-
patriotic
(National) products, first, last, foremost
Foreign should
be heavily taxed
cultural
Economic
Example: National Identity Defined as beliefs about importance of potential
determinants of membership of the nation. Theory suggests 2 dimensions
Civic: residence, citizenship, respect for law/institutions
Ethnic/ascriptive: birthplace, descent, religion Common example in methodological literatureDavidov, Eldad. 2009. “Measurement Equivalence of Nationalism and Constructive
Patriotism in the ISSP: 34 Countries in a Comparative Perspective.” Political Analysis 17 (1) (December 21): 64–82. doi:10.1093/pan/mpn014.
Heath, Anthony, Jean Martin, and Thees Spreckelsen. 2009. “Cross-national Comparability of Survey Attitude Measures.” International Journal of Public Opinion Research 21 (3) (September 21): 293–315. doi:10.1093/ijpor/edp034.
Loner, Enzo, and Pierangelo Peri. 2009. “Ethnic Identification in the Former Soviet Union: Hypotheses and Analyses.” Europe-Asia Studies 61 (8): 1341–1370. doi:10.1080/09668130903134798.
Kunovich, Robert M. 2009. “The Sources and Consequences of National Identification.” American Sociological Review 74 (4) (August 1): 573–593. doi:10.1177/000312240907400404.
Sarrasin, Oriane, Eva G. T. Green, André Berchtold, and Eldad Davidov. 2012. “Measurement Equivalence Across Subnational Groups: An Analysis of the Conception of Nationhood in Switzerland.” International Journal of Public Opinion Research (October 18). doi:10.1093/ijpor/eds033.
Measurement model for pooled ISSP data
Source: Heath 2009
Civic
Ethnic
Born in nation
Nat’l ancestr
y
Nat’l religion
Life-long
resident
Citizen-ship
Respect laws
Speak languag
e
Feel nat’l
identity
Citizenship as ethnic
Source: Heath 2009
Countries where citizenship rules restrictive, ascriptiveCountries that are ethnically homogenousIncludes most former Soviet countries, Austria, Switzerland
Citizenship as civic
Source: Heath 2009
Countries where citizenship rules less restrictiveCountries with historical ethnic diversityIncludes Czech Republic, Spain, Australia, Israel
Formal tests for measurement invariance Configural invariance (weak): factor structure
equivalent (same items load on same latent variables) Metric invariance (strong): factor loadings equivalent:
necessary to compare relationships between constructs (e.g. regression coefficients).
Scale invariance (strict): factor loadings and intercepts equivalent; necessary to compare means
Partial invariance: at least two items load equally on each construct.Byrne, Barbara M., and Fons J. R. van de Vijver. 2010. “Testing for Measurement
and Structural Equivalence in Large-Scale Cross-Cultural Studies: Addressing the Issue of Nonequivalence.” International Journal of Testing 10 (2): 107–132. doi:10.1080/15305051003637306.
Cheung, Gordon W. 2008. “Testing Equivalence in the Structure, Means, and Variances of Higher-Order Constructs With Structural Equation Modeling.” Organizational Research Methods 11 (3) (July 1): 593–613. doi:10.1177/1094428106298973.
Steenkamp, Jan‐Benedict E. M., and Hans Baumgartner. 1998. “Assessing Measurement Invariance in Cross‐National Consumer Research.” Journal of Consumer Research 25 (1) (June 1): 78–107. doi:10.1086/209528.
What to do when invariance not achieved Delete problematic groups or countries
from comparison set Delete problematic items from
measurement model Settle for partial invariance: configural
invariance, plus at least 2 indicators per latent variable with equal loadings and/or intercepts
Problem with formal tests of invariance Formal strictness can undermine substance “Just as cross-national researchers
recognize that indicators may need non-literal translation to maximize the comparability between countries, constructs also may need non-literal translation.” (Medina et al 2009)
Achieving functional equivalence Statistical approaches
“Locally-conditioned models”: control for reasons for invariance (Medina 2009)
Introduce contextual predictor variables – i.e. directly model cross-group differences (Davidov 2012)
Qualitative context Investigate sources of invariance through
cognitive interviews, observation (Carnaghan 2011)Davidov, Eldad, et al. 2012. “Using a Multilevel Structural Equation
Modeling Approach to Explain Cross-Cultural Measurement Noninvariance.” Journal of Cross-Cultural Psychology 43 (4): 558–575. doi:10.1177/0022022112438397.
Carnaghan, Ellen. 2011. “The Difficulty of Measuring Support for Democracy in a Changing Society: Evidence from Russia.” Democratization 18 (3): 682–706. doi:10.1080/13510347.2011.563113.
RESPONSE STYLE BIAS
Response styles Tendencies in responses, independent of true
belief. Types of response styles
Tendency to agree (acquiescence bias) Tendency to moderate responses Tendency to extreme responses
Create bias in likert scales (e.g. agree-disagree), yes/no questions
Problematic when tendency to acquiesce varies across groups to be comparedHarzing, Anne-Wil. 2006. “Response Styles in Cross-national Survey Research A 26-
country Study.” International Journal of Cross Cultural Management 6 (2) (August 1): 243–266. doi:10.1177/1470595806066332.
Kieruj, Natalia D., and Guy Moors. 2013. “Response Style Behavior: Question Format Dependent or Personal Style?” Quality & Quantity 47 (1) (January 1): 193–211. doi:10.1007/s11135-011-9511-4.
Tobi, Hilde, and Jarl K. Kampen. 2013. “Survey Error in an International Context: An Empirical Assessment of Cross-cultural Differences Regarding Scale Effects.” Quality & Quantity 47 (1) (January 1): 553–559. doi:10.1007/s11135-011-9476-3.
Example: acquiescence in Kazakhstan Survey experiments on tendency to agree
among Kazakh vs. Russian respondents in Kazakhstan
Javeline, Debra. 1999. “Response Effects in Polite Cultures: A Test of Acquiescence in Kazakhstan.” The Public Opinion Quarterly 63 (1) (April 1): 1–28. doi:10.2307/2991267.
Possible causes of acquiescence bias Cultural significance of hospitality,
deference, avoiding offense Uncertainty about answer; assume
statements contain cues about correct answer
Cognitive burden Must infer counterarguments If agree with inferred counterargument, must
disagree with statement (yes/no) Satisficing due to fatigue, disinterest
Example: freedom versus order Version A: “People should be free to say whatever they
want, even if what they say increases tensions in society.” Version B: “Public order should be maintained above all,
even if it requires limiting freedom of speech.” Version C: “Certain people think that it is better to live in a
society with strict order, even if it requires limiting freedom of speech. Others think that people should be free to say whatever they want, even if what they say increases tensions in society. Which view is closer to your own? Do you feel this way strongly or only somewhat?”
Response options for versions A & B are: strongly agree/ somewhat agree/ somewhat disagree/ strongly disagree
Version C is a “forced choice” format question
Evidence for acquiescence bias
freedom -- very freedom -- somewhat
order -- somewhat
order -- very0
5
10
15
20
25
30
35
40
45
A: pro-freedomB: pro-orderC: forced choice
In absence of bias, proportions should be the same for all 3 versions.Acquiescence bias overstates support for freedom in version A, understates
support for freedom in version B.
Comparison of Kazakhs & Russians
freedom -- very
freedom -- somewhat
order -- somewhat
order -- very
0
5
10
15
20
25
30
35
40
45
freedom -- very
freedom -- somewhat
order -- somewhat
order -- very
0
5
10
15
20
25
30
35
40
45
KazakhRussian
VERSION A VERSION B
VERSION C
CONCLUSION:Acquiescence bias inVersion Aunderstates differencesBetween groups
freedom -- very
freedom -- somewhat
order -- somewhat
order -- very
0
5
10
15
20
25
30
35
40
KazakhRussian
What to do Avoid likert scales (esp agree/disagree) Use forced choice formats If using likert scale: Randomize direction of
question