Cultural differences as statistical artefacts?

Reanalysing cross-national data with more advanced techniques

QMSS Conference Prague 21/06/2007

Dr Michael HoelscherDepartment of Education

University of Oxford

Overview

Context of the study Introduction to data Cultural differences within Europe – a first

approach Reanalysing the data with CFA Applying a correction for measurement errors Conclusions

1. Context of the study

European integration and enlargement often discussed in economic terms only

However: Cultural influences might play a crucial role Comparison of values in different spheres for all

countries in the EU (Religion, Family and Gender, Economy, Welfare State, Democracy)

“Normative” starting point: Position of the EU institutions, as found in its body of law and the treaties

Three year project, financed by VolkswagenStiftung

2. Introduction to data

European Values Study– 1999/2000– Wide variety of topics– Including all member and applicant countries of the EU

(except Cyprus)

28 countries are compared in our study Today the focus is on “Democracy and Civic Society” Secondary analysis

– Indicators are not always “perfect”

2. Introduction to data

Democracy: 4 Indicators

– “Having a strong leader” (v216)– “Having the army ruling” (v218)– “Having a democratic political system” (v219)– “Democracy may have problems, but best form of

government” (v220)

(all measured on a scale with 4 categories)

2. Introduction to data

Civic Society:

2 Indicators

– “People can be trusted” (v66)– “Membership in voluntary organisations”

(Index of membership in 14 groups; trade union membership is ignored)

3. Cultural differences within Europe – a first approach

Methods Comparisons of raw country means for each

indicator Integration of single indicators by using a

discriminant analysis

(see Fuchs/Klingemann 2002 in “West European Politics”)

Explanation of differences on the individual level by multiple regressions

3. Cultural differences within Europe – a first approach

Results Large differences between the countries, but

also within the countries Old member countries support position of EU

most, followed by new members Bulgaria, and especially Romania and Turkey

showed much lower support

3. Cultural differences – a first approach

Sweden 1Netherlands 2Denmark 3Finland 4Austria 5Belgium 6Germany_West 7Greece 8Luxembourg 9Ireland 10Czech Republik 11Germany_East 12Italy 13Malta 14France 15Slovenia 16Spain 17Slovakia 18Great Britain 19Estonia 20Hungary 21Bulgaria 22Portugal 23Poland 24Lithuania 25Latvia 26Romania 27Turkey 28

Overall support for the EU’s position in the field of Democracy/Civic Society

(RANK)

4. Re-analysing the data with CFA

Aim – To compare two different methods– Not: Building the best model!

Balance of model fit and equivalence of approaches is needed

4. Re-analysing the data with CFA

Advantages of CFA

Generally– CFA is the more appropriate technique– More flexible– Can easily be extended to an explanatory SEM

4. Re-analysing the data with CFA

Advantages

Measurement model– Test if measurement is the same in different

countries and therefore a comparison is appropriate

– Correction for measurement error possible (Saris/Gallhofer 2007)

4. Re-analysing the data with CFA

Advantages

Structural model– Relationship between “democracy” and “civic

society” can be estimated

4. Re-analysing the data with CFA

Great Britain, N = 728

Democracy

Zv216 e1.50

Zv218 e2.39

Zv219r e3.73

Zv220r e4.59

Civic SocietyZv66r e5

Zlmember e6

.53

.34

.39

.23

Standardized estimateschi-square=9.505 df=7 p-value=.218

gfi=.996 agfi=.987 rmsea=.022

4. Re-analysing the data with CFA

Running the model for all 25 countries without constraints

Chi-square = 333.29, df = 175, p-value=.000 CFI = .988 RMSEA = .006 (adjusted: 0.032)

All modification indices within the countries are well below 20, in most cases below 5

One can assume configural invariance

Introducing constraints: Model comparison

4. Re-analysing the data with CFA

2. Model “Equal Measurement Weights”Chi-square= 725,5 df = 271CFI = .965RMSEA = .008 (adjusted .04)

3. Model “Equal Measurement Weights and Intercepts”Chi-square= 4752.256 df = 367CFI = .666RMSEA = .023 (adjusted .115)

1. Model “Unconstrained”Chi-square= 333,29 df = 175CFI = .988RMSEA = .006 (adjusted .032)

Introducing constraints: Model comparison

4. Re-analysing the data with CFA

2. Model “Equal Measurement Weights”

=> Metric invariance can be assumed

3. Model “Equal Measurement Weights and Intercepts”

=> Scalar invariance can not be assumed!=> Mean comparison is (in priniciple) not appropriate with this model=> Adjustments (freeing some parameters)

1. Model “Unconstrained”

=> Configural invariance can be assumed

4. Re-analysing the data with CFA

SwedenCFA-Rank

1RANK CFA-Book

0Denmark 2 -1Netherlands 3 1Austria 4 -1Luxembourg 5 -3Germany_West 6 -1Belgium 7 1Italy 8 -4Ireland 9 0Finland 10 6Czech Republic 11 1Malta 12 -1France 13 -1Romania 14 -11Spain 15 -1Slovakia 16 -1Slovenia 17 2Germany_East 18 7Great Britain 19 1Bulgaria 20 -1Hungary 21 1Estonia 22 3Lithuania 23 0Poland 24 2Latvia 25 1

Comparison of ranks

5. Correction for measurement errors

SEM allows to correct for measurement errors Saris, Gallhofer et al. (2007) have introduced a tool

to estimate the quality (reliability and validity) of an instrument

From a huge amount of MTMM experiments they estimated the influence of certain characteristics on the quality

By coding one’s own questions one can predict their quality

=> http://www.sqp.nl/

5. Correction for measurement errors

Idea:– What has to be equal for cross-country-comparisons

is the factor structure– The quality of the instrument might influence this

factor structure, if one does not correct for measurement error if the quality is different in different countries

“We suggest that equivalence should (…) be tested by the equality of loadings based on the observed covariance matrix corrected for measurement error”

5. Correction for measurement errors

y1 T1

F

e1

q1

λ1

y2e2

q2 T2

y3e3

q3 T3

λ2

λ3

Indicators True scores Latent concept (by definition)

5. Correction for measurement errors

Applying the correction to a subsample of 9 countries:

- “Democracy”-indicators - The validity was nearly 1 for all countries- Reliability is different in countries, but reasonably

good- Problems with the “Civic Society”-indicators

- Unable to code the quality of the index straightforward

- Low quality of the “Trust” variable

5. Correction for measurement errors

Results:

- Factor loadings increase- Model fit decreases very slightly- At least for this specific subsample the ranks

do not change - Check for whole sample, especially the

“difficult” cases, is still missing

6. Conclusions

Advantages of the SEM approach

More appropriate More flexible (integration of additional indicators) Can detect problems with measurement model Easily extendable to an explanatory model Relationship between the latent constructs can

be estimated

6. Conclusions

“Problems” of the SEM approach

More demanding (data quality) Is it realistic to assume equal means and factor

loadings over so many countries? Partial invariance?

Taking requirements very seriously wouldn’t allow a comparison of all countries

6. Conclusions

Comparing the “outcome” of the three methods:

Small differences for the overall ranking

The methods seem to come to pretty similar results

However: Some extreme cases (Turkey), couldn’t be included or shifted quite a lot (Romania)

Thank you!

Dr Michael HoelscherDepartment of Education

University of Oxfordmichael.holscher@edstud.ox.ac.uk

Quantitative Methods in the Social Sciences Conference, Prague, 20-23 June 2007

Literature

Michael Hoelscher (2006): Wirtschaftskulturen in der erweiterten EU. Die Einstellungen der Buergerinnen und Buerger im europaeischen Vergleich. Wiesbaden: VS Verlag

Juergen Gerhards (unter Mitarbeit von Michael Hoelscher) (2005, second edition 2006): Kulturelle Unterschiede in der Europaeischen Union. Wiesbaden: VS Verlag

Dieter Fuchs/Hans-Dieter Klingemann (2002): Eastward Enlargement of the European Union and the Identity of Europe. West European Politics, 25, 2: 19-54.

Willem E. Saris/Irmtraud Gallhofer (2007): Design, Evaluation, and Analysis of Questionnaires for Survey Research. Wiley.