Date post: | 12-Jan-2016 |
Category: |
Documents |
Upload: | charity-black |
View: | 216 times |
Download: | 0 times |
Quality strategies in cross-national surveys
The case of the European Social Survey
Ineke Stoop
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Quality and comparability
Harmonisation strategies and cross-national surveys
Optimal comparability
European Social Survey
Nonresponse
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Harmonisation strategies(Körner and Meyer, 2005)
Harmonised concept
Harmonised concept
Harmonised concept
Measurementprocedure
Measurementprocedure
Measurementprocedure
National survey/Specific concept
National survey/Specific concept
National survey/Specific concept
Input harmonisationEx-ante outputharmonisation
Ex-post outputharmonisation
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Type of harmonisation
Ex post output harmonisation• Use existing sources
Ex ante output harmonisation• Set up new survey (or develop new instrument)
Output harmonisation• Best national quality• Or, national survey tradition• But, what about optimal comparability
Input harmonisation• Design new survey• Optimal comparability• Or, are identical methods and instruments really equivalent in different countries?• But, what about optimal quality
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Strategies in comparative survey
Quality
Optimal national approach• Comparability?• Poor performance
accepted
Some restrictions and standards
Consistency
Identical approach• Lowest standards
• Possible?• Pull down level
• Highest standards• Possible?• Pull up level
High standards and optimal national approach
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Mixed approach
Focus on those key aspects that may hamper comparability• Data collection mode• Sampling approach• Response rates• Question formulation
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
European Social Survey
Attitudes, values and beliefs
Bi-annual: 2002, 2004, 2006, 2008
Fixed core, rotating modules
Effective size 1500 or more
Central co-ordination
National implementation• 30 National Coordinators responsible for fieldwork
Face-to-face
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Aims European Social Survey
To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions
• Provide data on beliefs, attitudes and values for scientific and policy making purposes
• Measure attitude change in a changing Europe
To advance and consolidate improved methods of cross-national survey measurement in Europe and beyond
• Collect data according to highest standards
• Generate methodological research
• Develop and disseminate new best practices
• Develop and improve social survey research infrastructure in Europe
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
EVERYTHING DOCUMENTED
EVERYTHING AVAILABLE
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Nonresponse: consistency
Strict probability sampling
No substitution
Fieldwork monitoring
Guidelines response enhancement• Incentives• Brochure
Guidelines interviewer training
Target noncontact rate: 3%• Minimum number of calls• Timing of calls (evening,
weekend)
Target response rate: 70%• Refusal conversion
suggested
Contact form and call records
Standard response rate calculation
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Nonresponse: divergence
Population• Residential population
• Minority languages
Sampling frame• Addresses, households
• Individuals
At home pattern• Female employment
• When does the evening start?
Fieldwork organisation• Field director’s meeting
Interviewers• Experience
• F2F
• Random sampling
• Training and briefing
• Remuneration• Response, hours, staff
Mode of contacting
Use of advance letters
Incentives
Contact forms
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Response rates ESS (deviation 70%)
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Why do response rates differ?
Survey climate• Used to surveys?
Popularity topic• Europe?
Country size, urbanicity and contactability
Survey modes• Predominantly telephone?
Fieldwork efforts
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
UP TILL NOW
NO UNIVOCAL PREDICTOR
OF
RESPONSE RATES
ACROSS COUNTRIES
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
% noncontact and refusal (R1, R2, R3)
% noncontact
% r
efu
sal
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Age, gender, education (Vehovar & Zupanic)
Age: underrepresented• 35+ (Austria)• 55+ (Belgium and Luxemburg)• 15-34 (Spain, Ireland, Netherlands, UK)
Education: underrepresented• Middle (Austria, France, Iceland, Luxembourg, the Netherlands, Norway, Sweden
and the United Kingdom)• Less (Switzerland, Czech Republic, Germany, Estonia, Hungary, Norway,
Slovenia, Slovakia and Ukraine)
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Results multivariate analysis: reluctance(Beullens, Billiet and Loosveldt)
Switzerland• Members of larger families
Germany• Female, aged, city dweller• Internet, unemployment history• Less political participation
Estonia• Female, not in village• Employed
Slovakia• Older, average education, church
life• Never job, less safe, comfortable
family income
Netherlands• Female, average education, more
TV, internet• Paid job, healthy• Immigrants threat, trust political
institutions• Civil obedience, political
participation• Socially isolated, dissatisfied
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Nonresponse bias in cross-national studies
How to measure bias?• Different type of auxiliary information available
• Sample frame/register information?
• Reluctant respondents?• Neighbourhood information?
Different size of bias?
Different type of bias?
Different bias over time?
ESS-experiments in several countries• Mixed results
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Is output harmonisation better?
Maybe, but in that case do you know about• Response rates (standard calculation)• Intensity and effect of field efforts• Nonresponse composition• Underrepresented groups• Nonresponse bias
Black box:• Differences between countries due to methodology or
substance?
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Are comparative surveys possible?
If not, national surveys are not possible either• Subgroups
• Socio-demographic• Survey interest• Topic relevance
• Regions• National languages• Interviewers
There is so much we don’t know, and now we know so much more
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Thank you for your attention