Centraal Bureau voor de Statistiek
Challenges of redesigning household surveys and maintaining output quality
Menno CuppenPaul van der LaanWim van Nunspeet
Statistics Netherlands
Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
May 6, 2010
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Outline
• Driving forces
• Approach
• Maintaining quality
• Results and perspectives
• Conclusions
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Driving forces
• Efficiency
• Quality
• Flexibility
Constraints:
• (European) quality / user satisfaction
• Solution for discontinuities
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Approach
Means:
• More use of registrative data
• More use of CAWI and mixed mode designs
• Integration of surveys in one survey system
• “Re-use” of respondents
• Implementation of advanced estimation
techniques
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Survey Model
Optimal use of these means in integrated design for household surveys:
• Core module and topic modules
• Well-balanced use of mixed mode (CAWI - PAPI - CATI - CAPI)
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Core Questionnaire
• Household composition• Labour status• Health• Religion• Voting behaviour• Occupation in employment• Economic sector in employment• Highest level of education completed• Screening, telephone numbers, e-mail adresses
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Standard mixed mode strategy
CAWI (+ PAPI)
CATI CAPI
If tel.nr. available If not tel.nr. available
25 – 35 %
± 45% ± 35%
55 - 65% 35 - 45%
Refusal No contact
± 5%
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Mode effects
Design principle: Unimode questionnaire design
• Selection effects
• Context effects
• Satisficing
• Interviewer effects
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Maintaining output quality
• Explain discontinuities for core
variables
• Quality indicators (framework)
• Pilots and experiments
• Parallel designs
• Time series modelling
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Quality indicators
• Relevance
• Accuracy
• Timeliness
• Accessibility
• Comparability
• Coherence
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Results
2009:
• Core questionnaire in Health Survey
• LFS sample size reduction by using data from unemployment offices
• Structural time series estimators for monthly unemployment
• Small area estimators for sub-regional unemployment statistics
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Results
2010:
• Core questionnaire in LFS
• Mixed mode in LFS (CAPI – CATI)
• Mixed mode in Health Survey (CAWI – CATI – CAPI)
• Topic modules after Health Survey
2011/2012:
• CAWI in LFS
• More topic modules after core questionnaire
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Parallel designs
LFS [1] CATI
LFS [2] CATI
LFS [3] CATI
LFS [4] CATI
LFS [1] CAPI
2010
Q1 Q2 Q3 Q4
New design
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Parallel designs
LFS [3] CATI
LFS [4] CATI
LFS [5] CATI
LFS [1] CATI
LFS [2] CATI
LFS [3] CATI
LFS [4] CATI
LFS [1] CAPI
2010
Q1 Q2 Q3 Q4
LFS [2] CATINew design
Old design
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Parallel designs
LFS [1] CAPI
LFS [2] CATI
LFS [3] CATI
LFS [4] CATI
LFS [5] CATI
LFS [1] CATI
LFS [2] CATI
LFS [3] CATI
LFS [4] CATI
LFS [1] CAPI
2010
Q1 Q2 Q3 Q4
LFS [2] CATINew design
Old design, parallel run
Old design
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Issues
• Unpredictibility CAWI results and impact on fieldwork planning in a mixed mode design
• Changing geographical distribution CAPI-samples
• Mode effects (methodology & best practices)
• Survey integration
• Creating uni-mode questionnaires
• Households/LFS and CAWI
• Flexibility in operation (methodology – survey case management system)
Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands
Conclusions
• Quality can be maintained
• Efficiency targets met
• Flexible in meeting new national and
European needs
Centraal Bureau voor de Statistiek
Challenges of redesigning household surveys and maintaining output quality
Menno Cuppen May 6, 2010
Statistics Netherlands
Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands