52
24-3-2009 1
24-3-2009
1
24-3-2009
2
MESS project
Measurement and Experimentation in the Social Sciences
Marcel Das
24-3-2009
3
2005: Innovation platform Investment in Large-Scale Research Facilities
42 proposals, adding up to 1.5 billion Euros
MESS Project: one of five projects which received funding
A brief history...
24-3-2009
4
Online panel of 5000 households8000 individuals (>= 16 years)
Questionnaires each month, 30 min.
Incentive 15 euro an hour (average)
For scientific use, at no cost
Key element: LISS panel
24-3-2009
5
Why use the LISS panel?
24-3-2009
6
Disadvantages online surveyssurveys
Noncoverage:
Persons without internet are usually excluded
No probability sample:
Visitors of specific websites
Selfselection (interested, strongopinions, specific goal)
24-3-2009
7
Online interviews as method, but:
Probability sample drawn from address sampling frame of Statistics Netherlands
Includes households without internet access (less than 15%): CentERdata provides equipment
Contacted by letter, telephone or visit
LISS panel
Households without internetinternet
?
24-3-2009
9
SimPC
24-3-2009
10
Very small and silent
Only the most frequently used functions
Automatic maintenance, safety
Simple operation and readable screens
Installation and support
24-3-2009
11
24-3-2009
12
Internet interviews
Letter and brochure
Short interview
SimPCand adsl
Households with internet
Households without internet
Confirmation
Panel question
Recruitment
24-3-2009
13
Internet interviews
Letter and brochure
Short interview
SimPCand adsl
Households with internet
Households without internet
Confirmation
Panel question
Response
Primary response
target: 80%
Secondary response
target: 60%
Tertiary response
target: 53%
24-3-2009
14
Experimental design CATI F2F N
Special letter
Standard letter
Special letter
Standard letter
Prepaid incentive
Introduce panel in letter 0 euro 10 euro 200
Introduce panel after interview 20 euro 50 euro 300
Promised incentive
Introduce panel in letter 0 euro 10 euro 400
Introduce panel after interview 50 euro 20 euro 300
N 200 200 400 400 1200
Recruitment experiment
24-3-2009
15
Pilot experimental groups1. CATI: special letter - no incentive - Panel in letter and
brochure
2. CATI: special letter - promised incentive E. 50 - Panel introduced after interview
3. CATI: Standard letter - prepaid incentive E. 10 - Panel in letter and brochure
4. CATI: Standard letter - prepaid incentive E. 20 – Panel introduced after interview
5. CAPI: special letter - prepaid incentive E. 50 – Panel introduced after interview
6. CAPI: special letter - no incentive - Panel in letter and brochure
7. CAPI: Standard letter - promised incentive E. 10 - Panel in letter and brochure
8. CAPI: Standard letter - promised incentive E. 20 - Panel introduced after interview
24-3-2009
16
Pilot natural groups
CAPI
1. Unknown or no telephone number: 30%
CATI and CAPI
2. Known (landline) telephone number: 70%
24-3-2009
17
Observed response in CATI versus CAPI by subpopulation CATI
households with telnr
%
CAPI Households with telnr
%
CAPI Households
without telnr%
Primary response 65 58 44 Primary refusals 22 25 25 Primary not reached 8 11 20 Primary not usable2 3 3 5 Primary still in field 1 3 5
Effect of mode (1)
24-3-2009
18
Observed response in CATI versus CAPI by subpopulation CATI
households with telnr
%
CAPI Households with telnr
%
CAPI Households
without telnr%
Primary response 65 58 44 Secondary response conditional on primary
67 84 88
Primary * secondary 45 50 39 Tertiary response conditional on secondary
80 72 64
Primary * tertiary 35 35 25
Effect of mode (2)
24-3-2009
19
incentives increase response rates
- effect is large from 0 to 10 euro (increase of 15%)
- effect is not significant from 10 to 20, 20 to 50 euro (about 3%)
prepaid works better than promised
Effect of incentives
24-3-2009
20
Effect of information
no effect of letter content
no effect of timing panel introduction
effect of seeing/reading brochure
24-3-2009
21
Set up main recruitment
• Combination CATI – CAPI, follow up CAPI
• Prepaid 10 euro incentive
• Promised extra 10 euro incentive for starting
• Attention to design letter and brochure
24-3-2009
22
ResultsPilot versus main response rates
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
primary secondary tertiary
response
prop
ortio
n of
tota
l
Pilot
Main
24-3-2009
23
How well have we done?
Still selection bias possible due to nonresponse
24-3-2009
24
LISS panel compared to population
Group
LISS Panel%
Population%
Bias
Age >=70 7 13 -6
Living alone 15 20 -5
Non-western immigrant 4 7 -3
Did not vote at election 13 20 -7
Voted SP at election 17 17 0
Voted CDA at election 25 26 -1
Has Internet access 93 85 +8
24-3-2009
25
LISS panel compared to DPES
LISS Panel Bias
DPESBias
Group
Age >=70 -6 1
Living alone -5 -4
Non-western immigrant -3 -2
Did not vote at election -7 -13
Voted SP at election 0 1
Voted CDA at election -1 1
Has Internet access 8 2
DPES = Dutch Parliamentary Electoral Studies
24-3-2009
26
LISS panel compared to DPES
LISS Panel Bias
DPESBias
Group
Age >=70 -6 1
Living alone -5 -4
Non-western immigrant -3 -2
Did not vote at election -7 -13
Voted SP at election 0 1
Voted CDA at election -1 1
Has Internet access 8 2
DPES = Dutch Parliamentary Electoral Studies
24-3-2009
27
Use of the LISS panel
24-3-2009
28
Questionnaires
The “household box”
Core questionnaire
Rotating questionnaires open to academic researchers at no costs
24-3-2009
29
Examples
Standard question
24-3-2009
31
Examples
Visual tools
24-3-2009
33
Examples
Paradata
24-3-2009
34
24-3-2009
35
Examples
Video instruction
24-3-2009
37
Examples: subsample
Effects of illness babies in first year on labour participation parents:
Pregnant women, follow 3 years!
Adolescent/teen identity and the internet:
12-18 year olds
Random subsample or experimental groups
24-3-2009
38
Subprojects
Special groups
Biomarkers: Erasmus MC
Experience sampling (PDA)
Enrichment with administrative data (Statistics Netherlands)
24-3-2009
39
How to use the LISS panel ......
24-3-2009
40
Two ways ...
1. Send in proposal for collecting (new)
data with the LISS panel
2. Use already collected data for your
research project
24-3-2009
41
1. Data collection LISS panelpanel
Who can use the LISS panel:every researcher who wants to collectdata for scientific, policy or societalrelevant research
Scientific research: Academic researchers, irrespective of nationality, can use the LISS a panel at no cost
Proposals can be submitted throughoutth
24-3-2009
42
Procedure:(http://www.lissdata.nl/lissdata/Proposals/How_to_Submit_a_Proposal)
Submit to: [email protected]
Typically be between 2 and 5 pages
Evaluation: Board of Overseers
Decision: 1-3 monthsAcceptRevise and ResubmitReject
24-3-2009
43
Criteria:(http://www.lissdata.nl/lissdata/Proposals/How_to_Submit_a_Proposal)
Scientific potential: predicated on literature; sound analyticalframework
Fit in LISS panel: value of using an Internet survey
Feasibility:questions doable for respondents?
Burden:length and target sample
24-3-2009
44
Requirements:
Data protection statement
Single user access
Publication list
Data available to other researchers!
24-3-2009
45
Proposals: SummaryBoard decision Count
Undecided (in review or revision) 7
Accepted 25
Rejected 4
Total 36
24-3-2009
46
Proposals: ExamplesProposal Affiliation(s) Perceptions of nutrigenomics Twente University Mental health Radboud University,
Nijmegen; Emory University, Atlanta, GA
Subjective well-being Max Planck Institute for Human Development, Berlin
Computer crime victimization Leiden University Mobility in social networks University of
Amsterdam Open-ended questions University of Michigan
24-3-2009
47
2. LISS dataWho can use the LISS data:
every researcher who wants to use data for scientific, policy or societal relevant research
Use of data is free of charge (unless data are used for research funded by means of third sources)
Available data: longitudinal core studyproposed studies
24-3-2009
48
Longitudinal core studyHousehold and familyEconomic situation and housingWork and schoolingSocial integration and leisureHealthPersonalityReligion and ethnicityPolitics and values
24-3-2009
49
LISS data: Politics
• Government policy satisfaction• Political interest• Trust in institutions• External political efficacy• Internal political efficacy• Voting behaviour• Vote intention• Trust in voting methods• Civic political participation• Sympathy for political parties• Sympathy for politicians• Party membership• Left-right self-rating• Political position issues• Political goals
24-3-2009
50
LISS data: Economic SituationSituation• Subjective standard of living• Satisfaction• Employment income / Earnings• Self-employment income• Non-employment income: pension• Non-employment income: benefits• Non-employment income: other• Credit, debt, payments• Total income• Financial/material well-being• Financial expectations• Relative deprivation• Financial management• Material deprivation• Assets
24-3-2009
51
LISS data
Disseminated through website
Documentation available
Same requirements apply as with using the LISS panel to collect data
http://www.lissdata.nl
