Date post: | 13-Apr-2017 |
Category: |
Marketing |
Upload: | ray-poynter |
View: | 221 times |
Download: | 2 times |
Jeffrey Henning, Researchscape, April Lecture Series 2014
Improving the Representativeness
of Online Surveys
Jeffrey Henning
Researchscape International
Event Sponsors
Jeffrey Henning, Researchscape, April Lecture Series 2014
Respondent Selection Issues
Sampling Error
Coverage Error
Nonresponse Error at
Unit
Response Accuracy Issues
Nonresponse Error at Item
Measurement Error due to Respondents
Measurement Error due to Interviewers
Survey Administration Issues
Post-Survey Error
Mode Effects
Comparability Effects
Total Survey Error
Jeffrey Henning, Researchscape, April Lecture Series 2014
Niche Survey
Topline Survey
Probability Survey
Mode Online Online Telephone
Target > 5% incidence
> 20% incidence
General population
Respondents 100 400 400
Length 15 questions 25 questions
5 minutes
Cost/response $5 $5 $20
Price $495 $1,995 $7,995
Comparing Prices
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability sampling
Probability online panels
Open online panels
Weighting Quota
sampling Sample
matching
River sampling
Intercept samples
Practical ramifications
Agenda
Jeffrey Henning, Researchscape, April Lecture Series 2014
2.0% 3.3% 4.1% 4.7% 5.0% 5.3% 6.4% 6.4%
12.0%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
A B C D E F G H I
Average Absolute Errors
Source: Yeager, Krosnick, et al, 2011
Probability Non-probability
Jeffrey Henning, Researchscape, April Lecture Series 2014
6.0
3.6
2.9 2.6 2.4 2.3
1.9 1.9
1.0
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
A B C D E F G H I
Accuracy = Value
Base = worst performing survey’s average absolute error
Probability Non-probability
Jeffrey Henning, Researchscape, April Lecture Series 2014
9.6% 11.7%
13.2% 13.7% 15.3% 15.6% 16.0%
18.0%
35.5%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
A B E F H G D C I
Largest Absolute Errors
Probability Non-probability
Source: Yeager, Krosnick, et al, 2011
Jeffrey Henning, Researchscape, April Lecture Series 2014
9.6% 11.7%
13.2% 13.7% 15.3% 15.6% 16.0%
18.0%
35.5%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
A B E F H G D C I
Source: Yeager, Krosnick, et al, 2011
Probability Non-probability
Largest Absolute Errors
Jeffrey Henning, Researchscape, April Lecture Series 2014
Key Elements of
Probability Sampling
Coverage
• Known non-zero chance of selecting any member of the target population
External selection
• Random selection of members of the population to participate in the survey
Jeffrey Henning, Researchscape, April Lecture Series 2014
Robustness?
Any method with a low response rate is not a random probability sample. We can’t assume a known and non-zero chance of selection. This is true of telephone, so for most studies the gold standard is not a practical option, even if money were no object. – Ray Poynter, director, Vision Critical, 2013
What about the vast majority of research that has 90% opt-out rates? Do we decide that those people weren’t part of the population to begin with? ...I’m just having a hard time understanding the ongoing push to prove we are using probability samples. – Annie Pettit, Research Now, 2010
Jeffrey Henning, Researchscape, April Lecture Series 2014
Robustness?
Any method with a low response rate is not a random probability sample. We can’t assume a known and non-zero chance of selection. This is true of telephone, so for most studies the gold standard is not a practical option, even if money were no object. – Ray Poynter, director, Vision Critical, 2013
What about the vast majority of research that has 90% opt-out rates? Do we decide that those people weren’t part of the population to begin with? ...I’m just having a hard time understanding the ongoing push to prove we are using probability samples. – Annie Pettit, Research Now, 2010
Response rates were positively associated with demographic representativeness, but only very weakly... In general population RDD telephone surveys, lower response rates do not notably reduce the quality of survey demographic estimates. – Holbrook, Krosnick, Pfent, 2008
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability sampling
Probability online panels
Open online panels
Weighting Quota
sampling Sample
matching
River sampling
Intercept samples
Practical ramifications
Agenda
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability Online Panels
• Build a large panel using Address Based Sampling – Relentlessly invite candidates to join the panel – Provide computers or tablets and Internet
connectivity if needed
• Consistently perform as well as RDD – Transitive property of probability sampling: a random
sample of a random sample is highly accurate even though net response rates are low
– $900 per question from Knowledge Networks – Perhaps the rise of smartphones will lead to new
mobile probability panels that hit a lower price point
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability Online Panels
• Build a large panel using Address Based Sampling – Relentlessly invite candidates to join the panel – Provide computers or tablets and Internet
connectivity if needed
• Consistently perform as well as RDD – Transitive property of probability sampling: a random
sample of a random sample is highly accurate even though net response rates are low
– $900 per question from Knowledge Networks – Perhaps the rise of smartphones will lead to new
mobile probability panels that hit a lower price point
Jeffrey Henning, Researchscape, April Lecture Series 2014
Impractical for Low Incidence Mothers of children 4 and under
Families with chronically ill members
Women who do yoga workouts
Adventure racing enthusiasts
Video game players
Board and card game purchasers
Purchasers of apps for smartphones and tablets
E-book purchasers
Purchasers of self-help books
Golfers
Small-business owners
Middle managers
Jeffrey Henning, Researchscape, April Lecture Series 2014
Bye, Bye, Probability
But where randomized treatments are not possible... we must do the best we can with what is available to us. - Donald T. Campbell, social scientist, 1969
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability sampling
Probability online panels
Open online panels
Weighting Quota
sampling Sample
matching
River sampling
Intercept samples
Practical ramifications
Agenda
Jeffrey Henning, Researchscape, April Lecture Series 2014
Open Online Panels
• Anyone can join the panel
– Panelists join for cash or prizes
– Many field surveys through web intercepts, collecting responses from non-panelists
• Inconsistent results
– A random sample of a convenience sample is still a convenience sample
– Random sampling does produce greater consistency for longitudinal studies
Jeffrey Henning, Researchscape, April Lecture Series 2014
Examples of Online Panel Results
17%
69%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Panel U.S. Census
35+
18-34
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability sampling
Probability online panels
Open online panels
Weighting Quota
sampling Sample
matching
River sampling
Intercept samples
Practical ramifications
Agenda
Jeffrey Henning, Researchscape, April Lecture Series 2014
Weighting
• Post-stratification weighting viewed as a common solution to removing sampling bias from convenience samples
• Often misrepresented as a simple process of arithmetic
Jeffrey Henning, Researchscape, April Lecture Series 2014
Cell Weighting
Men Women
18 to 54
79,184,164 169 responses 469K weight
79,017,200 199 responses 397K weight
55+
36,301,576 15 responses
2,420K weight
43,154,705 17 responses
2,539K weight
Jeffrey Henning, Researchscape, April Lecture Series 2014
Rim Weighting / Raking
Age
Gender
Region
Race/ethnicity Education
level
Household income
Proprietary measure
Jeffrey Henning, Researchscape, April Lecture Series 2014
Weighting
• Implicit assumption is people we did survey in a particular demographic group are representative of the people we did not survey in that group
• Many researchers weight convenience samples...
– In the hope it does no harm
– In the belief it improves quality
– For the fact it redistributes demographics to match target population
Jeffrey Henning, Researchscape, April Lecture Series 2014
Wait, Wait
Waiting until the weighting stage to adjust is too late. The combination of coverage error and nonresponse in online panels generally creates a sample that is beyond fixing post hoc. We need to do more at the selection stage. - Reg Baker, former president and COO of Market Strategies, 2013
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability sampling
Probability online panels
Open online panels
Weighting Quota
sampling Sample
matching
River sampling
Intercept samples
Practical ramifications
Agenda
Jeffrey Henning, Researchscape, April Lecture Series 2014
Quota Sampling
Men Women
18 to 54 79,184,164 169 133 responses
595K weight
79,017,200 199 132 responses
599K weight
55+ 36,301,576 15 61 responses
595K weight
43,154,705 17 72 responses
599K weight
Jeffrey Henning, Researchscape, April Lecture Series 2014
• Divide the sample into cells and recruit to fill those cells
• Once 51% of respondents are women, stop accepting responses from women
• Each quota increases price: – $1,000 for no quota – $1,500 for 3 quota cells – $2,000 for 12 quota cells
• Bad reputation among public opinion researchers, good reputation among corporate researchers
Quota Sampling
Jeffrey Henning, Researchscape, April Lecture Series 2014
Quota Sampling
• Good reputation among corporate researchers
• Bad reputation among public opinion researchers
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability sampling
Probability online panels
Open online panels
Weighting Quota
sampling Sample
matching
River sampling
Intercept samples
Practical ramifications
Agenda
Jeffrey Henning, Researchscape, April Lecture Series 2014
Sample Matching
• Rim weighting : cell weighting = Sample matching : quota sampling
• Imagine trying to fill 400 cells:
– 57-year old African American woman with associates degree in Newton Falls, OH
– 21-year old white male high school graduate in Worcester, MA
• Where do the “cells” come from?
Jeffrey Henning, Researchscape, April Lecture Series 2014
YouGov Model of U.S. Population
• 2010 American Community Survey – Age – Gender – Race – Education – Region
• Imputation from Registration and Voting Supplements – Voter registration
• Imputation from Pew Religion in American Life Survey – Religion – Political interest – Minor party identification – Non-placement on an ideology scale
Jeffrey Henning, Researchscape, April Lecture Series 2014
Sample Matching
• Finding 57-year old African American woman with associates degree in Newton Falls, OH
• Proximity function tests all members of panel, calculating distance from target (distance in age, gender, physical location, etc.)
• Invite 59-year old African American woman with GED in Warren, OH
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability sampling
Probability online panels
Open online panels
Weighting Quota
sampling Sample
matching
River sampling
Intercept samples
Practical ramifications
Agenda
Jeffrey Henning, Researchscape, April Lecture Series 2014
Pros & Cons of Steady Panel Participation
Practice Effects (Major)
• Regularly answering surveys may improve accuracy of responses
• Panel members may become more introspective and self-aware, improving their reporting
• Respondents’ answers to attitudinal questions improve with practice
Panel Conditioning (Minor) • “Stimulus hypothesis”
that acting about future activity prompts that activity
• Past surveys makes panelists less like general population
• Panelist attrition nonrandomly affects panel representativeness
Source: Chang & Krosnick, 2008
Jeffrey Henning, Researchscape, April Lecture Series 2014
70% of NPD Panelists are
Introverts vs. 50% in U.S.
[Diligent panelists are] high on introversion, have a high need for cognition, enjoy thinking, and prefer complex to simple problems, and they like surveys – they find surveys worthwhile. - Inna Burdein, direct of panel analytics for NPD, 2013
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability sampling
Probability online panels
Open online panels
Weighting Quota
sampling Sample
matching
River sampling
Intercept samples
Practical ramifications
Agenda
Jeffrey Henning, Researchscape, April Lecture Series 2014
Key Elements of Probability Sampling?
Coverage
• Known non-zero chance of selecting any member of the target population
External selection
• Random selection of members of the population to participate in the survey
Jeffrey Henning, Researchscape, April Lecture Series 2014
Probability sampling
Probability online panels
Open online panels
Weighting Quota
sampling Sample
matching
River sampling
Intercept samples
Practical ramifications
Agenda
Jeffrey Henning, Researchscape, April Lecture Series 2014
Mimicking Probability Sampling
Coverage
• Known non-zero chance of selecting any member of the target population
External selection
• Random selection of members of the population to participate in the survey
Sample matching – Random selection of members of the population to match in the panel
Probability panel – Random selection of randomly recruited panelists
Weighting – Correcting for demographic underrepresentation
Margin of error – AAPOR is against reporting margin of error for non-probability samples
Open panel – Random selection of panelists
Jeffrey Henning, Researchscape, April Lecture Series 2014
Recommendations
• When sourcing sample, ask for steps taken to minimize sampling bias
• When evaluating panels, ask how they select respondents for a given study
• Don’t use weighting if sample was significantly demographically unbalanced
• Don’t report sampling error but do consider reporting de factor error ranges
Jeffrey Henning, Researchscape, April Lecture Series 2014
For Further Reading
Free 125-page report from the American Association for Public Opinion Research:
http://bit.ly/AAPOR2013
Jeffrey Henning, Researchscape, April Lecture Series 2014
Respondent Selection Issues
Sampling Error
Coverage Error
Nonresponse Error at
Unit
Response Accuracy Issues
Nonresponse Error at Item
Measurement Error due to Respondents
Measurement Error due to Interviewers
Survey Administration Issues
Post-Survey Error
Mode Effects
Comparability Effects
Total Survey Error
The Sponsors for this Event
If you are interested in sponsoring a future NewMR event Email [email protected]
Event Sponsors
Jeffrey Henning, Researchscape, April Lecture Series 2014
Jeffrey Henning, PRC
Researchscape International
Up-to-date Research on the Changing Consumer
Toll Free: +1 (888) 983-1675 x 701
http://www.researchscape.com/
http://www.twitter.com/jhenning