© 2015 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary informationand may not be disclosed or reproduced without the prior written consent of Ipsos.
May 17th, 2015
AAPOR 2015 Mini-ConferenceReassessing Today's Survey Methods:Voters, Election and Non-Probability Samples
Meta-Analysis of Online Paneland Non-Panel Sampling:Electoral and Non-ElectoralBehavior Metrics
2
The survey research industry is in an era of transition and flux: ‘traditional’ methodologies (now defined aslive interviewer landline phone and cellular phone) are waning in prevalence – if not importance – whilenewer methodologies (online, IVR, blended) are gaining prominence. The impact of this on measurementaccuracy is still being explored, with new information emerging all the time.
To further this body of knowledge specifically within online work, we analyze behavior measurementaccuracy of the two main sampling approaches for internet-based research: panel only and non-panel(River and other approaches) sampling. Our paper will comprise a meta-analysis of nonprobablity onlinesampling, looking across over 400 data points, at election and non-election behaviors as validating points.Behavior measurement accuracy will be assessed against external benchmarks from market-accepted datasources such as the US Census, Current Population Survey (CPS), and the General Social Survey (GSS).
The analysis will focus specifically on differences in accuracy between sample sources or blendedcompositions, as well as which socio-demographic subgroups are most accurate across a range of differentvalidation measures.
Abstract
3
Dr Clifford YoungPresident
US Public Affairs
Dr Robert PetrinVice President
US Public Affairs
Julia ClarkSenior Vice President
US Public Affairs
4
5
Our world is changing…has changed
66
7
Our client wanted their data…
Interactive (visuals & video)Faster (same-day & daily)Granular (huuuuge samples)
We gave them a RIVER of data...
11,000 interviews/month (c. 350/day), 365 days/yearQuestionnaire changes dailyAssessing same-day events (parsing)
Responding to Client Needs
8
So – we innovated
IPSOS BLENDED SAMPLING(aka: panel + non-panel)
9
Types of Nonprobability Sample
1. ‘Known’ panels from which we draw targeted sample– Our own panel (iSay)– Other panels
2. ‘Ad hoc’ panelists REALLOCATED from other surveys
3. Non-panel aka River– ALL non-panel sample, including social media,
communities, gaming, reward & loyalty programs, and adnetworks
All goes into the ROUTER
10
Think of the router as a very clever funnel
Your Survey
Respondent
Respondent
Respondent
RespondentRespondentRespondent
OtherSurveys
OtherSurveys
QualifyDoes NotQualifyR
EALLOCATION
11
This approach requires a HIGH volume of survey work
o c. 50 river suppliers from over 200 sites
o Screen c. 130,000 respondents /day
o Yields c. 4 million respondents each month
65% -75% of these are unique respondents
o Over 48,000,000 respondents a year
12
We know it works(and, for many of our clients, ‘knowing it works’ is good enough…)
13
• Measuring incidence of people who received flu shot in January 2013• 40% of Americans stated that they already got a flu shot this season;
CDC figure 41.5%
Flu Shot Study (also: 2012 and 2014 elections)
14
Research Question:What are the differences from anaccuracy standpoint between thesenonprobability sample types?
15
Aggregate data from May 1st 2014 through May 1st
2015
Focus on 37 questions with 102 variables 612cases for which there is a robust market comparator(Census, CPS, other government statistics, largepublished probability-based survey, etc.)Looked at Average Absolute Difference
Methodology
16
133,623 interviews total…
Ipsos Panel(iSay), 25%
ReallocatedPanel, 25%
River, 41%
OtherPanels,
9%
Ipsos Panel (iSay)Reallocated PanelRiverOther Panels
17
133,623 interviews total…
Ipsos Panel(iSay), 25%
ReallocatedPanel, 25%
River, 41%
OtherPanels,
9%
Ipsos Panel (iSay)Reallocated PanelRiverOther Panels
18
FINDINGS
19
First, some important dynamics re. online generally, and ‘market comparisons’
QUESTION RESPONSES WEIGHTED UNWEIGHTED MARKET TOTAL DIFF ISAY DIFF AMPARIO DIFF
TotalWeighted
WGT iSayallocated
WGT Ampario(re-allocated)
TotalUnweighted
UNWGTiSay
allocated
UNWGTAmpario (re-
allocated) MARKET WGT UNWGT WGT UNWGT WGT UNWGTWhich of the options below bestdescribes how your think of yourself?
Heterosexual/straight
91% 92% 91% 92% 93% 92% 96.6% 6.1% 4.2% 5.0% 3.7% 6.1% 4.3%Gay/lesbian 3% 4% 3% 3% 3% 2% 1.6% 1.7% 1.1% 1.9% 1.4% 1.2% 0.8%Bisexual 4% 3% 4% 3% 3% 3% 0.7% 3.0% 2.3% 2.4% 2.0% 3.1% 2.3%Other 3% 2% 3% 2% 1% 2%Total 133623 33320 54935 133623 33320 54935
How often do you attend religiousservices?
Seldom/never 41% 41% 41% 39% 39% 39% 29.0% 12.3% 10.4% 11.6% 9.9% 12.2% 10.4%Monthly or yearly 34% 34% 33% 35% 35% 34% 33.0% 0.9% 1.7% 1.5% 1.9% 0.0% 0.9%Weekly or more 19% 20% 18% 20% 22% 20% 37.0% 18.3% 16.5% 16.8% 15.1% 18.6% 17.0%Don’t know/noanswer
6% 5% 8% 5% 4% 7% 1.0% 5.2% 4.4% 3.8% 3.2% 6.5% 5.6%Total 133623 33320 54935 133623 33320 54935
Would you describe yourself as aborn-again or evangelical Christian?
Yes 31% 30% 31% 29% 28% 30% 35.0% 4.3% 5.6% 4.7% 6.7% 4.1% 4.8%No 69% 70% 69% 71% 72% 70% 65.0% 4.3% 5.6% 4.7% 6.7% 4.1% 4.8%Total 78051 16933 32384 78051 16933 32384
TM491Y15_TOT - Do you currentlyhave health insurance from a stategovernment health coverageexchange or throughHealthcare.gov? (BASED TO TOTALRESPONDENTS, ASKED OF OUTOF POCKET OR MEDICAID ONLY)
Yes 10% 9% 11% 9% 9% 9% 8.8% 1.4% 0.1% 0.7% 0.2% 1.9% 0.3%No+Not Sure+notapplicable
90% 91% 89% 91% 91% 91% 91.2% 1.4% 0.1% 0.7% 0.2% 1.9% 0.3%Total 23267 4951 9924 23267 4951 9924
20
FINDING 1:Panelists are more ‘engaged’ (RVs; Twitter users)
QUESTION RESPONSES WEIGHTED
MARKET
TOTALDIFF ISAY DIFF RIVER DIFF
Twitter - Howfrequently, if at all,do you use oraccess the followingsites?
TOTAL PANEL RIVER TOTAL PANEL RIVERUse TOTAL 40% 44% 38% 23.0% 16.9% 20.6% 14.6%
Do not use 60% 56% 62% 77.0% 16.9% 20.6% 14.6%
Are you currentlyregistered to vote,or not?
Yes 79% 82% 78% 71.2% 7.5% 10.6% 6.5%
No 18% 16% 18% 28.8% 10.7% 12.7% 10.4%
21
FINDING 2:River sample more likely to say ‘Don’t know’
QUESTION RESPONSES WEIGHTED
PANEL RIVERAPPROVAL2 -Summary
TOTAL APPROVE 40% 38%TOTALDISAPPROVE
54% 54%
Not sure 6% 8%Total 33320 54935
Are you currentlyregistered to vote, ornot?
Yes 82% 78%No 16% 18%Don’t know /Refused
2% 4%
Total 33320 54935How often do youattend religiousservices?
Seldom/never 41% 41%Monthly or yearly 34% 33%Weekly or more 20% 18%Don’t know/noanswer
5% 8%
Total 33320 54935
QUESTION RESPONSES WEIGHTED
PANEL RIVERFor whom did youvote for President in2012?
Barack Obama(Democrat)
51% 52%
Mitt Romney(Republican)
40% 38%
Other 6% 6%Unsure / Refused 3% 5%Total 26857 41117
Generally speaking,would you say thingsin this country areheading in the rightdirection, or are theyoff on the wrongtrack?
Right direction 25% 23%Wrong track 61% 60%Don’t know 14% 16%Total 33320 54935
22
FINDING 3:Panel and Non-Panel also showed strong similarities for
UNWEIGHTED demographics
DEMO UNWEIGHTED
TOTAL PANEL RIVER
DEMOG AVERAGE 6.8% 7.4% 6.8%
TOTAL AVERAGE 5.7% 6.2% 5.4%
23
FINDING 4:Panel and Non-Panel were surprisingly similar, with the biggest
gap on attitudinal variables(e.g. on marijuana legalization; ACA)
NON DEMO WEIGHTED
TOTAL PANEL RIVER
SOCIAL AVERAGE 5.2% 5.4% 4.7%
POLITICAL AVERAGE 4.6% 5.3% 4.5%
ATTITUD AVERAGE 4.3% 5.2% 3.8%
TOTAL AVERAGE 4.2% 5.1% 4.1%