31
A Sampler of Methodological Issues: Spanish Language, Cell Phones, and Address Based Sampling Sarah Cho PAPOR Mini-Conference June 24, 2011
A Sampler of Methodological Issues: Spanish Language, Cell Phones, and
Address Based Sampling
Sarah ChoPAPOR Mini-Conference
June 24, 2011
THE BENEFITS & CHALLENGES OF ADDRESS-BASED SAMPLING
DESIGNS
David Dutwin, PhDVice PresidentSocial Science Research Solutions
Michael W. Link, PhDChief MethodologistThe Nielsen Company
66th Annual Conference of the American Association for Public Opinion ResearchPhoenix, AZMay 2011
“Address-Based Sampling” is the sampling of addresses from a database with near universal coverage of residential homes
ABS helps solve coverage issues, but not response rate
While mail surveys have been around for decades, use of a residential mailing frame for sampling the general population is relatively new◦ Before 2000: U.S. Census begins developing a Master Address File for the 2000 Census
from U.S. Post Office◦ 2003: First published evaluation of use of mailing addresses for in-person survey◦ 2008: First published use of the term “Address-Based Sampling”
ABS is now the basis or key component of many critical studies (National Election Survey, Knowledge Networks Panels, Nielsen Television Audience Measurements, some State/City-Level Health Interview Studies)
ABS is a sampling methodology, NOT a single survey design and can be used as a foundation for multimode data collection designs
Address-Based Sampling
63% of the public has at least one listed LL telephone number◦ 10% are zero listed landline banks and typically
excluded from most RDD samples, 25% are CPO, and 2% have no telephone
◦ LL only RDD sample systematically excludes 37% of the public
With ABS, coverage is 98% if you include all addresses (including PO Boxes)◦ Note: there are regional variations in coverage
ABS vs. RDD
5
Primary source: USPS Address Management System◦ Computerized Delivery Sequence File (CDSF)
Only licensed vendors can provide addresses from this product
Weekly or monthly updates from USPS available◦ Delivery Sequence File Second Generation (DSF2)
Not a product, but a process Can only be applied to a mailer’s existing address list
to clean the list of erroneous addresses and cannot be used to generate an address list
Sample Frame
6
House number Apt number Street name Street Suffix (ex. Ave, Blvd) Directional (ex. NE, W) Zip Zip+4 City name City code State code State Name Tract
CDSF Frame Elements (Raw File)
Block County Walk sequence number
(Postal route) Route type Delivery type code (ex.
door slot, curb) Vacant code Seasonal code Drop count (# people) PO Box
7
A key benefit of having address as the base sample unit is the extensive amount of information that can be appended to the sample file◦ Geographic data◦ Demographic data◦ Names and telephone numbers
Percent name append on average is over 90% Percent phone append on average is about 65%
All varies by geography and the company you use to append the CDSF file
Appending Data to CDSF
8
Two key decisions after sampling◦ Mode of contact for recruitment (mail, in-person,
telephone) ◦ Mode of interviewing (in-person, mail, telephone,
web, dairies)◦ A combination of multiple modes of contact or
interviewing can be utilized
After Appending Data to the Sample
9
ABS can achieve near-universal coverage
Need to work with a licensed vendor
New terminology associated with ABS
Information can be appended to the sample file to help facilitative sampling stratification, specialized data collection, backend analyses
Can support an array of different survey designs
ABS is still in its infancy, but growing number of studies adopting and testing this approach will allow industry to rapidly optimize use of ABS
Summary
10
Check out the first AAPOR webinar on June 30th, 10am-11:30am
◦ Instructor: David Dutwin◦ Register on the AAPOR website to view the
webinar live or to view a recorded version at your own pace
◦ Discounts available for students
For more information…
Factors Contributing to Differences in Reported Party Identification Across Polls in 2010
Liz Hamel, Bianca DiJulio, Sarah Cho,Claudia Deane, Mollyann Brodie
Kaiser Family FoundationMenlo Park, CA
www.kff.org
Larry Hugick, Jonathan Best
Princeton Survey Research Assoc. Int’l.Princeton, NJ
www.psrai.com
• Prior to 2007, Kaiser Health Tracking polls generally reported a party ID distribution quite similar to that of other publicly released polls.
• In 2009, we began to notice that the party ID distribution in Kaiser tracking surveys was leaning more Democratic compared with most other publicly released polls.
The Issue
More Details on The Issue: 2010
Poll/Organization # polls Dem (%) Rep (%) Dem-Rep (% pts.)
Kaiser Health Tracking Poll 12 35.8 22.8 13.0
Newsweek 5 35.2 26.4 8.8Ipsos (w/Reuters and/or McClatchy) 8 30.9 22.3 8.6Washington Post/ABC News 10 32.4 24.2 8.2NBC News/Wall Street Journal 11 31.9 23.9 8.0Pew Research Center (political surveys) 12 32.7 25.2 7.5Associated Press/GfK 11 32.0 24.6 7.4CNN 16 33.5 26.1 7.4PSRAI weekly omnibus (May-Dec.) 32 31.6 24.5 7.1CBS News and CBS/New York Times 13 33.2 26.7 6.5Gallup (Daily Tracking avg. by month) n/a 31.9 27.6 4.3Gallup Social Series + USA Today/Gallup 22 31.2 28.5 2.7 Avg. of all organizations excluding KFF 32.4 25.4 7.0Avg. of all excluding KFF and Gallup 32.6 24.9 7.7
Difference between KFF and avg. +3.4 -2.6 +6.0Difference between KFF and avg. excluding Gallup +3.2 -2.0 +5.3
HYPOTHESIS 1: Offering respondents the opportunity to be interviewed in Spanish
leads to a higher Dem-Rep gap compared to surveys conducted in English only.
Hypothesis 1
Party ID among Hispanics in 6 Kaiser tracking polls, June-Dec. 2010
Hypothesis 1 (Spanish interviewing): Internal analysis of KFF data
48%
43%
7%
18%
45%
39%
RepublicanDemocrat Independent/other/DK
Hispanics interviewed in
Spanish
Hispanics interviewed in
English
Dem-Rep = 41
Dem-Rep = 25
Hypothesis 1 (Spanish interviewing): Internal analysis of KFF data
6-month average:
Original data
6-month average: Data weighted without Spanish interviews
(replicating English-only design)Diff.
(% pts.)% Democrat 36.6 36.2 -0.4% Republican 23.6 24.5 +0.9% Independent 30.8 31.0 +0.2% Other 4.7 4.4 -0.3% DK/Refused 4.6 3.9 -0.7Dem-Rep Gap (% pts.) 13.0 11.7 -1.3
Average party ID in Kaiser tracking polls July-Dec. 2010:Original weighting versus English-only weighting
Poll/Organization Dem-Rep(% pts.) Spanish? % interviews
Spanish (unwtd.)
Kaiser Health Tracking Poll 13.0 Yes 5.6
Newsweek 8.8 No
Ipsos (w/Reuters and/or McClatchy) 8.6 Yes ?Washington Post/ABC News 8.2 No
NBC News/Wall Street Journal 8.0 No
Pew Research Center (political surveys) 7.5 Some 2.7Associated Press/GfK 7.4 Yes 0.9CNN 7.4 No
PSRAI weekly omnibus (May-Dec.) 7.1 No
CBS News and CBS/New York Times 6.5 No
Hypothesis 1 (Spanish interviewing): Does # of Spanish interviews matter?
HYPOTHESIS 2: Using an “overlapping” dual frame design leads to a higher Dem-Rep gap
compared with surveys that use a “segmented” dual frame design or a
landline-only design.
Hypothesis 2
Hypothesis 2 (Cell phone frame): Internal analysis of KFF data
Average party ID in Kaiser tracking polls July-Dec. 2010:Original weighting vs. “segmented” dual frame weighting
Original data
Data weighted without dual users reached on cell (replicating segmented dual frame design)
6-month average
6-monthaverage
Difference fromoriginal (% pts.)
% Dem 36.6 35.2 -1.4
% Rep 23.6 23.4 -0.2
% Ind 30.8 31.6 +0.8
% Other 4.7 5.2 +0.5
% DK/Ref. 4.6 4.7 +0.1
Dem-Rep gap (% pts.) 13.0 11.8 -1.2
Hypothesis 2 (Cell phone frame): Comparison to other orgs. inconclusive
Dem (%)
Rep (%)
Dem-Rep gap(% pts.)
Kaiser tracking poll(overlapping dual frame design) 35.8 22.8 13.0
Others using overlapping dual frame designs(Pew, Newsweek, AP/GfK, CBS/NYT, PSRAI) 32.9 25.5 7.4
Organizations using segmented dual frame designs(Post/ABC and NBC/WSJ) 32.2 24.1 8.1
Organizations using landline only samples(CNN, Ipsos) 32.2 24.2 8.0
2010 Party ID averages across survey organizations, grouped by type of telephone frame
HYPOTHESIS 3: Differences in question wording and rotation of the terms “Democrat” and “Republican” can explain the different results between organizations. MAYBE
HYPOTHESIS 4: The topic of the questions asked in Kaiser surveys (health care and health policy) leads to a higher Dem-Rep gap compared to other polls. NO
HYPOTHESIS 5: Something about the way Kaiser survey data is weighted to national parameters is contributing to the difference. NO
HYPOTHESIS 6: “Field house effects” are contributing to the difference. UNKNOWN
Other Hypotheses Tested
• Different factors can work together in the same direction and lead to measurable differences in survey outcome measures
• Cell phone frame: Evidence suggests type of cell phone frame can have an impact. Important for researchers to continue investigating as survey industry tries to settle on “best practices” for including cell phones in GP samples
• Spanish language interviewing: Researchers should carefully consider question of whether to offer Spanish, and also questions of how many Spanish interviews, translation methods, etc.
Implications
Hispanic Attitudes Toward Immigration and the Language of the Interview
Kate Kenski, Ph.Dand
Marisa EnriquezDepartment of Communication
The University of Arizona
Why study Hispanics?• Latinos projected to be 24.4% of population
by 2050 (US Census, 2006)• Growth rate– From 2000-2010 Latino population grew 43%
(US Census, 2010)
• Language– 6% Hispanic voters get news only in Spanish– 40% Hispanic voters get news in both English
and Spanish (Suro, 2004) Kenski & Enriquez (2011) AAPOR
Data Source and Research Question
• 2008 National Annenberg Election Survey– RDD telephone survey– AAPOR RR1: 19%
• Examined differences between Hispanic respondents who completed the interview in Spanish versus those who completed the interview in English on three topics related to immigration– Path to citizenship– Border fence– Driver’s license
Kenski & Enriquez (2011) AAPOR
Variables
• Independent Variables• Language of Interview, demographic characteristics,
citizenship, political ideology, party id• Dependent Variables– 5-point scale: 1=strongly oppose to 5=strongly favor
• Provide a path to citizenship for some illegal aliens who agree to return to their home country for a period of time and pay substantial fines • Increase border security by building a fence along part
of the U.S. border with Mexico.• Driver’s licenses to undocumented or illegal immigrants
Kenski & Enriquez (2011) AAPOR
Hypotheses, part 1COMPARED TO HISPANICS INTERVIEWED IN SPANISH, THOSE
INTERVIEWED IN ENGLISH ARE - • H1a: more educated• H1b: have higher household incomes• H1c: more likely to be born in the U.S.• H1d: less likely to attend religious services• H1e: more likely to express a political party affiliation
Kenski & Enriquez (2011) AAPOR
Hypotheses, part 2COMPARED TO HISPANICS INTERVIEWED IN SPANISH, THOSE
INTERVIEWED IN ENGLISH ARE - • H2a: less likely to support a path to citizenship• H2b: more likely to support a border fence• H2c: less likely to support allowing illegal immigrants to have a
driver’s license
Kenski & Enriquez (2011) AAPOR
Results• H1a (education): Supported, bivariate and multivariate• H1b (income): Supported, bivariate and multivariate• H1c (U.S born): Supported, bivariate and multivariate• H1d (religious attendance): Supported, bivariate• H1e (party affiliation): Supported, bivariate
• H2a (path to citizenship): Not supported, significant in opposite direction from prediction
• H2b (border fence): Supported, bivariate and multivariate• H2c (driver’s license): Supported, bivariate and multivariate
Kenski & Enriquez (2011) AAPOR
Conclusion• Should not assume English speaking
Hispanics represent entire group
• No opportunity for Hispanics to answer in Spanish may systematically bias survey estimates
Kenski & Enriquez (2011) AAPOR
