0icfi.com |
Key words: incentives; call-backs; text-messages
Cell Phone Operational Efficiencies for a Survey of
Young Adults
May 19, 2012
Prepared by:Ashley Mark, Randal ZuWallack; ICF InternationalMichelle Bover Manderski, Daniel Gundersen, CristineDelnevo; UMDNJ School of Public Health
**This work was supported in part by a grant from the National Cancer Institute (R01CA149705, PI: Delnevo)**
1icfi.com |
Outline
Introduction to the National Young Adult Health Survey Motivation for Improvements Dropped Calls:
• A Common Sense Solution• Methods• Results
Administering Incentives:• Advantages and Disadvantages• Discovering the Problem• A Common Sense Solution• Results
2icfi.com |
The National Young Adult Health Survey
The University of Medicine and Dentistry of New Jersey (UMDNJ) contracted ICF International to conduct the NYAHS
What: National, RDD, cell phone study of young adults aged 18-34
Purpose: Improve existing data on smoking trends in young adult populations
Cost-effective: A cell phone survey of 18-34 year-olds is less expensive than a traditional landline survey as you’re better able to reach the target population (40%)
High coverage: 96% of young adults 18-29 own cell phones (Pew, 2010)
3icfi.com |
Chart or Table Title
Why Should You Care?
4icfi.com | 4
In a cell phone survey...a ―breakoff may also occur as the result of a dropped call or other technical problems and may have nothing to do with the respondent actively deciding to break off from the interview.
--AAPOR Cell Phone Task Force Report, 2010
Dropped Calls A unique obstacle for cell phone studies
5icfi.com |
Natural situation:We call the person right back.
Research environment: We give the record a hang-up disposition and put it back in the calling stack to re-attempt the number a few hours later (up to a couple of days later).
A Common Sense SolutionDropped calls - What do we normally do?
Why are we doing this?• We're using the pre-existing landline model, which doesn't
account for dropped calls.
Instead – Call back respondents when the call is dropped!
6icfi.com |
Implementation
Trained interviewers to distinguish between a hang-up (likely a refusal) and a dropped call
Programmed custom scripts when able to successfully reach a dropped-call respondent
7icfi.com |
Interviewer TrainingDropped Call v. Hang-up/Refusal
Dropped Call Hang-up – all mid-survey hang-ups should be dispositionedas a *REFUSAL*
SR is in the middle of a sentence SR gives indication that they are not interested in the survey
The respondent states that they are entering into an elevator, building, mine etc. and the call might be lost
Any verbal queues that the SR is going to hang up
Quality of call gradually diminishes
SR is engaged in the survey and the call drops suddenly
What to do with a dropped call
Type “special” Choose option 1, redialIf call is connected Choose option 1 and resume the interviewIf sent to VM Leave the appropriate VM message
If call is not connected Go back to the question where the dropped call occurred, type suspend and disposition the call as an unscheduled callback.
8icfi.com |
Custom Call-back ScriptInterviewers typed “special” which brought them to a screen with three options:
INTERVIEWER: WAS THE CALL CONNECTED?01 Yes, Person on line02 Yes, Answering machine03 NO1. “Hello, my name is (name). We were just in the middle of completing
the National Young Adult Health Survey, and it appears that the call was dropped. May we continue now?”
2. “I am calling you back to complete the National Young Adult Health Survey. Your participation is important to us. Please call us to finish the survey at 866-784-7292. Thank you.”
3. INTERVIEWER, PRESS ENTER, YOU WILL GO BACK TO THE QUESTION WHERE THE CALL DROPPED. TYPE “SUSPEND” THEN DISPOSITION THE CALL AS AN UNSCHEDULED CALLBACK.
9icfi.com |
Results
Interviewers invoked callback 187 times
Reconnected with the respondent 31.0% of the time (54 cases)• 34 completed interviews (63%)• 6 refusals (11%)• 6 callbacks, never completed (11%)• 8 not age eligible (15%)
Connected to voicemail 50.6% (88 cases)• 3 completed interviews (eventually)
Call not connected 18.4% (32 cases)
10icfi.com |
Anecdotal Results – Interviewer Feedback
At the end of fielding, interviewers were sent a number of questions to evaluate the call-back feature:
• Were there times you could have used the feature but didn’t?• How did you choose to use the feature/how did you differentiate
between a hang-up and a dropped call?• How can we make it better?• Did you find it useful and efficient?• How did respondents react when you called them back immediately?• Any other useful information or commentary on the feature?
11icfi.com |
Anecdotal Results – Interviewer Feedback
“Sometimes you could differentiate between a hang up or a dropped call by whether you actually heard a click. Then it was a hang-up. Sometimes you could tell by the respondent's attitude prior to the call dropping. That is why I liked the feature. It eliminates the need to continue calling the respondent. If they answered the callback it could be dispositioned correctly and resolved or coded out if necessary.”
“[The feature was useful] because you were able to get a complete with those respondents who were willing to do the survey and their call just happened to drop.”
“I was able to get a lot of completes this way that otherwise I wouldn't have gotten. Also, I was able to convert some of those who were not willing to do the survey before when I called them back.”
12icfi.com | 12
AAPOR guidelines recommend offering compensation/reimbursement to cell phone respondents to offset cellphone-related fees.
Administering Gift Codes
13icfi.com |
Advantages and Disadvantages to Gift Codes
Disadvantages:• Typically involves a lengthy, alphanumeric code (example: 35AZ-
YTBN2X-NBNL)• Providing them over the phone can result in transcription errors• Many respondents don’t have the ability to write down the number• Increases survey length• If respondents lost the code, they had to call us back
Advantages:• Don’t have to collect contact information from the respondent – PII• Saves time (or so we thought…)• Reduces costs (no mailings required)• Instant gratification• Easy to track
We offer respondents three options for distribution of the gift codes: 1. Over the telephone 2. Text messages 3. Respondents could call back at a later time
14icfi.com |
NYAHS survey was running long
Inserted timers onto all questions to identify certain ‘problem’ questions
Found that dispensing the gift code over the phone was taking, on average, 80 seconds!
Only 15% of respondents requested the gift code via text message
• This seemed very low given the text-messaging behaviors of this population
Discovering the Problem
15icfi.com |
Why Does It Take So Long?
You hear “m” but it’s actually an “n” –repeat that, please!
You’re sitting in a park, and a group of kids are making a lot of noise – you can’t hear the code!
You don’t have a pen or pencil to write down the code – could you memorize this? 35AW-YTBN2X-NBNL
16icfi.com |
A Common Sense Solution
Original script: Those are all the questions I have for you. In appreciation for the time you have spent answering our questions, we would like to give you a $10 Amazon.com® gift certificate. Would you like the code for the Amazon.com® gift certificate now?
• Only when respondents said “no” did we offer to text the gift code.
Revised script: Those are all the questions I have for you. In appreciation for the time you have spent answering our questions, we would like to give you a $10 Amazon.com® gift certificate. Would you like to receive the code via text message?
17icfi.com |
Results
Length of time to administer an incentive was reduced by nearly 50% - 80 seconds down to 42 seconds
Texting went from 15% to 78%
Zero complaints were lodged regarding receiving a non-working gift code when incentive codes were texted to respondents
18icfi.com |
Summary
Use common sense
Update the pre-existing landline model for cellphone data collection
Utilize low-cost solutions
19icfi.com |
Q&A
20icfi.com |
Future Uses – Call-back Feature
Took the call-back feature one step further on the Mt Sinai Adolescent Health Behavior Survey, an RDD study of youth aged 12-21
We are required to obtain consent for youth aged 12-14
We built the cell phone instrument to store new numbers • If we reached a 12 year old who was not with their parent, we were
able to store the number in the record; dial the new phone number right away, obtain consent, and call the child back
There are many ways to utilize this feature in more advanced ways
21icfi.com |