0icfi.com |

Key words: incentives; call-backs; text-messages

Cell Phone Operational Efficiencies for a Survey of

Young Adults

May 19, 2012

Prepared by:Ashley Mark, Randal ZuWallack; ICF InternationalMichelle Bover Manderski, Daniel Gundersen, CristineDelnevo; UMDNJ School of Public Health

**This work was supported in part by a grant from the National Cancer Institute (R01CA149705, PI: Delnevo)**

1icfi.com |

Outline

Introduction to the National Young Adult Health Survey Motivation for Improvements Dropped Calls:

• A Common Sense Solution• Methods• Results

Administering Incentives:• Advantages and Disadvantages• Discovering the Problem• A Common Sense Solution• Results

2icfi.com |

The National Young Adult Health Survey

The University of Medicine and Dentistry of New Jersey (UMDNJ) contracted ICF International to conduct the NYAHS

What: National, RDD, cell phone study of young adults aged 18-34

Purpose: Improve existing data on smoking trends in young adult populations

Cost-effective: A cell phone survey of 18-34 year-olds is less expensive than a traditional landline survey as you’re better able to reach the target population (40%)

High coverage: 96% of young adults 18-29 own cell phones (Pew, 2010)

3icfi.com |

Chart or Table Title

Why Should You Care?

4icfi.com | 4

In a cell phone survey...a ―breakoff may also occur as the result of a dropped call or other technical problems and may have nothing to do with the respondent actively deciding to break off from the interview.

--AAPOR Cell Phone Task Force Report, 2010

Dropped Calls A unique obstacle for cell phone studies

5icfi.com |

Natural situation:We call the person right back.

Research environment: We give the record a hang-up disposition and put it back in the calling stack to re-attempt the number a few hours later (up to a couple of days later).

A Common Sense SolutionDropped calls - What do we normally do?

Why are we doing this?• We're using the pre-existing landline model, which doesn't

account for dropped calls.

Instead – Call back respondents when the call is dropped!

6icfi.com |

Implementation

Trained interviewers to distinguish between a hang-up (likely a refusal) and a dropped call

Programmed custom scripts when able to successfully reach a dropped-call respondent

7icfi.com |

Interviewer TrainingDropped Call v. Hang-up/Refusal

Dropped Call Hang-up – all mid-survey hang-ups should be dispositionedas a *REFUSAL*

SR is in the middle of a sentence SR gives indication that they are not interested in the survey

The respondent states that they are entering into an elevator, building, mine etc. and the call might be lost

Any verbal queues that the SR is going to hang up

Quality of call gradually diminishes

SR is engaged in the survey and the call drops suddenly

What to do with a dropped call

Type “special” Choose option 1, redialIf call is connected Choose option 1 and resume the interviewIf sent to VM Leave the appropriate VM message

If call is not connected Go back to the question where the dropped call occurred, type suspend and disposition the call as an unscheduled callback.

8icfi.com |

Custom Call-back ScriptInterviewers typed “special” which brought them to a screen with three options:

INTERVIEWER: WAS THE CALL CONNECTED?01 Yes, Person on line02 Yes, Answering machine03 NO1. “Hello, my name is (name). We were just in the middle of completing

the National Young Adult Health Survey, and it appears that the call was dropped. May we continue now?”

2. “I am calling you back to complete the National Young Adult Health Survey. Your participation is important to us. Please call us to finish the survey at 866-784-7292. Thank you.”

3. INTERVIEWER, PRESS ENTER, YOU WILL GO BACK TO THE QUESTION WHERE THE CALL DROPPED. TYPE “SUSPEND” THEN DISPOSITION THE CALL AS AN UNSCHEDULED CALLBACK.

9icfi.com |

Results

Interviewers invoked callback 187 times

Reconnected with the respondent 31.0% of the time (54 cases)• 34 completed interviews (63%)• 6 refusals (11%)• 6 callbacks, never completed (11%)• 8 not age eligible (15%)

Connected to voicemail 50.6% (88 cases)• 3 completed interviews (eventually)

Call not connected 18.4% (32 cases)

10icfi.com |

Anecdotal Results – Interviewer Feedback

At the end of fielding, interviewers were sent a number of questions to evaluate the call-back feature:

• Were there times you could have used the feature but didn’t?• How did you choose to use the feature/how did you differentiate

between a hang-up and a dropped call?• How can we make it better?• Did you find it useful and efficient?• How did respondents react when you called them back immediately?• Any other useful information or commentary on the feature?

11icfi.com |

Anecdotal Results – Interviewer Feedback

“Sometimes you could differentiate between a hang up or a dropped call by whether you actually heard a click. Then it was a hang-up. Sometimes you could tell by the respondent's attitude prior to the call dropping. That is why I liked the feature. It eliminates the need to continue calling the respondent. If they answered the callback it could be dispositioned correctly and resolved or coded out if necessary.”

“[The feature was useful] because you were able to get a complete with those respondents who were willing to do the survey and their call just happened to drop.”

“I was able to get a lot of completes this way that otherwise I wouldn't have gotten. Also, I was able to convert some of those who were not willing to do the survey before when I called them back.”

12icfi.com | 12

AAPOR guidelines recommend offering compensation/reimbursement to cell phone respondents to offset cellphone-related fees.

Administering Gift Codes

13icfi.com |

Advantages and Disadvantages to Gift Codes

Disadvantages:• Typically involves a lengthy, alphanumeric code (example: 35AZ-

YTBN2X-NBNL)• Providing them over the phone can result in transcription errors• Many respondents don’t have the ability to write down the number• Increases survey length• If respondents lost the code, they had to call us back

Advantages:• Don’t have to collect contact information from the respondent – PII• Saves time (or so we thought…)• Reduces costs (no mailings required)• Instant gratification• Easy to track

We offer respondents three options for distribution of the gift codes: 1. Over the telephone 2. Text messages 3. Respondents could call back at a later time

14icfi.com |

NYAHS survey was running long

Inserted timers onto all questions to identify certain ‘problem’ questions

Found that dispensing the gift code over the phone was taking, on average, 80 seconds!

Only 15% of respondents requested the gift code via text message

• This seemed very low given the text-messaging behaviors of this population

Discovering the Problem

15icfi.com |

Why Does It Take So Long?

You hear “m” but it’s actually an “n” –repeat that, please!

You’re sitting in a park, and a group of kids are making a lot of noise – you can’t hear the code!

You don’t have a pen or pencil to write down the code – could you memorize this? 35AW-YTBN2X-NBNL

16icfi.com |

A Common Sense Solution

Original script: Those are all the questions I have for you. In appreciation for the time you have spent answering our questions, we would like to give you a $10 Amazon.com® gift certificate. Would you like the code for the Amazon.com® gift certificate now?

• Only when respondents said “no” did we offer to text the gift code.

Revised script: Those are all the questions I have for you. In appreciation for the time you have spent answering our questions, we would like to give you a $10 Amazon.com® gift certificate. Would you like to receive the code via text message?

17icfi.com |

Results

Length of time to administer an incentive was reduced by nearly 50% - 80 seconds down to 42 seconds

Texting went from 15% to 78%

Zero complaints were lodged regarding receiving a non-working gift code when incentive codes were texted to respondents

18icfi.com |

Summary

Use common sense

Update the pre-existing landline model for cellphone data collection

Utilize low-cost solutions

19icfi.com |

Q&A

20icfi.com |

Future Uses – Call-back Feature

Took the call-back feature one step further on the Mt Sinai Adolescent Health Behavior Survey, an RDD study of youth aged 12-21

We are required to obtain consent for youth aged 12-14

We built the cell phone instrument to store new numbers • If we reached a 12 year old who was not with their parent, we were

able to store the number in the record; dial the new phone number right away, obtain consent, and call the child back

There are many ways to utilize this feature in more advanced ways

21icfi.com |