Survey and Questionnaire Design

MAR 6648: Marketing ResearchJanuary 20, 2010

Overview

• How do you write survey questions?– What are some common traps survey writers fall

into?– How do we avoid them?

• What about special circumstances?– Embarrassing questions, etc.?

Survey Research

• Descriptive research• Survey modes and methods:– Telephone surveys– Mail surveys– Face-to-face surveys• At home• Mall intercepts

– Web-based surveys

Advantages of Surveys

• Can collect a great deal of data, including– Attitudes, interests, opinions, behaviors– Classification variables

• Can be employed in virtually any setting• Can be inexpensive, depending on format

Disadvantages of Surveys

• Effective implementation requires considerable judgment

• Social interaction of interview may lead to errors

• Subject to demand and honesty concerns

First Big Point: Biases

• Basic Tenet: Ask a question, and you’ll get a response!

• Translation: Garbage in, garbage out!

Typical problems in wording questions

How questions create answers

How questions create answers

How questions create answers• A survey for Black Flag said:

– “A roach disk… poisons a roach slowly. The dying roach returns to the nest and after it dies is eaten by other roaches. In turn, these roaches become poisoned and die. How effective do you think this type of product would be in killing roaches?”

– Not surprisingly, 79% said effective!• A Gallup poll sponsored by the disposable-diaper industry

asked:– “It is estimated that disposable diapers account for less than 2% of the

trash in today’s landfills. In contrast, beverage containers, third-class mail and yard waste to account for about 21% of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers?”

– 84% said no!

How questions shape answers

• Some examples– Behavioral frequency questions• “How often do you…?”

Behavioral Frequency Questions

• How many hours of TV do you watch on an average week day? Please choose the category from the list below that best describes your behavior.

Marketers Question: What % of class watches more than 2.5 hrs of TV?

Behavioral Frequency Questions• Scale represents

distribution in ‘real’ world– Values in middle reflect

“average” / “usual” behaviors

– Values at an end (or ends) reflect extreme behavior

• Low Frequency Scale– Lower estimates of

“average”– Avoid marking extreme high

box• High Frequency Scale

– Makes marking “>2.5 hours” more acceptable

• “Response Scale Effect”

How questions shape answers

• How much money do you have in savings?– 1-7; 1 = $0, 7 = $500 and

over– 1-7; 1 = $500 and under,

7 = $500,000 and over

• How wealthy are you?

How questions shape answers

• Some examples– Behavioral frequency questions– Aided versus unaided questions

Aided and Unaided Questions

• What do you consider to be the most important thing for children to prepare them for life?– 1. _______________– 2. _______________– 3. _______________– 4. _______________

• What do you consider to be the most important thing for children to prepare them for life?– __Good education– __Inheritance– __Moral values– __To think for themselves– __Good friendships– __Others (please specify)

How questions shape answers

• Some examples– Behavioral frequency questions– Aided versus unaided questions– Framing of questions

Framing Effects

• Question forms that lead respondents to give a certain response– Example: Allow versus forbid

Q: Do you think the U.S. should forbid public speeches against democracy?

Q: Do you think the U.S. should allow public speeches against democracy?

Yes No

5% 95%

81% 19%

Framing Effects

• Question form• Choice of example/source

Framing Effects

• Question form• Choice of example/source– Surveyor power: Mispredicting political outcomes

(Bischoping & Schuman 1992)– Perhaps partly motivational?• People who feel good about themselves show less of

this bias (Cohen 2003)

How questions shape answers

• Some examples– Behavioral frequency questions– Aided versus unaided questions– Framing of questions– Scales associated with questions• Odd versus even numbered scales• Semantic versus Stapel scales

How questions shape answers

• Some examples– Behavioral frequency questions– Aided versus unaided questions– Framing of questions– Scales associated with questions– Sequence of questions

Sequence of Questions

• Specific – General Questions– Happiness and dating (How many dates have you had

recently?)• Order 1 (Happiness then dates): Correlation = 0.03• Order 2 (Dates then happiness): Correlation = 0.67

Q1: General Colin Powell, the hero of the Gulf War, recently decided to become a member of a political party. Do you happen to know which party that is?

Q2: Now rate this party.

Q1: General Colin Powell, the hero of the Gulf War, recently decided to become a member of a political party, but decided not to run for the presidential elections. Do you happen to know which party that is?

Q2: Now rate this party.

Sequence of Questions

• Difficultly in answering one question can influence subsequent answers

• List three (twelve) ways in which you are happy (unhappy) with your partner.

• How happy are you with your partner?

Sensitive Questions

• Respondents may not be willing to answer some questions, due to embarrassment, legal issues, etc.

• This may result in – Item nonresponse– Termination– Distortion

Asking the Embarrassing Question• The casual approach

– “Do you happen to have killed your wife?”• The numbered card approach

– “Would you please read off the number on this card which corresponds to what became of your wife?:• 1: Natural Death• 2: I killed her• 3: Other

• The everybody approach (counter biasing method)– “As you know, many people have been killing their wives these

days. Do you happen to have killed yours?”• The other people approach

– “Do you know any people who have murdered their wives? How about yourself?”

Barton, POQ, 1958

How to ask a sensitive question

• Personal information – Build rapport, legitimize yourself

• Embarrassing info – Downward bias– Counterbiasing statement• “Recent studies indicate that men often use their

wives’ hair spray. Do you use your wife’s hair spray?”

– Projective techniques

Projective Techniques

Key Points

• When wording questions…– Use simple words– Use clear words– Avoid leading questions– Avoid implicit alternatives– Avoid implicit assumptions– Avoid double-barreled questions– Consider frame of reference

Key Points

• Don’t forget about question sequence– It can influence the nature of the respondent’s

answers and can be the cause of serious error in the results• Use a simple and interesting opening question• Ask general questions first• Place uninteresting and difficult questions late in the

sequence• Arrange questions in logical order

The Basic Steps of Questionnaire Design

• Decide on question content• Decide how to ask each question– Response format– Question wording

• Decide on question sequence– Funnel approach (general to specific)

• Decide on physical characteristics• Carry out pretest, revise, and final draft

So…

• What is a good survey question?• How are we to interpret survey question

output?

A Managerial Problem

• Temptation: – Report those numbers to

your manager as is

• Problem:– Those numbers have no

direct value• What does “somewhat

likely” mean in real life?

In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely

is it that you will buy a Verizon iPhone?

In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely

is it that you will buy a Verizon iPhone?Very

UnlikelySomewhat

UnlikelySomewhat

LikelyVeryLikely

15% 35% 35% 15%

In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely

is it that you will buy a Verizon iPhone?

In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely

is it that you will buy a Verizon iPhone?Very

UnlikelySomewhat

UnlikelySomewhat

LikelyVeryLikely

15% 35% 35% 15%

A Managerial Problem

• Solutions:– 1. Benchmark

• What do past surveys say?

AT&T iPhone 12% 28% 30% 30% 9%

VZ Droid Incredible 20% 25% 30% 25% 5%

Actual Purchase Rate

A Managerial Problem

• Solutions:– 1. Benchmark

• What do past surveys say?

– 2. Focus on questions you can answer• Relative responses on the

same items

In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely

is it that you will buy a Verizon iPhone?

In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely

is it that you will buy a Verizon iPhone?Very

UnlikelySomewhat

UnlikelySomewhat

LikelyVeryLikely

15% 35% 35% 15%

Verizon Customers 35% 35% 20% 10%

AT&T customers 15% 25% 30% 30%

A Managerial Problem

• Solutions:– 1. Benchmark

• What do past surveys say?

– 2. Focus on questions you can answer• Relative responses on the

same items

– 3. Ask other questions/use other methods

In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely

is it that you will buy a Verizon iPhone?

In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely

is it that you will buy a Verizon iPhone?Very

UnlikelySomewhat

UnlikelySomewhat

LikelyVeryLikely

15% 35% 35% 15%

Summary

• Survey design is harder than it looks– Be sure to think about what the question suggests

about the answer• Order and scale points matter, too

– Be sure to keep respondents’ comfort in mind