NEWMOA Workshop on Survey Instrument Design

Linda Piccinino

The Importance of Questionnaire Design

• “Planning the questionnaire is one of the most critical stages in the survey development process.”

– American Statistical Association (1997)

Purposes of Surveys for the EPA

• Measure results of pollution prevention and compliance assistance efforts

• Show level of compliance with regulations

• Determine assistance needs

• Demonstrate results in a quantitative manner

Types of Survey Mode

• Mail (letter, email)

• Telephone

• In-person

Group Exercise 1

Learning How to Define Survey Content and Scope

– Writing survey questions to address research questions

Types of Survey Administration

• Group interviews

• Structured interviews

• Semi-structured interviews

• Self-administered questionnaires

Other Factors to Consider

• Questionnaire length

• Questionnaire format and appearance, physical size

• Question order

Question Wording and Structure

Comprehension and Clarification

• The question should define words in the way the researcher intended

• The question wording and structure should be clear about what event is being asked about

Comprehension and Clarification

Example:

– Were you robbed during the last six months?

Comprehension and Clarification

Revision:

• Was something belonging to you stolen, such as –

a) Things that you carry, like luggage, a wallet, purse, briefcase, book

b) Clothing, jewelry or calculator

c) Bicycle or sports equipment

d) Things in your home – like a TV, stereo or tools

e) Things from a vehicle, such as a package, groceries, camera

OR

f) Did anyone ATTEMPT to steal anything belonging to you?

Source: National Crime Victim Survey Questionnaire

Comprehension and Clarification

Examples:

– How many hours did you work LAST WEEK at all jobs?

– Did you lose any time or take any time off LAST WEEK for any reason such as illness, holiday or slack work?

– Did you you work any overtime or at more than one job LAST WEEK?

– Source: Schaeffer, 1993

Problems:

– Respondents include reports about usual hours worked

– Questions about hours lost and extra hours follow request for total hours

Comprehension and Clarification

Revision:

– How many hours do you USUALLY work at your (main) job?

– LAST WEEK, did you lose or take off any hours from (work/your main job) for any reason such as illness, vacation, holiday, labor dispute or layoff?

– LAST WEEK did you work any overtime or extra hours (at your main job) that you don’t usually work?

– So, for LAST WEEK, how many hours did you ACTUALLY work at your (main job/other jobs)?

Specifying the Reference Period

• The reference period should be as clear and concise as possible.

• Memory aids such as a calendar or markers such as important dates (holidays, birthdays, etc.) might help respondents to think about the reference period you are asking about.

Specifying the Reference Period

Example:

• In the last 7 days, not counting today, how many times did you smoke at work?

• Source: Belson, 1989

Possible Solution:

• I am going to ask a few questions about things you might have done while at work LAST WEEK. By last week I mean the week beginning on Sunday June 4, 2006, and ending on Saturday June 10, 2006.

– In the LAST WEEK, how many times did you take a cigarette break during work hours?

Tips

• Look at each question and ask yourself “How will this information be used?”

• Put yourself in the respondent’s place – how would you answer the question?

Stages of the Survey Response Process

The Survey Response Process

Four steps in generating a response:

1) Comprehension

2) Retrieval of information

3) Response formation

4) Response mapping and reporting

The Survey Response Process

Extent to which each stage in the response process is difficult depends on both the question and the respondent.

– How many bowls of cereal did you consume in the past week? (retrieval process)

– Have you gained 5 or more pounds in the past year? (judgment process)

Respondent Cognitive Tasks

Comprehension:

• What does the respondent think the question is asking?• What do specific words or phrases mean?

Retrieval: • What type of information does the respondent need to recall? Does

the respondent possess the information required? • What are some difficulties associated with the respondent retrieving

information? What types of strategies does he/she use (e.g., counting events by recalling each one individually or providing an estimate based on usual behavior)

Source: Willis, G. Cognitive Interviewing.

Response Formation:

•Does the respondent have the motivation to use the retrieved information to form a response?

•Does response formation pose particularly difficult tasks for the respondent?

•Does the respondent tell the truth? Does he/she falsify his/her response? Does the question raise privacy concerns?

Response Mapping:

•Can the respondent match his/her answer to the response options provided?

Respondent Cognitive Tasks

Tips

• We focused on comprehension, but respondents can have difficulties at any stage of the process

• If you know the stage that is causing difficulty, you have a better chance of fixing the problem

Response Categories

• Frequencies: Absolute and Relative

• Rating Scales

• Dates

• Open and Closed Questions

Frequencies

• How many boats do you shrink wrap a year?

• Source: New Hampshire Dept. of Environmental Services Marina Questionnaire

• How often do you dispose of unwanted mercury from your dental office?

Rating Scales

• Bipolar

• Interval

• Categorical

Dates

• Exact dates

• Ranges: Approximate dates, durations and estimates

Open and Closed Questions

• Open-ended:

– How did you learn about this program?

• Closed:

– What is your RCRA hazardous waste generator status?

• Large Quantity Generator

• Small Quantity Generator

• Conditionally Exempt Small Quantity Generator

• Don’t know

Common Problems with Survey Questions

Common Question Problems:

•Complicated questions

•Unfamiliar or technical terms

•Ambiguous concepts or words

Problems with Survey Questions

Question Problems: Complicated Questions

Example:

Over the last month, how many times per month, week, or day did you drink 100% fruit juice such as orange, apple, grape or grapefruit juice? Do not count fruit drinks like Kool-Aid, lemonade, HI-C, cranberry juice drink, Tang, and Twister. Include juice you drank at all mealtimes and between meals. (never, 1-3 times last month, 1-2 times per week, 3-4 times per week, 1 time per day, 2 times per day, 3 times per day, 4 times per day, 5 or more times per day).

Source: NCI All-Day Screener

Example: Do you think that a compliance assessment is a satisfactory tool to measure results of compliance assistance efforts?

Example: I have a sweet tooth. (strongly disagree, disagree, neutral/NA, agree, strongly agree)

Source: Eating Behavior Patterns Questionnaire (2003)

Question Problems: Unfamiliar Terms

Examples:• In the past 3 months, how often did you eat a

vegetarian dinner? (usually/always, sometimes, rarely, never)

Source: Fat and Fiber Behavior Questionnaire (1997)

• Do you now…• Spend half an hour or more in moderate or

strenuous physical activity at least 3 times a week? (Yes/No)

Source: NMES - SAQ

Question Problems: Ambiguous Concepts or Words

•Question assumptions not applicable to respondent

•Loaded questions

•Double-barreled questions

Question Problems

Examples: About how many gallons of “perc” do you use in one

year? Source: EPA Dry Cleaners Survey (2004)

Do you have a Non-Contact Cooling Water Discharge of 5,000 or more gallons per day?(yes/no/not applicable/don’t know)

Source: Garment Care Business Information (2004)

How long have you been limiting the amount of fat in your diet? (less than 30 days, 1-6 months, 7-12 months, over a year)

Source: National Food Stamp Program (1996)

Common Question Problems: Assumptions

Examples: Would you say you are now complying with EPA

standards?

Are you using environmentally preferable products?

Common Question Problems: Loaded Questions

Common Question Problems: Double-Barreled

Examples: What did you do to lose weight and maintain your

weight loss? Source: California Dietary Practices Survey (2001)

During the winterization process, did you store engines “wet” and “dry?”

•Overlapping response categories

•Recall difficulty

•Sensitive questions

Question Problems

Common Question Problems: Overlapping Response Options

Examples:How many boats do you sand per year?

Fewer than 5 boats5 to 10 boats10 to 20 boats More than 20 boats

Which of the following main dishes do you think is lowest in fat? (fish sticks, pork ribs, barbeque ribs, chicken)

Source: National Food Stamp Program Survey (1996)

Common Question Problems: Recall Difficulty

Examples:

How many newspapers did you recycle in the past month?

How many glass bottles, cans of paint, cardboard boxes or batteries did you dispose of since you started working at the hospital maintenance division?

Common Question Problems: Sensitive Questions

Examples:

If you are not using EPP (environmentally preferable) or bio-based products, why not?

Source: Connecticut Hospital Environmental Roundtable Follow-up Survey (2005)

Do you maintain your weight at recommended levels? (Y,N)

Source: Health Behaviors Module (2000)

Identify problems with the following survey question:

1) Would you describe the amount of butter or margarine you usually spread on breads and muffins as ____? (none, light, moderate, generous)

Source: Diet and Health Knowledge Survey (DHKS) 1994-1996

Practice Example

Group Exercise 2

• Critique of Existing Survey Questions

– Based on what has been learned so far, identify problems and offer solutions

– Revise questions from Exercise 1

Tips

• Look at each question – Can it be simplified?

• Reconsider words that are not recognized by your spellchecker

Pretesting

Pretesting Methods

• Methods

– Low cost options

• Consequences of not pretesting

Debriefing

• Final questioning that occurs after the respondent has completed the questionnaire and reviewed all materials

• Interviewer may ask open-ended question to probe further into certain comments or observations the respondent made during the interview

• Respondents may also be asked to provide an assessment or overall critique of the instrument

Sample Debriefing Questions

• Think about the questionnaire as a whole. Does anything in particular come to mind that you want to comment on? What are your general feelings about it? Did you feel you were prepared for the types of questions asked?

• In general, did you have difficulty remembering what you did?

Sample Debriefing Questions

• I noticed that you chuckled at several points. Can you tell me what caused you to do so?

• Were there any specific questions that you had trouble answering? Why?

• Were there any questions that you felt uncomfortable answering? Why?

• What questions do you think most needed to be revised? Do you have any suggestions for how they might be better written?

Group Exercise 3

• Pretesting of Questionnaire

• Debriefing

Group Exercise 4

• Revise and retest questions from Exercise 3

• Review of entire questionnaire development process – what are the key points and problems associated with each step?

Tips

• Always do some pretesting with some members of the target population

• Test any changes that you make

Suggested Readings

Converse, JM and S Presser. 1986. Handcrafting the Standardized Questionnaire. Thousand Oaks, CA: Sage Publications, Inc.

EPA. 2002. Guide for Measuring Compliance Assistance Outcomes. Revised. Section IV. Washington, DC: EPA, pp. 23-50.

Groves, RM, et al. 2004. Survey Methodology. Hoboken, NJ: John Wiley & Sons, Inc.

Payne, S. 1980. The Art of Asking Questions. Princeton, NJ: Princeton University Press.

Sudman, S and NM Bradburn. 1982. Asking Questions. A Practical Guide to Questionnaire Design. San Francisco, CA: Jossey-Bass Publishers, Inc.

  Willis, GB. 2004. Cognitive Interviewing. A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications, Inc.