Questionnaire Design & Survey Methodology for Medical Education Research Proposals

ADE Medical Education Research

Faculty Development Program

February 14, 2006

Faculty Introductions

Wendy Cohn, Ph.D. Jim Martindale, Ph.D. Natalie May, Ph. D. Lisa Rollins, Ph.D.

Goals of Session I

How to write a questionnaire Best ways to administer a survey Measurement and Analysis Q & A Prepare for Session II

What You Must Consider

What is the question? Has the question been addressed previously? If so,

how? What is the appropriate methodology? Whom do you need to sample and how? What do you need to ask? How should you ask it? Is there an existing survey/validation? How should the survey data be analyzed? How should the results be reported?

Finding Your Question

Begin with a purpose! For example: “The purpose of this study is to

test (the theory) by relating (the independent variable) to (the dependent variable) for (participant group) at (the research site).

Finding Your Question (cont’d) Pose a question Begin with “how,” “what,” or “why” Specify your variables Use words to describe the connection

between/among variables (describe, compare, relate, etc.)

Indicate the participants and setting

Conceptual Survey Outline

A conceptual map of your questionnaire. Helps guide the development of your

questionnaire so that you include relevant questions and exclude irrelevant questions.

Key elements: Concept; term; operational definition;

measurement; priority

Types of Questions

Dichotomous Questions Nominal Questions Ordinal Questions Interval/Ratio Questions Filter or Contingency Questions

Question Formats

Structured Fill-in the blank Rating Likert Scale Check all that apply

Unstructured Open ended question

I oppose electronic prescribing and electronic medical records for

patients.

SA A N D SD

Which of these five statements best describes your chairman?

Innovative but lacking in leadership qualities

About the same on innovation and leadership qualities

Stronger on leadership than innovation

A born leader

A real innovator

How do you feel about healthcare in the United

States?

About how many books have you read for leisure during the past year?

___________ Number of books

Don’t you agree that the new resident work hour regulations are

limiting to your education?

SA A N D SD

How supportive is your spouse about your efforts to quit smoking?

Very Supportive Somewhat Not at allsupportive supportive supportive

If you fixed dinner at home last night, did you eat meat as part of that meal?

Yes

No

Please identify your insurance carrier:

Aetna Blue Cross Health South Self Pay

Do you incorporate obstetrics/ gynecology into your practice?

SA A N D SD

Do you favor or oppose not allowing the state to raise taxes without approval of 60% of the voters?

Favor

Oppose

Summary of Typical Problems Double-barreled questions Vague questions Leading questions Premature assumptions/bias Insufficient alternatives with forced-choice options Answer options don’t fit with the question Double negatives

Issues to Consider Regarding Content Is the question necessary/useful? Are several questions needed? Do respondents have the needed info? Does the question need to be more specific? Is the question biased or loaded? Is the question asking about sensitive

information?

Layout Issues

Self-explanatory Visually clear and uncluttered Group tasks/types of questions Consistent and clear response options

Layout Issues (cont’d)

Consider question placement Skip patterns kept to a minimum (if needed,

use arrows and boxes) Beware of using double-sided pages There is elegance in simplicity!

Tips to Reduce Non-Response Professional, personalized, attractive, easy to

complete Tasks/directions should be clear Easy to read, uncluttered, visually appealing Response task should be easy Length of form

Survey Implementation

A well designed questionnaire is critical…however: Implementation procedures have a greater

influence on response rates.

Experimental research has identified the factors that influence survey response.

Choices for implementation

Telephone Costly; need expertise

Web Efficient; cheap; limited range of question types;

hard to apply full range of implementation procedures

Mail Very well studied; moderately costly; can do well

Multi-method Can be done but complicated; get help

Implementation procedures that affect response Multiple contacts Contents of letters Appearance of envelopes Incentives Personalization Sponsorship & its explanation

The five most important elements for achieving high response rates Respondent-friendly questionnaire Four contacts by first class mail

Pre-notice letter; questionnaire; thank you postcard; replacement questionnaire; final

Return envelopes with real first class stamps Personalization of correspondence Token prepaid financial incentives

First Contact: Pre-Notice Letter Provides positive notice that the recipient will

be receiving a questionnaire Important characteristics:

Brief Personalized Positively worded Aimed at building anticipation vs. providing too

many details Sent about 1 week in advance

Second Contact: The Questionnaire Mail Out

• Cover letter (1 page; date; purpose of letter; why request is important; confidentiality; voluntary participation; enclosures of stamped envelop and incentives; who to contact with questions

• Questionnaire• Return envelope with stamp• Assembling the packet

Third Contact: The Postcard Thank You/Reminder Written to remind participants that a

questionnaire was sent to them Elements:

Reminder that questionnaire was sent Thank you to those who have returned; request

for others to do so. Invitation to ask for a replacement questionnaire.

Fourth Contact: The First Replacement Questionnaire Elements:

Haven’t heard from you Others have responded and answers are

important Eligibility Confidentiality Voluntary

The Fifth Contact: Special Procedures Less frequently done in practice Last attempt; most intense

Certified mail Telephone

Sampling

The survey population consists of all units (e.g. households, individuals) to which one desires to generalize survey results.

The sampling frame is the list from which a sample is to be drawn to represent the population.

A sample are all units of the population that are drawn for inclusion in the survey.

Probability Sampling

Get help! How large of a sample do you need?

How precise do you need your estimates? Size of the population How varied the population is with respect to your

characteristic of interest Amount of confidence you wish to have in the

estimates of the entire population.

What Am I Measuring? A Look At Reliability Reliability has to do with the quality of

measurement. Practically speaking, reliability is the "consistency" or "repeatability" of your measures.

Internal Consistency is one type of reliability measure. Cronbach's alpha measures internal consistency by how well a set of items (or variables) measures a single uni-dimensional latent construct.

What Am I Measuring?A Look at Validity Construct-the degree to which inferences can

legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based.

External-the degree to which the conclusions in your study would hold for other persons in other places and at other times.

Summary Points

There are a lot of things to consider when developing surveys

Leave enough time for the process See if it has been done before Get help if you need it No matter what, another set of eyes can be

helpful Length and simplicity

Next Session (Feb. 21)

You bring: Questionnaire drafts Research questions Ideas Questions

We’ll bring: List of resources IRB contacts & information

Other Ideas for Final Session?

Contact Information

Wendy Cohn wfc2r@virginia.edu Jim Martindale jrm7e@virginia.edu Natalie May nlb7r@virginia.edu Lisa Rollins lkr2h@virginia.edu