Applied Opinion Research Training Workshop Day 2.

Applied Opinion Research Training Workshop

Day 2

Designing Quantitative Instruments

Mary McIntosh

Quantitative Instruments

Designing a Questionnaire – Keep in Mind: Don’t forget the big picture

Follow your objectives and hypotheses to determine the content of your questions

The quality of your research depends on the quality of your questionnaire

Ask others to review and proofread your survey Always pretest, do a trial run Utilize other sources to assist you in designing your

questionnairePrevious studies, expert input, etc.

Make instructions as clear as possible


Designing a Questionnaire – Step by Step: Screening respondents Wording questions Response options Order of questions Length of questionnaire How the questionnaire will be administered


Screening Respondents: Sample Parameters First, you want to decide if you will need to specify

certain criteria that your respondents need to satisfy• Age (e.g., over 18, under 65)• Profession (e.g., government official, private sector, etc.)• Status in household (e.g., mother)

These questions should come first


Wording Questions Second, start writing your questions, keeping in

mind: Questions should be kept short Use wording appropriate to your respondents’ level of

education Make all definitions, qualifiers, and assumptions explicit

(e.g., Do you think the Bank’s program was effective? Which program?)


Wording Questions Be careful when using abbreviations, acronyms, and

jargon Be careful with word choice Highlight words that require extra emphasis Make sure your questions are technically accurate Make sure questions are relevant and applicable to

your respondents


Wording Questions The time span for recall in a question should reflect

the saliency of the topic Also be careful about asking questions about future

intentions; keep to the short-term future Time spans, past or future, should be as specific as

possible (e.g., don’t use “lately”)


Wording Questions Do not use leading questions that suggest a particular answer or

the researcher’s viewpoint (e.g., Why does working in the public sector make people lazy

and careless?) Do not use loaded questions that bias people towards particular

answers (e.g., Do you think it is important to have a strong police force in

this time of crisis?) Be wary of double-barreled questions. Rephrase them into two

questions whenever possible (e.g., Should the government reduce its financial help and its

technical assistance to other countries?)


Wording Questions Avoid double negatives (e.g., not prohibit), they can be

confusing. If absolutely necessary, emphasize the “not” by underlining it

Make sure your questions are valid. The question must ask for information which the respondent is capable of providing an informed response

Make the question as specific as possible(e.g., instead of asking “Is privatization a good idea?” ask “Is privatization of utilities a good idea?”)


Wording Questions Explicitly state alternatives

(e.g., instead of asking “To balance the budget, would you be in favor of a higher income tax?” ask “To balance the budget, would you be in favor of a higher income tax, a higher sales tax, or reduced government services?”)

Use face-saving phrasing (e.g., instead of “Did you vote?” ask “Were you able to vote?”)

Be sure the wording of your question matches your response options

Consider “Don’t Know” options


Response OptionsCommon response options are:

– Likert/Rating scale– Dichotomous choice/Trade-off– Multiple choice– Open-ended– Ranking


Likert/Rating Scale Shouldn’t use more than 2-3 scales in each survey effortNumbered Scales Asks for responses on a numbered scale with descriptions for what

the numbers represent (even and odd numbered scales) e.g., Please indicate how favorable your impression of the World Bank is

on a scale of 1-10, 1 being very unfavorable, 10 being very favorable Numbered scales work well in written surveys Numbered scales work well with educated population Numbered scales must be appropriate for culture (e.g., 10 point

scale vs. 7 or 8 point scale)


Likert/Rating ScaleWord Scales Word scales are similar but meet different respondent needs.

e.g., Please indicate how much you agree with this statement, using a scale of “strongly disagree”, “somewhat disagree”, “neither agree nor disagree”, “somewhat agree”, “strongly agree”

What is your opinion of the Bank’s work in the area of infrastructure development in your country. Is it “very good,” “somewhat good,” “somewhat bad,” or “very bad.”

Word scales are usually 3, 4, or 5 point scales Very common, easily used in written form

or verbally Word scales sometimes preferred because the opinion is clearly

defined for respondent


Likert/Rating Scale Provides a great deal of information, quickly and

easily Can be used to compute summary statistics (e.g.,

frequencies, means) and significance tests Best used to measure opinions, attitudes, extent of

agreement/intention, etc. Not as good for measuring facts

Consider a “Don’t Know” or “Refuse” response


Likert/Rating Scale Common formats are 4-points, 5-points, 7-points, and 10-points

(will vary based on how much knowledge respondents have; how nuanced their opinions may be. When in doubt, keep it simple (4 point scale.)

The greater the number of points, the greater variation you will be able to measure in respondents’ responses

An odd number of points allows respondents to give a middle or “neutral” response. Even numbered scale forces respondents to commit.

Odd point middle point can have a number of meanings: don’t know; neutral; neither positive nor negative


Likert/Rating Scale Respondents need to be comfortable with the scales you

utilize If possible, use a balanced scale

• e.g., 1-not at all effective, 5-very effective• If it is not possible to use a balanced scale, make sure your endpoints

are clear antonyms In your number descriptions, 1 should describe the smallest,

most negative response and 4/5/7/10 should describe the largest, most positive response


Dichotomous Choice/Trade-Off Asks for an either/or response

• e.g., Were you able to vote in the last election? Yes/No• e.g., Which of the following two statements best represents your

views on taxation in our country? 1) I believe our tax dollars are not used wisely by

our government.2) I believe our tax dollars are used appropriately

by our government. Flexible and easy to administer in written or verbal format


Dichotomous Choice/Trade-Off Provides limited information - Does not tell you the

extent of opinion Yields dichotomous data Can be used to compare response percentages and

to examine how different groups responded Best used to measure facts. Not good for

measuring opinions, attitudes


Multiple Choice Gives respondent a variety of responses to choose from

e.g., Which organization do you trust the most? (Oxfam, Save The Children, Catholic Charities)

e.g., What do you consider the most critical development challenge facing your country? (Poverty, Education, Corruption, etc.)

e.g., Which of the following describes your position? (Government official, Private sector, NGO, etc.)


Multiple Choice Very common, flexible and easily used in written

form; verbal ease depends on the number of choices Provides a great deal of information, quickly and

easily Yields categorical or ordered data Can be used to compare response percentages and

to examine how different groups responded


Multiple Choice Under some circumstances, you may want the respondent to

be able to choose more than one answer• e.g., In your opinion, which are the most important Bank programs in

your country? (choose up to 3) Response options must be mutually exclusive

• e.g., How much of your country’s GDP do you think is spent on foreign assistance:

< 1% 1-4% 5-8% 9+%


Multiple Choice Response options need to be exhaustive, but not

overwhelming• Include an “Other” choice in which respondents can write

in a response that you did not include When applicable, order choices from smallest to

largest, most negative to positive• “Other” and “Don’t Know” options should be last


Open-Ended Allows respondent to provide any response that they want

• e.g., Why do you think economic reforms have not been implemented by the government?

• e.g., How would you improve the effectiveness of Bank programs? Common, however, must be transcribed and, if not in English,

translated Limited to skill of interviewer and knowledge of respondent


Open-Ended Provides a great deal of information Yields data unique to each respondent;

once it is coded, response percentages can be computed and compared

• Risk: coding is subjective, time consuming, and expensive

Best used for exploratory research • Use when many answers are possible and multiple-choice options are

too extensive or unknown


RankingAsks respondent to rank a variety of responses

according to some criterion• e.g., Please rank the following in order of priority from 1-

10, 1 being top priority: Education, Communicable Disease, Weak Institutions, The Economy, Corruption, etc.

Less common, best used in written form, difficult to administer verbally


Ranking Can be difficult for respondents to complete Provides limited information; assumes that

respondents feel differently about each item Data is difficult to analyze and interpret Best used to measure limited attitudes and opinions


Order of Questions If necessary, screening questions should be first to

eliminate ineligible respondents early Then, using the funnel approach, proceed from

broad, general questions to more specific questions Opening questions should be easy and

non-threatening to keep respondents’ interest If possible, make the opening questions interesting

to peak respondents’ interest


Order of Questions Consider the advantages/disadvantages of starting a

questionnaire with an open-ended question Avoid questions that get monotonous Consider the advantages/disadvantages of completing all

questions on an issue before moving to a new issue Order issues in a logical fashion to facilitate the flow of the

questionnaire Obtain historical information in chronological order, either

forward or backward


Order of Questions Use transition statements to facilitate shifts from

one issue to another or one series of questions to another

Eliminate order bias due to the sequencing of questions as much as possible• e.g., do not ask about economic growth as a major

development challenge before you ask respondents what they consider to be the major development challenges for their country


Order of Questions Do not ask sensitive or difficult questions until a rapport has

been established with the respondent Any demographic/classification questions should be asked at

the end of the questionnaire, unless required for screening or cultural tradition

• Demographic/classification questions should be ordered from the least to the most sensitive items

Avoid complex or frustrating skip patterns (if self-administered or face-to-face)


Length of Questionnaire Don’t burden respondents with a long questionnaire

• The shorter the questionnaire, the higher the quality of the data Only ask questions that meet your specific research

objectives• For each question, ask is this question necessary?

Pre-test your questionnaire, even if it is just among your colleagues, to determine how long it will take respondents to complete it• Language expansion factor• Cultural traditions of long introduction


Administering the QuestionnaireConfidentiality must be ensuredConsider not using a political polling firmKeep in mind how you will be administering the

questionnaire (e.g., mail-in, phone, in person, within groups, electronic) when deciding on the wording and length of the questionnaire


Administering the Questionnaire Mail-in Questionnaires:

• Best for longer questionnaires that may include sensitive issues for which respondents will want to maintain confidentiality

• Should include postage-paid return envelopes to encourage participation

Phone Surveys:• Best for moderate length questionnaires, around twenty minutes• Should be done at appropriate times with interviewers fluent in the

respondents’ language


Administering the Questionnaire In-person Surveys:

Best for longer questionnaires Should be done at appropriate times with local

interviewers who are fluent in the respondents’ language Skill of the interviewer will influence the quality of the data

Hands-on-Work:Designing a Quantitative Questionnaire

Designing Qualitative Instruments

Sharon Felzer

Qualitative Instruments

Designing a Guideline – Keep in Mind: This is a questionnaire for discussion Don’t forget the big picture

• Follow your objectives and hypotheses to determine the content of your questions

The quality of your research depends on the quality of your guideline and how well the guideline is administered

Ask others to review your guideline Utilize other sources to assist you in designing your guideline

• Previous studies, expert input, etc.


Designing a Guideline – Step by Step: IntroductionWording questionsOrder of questionsLength of guidelineAdministration of guideline


IntroductionThe point of the introduction period is to make the

respondent(s) feel more comfortable in the situation The researcher should introduce themselves and the

research Discuss anonymity and confidentiality Introduce respondent(s)


Wording QuestionsWriting questions for a guideline is much like writing

a questionnaire with all open-ended questions; therefore, many of the recommendations are the same, keeping in mind:

• You want a logical, systematic conversation• Replicate as much as possible by other interviewers


Wording QuestionsMost importantly:

• Your questions must encourage discussion:– Why? – How? – What do you think about…?– In what ways…?– What do you know about…?

» e.g., What is your perspective on privatizing the airlines in our country?


Wording QuestionsProvide probes for the researcher to use if the initial question does not elicit sufficient discussion. For instance:

– Question: What do you think are the major development challenges facing your country?

– Probe: Why are these more pressing than other challenges?

– Probe: Describe how the government is meeting these challenges (probe: resources, partnerships, etc.?)

Qualitative InstrumentsWording Questions

Occasionally, a close-ended question may be necessary, just be sure to include follow-ups

e.g., Do you believe families should have to pay for their children’s education? Why/why not?

Although you want to encourage discussion, make sure your questions are sufficiently focused that respondents do not go too far astray

You should set approximate time allotments for each section to be sure the researcher covers all the issues

For certain questions, you may want to take notes on a board for the respondents to see; indicate this in the guideline


Order of Questions As with quantitative questionnaires, use the funnel

approach, proceeding from broad, general questions to more specific questions

Opening questions should be easy and non-threatening


Length of Guideline Keep in mind, the objective of this research is to encourage

discussion. You should not expect to get through a lengthy guideline• The shorter the guideline, the more you will be able to encourage a

thorough discussion Only ask questions that meet your specific research

objectives. For each question, ask:• Is this question necessary?• Will this question further the relevant discussion?• Will it meet my objectives?


Administering the Guideline A skilled moderator/interviewer is essential to gathering

quality data Confidentiality must be ensured Consider not using political polling firms Keep in mind how you will be administering the

guideline (e.g., focus group, in-depth interview) when deciding on the wording and length of the guideline


Administering the GuidelineFocus Group:

Professional moderators Best for encouraging discussion of a range of respondents’

opinions Guidelines should be short to moderate in length.

To keep a lively discussion, focus groups should not go too long (about 90 minutes, depending on the culture)

Be careful not to include sensitive issues about which respondents will want to maintain confidentiality and may be reluctant to discuss


Administering the GuidelineFocus Group:

Moderators should be instructed on what to do if the discussion goes too far astray

Moderators must also maintain objectivity, so that respondents do not feel that they should respond in a certain manner

Moderators must be fluent in the respondents’ language


Administering the GuidelineIn-depth Interview:

Professional interviewer Best for a longer guideline, a guideline that does not

lend itself to group discussion, or when a group format would not be appropriate with the desired respondents (e.g., elite government officials)

Confidentiality must be ensured Should be done at appropriate times with interviewers

fluent in the respondent’s language


Administering the GuidelineIn-depth Interview:

Be careful asking about sensitive issues as responses may be vague or respondent may be uncomfortable responding, impacting the entire interview

As with focus groups, interviewers should be instructed on what to do if the discussion goes too far astray and how to maintain objectivity

Hands-on-WorkDesigning a Qualitative Guideline

Date post:	27-Dec-2015
Category:	Documents
Upload:	kenneth-mccoy
View:	212 times
Download:	0 times

Applied Opinion Research Training Workshop Day 2.

Documents