Survey ResearchFrankfort-Nachmias & Nachmias (Chapter 10 – Survey Research, Chapter 11 – Questionnaire Construction)

Applications:James H. Kuklinski; Michael D. Cobb; Martin Gilens, “Racial Attitudes and the "New South." The Journal of Politics Vol. 59, No. 2 (May, 1997), pp. 323-349.

Canache, Damarys, Mondak, Jeffery J., & Seligson, Mitchell A. (2001). “Meaning and measurement in cross-national research on satisfaction with democracy.” Public Opinion Quarterly, 65, 506-528.

Survey Research

Two types of error – Sampling error– Response error

Error in Surveys

Survey experts agree – BY FAR the largest amount of error in survey research comes from response error

Sources of Response Error Survey Administration Question Order Question Type Question Wording

Survey Administration

There are three typical methods used to administer surveys

– Interview– Telephone– Mail

Personal/Telephone Interviewing

Advantages– High response rate– Flexibility/control in questioning– Observation (personal interviewing)– Supervision of interviewers

(telephone)

Personal/Telephone Interviewing

Disadvantages– Expensive– Lack of anonymity – Characteristics of interviewer may

introduce bias

Mail Surveys

Advantages– Cheap!!!– Often avoid bias due to interviewer

characteristics / social desirability– Allows respondent time to answer

Mail Surveys

Disadvantages– Low response rates– Cannot probe– Cannot clarify– Cannot be sure who completes

questionnaire

Dillman’s Total Design Method for Mail Surveys First Mailing

– Typically, when you send a survey out by mail, about 24% of those you sent it to will respond.

Why do we care about the response rate so much???

Dillman Method

After you send the first survey, wait about two weeks and then send a follow-up postcard

The postcard should be sent only to non-respondents to save money

Disadvantage: Loss of anonymity

Dillman Method

The postcard should say something like, “In the past week or so you received a survey from me about XYZ. I know you are a very busy person, but it is important for me to include your opinions in my research. If you still have the survey, could you please complete it and send it to me in the stamped addressed envelope I included? If it got misplaced, I am happy to send you a replacement survey. Please call or email me at ABC and I will send a replacement to you right away. Thank you so much for helping me with this important project. Sincerely, Jane Doe.”

The postcard follow-up typically increases response rate to about 42%.

Dillman Method

About one to two weeks after you send the postcard, send the first replacement questionnaire

Attribute the lack of response to the first survey getting lost in the mail, accidentally misplaced or discarded, or lack of time, etc. Stress the importance of that person’s response to your project.

The first replacement survey will usually get your response rate up to about 59%.

Dillman Method

Send a second replacement survey by certified mail about two weeks after the first replacement survey was sent. This will increase the response rate to about 72%

Dillman Method

Other pieces of advice to maximize response rate– Keep it simple– Make it look like personal mail– Get a sponsor– Offer inducements

The New Frontier in Survey Administration:

The Internet Email surveys Web surveys

Question Order

The order in which you place your questions can cause the error of a response set.

A response set may occur when a series of questions have the same answer choices. This is most likely to happen if questions are on related subjects.

Question OrderExample: Response Set (from the

GSS) “Listed below are various areas of

government spending. Please indicate whether you would like to see more or less government spending in each area. Remember that if you say ‘much more,’ it might require a tax increase to pay for it.

A. The environment. 1. SPEND MUCH MORE 2. SPEND MORE 3. SPEND SAME 4. SPEND LESS 5. SPEND MUCH LESS 6. CANT CHOOSE

Question OrderExample: Response Set (from the

GSS)

B. Health Care. 1. SPEND MUCH MORE 2. SPEND MORE 3. SPEND SAME 4. SPEND LESS 5. SPEND MUCH LESS 6. CANT CHOOSE

Question OrderExample: Response Set (from the

GSS)

C. Education. 1. SPEND MUCH MORE 2. SPEND MORE 3. SPEND SAME 4. SPEND LESS 5. SPEND MUCH LESS 6. CANT CHOOSE

Question OrderExample: Response Set (from the

GSS)

D. Unemployment Benefits. 1. SPEND MUCH MORE 2. SPEND MORE 3. SPEND SAME 4. SPEND LESS 5. SPEND MUCH LESS 6. CANT CHOOSE

Question Order

Etc. What is the danger in a response

set and what can we do about it???

Acquiescence Response-set A type of response-set pattern in

which the respondent tends to “agree” with a series of statements due to their desire to please the interviewer (or respond with the “right” answer)

Acquiescence Response-set and the California F-scale

Authoritarianism personality – “implicit antidemocratic tendencies and fascist potential”

F-scale: http://www.anesi.com/fscale.htm

THE GENERAL ATTITUDE TOWARDS INSTITUTIONAL AUTHORITY SCALE

For each of the following 16 opinions, please state your personal reaction by crossing (x) the appropriate answer for each question as: "agree strongly", "uncertain", "disagree strongly", "agree but not strongly", "disagree but not strongly"

1. The Police are pretty trustworthy 2. A Person should obey only those laws that seem reasonable 3. The Army develops initiative 4. It is unreasonable to say that as a rule teachers work in the best interests of their students 5. The Police are quite unfair in their treatment of certain groups of society 6. The law is the embodiment of justice and equality 7. I disagree with what the Army stands for 8. The Police have a hard job which they carry out well 9. Laws are so often made for the benefit of small, selfish groups that one cannot respect the law 10. The Army reduces men to robots 11. Teachers do not respect the individual personalities of the students 12. Policemen like to bully people 13. Teachers are usually ready to take seriously whatever students feel earnest about 14. Obedience to the law constitutes a value indicative of the highest citizenship 15. In this day and age students should not be expected to call a teacher "Sir" 16. People should feel proud to serve in the Army

Question Order

The order in which you place your survey questions can cause the error of a saliency effect

Specific mention of an issue in a survey may cause a respondent to associate the issue in connection with a later question.

Fox News and Public Opinion on Health Care Reform

“Based on what you know about the health care reform legislation being considered right now, do you favor or oppose the plan?”

– Fox – 35% support, 49% oppose– Other polls – 43% support, 45%

oppose

– Why the difference?

Fox News and Public Opinion on Health Care Reform

3. Do you think Barack Obama's travel and speaking schedule makes him look more like he is a candidate on the campaign trail or more like he is the president of the United States?

4. Do you think President Obama apologizes too much to the rest of the world for past U.S. policies?

5. Do you think the Obama administration is proposing more government spending than American taxpayers can afford, or not?

6. Do you think the size of the national debt is so large it is hurting the future of the country?

7. Would you rather: [ROTATE OPTIONS 1 and 2]Cut spending now so future generations don't have to payKeep spending at current levels and let future generations pay

20. When Barack Obama was a candidate campaigning for the presidency, he spoke of the urgent need to finish the fight in Afghanistan, which he called the central front on the war on terrorism. Do you think that, as president, Obama is doing what it takes to win in Afghanistan?

Question Order

Consistency is another problem we might encounter due to question order.

Respondents may strive to appear consistent in their answers: an answer to a question may be constrained by an answer given earlier.

Question OrderExample: Consistency

“And what about the federal government, does it have too much power or too little power?”

FAR TOO MUCH TOO MUCH RIGHT AMNT OF POWER TOO LITTLE POWER FAR TOO LITTLE POWER CANT CHOOSE

What if this question preceded the government spending questions?

Question Order

A final concern about question order is fatigue

Respondents may give perfunctory answers to questions late in the survey especially if it is long

In addition, response sets are often a problem in lengthy surveys toward the end

Question Order

Some final thoughts on question order– The first few questions should “break the

ice” – don’t jump in with a big question.– Move from general questions to more

specific questions.– Questions on personal or sensitive topics

(such as income, race, age) should be asked at the end with a big disclaimer about why you are asking those questions

Question Type

Closed-ended questions are ones that have response options provided by the researcher.

What are some of the advantages and disadvantages to closed-ended questions?

Question Type

Advantages of Closed-Ended:

– Easy to answer– Answers are easy to compare– Respondents willing to provide more

personal information– May help clarify the question

Question Type

Disadvantages of Closed-Ended:

– Forces people to accept the categories or puts too many responses into the “other” category.

– Choices listed may communicate the kind of response wanted.

Question Type

Open-ended questions allow the respondent to write in an answer.

What are some of the advantages and disadvantages to open-ended questions?

Question Type

Advantages of Open-ended:

– May better indicate respondents thoughts/beliefs/opinions.

– Better when the list of responses might be excessive.

– Might get unanticipated answers.

Question Type

Disadvantages of Open-ended:

– Respondent may say too much or too little.

– Difficult to code all the various answers and analyze them.

– Can be expensive and time-consuming.

Question Type

Single-sided vs. Two-sided questions.

Single-sided: a question that asks respondents to agree or disagree with a single substantive statement.

Two-sided: a question that offers the respondent two substantive choices.

Question Type

Single-sided example:

Do you agree or disagree with the idea that the government should see to it that every person has a job with a good standard of living?

Question Type

Two-sided example:

Do you think that the government should see to it that every person has a job with a good standard of living, or should it let each person get ahead on his or her own?

Question Type

With a single-sided question a larger percentage of respondents tend to agree with the statement given.

Question Type

44% of respondents agreed with the single-sided statement.

Only 30.3% of respondents agreed government should guarantee employment and standard of living in the two-sided question.

Two-sided questions reduce the proportion of respondents who give no opinion.

Question Wording

Double-Barreled Questions are two questions in one. Example:

Do you agree with the statement that the situation in Iraq is deteriorating and that the United States should increase the number of troops in Iraq?

What is wrong with how this question is worded?

Question Wording

Ambiguous Question: one that contains a concept that is not clearly defined. Example:

– What is your income?

What is the problem with this survey question???

Question Wording

The choices you provide on a frequency scale can influence how respondents answer. Consider this question:

How much tv do you watch per day?

Question Wording

When the response options were:A) Up to .5 hoursB) .5 to 1 hourC) 1-2 hoursD) 2-2.5 hoursE) 2.5+ hours

16% responded they watched 2.5+ hours.

Question Wording

When the response options were:A) Up to 2.5 hoursB) 2.5 to 3 hoursC) 3-4 hoursD) 4-4.5 hoursE) 4.5+ hours

38% responded they watched 2.5+ hours.

(From Schwarz et al., 1985)

Question Wording

Leading Questions: Encourages respondents to choose a particular response because the question indicates the researcher expects it.

“Don’t you think that global warming is a serious environmental problem?”

Question Wording

Polls conducted by political organizations and politicians often include leading questions.

1980 poll for the Republican National Committee read as follows on the next slide (a dated, but classic example that is widely used).

Question Wording

“Recently the Soviet armed forces openly invaded the independent country of Afghanistan. Do you think the U.S. should supply military equipment to the rebel freedom fighters?”

Question Wording

Word Choice may lead respondents to an answer as well. You have to really think about what words you use to describe people, things, and events.

Question Wording

Helping “the needy” vs. those “on welfare”.

“Socialized Medicine” vs. “National Health Insurance”

“Fight Against Terrorism” vs. “Going to War”

Question Wording

Linking personalities or institutions to issues can also affect responses.

“Would you say that Governor Beshear’s program for promoting economic development has been very effective, fairly effective . . .”

A respondent’s feelings about Governor Beshear might affect his or her response to the question about economic development.

Question Wording

Push Polls: These are relatively new and conducted by campaigns, parties, and political organizations.

Interviewers feed respondents false and damaging information about a candidate or cause under the guise of asking a question.

Question Wording

“Do you agree or disagree with Candidate X’s willingness to tolerate terrorism in our country?”

The goal is NOT to conduct research but to use innuendo to spread rumors and lies.

This is a politically risky strategy and if it’s too blatant, it can back-fire.

Question Wording

Confusing Questions should be avoided. Yes, this seems so obvious, but even the experts make mistakes sometimes. Double-negatives are especially confusing.

The following question was asked of a national random sample by the Roper Center, a respected polling organization.

Question Wording

Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?

A) Impossible it never happenedB) Don't knowC) Possible it never happened

Question Wording

The responses were as follows:

Impossible it never happened: 65%Don't know: 12%Possible it never happened: 22%

Question Wording

Does it seem possible to you that the Nazi extermination of the Jews never happened or do you feel certain that it happened?

A) Certain it happenedB) Don't knowC) Possible it never happened

Question Wording

The responses were as follows:

Certain it happened: 91%Don't know: 8%Possible it never happened: 1%

Question Type

Social Desirability – a form of response error in which respondents are not truthful due to the fact that they perceive their answer to be out of step with the mainstream of society

Race Relations and the “New South”

Race Relations and the “New South”

Experimental Design– Random assignment of individuals

into two groups (national random sample of white Americans)

Race Relations and the “New South”

Race Relations and the “New South”

Race Relations and the “New South”

Result– South – 42% angry– Non-South – 11% angry

“Equlivalence” in Cross-National Research What is equivalence?

“Satisfaction with Democracy” Canache, Mondak and Seligson “On the whole, are you very

satisfied, fairly satisfied, not very satisfied, or not at all satisfied with the way democracy works in [country]?”

“Satisfaction with Democracy”Canache, Mondak and Seligson

“Satisfaction with Democracy”Canache, Mondak and Seligson