Date post: | 05-Jan-2016 |
Category: |
Documents |
Upload: | silvester-stafford |
View: | 215 times |
Download: | 1 times |
Non-Experimental designs: Surveys
Psych 231: Research Methods in Psychology
Non-Experimental designs
Sometimes you just can’t perform a fully controlled experiment Because of the issue of interest Limited resources (not enough subjects, observations are too
costly, etc). • Surveys
• Correlational studies
• Quasi-Experiments
• Developmental designs
• Small-N designs
This does NOT imply that they are bad designs Just remember the advantages and disadvantages of each
Surveys
What are they? Questionnaires and interviews that ask people to
provide information about themselves Why conduct them?
Best way to collect some kinds of information:• Descriptive, behavioral, and preferential
• e.g., demographic information, recreational behavior, and attitudes
To compliment experimental work• Good/common first step, can collect a lot of data about a
lot of variables• Do not have to directly observe behaviors
Surveys
Advantages One can investigate internal events (for example,
attitudes & opinions) Can generalize about an entire population based
on relatively small samples of individuals Large amounts of data can be collected quickly
with relatively little cost (effort, time, etc.)• But they’re often not as “cheap” as you may think
Surveys
Disadvantages Correlational: causal claims shouldn’t be made
• Note: surveys are sometimes used as part of an experimental design, which may allow testing of causal claims
Non-response bias• Why doesn’t everybody respond?• Does response rate interact with variables of interest?
Large data sets are sometimes difficult to analyze Self-reports may not be truthful
• Response set - tendency to respond from a particular perspective
• Social desirability bias (e.g., how a “moral” person would answer)
Stages of survey research
Stage 1) Identify the focus of the study and select your research method What are the objectives of the research? Is a survey method the best approach? What kind of survey should be used?
• Group administration• Mail surveys• Internet surveys• Telephone surveys• Face-to-face interviews• Focus group interviews
Stages of survey research cont.
Stage 2) Determining the research schedule and budget
Stage 3) Establishing an information base Find out what’s been done, what’s known
• E.g., Find other related surveys
Stage 4) Identify the sampling frame The actual population that the sample is drawn
from (as opposed to the ideal population)• Think of it as operationalizing the conceptual level
population• Be aware of potential coverage error – when the
sampling doesn’t lead to a good representativeness
Stages of survey research cont.
Stage 5) Determining the sample method and sampling size Review Probability and Non-Probability methods
• Voluntary response method
Importance of sample size
Voluntary response methods
A kind of convenience sampling methods commonly used
• Problem: Typically only individuals with strong opinions respond, so the results are often extremely biased
Should leftover Halloween candy be given out to students who get an A on the exam?
Should leftover Halloween candy be given out to students who get an A on the exam?
Call 123-NONO if you think NOCall 123-4YES if you think YES
Daily show clip Another fun poll result
Importance of sample size
Sampling error - how is the sample different from the population? Confidence intervals
• An estimate of the mean or percentage of the population, based on the sample data
• “John Doe has 55% of the vote, with a margin of error ± 3%”
• Margin of error (that “± 3%” part)• The larger your sample size, the smaller your margin of
error will be. • Which would you be more likely to believe
• “We asked 10 people …”• “We asked 1000 people …”
Importance of sample size
Sampling error - how is the sample different from the population?
Response rate • What proportion of the sample actually responded to
the survey?• Hidden costs here - what can you do to increase
response rates
• Non-response error (bias)• Is there something special about the data that you’re
missing (From the people who didn’t respond)?
Stages of survey research cont.
Stage 6) Designing the survey instrument Question construction: How the questions are
written is very important• Clearly identify the research objectives
• Do your questions really target those research objectives (think Internal and External Validity)?
• Take care wording of the questions• Keep it simple, don’t ask two things at once, avoid
loaded or biased questions, etc.• How should questions be answered (question type)?
Good and poor questions
GoodPoorWas the FDC negligent by ignoring the warnings about Vioxx during testingand approving it for sale?
a) Yesb) Noc) Unsure
Do you favor eliminating the wasteful excess in the public school budget?
a) Yesb) Noc) Unsure
If the FDC knew that Vioxx caused serious side effects during testing,what should it have done?
a)Ban it from ever being soldb)Require more testing before approving itc)Unsure
Do you favor reducing the public school budget?
a)Yesb)Noc)Unsure
Problem: emotionally charged words
Problem: emotionally charged words
Good and poor questions
GoodPoorShould senior citizens be given moremoney for recreation centers and food assistance programs?
a)Yesb)Noc)Unsure
Should senior citizens be given more money for recreation centers?
a)Yesb)Noc)Unsure
Should senior citizens be given more money for food assistance programs?
a)Yesb)Noc)Unsure
Problem: asks two different questions
Problem: asks two different questions
Good and poor questions
GoodPoorAre you against same sex marriage and in favor of a constitutional amendment to ban it?
a) Yesb) Noc) Unsure
What is your view on same sex marriage?
a) I think marriage is a matter of personal choice
b) I’m against it but don’t want a constitutional amendment
c) I want a constitutional amendment banning it
Problem: Biased inmore than one direction
Problem: Biased inmore than one direction
Problem: Asks two questions
Problem: Asks two questions
Survey Questions
Question types Open-ended (fill in the blank, short answer)
• Can get a lot of information, but• Coding is time intensive and potentially ambiguous
Close-ended (pick best answer, pick all that apply)• Easier to code• Response alternatives are the same for everyone
Rating scales • Used for “how much” judgments
• e.g., measures attitudes, agree/disagree
• Take care with your labels• Range of scores, anchors
Closed-ended Open-endedWhat is the best thing about
ISU? (choose one) 1. Location 2. Academics 3. Dorm food 4. People who sell
things between Milner and the Bone
What is the best thing about ISU?
Survey Questions
if closed-ended decide number/label of response alternatives
• should use odd number (5 or 7 best)
• labels should be clear decide scale
• rating: PSY 231 is an important course in the major.1 2 3 4
5
Strongly Agree Neutral Disagree Strongly
Agree Disagree
Survey Questions
if closed-ended decide scale
• semantic differential: PSY 231 Important _____: _____: _____: _____: _____: Unimportant
Boring _____: _____: _____: _____: _____: Interesting
• nonverbal scale for children:
Point to the face that shows how you feel about the toy.
Survey Questions
Stages of survey research cont.
Stage 7) Pre-testing the survey instrument Fix what doesn’t seem to be working
Stage 8) Selecting and training interviewers For telephone and in-person surveys Need to avoid interviewer bias
Stage 9) Implementing the survey Stage 10) Coding and entering the data Stage 11) Analyzing the data and preparing a
final report