CMNS 260: Empirical Communication Research Methods 13-Review and Overview of the Course
CMNS 260: Empirical Communication Research Methods 13-Review and Overview of the Course
Professor: Jan Marontate Teaching Assistants: Nawal Musleh-Motut, Megan Robertson
Lab Instructor: Chris JeschelnikSchool of Communication.
Simon Fraser UniversityFall 2011
Outline of Class Activities Today• Syllabus & Outline of Class Sessions– Objectives
• Selected excerpts of lecture material to review for final examination
• Study tips for final examination• Discussion of last assignment
Course contentCourse content
• Introduce different forms of research • Analyze relationships between goals,
assumptions, theories and methods• Study basic data collection and analysis
techniques• Research process—focusing on empirical
methods
Why study methods? Practical aspects– learn to read other people’s research & critically
evaluate it– learn ways to find your own “data” to answer your
own research questions– acquire skills potential employers seek– self-defense (against misinformation) &
responsible citizenship
Babbie (1995: 101)
The Research Process
Why study methods? – “Knowledge is power” (to acquire skills for social
action or change)• “Savoir pour pouvoir, Pouvoir pour prévoir” (Auguste
Comte)• «To know to do (have power), to do (have power) in
order to predict the future and plan for it »
– « Knowledge is understanding »• “décrire, comprendre, expliquer ” (Gilles Gaston Granger)• “to describe, to understand and to explain”
Research has the potential to inform and misinform
• even well-done research is not always used accurately
• some research is technically flawed• knowledge of methods an important tool for
understanding logic and limits of claims about research
Research Methodology (Scholarly Perspectives)
• Process– methods– logic of inquiry (assumptions & hypotheses)
• Produces– laws, principles and theories that can be tested• (Karl Popper & notion of falsifiability for politically
engaged scholars interested in the fight against genocide in the early 20th century)
Research has the potential to inform and misinform
• even well-done research is not always used accurately
• some research is technically flawed• knowledge of methods an important tool for
understanding logic and limits of claims about research
Other Ways of Knowing
– authority (parents, teachers, religious leaders, media gurus)
– tradition (past practices)
– common sense– media (TV. etc.)– personal experience
Talk show host Oprah Winfrey
Cory DoctorowElectronic Frontier Assoc. & Boingboing.net
Ordinary Inquiry vs. Scholarly Inquiry
Risks of “Errors” associated with non-scholarly knowledge
• selective observation--only notice some phenomena-- miss others
• overgeneralization-evidence applied to too wide a range of conditions
• premature closure--jumping to conclusions• halo effect--idea of being influenced by prestige
Communication as a Science?
• Field more recent – affiliations with the sciences, social sciences & the
humanities
• Scholarly work (like old ideas of science) distinguished from mythology by methods AND goals
• many different approaches
Relations between theory and empirical observation
• Theory and empirical research– Testing theories through empirical observation
(deductive)– Using empirical observation to develop theories
(Inductive)
Theories
EmpiricalGeneralizations
Observations
Predictions(Hypotheses)
TheScientificProcess
Empirical and LogicalFoundations of Research
(does not have to start with theory)
Source: Singleton & Straits (1999: 27); Babbie (1995: 55)
Scholarly Communities--Norms
• universalism -- research judged on “scientific” merit
• organized scepticism -- challenge and question research
• disinterestedness-- openness to new ideas, non-partisan
• communalism--sharing with others• honesty
Research Questions
• Questions researchers ask themselves, not the questions they ask their informants
• Must be empirically testable• Not– too vague– too general– untestable (with implicit, untested assumed outcomes)
Developing research topics
““Dimensions” of ResearchDimensions” of Research
Neuman (2000: 37)
Purpose ofPurpose of
StudyStudy
Intended Use Intended Use of Studyof Study
Treatment of Time Treatment of Time in Studyin Study
Space Unit of Space Unit of
Analysis Analysis
(examples)(examples)
ExploratoryExploratory
DescriptiveDescriptive
ExplanatoryExplanatory
BasicBasic
AppliedApplied
-Action-Action
-Impact-Impact
-Evaluation-Evaluation
Cross-sectionalCross-sectional
LongitudinalLongitudinal
-Panel-Panel
-Time series-Time series
-Cohort analysis -Cohort analysis
-Case Study-Case Study
--Trend studyTrend study
-dependent -individual-dependent -individual
-independent -family-independent -family
-household-household
-artifact-artifact
(media, (media,
technology)technology)
Exploratory ResearchExploratory Research
• When not much is known about topic• Surprises (e.g. Serendipity effect)• Acquire familiarity with basic concerns
and develop a picture• Explore feasibility of additional
research• Develop questions
Descriptive ResearchDescriptive Research
• Focuses on “who”, “what” and “how”• Background information, to stimulate new
ways of thinking, to classify types, etc.
Explanatory ResearchExplanatory Research
• To test theories, predictions, etc…• Idea of “advancing” knowledge
Intended Use of StudyIntended Use of Study
• Basic• Applied– action research (We can make a difference)– social impact assessment (What will be the
effects?)– evaluation research (Did it work?)– needs assessment (Who needs what?)– cost-benefit analysis (What is it worth?)
Basic or Fundamental ResearchBasic or Fundamental Research
• Concerns of scholarly community• Inner logic and relation to theoretical issues
in field
Applied ResearchApplied Research
• commissioned/judged/used by people outside the field of communication
• goal of practical applications– usefulness of results
Types of Applied ResearchTypes of Applied Research
Action Research Social Impact Assessment Needs Assessment Evaluation Research
• formative (built in)• summative (final outcomes)
Cost-benefit analysis
Treatment of TimeTreatment of Time Cross-sectional(one point in time)
Longitudinal (more than one point in time)
Main Types of Longitudinal StudiesMain Types of Longitudinal Studies• Panel study
– Exactly the same people, at least twice• Cohort Analysis
– same category of people or things (but not exactly same individuals) who/which shared an experience at at least two times
– Examples: Birth cohorts. Graduating Classes, Video games invented in the same year2000 2010
41-50 41-5051-60 51-6061-70 61-7071-80 71-80
• Time-series– same type of info., not exactly same people, multiple time periods, e.g. Same place
2006 2011Burnaby residents Burnaby residents
• Case Studies may be longitudinal or cross-sectional
Lexis Diagram (To study Cohort Survival)
Importance of Choosing Appropriate Unit of Analysis
• example: Ecological Fallacy (cheating)
Ecological Fallacy
Ecological Fallacy
Ecological Fallacy & Reductionism
ecological fallacy--wrong unit of analysis (too high)
reductionism--wrong unit of analysis (too low)reductionism--wrong unit of analysis (too low)
Relationship of Theory & Empirical Observation (Wheel of Science)
Deductive & Inductive Methods (p. 71)
Conceptualization & Operationalization of Conceptualization & Operationalization of Research questionsResearch questions
• Conceptualization:
Development of abstract concepts
• Operationalization:
Finding concrete ways to do research
Reliability & Validity
Reliability dependability is the indicator consistent? same result every time?
Validity measurement validity - how well the conceptual and
operational definitions mesh with each other does measurement tool measure what we think ?
Hypothesis Testing
Possible outcomes in Testing Hypotheses (using empirical research)
• support (confirm) hypothesis• reject (not support) hypothesis• partially confirm or fail to
support• avoid use of PROVE
Causal diagrams
X Y
X Y
Direct relationship (positive correlation)
Indirect relationship (negative correlation)
Causal Diagrams
YX+
X1
X2
Y+
_
X Z Y+ +
XY
Z
+
_
X1
X2
Z Y_+
_
+
Neuman (2000: 56)
Types of Errors in Causal Explanation
• ecological fallacy• reductionism• tautology• teleology• Spuriousness
Double-Barrelled Hypothesis & Interaction Effect
OR
Means one of THREE things
1
2
Interaction effect
Recall: Importance of Choosing Appropriate Unit of Analysis
• Recall example: Ecological Fallacy (cheating)
Ecological Fallacy (cheating)
Ecological Fallacy (cheating Box)
Ecological Fallacy & Reductionism
ecological fallacy--wrong unit of analysis (too high)
reductionism--wrong unit of analysis (too low)reductionism--wrong unit of analysis (too low)
Teleology & Tautology
tautology--circular reasoning (true by definition)teleology--too vague for testing
Neuman (2000: 140)
Spurious Relationship
spuriousness--false relationship (unseen third variable or simply not connected)
Neuman (2000: 140)
Example: Storks & Babies– Observations: – Lots of storks seen around apartment buildings
in a new neighbourhood with low cost housing– An increase in number of pregnancies– Did the storks bring the babies???
?
But...
• The relationship is spurious.– The storks liked the heat coming from the
smokestacks on the roof of the building, and so were more likely to be attracted to that building.
– The tenants of the building were mostly young newlyweds starting families.
– So…the storks didn’t bring the babies after all.
Causal Diagram for Storks
• Stork = S• Baby = B
• Newlywed = N• Chimneys on Building = C
S B+
N B+
C S+
Another example of spurious relationships: number of firefighters & damage
• The larger the number of firefighters, the greater the damage
But...
• A larger number of firefighters is necessary to fight a larger fire. A larger fire will cause more damage than a small one.
• Debate about Hockey Riots in Vancouver. – Did the size of the crowd & amount of drinking
cause the riots? – Did bad planning and inadequate policing cause
the fire?
Causal Diagram
• Firefighter = F• Damage = D
• Size of Fire = S
F D+
F
S+
+ D
BothMoral and
Legal
IllegalOnly
ImmoralOnly
BothImmoral
and Illegal
EthicalIll
egal
Legal
Unethical Source: figure adapted fromNeuman (2000:91)
Ethics & LegalityEthics & LegalityTypology of Legal and Moral Typology of Legal and Moral
Actions in ResearchActions in Research
Privacy, Anonymity, ConfidentialityPrivacy, Anonymity, Confidentiality
• privacy: a legal right (note : public vs. private domain)--even if subject is dead
• anonymity: subjects remain nameless & responses cannot be connected to them (problem in small samples)
• confidentiality: subjects’ identity may be known but not disclosed by researcher, identity can’t be linked to responses
4-Measurement—Scales & Indices (Part 2 of 2 slideshows)
4-Measurement—Scales & Indices (Part 2 of 2 slideshows)
Neuman & Robson Chapter 6
•systematic observation •can be replicated
Creating Measures
Measures must have response categories that are: mutually exclusive
possible observations must only fit in one category
exhaustive categories must cover all possibilities
Composite Measures
• Composite measures are instruments that use several questions to measure a given variable (construct).
• A composite measure unidimensional (all items measure the same construct)– Indices (plural form of index) and scales
Logic of Index Construction
actions combined in single measure, often an ordinal level of
measurement
Logic of Scalesactions ranked
Logic Index--example
Logic Scale-example
Treatment of Missing Data
• eliminate cases with missing data?• substitute average score ?• Guess ?• insert random value ?
• deciding what measure to use for reference populations example: employment rates
Rates & Standardization:
Sampling: key ideas & terms
Bad sampling frame
= parameters do not accurately represent target population– e.g., a list of people in the phone directory
does not reflect all the people in a town because not everyone has a phone or is listed in the directory.
Types of NonprobabilitySamples
4
16
Types of Probability Sampleslink to useful webpage: http://www.socialresearchmethods.net/kb/sampprob.php
Stratified
Evaluating Sampling
• Is the sample representative of the population under
study?
• Assessing Equal chance of being chosen
• Examine Sampling distribution of parameters of
population
• Use Central Limit Theorem to calculate Confidence
Intervals and estimate Margin of Error
Asking Questions
that can be answered
Types of Surveys & Survey Instruments• Self-administered Surveys
• Mail• Web
• Surveys based on Interactive Interviews• Telephone• Online (interactive)• Face-to-face
– Individuals– Focus groups
• Survey Instruments:– Questionnaires
• self-administered • Respondent reads questions & records answers
– Interview Schedules • interviewer reads questions & records responses
Main Types of Unobtrusive Measures
• Physical traces– Erosion (ex. wear on floor in museum
displays as measure of popularity of display)– Accretion (ex. garbage)
• Simple observation• Media analysis such as content analysis,
critical discourse analysis (ex. advertisements, news reports, films, music lyrics etc…)
• Analysis of archives, existing statistics & running records (ex. shoppers’ records, library borrowers’ histories)
• Simple observation
Types of Equivalence for comparative research using existing statistics
Types of Equivalence for comparative research using existing statistics
• lexicon equivalence (technique of back translation)
• contextual equivalence (ex. role of religious leaders in different societies)
• conceptual equivalence (ex. income)• measurement equivalence (ex. different
measure for same concept)
Discrete & Continuous Variables
• Continuous– Variable can take infinite (or large) number of values
within range• Ex. Age measured by exact date of birth
• Discrete– Attributes of variable that are distinct but not
necessarily continuous• Ex. Age measured by age groups (Note: techniques exist
for making assumptions about discrete variables in order to use techniques developed for continuous variables)
Cleaning Data
• checking accuracy & removing errors –Possible Code Cleaning• check for impossible codes (errors)
– Some software checks at data entry– Examine distributions to look for impossible codes
– Contingency cleaning• inconsistencies between answers (impossible
logical combinations, illogical responses to skip or contingency questions)
Treatment of Missing Data (%)• Comparison with medium & low collapsed
Table 5-1 Alienation of Workers
Level of Alienation F %High 30 14 Medium & Low 120 58 No Response 60 29
(Total) 210 100
Table 5-1 Alienation of Workers
Level of Alienation F %High 30 20 Medium & Low 120 80
(Total) 150 100
Non-respondents included Non-respondents eliminated
Grouping Response Categories(%)
• Comparison of with high & medium response categories collapsed
Table 5-1 Alienation of Workers
Level of Alienation Freq % High& medium 87Low 13
(Total) 150
Table 5-1 Alienation of Workers
Level of Alienation Freq %High & Medium 62Low 10No Response 29
(Total) 210 100
Core Notions in Basic Univariate Statistics
Ways of describing data about one variable (“uni”=one)–Measures of central tendency• Summarize information about one variable • three types of “averages”: arithmetic mean,
median, mode
–Measures of dispersion• Analyze Variations or “spread”• Range, standard deviation, percentiles, z-scores
Normal & Skewed Distributions
Details on the Calculation of Standard Deviation
Neuman (2000: 321)Neuman (2000: 321)
The Bell Curve & standard deviation
If Time: Begin Bivariate Statistics (Results with two variables)
• Types of relationships between two variables:– Correlation (or covariation)• when two variables ‘vary together’
– a type of association– Not necessarily causal
• Can be same direction (positive correlation or direct relationship)• Can be in different directions (negative correlation or
indirect relationship)– Independence• No correlation, no relationship• Cases with values in one variable do not have any
particular value on the other variable
Recall (Lecture 2) *Types of variables*
• independent variable (cause)• dependent variable (effect)• intervening variable – (occurs between the independent and the
dependent variable temporally)
• control variable – (temporal occurance varies, illustrations later
today)
Causal Relationships
• proposed for testing (NOT like assumptions)• 5 characteristics of causal hypothesis (p.128)
– at least 2 variables– cause-effect relationship (cause must come before
effect)– can be expressed as prediction– logically linked to research question+ a theory– falsifiable
Types of Correlations & Causal Relationships between Two Variables
X=independent variable Y=dependent variable
• Positive Correlation (Direct relationship)– when X increases Y increases or vice versa
• Negative Correlation (Indirect or inverse relationship)– when X increases Y decreases or vice versa
• Independence – no relationship (null hypothesis)
• Co-variation – vary together ( a type of association but not necessarily causal)
YX-
YX+
Five Common Measures of Association between Two Variables
General Idea of Statistical Significance
• In general English ‘significance’ means important or meaningful but this is NOT how the term is used in statistics
• Tests of statistical significance show you how likely a result is due to chance.
Multi-variate Statistics: Elaboration Paradigm (Types of Patterns)
• Replication: same relationship in both partials as in bivariate table
• Specification: bivariate relationship only seen in one of the partial tables
• Interpretation: bivariate relationship weakens greatly or disappears in partial tables (control variable is intervening—happens in between independent & dependent)
• Explanation: Bivariate relationship weakens or diappears in partial table (control variable is before independent variable)
• Suppressor: No bivariate relationship; relationshp only appears in partial tables.
Elaboration Paradigm Summary
Study Tips for Final Exam
• Practice questions• Other ideas for preparation