Day 6: Non-Experimental & Day 6: Non-Experimental & Experimental DesignExperimental Design
Where are the beakers??Where are the beakers??
What kind of research is considered What kind of research is considered the “gold standard” by the Institute of the “gold standard” by the Institute of Education Sciences?Education Sciences?
A.A. DescriptiveDescriptiveB.B. Causal-ComparativeCausal-ComparativeC.C. CorrelationalCorrelationalD.D. ExperimentalExperimental
Why?Why?
Why does most educational Why does most educational research use non-experimental research use non-experimental
designs?designs?
What is the purpose of What is the purpose of non-experimental designs?non-experimental designs?
Causal-Comparative ExampleCausal-Comparative Example
Green & Jaquess (1987) Green & Jaquess (1987) – Interested in the effect of high school Interested in the effect of high school
students’ part-time employment on their students’ part-time employment on their academic achievement.academic achievement.
– Sample: 477 high school juniors who were Sample: 477 high school juniors who were unemployed or employed > 10 hours/wk.unemployed or employed > 10 hours/wk.
Causal-Comparative DesignCausal-Comparative Design
A study in which the researcher attempts to A study in which the researcher attempts to determine the cause, or reason, for pre-determine the cause, or reason, for pre-existing differences in groups of individualsexisting differences in groups of individuals
At least two different groups are compared on At least two different groups are compared on a a dependent variabledependent variable or measure of or measure of performance (called the “effect”) because the performance (called the “effect”) because the independent variableindependent variable (called the “cause”) has (called the “cause”) has already occurred or cannot be manipulated already occurred or cannot be manipulated
Causal-Comparative DesignCausal-Comparative Design
Ex-post factoEx-post facto– Causes studied after they have exerted Causes studied after they have exerted
their effect on another variable.their effect on another variable.
Causal-Comparative DesignCausal-Comparative Design
DrawbacksDrawbacks– Difficult to establish causality based on Difficult to establish causality based on
collected data.collected data.– Unmeasured variables (Unmeasured variables (confoundingconfounding
variables) are always a source of potential variables) are always a source of potential alternative causal explanations.alternative causal explanations.
Some Thought Questions…Some Thought Questions…
Correlational DesignCorrelational Design
Determines whether and to what degree Determines whether and to what degree a relationship exists between two or a relationship exists between two or more quantifiable variables.more quantifiable variables.
Correlational DesignCorrelational Design
The degree of the relationship is The degree of the relationship is expressed as a coefficient of correlationexpressed as a coefficient of correlationExamplesExamples– Relationship between math achievement Relationship between math achievement
and math attitudeand math attitude– Relationship between degree of a school’s Relationship between degree of a school’s
racial diversity and student use of racial diversity and student use of stereotypical languagestereotypical language
– Your topics?Your topics?
Correlation coefficient…Correlation coefficient…
-1.00 +1.00
strong negative strong positive
0.00
no relationship
Advantages of Correlational DesignAdvantages of Correlational Design
Analysis of relationships among a large Analysis of relationships among a large number of variables in a single studynumber of variables in a single studyInformation about the Information about the degreedegree of the of the relationship between the variables being relationship between the variables being studiedstudied
CautionsCautions
A relationship between two variables A relationship between two variables does not mean one causes the other does not mean one causes the other (Think about the reading achievement (Think about the reading achievement and body weight correlations)and body weight correlations)Possibility of low reliability of the Possibility of low reliability of the instruments makes it difficult to identify instruments makes it difficult to identify relationshipsrelationships
CautionsCautions
Lack of variability in scores (e.g. Lack of variability in scores (e.g. everyone scoring very, very low; everyone scoring very, very low; everyone scoring very, very high; etc.) everyone scoring very, very high; etc.) makes it difficult to identify relationshipsmakes it difficult to identify relationshipsLarge sample sizes and/or using many Large sample sizes and/or using many variables can identify significant variables can identify significant relationships for statistical reasons and relationships for statistical reasons and not because the relationships really exist not because the relationships really exist (Avoid (Avoid shotgunshotgun approach) approach)
CautionsCautions
Need to identify your sample to know Need to identify your sample to know what is actually being compared.what is actually being compared.If using predictor variables, time interval If using predictor variables, time interval between collecting the predictor and between collecting the predictor and criterion variable data is important.criterion variable data is important.
Correlational DesignsCorrelational Designs
Guidelines for interpreting the size of Guidelines for interpreting the size of correlation coefficientscorrelation coefficients– Much larger correlations are needed for Much larger correlations are needed for
predictions with individuals than with groupspredictions with individuals than with groupsCrude group predictions can be made with Crude group predictions can be made with correlations as low as .40 to .60correlations as low as .40 to .60Predictions for individuals require Predictions for individuals require correlations above .75correlations above .75
Correlational DesignsCorrelational Designs
Guidelines for interpreting the size of Guidelines for interpreting the size of correlation coefficientscorrelation coefficients– Exploratory studiesExploratory studies
Correlations of .25 to .40 indicate the need Correlations of .25 to .40 indicate the need for further researchfor further researchMuch higher correlations are needed to Much higher correlations are needed to confirm or test hypotheses confirm or test hypotheses
Correlational DesignsCorrelational Designs
Criteria for evaluating correlational studiesCriteria for evaluating correlational studies– Causation should not be inferred from Causation should not be inferred from
correlational studiescorrelational studies– Practical significance should not be confused Practical significance should not be confused
with statistical significancewith statistical significance– The size of the correlation should be The size of the correlation should be
sufficient for the use of the results sufficient for the use of the results (individuals vs groups)(individuals vs groups)
Think…Think…
If you were going to take your action If you were going to take your action research topic, and create a causal-research topic, and create a causal-comparative study, what would it look comparative study, what would it look like?like?
--OR----OR--If you were going to take your action If you were going to take your action research project, and create a research project, and create a correlational study, what would it look correlational study, what would it look like?like?
Experimental DesignExperimental Design
The Gold Standard?The Gold Standard?
To ReviewTo Review
Why is most educational research Why is most educational research comprised of comprised of nonnon-experimental research -experimental research designs?designs?
To ReviewTo Review
What is the purpose of What is the purpose of nonnon-experimental -experimental research?research?
To ReviewTo Review
How does the independent variable How does the independent variable function in function in nonnon-experimental research?-experimental research?
To ReviewTo Review
Can non-experimental research claim Can non-experimental research claim causality?causality?
An exampleAn example
Read the example given in class and in Read the example given in class and in pairs respond to the questionspairs respond to the questions
Experimental ResearchExperimental Research
PurposePurpose– To make To make causalcausal inferences about the relationship inferences about the relationship
between the independent and dependent variablesbetween the independent and dependent variables
CharacteristicsCharacteristics– Direct manipulationDirect manipulation of the independent variable of the independent variable– Control of extraneous variablesControl of extraneous variables
Experimental DesignsExperimental Designs
Single Group Post-testSingle Group Post-testSingle Group Pre-test Post-test Single Group Pre-test Post-test Non-Equivalent Groups Post-testNon-Equivalent Groups Post-testQuasi-Experimental DesignQuasi-Experimental DesignRandomized Post-test onlyRandomized Post-test onlyRandomized Pre-test Post-testRandomized Pre-test Post-testFactorialFactorial
Examples
Experimental ValidityExperimental Validity
Internal validityInternal validity– The extent to which the The extent to which the independent independent
variablevariable, and not other , and not other extraneous extraneous variablesvariables , produced the observed effect on , produced the observed effect on the dependent variablethe dependent variable
External validityExternal validity– The extent to which the results are The extent to which the results are
generalizablegeneralizable
Internal ValidityInternal Validity
Threats that reduce the level of confidence in Threats that reduce the level of confidence in any causal conclusionsany causal conclusionsKey Question: Is this a Key Question: Is this a plausibleplausible threat to the threat to the internal validity of the study?internal validity of the study?
Threats to Internal ValidityThreats to Internal ValidityHistoryHistory– Extraneous events have an effect on the subjects’ Extraneous events have an effect on the subjects’
performance on the dependent variableperformance on the dependent variable– Ex - The crash of the stock market, 9-11, the Ex - The crash of the stock market, 9-11, the
invasion of Iraq, etc.invasion of Iraq, etc.
SelectionSelection– Groups that are initially Groups that are initially notnot equal due to equal due to
differences in the subjects in those groupsdifferences in the subjects in those groups– Ex - Positive and negative attitudes, high and low Ex - Positive and negative attitudes, high and low
achievers, etc.achievers, etc.
Threats to Internal ValidityThreats to Internal Validity
MaturationMaturation– Changes experienced within the subject over Changes experienced within the subject over
timetime
PretestingPretesting– The effect of having taken a pretestThe effect of having taken a pretest
InstrumentationInstrumentation– Poor technical quality (i.e. validity, reliability) Poor technical quality (i.e. validity, reliability)
or changes in instrumentationor changes in instrumentation
Threats to Internal ValidityThreats to Internal ValiditySubject attritionSubject attrition– Differential loss of subjects from groupsDifferential loss of subjects from groups
Statistical regressionStatistical regression– The natural movement of extreme scores toward the The natural movement of extreme scores toward the
meanmean
Diffusion of treatmentDiffusion of treatment– The treatment is given to the control groupThe treatment is given to the control group
Experimenter effectsExperimenter effects– Different characteristics or expectations of those Different characteristics or expectations of those
implementing the treatments across groupsimplementing the treatments across groups
Threats to Internal ValidityThreats to Internal Validity
Subject effectsSubject effects– The effects of being aware that one is The effects of being aware that one is
involved in a studyinvolved in a study– Four typesFour types
Hawthorne effectHawthorne effectJohn Henry effectJohn Henry effectResentful demoralizationResentful demoralizationNovelty effectNovelty effect
Internal ValidityInternal Validity
Key Point: Ultimately, validity is a matter Key Point: Ultimately, validity is a matter of judgment. Ask if it is of judgment. Ask if it is reasonablereasonable that that possiblepossible threats are threats are likelylikely to affect the to affect the results.results.
External Validity
The extent to which results can be generalized from a sample to a particular population.Question – Why would really good internal validity often result in poor external validity?
External ValidityExternal Validity
Factors affecting external validityFactors affecting external validity– SubjectsSubjects
Representativeness of the sample in Representativeness of the sample in comparison to the populationcomparison to the populationPersonal characteristics of the subjects Personal characteristics of the subjects
– Situations - characteristics of the settingSituations - characteristics of the settingSpecific environmentSpecific environmentSpecial situationSpecial situationParticular schoolParticular school
External ValidityExternal Validity
Importance of explanation of sampling Importance of explanation of sampling proceduresprocedures
Experimental DesignsExperimental DesignsSingle Group Post-testSingle Group Post-testSingle Group Pre-test Post-test – Single Group Pre-test Post-test – Libby, DebLibby, Deb
Non-Equivalent Groups Post-test – Non-Equivalent Groups Post-test – Mary, CherylMary, Cheryl
Quasi-Experimental Design – Quasi-Experimental Design – Pete, LauraPete, Laura
Randomized Post-test only –Randomized Post-test only – Amanda, Nicole, Tam Amanda, Nicole, Tam
Randomized Pre-test Post-test – Randomized Pre-test Post-test – Karen, Jen, Justin Karen, Jen, Justin
Examples
Your TaskYour Task
Based on the topic of your proposal, Based on the topic of your proposal, design an experimental study using the design an experimental study using the design you were assigned.design you were assigned.– Write a research question and hypothesis.Write a research question and hypothesis.– Sketch out the methods.Sketch out the methods.
Identify strengths and weaknesses of Identify strengths and weaknesses of each design.each design.
Experimental DesignsExperimental Designs
NotationNotation– RR indicates random selection or random indicates random selection or random
assignmentassignment– OO indicates an observation indicates an observation
TestTestObservation scoreObservation scoreScale scoreScale score
– XX indicates a treatment indicates a treatment– AA,, B B, , CC, ... indicates a group, ... indicates a group
Pre-Experimental DesignsPre-Experimental Designs
No pre-experimental design controls internal No pre-experimental design controls internal validity threats well validity threats well Single group pretest onlySingle group pretest only– A X OA X O– Internal validity threatsInternal validity threats
History, maturation, attrition, experimenter effects, subject History, maturation, attrition, experimenter effects, subject effects, and instrumentation are viable threatseffects, and instrumentation are viable threatsUseful only when the research is sure of the status of the Useful only when the research is sure of the status of the knowledge, skill, or attitude being changed knowledge, skill, or attitude being changed andand there are there are no extraneous variables affecting the resultsno extraneous variables affecting the results
Pre-Experimental DesignsPre-Experimental Designs
Single group pretest post-testSingle group pretest post-test– A O X OA O X O– Internal validity threatsInternal validity threats
Maturation and pretesting are threatsMaturation and pretesting are threatsHistory and instrumentation are potential threatsHistory and instrumentation are potential threats
– Useful when subject effects will not influence the Useful when subject effects will not influence the results, history effects can be minimized, and results, history effects can be minimized, and multiple pretests and post-tests are usedmultiple pretests and post-tests are used
Pre-Experimental DesignsPre-Experimental Designs
Non-equivalent groups post-test onlyNon-equivalent groups post-test only– A X O A X O
B OB O– Internal validity threatsInternal validity threats
Definite Threat: Selection Definite Threat: Selection Potential Threats: History, maturation, and Potential Threats: History, maturation, and instrumentationinstrumentation
– Useful when groups are comparable and subjects Useful when groups are comparable and subjects can be assumed to be about the same at the can be assumed to be about the same at the beginning of the studybeginning of the study
Quasi-Experimental DesignsQuasi-Experimental DesignsTypesTypes– Non-equivalent pretest/post-test, experimental Non-equivalent pretest/post-test, experimental
control groupscontrol groupsA O X O A O X O B O OB O O
– Non-equivalent pretest/post-test, multiple treatment Non-equivalent pretest/post-test, multiple treatment groupsgroups
A O XA O X11 O O B O XB O X22 O O
Useful when subjects are in pre-existing Useful when subjects are in pre-existing groups (e.g. classes, schools, teams, etc.)groups (e.g. classes, schools, teams, etc.)
Quasi-Experimental DesignsQuasi-Experimental Designs
Threats to internal validityThreats to internal validity– Selection is the major concernSelection is the major concern– Likely to control for most other threats, Likely to control for most other threats,
provided the groups are not significantly provided the groups are not significantly different from one anotherdifferent from one another
– See Table 9.2 for specific threats related to See Table 9.2 for specific threats related to each designeach design
True Experimental DesignsTrue Experimental Designs
Important terminologyImportant terminology– Random assignmentRandom assignment
Subjects placed into groups by randomSubjects placed into groups by randomEnsures equivalency of the groupsEnsures equivalency of the groups
– Random selection of subjectsRandom selection of subjectsSubjects chosen from population by randomSubjects chosen from population by randomEnsures generalizability to the population from Ensures generalizability to the population from which the subjects were selected (i.e. external which the subjects were selected (i.e. external validity)validity)
True Experimental DesignsTrue Experimental DesignsTypesTypes– Randomized post-test only experimental control Randomized post-test only experimental control
groupsgroupsR A X O R A X O R B OR B O
– Randomized post-test only multiple treatment Randomized post-test only multiple treatment groupsgroups
R A XR A X11 O O R B XR B X22 O O
True Experimental DesignsTrue Experimental Designs
Types (continued)Types (continued)– Randomized pretest/post-test multiple Randomized pretest/post-test multiple
treatment groupstreatment groupsR A O XR A O X11 O O R B O X R B O X22 O O
– Randomized pretest/post-test experimental Randomized pretest/post-test experimental control groupscontrol groups
R A O X O R A O X O R B O O R B O O
True Experimental DesignsTrue Experimental Designs
Threats to internal validityThreats to internal validity– Controls for selection, maturation, and Controls for selection, maturation, and
statistical regressionstatistical regression– Likely to control for most other threatsLikely to control for most other threats– See Table 9.2 for specific threats related to See Table 9.2 for specific threats related to
each designeach design
Evaluating Experimental Evaluating Experimental DesignsDesigns
Criteria for evaluating experimental Criteria for evaluating experimental researchresearch– The primary purpose is to test causal The primary purpose is to test causal
hypotheseshypotheses– There should be direct manipulation of the There should be direct manipulation of the
independent variableindependent variable– There should be clear identification of the There should be clear identification of the
specific research designspecific research design
Evaluating Experimental Evaluating Experimental DesignsDesigns
Criteria for evaluating experimental Criteria for evaluating experimental researchresearch– The design should provide maximum The design should provide maximum
control of extraneous variablescontrol of extraneous variables– Treatments are substantively different from Treatments are substantively different from
one anotherone another– The number of subjects is dependent on or The number of subjects is dependent on or
equal to the number of treatment equal to the number of treatment replicationsreplications