Date post: | 11-Jan-2016 |
Category: |
Documents |
Upload: | vivien-mccormick |
View: | 222 times |
Download: | 1 times |
META-ANALYSIS
Dr. Ken Kavale
META-ANALYSIS
Quantitative Research Synthesis
“What the research says”
Knowledge Accumulation
Usable Knowledge
METHODS OF RESEARCH SYNTHESIS
Narrative ReviewSerial Presentation
No Accumulation
Findings remain independent and isolatedIf Accumulated
SUBJECTIVE
QUANTITATIVE REVIEWS
Voting MethodBox-Score Analysis
Studies classified by statistical significance or non-significance
What is statistical significance?Publication Bias
(“File Drawer Problem”)
META-ANALYSIS
Advantages• Uses quantitative (statistical) methods for
organizing and extracting information• Eliminates study selection bias – no
prejudgments about research quality are made• Makes use of all information – findings are
transformed to commensurate expressions of effect magnitude
META-ANALYSIS (con’t)
Detects interactions – study characteristics that may mediate findings are defined and measured, and their covariation is investigated
Seeks general conclusions – practical simplicity
OBJECTIVITY
VERIFIABILITY
REPRODUCABILITY
METHODS OF META-ANALYSIS
Scientific Method
Formulating ProblemsBroadly defined“Is Method x effective?”
SamplingLiterature search- Comprehensive- Published and unpublished
METHODS OF META-ANALYSIS (con’t)
Classifying and Coding StudiesStudy features
Data AnalysisEffect Size – z score (standard deviation units)
__ __ ES = XE – XC
SDC
METHODS OF META-ANALYSIS (con’t)
Effect Size Interpretation• Overlapping distributions and comparable
percentiles• Comparative context• Statistical power – small (.20), medium (.50),
large (.80>)• Binomial effect size display (BESD) – practical
significance• Common language effect size – research
significance
SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE
Intervention ES Number of Studies
Large
Mnemonic strategies
(Mastropieri & Scruggs, 1989)
1.62 24
Reading-comprehension strategies
(Talbot, Lloyd, & Tankersley, 1994)
(Mastropieri et al., 1996)
(Swanson, 1999)
1.13
0.98
0.72
48
68
58
Behavior modification
(Skiba & Casey, 1985) 0.93 41
Direct instruction
(White, 1988) 0.84 25
SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE (con’t)
Intervention ES Number of Studies
Medium
Cognitive behavior modification
(Robinson, Smith, Miller & Brownell, 1999) 0.74 23
Psychotherapy
(Weisz & Weiss, 1993) 0.71 110
Formative evaluation
(Fuchs & Fuchs, 1986) 0.70 21
Early intervention
(Casto & Mastropieri, 1986) 0.68 74
Stimulant medication
(Crenshaw, Kavale, Forness, & Reeve, 1999)
(Kavale, 1982)
0.67
0.58
115
135
SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE (con’t)
Intervention ES Number of Studies
Medium
Computer-assisted instruction
(Schmidt, Weinstein, Niemie, & Walberg, 1985-86)
0.66 18
Peer tutoring
(Cook, Scruggs, Mastropieri, & Casto, 1985-86)
0.58 19
Word recognition strategies
(Swanson, 1999) 0.57 54
SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE (con’t)
Intervention ES Number of Studies
Small
Psycholinguistic training
(Kavale, 1981) 0.39 34
Reducing class size
(Glass & Smith, 1979) 0.31 77
Psychotropic medication
(Kavale & Nye, 1984) 0.30 70
Social-skills training
(Forness & Kavale, 1996)
(Quinn, Kavale, Mathur, Rutherford, & Forness, 1999)
0.21
0.20
53
35
SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE (con’t)
Intervention ES Number of Studies
Small
Modality instruction
(Kavale & Forness, 1987) 0.14 39
Diet restrictions
(Kavale & Forness, 1983) 0.12 23
Perceptual training
(Kavale & Mattson, 1983) 0.08 180
Special class training
(Carlberg & Kavale, 1980) -0.12 50
PERCEPTUAL-MOTOR TRAINING
EFFECT SIZE.08
Table 1
Average ES for Perceptual-Motor Outcome Classes
Outcome Class Mean ES Power Analysis
Perceptual/Sensory Motor
.166 Small
Academic Achievement
.013 Negligible
Cognitive .028 Negligible
PERCEPTUAL-MOTOR TRAINING (con’t)
Table 2
Average ES for Perceptual-Motor General Outcome Categories
General Outcome Categories Mean ES Power Analysis
Perceptual/Sensory Motor
Gross motor
Fine motor
Visual perception
Auditory perception
.214
.178
.149
.122
Small
Small
Small
Small
PERCEPTUAL-MOTOR TRAINING (con’t)
Table 2
Average ES for Perceptual-Motor General Outcome Categories
General Outcome Categories Mean ES Power Analysis
Academic Achievement
Readiness
Reading
Arithmetic
Language
Spelling
Handwriting
.076
-.039
.095
.031
.021
.053
Negligible
Negative
Negligible
Negligible
Negligible
Negligible
Cognitive/Aptitude
Verbal IQ
Performance IQ
-.007
.068
Negative
Negligible
PERCEPTUAL-MOTOR TRAINING (con’t)
Table 3
Average ES for Perceptual-Motor Specific Outcome Categories
General Outcome Categories Mean ES Power Analysis
Gross Motor Skills
Body awareness/image
Balance/posture
Locomotor skills
.256
.263
-.017
Small
Small
Negative
Visual Perceptual Skills
Visual discrimination
Figure-ground discrimination
Visual-motor ability
Visual integration
Visual spatial perception
Visual memory
.146
.173
.222
.086
.144
.062
Small
Small
Small
Negligible
Small
Negligible
PERCEPTUAL-MOTOR TRAINING (con’t)
Table 3
Average ES for Perceptual-Motor Specific Outcome Categories
General Outcome Categories Mean ES Power Analysis
Reading Achievement
Word recognition
Comprehension
Oral reading
Vocabulary
Speed/rate
-.016
-.055
-.037
-.012
-.038
Negative
Negative
Negative
Negative
Negative
PERCEPTUAL-MOTOR TRAINING (con’t)
Table 4
Average ES for Subject Groups
Subject Mean ES Power Analysis
Normal
Educable Mentally Retarded
(IQ = 50-75)
Trainable Mentally Retarded
(IQ = 25-50)
Slow Learner (IQ = 75-90)
Culturally Disadvantaged
Learning Disabled
Reading Disabled
Motor Disabled
.054
.132
.147
.098
.045
.018
-.007
.121
Negligible
Small
Small
Negligible
Negligible
Negligible
Negative
Small
PERCEPTUAL-MOTOR TRAINING (con’t)
Table 5
Average ES for Grade Level
Level Mean ES
Power Analysis
Preschool
Kindergarten
Primary (Grades 1-3)
Middle (Grades 4-6)
Junior high school
High school
.053
.099
.079
.066
.085
.088
Negligible
Negligible
Negligible
Negligible
Negligible
Negligible
MODALITY-MATCHED INSTRUCTION(Learning Styles)
EFFECT SIZE.14
Table 1
Effects of Modality Assessments
Modality Mean ES
(uncorrected)
Mean ES
(corrected)
% of Subjects Differentiated from Comparison Group
Auditory .925 .552 71%
Visual .899 .506 70%
Kinesthetic .970 .430 67%
MODALITY-MATCHED INSTRUCTION(Learning Styles) (con’t)
Table 2
Effects of Modality Matched Instruction
Method Number of ES
Mean ES
Standard Error of ES
% of Subjects Differentiated from Comparison Group
Auditory 80 .184 .028 57
Visual 81 .086 .037 54
Kinesthetic 44 .175 .045 57
MODALITY-MATCHED INSTRUCTION(Learning Styles) (con’t)
Table 2
Effects of Modality Matched Instruction
Modality
Total Auditory Visual Kinesthetic
ES % ES % ES % ES %
Word
Recognition .150 56 .203 58 .081 53 .197 58
Comprehension .046 52 .062 52 .034 51 .041 52
Vocabulary .174 57 .194 58 .141 56 .185 58
Spelling .184 57 .249 60 .088 54 .216 59
STIMULANT MEDICATION
EFFECT SIZE.58.67
STIMULANT MEDICATION (con’t)
Table 1
Average Effect Sizes for Stimulant Medication Research Outcome Classes
Mean ES Power Analysis
Behavioral
Global improvement ratings .886 Large
Rating scales and checklist .837 Large
Activity level .846 Large
Attention and concentration .782 Large
Behavior (social and classroom) .634 Medium-Large
Anxiety .118 Small
STIMULANT MEDICATION (con’t)
Table 1
Average Effect Sizes for Stimulant Medication Research Outcome Classes
Mean ES Power Analysis
Cognitive
Intelligence .391 Medium
Achievement .383 Medium
Drawing and copying .467 Medium
Perceptual, memory and motor .412 Medium
Learning characteristics .367 Small-Medium
FEINGOLD DIET
EFFECT SIZE.12
Table 1
Average Effect Sizes for Feingold Diet Research Outcome Categories
Category Mean ES Power Analysis
Conners Scale-Parents .156 Small
Conners Scale-Teachers .268 Small
Global improvement .128 Small
Hyperkinesis rating .293 Small
Attention .015 Negligible
Disruptive behavior .052 Negligible
Impulsivity .153 Small
Learning ability -.055 Negative
INTERVENTIONS FOR ADHD
Stimulant Medication Feingold Diet Effect Size Effect Size
.63 .12
Treatment DurationStimulant Medication Feingold Diet
10 weeks 39 weeks
ComparisonStimulant medication better than 5X more effective than the
Feingold diet in ¼ the time
SPECIAL CLASS PLACEMENT
EFFECT SIZE-.12
Table 1
Average ES by Special Education Classification
Diagnosis Mean ES
Mental retardation (IQ 50-75) -.14
Slow learning (IQ 75-90) -.34
Learning disability or
Emotional/behavioral disorder
.29
SOCIAL SKILLS TRAINING
EFFECT SIZE
.21
.20
SOCIAL SKILLS TRAINING (con’t)
Table 1
Average ES for Teacher Assessments of Social Skills Training for Students with Learning Disabilities
Training Program Mean ES % Equivalent
Adjustment .29 62
Dependency .25 60
Conduct disorder .22 59
Interaction .11 54
Hyperactivity .07 53
Academic competence .05 52
Teacher assessment composite mean .16 56
SOCIAL SKILLS TRAINING (con’t)
Table 2
Average ES for General Education Peer Assessments of Social Skills Training for Students with Learning Disabilities
Components Mean ES % Equivalent
Communication .25 60
Acceptance .23 59
Cooperation .22 59
Friendship .22 59
Rejection .20 58
Interaction .20 58
Social status .13 55
Peer assessment composite mean .21 58
SOCIAL SKILLS TRAINING (con’t)
Table 3
Average ES for Self-Assessments of Social Skills Training for Students with Learning Disabilities
Components Mean ES % Equivalent
Social status .38 65
Self-concept .28 61
Social problem solving .28 61
Social competence .27 61
Interaction .19 58
Locus of control .08 53
Self-assessment composite mean .24 59
SOCIAL SKILLS TRAINING (con’t)
Table 4
Average ES of Social Skills Training by Teacher, Peer, Self, Experimenter, and Parent Assessments of Students with
Emotional/Behavioral Disorders
Rater Mean ES % Equivalent
Teacher .22 59
Peer .22 59
Self .22 59
Experimenter .19 58
Parent .15 56
Rater composite mean .20 58
SOCIAL SKILLS TRAINING (con’t)
Table 5
Average ES of Training on Social Skill Dimensions for Students with Emotional/Behavioral Disorders
Dimension Mean ES % Equivalent
Social behavior .27 61
Social relations .27 61
Social problem solving .26 60
Social competence .22 59
SOCIAL SKILLS TRAINING (con’t)
Table 6
Average ES of Training on Social Skill Areas for Students with Emotional/Behavioral Disorders
Area Mean ES % Equivalent
Family relations .20 58
Communication .18 57
School behavior .18 57
Conduct disorder .13 55
SOCIAL SKILLS TRAINING (con’t)
Table 7
Average ES of Training on Social Skill Variables for Students with Emotional/Behavioral Disorders
Variable Mean ES % Equivalent
Anxiety .42 66
Adjustment .27 61
Cooperation .26 60
Interaction .24 59
Self-concept/esteem .16 56
Aggression .13 55