Date post: | 17-Jan-2016 |
Category: |
Documents |
Upload: | alfred-beasley |
View: | 215 times |
Download: | 0 times |
LECTURE 4EPSY 652 FALL 2009
Computing Effect Sizes- Mean Difference EffectsGlass: e = (MeanExperimental – MeanControl)/SD
o SD = Square Root (average of two variances) for randomized designs
o SD = Control standard deviation when treatment might affect variation (causes statistical problems in estimation)
Hedges: Correct for sampling bias: g = e[ 1 – 3/(4N – 9) ]
where N=total # in experimental and control groups
Sg = [ (Ne + Nc)/NgNc + g2/(2(Ne + Nc) ]½
Computing Effect Sizes- Mean Difference Effects Example from Spencer ADHD Adult study
Glass: e = (MeanExperimental – MeanControl)/SD
= (82 – 101)/21.55= .8817
Hedges: Correct for sampling bias: g = e[ 1 – 3/(4N – 9) ]
= .8817 (1 – 3/(4*110 – 9) = .8762
Note: SD computed from t-statistic of 4.2 given in article:e = t*(1/NE + 1/NC )½
123456789101112
A B C D E F G H I J K L
effect Mean E Mean C SDE SDC d
Hedges g Ctrl N Trmt N N w wd
1 1 0.2 1 1 0.60 0.58 10 13 23 5.43 3.142 0.3 -0.4 1 1 -0.05 -0.05 21 20 41 10.24 -0.533 0.8 0.28 1 1 0.54 0.52 9 9 18 4.35 2.254 0.5 -0.46 1 1 0.02 0.02 18 21 39 9.69 0.205 0.2 -0.8 1 1 -0.30 -0.30 73 94 167 40.65 -12.106 0.4 -0.12 1 1 0.14 0.14 52 71 123 29.94 4.207 1 0.36 1 1 0.68 0.68 117 115 232 54.85 37.178 0.46 -0.5 1 1 -0.02 -0.02 8 8 16 4.00 -0.06
mean 0.2154 38.50 43.88 82.38 159.15 34.28s(mean) 0.0793
Computing Mean Difference Effect Sizes from Summary Statisticst-statistic: e = t*(1/NE + 1/NC )½ F(1,dferror): e = F½ *(1/NE + 1/NC )½ Point-biserial correlation:
e = r*(dfe/(1-r2 ))½ *(1/NE + 1/NC )½ Chi Square (Pearson association):
= 2/(2 + N) e = ½*(N/(1-))½ *(1/NE + 1/NC )½
ANOVA results: Compute R2 = SSTreatment/Sstotal
Treat R as a point biserial correlation
Excel workbook for Mean difference computation
STUDY# OUTCOME#STATISTIC TYPE MEAN E MEAN C SD E SD C Ne Nc N
SUMMARY STATISTIC
COMPUTATION d
intermediate computation
intermediate computation hedges g
1 1means, SDs 101 82 22 21 78 32 110 19 0.8817 0.8817 0.875563
2 1 t-statistic 101 82 78 32 110 4.2 0.881705 0.881705 0.875568
3 1 F-statistic 78 32 110 17.64 0.881705 0.881705 4.2 0.875568
4 1point-biserial r 78 32 110 0.374701 0.881705 0.881705 17.64 4.2 0.875568
5 1 chi square 47 76 123 3.66 0.654634 0.654634 0.169989 0.169989 0.650568
6p(t-statistic) 47 76 123 0.05 0.654634 0.654634 1.979764 0.650568
Story Book ReadingReferences1 Wasik & Bond: Beyond the Pages of a Book: Interactive Book Reading and Language Development in Preschool Classrooms. J. Ed Psych 20012 Justice & Ezell. Use of Storybook Reading to Increase Print Awareness in At-Risk Children. Am J Speech-Language Path 20023 Coyne, Simmons, Kame’enui, & Stoolmiller. Teaching Vocabulary During Shared Storybook Readings: An Examination of Differential Effects. Exceptionality 20044 Fielding-Barnsley & Purdie. Early Intervention in the Home for Children at Risk of Reading Failure. Support for Learning 2003
Coding the Outcome1 open Wasik & Bond pdf2 open excel file “computing mean effects
example”3 in Wasik find Ne and Nc4 decide on effect(s) to be used- three outcomes
are reported: PPVT, receptive, and expressive vocabulary at classroom and student level: what is the unit to be focused on? Multilevel issue of student in classroom, too few classrooms for reasonable MLM estimation, classroom level is too small for good power- use student data
Coding the Outcome5 Determine which reported data is usable: here
the AM and PM data are not usable because we don’t have the breakdowns by teacher-classroom- only summary tests can be used
6 Data for PPVT were analyzed as a pre-post treatment design, approximating a covariance analysis; thus, the interaction is the only usable summary statistic, since it is the differential effect of treatment vs. control adjusting for pretest differences with a regression weight of 1 (ANCOVA with a restricted covariance weight):
Interactionij = Grand Mean – Treat effect –pretest effect = Y… - ai.. – b.j.
Graphically, the Difference of Gain inTreat(post-pre) and Gain in Control (post –pre)
• F for the interaction was F(l,120) = 13.69, p < .001.• Convert this to an effect size using excel file Outcomes
Computation• What do you get? (.6527)
Coding the OutcomeY
Control Treatment
gains
Gain not “predicted” from control
pre
post
Coding the Outcome7 For Expressive and Receptive Vocabulary,
only the F-tests for Treatment-Control posttest results are given:Receptive: F(l, 120) = 76.61, p < .001Expressive: F(l, 120) =128.43, p< .001
What are the effect sizes? Use Outcomes Computation1.5441.999
Getting a Study Effect• Should we average the outcomes to get a single
study effect or• Keep the effects separate as different constructs
to evaluate later (Expressive, Receptive) or• Average the PPVT and receptive outcome as a
total receptive vocabulary effect?Comment- since each effect is based on the same
sample size, the effects here can simply be averaged. If missing data had been involved, then we would need to use the weighted effect size equation, weighting the effects by their respective sample size within the study
Getting a Study EffectFor this example, let’s average the three
effects to put into the Computing mean effects example excel file- note that since we do not have means and SDs, we can put MeanC=0, and MeanE as the effect size we calculated, put in the SDs as 1, and put in the correct sample sizes to get the Hedges g, etc.
(.6567 + 1.553 + 2.01)/3 = 1.4036
2 Justice & EzellReceptive: 0.403Expressive: 0.8606Average = 0.6303
3 Coyne et al• Taught Vocab: 0.9385• Untaught Vocab: 0.3262• Average = 0.6323
4 Fielding• PPVT: -0.0764
Computing mean effect sizeUse e:\\Computing mean effects1.xls
12345678
A B C D E F G H I J K L
Study Mean E Mean C SDE SDC d
Hedges g Ctrl N Trmt N N w wd
1 1.4036 0 1 1 0.65 1.40 61 63 124 24.87 34.912 0.6303 0 1 1 0.63 0.61 15 15 30 7.16 4.393 0.6323 0 1 1 0.63 0.62 30 34 64 15.20 9.494 0.5 -0.46 1 1 0.02 -0.08 23 26 49 12.20 -0.93
mean 0.8054 32.25 34.50 66.75 59.43 47.86s(mean) 0.1297
Mean
Computing Correlation Effect SizesReported Pearson correlation- use thatRegression b-weight: use t-statistic reported,
e = t*(1/NE + 1/NC )½ t-statistics: r = [ t2 / (t2 + dferror) ] ½ Sums of Squares from ANOVA or ANCOVA:
r = (R2partial) ½ R2partial = SSTreatment/Sstotal
Note: Partial ANOVA or ANCOVA results should be noted as such and compared with unadjusted effects
Computing Correlation Effect SizesTo compute correlation-based effects, you can
use the excel program “Outcomes Computation correlations”
The next slide gives an example.Emphasis is on disaggregating effects of
unreliability and sample-based attenuation, and correcting sample-specific bias in correlation estimation
For more information, see Hunter and Schmidt (2004): Methods of Meta-Analysis. Sage.
Correlational meta-analyses have focused more on validity issues for particular tests vs. treatment or status effects using means
Computing Correlation Effects Example
STUDY# OUTCOME# x alpha y alpha Ne Nc N r rcorrected s(r ) Nr N*(r-rmean)disattenuated r Ndisr s(edis)
reliabiltiy reliabiltiy1 1 0.80 0.77 47 76 123 0.352646 0.351381 0.079277 43.21983 -5.1631 0.449313 55.26549 0.1010082 1 0.70 0.80 33 55 88 0.323444 0.32178 0.095995 28.31665 -6.26369 0.432221 38.03544 0.1282793 1 0.75 0.90 22 45 67 0.190571 0.18918 0.118621 12.67504 -13.6715 0.231956 15.54103 0.1443814 1 0.88 0.70 111 111 222 0.67 0.669165 0.037071 148.5545 61.13375 0.853659 189.5123 0.0472335 1 0.77 0.85 34 34 68 0.169989 0.168757 0.118639 11.47548 -15.2751 0.210119 14.28811 0.1466476 1 0.90 0.78 47 45 92 0.177133 0.17619 0.101539 16.20946 -20.0091 0.211412 19.44991 0.12119
N(r-rmean) N(rdis-rdismean)0.229994041 0.356773 r(mean)= 0.3946230.466932722 0.442974 rdis(mean)= 0.503172.827858564 4.92834 Var(rmean )= 0.00113816.73286084 27.27103 s(rmean)= 0.0337393.469040187 5.8397574.389591025 7.831295 s(emean)= 0.080498
s(edismean)=
EFFECT SIZE DISTRIBUTION
Hypothesis: All effects come from the same distribution
What does this look like for studies with different sample sizes?
Funnel plot- originally used to detect bias, can show what the confidence interval around a given mean effect size looks likeNote: it is NOT smooth, since CI depends on
both sample sizes AND the effect size magnitude
EFFECT SIZE DISTRIBUTIONEach mean effect SE can be computed from
SE = 1/ (w)
For our 4 effects: 1: 0.200525 2: 0.373633 3: 0.256502 4: 0.286355
These are used to construct a 95% confidence interval around each effect
EFFECT SIZE DISTRIBUTION- SE of Overall MeanOverall mean effect SE can be computed
fromSE = 1/ (w)
For our effect mean of 0.8054, SE = 0.1297Thus, a 95% CI is approximately (.54, 1.07)The funnel plot can be constructed by
constructing a SE for each sample size pair around the overall mean- this is how the figure below was constructed in SPSS, along with each article effect mean and its CI
EFFECT SIZE DISTRIBUTION- Statistical testHypothesis: All effects come from the same
distribution: Q-testQ is a chi-square statistic based on the
variation of the effects around the mean effect
Q = wi ( g – gmean)2
Q 2 (k-1)
k
Example Computing Q Excel file
effect d w Qi prob(Qi) sig?
1 0.58 5.43 0.7151598 0.397736175no
2 -0.05 10.24 0.7326248 0.392033721no
3 0.52 4.35 0.3957949 0.52926895no
4 0.02 9.69 0.366319 0.545017585no
5 -0.30 40.65 10.697349 0.001072891yes
6 0.14 29.94 0.1686616 0.681304025no
7 0.68 54.85 11.727452 0.000615849yes
8 -0.02 4.00 0.2125622 0.644766516no
0.2154 Q= 25.015924
df 7
prob(Q)= 0.0007539
Computational Excel fileOpen excel file: Computing QEnter the effects for the 4 studies, w for each
study (you can delete the extra lines or add new ones by inserting as needed)
from the Computing mean effect excel fileWhat Q do you get? Q = 39.57 df=3 p<.001
Interpreting QNonsignificant Q means all effects could have
come from the same distribution with a common mean
Significant Q means one or more effects or a linear combination of effects came from two different (or more) distributions
Effect component Q-statistic gives evidence for variation from the mean hypothesized effect
Interpreting Q- nonsignificantSome theorists state you should stop-
incorrect.Homogeneity of overall distribution does not
imply homogeneity with respect to hypotheses regarding mediators or moderators
Example- homogeneous means correlate perfectly with year of publication (ie. r= 1.0, p< .001)
Interpreting Q- significantSignificance means there may be
relationships with hypothesized mediators or moderators
Funnel plot and effect Q-statistics can give evidence for nonconforming effects that may or may not have characteristics you selected and coded for