Date post: | 21-Dec-2015 |
Category: |
Documents |
View: | 213 times |
Download: | 0 times |
Center for Biofilm Engineering
Marty HamiltonProfessor Emeritus of StatisticsMontana State University
Statistical design & analysis for assessing the efficacy of
instructional modules
CS 580 April 24, 2006
Why Statistics?
Provide convincing results
Improve communication
“...I do not mean to suggest that computers eliminate stupidity---they may in fact encourage it.”Robert P. Abelson, in Statistics as Principled Argument(cited on Rocky Ross’s CS 580 home page)
After-treatment score
A student used the modules, then scored 80% on the test
Conclusion: modules have high efficacy
Data: Choosing thequantity to measure
Reliable tests of knowledge:before-treatment testafter-treatment test
Quantitative response: difference in test scores, after-treatment minus before-treatment
Difference between before- and after-treatment scores
A student used the modules, then scored 50 points higher on the after-treatment test than on the before treatment test (Response = 50).
Conclusion: modules have high efficacy
Anticipating criticism: “natural” improvement
High
LowAfterBefore
Test
score without the
treatment
Response
Anticipating criticism
Before/after observations for just the “treated” student may not accurately represent the treatment effect
May need treated and untreated students (i.e., a control)
Control or comparison
The control can be either a negative control or positive control
A student taking a conventional classroom lecture/recitation course would provide a positive control or comparison
(placebo)(best conventional)
Difference scores for each of 12 students, 6 per group
100
0Treatedgroup
Controlgroup
D
iffere
nce
(aft
er
– b
efo
re)
Of practical importance?
Data: 20 students per group(randomly assigned?)
Treatment ResponseC -28.5096C 34.7186C -3.3184C -13.9297C -5.7949C 29.0260C 15.4682C 29.1025C -10.8522C -18.7876C -3.1457C 5.4531C -9.3185C 1.2575C -11.5470C -17.6932C 5.5314C 6.7628C -10.8001C 18.3930
Treatment ResponseT 53.4115T 75.9697T 8.3348T 33.3584T 42.5355T 58.2345T 47.9143T 58.6826T 48.3604T 68.2412T 91.1052T 42.8328T 48.9096T 67.1174T 39.2733T 68.9961T 52.2039T 39.2210T 31.1658T 36.4764
Analysis via Minitab 14
.Minitab: FirstStudy_CS580.MTWShow data layout ... matrix
Stat > Basic Statistics > Display Descriptive Statistics ... Ask for individual value plotStat > Basic Statistics > 2 Sample t ...Minitab outputTwo-Sample T-Test and CI: Response, Treatment Two-sample T for ResponseTreatment N Mean StDev SE MeanC 20 0.6 17.4 3.9T 20 50.6 18.4 4.1Difference = mu (C) - mu (T)Estimate for difference: -50.016495% CI for difference: (-61.4656, -38.5672)T-Test of difference = 0 (vs not =): T-Value = -8.84 P-Value = 0.000 DF = 38Both use Pooled StDev = 17.8846
Null hypothesis: true mean response for Treatment = true mean response for Control Conclusions:
1. Reject the null hypothesis because it is discredited by the data (p-value < 0.001)2. 95% confident that the treatment mean response is between 38.6 and 61.5 larger than
the true control mean response3. Is this efficacy repeatable?
Analysis via Minitab 14 (more)
Minitab: SixStudies_CS580.MTWShow data layout ... matrix
Stat > Tables > Descriptive Statistics Minitab outputTabulated statistics: Replicate, Treatment Rows: Replicate Columns: Treatment C T All
1 0.60 50.62 25.61 20 20 402 0.07 62.94 31.50 20 20 403 5.09 51.46 28.27 20 20 404 13.29 58.99 36.14 20 20 405 6.85 41.45 24.15 20 20 406 16.05 51.59 33.82 20 20 40All 6.99 52.84 29.92 120 120 240Cell Contents: Response: Mean Count
Analysis via Minitab 14 (more)
Treatment
Re
spo
nse
TC
100
50
0
-50
TC
100
50
0
-50TC
1 2 3
4 5 6
Panel variable: Replicate
SixStudies_CS580
Analysis via Minitab 14 (more)
Stat > ANOVA > General Linear Model ...Minitab outputGeneral Linear Model: Response versus Treatment, Replicate Factor Type Levels ValuesTreatment fixed 2 C, TReplicate(Treatment) random 12 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6Analysis of Variance for Response, using Adjusted SS for TestsSource DF Seq SS Adj SS Adj MS F PTreatment 1 126120 126120 126120 128.16 0.000Replicate(Treatment) 10 9840 9840 984 2.53 0.007Error 228 88786 88786 389Total 239 224746S = 19.7335
Variance Components, using Adjusted SS EstimatedSource ValueReplicate(Treatment) 29.73 Variance among replicate studiesError 389.41 Variance among students in same study and treatment---------added by Marty ----------Total variance 419.14 Repeatability Standard Deviation = 20.5 (single student)Repeatability Standard Deviation = 9.9 (mean of 20 treated students minus mean of 20 control students)
Stat > Basic Statistics > Normality Test... of residuals provides an evaluation of key statistical assumption underlying the ANOVA
Analysis via Minitab 14 (more)
Data copied from Tables output and pasted into the worksheet: Rep CntrlMean TrtMean Mean (Treatment minus Control) 1 0.60 50.62 50.02 2 0.07 62.94 62.87 3 5.09 51.46 46.37 4 13.29 58.99 45.70 5 6.85 41.45 34.60 6 16.05 51.59 35.54
Stat > Basic Statistics > 1 sample t ... analysis of 6 MeansConclusions:
1. Reject the null hypothesis because it is discredited by the data (p-value < 0.001)2. Estimated difference in mean responses = 45.93. 95% confident that the treatment mean response is between 36.9 and 54.9 larger than the true
control mean response4. 95% confident that the treatment mean response is at least 38.6 larger than the true control mean
response5. The efficacy measure is repeatable
Note: this straightforward analysis of the six means, one mean for each of the 6 repeated studies, using a 1-sample t-test provides nearly the same results as does the ANOVA variance component analysis approach.
Trade-offs: What is themain source of variability?
It is often more important to repeat the study
than to expend time and materialsfinding a precise efficacy estimate for a single study.