Calculations of Calculations of Reliability Reliability We are interested in We are interested in calculating the ICC calculating the ICC – First step: First step: Conduct a single-factor, within- Conduct a single-factor, within- subjects (repeated measures) ANOVA subjects (repeated measures) ANOVA – This is an inferential test for systematic This is an inferential test for systematic error error – All subsequent equations are derived from All subsequent equations are derived from the ANOVA table the ANOVA table
Transcript
Calculations of ReliabilityCalculations of Reliability
We are interested in calculating the ICCWe are interested in calculating the ICC– First step:First step:
Conduct a single-factor, within-subjects (repeated Conduct a single-factor, within-subjects (repeated measures) ANOVAmeasures) ANOVA
– This is an inferential test for systematic errorThis is an inferential test for systematic error– All subsequent equations are derived from the ANOVA All subsequent equations are derived from the ANOVA
Steps for calculation:Steps for calculation:1.1. Arrange the raw data (X) into tabular form, Arrange the raw data (X) into tabular form,
placing the data for subjects in rows (R), and placing the data for subjects in rows (R), and repeated measures in columns (C).repeated measures in columns (C).
Steps for calculation:Steps for calculation:15.15. Compute the sum of squares between columns (SSCompute the sum of squares between columns (SSCC), which is ), which is
the variability due to the repeated-measures treatment effect.the variability due to the repeated-measures treatment effect.– In this case, SSIn this case, SSCC is “systematic variability.” is “systematic variability.”
))((
)()()( 222
21
kN
X
N
CCSS T
C
)2)(8(
)2474(
8
)1226()1248( 222
CSS
25.30CSS
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:16.16. Compute the sum of squares between rows (SSCompute the sum of squares between rows (SSRR), ),
which is the variability due to differences among which is the variability due to differences among subjects.subjects.
))((
)()( 22
kN
X
k
RSS T
R
)2)(8(
)2474(
2
794464 2
RSS
75.14689RSS
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:17.17. Calculate the total sum of squares (SSCalculate the total sum of squares (SSTT), which is the ), which is the
variability due to subjects (rows), treatment variability due to subjects (rows), treatment (columns), and unexplained residual variability (columns), and unexplained residual variability (error).(error).
))((
)( 22
kN
XXSS T
T
25.382542397662 TSS
75.15119TSS
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:18.18. Calculate the total sum of squares due to error (SSCalculate the total sum of squares due to error (SSEE), ),
which is the unexplained variability due to error. This which is the unexplained variability due to error. This will be used in the denominator for the F ratio.will be used in the denominator for the F ratio.
75.1468925.3075.15119 ESS
RCTE SSSSSSSS
75.399ESS
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:19.19. Calculate the degrees of freedom for each source of Calculate the degrees of freedom for each source of
variance (dfvariance (dfCC, df, dfRR, df, dfEE, and df, and dfTT).).
1kdfC
)1)(1( NkdfE
1NdfR
1))(( kNdfT
12 Cdf
)18)(12( Edf
18 Rdf
1)2)(8( Tdf
1Cdf
7Edf
7Rdf
15Tdf
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:20.20. Construct an ANOVA table:Construct an ANOVA table:
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75
Within Subjects 8 430
Trials 1 30.25
Error 7 399.75
Total 15 15,119.75
dfRdfC dfE dfT SSRSSC SSE SST
Between Subjects = rows
Trials = columns
Within Subjects = Trials + Error
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:21.21. Calculate the mean square for each source of Calculate the mean square for each source of
variance (MSvariance (MSCC, MS, MSRR, and MS, and MSEE).).
C
CC df
SSMS
E
EE df
SSMS
R
RR df
SSMS
25.30CMS
12.57EMS
54.2098RMS
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54
Within Subjects 8 430 53.75
Trials 1 30.25 30.25
Error 7 399.75 57.11
Total 15 15,119.75
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:22.22. Calculate the F ratio for the treatment effect Calculate the F ratio for the treatment effect
(columns, F(columns, FCC).).
E
CsubjectsWithin MS
MSF
E
RsubjectsBetween MS
MSF
53.0 subjectsWithinF
75.36 subjectsBetweenF
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54 36.75
Within Subjects 8 430 53.75
Trials 1 30.25 30.25 0.53
Error 7 399.75 57.11
Total 15 15,119.75
Repeated Measures ANOVARepeated Measures ANOVA
Determining the Significance of F:Determining the Significance of F:– Use the F Distribution Critical Values table.Use the F Distribution Critical Values table.
dfdfCC = df = dfB B – columns across the top– columns across the top
dfdfEE = df = dfEE – rows down the side – rows down the side
– If your calculated F ratio is greater than the critical F If your calculated F ratio is greater than the critical F ratio, then reject the null hypothesis.ratio, then reject the null hypothesis.
There is a significant difference from Trial A1 to Trial A2There is a significant difference from Trial A1 to Trial A2
There is a significant systematic errorThere is a significant systematic error
– If your calculated F ratio is less than the critical F If your calculated F ratio is less than the critical F ratio, then accept the null hypothesis.ratio, then accept the null hypothesis.
There is no difference from Trial A1 to Trial A2There is no difference from Trial A1 to Trial A2
There is no systematic errorThere is no systematic error
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54 36.75 < 0.05
Within Subjects 8 430 53.75
Trials 1 30.25 30.25 0.53 > 0.05
Error 7 399.75 57.11
Total 15 15,119.75
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54 36.75 0.00005
Within Subjects 8 430 53.75
Trials 1 30.25 30.25 0.53 0.49035
Error 7 399.75 57.11
Total 15 15,119.75
Using ANOVA Table for ICCUsing ANOVA Table for ICC
2 sources of variability for ICC model 3,12 sources of variability for ICC model 3,1– Subjects (MSSubjects (MSSS))
Between-subjects variability (for calculating the ICC)Between-subjects variability (for calculating the ICC)
– Error (MSError (MSEE))
Random error (for calculating the ICC)Random error (for calculating the ICC)
ES
ES
MSkMS
MSMSICC
)1(1,3
ER
ER
MSkMS
MSMSICC
)1(1,3
Equation reported by Weir (2005) Same equation, but modified for our terminology (MSS = MSR).
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54 36.75 0.00005
Within Subjects 8 430 53.75
Trials 1 30.25 30.25 0.53 0.49035
Error 7 399.75 57.11
Total 15 15,119.75
MSE
MSR or MSS
Using ANOVA Table for ICCUsing ANOVA Table for ICC
Calculating the ICCCalculating the ICC3,13,1::
ER
ER
MSkMS
MSMSICC
)1(1,3
11.57)12(54.2098
11.5754.20981,3
ICC
947.01,3 ICC
Interpreting the ICCInterpreting the ICC
If ICC = 0.95If ICC = 0.95– 95% of the observed score variance is due to true 95% of the observed score variance is due to true
score variancescore variance– 5% of the observed score variance is due to error5% of the observed score variance is due to error
2 factors for examining the magnitude of the ICC2 factors for examining the magnitude of the ICC– Which version of the ICC was used?Which version of the ICC was used?– Magnitude of the ICC depends on the between-Magnitude of the ICC depends on the between-
subjects variability in the datasubjects variability in the dataBecause of the relationship between the ICC magnitude and Because of the relationship between the ICC magnitude and between-subjects variability, standard error of measurement between-subjects variability, standard error of measurement values (SEM) should be included with the ICCvalues (SEM) should be included with the ICC
Implications of a Low ICCImplications of a Low ICC
Low reliabilityLow reliabilityReal differencesReal differences– Argument to include SEM valuesArgument to include SEM values
Type I vs. Type II errorType I vs. Type II error– Type I error is rejecting HType I error is rejecting H00 when there was no effect when there was no effect
(i.e., H(i.e., H00 = 0) = 0)– Type II error is failing to reject the HType II error is failing to reject the H00 when there is an when there is an
effect (i.e., Heffect (i.e., H00 ≠ 0)≠ 0)
A low ICC means that more subjects will be A low ICC means that more subjects will be necessary to overcome the increased necessary to overcome the increased percentage of the observed score variance due percentage of the observed score variance due to error.to error.
Standard Error of MeasurementStandard Error of Measurement
ICC ICC relative measure of reliability relative measure of reliability– No unitsNo units
SEM SEM absolute index of reliability absolute index of reliability– Same units as the measurement of interestSame units as the measurement of interest
The SEM is the standard error in The SEM is the standard error in estimating observed scores from true estimating observed scores from true scores.scores.
Calculating the SEMCalculating the SEM
Calculating the SEMCalculating the SEM3,13,1::
EMSSEM
11.57SEM
56.7SEM
SEMSEM
We can report SEM values in addition to We can report SEM values in addition to the ICC values and the results of the the ICC values and the results of the ANOVAANOVA
We can calculate the minimum difference We can calculate the minimum difference (MD) that can be considered “real” (MD) that can be considered “real” between scoresbetween scores
Minimum DifferenceMinimum Difference
The SEM can be used to determine the The SEM can be used to determine the minimum difference (MD) to be considered minimum difference (MD) to be considered “real” and can be calculated as follows:“real” and can be calculated as follows:
296.1SEMMD
296.156.7MD
95.20MD
Example ProblemExample Problem
Now use your skills (by hand) to calculate Now use your skills (by hand) to calculate a repeated measures ANOVA, ICCa repeated measures ANOVA, ICC3,13,1, ,
SEMSEM3,13,1, and MD, and MD3,13,1 for Trials B1 and B2. for Trials B1 and B2.
– Report your results.Report your results.– Compare your results to Trials A1 and A2.Compare your results to Trials A1 and A2.
What is the primary difference?What is the primary difference?
Using the Reliability Worksheet Using the Reliability Worksheet OnlineOnline
Go to the course website and download Go to the course website and download the Reliability.xls worksheet.the Reliability.xls worksheet.– Calculate the ANOVA, ICC, SEM, and MD Calculate the ANOVA, ICC, SEM, and MD
values for both Trials A1 and A2 and Trials B1 values for both Trials A1 and A2 and Trials B1 and B2 and compare your results.and B2 and compare your results.
