Date post: | 28-Mar-2015 |
Category: |
Documents |
Upload: | rachelle-downard |
View: | 216 times |
Download: | 0 times |
John Ogier
Making summative unit surveys totally summative - risks or opportunities
Survey Timing
• Collings AHEEF 2007
• I misheard David’s method!!
• What are the implications of a long survey window?
• Into and after exams?
• Post grades?
• Response demographics??
• Immediate response
• Post exams
• Post grades
• General - gender, grades etc
• Online Survey Experiment
• Wanted a wide range of large “units”
• Divide each randomly into 4 groups
• But high female response rate AUSSE 2007 - so maintain gender balance
• Got small selection of units
• Group 1 - normal pre-exams 15 days
• Group 2 - day after exam 15 days
• Group 3 - day after grades 13 days
• Group 4 - open entire 7 weeks
• Day 1 - personalised invitation email
• Week before close - reminder email
Also some other units - MATH200, ECON200 and 300
Responses
Group 1 Group 2 Group 3 Course W1 Rem Non Total W1 Rem Non Total W1 Rem Non Total
AFI S204 11 6 69 86 21 9 56 86 20 5 61 86 ECON104 24 16 126 166 39 11 116 166 32 13 120 165 ECON105 2 3 33 38 3 4 31 38 5 2 32 39
Total 37 25 228 290 63 24 203 290 57 20 213 290 Group 4 W1 Mid Rem Non Total ECON104 36 12 8 112 168 ECON105 6 0 2 30 38
Total 42 12 10 142 206 Group 1 Group 2 Group 3
Others W1 Rem Non Total W1 Rem Non Total W1 Rem Non Total Excluded 100 74 515 689 45 26 217 288 53 20 213 286
(Whole Yr Group 4 Pass-Fail W1 Mid Rem Non Total
Teaching) 108 21 64 552 745
Table 1: Response Count by Group & Response Period
Variables
C_FdBck “I received helpful feedback on my progress”
C_Asmt “The assessments in this course measured my learning effectively”
C_OAll “Overall, this was a good quality course”
T_OAll “Overall, the lecturer is an effective teacher”
Gender F, M
Group 1, 2, 3, 4 (Survey Group)
NumGrade -1 to 9 (“A+” = 9 through to “E” = -1)
PostGrades PG (responded post grades/mid-year test return), NA (prior)
Week W1 (responded Week 1), Rem (post-reminder email), NA (between)
Also knew: -
Ethnicity, International Student status, Citizenship, Age, intended Qualification, Overall GPA, Response Day & Time and Time taken
First analysis
C_O
All
1
2
3
4
5
1 2 3 4
Group
Group Number Mean Std Error Lower 95% Upper 95% 1 178 3.80337 0.06818 3.6695 3.9372 2 157 3.93631 0.07260 3.7938 4.0788 3 150 4.02000 0.07427 3.8742 4.1658 4 254 3.90551 0.05708 3.7935 4.0176
Table 2: C_OAll Means for One-way ANOVA
What about grades?C_F
dB
ck
1
2
3
4
5
1 2 3 4
Group
Each PairStudent's t 0.05
C_A
smt
1
2
3
4
5
1 2 3 4
Group
Each PairStudent's t 0.05
C_O
All
2
3
4
5
1 2 3 4
Group
Each PairStudent's t 0.05
T_O
All
1
2
3
4
5
1 2 3 4
Group
Each PairStudent's t 0.05
Figure 4: ANOVA – C_FdBck, C_Asmt, C_OAll and T_OAll by Group
The dangers of “means”
• Dommeyer, Baum, Hanna & Chapman, 2004 - online surveys “do not produce significantly different mean evaluation scores” from paper
• UC Paper and Online survey history support this
• Not just means - also Std Deviations
• But are the responses “representative of the whole group”?? (Nulty, 2008)
Gender - a response bias?
p N p N N p N p N N p N p N NGroup 1 48 7 38 10 17 96 14 70 26 40 22 2 16 3 5Group 2 48 12 38 18 30 98 29 68 21 50 20 3 18 4 7Group 3 47 11 39 14 25 89 13 76 32 45 30 5 9 2 7Group 4 N/A N/A N/A N/A N/A 94 30 74 26 56 24 4 14 4 8
Total 143 30 115 42 72 377 86 288 105 191 96 14 57 13 27
p% N% p% N% N% p% N% p% N% N% p% N% p% N% N%Group 1 19% 3% 15% 4% 7% 14% 2% 11% 4% 6% 14% 1% 10% 2% 3%Group 2 19% 5% 15% 7% 12% 15% 4% 10% 3% 8% 13% 2% 12% 3% 5%Group 3 18% 4% 15% 5% 10% 13% 2% 11% 5% 7% 20% 3% 6% 1% 5%Group 4 N/A N/A N/A N/A 14% 5% 11% 4% 8% 16% 3% 9% 3% 5%
Total 55% 12% 45% 16% 28% 57% 13% 43% 16% 29% 63% 9% 37% 8% 18%
N x Gender 21% 37% 23% 36% 15% 23%p (M/F) 1.24 1.31 1.68N (M/F) 0.57 0.63 0.64
AFIS204 (258) ECON104 (665) ECON105 (153)
Survey Response Numbers and Perecentages by Gender
Male FemaleMale Female Male Female
No significant diff @ 95% CI in Likert means or across Groups by Gender
What about grades?
Across the main units analysed:
Bivariate fits
Correlations for “Overall” Qtns
• Course R2 = 0.047
• Teaching R2 = 0.0045
A+ is 9 ………… E is -1
1
2
3
4
5
C_O
All
-1 0 1 2 3 4 5 6 7 8 9NumGrade
1
2
3
4
5
T_O
All
-1 0 1 2 3 4 5 6 7 8 9NumGrade
Broken into Groups
Survey timing now apparent!
Group 2 - post exams
Group 3 - post grades
Why not the full picture?
Week=W1 Distributions GradeGrp
A B C D
Frequencies Level Count Prob A 69 0.34673 B 64 0.32161 C 35 0.17588 D 31 0.15578 Total 199 1.00000
Week=Rem Distributions GradeGrp
A B C D
Frequencies Level Count Prob A 14 0.17722 B 23 0.29114 C 27 0.34177 D 15 0.18987 Total 79 1.00000
Where to from here?
Oct 6 - online surveys - 40 units
Group 2 runs until 15th Dec. Grades released 5th Dec
Pre-exam Post-exam Post-grades
Group 1 - Control 11487 Standard Course Survey - no demographic questions
Group 2 - hint3469 Standard Course Survey - no demographic questions
Group 3 - no hint 3 3RGroup 4 - no hint 4 4RGroup 5 - no hint 5R
1472 Course Survey plus demographic questions1626 Course Survey plus demographic questions1305 Course Survey plus demographic questions
M=5660, F=3699, T=9359
5
2