Student Satisfaction Student Satisfaction with Online Courseswith Online Courses
Amber Settle, CTI, DePaul Amber Settle, CTI, DePaul UniversityUniversity
joint work with Chad Settle, University joint work with Chad Settle, University of Tulsaof Tulsa
CCSC: Southeast RegionCCSC: Southeast RegionNovember 11, 2005November 11, 2005
Satisfaction with distance Satisfaction with distance learninglearning
Distance learning is popular with CTI studentsDistance learning is popular with CTI students There are 8 M.S. and 1 M.A. degree online (out of There are 8 M.S. and 1 M.A. degree online (out of
10 M.S., 1 M.A., and multiple joint degrees)10 M.S., 1 M.A., and multiple joint degrees) Distance learning students are 21% of the student Distance learning students are 21% of the student
populationpopulation It has been asserted that while outcomes are It has been asserted that while outcomes are
similar in DL and traditional classes, DL similar in DL and traditional classes, DL classes are less satisfying to students (Carr classes are less satisfying to students (Carr 2000)2000)
Is DL less satisfying for CTI students? If so, Is DL less satisfying for CTI students? If so, how?how?
The test casesThe test cases The coursesThe courses
Foundations of Computer Science (CSC 415)Foundations of Computer Science (CSC 415) Graduate discrete mathematics Graduate discrete mathematics 9 sections between Fall 2001 and Fall 20039 sections between Fall 2001 and Fall 2003
Programming in Java I and II (CSC 211 and CSC 212)Programming in Java I and II (CSC 211 and CSC 212) Java I: 8 sections between Fall 2003 and Winter 2005Java I: 8 sections between Fall 2003 and Winter 2005 Java II: 7 sections between Spring 2003 and Fall 2004Java II: 7 sections between Spring 2003 and Fall 2004
The formatThe format TraditionalTraditional Sibling DL: Runs parallel to a traditional class; entire Sibling DL: Runs parallel to a traditional class; entire
classroom interaction is recorded automaticallyclassroom interaction is recorded automatically Pre-recorded DL: High production quality Pre-recorded DL: High production quality
independent of any live class; broken into five independent of any live class; broken into five modules (CSC 415 only)modules (CSC 415 only)
Course evaluationsCourse evaluations Conducted every quarter for every CTI courseConducted every quarter for every CTI course
Mandatory for all studentsMandatory for all students Online using secure login; anonymousOnline using secure login; anonymous Completed during the 8Completed during the 8thth and 9 and 9thth week of 10 week week of 10 week
quarterquarter Results are withheld from instructor until grades are Results are withheld from instructor until grades are
submitted; results are then published on the CTI web submitted; results are then published on the CTI web sitesite
Consists of 22 multiple choice questionsConsists of 22 multiple choice questions 10 questions about course-related factors; 12 questions 10 questions about course-related factors; 12 questions
about instructor-related factorsabout instructor-related factors Ratings on a scale from 0 to 10; a higher number Ratings on a scale from 0 to 10; a higher number
indicates greater satisfaction; 0 indicates the question is indicates greater satisfaction; 0 indicates the question is not applicablenot applicable
Summary of CSC 415 Summary of CSC 415 resultsresults
Similarities in evaluations: Overall scoresSimilarities in evaluations: Overall scores None of the coefficients for instructor-related None of the coefficients for instructor-related
or course-related questions were significantly or course-related questions were significantly different from 0 for DL sectionsdifferent from 0 for DL sections
Differences in evaluations: Not applicable Differences in evaluations: Not applicable response rateresponse rate Q-IR5 (Encourage participation)Q-IR5 (Encourage participation)
DL: 60%DL: 60% Traditional: 6%Traditional: 6%
Q-IR12 (Teaching effectiveness)Q-IR12 (Teaching effectiveness) DL: 15%DL: 15% Traditional: 2.5%Traditional: 2.5%
Potential explanationsPotential explanations Pre-recorded DL is better organized Pre-recorded DL is better organized
which compensates for the lack of which compensates for the lack of interaction (Swan 2001)interaction (Swan 2001)
DL students are not watching the DL students are not watching the recordingsrecordings
Small DL sample sizeSmall DL sample size
Summary of Java resultsSummary of Java results Course-related questionsCourse-related questions
Two out of 10 questions (Q-CR1: Course organization, Q-CR2: Two out of 10 questions (Q-CR1: Course organization, Q-CR2: Achieving course objectives) were statistically different from 0 for DL Achieving course objectives) were statistically different from 0 for DL studentsstudents
Both coefficients were negative, indicating less satisfactionBoth coefficients were negative, indicating less satisfaction Similar results were not seen in an analysis of only the live sectionsSimilar results were not seen in an analysis of only the live sections
Instructor-related questionsInstructor-related questions Nine out of 12 questions (all but Q-IR4, Q-IR7, Q-IR10) were Nine out of 12 questions (all but Q-IR4, Q-IR7, Q-IR10) were
statistically different from 0 for DL studentsstatistically different from 0 for DL students All coefficients on the 9 questions were negative (from -0.3 to -0.8)All coefficients on the 9 questions were negative (from -0.3 to -0.8) For live students only two questions were significant over time; one For live students only two questions were significant over time; one
was positive (Q-IR 5: student participation), the other negative (Q-IR 6: was positive (Q-IR 5: student participation), the other negative (Q-IR 6: availability)availability)
Sibling DL students are not as satisfied with the instructor as Sibling DL students are not as satisfied with the instructor as traditional students or students in live sibling sectionstraditional students or students in live sibling sections
Q-IR 12 (comparison to other instructors) is used for T/P and meritQ-IR 12 (comparison to other instructors) is used for T/P and merit Not applicable rate was fairly consistent between sections; only Not applicable rate was fairly consistent between sections; only
differences on Q-IR5 (7% for DL, < 1% for live) and Q-IR7 (6% for DL, differences on Q-IR5 (7% for DL, < 1% for live) and Q-IR7 (6% for DL, < 1% for live)< 1% for live)
Potential explanationsPotential explanations Larger data setLarger data set DL students are more sensitive to course DL students are more sensitive to course
organization because of the lack of interactionorganization because of the lack of interaction A more interactive sibling course will highlight A more interactive sibling course will highlight
the shortcomings of the class for DL studentsthe shortcomings of the class for DL students Watching the students in the sibling section Watching the students in the sibling section
participateparticipate Listening to instructor questions with no chance of Listening to instructor questions with no chance of
respondingresponding The course evaluations do not allow evaluation The course evaluations do not allow evaluation
of the course delivery technology, causing the of the course delivery technology, causing the instructor ratings to sufferinstructor ratings to suffer
AppendixAppendix Course evaluation questionsCourse evaluation questions
Course-related questionsCourse-related questions Instructor-related questionsInstructor-related questions
Statistical analysisStatistical analysis Least squares regressionLeast squares regression Data for CSC 415Data for CSC 415 Data for Java I and IIData for Java I and II
Course-related questionsCourse-related questions1.1. Was this course well organized?Was this course well organized?2.2. Do you feel the course objectives were Do you feel the course objectives were
accomplished?accomplished?3.3. The amount of work you performed outside of this The amount of work you performed outside of this
course was:course was:4.4. How difficult was this course material?How difficult was this course material?5.5. The textbook for this course was:The textbook for this course was:6.6. Supplementary reading for this course was:Supplementary reading for this course was:7.7. The assignments for this course were:The assignments for this course were:8.8. What is your overall estimate of this course?What is your overall estimate of this course?9.9. How valuable was this course in terms in your How valuable was this course in terms in your
technical development?technical development?10.10. Would you recommend this course to another Would you recommend this course to another
student?student?
Instructor-related Instructor-related questionsquestions
1.1. How would you characterize the instructor’s How would you characterize the instructor’s knowledge of this subject?knowledge of this subject?
2.2. How would you characterize the instructor’s How would you characterize the instructor’s ability to present and explain the material?ability to present and explain the material?
3.3. Does the instructor motivate student Does the instructor motivate student interest in the subject?interest in the subject?
4.4. How well does the instructor relate the How well does the instructor relate the course material to other fields?course material to other fields?
5.5. Did the instructor encourage participation Did the instructor encourage participation from the students?from the students?
6.6. Was the instructor accessible outside of Was the instructor accessible outside of class?class?
Instructor-related Instructor-related questionsquestions
continuedcontinued7.7. What was the instructor’s attitude? How did What was the instructor’s attitude? How did
he/she deal with you?he/she deal with you?8.8. How well did the instructor conduct, plan, and How well did the instructor conduct, plan, and
organize classes?organize classes?9.9. Were the instructor’s teaching methods Were the instructor’s teaching methods
effective?effective?10.10. How fair was the grading of the homework and How fair was the grading of the homework and
exams of this course?exams of this course?11.11. Would you take this instructor for another Would you take this instructor for another
course?course?12.12. Rate the teaching effectiveness of this instructor Rate the teaching effectiveness of this instructor
as compared to other faculty in the department.as compared to other faculty in the department.
Statistical analysisStatistical analysis Ordinary least squares regression: Ordinary least squares regression:
XX2i2i = 0 for traditional, X = 0 for traditional, X2i2i = 1 for DL = 1 for DL If If 22 is statistically different from 0, it is statistically different from 0, it
indicates a difference in how DL students view indicates a difference in how DL students view the course vs. traditional studentsthe course vs. traditional students
QQii = = 00 + + 11 X X1i1i + + 22XX2i2i + u + uii
Question iConstant Time DL Error
CSC 415: Course-related CSC 415: Course-related resultsresults
Question Time DL Question Time DLQ-CR1 -0.164**
(0.075)0.200
(0.393)Q-CR6 0.046
(0.164)0.302
(0.910)
Q-CR2 -0.120(0.079)
-0.023(0.416)
Q-CR7 -0.071(0.101)
0.237(0.533)
Q-CR3 0.064(0.086)
0.221(0.460)
Q-CR8 -0.179**(0.089)
0.594(0.467)
Q-CR4 0.305***(0.102)
-0.549(0.530)
Q-CR9 -0.199*(0.110)
0.620(0.573)
Q-CR5 0.185(0.144)
-0.261(0.758)
Q-CR10 -0.202**(0.092)
0.067(0.486)
Coefficient estimates are presented with standard errors in parentheses.*Statistically significant at the 10% level of a two-tailed test.**Statistically significant at the 5% level on a two-tailed test.***Statistically significant at the 1% level of a two-tailed test.
CSC 415: Instructor-related CSC 415: Instructor-related resultsresults
Question Time DL Question Time DLQ-IR1 -0.010
(0.072)0.065
(0.379)Q-IR7 -0.115
(0.084)0.013
(0.437)
Q-IR2 -0.069(0.069)
-0.029(0.360)
Q-IR8 -0.110(0.084)
0.042(0.442)
Q-IR3 -0.194**(0.083)
-0.016(0.437)
Q-IR9 -0.181**(0.088)
0.570(0.454)
Q-IR4 -0.189*(0.098)
0.043(0.515)
Q-IR10 0.010(0.087)
0.304(0.451)
Q-IR5 -0.267**(0.102)
0.481(0.654)
Q-IR11 -0.096(0.089)
0.411(0.461)
Q-IR6 -0.053(0.090)
-0.183(0.468)
Q-IR12 -0.114(0.082)
0.268(0.443)
Coefficient estimates are presented with standard errors in parentheses.*Statistically significant at the 10% level on a two-tailed test. **Statistically significant at the 5% level on a two-tailed test.
Java: Course-related Java: Course-related resultsresults
Question Time DL Question Time DLQ-CR1 -0.034
(0.040)-0.736***
(0.226)Q-CR6 -0.130**
(0.063)-0.524
(0.367)
Q-CR2 -0.038(0.037)
-0.431**(0.209)
Q-CR7 -0.047(0.044)
-0.037(0.252)
Q-CR3 -0.035(0.037)
-0.013(0.210)
Q-CR8 -0.017(0.043)
0.083(0.243)
Q-CR4 -0.096**(0.046)
-0.149(0.261)
Q-CR9 -0.068(0.049)
0.129(0.278)
Q-CR5 -0.114*(0.059)
0.357(0.333)
Q-CR10 0.023(0.051)
-0.425(0.286)
Coefficient estimates are presented with standard errors in parentheses.*Statistically significant at the 10% level of a two-tailed test.**Statistically significant at the 5% level on a two-tailed test.***Statistically significant at the 1% level of a two-tailed test.
Java: Instructor-related Java: Instructor-related resultsresultsQuestion Time DL Question Time DL
Q-IR1 0.017(0.028)
-0.394***(0.157)
Q-IR7 -0.018(0.044)
-0.304(0.252)
Q-IR2 0.023(0.042)
-0.648***(0.239)
Q-IR8 -0.009(0.039)
-0.874***(0.222)
Q-IR3 -0.019(0.048)
-0.622**(0.275)
Q-IR9 0.051(0.048)
-0.754***(0.275)
Q-IR4 -0.026(0.057)
-0.143(0.317)
Q-IR10 -0.069(0.042)
-0.143(0.239)
Q-IR5 0.075*(0.045)
-0.718***(0.257)
Q-IR11 0.034(0.055)
-0.770**(0.313)
Q-IR6 -0.071*(0.038)
-0.375*(0.217)
Q-IR12 0.010(0.044)
-0.635**(0.254)
Coefficient estimates are presented with standard errors in parentheses.*Statistically significant at the 10% level of a two-tailed test.**Statistically significant at the 5% level on a two-tailed test.***Statistically significant at the 1% level of a two-tailed test.
