Post on 22-Jan-2016
description
transcript
Weighing Costs versus Benefits: Evaluating the Impact
From Implementing a Course Management System
Richard E. WestUniversity of Georgia (rickwest@uga.edu)
Greg Waddoups, Meghan Kennedy, and Charles R. Graham
Brigham Young University
Technology improves everything … right?
Calvin and Hobbes by Bill Watterson
CMS
CMS
Research Questions
• How has the implementation of a CMS impacted teaching/learning at the university?
• What are the benefits/challenges from supporting this tool on an institution-wide basis?
Research Context
• Average university enrollment each semester is 30,000 students
• 8,000 academic courses for both fall and winter semesters in 2004
• There were slightly more than 4,000 Blackboard courses created during each semester
• Over 300,000 students and faculty are estimated to have used Blackboard at least once
Methods
• Open-ended/closed-ended survey questions (n=124 instructors; 163 students) administered to students and instructors over two semesters
• An analysis of the calls reported to the Blackboard support center on campus (n=1,341)
• Semi-structured interviews with instructors (n=53) and brief intercept interviews with students (n=17)
• Data was analyzed inductively, looking for patterns within major categories
Findings — Satisfaction
Table 1. (Satisfaction) — How satisfied students are with Blackboard and how comfortable they feel using the tool. Survey Question Population SD D N A SA Blackboard is easy to learn/use. Fall 04 Students 4.2% 6.3% 14.6% 68.8% 6.3% Winter 05 Students 6.2% 6.2% 14.2% 55.8% 17.7% I prefer that instructors use Blackboard in their courses. Fall 04 Students 10.2% 4.1% 18.4% 36.7% 30.6% Winter 05 Students 6.3% 7.1% 20.5% 42.9% 23.2% Note. SD=Strongly Disagree, D=Disagree, N=Neither, A=Agree, SA=Strongly Agree.
Findings — Satisfaction
Findings: Satisfaction
What Blackboard features do you like using in your class?
0.010.020.030.040.050.060.070.080.090.0
100.0
AnnouncementsCourse DocsDisc. BoardsAssessments
Virt. ClassroomLghtwght ChatGradebook
Assign ManagerDig. Dropbox
Fall '04 Students Fall '04 Faculty Winter '05 Students
Conclusions: Satisfaction• When Blackboard works without technical
difficulties, most students and instructors at our university reported being satisfied with the tool because it is convenient for them to use, easy to learn, and helpful in their studies.
• Satisfaction and usage is high for only four features (announcements, email, posting of course documents, and gradebook). Satisfaction is low for synchronous chatting tools, discussion boards, assessments, and the digital dropbox
Findings — Stability
When Was The Last Time You Encountered a Technical Problem With Blackboard?
26.5
14.2
23.925.7
9.7
34.0
21.3 21.323.4
0.00.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
1-2 days ago Less than aweek ago
Less than amonth ago
More than amonth ago
Never
Students Faculty
Conclusions: Stability
• Blackboard has been unstable in some of its most critical areas
• Stability is more than a matter of convenience. It impacts instructor/student relationships and the instructional effectiveness of activities– Case Study A
Findings — Efficiency
On average, Blackboard saves me time in teaching/working on my classes.
0.0
10.0
20.0
30.0
40.0
50.0
60.0
StronglyDisagree
Disagree Neither Agree StronglyAgree
F '04 Students F '04 Faculty
W '05 Students W '05 Faculty
Conclusions: Efficiency
• Using a CMS seems to improve efficiency somewhat for instructors and students, especially in 4 ways:– distributing class materials– updating documents electronically– keeping the instructor and course
organized– providing quicker and more frequent
feedback to students
Conclusions: Efficiency
• “The potential for me is that I can change it once in one place and not have to pass out 30 paper.”
• “It allows (and makes) professors be more organized. For the class that doesn't use Blackboard, I am often confused at what material I should know, assignments, etc.”
• “It saves time at the end of the semester so you don't get a whole bunch of students saying, ‘C+, I thought I had a B.’”
However, some efficiency traps…• Homework creep• Unrealistic expectations• Instructor reliability• Encouragement to miss class• Using the entire CMS tool for one small
feature
Findings — LearningBlackboard improves student learning.
0.0
10.0
20.0
30.0
40.0
50.0
60.0
StronglyDisagree
Disagree Neither Agree StronglyAgree
F '04 Students F '04 Faculty
W '05 Students W '05 Faculty
Conclusions: Learning
• Many instructors feel the CMS either did nothing or improved their instruction
• Most instructors who felt it improved instruction were using it to support teacher-centered instruction– Access to materials– Improved notetaking during lectures– Just-in-time preparation for teaching– Saving in-class time for other activities
Conclusions: Teacher-centered learning• “I think one of the biggest things that helps the
learning thing is having my lectures online so they can pull them down and review or else they can get it if they miss the class, and that's huge.”
• “I think it's allowed a little more freedom in the classroom for students to ask questions and not feel they're going to miss something while they're writing it down.”
• “It's interesting because I can get through a lecture now 30% faster because I don't have to write it all out on the board. . . . I can get through the material so I can get to discussion and application, which is really nice.”
Findings: Student-centered learning• Many of the instructors who do not find
Blackboard to be a helpful tool are those instructors who prefer more hands-on, exploratory, or student-centered approaches to instruction. – Case Vignette 2
• These instructors often feel that Blackboard’s benefits do not outweigh the costs associated with using the tool
• These instructors are typically seeking out alternatives to Blackboard to fulfill their online instructional needs, such as blogs, wikis, personal websites, and instant messenging services
Why?• All feature controls are open only to
the teacher– Examples: Discussions, grading, posting
of documents, no outside guests, creating/adding modules
• Blackboard is not easily adaptable– “It is not flexible”
Overall Conclusions
• Do the benefits of a CMS outweigh its disadvantages?– It depends! What type of instruction do you
want to encourage? Are you satisfied with only four features, or will it only be worth the price tag if every feature is being used?
• Caution: are you getting “married” to the tool?– Even if they only use one or two features,
faculty can grow dependent on the tool, making a switch very difficult.
Future Questions• Can a CMS be built to support
constructivist, student-centered, problem-based, or other types of pedagogies?
• How can we help stakeholders properly evaluate a technology for its pedagogical strengths before purchasing it?
• If we have already purchased one that is not meeting our needs, how can we more easily maneuver towards other tools without isolating the instructors who now feel tied to the tool?
Case Studies
• “It looks like it works great, but there are glitches.”
• “It's really cumbersome.”