1

Evaluating Online Tutorials for Data Structures and

Algorithms CoursesJune 24, 2013

Evaluating Online Tutorials for Data Structures and

Algorithms CoursesJune 24, 2013

Simin Hall, PhD Mechanical EngineeringProf. Clifford A. Shaffer, Computer Science

Eric Fouh, Computer ScienceMai Hassan ElShehaly, Computer Science

Daniel Breakiron, Computer Science

ASEE Annual Conference & Exposition 2013

MotivationMotivation

• Low usage of learning technologies for teaching in CS (other than programming tasks)

• Low usage of Algorithm Visualizations (AVs) despite their educational usefulness

• Modest number of assessment activities with immediate feedback due to the lack of time for grading.

Features of OpenDSAFeatures of OpenDSA• A collection of online, open-source tutorials that

combine textbook-quality text with interactive examples and randomly generated instances of exercises

• Provides unlimited practice• Provides automatic assessment and immediate

feedback• Free and open source• Web-accessible• Interactive, engaging, dynamic material• Content is continually updated and improved

OpenDSAOpenDSA

• Modules authored in reStructuredText and compiled into HTML5

• Exercises written in HTML5, CSS and JavaScript• Extensive use of jQuery and the JavaScript

Algorithm Visualization (JSAV) library• Mastery-based design• Concept of “proficiency”

Exercise TypesExercise Types

• Slideshows• Mini-slideshows• Algorithm visualizations (AVs)

• Proficiency exercises• Algorithm simulations• Calculators

• Khan Academy-style exercises• Mini-proficiency exercises• Summary exercises

Mini-slideshowMini-slideshow

Sorting a sublist in Shellsort

http://algoviz.org/OpenDSA/dev/OpenDSA/Books/OpenDSA

Algorithm VisualizationAlgorithm Visualization

http://algoviz.org/OpenDSA/dev/OpenDSA/Books/OpenDSA

Algorithm SimulationAlgorithm Simulation

http://algoviz.org/OpenDSA/dev/OpenDSA/Books/OpenDSA

CalculatorCalculator

Birthday problem calculator

Mid-square calculator

Mini-Proficiency ExerciseMini-Proficiency Exercise

http://algoviz.org/OpenDSA/dev/OpenDSA/Books/OpenDSA

Summary ExerciseSummary Exercise

http://algoviz.org/OpenDSA/dev/OpenDSA/Books/OpenDSA

Research QuestionsResearch Questions• Can students learn as well or better using

interactive tutorials instead of traditional lecture and textbook?

• Will students be accepting of a class focused on interactive tutorials rather than traditional lecture and textbook?

• Will our client/server infrastructure adequately support classroom use?

• What feedback do students have about the best approaches for using such interactive tutorials in courses?

Research MethodResearch Method• Mixed method evaluation model (quasi-

experimentation) – Fall 2012• Control group: received standard lecture and

textbook for three weeks on the topics of sorting and hashing

• Treatment group: spent their class time working through equivalent content in the form of online material

Research Method, Cont’dResearch Method, Cont’dData collection • Surveys before and after the experiments• Interview• Classroom observations • Exam results and OpenDSA interaction logs

Population: • DSA students (55 and 57 students in the

control and treatment groups, respectively).

Results Results • Most students in treatment group reached the

maximum number of points awarded for OpenDSA homework (5% of total class score).

• Students in treatment group preferred to be lectured during class time instead of only doing assignments.

• Control group mean and median on test was 70, treatment group mean was 75 and median was 79 (not statistically significant).

Results, cont’d Results, cont’d

• Students in treatment group expressed a positive experience using OpenDSA.

• Instructor of treatment section reported being able to spend a greater fraction of lecture time on content related to the more abstract and difficult topics, and less on the mechanics of sorting and hashing algorithms.

Time Required for ProficiencyTime Required for Proficiency

Median time required for mini-slideshows

Time Required for ProficiencyTime Required for Proficiency

Median time required for algorithm visualizations (AVs)

Time Required for ProficiencyTime Required for Proficiency

Median time required for algorithm simulations

Median time required calculators

Time Required for ProficiencyTime Required for Proficiency

Median time required for Khan Academy-style mini-proficiency exercises

Time Required for ProficiencyTime Required for Proficiency

Median time required for Khan Academy-style summary exercises

Summary of our results Summary of our results

• The approach of online tutorials was well received by our students.

• Made us reassess our planned teaching strategy for the next deployment, to include more in-class lecture on the more difficult concepts to reinforce online materials.

• Gave us insight on how to improve our server-side logging and scoring infrastructure.

Summary of our results, cont’d Summary of our results, cont’d

• Gave us insight on what log data to collect for our next round of testing, and what tools we need to analyze it.

• Allowed us to perform fine-grained tuning of specific exercises in terms of things like making sure that students take the right amount of time to complete and adequately cover the material.

AcknowledgementsAcknowledgements• NSF• Grant DUE-1139861

• Virginia Tech• the Institute of Distance & Distributed Learning, • the Center for Instructional Development and

Educational Research, • the Institute for Society, Culture, and Environment

25

Questions?Questions?

• Project web site: http://algoviz.org/OpenDSA/dev/OpenDSA/Books/OpenDSA

• Cliff Shaffer shaffer@cs.vt.edu• Simin Hall simin.hall@vt.edu