Labrador: A Tool for Automated Grading Support in Multi-Section Courses
Drexel University ProgrammingLearning Experience (DUPLEX)http://duplex.mcs.drexel.edu
Christopher D. Cera, Robert N. Lass, Bruce Char, Jeffrey L. Popyack, Nira Herrmann, Paul Zoski
Drexel UniversityMathematics and Computer Science
2
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
3
The Group
Bruce Char– Professor, Computer Science
Nira Herrmann– Professor and Head, Mathematics
Jeffrey L. Popyack– Associate Professor, Computer Science
Paul Zoski– Instructor, Math and Computer Science
Christopher D. Cera– Computer Science Graduate Student
Robert N. Lass– Computer Science Undergraduate Student
4
Who Am I
First TA in MCS to experiment with WebCT in December 2000
Courses I have TA’ed using WebCT– 3 x Introductory Programming I, II– 1 x Object Oriented Programming
Migrated course content to WebCT from previous course website– HTML assignments and labs into question database– Gradebook maintainer
Wrote demo’s and documentation to train other TAs
Developer of software supplements to WebCT to support course administration
5
The Course
Intro Programming I, II
Variety of Majors – Computer Science– Computer
Engineering– Information Systems– Mathematics– Digital Media
Class– 1 x 1 hour lecture– 2 x 1 hour labs
People– 250-300 Students– 2-3 Professors– 10-12 Teaching
Assistants
6
The Duplex Project: An Overview
Take advantage of advances in Information Technology to improve instruction and reduce costs for computer programming courses
– Modular Structure Multiple Entry Points Multiple Audiences Multiple levels of knowledge (Bloom’s Taxonomy)
– Computer Supported Cooperative Work (CSCW) in student labs
– Online Services – Today’s Topic
7
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
8
Before WebCT
Hundreds of people involved in paper exchanges of handwritten assignments and quizzes
Testing programs required floppy exchanges No Chat No newsgroup-style threads Feedback and grades are not online
9
Before WebCT
Large Classes
– Managing several hundred students is difficult due to the logistics involved with organizing paperwork and people.
10
WebCT
Service from Information Resources & Technology (IRT) department (v3.1)
MCS started Dec. 2000 (v3.5)
Upgraded to v3.7 recently
Transition to v3.8 planned in Fall
11
Noteworthy Features
General Course Website
Labs: Online Quizzes with Automated Grading
Homework: Online Assignments
Joint Staff--Student Chat
Discussion Threads
13
Software Design Goals
Bulk Assignment Downloading (prior to 3.7)
Bulk Quiz Downloading
Section Sorting for bulk downloads, or only one section
14
Software Design Goals
Post-Processing student files– Archive Extraction (tar, zip)– Decompress (gz, zip)– Decode (uue)
Minimal staff intervention when transferring submissions between systems, eg. Plagiarism Detection Systems (PDS)
Automatically collate source code
Generate electronic documents to facilitate grading and archiving
Remote Execution– downloading to computer x (on
campus) while operating at computer y (off campus)
15
The Bigger Picture
Course-Specific Tasks– Not all processing should be done on the server
Select files have to be transferred to a different system for further processing and analysis by staff
Client-side support is needed to perform this, preferably automated and not necessarily using a web browser.
16
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
17
Labrador: Our Solution
Client-side WebCT Supplement
Implemented in Perl
Works for users with TA and Designer access to WebCT
18
Perl
Practical Extraction and Reporting Language
Created in 1987 to replicate Unix shell functionality
Powerful text manipulation
High-level: rapid prototyping and development
Large library of modules and active development community
Perl is used by many applications, including WebCT
19
A Taste of Labrador Perl Programming
20
Labrador works on Windows, Mac, Linux, Other Unix Variants, etc.
Perl has been ported to all major operating systems
Perl programs can be compiled:
– No need for the end user to install Perl
– Perl2exe for Windows support (www.perl2exe.com)
21
User Interface
Different users / Different UI preferences
Command-line
Interactive
Configuration File
GUI (in development)
22
Required Info
http://webct.drexel.edu/SCRIPT/CS164_Fall2002/scripts/designer/dropbox_edit.pl?DROPBOX_ASSN_VIEW+_side_nav++1006285909
Username/Password
Server URL webct.drexel.edu
Course ID CS164_Fall2001
Submission Name or ID 1006285909
Optional Username List
23
Components
SubmissionDownloader
SectionSorting
PostProcessing
Moss
JPlag
PDFGenerator
24
Bulk Downloader
Web-Crawler: simulates clicks of actual staff member
– Downloads actual web content (HTML)
– Parses HTML to find desired text or URLs to crawl to next
– Uses HTTP operations
Works on assignments and quizzes
Only component in Labrador which interacts with WebCT
25
Organizing Submissions by Section
Not supported in original Labrador prototype, added later on.
Organizes files into directories
3 possible input methods:– Creating a Section column in
gradebook– “Username, Section” CSV File– Username file
26
PDF Generation for Electronic Mark-up
Adobe Portable Document Format can be viewed on all major platforms
With Adobe Acrobat, PDFs can be annotated by graders
Sony VAIO Slimtop PC (PCV-LX920)
Wacom Pen Tablet.
27
VAIO / Wacom
28
PDF Markup Example
29
Redistribution
How can we return annotated PDF’s back to students using WebCT?
WebCT Mail
WebCT Groups
Version 3.8 now addresses this issue
30
Heterogeneous Systems
Different systems want different formats
Plagiarism Detection Systems– Moss has been used most extensively– Began experimenting with JPlag more recently
Future work: Automatic program compiling and testing
31
Automated Plagiarism Detection
Digital formats make "borrowing" easy
– Browsing similar works needs a simple and quick user interface.
– Careful review by faculty to assess results and present to students
32
Moss
Processes C, C++, Java, ML, Lisp, Scheme, Pascal, and Ada programs.
Common code feature reduces false positives
33
Moss Interface
34
JPlag
Processes C, C++, Scheme, and Java programs
For plain text files, it matches a user specified number of words appearing in succession
Could be used for any course grading written (text) documents
35
Command Line Invocation
36
Interactive
37
Quiz Download with Config File
38
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
39
Recent WebCT Enhancements
Relevant to this talk:
– 3.7 Addressed bulk download issue for assignments
– 3.8 Attaching documents to an assignment
40
Future WebCT Enhancements
Power users will need functionality not yet supported
Every domain will also require additional functionality
Not feasible for all domain-specific functionality to run on the WebCT server
41
Wanted
API for non-administrators
Stateful protocol so clients can be built by 3rd parties
42
HTTP: Insufficient Protocol
Inconvenient content interchange Heavy client interaction
A HTTP based approach will be sensitive to the exact location of web pages, and format of text within them.
43
Labrador Availability
Contact Us
http://duplex.mcs.drexel.edu
44
Project Support
The Pew Learning and Technology Program at the Center for Academic Transformation
National Science Foundation, Division of Undergraduate Education, DUE-#0089009
The Ramsey-McCluskey Family Foundation, Margaret Ramsey, '84
Drexel University
45
References
[1] Alex Aiken. Moss: A system for detecting software plagiarism (unpublished), http://www.cs.berkeley.edu/~aiken/moss.html.
[2] Jeffrey L Popyack, Bruce Char, Paul Zoski, Nira Herrmann, and Christopher D. Cera. Managing course management systems. In Proceedings of the thirty-third SIGCSE technical symposium on Computer Science Education, Birds-of-a-Feather Sessions, page 423, 2002.
[3] L. Prechelt, G. Malpohl, and M. Philippsen. Jplag: Finding plagiarisms among a set of programs. Technical Report 2000-1, Fakultat fur Informatik, Universitat Karlsruhe, Germany, March 2000.
[4] Larry Wall, Tom Christiansen, and Jon Orwant. Programming with Perl. O’Reilly and Associates, 3rd edition, 2000.
[5] Michael J. Wise. Yap3: Improved detection of similarities in computer program and other texts. In Proceedings of the twenty-seventh SIGCSE technical symposium on Computer Science Education, pages 130–134. ACM Press, 1996.
46
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
Bonus Slides
47
Blooms Taxonomy
Knowledge– remembering of previously learned material; recall (facts or whole theories); bringing to mind.
Comprehension– grasping the meaning of material; interpreting (explaining or summarizing); predicting outcome and
effects (estimating future trends). Application
– ability to use learned material in a new situation; apply rules, laws, methods, theories. Analysis
– breaking down into parts; understanding organization, clarifying, concluding. Synthesis
– ability to put parts together to form a new whole; unique communication; set of abstract relations. Evaluation
– Ability to judge value for purpose; base on criteria; support judgment with reason. (No guessing).