www.sakaiproject.org 1

Course Evaluation

Amitava ‘Babi’ Mitra, MIT

Maneesha Aggarwal & Robert Cartolano,

Columbia University

William Plymale and Aaron Zeckoski, Virginia Tech

www.sakaiproject.org 2

Agenda

• Introduction

• Review Common Ground

• Features at Columbia and Virginia Tech

• Next Steps

• Q & A

www.sakaiproject.org 3

Introduction• How did this begin ?

– Late Feb 05 began exploring who was using/planning to use online course eval within SEPP

– March 05, SEPP Course Eval Working Group set up

• What’s the WG been doing since then ?– Decided to use Columbia's existing feature set as a baseline to start with– CU and VT prepared a detailed functional layout and work flow, 

and posted it in April so that we can clearly show how the existing systems might operate. MIT, e.g., has studied this and responded with their own needs

– CU and VT have made sandboxes available

• Why are we doing this ?– Getting to the common ground - the baseline that most of us can agree

on

www.sakaiproject.org 4

What is Course Evaluation?• Administration of survey to students across one or more

courses– May span course, department, school, university– Hierarchy, permissions, access levels are important

• Provide results to University, School, Department, Faculty, Students to evaluate:– Instructors– Curriculum– Academic Program (Program Evaluation)– Department, School, University (accreditation)

www.sakaiproject.org 5

Course Evaluation

• Evaluation has historical aspects at each school• There are different motivations for evaluation• System must be flexible enough to deal with

different needs, but…• There are common features that we have

identified among our working group members.

www.sakaiproject.org 6

Common Features --- from the WG’s discussions• Anonymity - student privacy must be preserved• Evaluation administrator - someone with appropriate

authority must create and administer evaluation.• Course Selection - must be very flexible

– As little as 1 course…– Or the entire department, program, school, etc.

www.sakaiproject.org 7

More Common Features --- from the WG’s discussions

• Ability to combine core and specific questions– Course-specific, instructor-specific, etc.

• Requires extensive reporting capabilities– Administrator, Instructors, Students, Public accessible

• Requires workflow and automation to make it easy, reduce burden to administrators.

www.sakaiproject.org 8

Columbia - Current System– In production since Fall 2003– Built into CourseWorks, campus CMS– Flexible approach needed to support multiple

schools, multiple departments, etc.• Carrot vs. Stick• Open vs. Closed reports

• Many Reporting types; built-in reports and export data

www.sakaiproject.org 9

CU Current System (cont’d.) – E-mail tool very important; both auto and

manual– Aggregate data across semesters to support

accreditation review– Very popular, well-received by faculty,

students, and administrators• 87,825 evaluations administered to date• 54,480 evaluations completed• 157 evaluation templates created

www.sakaiproject.org 10

Columbia Live Demo

www.sakaiproject.org 11

--- Current System

www.sakaiproject.org 12

--- Current System

• Implemented - 1981• Different forms for different colleges

– Business, Architecture, Engineering

• Statistics– 500 class sections per term– 230,000 sheets per year

• Response Rate: 78% during past 3 years

www.sakaiproject.org 13

SEPP Contribution – VT / CU• Columbia University Partnership

– SEPP Conference – Summer, 2004 – Denver– Robert Cartolano / Maneesha Aggarwal

• Virginia Tech Development– Aaron Zeckoski– Kapil Ahuja / Justin Gawrilow

• SEPP Course Eval Discussion Group• VT / CU Evaluation System Summary• VT Evaluation System – “sandbox” release

– https://courseeval-dev.cc.vt.edu

www.sakaiproject.org 14

SEPP Contribution – VT / CU• Virginia Tech extensions

– Administrative hierarchy (Super, University, School, Dept, Instructor)

– Dynamic reporting (flexible formatting, trend analysis, sub-setting)

– Ability to incorporate existing data collected with VT’s current evaluation system.

– Oracle database tuning

www.sakaiproject.org 15

--- Pilot System

• VT Pilot release – May 2005– Columbia System with VT extensions – Very limited response to VT pilot study due to late

release.– VT pilot study will resume Fall 2005

www.sakaiproject.org 16

--- Pilot System• Faculty training will be incorporated into FDI

• Change to online evaluation system needs approval by Deans and Department Heads

• Online system will provide better access for Institutional Research and the administration

• Online system offers faculty flexibility in adding questions to their evaluations

www.sakaiproject.org 17

--- Demo

www.sakaiproject.org 18

--- Current scenario• Institutional subject evaluation evaluates between 650-700 subjects each

term, about 2/3 of which are undergraduate subjects and 1/3 graduate.• This is out of approximately 1700 subjects with students enrolled and 3000

total subjects offered for credit each term.• Evaluations are end-of-term only, on 2 paper forms separately for

Science/Engineering subjects and for Humanities, Arts, and Social Science subjects, as chosen by the department. Quantitative data are captured and analyzed, written comments are available to each department from originals.

• Three departments, Electrical Engineering and Computer Science, Management, and Aeronautics and Astronautics (online) have their own forms and systems for subject evaluations.

www.sakaiproject.org 19

Needs --- Must havesMust haves for a new system (to replicate current system):• Stand alone system (with the capability of being tied to the Course Management

System)• Anonymity, confidentiality, security• Overall institutional administrator, plus levels of school/departmental administrators• At least 2 different forms (1 for Science/Engineering subjects and 1 for Humanities, Arts,

and Social Sciences subjects) with preference for many• Some questions consistent across all schools with flexibility within school/dept./subject.

Flexible questions would need to be administered at the local level but available to all.• Rating both subject and instructors on the same form• Access to individualized reports for each person evaluated • Assigning roles to instructors (lecturer, recitation instructor, etc.)• Integration with other MIT systems for data downloads (student, subject, and instructor

information) and appropriate access (e.g., only to students registered in subject)• A 7 point rating scale, with flexibility to change that if needed• Blocking access to data/reports until after grades are turned in• Analysis and reporting at a centralized point• Student access to appropriate report information through a centralized site

www.sakaiproject.org 20

Needs --- NecessaryNecessary in any new system:• Searching data/reports on instructor name• Assigning rank/status to instructors (prof./assoc. prof./assist.

prof./lecturer/grad. student, etc.)• Individual and other reports that include overall subject, department, school,

and institutional data• Longitudinal data/reports• Analysis and reporting at local points in addition to centralized one, with

capability to print subject/department/school level with 1 click• Linking to evaluation results by department and by subject• Linking to evaluation results from subject listings/selection (i.e., our on-line

catalog)* Integration with course management system with necessary security• Having students go to a single site for all their subjects to be evaluated

www.sakaiproject.org 21

Needs --- Nice to haveNice to have capability for:• Evaluating sections of a subject (with the capacity for

students to enter instructor's name)• Instructors' photos together with their name• Mid-term evaluations• Evaluations that only faculty would have access to• Capturing, editing, and distributing students'

comments• Having a single email sent to a student listing all

subjects to be evaluated by that student

www.sakaiproject.org 22

What’s Next• Create a functional specification for ‘Course Eval

ver 1.0’ built upon Columbia base and Virginia Tech extensions

• Develop Java-based version based on Sakai framework

• Run as standalone or within Sakai• Consider using SAMigo as assessment engine,

but probably not in version Course Eval ver 1.0

www.sakaiproject.org 23

Open Discussion

• Questions?

• See Demonstrations at 5:30 today

• Join Evaluation Working Group– Contacts:

• Robert Cartolano rtc@columbia.edu

• Amitava ‘Babi’ Mitra babi@mit.edu