Measuring Program Objectives Via Online Assessment

Julia Cronin-Gilmore, Ph.DTracy Gies

Bellevue University

International Assembly for Collegiate Business Education

About your Presenters…Julia

About your Presenters…Tracy

We are from…

Some think of it as…

Some think of it as…

Gates - $81.6 Billion, Buffett $74.8 Billion

Some even think of it as…

Background• New Degree, M.S., Marketing• 12 Courses

Create Program Objectives• Demonstrate problem solving and critical

thinking skills in order to be successful in the business world

• effective communication skills and professionalism when working with others in a business setting.

• Demonstrate effective interpersonal skills and an appreciation of business ethics.

• Demonstrate the ability to apply content knowledge to create value as business leaders in an organization.

Two Direct & Two Indirect Measures

Direct measures:Entrance & Exit Exams: 80% of students will score at least 73% or greater, which is at a level of proficiency.

Two Direct & Two Indirect Measures

Indirect Measures:Mid-term and End of Program Surveys: At least 80% of the students' will note the ability to apply program learning outcomes as noted by their responses of "Agree" or "Strongly Agree".

Measure Program ObjectivesTwo Direct Measures• In other programs, we used a

business plan and a group of professors would evaluate their specialty area (marketing, finance,…).

• Found to be a time consuming process.

• Needed something manageable for one person

• Decided to try analyzing a case study

Case AnalysisFour Cases Total• All four are equal in length – about ¾

page.• Each has multiple things going on

Three Central Questions• What are the relevant facts of the case?• What are the ethical issues?• What are the ethical alternatives?

Exam #1 ResultsStudents didn’t score so well…

12.21 16.59 23.65

30 Possible Points

Test ResultsTest Results were Skewed…• Students are used to answering once

on a multiple choice exam.• There were several answers that were

correct.• If students answered correctly, they

received a point• If students missed a correct answer, it

deducted points from ones they earned.

Going Forward• Questions reflect how many correct

answers are possible.• Students will earn one point per

correct answer• Students will still loose points earned

for incorrect answers (couldn’t fix in the system).

Test ResultsTest Results were Skewed…• Exam #1 was extra credit• Eliminated the issue with it being a

major portion of their grade or• Something I had to prepare them for.• Most were encouraged to take• Not every student completed the

exam.

Measuring Program Objectives Via Online Assessment

Part IISetting It Up in the LMS

Agenda

• Why use cases online?• Mapping• Creating the tests in the LMS

Why Use Cases

• Authenticity (Suskie, 2009)

• Effectiveness and student acceptance of CBL (Barnes et al, 1994; Srinivasan et al, 2007)

• Program curriculum favors CBL learning

• CBL in assessment good tie-in

Issues

• Grading responses to cases can be very time consuming

• There can be a wide range of correct answers

• Setting up auto graded tests that capture a wide range of correct answers can be difficult

Our Approach

• Use Multiple Answer questions• Try to capture as broad of a range as

possible of correct answers in terms of ethical considerations, and resolutions

What We’ve Learned So Far

• Generally well received by students• Refine instructions

Mapping

• Provides a common document to prompt discussion for all parties involved in course design, development and review

• Keeps the two main components of learning in a program (program outcomes and learning objectives) a the forefront of design and development discussions

• Helps ensure alignment of program outcomes and course objectives

How to use the Excel Mapping Tool

• Some cells are linked so you can enter info on the Data Entry worksheet, and it will fill the appropriate cells on the other worksheets.

• Demo

Setting up the Test

• Blackboard 9.1.201404.160205• Fairly standard Multiple Answer set

up• Partial Credit• Group cases and MC questions

Set Up in Blackboard

Questions?