The New Teaching Institute ProgramEvaluation Report

Lin ChengyanEric Whitmer

Anastasia Kalmykova

Agenda

• Introduction• Evaluation plan description• Description of Materials Evaluated• Data collection tools• Results

Introduction

Name of the Program: The New Teaching Institute.

Time range: November,2014 -- March,2015.

Purpose of the course: Incorporate the new technology into their teaching methods.

Participants: 11 Faculty & Professors.

Motivation for participants:

– Upgrade Teaching Skills and Methods

– Additional Reward.

Evaluation plan description

The objectives of the course:

• Generate a fully functional hybrid course shell including a variety of course materials designed to engage students in achieving the course goals

• Generate concise learning goals and objectives appropriate for a hybrid course

• Generate assessments appropriate for evaluating the stated course outcomes

• Explain the ADDIE model of instructional design and its applicability in a hybrid course

• Evaluate the effectiveness of other instructors’ hybrid courses

Evaluation plan description

Effectiveness of Course

Efficiency of Course

Usability of Course

Appeal of Course

Four main factors to evaluate the module

Description of Materials Evaluated

Accessed via Blackboard Start Here page

Blocks provide sense of community and social presence for learners

Blocks provide important information of the course

• Contact Information of the Course Facilitator

• Introductions • Cyber Café

• Navigating the Course• Syllabus• Course Overview• Netiquette • UT Hybrid Course Rubric

Description of Materials Evaluated

The module is organized in hierarchical structure represented in a form of tree. There are main sections menu of the module available on each page. Each section contains a set of topics with specific contents.

Data collection tools

• Entry Skills survey

• Usability survey

• Attitudes and Acceptance survey

• Course Review Rubric data

Tool Administration

• Surveymonkey

Data collection tools

Results of Evaluation of New Teaching Institute Course

How Effectiveness of NTI Module was Evaluated?

Conducted a paired samples t-test    Mean N

Entry skills testCourse review data

2.362.91

77

Course review data Entry skills test

3

2

1 Developed questions of the entry skills test based on the course objectives and course review rubrics.

What is the Result of NTI Module Effectiveness?

The course achieved its objectives with the intended audience

The objective “Evaluate the effectiveness of other instructors’ hybrid courses” was measured by indirect observation of work performance

How Efficiency of NTI Module was Evaluated?

Course review data

Calendar

What is the Result of NTI Module Efficiency?

How Usability of NTI Module was Evaluated?

• Developed survey questions• Conduct the survey• Analyze the results

What is the Result of NTI Module Usability?

Question Unsatisfactory Acceptable Excellent1. Comfortable to Use X XX X

2. Navigation works, easy to use, & consistent

XXX X

3. Clear Organization XXX X

4. Practice Opportunities Appropriate XXX X

5. Expected Functions & Capabilities X XX X

6. Easily Accessible in the event accidental termination

XX X

7. Overall Satisfaction X XX X

How Appeal of NTI Module was Evaluated?

• Developed survey questions• Conduct the survey• Analyze the results

NTI Attitudes & Acceptance

Welcome to the NTI Attitudes & Acceptance Survey

What is the Result of NTI Module Appeal?

Question Strongly Disagree

Disagree Undecided Agree Strongly Agree

1. Maintains Interest X XX X

2. Helps Generate Fully Functional Hybrid Course

XX XX

3. Assignments Effective & Assist in Learning

X XXX

4. Course Follows Objectives

X XX X

5. Helps to Evaluate Other Instructors' Hybrid Courses

X XXX

Part1

What is the Result of NTI Module Appeal?

Question Strongly Disagree

Disagree Undecided Agree Strongly Agree

6. Provided Proper Tools to Generate Assessments for Evaluating Course Outcomes

XX XX

7. Learner was able to Generate Concise Learning Goals & Objectives for a Hybrid Course

X X X X

8. ADDIE Model Effective in Delivering Content

XX X X

9. Course has been Beneficial XX X X

10. Course is Challenging X X X X

Part2

Pattern of Like or Dislike

Expert experience

Boring

Entry skills survey Appeal survey

No experience Interesting

Recommendations

• Conduct the regular and systematic evaluation– Develop and conduct pre/post tests– Conduct entry skills survey– Conduct usability survey– Conduct attitudes and acceptance survey

• Make subsequent changes based on the resultsif following evaluation will indicate that the majority of learners have high level of entry skills we recommend:

– Add more challenging practice items– Reduce time to complete courseif following evaluation will indicate that the majority of learners have problem finding information we recommended:

– Make navigation simpler

Thank you.

Questions?