Using IDEA for Faculty Evaluation

Shelley A. Chapman, PhDTexas A & M University February 2013

Plan for this Session “Teaching Effectiveness” What it is Uniqueness of IDEA Conditions for the Good Use of IDEA 3-Phase Process for Faculty

Evaluation Using Reports to Improve Teaching

Most SurveysHow well do the instructor’s methods resemble those of a “model” teacher?

How well do students rate their progress on the types of learning the instructor targeted?

Teaching Effectiveness

What makes IDEA unique?1. Focus on Student

Learning 2. Focus on Instructor’s

Purpose3. Adjustments for

Extraneous Influences4. Validity and Reliability5. Comparison Data6. Flexibility

Conditions for Good Use

The instrument o Focuses on learning o Provides suggested

action steps

Conditions for Good Use

The Facultyo Trust the processo Value student

feedbacko Are motivated to

make improvements

Conditions for Good Use

Campus Cultureo Teaching excellence - high

priorityo Resources to improve -

providedo Student ratings -

appropriate weight

Conditions for Good Use

The Evaluation Processo 30-50% of evaluation of teachingo 6-8 classes, more if small (<10)oNot over-interpreted

(3-5 performance categories)

Underlying Philosophy of IDEA

Teaching effectiveness is determined primarily by students’ progress on the types of learning the instructor targets.

Faculty Information Form

Diagnostic Report Overview

1. How did students rate their learning experience?

2. What contextual factors impacted those ratings?

3. How do my scores compare to: IDEA, discipline, and institution?

4. What might I do to facilitate better learning for my students next time?

Wisdom

Knowledge

InformationData

What the Report Can Provide

Calculation of Scores

Context: Variables and Comparisons

Suggested Action Steps

Using IDEA

As Part of a Faculty Evaluation Process

Multiple forms of Assessment

Student Ratings

External PerspectiveArtifacts

BalancedPlan for SummativeEvaluation

Evidence of Good Teaching

Artifacts

• Syllabi• Graphic Organizers• Assignments and project

descriptions• Rubrics• Written Teaching

Philosophy/Reflections• Samples of Student Work• CATs and results

Evidence of Good Teaching

• Classroom Observation• Classroom Visitation• Invited Presentations• Alumni Surveys• Focus Groups of

Graduating Students

External Perspectiv

e

Classroom ObservationsTime What Happened What Was Said

Classroom ObservationsTime What Happened What Was Said

8:05

8:10

8:15

Instructor shut door

Students are shuffling papers, opening books.

Student comes in late

Several students raise hands

Female in first row is called on

Instructor (I): OK, Class. Let’s begin. Make sure you turned in your homework as you came in.

Today we will begin our discussion on the brain. Turn in your textbooks to chapter 5.

Is your brain more like a computer or a jungle? Who would like to respond first?

Student (S) My brain is a jungle! I am so unorganized! (class laughs)…

Flow of Communication MapInstructor

M

F

F

F

M M M M M

M M

M

F F F F

F

F

F

F F

Evidence of Good Teaching

• Administer Appropriately• Collect 6-8 Reports (more if class size is <10)• 30-50% of Overall Evaluation

Student

Ratings

Evidence of Good Teaching

• Student Comments-formative • Be mindful of standard error of

measurement (±.3)• Use 3-5 Performance Categories

Student

Ratings

Three-Phase Process for Faculty Evaluation

Set Expectations

Collect Data

Use Data

Three-Phase Process for Faculty Evaluation

Set Expectations

Collect Data

Use Data

I. Set ExpectationsWhat does this entail regarding IDEA?

Criterion-Referenced Standards

Criteriono Use averages on 5-

point scaleo Recognize that

some objectives are more difficult to achieve

o “Authenticate” objectives

Page 1

Norm-Referenced Standards

Use Converted Averages

o IDEAo Disciplineo Institutio

n

Comparison Information: Converted Averages

T-Score Distribution

40%

Similar

Lower20%10%

Much Lower

10%Much Higher

20%Higher

Gray Band

Comparison Scores

Create Categories of Performance

Below Acceptable Standards

Marginal, Needs Improvement

MeetsExpectations

ExceedsExpectations

Outstanding

Does Not Meet Expectations Meets Expectations Exceeds Expectations

Performance Categories: EXAMPLECriterionAverage Rating

Effectiveness Category

NormativeT-Score

Below 3.0 Below acceptable standards Below 38

3.0-3.4 Marginal, improvement needed

38-44

3.5-3.9 Meets expectations 45-54

4.0-4.4 Exceeds expectations 55-62

4.5 or higher Outstanding 63 or higher

Three-Phase Process for Faculty Evaluation

Set Expectations

Collect Data

Use Data

II. Collect DataWhat do you look for regarding IDEA?

Improving Online Response Rates

• Create value for student feedback

• Monitor and Communicate through multiple modalities:• Twitter• Facebook• Other

• Prepare Students• Talk about it• Syllabus

Example: Course Syllabus

Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions)

Students will be able to apply the methods, processes, and principles of earth science to understanding natural phenomenaStudents will think more critically about the earth and environment

Objective 8: Developing skill in expressing myself orally or in writing

Students will be able to present scientific results in written and oral forms

IDEA Center Learning Objective

Course Learning Outcomes

For Personnel Decisions

Pages 1 and 2 What were students’ perceptions of the

course and their learning?

Things toConsider…

Were the appropriate objectives selected?

• How many?• Do they match the course?

• How might you “authenticate” the objectives selected?

FIF: Selecting Objectives• 3-5 as “Essential” or

“Important”o Is it a significant part of the

course?o Do you do something

specific to help students accomplish the objective?

o Does the student’s progress on the objective influence his or her grade?Be true to your course.

Things to Consider…

What were the students’ perceptions of their course and their learning?

How Did Students Rate Their Learning?

Your Average (5-point Scale)

Raw Adj.

A. Progress on Relevant Objectives1

Four objectives were selected as relevant (Important or Essential—see page 2)

4.1 4.3

1If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.

How Did Students Rate Their Learning?

ProgressOnRelevantObjectives

4

4.3 + 4.34.14.23.6 5

Summary Evaluation: Five-Point Scale

Report Page 1

Your Average Score

(5-point scale)

Raw Adj.

A. Progress on Relevant ObjectivesFour objectives were selected as relevant (Important or Essential—see page 2)

4.1 4.3

Overall Ratings B. Excellent Teacher 4.7 4.9 C. Excellent Course 4.1 4.4

D. Average of B & C 4.4 4.7

Summary Evaluation(Average of A & D) 4.3 4.5

50%

25%25%

Understanding Adjusted Scores

Work Habits

(Item 43)

Student Motivation (Item 39)

 

HighHigh Avg.

 

Avg.LowAvg.

 

Low

High 4.48 4.38 4.28 4.13 4.04

High Avg. 4.38 4.29 4.14 3.96 3.76

Average 4.28 4.14 4.01 3.83 3.64

Low Avg. 4.15 4.05 3.88 3.70 3.51

Low 4.11 3.96 3.78 3.58 3.38

Impact of Extraneous Factors

Gaining Factual Knowledge – Average Progress Ratings

Technical Report 12, page 40

Work Habits

(Item 43)

Student Motivation (Item 39)

 

HighHigh Avg.

 

Avg.LowAvg.

 

Low

High 4.48 4.38

High Avg. 4.38 4.29

Average 4.01

Low Avg. 3.70 3.51

Low 3.58 3.38

Impact of Extraneous Factors

Gaining Factual Knowledge – Average Progress Ratings

Technical Report 12, page 40

Raw or Adjusted Scores

Purpose Raw or Adjusted?How much did students learn? Raw

What were the instructor’s contributions to learning?

Adjusted

How do faculty compare?

Adjusted

Doraw

scores meet or exceed

Expectations?*

Are adjusted

scores lower or higher than

rawscores?

Use adjusted

scoresUse

raw scores

Lower Yes

Higher

When to Use Adjusted Scores

*Expectations defined by your unit.

No

Three-Phase Process for Faculty Evaluation

Set Expectations

Collect Data

Use Data

III. Use DataWhich data will you use and how?

.

IDEA Faculty Worksheet

Keep track of reports Look for longitudinal trends Use for promotion and tenure

Created by Pam Milloy, Grand View University

Available from The IDEA Center Website

Using the DataSummative (pp.1-

2) Criterion or Norm-

referenced Adjusted or raw Categories of

Performance 30-50% of Teaching

Evaluation 6-8 Classes (more if

small)

Formative (p.3) Identify areas to

improve Access applicable

resources from IDEA website

Read and have conversations

Implement new ideas

Reflective Practice

Collect Feedback

Interpret Results

Read & Learn

Reflect & Discuss

Improve

IDEA resources that are keyed to reports

Talk with colleagues

Try new ideas Online, Paper

What the reports sayand what they mean

Research

Relationship of Learning Objectives to Teaching

Methods

Information Improvement

Information Improvement

Questions?