A presentation by:

The University Student Evaluation of Teaching Task

Force

August, 2014

Take a moment…

Reflect upon feedback you have received from your student ratings of your teaching. Were the comments/ratings

useful?Did they help you improve?Were they demoralizing?

Comments From Student Ratings…The entire structure of this class is messed

up! The 10 minute quizzes were bull***t. She taught this class like it was a SWS class but its not SWS. Therefore she needs to change that. This final exam is bull***t. We have never been tested on the material yet.

Does not tolerate logical discussion from students.

He just rubbed me the wrong way.

Comments Continued…Best teacher at Grand Valley; he

should get a big raise.

She was cool. I’d like to set her up with my dad.

Student Ratings of Teaching at GVSU

GVSU’s primary identity is that of a teaching institution, and we use student ratings for important personnel and salary decisions.

Student ratings of teaching performance should be a tool that give faculty useful information to help them succeed.

Yet, we are not measuring student perceptions of teaching effectiveness in a consistent, empirically validated manner.

Concerns with Student Rating Instruments at GVSU

25 different instruments that measure 180 constructs/competencies are used across the university.

Most instruments are homemade and have not been tested for validity or reliability.

Concerns With Interpreting Student Ratings of Teaching at GVSU

Student ratings of teaching are a subjective and perceptual measure, but they are often treated as a finite and objective measure.

Interpretation Concerns continued…

Some units place too much weight on small differences in rating scores. For example, there is not much empirical difference between scores of 4.1 and 4.3, but such gaps are often treated as meaningful.

Student comments are sometimes being used haphazardly and inconsistently. Interpreting rating form comments necessitates specific questions and a validated coding system.

Concerns With the Use of Student Rating Data at GVSU

Student ratings of teaching are not a complete measure of teaching effectiveness, and they are sometimes given too much importance in the instructor evaluation process.

There is no consensus across the university about the weight that should be assigned to student ratings.

Use Concerns continued…

The outputs of student ratings are used inconsistently across the university. Some units use only scores, some use only comments, and some use both. Some units have no specific guidelines about what should be used.

Past Work on Student Rating Systems

A 2007 task force studied the feasibility of a university-wide student rating system. The task force concluded there was a lack of understanding at GVSU regarding the purposes and use of student rating systems, and education of faculty and administrators was needed before a university-wide system was adopted.

From 2010-2013, the Pew FTLC Advisory Committee conducted an extensive investigation into available student rating instruments. It recommended that GVSU should adopt a standardized instrument. It further noted that faculty held many incorrect notions about student ratings of teaching.

ECS held a number of town hall meetings during AY 2013-2014 to give faculty an opportunity to learn about and discuss the issue.

ECS/UAS MotionOn January 31, 2014, the UAS approved the following

motion:

The university should adopt a standardized measure of student evaluations of faculty teaching that meets contemporary standards for reliability and validity in psychometric measurement. The measure should yield both quantitative and qualitative results. The university should also adopt a standardized platform for administration of the measure (e.g., online software). The measure and platform should be used in all units and colleges. Units and colleges may employ additional instruments separately from the university standard.

Current Task Force Formed

In April of 2014, ECS appointed a task force

charged with recommending:1. A common student evaluation form2. Guidelines concerning the use of the

form3. An implementation procedure.

Their report to ECS is due in December, 2014.

Task Force MembersThe task force is comprised of

people who have expertise in test and measurement, statistics, technology, performance rating, and teaching scholarship.

You can find their names on the USETI website.

Research about Student Ratings of Teaching

It is one of the most studied aspects of college teaching; there are currently over 3,000 citations.

We relied heavily on reviews of the literature for research information.

Links to this material are posted on the USETI Website

Research About Consistency

Starting at mid-career, student ratings for an instructor in a particular course remain fairly stable over time if a valid instrument is used.

Individual student ratings of an instructor remain the same five years after graduation.

Research About Validity…

Student ratings do measure learning – in multi-section courses that use a common final exam, students of teachers with higher ratings do better on the exam.

Students do separate the teacher from the course when rating.

Factors That Are Associated With Lower Numerical Ratings

◦Teachers of math and science courses are rated lower than teachers of humanities courses.

◦Teachers of lower-level courses are rated lower than teachers of higher-level courses.

◦Teachers of required classes are rated lower than teachers of elective classes.

Factors That Are Not Consistently or Strongly Related to Ratings

◦Gender of student or instructor

◦Time of day class is taught

◦How many majors are in the course

◦The scholarly output of the instructor

Factors Not Consistently or Strongly Related continued…

Assigning higher grades does not result in better evaluations. Hundreds of studies have examined this issue; the average correlation across studies is zero.

Note: The difficulty of the course actually has a POSITIVE relationship with higher ratings.

What Can Be Learned From Student Ratings

Student ratings can serve as a useful global indicator of teaching effectiveness.

Students are qualified to rate how much they learned in a course.

Students are qualified to rate observable

teacher behaviors, such as clarity, fairness, organization, presentation skills, etc.

The Limits of Student Ratings

Student ratings cannot precisely or completely appraise teacher effectiveness.

Student ratings of teaching inherently contain error; they are a form of performance appraisal, and as such, are susceptible to all the known rater errors.

Students cannot accurately evaluate all important aspects of teaching. For example, students are not qualified to evaluate instructor knowledge or depth of subject coverage.

What’s Next…As charged by ECS, by the end of December, 2014,

the task force will:

1. Recommend an instrument for adoption by all units at GVSU. Only instruments with established reliability and validity are being considered.

2. Recommend a method of delivery.

3. Recommend parameters addressing the use of student ratings.

4. Recommend an implementation plan.

What’s Next continued:

ECS and UAS will consider, discuss, and vote on these recommendations.

Available Resources

The task force has constructed a website about student ratings of teaching and its work:

gvsu.edu/useti

You can also contact Ed Aboufadel, Chair of USETI.