Dorothy Spiller Teaching Development Unit University of Waikato, Aotearoa/ New Zealand Student Evaluations of teaching: Do they matter?
Transcript
Slide 1
Dorothy Spiller Teaching Development Unit University of
Waikato, Aotearoa/ New Zealand
Slide 2
Mihi E nga mana, e nga reo To those gathered here from diverse
backgrounds E rau rangatira ma To my esteemed audience Tenei te
mihi ki a koutou I greet you
Slide 3
Mihi Ka nui te hari me te koa Im so happy Ki te hoki mai ki
tenei whenua Im so happy to return to this land Ki taku ukaipo, ki
taku whenua tapu The land that provided me with nurturing and
sustenance, the land of my birth
Slide 4
Mihi Ka nui hoki te mihi ki nga iwi o tenei whenua I also wish
to express my thanks to the people of this land Ki aku matua me aku
tamariki To my parents and my children Mo nga taonga kua takoha mai
ki ahau For the gifts that have been given to me No reira, tena
koutou, tena tatou katoa Greetings once again to all of you/us who
are gathered here
Slide 5
Do student evaluations matter? For who? To who? For what? Why?
When? How?
Slide 6
The Quest What do teachers think about them? Why do they have
these views? How do teachers engage with them? How can we enhance
student evaluation processes to optimise their potential to improve
teaching and learning?
Slide 7
The Story The Beginnings The Quest The Equipment The Landscape
Re-designing the Landscape The Holy Grail
Slide 8
The Story Once upon a time... How did it all begin?
Slide 9
The Story - Myth or Reality? The institution uses this to judge
me People manipulate appraisals Students are not in a position to
make judgements The instrument is invalid I know my discipline and
what they need to know Are the questions relevant? I can't try
anything new because of the risk to my promotion It is a popularity
contest
Slide 10
The Story The Quest What do teachers really think? How does it
make a difference to them, their teaching and their learners?
Slide 11
The Story The Quest How do current evaluation processes and
practices influence teachers thinking and behaviours in relation to
student learning at all stages of the teaching and learning cycle?
What are the perceptions that tertiary teachers hold about student
evaluations? What factors affect these views? How do tertiary
teachers engage with student evaluations?
Slide 12
The Story - The Equipment Ako Aotearoa funded NPF research Two
year project Three institutions Otago University, Otago Polytech
and the University of Waikato Researchers were Academic Developers
and Evaluations personnel from these institutions
Slide 13
The Story - Count and find out... Survey online, run over three
weeks with weekly reminders Participants goal to reach as broad a
sample of the academic community for trends rather than in depth
analysis InstitutionTotal StaffNumber of responses Response rate
University of Otago (OU)144367046% University of Waikato
(WU)66324237% Otago Polytechnic (OP)32015348% Total2426106544%
Slide 14
The Story - Ask and ye shall find... Interviews
Semi-structured. Questions developed on the basis of themes that
emerged from questionnaire data Participants Called for volunteers
selected based on demographics to get a range of disciplines and
years experience Literature review - to sharpen interview questions
Environmental scan of institutions, around appraisal practice,
mission statements, teaching policies
Slide 15
The Story - Sense making Survey Quantitative analysis of
summative questions and thematic analysis of associated comments
(interpretivist paradigm) Interviews Thematic analysis
Slide 16
Discoveries What are teachers perceptions of student
evaluations? Why do they hold these views? views ?
Slide 17
Discoveries: Appraisals Landscape Widespread recognition of the
worth of collecting this data : 73% Main uses and limitations of
appraisal: UseResponses Informs teacher/course development19% Helps
identify students learning needs19% Limitations Shortcomings in the
current appraisal system14% Quality of responses by students10% Use
for quality/summative processes7%
Slide 18
The Story - Teachers Narratives Appraisals are okay, but but
but (mistrust of students judgement, of the instrument and the
institution) Read what students say Appraisals can help me with my
courses I feel strongly about this The timing is hopeless The
process ends here I work alone We dont talk about them
Slide 19
The Story: Teachers Narratives Beliefs about teaching and
learning Teacher manipulation Status of teaching Research focus
Need for multiple sources of evaluation
Slide 20
The story Emotional Terrain OU 10 indicated some form of
emotional reaction positive and negative. Language suggests
emotional intensity WU 4 general emotional significance and 3
directly negative OP 6 openly discussed feelings of
vulnerability.
Slide 21
Discussion: join the quest! 1. What perceptions of evaluations
have you encountered? 2. What are the reasons given for these
perceptions?
Slide 22
The Story Appraisals landscape How do the respondents engage
with appraisal data? For WU (92%) and OU (95%) - ranked first to
get feedback on students learning experiences OP: Because it is
required Interpreting and responding to the data For all
institutions the majority of participants rank reading the open
question/comments as their first response, and high numbers of
participants spend time going over the data, look for feedback on
teaching and compare with previous results. However percentages are
far lower when looking at active engagement and discussion with
colleagues and students. This is most noticeable in the
universities as opposed to Otago Polytech
Slide 23
Survey of the landscape Use 1. Read the open question/comments
made by students 2. Spend time going over the data and responses
3.= Actively looking for feedback about teaching and assessment
Compare the data with previous evaluations 5. Discuss the results
with colleagues/teaching teams 6. Provide students with feedback on
the results 7. Seeking assistance with interpreting the results
from others
Slide 24
The Story Appraisals Landscape How do teachers feed the
outcomes back to students? One of the lowest ranked use of
appraisal data was to provide feedback to students. The
universities were significantly lower than the polytech in this
area. The most preferred way of communicating with students was
through paper outlines The most commonly cited reason for not
feeding back to students was the timing of appraisal
Slide 25
How do teachers use evaluations for professional development?
47% discussed with colleagues mainly on course and paper
refinements Institutional use for promotion inhibits professional
development Improvements for professional development include
flexibility in the instrument and the use of other evaluation
methods
Slide 26
Stories from the interviews Across the board there is a small
percentage of staff who have systematic and deliberate strategies
for feedback to students although a number have informal approaches
and/or report to the next cohort Timing of evaluations is cited as
a major problems in terms of feedback to students Discussion with
colleagues is very limited, with a fairly strong culture of privacy
around evaluations More engagement with feedback at the
polytechnic
Slide 27
Join the journey: discussion To what extent do you consider
staff at your institution engage with formal evaluations? How do
staff provide feedback to students on their response to student
evaluations? What strategies can we use to enhance engagement with
evaluations feedback for professional development and student
learning?
Slide 28
The story the holy grail 1. That institutions ensure that there
is clear alignment between their vision/policy statements
concerning the auditing and developmental purposes of student
evaluation systems and their implementation. 2. That institutions
implement a professional development policy that includes explicit
support for the education of staff and students about the purposes
of student evaluation for curriculum and teaching, and the
institutional intents and purposes of its student evaluation
system.
Slide 29
The Holy Grail That those who administer student evaluation
systems recognise the variety of staff perceptions about student
evaluations and provide communication, support and resources that
address teacher expectation and needs, without compromising
institutional intents and purposes. That institutions ensure
expectations about teacher and student roles and responsibilities
in evaluations are unambiguous, and connections among performance,
evaluation and rewards are clearly understood.
Slide 30
The Holy Grail That teachers/faculties/departments schools and
institutions embed within the evaluation policies and practices the
notion that a well-rounded representation of teaching and courses
is more likely to be achieved by drawing on multiple forms of
evaluation data. That professional development and course
enhancement are firmly ensconced as the foundation and foci of
student evaluation processes and practices.
Slide 31
Policies, perspectives, practices Do policies highlight and
support the developmental as well as the auditing purposes of
student evaluation? Do evaluation instruments and processes fit the
needs and context of different programmes and levels? Are
evaluation questions informed by current research on teaching and
learning? Is there a culture that promotes conversation around
teaching, learning and evaluation? Is the evaluation process an
integral part of the entire teaching and learning process?
Slide 32
Policies, perspectives, practices. Does the institution provide
support and guidance in relation to the interpretation of
evaluation feedback? Is there an institutional requirement for
teachers to communicate with students about their responses to
student feedback? Are students provided with education about
evaluations and their use? Does the institution include evaluations
education in staff professional development requirements? Are staff
required to demonstrate their response to evaluations feedback and
undertake appropriate PD?
Slide 33
The case for mid-semester evaluations Closing the loop Feedback
Dialogue Student engagement Integral to teaching and learning
process
Slide 34
Closing the loop Student feedback Analysis Action Feedback to
students