1

Using SETU data:some thoughts from the trenches

Stuart Palmer

Institute of Teaching and Learning

2

Or, How I

Learned to

Stop Worrying

and Love Like

At Least

Not Hate the

Thing

Dr Set-youDr Set-you

Source: http://en.wikipedia.org/wiki/Image:Castle_Romeo.jpg

3

THE HITCH-THE HITCH-HIKERS GUIDE HIKERS GUIDE TO SETUTO SETU

Don’t panic!Don’t panic!

Mostly harmlessMostly harmless

4

A context-free SETU comment

“I found that Stuart seemed to be complacent about this unit, his lectures where not worth turning up too, as it involved sitting there listening to him talk from the unit guide, which I did by myself at home anyway.”

5

> Liberte

6

> Je Remasse

7

> Le Tour Eiffel

8

> Gendarmarie

9

> Animalerie

10

> Stonehenge

11

Student evaluation of teaching (SET)

Work by Ramsden and others in the 1980s (re-)established a link between student perceptions of their learning environment and:

> the level of their engagement with their

studies; (hence, generally,)> the ‘quality’ of their learning/outcomes

12

What is quality?

?

13

What is quality in a pen?

Source: http://en.wikipedia.org/wiki/Image:Pen.jpg

14

What is quality in a pen?

> Writes smoothly?> Lasts a long time?> Doesn’t break?> Good grip?> Has a highlighter on one end?> Doesn’t smudge?> Has a gold nib?> Writes in space?

15

Student evaluation of teaching

Reflecting the modern understanding of the multi-faceted nature of quality and the finding that it is the entirety of the university experience that contributes to the student ‘course experience’, virtually all authors examining the value of the SET process recognise that SET data are only one of many sources of information that should be called upon when evaluating teaching

16

Triangulation

Source: http://en.wikipedia.org/wiki/Orthographic_projection

17

Triangulation

18

Triangulation

19

Triangulation

20

Evaluation of teaching

Other valuable data sources include:> objective measures of student learning (such

as unit marks)> reflective (self) assessment of teaching

performance (what worked, what didn’t)> peer assessment of teaching> student focus groups

21

Evaluation of teaching

> etc. …maybe Facebook?

22

Interpreting SETU

USA Today has come out with a new survey - apparently, three out of every four people make up 75% of the population

David Letterman

23

Interpreting SETU

> a reasonable number of responses;> a reasonable overall response rate; and > a reasonably equal rate of responses

between campuses

…provide some confidence that the survey sample is representative

24

Consistency

> a small standard deviation on a mean item rating estimate indicates that most respondents were in agreement

> a large positive difference between the proportion estimates of agreement and disagreement indicates that substantially more students agreed

> if the written open-ended comments reveal common, repeated themes, these are worthy of further investigation

> etc.

25

Discrepancy

> a small number of ratings that are different from the majority can influence the reported mean estimate

> a small number of adverse comments that seem at odds with the majority of written feedback are probably not worth losing a lot of sleep over

> a rating in one item that is significantly different from the others in the SETU results for the same unit might be an indicator worth investigating further

> etc.

26

Interpreting SETU – in general

SETU is only one measure of the ‘quality’ of teaching

SETU is a pretty blunt instrument

Used in an rational manner, the quantitative and qualitative results from SETU can be an aid in the evaluation of teaching and learning

27

Interpreting SETU – in general

The larger the number of responses, the more reliable the data

Trends (as long as they are real) can be informative

28

Mean SETU responses 2003-2004

2.5

3

3.5

4

4.5

1 2 5 6 17 18

SETU question

Mea

n ra

ting

2003(n=51, 17%)

2004(n=73, 42%)

2005(n=47, 32%)

2006(n=32, 37%)

1 The teaching staff of this unit stimulated my interest in further learning.2 The teaching staff of this unit motivated me to do my best work.5 This unit was well taught.6 I had a clear idea of what was expected me in this unit.17 I would recommend this unit to other students.18 The use of on-line technologies in this unit enhanced my learning experience.

29

Mean SETU responses 2003-2005

1 The teaching staff of this unit stimulated my interest in further learning.2 The teaching staff of this unit motivated me to do my best work.5 This unit was well taught.6 I had a clear idea of what was expected me in this unit.17 I would recommend this unit to other students.18 The use of on-line technologies in this unit enhanced my learning experience.

2.5

3

3.5

4

4.5

1 2 5 6 17 18

SETU question

Mea

n ra

ting

2003(n=51, 17%)

2004(n=73, 42%)

2005(n=47, 32%)

2006(n=32, 37%)

30

Mean SETU responses 2003-2005

1 The teaching staff of this unit stimulated my interest in further learning.2 The teaching staff of this unit motivated me to do my best work.5 This unit was well taught.6 I had a clear idea of what was expected me in this unit.17 I would recommend this unit to other students.18 The use of on-line technologies in this unit enhanced my learning experience.

2.5

3

3.5

4

4.5

1 2 5 6 17 18

SETU question

Mea

n ra

ting

2003(n=51, 17%)

2004(n=73, 42%)

2005(n=47, 32%)

2006(n=32, 37%)

31

Mean SETU responses 2003-2006

2.5

3

3.5

4

4.5

1 2 5 6 17 18

SETU question

Mea

n ra

ting

2003(n=51, 17%)

2004(n=73, 42%)

2005(n=47, 32%)

2006(n=32, 37%)

1 The teaching staff of this unit stimulated my interest in further learning.2 The teaching staff of this unit motivated me to do my best work.5 This unit was well taught.6 I had a clear idea of what was expected me in this unit.17 I would recommend this unit to other students.18 The use of on-line technologies in this unit enhanced my learning experience.

32

SETU teaching comments

“I found that Stuart seemed to be complacent about this unit, his lectures where not worth turning up too, as it involved sitting there listening to him talk from the unit guide, which I did by myself at home anyway.”

33

SETU teaching comments

> An exceptionally hardworking teacher. Always responding promptly to enquiries. Thank you Stuart for your efforts.

> Excellent teaching of course work> Excellent teaching, great feedback, showed interest

in progress. Best I have come across.> Good lecturer.> Great course content and well delivered

34

SETU teaching comments

> It was good how Stuart used videos to illustrate things in the real world. His lectures are well structured as all the unit material is covered in lecturers which are always informative. It was nice to get the reminder email about assignment1 and 3, this shows he has compassion towards the students.

> The lectures were good and didnt drag on too much.> The teaching support offered as part of this unit was

excellent.

35

SETU teaching comments

> This unit was very well taught. I could not fault any part of it.

> Very easy to listen to, and teaches well> Very promptuos lecturer> Well structured class.> very interesting to listen to and knows how to teach.

was very helpful to me when i had questions> very well taught, easily understood, madde the unit

enjoyable.

36

SETU teaching comments

> Yes stuart palmer is dedicated lecturer who is willing to help and puts an effort into helping students. His lectures cover alot of content with plenty of examples.

> Good uses of informations and extra material to help get a point across

> sometimes it feels very long to has class for 2 hours.

37

Be careful of the prominent outlier…

38

2

2.5

3

3.5

4

4.5

5

Well t

augh

t

Mat

erial

s

Workl

oad

Require

ments

Feedba

ck

Libra

ry

Recom

men

d

Online

OK

O/l enh

ance

d

Faculty

School

Prereq

Similar

Unit

39

2

2.5

3

3.5

4

4.5

5

Well t

augh

t

Mat

erial

s

Workl

oad

Require

ments

Feedba

ck

Libra

ry

Recom

men

d

Online

OK

O/l enh

ance

d

Faculty

School

Prereq

Similar

Unit

40

2

2.5

3

3.5

4

4.5

5

Well t

augh

t

Mat

erial

s

Workl

oad

Require

ments

Feedba

ck

Libra

ry

Recom

men

d

Online

OK

O/l enh

ance

d

Faculty

School

Prereq

Similar

Unit

41

2

2.5

3

3.5

4

4.5

5

Well t

augh

t

Mat

erial

s

Workl

oad

Require

ments

Feedba

ck

Libra

ry

Recom

men

d

Online

OK

O/l enh

ance

d

Faculty

School

Prereq

Similar

Unit

42

2

2.5

3

3.5

4

4.5

5

Well

taug

ht

Mate

rials

Wor

kload

Requir

emen

ts

Feedb

ack

Libra

ry

Recom

men

d

Online

OK

O/l enh

ance

d

Faculty

School

Prereq

Similar

Unit

43

44

Using SETU

The circumstances, context and influences on a particular unit are individual in nature

Standard, boilerplate responses probably aren’t that useful

SETU might offer some insights of value

45

46

Online PD materials

47

Online PD materials

Online PD modules:

www.deakin.edu.au/itl/pd/tl-modules

Enhancing teaching and learning:

www.deakin.edu.au/itl/pd/tl-modules/scholarly