EVS: The Good, the Bad and the UglyMichaela Cottee & Fiona Robins

ContextEVS = Electronic Voting System(s) Used with TurningPoint software, which links with

Powerpoint presentations.

University-wide, centrally funded, assessment project during 2010/11.

UHBS given 1000 EVS handsets for use in Semester B.

500+ EVS handsets were issued to Level 4 students.Also used with Level 5 students.Additional handsets were used for one-off test and

revision sessions (various levels).

EVS: Four advantages1. Promotion of active learning [through

anonymity]2. Generation of [prompt] student feedback3. Generation of staff feedback4. Encourages attendance and engagement

Iain Cross, Life Sciences“Blended Learning in Practice” May 2011

EVS: Good practiceSeven principles for good practice in UG education:1. EVS supports active learning through anonymity2. EVS facilitates prompt student feedback3. EVS encourages contact between students and staff4. EVS develops reciprocity and cooperation among

students5. EVS emphasises time on task6. EVS communicates high expectations7. EVS respects diverse talents and ways of learning

Chickering & Gamson (1987)

OverviewFiona Robins4BUS0154 – Decision Making Tools for BusinessN = 82 studentsEVS used in LECTURES to provide both formative

and summative feedbackMichaela Cottee4BUS1017 – Quantitative Analysis for Accounting

and FinanceN = 146 studentsEVS used in TUTORIALS to monitor attendance and

to provide summative feedback

Fiona RobinsUsing EVS in Lectures

Presentation

Using EVS in LecturesEVS used in a level 4 quantitative

moduleEVS accounted for 5% of the marks

for the moduleUsed for both formative and

summative feedback

Why did I use EVSTo improve attendance

0 1 2 3 4 5 6 7 8 90%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

FINAL Grade

Number of Lectures missed

Final Grade (%)

Why did I use EVSTo check understanding

Basil ... “there is too much butter on those

trays”

Manuel ...“no no, senor, not on those tres” ....” no sir “un, dos, tres

Why did I use EVSTo inject some fun

The goodFeedback to the lecturer on students learningInstant feedback to student on their

understandingCorrect their understandingCement learning – three times to go into

brainRecords results

The Bad

Time taken from lecture

Some students too embarrassed to vote

Stressful to manage lecture and EVS

The Ugly

Managing the Technology

The Ugly

0 1 2 3 4 5 6 7 8 90%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

FINAL Grade

Number of Lectures missed

Final Grade (%)

Attendance didn’t improve

Michaela CotteeUsing EVS in Tutorials

Assessment diet20% EVS tests (8 weekly in-class tests – best

4 results)20% StudyNet quizzes (9 weekly on-line

quizzes)20% Mid-term test (written)40% End-of-module test (on-line multiple-

choice)

Module Handbook infoThere will be 8 weekly in-class tests requiring the use of the EVS handsets. The best 4 results will be used to determine the mark for this assessment. The tests will be in the form of multiple choice questions which must be answered using the EVS handset. Please note:

You should bring your EVS handset (and calculator) with you to every tutorial.

You must attempt a test in order to be recorded as present at that tutorial. (In accordance with Business School policy, students who miss more than three tutorials may fail the module through non-attendance.)

No allowance will be made for forgetting your EVS handset or if the battery runs out.

Lending your EVS handset to another student will be classed as cheating. As the best 4 results out of 8 will be used, if you miss one week (e.g.

through illness) we will take the best 4 results out of 7. You will be loaned an EVS handset for the duration of the module. Failure

to return it at the end of the Semester will result in a debt to the University.

Solve the following equation for x:x – 4 = –2

1. –22. 23. 44. –45. None of the above

45

If a Normal distribution has mean 10 and standard deviation 2,

what is P(X < 12)?

1. 0.15872. 0.46023. 0.53984. 0.84135. None of the above

60

Test InstructionsYou will need a pen, paper and calculator.You will need a copy of the standard Normal

distribution table.You will need to use the following formula –

write it down now:

.d.s

meanXz

Performance (excluding zeros)End-of-module test average: 54%Mid-term test average: 59%StudyNet quiz average: 69%EVS test average: 79%

Overall average: 62%

Pass rate (before resits): 88%

Tutorial Attendance

* Test 8 took place on the last Friday afternoon of term!

Test Number Number Absent Percentage Attendance

1 30 79%

2 29 80%

3 33 77%

4 18 88%

5 35 76%

6 20 86%

7 48 67%

8 70 48%*

Attendance & EVS Test Performance

N = 146

Number of absences Number of students “ Best 4” average0 35 89%1 27 87%2 37 81%3 24 77%4 14 64%5 3 49%6 4 26%7 0 -8 2 0%

The Good...The system lends itself to multiple choice testing – as

does Quants.Quick and easy to create and incorporate questions

into Powerpoint using TurningPoint.The students had very few problems learning to use

the EVS handsets.Taking the “best X out of Y” results gave me a very

easy way of coping with missed tutorials / forgotten handsets / faulty handsets / etc.

Improved attendance at tutorials – and easier monitoring of attendance.

Good student engagement.

The Good...Students had EVS tests on all but two topics

and so had to revise as they went along.Students met each topic up to FIVE times –

first of all in the lecture, then in the tutorial, then in the StudyNet Quiz, then in the EVS test and then finally in either the mid-term test or the end-of-module test.

Good marks – both for the EVS tests and overall.

Weekly feedback via StudyNet – League table (using SRNs).

The Bad...Getting started – one more thing to learn

incorporate into a new module in a very busy Semester!

I didn’t seek feedback from students about the use of EVS in the module.

I didn’t use the results to inform my own teaching or to chase or support weak students.

The testing format suited some types / styles of question better than others.

NB. The above points reflect badly on me, but not on the use of EVS!

The Ugly...It took several weeks for all students to collect

handsets from SSG. The tutors had to be trained in the use of EVS –

including new tutors who took over towards the end of the module.

The tutors felt that the EVS tests ate into the tutorial time and distracted from the tutorial activity.

There was signal interference between the tutorial rooms - but coping with the consequences of this was thought to be easier than trying to get the students to change channel on the EVS handsets!

The Ugly (continued)...Reports from TurningPoint were only partially helpful

(but not necessarily TurningPoint’s fault!)Results had to be entered manually into a spreadsheet.Good support available within UHBS (Glenn Alie,

Karen Robins, Richard Balawender, etc.) – but not always when it was needed!

Nothing went seriously wrong – the responses were successfully saved each week. Phew!

Allowance could not be made for students who forgot their handsets or whose handset wasn’t working.

Inclusive practice: How to cope with “extra time” students?

A Third PerspectiveKaren Robins5BUS0133 – OR ModelsN = 64 studentsEVS used in lectures to provide summative

feedback – best 5 out of 7 (worth 10%)

Student Feedback – Qualitative & QuantitativeN= 44 respondents

Student Feedback“EVS is easy to use” – 98% agree“I would recommend using EVS for this module next year” –

95% agree“I would recommend use of EVS technology in other

Business School modules” – 90% agree“Using EVS helped my understanding of the module” – 74%

agree“I was happy to be provided with an EVS handset, knowing

that if I did not return it I would have to pay £35 to replace it” – 40% agree; 60% disagree

“Weekly EVS questions were the only reason for attending lectures – 28% agree; 72% disagree

“To use EVS technology the user must be computer literate” – 21% agree; 79% disagree

Negative Comments“The software would not always work”“It’s annoying because lectures are not meant to be

compulsory but I have to attend”“If lost the handset it is too expensive to pay for it” [£35]“Sometimes gets confusing to operate (changing

channels)”“Usually forget to take it to the class”“When there were problems with the pad it either took

too long to get fixed or trouble to get help and a new one”“It keeps me attending lectures and puts me under

pressure when I am late, plus it’s just one lecture a week which is not friendly!!!!”

Positive Comments“Simple and straightforward to use the handsets”“A good way to enhance understanding by ensuring

revision on a weekly basis”“Is fun at the same time as it is when working hard”“Helps the teacher to have a better understanding

of which students struggle more than others”“Using EVS makes individuals come to the lecture

as it counts towards the final grade”“It’s something new and innovative”“Save paper” “Save the environment”“Quick answers/results”