Does the Medium have an Effect on

Assessment?

George McGuireMartin YoungsonDouglas McMillan

with Dave Fiddes and Athol Korabinski

Paper - current exam delivery

Computer - growing trend For Mathematics, attractions include automatic (instantaneous) marking logging of results

Are exam results affected by mode of delivery?

Modes of exam delivery

Rewording effect - to fit the assessment package

Medium effect - questions read from the screen input of answers by typing

‘on’ the screen

What is the significance of each of these effects for exam marks?

Pilot study to investigate this question.

Changing the mode of delivery

The questions in the pilot study

Typical of the short questions in paper 1 of the Scottish Higher

exam

Require few steps to reach the answer but still involve a reasonable amount of mathematics.

Paper version of a test (P Format)

Example of a paper question

1. If f (x) = 2x - 3 and g (x) = 2x2 - 3

find an expression for g (f (x)). Write

your answer in the form ax2 + bx + c.

Computer version (ICT format)

Example of a computer question

1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 then g (f (x)) = ax2 + bx + c. What are a, b and c?

1. What is a?

2. What is b?

3. What is c?

CUE

Determination of the medium effect

Comparison between paper and computer versions would have

to take account of both rewording and medium effects

Exact paper copy of the computer version (RT format) eliminates any rewording effect

ICT versus RT for medium effect only

Reverse translation version (RT format)

Example of an RT question

1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 then g (f (x)) = ax2 + bx + c. What are a, b and c?

1. What is a?

2. What is b?

3. What is c?

Determination of the rewording effect

No medium effect between P format

and RT format (both paper)

Working influences marking in a

paper test

Modification of the RT test, RTW

format, allows space for working

Question in (RTW format)

Rough Working1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 then g (f (x)) = ax2 + bx + c.

What are a, b and c?

1. What is a?

2. What is b?

3. What is c?

Pilot study 90 minutes in April 2001

Two schools with Higher Mathematics pupils

The Tests:

1 P format - Three 30 minute tests with SQA

questions

2 ICT format - Three computer tests using CUE

3 RT format - Three reverse translations of the

computer tests

Each candidate sat the 3 tests, one in each

format

Preparation for the pilot study

Pupils familiar with paper exams but generally unfamiliar with computer exams

A visit to each school before the day of the pilot study

Explain what would take place on the day of the pilot

Practice test of some simple questions to give pupils some familiarity with CUE

CUE

Setting up the groups for the test

SQA split the pupils into 3 groups based on gender mathematical ability

Mathematical ability measured by: standard grade results higher prelim results

Running the tests Each candidate took each test -

one in each format

Group sat tests in different orders

At any time no group was sitting

the same test as another group

a test in the same format as

another group

Time was not an issue

Setting up the pairings for the analysis

62 matched pairs in total

59 matched for gender and ability

3 matched for ability not gender

From different groups, matched pairs were set up:

Marking of the tests Paper tests marked by SQA

ICT tests marked automatically by computer

RT tests marked by me, looking only at final answer

RTW tests marked by me, taking account of rough working

Average mark over all tests: 39%

Individual marks range from 0% to 100%

Analysis of marks

2 statistical analyses: P - RTW and ITC -

RTC

Both done on 62 differences in marks

from matched pairs

Null hypothesis is true underlying mean

is zero

One sample t-test performed

62 is large enough to validate analysis

The results for the rewording effect

Well outside the 95% confidence

interval

Strong evidence of a difference

Was this a marker effect?

Marker effect eliminated by P papers

remarked by me

Still strong evidence of a difference

The results for the medium effect

Well inside the 95% confidence interval

No evidence of a difference

The gender effects

Analysis repeated for male and

female pairs

Same results as before

Conclusions on the rewording effect

Differences may have been due to

Rewording

Reformatting

The way the answers were set out

Pupils less inhibited doing rough

working

More detailed study needed to clarify

Conclusions from the medium effect

Candidates had only limited prior

experience of computer tests

There is no evidence of a

difference due to the medium

effect

Thank you

V

Does the Medium have an Effect on Assessment?

Directions for further study

Partial credit using steps

Steps may give strategy

May be able to test areas not

currently tested

CUE