Methods for evaluation of a prescribing curriculum for Australian senior medical students

AJ Smith, SR Hill, T Tasioulas,

NL Cockayne.

Objectives

To design, field-test, implement and evaluate the impact of a modular, problem-based web-interactive prescribing curriculum based on the WHO “Guide to Good Prescribing”.

Summary of methods Curriculum developed by a consortium

from all medical schools. Made web-based, interactive by

“Knowledge South” and the University of South Australia.

Field-tested for content and ease of navigation by medical students and staff.

Made available to medical schools -could not be mandated.

Evaluative methods – the ideal

A pre-planned randomised allocation of medical schools to receive/not receive the curriculum with pre- and post-testing of each group on ability to select medicines discussed in the modules.

OR Randomisation within each school with

delayed introduction of the curriculum to one of the two student groups – but potential problem with “contamination”

Obstacles to the ideal. No advance knowledge of which, if

any, school would use the curriculum No knowledge of whether all or only

part of the curriculum would be used No ability to do delayed introduction

as students needed to be moving through at the same rate if the curriculum replaced conventional teaching, which was what teachers wanted.

What was feasible ?

“Hits” on the website organised by school

In-depth interviews with staff, on-line feedback from 107 med. students representing the schools that used the curriculum

Before-and-after testing of a cohort of students from one school.

What were the problems in interpretation of the results ? “Hits” only mean that the page has been

opened – not an assessment of learning. Staff respondents were uniformly

supportive Student responses may have only come

from the enthusiasts and not be fully representative.

Before-after results show that compulsion works but are they a fair test of the program’s ability to change prescribing ability ? Were there any other influences working at the same time? No control group.

How would we do it differently ?

Difficult unless we can construct control groups and, perhaps, conduct the testing over a shorter period.

If schools do not take up the curriculum, seek to use their students as controls for further before-after tests.

A PRESCRIBING CURRICULUM FOR AUSTRALIAN MEDICAL STUDENTS

Smith AJ, Hill SR,Tasioulas T,Cockayne NL.National Prescribing Service, Sydney & Clinical Pharmacology Unit, University of

Newcastle, Australia

Background The Australian National Prescribing Service

(NPS) not-for-profit, independent organisation

Activities aim to improve the overall quality of use of medicines in Australia

Development of curriculum as a response to: Documented sub-optimal prescribing by doctors A perceived deficit in prescribing education for

senior students in medical schools - despite good programs in pharmacology/clinical pharmacology

Objectives To design, field-test, implement and

evaluate the impact of a modular, problem-based web-interactive prescribing curriculum based on the WHO “Guide to Good Prescribing”

Methods (1)Design Jointly sponsored by NPS and the Australasian

Society for Clinical and Experimental Pharmacologists and Toxicologists (ASCEPT)

Collaboration with teachers from each of the Australian medical schools

Clinical conditions were selected from those that interns claimed were common and raised prescribing issues

The 12 written cases were converted to web-based interactive form by Knowledge South and the University of South Australia

Methods (2)

Field-testing and implementation The 12 modules were field-tested for ease

of navigation and for content by medical students and staff

The curriculum could not be mandated and uptake was dependent on NPS advocacy and the influence of the teachers who had helped construct the program

The completed modules were made available to each of the 11 Australian medical schools in January, 2002

Methods (3)Evaluation Measurement of ‘hits’ on the website by each

medical school Structured interviews with 15 teachers from 9

medical schools and online survey of 121 medical students from 5 medical schools

An assessment of students’ ability to select drugs for four common conditions: before using the curriculum after one year’s voluntary use after a further one year’s compulsory use

Results (1)

Annual 'Hits' to web site by each Medical School*

0

20000

40000

60000

80000

100000

120000

1 2 3 4 5 6 7 8 9

2002 2003

*Medical Schools have been de-identified

Results (2)

Qualitative feedback: In-depth interviews with teaching staff

All used it as a tool for individual self-directed learning tool five with added case-discussions

Five schools made the curriculum assessable in final examinations

Issues/problems: Technical aspects e.g. navigation and browser capability “Need to engage more teaching staff within faculty”

Generally, “an excellent, practical resource”

Results (3)

Qualitative feedback: student online survey

97/107 (91%) therapeutic content appropriate to needs

92/107 (86%) felt better able to prescribe after using the curriculum

57 (54%) had developed their own personal formulary – a major goal of the program

Results (4)

Improvements in selecting appropriate drugs in each condition were significant (2=37, p=0.01)

Student Assessment of Ability to Select Appropriate Drugs in

Four Conditions (Pre and Post Testing)

0

20

40

60

80

100

Condition 1 Condition 2 Condition 3 Condition 4

% C

orr

ect Res

ponse

Start Year 4 (Curriculum voluntary, n= 54) Start Year 5 (Curriculum compulsory, n=58) End Year 5 (Curriculum compulsory, n=24)

Conclusions

The program has proved very popular with students and staff

Uptake in the first two years has been better than predicted but the program needs more ‘marketing’

The modules have now been revised and plans are in place for more thorough evaluation in 2004

Website URL: http://nps.unisa.edu.au