2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

Evaluating and reporting at a distance: quality experiences with

cost effective web supported evaluations

Dr Susan Shannon

Spencer Gulf Rural Health School

The University of Adelaide and The University of South Australia

susan.shannon@adelaide.edu.au

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

Evaluation ClientsCASE STUDY 1

N U P H I S S

National Undergraduate Public Health Internship and Scholarship Program, December 2001 to February 2002

4 Universities, placements in WA, Qld, SA and ACT

6 week placements for end-3rd year students in private or public clinical setting. Students undertook research projects, under supervision.

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

Evaluation ClientsCASE STUDY 2

S G R H S Spencer Gulf Rural Health School

All 1st year medical students spend a Rural Week in one of two country districts in April, July or Sept

All 2nd year students undertake clinical shadowing for a week in a rural health setting

2003 Ten 5th years spent their year in the Spencer Gulf Learning in 5 locations

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

External Requirement for evaluation

Case Study 1

NUPHISSOngoing funding contingent on successful pilot program. External evaluation mandated.

Case Study 2

SGRHS Parameter of Commonwealth Contract for Rural Clinical School …‘transparent internal evaluation”

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

The Web to the rescue!CASE STUDY 1 NUPHISS

VGallery web-based Survey tool

Web based Discussion Board

CASE STUDY 2 SGRHS

Email Evaluation Questions

Optical Mark Recognition Software Surveys

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

V Gallery online survey tool

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

vGallery Advantages

• Professional appearance;• Speed of feedback from respondents and • Reporting to Excel automatically conducted;• No data keying by evaluation personnel;• Evaluation time can be spent on evaluation, ie understanding what data means rather than gathering data;• Good response rates (students 89%; academics 100%; work placement supervisors 68%)

Disadvantages• universal web access amongst respondents is required• keyboard familiarity is assumed

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

Web based Discussion Board

Advantages• Interactive• Responsive• Group focused • Threaded by evaluator to add further qualitative data

Disadvantages• Asynchronous• Universal Web access required• Key board familiarity is assumed• Can be talking to an empty “room” • Whole question of the evaluator as correspondent with the evaluatee

comes more vexed

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

Smart Board and Video Conferencing

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

Optical Mark Recognition Instrumentspurpose made evaluations

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003

Tool or Technique Advantages Disadvantages

VGallery web-based Survey tool Self-preparedProfessional appearanceSpeed of feedback from respondentsReports automatically to ExcelNo data keying by evaluation personnelEvaluation time can be spent productivelyGood response ratesNo Paper – nothing to loseSupports any type of question

Universal Web access requiredKey board familiarity is assumedSurvey form must be carefully prepared to avoid glitchesAll respondents must be enrolled as vGallery users on University of Adelaide PeopleSoft database to access Survey (provided with a username and password)

Web based Discussion Board InteractiveResponsiveGroup focused

AsynchronousUniversal Web access requiredKey board familiarity is assumedCan be talking to an empty “room”

Email Evaluation Questions Ubiquity of email as relevant and informal communication protocolCheapRequires no particular platform – hotmail and private isp providers utilised by students as preferred email addresses as well as official University student emailNo paper – nothing to lose

Email access problems at some sitesAttachments would not upload at some remote learning centres (bandwidth issues)Response rates diminished during the yearNo automatic accumulation of responses to database

Optical Mark Recognition Software Surveys

Response rtes remained highOMR software collates, analyses and reports digitallySpeed (potentially) and accuracy ensured in reportingReports emailedNo loss of data richnessUniversity- supported system

Paper based – document storage issues with originalsCurrent Software set-up only accommodates some types of questionsNeed to involve LTDU in process of preparing Surveys, analysing etc.