2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
Evaluating and reporting at a distance: quality experiences with
cost effective web supported evaluations
Dr Susan Shannon
Spencer Gulf Rural Health School
The University of Adelaide and The University of South Australia
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
Evaluation ClientsCASE STUDY 1
N U P H I S S
National Undergraduate Public Health Internship and Scholarship Program, December 2001 to February 2002
4 Universities, placements in WA, Qld, SA and ACT
6 week placements for end-3rd year students in private or public clinical setting. Students undertook research projects, under supervision.
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
Evaluation ClientsCASE STUDY 2
S G R H S Spencer Gulf Rural Health School
All 1st year medical students spend a Rural Week in one of two country districts in April, July or Sept
All 2nd year students undertake clinical shadowing for a week in a rural health setting
2003 Ten 5th years spent their year in the Spencer Gulf Learning in 5 locations
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
External Requirement for evaluation
Case Study 1
NUPHISSOngoing funding contingent on successful pilot program. External evaluation mandated.
Case Study 2
SGRHS Parameter of Commonwealth Contract for Rural Clinical School …‘transparent internal evaluation”
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
The Web to the rescue!CASE STUDY 1 NUPHISS
VGallery web-based Survey tool
Web based Discussion Board
CASE STUDY 2 SGRHS
Email Evaluation Questions
Optical Mark Recognition Software Surveys
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
V Gallery online survey tool
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
vGallery Advantages
• Professional appearance;• Speed of feedback from respondents and • Reporting to Excel automatically conducted;• No data keying by evaluation personnel;• Evaluation time can be spent on evaluation, ie understanding what data means rather than gathering data;• Good response rates (students 89%; academics 100%; work placement supervisors 68%)
Disadvantages• universal web access amongst respondents is required• keyboard familiarity is assumed
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
Web based Discussion Board
Advantages• Interactive• Responsive• Group focused • Threaded by evaluator to add further qualitative data
Disadvantages• Asynchronous• Universal Web access required• Key board familiarity is assumed• Can be talking to an empty “room” • Whole question of the evaluator as correspondent with the evaluatee
comes more vexed
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
Smart Board and Video Conferencing
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
Optical Mark Recognition Instrumentspurpose made evaluations
2003 ALN Evaluation and Assessment Conference: a commitment to quality Adelaide 24-25 Nov 2003
Tool or Technique Advantages Disadvantages
VGallery web-based Survey tool Self-preparedProfessional appearanceSpeed of feedback from respondentsReports automatically to ExcelNo data keying by evaluation personnelEvaluation time can be spent productivelyGood response ratesNo Paper – nothing to loseSupports any type of question
Universal Web access requiredKey board familiarity is assumedSurvey form must be carefully prepared to avoid glitchesAll respondents must be enrolled as vGallery users on University of Adelaide PeopleSoft database to access Survey (provided with a username and password)
Web based Discussion Board InteractiveResponsiveGroup focused
AsynchronousUniversal Web access requiredKey board familiarity is assumedCan be talking to an empty “room”
Email Evaluation Questions Ubiquity of email as relevant and informal communication protocolCheapRequires no particular platform – hotmail and private isp providers utilised by students as preferred email addresses as well as official University student emailNo paper – nothing to lose
Email access problems at some sitesAttachments would not upload at some remote learning centres (bandwidth issues)Response rates diminished during the yearNo automatic accumulation of responses to database
Optical Mark Recognition Software Surveys
Response rtes remained highOMR software collates, analyses and reports digitallySpeed (potentially) and accuracy ensured in reportingReports emailedNo loss of data richnessUniversity- supported system
Paper based – document storage issues with originalsCurrent Software set-up only accommodates some types of questionsNeed to involve LTDU in process of preparing Surveys, analysing etc.