Electronic Management of
Assessment – Critical Success Factors in
Institutional ChangeDr Barbara Newland, University of Brighton
Dr Neil Ringan, Manchester Metropolitan UniversityLindsey Martin, Edgehill University
Electronic Management of Assessment (EMA) includes a wide range of activities so the following definitions are being used:
Definitions
eSubmission electronic submission of an assignment
eMarking electronic marking (including offline marking eg in Word)
eFeedback electronic feedback (ie text, audio, video but not hard copy)
eReturn electronic return of marks
A network of senior staff in institutions engaged in promoting, supporting and developing technology enhanced learning
Over 130 nominated Heads from UK Higher Education institutions
A regular programme of well attended events
Represents the interests of its members to various national bodies and agencies including the Higher Education Academy and JISCwww.helf.ac.uk
Heads of eLearning Forum (HeLF)
AimTo identify critical success factors relating to
institutional change with EMA from longitudinal research over the past three years which has: Identified current practice with regard to EMA in
UK HEProvided a snapshot of the strategic overview
identifying key issues relating to strategic change, policies and practices
HeLF members were surveyed each March for 3 years from 2011 and asked to respond with regard to their knowledge of their own institution
The questions were a mixture of closed multiple-choice/multiple-answer and multiple selection as well as open response type
Participants were assured that all data collected in the survey would be held anonymously and securely
No personal data was asked for or retained unless the participant indicated a willingness to be contacted in the future
The results were analysed using quantitative and qualitative methods
Methodology
ResultsThe results emphasised the importance of assessment
It is mission critical as it determines whether students achieve, progress and gain awards (and at which level)
Consequently, all stakeholders in the assessment process are wary of making errors and need to be convinced of the merits in changing long-established practices
The change in practice supported by technology is only worthwhile if it effectively enhances stakeholders’ experiences and/or improves the overall process workflow
ResultsThe following comments illustrate the scale of
the change being undertaken:
‘e-Submission is quite straightforward. However, we could have managed the process of e-marking and e-feedback better. Academic staff need a lot of time to come round to the idea if they are changing years of established practice.’
‘Never underestimate the effort involved with winning hearts and minds of colleagues.’ (HeLF, 2014)
Critical Success FactorsInstitutional
culture Regulations Policy Guidelines
Support Implementation Plan
Communication
Responsibilities
Monitoring Risks Technology Health and safety
Resourcing – support staff
Attitudes of academics,
administrators, students
Quality assurance
Institutional culture An institutional culture encompasses all the critical success factors. The
change process needs to align with this culture. In some universities there has have been mandatory policies and in others a more flexible approach or a mixture of both:
‘It depends very much on the culture of your organisation. Here a partnership model between central services, and School / faculty admin and academics tends to work. Top down imposition of systems is less successful. In other institutional cultures this may be different.’
‘Policy led the mandatory eSubmission. eFeedback, while voluntary, is being adopted by staff as it is available in the same interface.’
‘One of the emerging ideas is to see submission, marking and feedback as separate entities, i.e. enabling students to submit and receive feedback online but staff to mark by whatever method is comfortable. Also not to insist on a particular feedback tool as long as the virtual point students pick up the feedback is consistent.’ (HeLF, 2014)
Institutional cultureThe EMA process involves everyone in a
university so identifying a leader of change can bring all the people together.
‘Having a central 'head of elearning' role to bring everything together across the institution: to scour the Regulations and relevant policies to produce a set of key questions for consideration by each of the key stakeholder groups.’ (HeLF, 2014)
Institutional cultureConsultation of all stakeholders and support from
senior management as well as the efforts of departmental champions and students is key:
‘Consultation was important to choose a system that would fulfil the criteria of all stakeholders. The stakeholders themselves shaped that system. The drive from students was critical in making sure people engaged with this’.
‘Senior Management support. Departmental champions. Peer and student pressure on academic staff concerning provision of eFeedback and eMarking.’ (HeLF, 2014)
RegulationsAssessment regulations will need to include EMA processes.
Often this requires just a small amendment to existing regulations. These will need to go through the committee process, which may take some time.
It may be helpful to consider how the current policy handles situations in which a student is unable to hand in a hard copy. For example, the outcome for a student who hands in work late
because his own computer fails could be the same as if the reason late submission had been missing a bus
Similarly, the outcome for a student who cannot hand in work as the university is shut due to snow could be the same if the student cannot hand in work because a university system has failed.
Policy and guidelinesPolicies
clear, up-to-date have to be agreed and approved by committees. This
may take some time
Guidelines: need to be made easily available to students,
academics, administrators and departments give clear instructions on effective ways of using the
systems and explain the responsibilities in alignment with the regulations and policies
describe contingency plans for particular scenarios occurring.
Technology Both the end user hardware and software and the central systems
need to be robust, reliable and secure
User software should be user-friendly and have the features and functionality to improve the assessment process for students, academics and administrators
There are benefits if the software integrates with other central systems such as the student record system and the Virtual Learning Environment.
‘Reliability / availability of both the VLE and Turnitin essential.’
‘Achieving a smooth system that supports the process well (technological perspective’ (HeLF, 2014)
Some departments provide second monitors and/or laptops to academics for eMarking and eFeedback.
Support Support and training both face-to-face and online is required for
academic and administrative staff as well as Helpdesk support.
Learning technologists have a key role and the move to eSubmission has generally created more work for them (Newland, 2011)
Students may require support in accessing and understanding the method of eFeedback
Support will be required both in advance and then to handle urgent queries from students and staff as the deadline approaches and stress levels increase with the urgency of submitting on time and overcoming any problems in this process
Pedagogical support to encourage effective feedback practices can enhance student learning. The quality of eFeedback is dependent on the quality of the feedback
Implementation PlanIdentifies what, when, how and who is
responsible
It needs to include the detail of the EMA processes e.g.How to handle extensions, second marking, blind
marking, re-sits. Is it e-only Submission or e and paper? If eFeedback is optional then who prints and who
pays for it.
CommunicationAlways important during a change process
Not only about the process of the change but also highlighting the benefits of engaging in the change.
Needs to be two way so that the implementation can be modified as the result of feedback and people participating in the change process are aware of the impact of their feedback
This helps to develop a sense of ownership/partnership with the change rather than it being perceived as something enforced.
ResponsibilitiesEMA will change the roles of academics,
administrators and learning technologists
This change may vary within courses and departments within a university
The consideration of responsibilities needs to be within a university’s EMA guidelines.
Monitoring and risksMonitoring
The implementation of EMA needs to be monitored and exceptions, for example, to e-only Submission to be identified and reported
The monitoring can also highlight effective practice.
RisksAssessment is mission critical and high risk in every
aspect.Appropriate planning for, and mitigation against,
potential identified risks occurring should be embedded within EMA policies.
Health and safetyConcerns expressed by certain staff regarding
the time spent looking at a screen. Usually, these concerns are already addressed through a university’s health and safety procedures
Many academics undertake eFeedback at home on their own computers, and appropriate consideration should be given to this aspect.
ResourcingImplementing change requires resourcing not
only in hardware and software but also for support and engagement in reviewing the whole EMA process.
AttitudesThe move to eSubmission is generally supported by
students, academics and administrators. However, many academics are initially reluctant to engage with eMarking and eFeedback. Positive experiences from students and staff encourage the change.
‘Positive student feedback through module evaluations and staff, student consultative committees. Exams & Assessments dept are positive about new processes.’ (HeLF, 2014)
Academic staff attitudes to eMarking are steadily changing as there is some recognition of the perceived value, benefits and efficiency and acknowledgment that it will happen in future.
Quality AssuranceeMarking and eFeedback can provide a more
consistent student experience. For example, by using a course or departmental rubric
However, greater standardisation may deter innovation in assessment
ConclusionThe critical success factors in the use of EMA as it
has moved from individual early adopters to more formalised institution–wide usage have been identified through a series of annual HeLF surveys
Consideration of these factors within an institutional context can enable the effective implementation of EMA
These factors involve everyone in a university and need to be considered simultaneously for effective implementation of, and change to, EMA
References Gibbs, G., Simpson, C., & Macdonald, R. (2003). Improving student
learning through changing assessment https://www.open.ac.uk/fast/pdfs/Gibbs%20and%20Simpson%202003.pdf
HeLF, (2014), www.helf.ac.uk
Newland, B., Martin, L., Ramsden, A. ( 2011). eSubmission – UK Policies, Practice and Support. Proceedings of the European Conference on e-Learning: 74-82.
Newland, B., Martin, L., Bird, A., & Masika, R. (2013). HeLF Electronic Management of Assessment Report, www.helf.ac.uk
Nicol, D., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31: 2, 199 — 218.
Sadler, D.R. (1989). Formative assessment and the design of instructional systems, Instructional Science, 18, 119-144.
Dr Barbara Newland [email protected]
Dr Neil [email protected]
Lindsey [email protected]
Contact details