John Winkley May 2014 Reflections on innovations in e-assessment
Transcript
Slide 1
Slide 2
John Winkley May 2014 Reflections on innovations in
e-assessment
Slide 3
About AlphaPlus Specialist education and assessment consultancy
Particular strength in qualifications, assessment and examinations
Extensive work in technology-supported learning and assessment
Based in the UK, with an international portfolio of work, covering
Primary, secondary and high school examinations for progression to
college university Vocational and technical qualifications
Professional certifications
Slide 4
About me! Started out as an engineer/technologist two
organisations providing e- assessment and e-learning. www.btl.com
(millions of online exams each year)www.btl.com
www.virtual-college.co.uk (recently enrolled its 1.3 millionth
learner)www.virtual-college.co.uk Then moved in to e-assessment,
and then assessment more generally. Its not really about the
technology any more I work with educators, test writers,
certification agencies and governments to help make qualifications
and assessment work better
Slide 5
Potential of e-assessment for large-scale testing Bennett
(2001) In the same way that it is already helping to revolutionize
commerce, education, and even social interaction, the Internet will
help revolutionize the business and substance of large-scale
assessment.
Slide 6
But has potential gone unrealised? Isabel Nisbet, outgoing
chief executive of Ofqual February 2011
Slide 7
The way that students are tested should be the same as the way
that they learn, otherwise its not a very good test the way that
they use technology to learn that thats just native to them sitting
at desks writing down, that may not be testing what theyve learnt.
lets get this off the too difficult pile and just think a little
bit about the future, how are we going to make sure that tomorrows
kids have the right kind of tests? Isabel in text
Slide 8
Ofqual in 2013 Jeremy Benson, Director of Policy, Ofqual at FAB
national conference 2013 The starting point for my speech today is
standards in vocational qualifications. There has [not] been nearly
enough [talk] about standards. But in my view, standards are as
important in vocational qualifications as in general
qualifications. Standards covers not only the skills that have to
be demonstrated to pass the qualification, but also the level of
demand from the design and structure of assessments, and how well
candidates have to perform. How can we, as regulator, be confident
that your assessment developers and your assessors have the right
skills, so you can be confident in the standards and validity of
your awards? Where will the assessors of the future come from?
particularly in an increasingly e-enabled world?
Slide 9
Focus of this talk Maybe high-stakes testing in schools isnt
where e-assessment can have the most (immediate) benefit. Maybe we
can get more wins from non-testing, less schools technologies and
services. Some case studies: Changes in traditional e-assessment
On-screen essays Elephants dont forget! Secure exam delivery by pdf
Low volume marking Statistical services for AOs I will illustrate
the case studies and explain why they are successful (or at least
interesting) e-assessment innovations.
Slide 10
The case studies
Slide 11
CASE STUDY 1: on screen tests in 2014 Actually, a few disparate
experiences from various projects its really not about the
technology any more Our own experience in deploying e-assessment
for a major project for BIS Our experience in running a training
course recently in Saudi Arabia Falling price of e-assessment
per-test. Falling minimum volumes too. Cloud-based delivery appears
to have solved a lot of problems for centres
Slide 12
CASE STUDY 2: on screen essays The brief: Specialist
professional body offering international examinations Small
volumes, limited internal resources previous investigations of
traditional e-assessment not fruitful But board level drive to
undertake service modernisation to support growth Examinations take
the form of lengthy essay-based papers. Changing the nature of the
examination was outside the scope of the project. The
administration involves moving a lot of paper around the world.
Candidates are typically mid 30s busy professionals working in
large companies
Slide 13
CASE STUDY 2: on screen essays What we helped the professional
body to do: Review the options in the market including some of the
less trodden paths Deploy a Bring Your Own Device e-assessment
solution using the Exam4 armored word processor Support centres and
learners participating in the live pilot Obtain detailed feedback
on how they found the assessment Deploy a secure exam paper
delivery system Support centres using the technology
Slide 14
Slide 15
Slide 16
CASE STUDY 2: on screen essays What we learned: Students really
valued being able to use their own machine Familiarity Able to
type-and-edit faster than they could write Removed concerns about
untidy and illegible scripts Many students were willing and able to
work through the setup in advance Technology effective and
inexpensive Start to finish in about 6 months
Slide 17
CASE STUDY 3: Elephants dont forget Employer ownership pilots
are showing interest in unfamiliar approaches to assessment and
awarding, particularly in areas where employers have existing
assessment (usually internal compliance or safety-related) which is
similar to external assessment. Seems likely that BIS will want
Ofqual to consider approval of such innovative approaches. Despite
the jokey website, this approach could potentially deliver very
interesting effects in terms of assessment predictive validity (we
think).
Slide 18
CASE STUDY 4: Secure exam delivery Awarding organisations:
Upload Exam Packs to the system. An Exam Pack contains all the PDF
files you want the centre to be able to download in advance of the
examination. Create Examination Windows. An Examination Window
specifies when an Exam Pack becomes available for download, and
when the download window closes. Allocate the Examination Window to
one or more Centres. Mass email the Centres (via SecurePDF) to let
them know about their upcoming Examination Windows.
Slide 19
Monitor the status of downloads
Slide 20
How does it work for centres?
Slide 21
CASE STUDY 4: Secure paper delivery What we learned: Greatly
increased security for delivery of examination documents (versus
postal delivery or website/portal download) Awarding organisation
has complete control over when centres download examination
documents SecurePDF maintains a record of downloads, identifying
users and centres that downloaded each Exam Pack Centres were happy
to use the system and valued the efforts made to improve
security
Slide 22
CASE STUDY 5: Low volume marking There are four key problems
when it comes to marking digital files: How do we view all the
different file types without lots of software? Files can be very
large e.g. PowerPoint How do we actually mark/annotate the files
without changing the work? How do we track different users
annotations clearly?
Slide 23
Red pen how it works The Benefits: Mark and annotate a variety
of file types completely online No need for a multitude of software
- just a Flash plug-in Does not alter the work Links to mark
schemes Provision for student self, peer, teacher and verifier
assessment.
Slide 24
Case Study Red Pen Tool for Moodle Oldham Hulme Grammar School
Using Moodle to support work at KS3 & 4 Found that there was
still too much printing, especially for marking work They started
using Red Pen Tool in February 2011 Primarily using the Red Pen
Tool to support coursework for GCSE ICT (Module 2: controlled
assessment - WJEC) Proved to be a huge success and as a result they
are planning to extend use this coming year.
Slide 25
CASE STUDY 4: Statistical analysis for AOs AOs who use
on-screen tests will have large quantities of data. It is possible
to use such data to do a wide range of analysis. Such analyses are
very useful as a reality check. Helps you understand tests
performance empirically. Experts consider this item difficult, but
when we presented it to 5,000 students, the facility value was
0.75, so it's pretty easy. Statistical analysis does not replace
expert judgement, it supplements it. Its part of a general
commitment to quality. Helps AOs to spot problems before anyone
else does! There are pushes as well as pulls Ofqual increasingly
looking into this area. Likely to ask do you know how reliable your
tests are? Parallel forms an area of particular weakness, we
believe
Slide 26
Item difficulty vs curriculum level
Slide 27
Reflections on innovations in e-assessment Conclusions
Slide 28
Conclusions Lower cost, lower volume solutions ever more viable
for organisations that would not consider themselves as experts in
IT The view of on-screen assessment as a commodity is becoming more
prevalent (and more true) but is not the whole story Diversity of
options increasing too. Careful market searches identify tools that
solve different problems to the mainstream products Technology
offers AOs proven competitive advantage, at a time of extraordinary
market turbulence (for the Ofqual regulated sector)
Slide 29
John Winkley May 2014 Reflections on innovations in
e-assessment Thank you for listening. Questions?