Date post: | 01-Nov-2014 |
Category: |
Education |
Upload: | paul-bacsich |
View: | 521 times |
Download: | 0 times |
EFQUEL Academy 1
Quality in Virtual Schools and Colleges – a view from VISCEDPaul Bacsich ([email protected])
VISCED – www.virtualschoolsandcolleges.info
EFQUEL Academy 2
Introduction
EFQUEL Academy 3
Topics
• Historical overview• The Pick&Mix system• Supplementary criteria, local criteria and slices• Scoring, evidence and documentation
EFQUEL Academy 4
Benchmarking and Quality in e-learning: My part of the story
Overview and history
EFQUEL Academy 5
Benchmarking e-learning (UK)
• Foreseen in HEFCE e-learning strategy• Higher Education Academy oversaw it• Four phases – 82 institutions (2005-07)• Justified entry to Pathfinder and Enhancement
programmes - and useful for JISC bids (Curriculum Design etc)
• Was leveraged into refresh of learning and teaching strategy (Leicester etc)
EFQUEL Academy 6
What is benchmarking? (HEFCE)
“[for] individual institutions [to] understand their own positions on e-learning, to set their aspirations and goals for embedding e-learning… to benchmark themselves and their progress against institutions with similar goals, and across the sector [world]”
Can replace e-learning by “distance learning”, “OER”, “podcasting” ,etc
EFQUEL Academy 7
Methodologies: Pick&Mix• First used for Manchester Business School 2005• Has been used in 36 institutions, mainly new universities
and university colleges– Including 4 very diverse institutions in Wales (2007-09)– And 5 in the Distance learning benchmarking Club (UK, Sweden,
Canada) plus two more in UK (2009-12)– And basis for the Critical Success Factors schemes in two EU
projects: Re.ViCa (2007-09: developments fed back into Pick&Mix) and now VISCED (2011-12)
– Also an OER version of the scheme ready for use• Wording slightly revised for schools and colleges
EFQUEL Academy 8
Pick&Mix
Criteria and metrics
EFQUEL Academy 9
Criteria
• Criteria are “statements of practice” which are scored into a number of performance levels from “bad” to “best” to “excellent”
• These statements are in the public domain – to allow analysis, refinement etc (cf.OER)
• Pick&Mix has over 80 criteria…– But typically a set of 20-30 is chosen – this range is based
on analysis from the literature (ABC, BS etc) and experience in many scoring meetings
EFQUEL Academy 10
Pick&Mix Metrics
• Use a 6-point scale (1-6) – ignore 6 if you wish– 5 (cf Likert, MIT90s levels) plus 1 more for “excellence”
• Backed up by continuous metrics where possible• Also contextualised by narrative• The 6 levels are mapped to 4 colours in a “traffic
lights” system (red, amber, olive, green)
EFQUEL Academy 11
Pick&Mix
Three sample criteria
EFQUEL Academy 12
P01 “Adoption” (Rogers) – not used now
1. Innovators only2. Early adopters taking it up3. Early adopters adopted; early majority taking
it up4. Early majority adopted; late majority taking it
up5. All taken up except laggards, who are now
taking it up (or retiring or leaving)6. First wave embedded, second wave under way
(e.g. m-learning after e-learning)
EFQUEL Academy 13
P10 “Training”1. No systematic training for e-learning2. Some systematic training, e.g. in some projects and
departments3. Institution-wide training programme but little
monitoring of attendance or encouragement to go4. Institution-wide training programme, monitored and
incentivised5. All staff trained in VLE use, training appropriate to job
type – and retrained when needed6. Staff increasingly keep themselves up to date in a “just
in time, just for me” fashion except in situations of discontinuous change
EFQUEL Academy 14
P05 “Accessibility” – tough one
1. e-learning material and services is not accessible2. Much e-learning material and most services conform to
minimum standards of accessibility3. Almost all e-learning material and services conform to minimum
standards of accessibility4. All e-learning material and services conform to at least minimum
standards of accessibility, much to higher standards5. e-learning material and services are accessible, and key
components validated by external agencies6. Strong evidence of conformance with letter & spirit of
accessibility in all countries where students study
EFQUEL Academy 15
Pause for questions
EFQUEL Academy 16
Extensions, local criteria and slices
EFQUEL Academy 17
Extensions
• Pick&Mix started with 20 core criteria – but now there are over 80 – and a different core (2.0)
• There is room for more supplementary criteria – and further analyses are always being done to ensure topicality
• Web 2.0 aspects are already covered• But it is mainly the institutions who have kept
and will help to keep the system up to date• And some EU and other projects
EFQUEL Academy 18
Supplementary criteria - examples
• They include:– IT reliability– Market research, competitor research– IPR– Help Desk– Management of student expectations– Student satisfaction
EFQUEL Academy 19
Slices
• As well as benchmarking the whole institution it is wise in large institutions to look at a few “slices”:
• Useful to give a context to scores• Slices need not be organisational or subject-based– Distance learning (is a natural one)– Thematic or dimensional like HR, costs…
EFQUEL Academy 20
Scoring
• Scores are usually in the range 1-5, point 6 is exceptional and cannot be planned for
• Focus not only on best practice but also bad practice and real practice
• Do not try to ensure that your institution scores 5 overall
• 6 sometimes has aspects of 1
EFQUEL Academy 21
Use in Distance Learning (such as Virtual Schools)
details
EFQUEL Academy 22
Distance learning• A variant “mood” of the default Pick&Mix was
developed to benchmark distance learning• This was used in the Distance Learning
Benchmarking Club:– University of Leicester– Thompson Rivers University– Lund University– Royal Swedish Technical Institute (KTH)
• and also at the University of Gotland• Most recently at University of Northampton
EFQUEL Academy 23
Evidence and documentation• Normally institutions want a final report but but the
aim is NOT to recreate the QAA-style SED or DATs –the report is an outcome of the benchmarking process not the start of it
• Have a “file” for each criterion• Some narrative will be needed• Institutions normally group criteria according to their
own L&T strategy or in terms of “owning” departments– We also supply some standard groupings based on MIT90s
but few use these
EFQUEL Academy 24
Documentary support
• There are many reports available – many but not all are public, others are available to those participating in Pick&Mix
• The HE Academy wiki “was” a good source of information and the Academy blog structure “was to be” maintained indefinitely…. Not now
• We store all Pick&Mix key material separately from the HE Academy
• There are now many papers and a bibliography is available
EFQUEL Academy 25
Ongoing work
• Evolving Pick&Mix into a benchmarking and quality scheme for virtual schools:– With specialist help from EFQUEL– And a research team at KU Leuven– And correlation with iNACOL scheme in US
• Version of Pick&Mix ready to benchmark OER uptake (developed with Bieke Schreurs, OUNL)
• Variant moods for accessibility and employability (Wales)
EFQUEL Academy 26
Thank you for listening
email for this work is [email protected]
Benchmarking E-Learning in UK Universities: Lessons from and for theInternational Context , published in http://www.openpraxis.com/files/Bacsich%20et%20al..pdf
http://www.virtualcampuses.eu/index.php/Bibliography_of_benchmarkingandhttp://www.mendeley.com/groups/1075191/benchmarking-e-learning/