Benchmarking for technology enhanced learning: Longer term benefits
Associate Professor Michael SankeyDirector, Learning Environments and MediaUniversity of Southern QueenslandMember ACODE Executive
Introduction It is one thing to undertake Benchmarking in
TEL as a one-off activity. It’s quite another to build this into your strategy for
future and long-term growth. The paper reports on a follow-up study conducted in
March across 22 institutions who participated in the 2014 ACODE Benchmarking Summit on TEL.
Conducted 8 months after the event to see if it had been used this to build capacity for future growth.
It will provide evidence of the longer-term value of this type of activity to enhance institutional capacity in the rapidly changing TEL space.
www.acode.edu.au
ACODE
www.acode.edu.au
What is ACODE
ACODE's mission is to enhance policy and practice in open, distance and e-learning in the Australasian higher education sector by: disseminating and sharing knowledge and expertise; supporting professional development and providing
networking opportunities; investigating, developing and evaluating new
approaches; advising and influencing key bodies in higher
education; and promoting best practice..
Benchmarks
In 2014 we shifted the focus away from ‘eLearning’ to ‘Technology Enhanced Learning’ (TEL)
New Self-Assessment template
New guidelines for the use of these instruments
Bi-yearly opportunities for institutions to formally engage with this tool
www.acode.edu.au
The 8 Benchmarks for TEL1. Institution-wide policy and governance for technology
enhanced learning;2. Planning for institution-wide quality improvement of
technology enhanced learning;3. Information technology systems, services and support for
technology enhanced learning;4. The application of technology enhanced learning
services;5. Staff professional development for the effective use of
technology enhanced learning;6. Staff support for the use of technology enhanced
learning;7. Student training for the effective use of technology
enhanced learning;8. Student support for the use of technology enhanced
learning.
1st ACODE Inter-institutional Benchmarking Summit
Having the conversation
ACODE Benchmarking
June 2014 ACODE held a major Benchmarking Summit at Macquarie Uni using their Benchmarks for Technology Enhanced Learning (TEL).
This was unprecedented, with 24 institutions from 5 countries benchmark their capacity in TEL.
15 Australian, 6 New Zealand, 1 from UK, South Africa and Fiji.
A paper describing this event in more detail, along with the finding of a major evaluation survey was presented at last year’s ascilite (Dunedin).
Why the benchmarks?
Benchmarking in TEL is becoming an increasingly important part of how we can mediate a level of quality in our practices and align this with the practice of other institutions.
Further, many of the issues being faced by Australasian uni’s, particularly in the online space, are coming under increased scruitin: Few university policy cycles include the value-adding stages of
monitoring, evaluation and benchmarking. This suggests that many Australian universities will face challenges meeting the Australian tertiary sector regulators’ requirements regarding evidence of implementation of policy, and improvement of policy over time. (Freeman, 2014, P. 84)
To support the continuous QI of TEL. To integrate pedagogy, planning, staff dev &
infrastructure provision. To be used at an enterprise level, or business unit, or
self-assessment, or for a broader BM activity. Each has to participate in at least 2 benchmarks,
with some doing 3, 4 or 5, or all 8. Those involved first undertook a self-assessment,
then shared that with the others involved. Based on this, each institution was then able to
make a judgment call on their self-assessment. This allows us to form clear judgments around our
own practice, to then make a qualitative judgment as to the accuracy of our self-assessment.
Why and what?
Initial Eval - some basic stats
0
5
10
15
20
25
30
35
Strongly disagree
Disagree
Neither agree, or disagree
Agree
Strongly agree
N = 33
However
As with many things, once it’s all-over we go back to our day jobs and tend to forget about things like this – until it is time to do it again. But…
We wanted to get a better understanding of the longer-term benefits gained by participating, 1st in the internal activity, 2nd by attending the Summit and 3rd what happened after that.
In March 2015 we did a follow-up survey to understand the level of ongoing activity that may have been generated by participating in the Summit.
So 8 months on
25 participants from 22 (originally 24) institutions responded.
2 institutions did not participate as the people involved had left their institution.
The survey contained 7 questions, plus demographic data to allow for the alignment with the previous study.
Each was asked to comment on:1. their experience and how useful it had been
personally & for their institution,2. what they had done within your institution since the
activity. 3. how useful the ACODE follow-up documents had
been.4. how ACODE will would now facilitate a formal
benchmarking activity every 2 years. 5. the usefulness of an online tool to assist them in the
collection and reporting their data.6. whether these data should be shareable with other
participating institutions, and 7. the value in hearing what other institutions were
doing across all of these areas.
So 8 months on… There was still a significantly positive
affirmation of the usefulness of the activity both at a personal and institutional level.
All 22 had continued their affiliation with the tool, using it to base-lining ongoing TEL activities, or to provide a solid platform to advocate from.
There had been some follow-up activities within institutions, while others would return to using the benchmarks again in the near future.
1/3 said this had served as a confirmatory activity, particularly in support of their current direction. The other 2/3 had found them useful in providing evidence to their institution, with a view to promote further growth.
It’s was not just the Summit that is helpful, but the activities they did to prepare for this, and aligning this with their ongoing strategic approach.
In 5 of 22 cases formal reports had been written and presented to senior management, Although, most had brought this to the attention of their
senior managers. More common was using the findings for internal
reporting and informing future practice, at a practical and stategic unit level.
Not surprisingly, some institutions were undergoing some form of restructure, so the value was more localised.
ACODE provided a formal report on the activity, containing a description of the activity, a list of who undertook what and provided anonymised data.
The vast majority found this ‘very helpful’ and where not concerned by the data being anonymous.
When mooted that ACODE would run this event every 2 years, all agreed this was an excellent idea.
More importantly, the proposed addition of an online tool to help gather and collate benchmarking data was overwhelmingly supported (95%).
In agreeing to this each institution was also very cautious about making their finding too transparent.
At the Summit all institutions sat in on all the discussions, regardless of their Benchmarks.
Some said that this was too much, proposing alternate models for more focused discussions.
It was important to see, in planing the next event, if there had been any changes to this sentiment.
77% agreed it was better to sit in on all the sessions, 15% prefered more focused sessions, the remaining 8% made alternate suggestions.
Feedback has provided a clear way forward. Not only did institutions benafit from this activity, but they have incorporated much of their learnings in creating posative outcome for their institution.
Conclusion Many of the issues we face can be remediated by
simply taking the time to self-assess against a set of quality performance indicators.
When we extend that self-reflection by sharing our practices with those in similar circumstances, this provides for a truly dynamic learning activity.
This study confirmes there are both short and longer term benefits to benchmarking around TEL.
This is important, as regulatory bodies develop formal measures for ensuring institutions are meeting their obligations in relation to TEL.
An activity, like the ACODE Benchmarking Summit, provides an opportunity for institutions to meet the unique challenges of building a strong TEL future.
www.acode.edu.au
27-29 June 2016University of Canberra
Email: [email protected] to register your interest
2nd Inter-Institutional Benchmarking Summit