Benchmarking for technology enhanced learning: Longer term benefits

Associate Professor Michael SankeyDirector, Learning Environments and MediaUniversity of Southern QueenslandMember ACODE Executive

Introduction It is one thing to undertake Benchmarking in

TEL as a one-off activity. It’s quite another to build this into your strategy for

future and long-term growth. The paper reports on a follow-up study conducted in

March across 22 institutions who participated in the 2014 ACODE Benchmarking Summit on TEL.

Conducted 8 months after the event to see if it had been used this to build capacity for future growth.

It will provide evidence of the longer-term value of this type of activity to enhance institutional capacity in the rapidly changing TEL space.

www.acode.edu.au

ACODE

www.acode.edu.au

What is ACODE

ACODE's mission is to enhance policy and practice in open, distance and e-learning in the Australasian higher education sector by: disseminating and sharing knowledge and expertise; supporting professional development and providing

networking opportunities; investigating, developing and evaluating new

approaches; advising and influencing key bodies in higher

education; and promoting best practice..

Benchmarks

In 2014 we shifted the focus away from ‘eLearning’ to ‘Technology Enhanced Learning’ (TEL)

New Self-Assessment template

New guidelines for the use of these instruments

Bi-yearly opportunities for institutions to formally engage with this tool

www.acode.edu.au

The 8 Benchmarks for TEL1. Institution-wide policy and governance for technology

enhanced learning;2. Planning for institution-wide quality improvement of

technology enhanced learning;3. Information technology systems, services and support for

technology enhanced learning;4. The application of technology enhanced learning

services;5. Staff professional development for the effective use of

technology enhanced learning;6. Staff support for the use of technology enhanced

learning;7. Student training for the effective use of technology

enhanced learning;8. Student support for the use of technology enhanced

learning.

1st ACODE Inter-institutional Benchmarking Summit

Having the conversation

ACODE Benchmarking

June 2014 ACODE held a major Benchmarking Summit at Macquarie Uni using their Benchmarks for Technology Enhanced Learning (TEL).

This was unprecedented, with 24 institutions from 5 countries benchmark their capacity in TEL.

15 Australian, 6 New Zealand, 1 from UK, South Africa and Fiji.

A paper describing this event in more detail, along with the finding of a major evaluation survey was presented at last year’s ascilite (Dunedin).

Why the benchmarks?

Benchmarking in TEL is becoming an increasingly important part of how we can mediate a level of quality in our practices and align this with the practice of other institutions.

Further, many of the issues being faced by Australasian uni’s, particularly in the online space, are coming under increased scruitin: Few university policy cycles include the value-adding stages of

monitoring, evaluation and benchmarking. This suggests that many Australian universities will face challenges meeting the Australian tertiary sector regulators’ requirements regarding evidence of implementation of policy, and improvement of policy over time. (Freeman, 2014, P. 84)

To support the continuous QI of TEL. To integrate pedagogy, planning, staff dev &

infrastructure provision. To be used at an enterprise level, or business unit, or

self-assessment, or for a broader BM activity. Each has to participate in at least 2 benchmarks,

with some doing 3, 4 or 5, or all 8. Those involved first undertook a self-assessment,

then shared that with the others involved. Based on this, each institution was then able to

make a judgment call on their self-assessment. This allows us to form clear judgments around our

own practice, to then make a qualitative judgment as to the accuracy of our self-assessment.

Why and what?

Initial Eval - some basic stats

0

5

10

15

20

25

30

35

Strongly disagree

Disagree

Neither agree, or disagree

Agree

Strongly agree

N = 33

However

As with many things, once it’s all-over we go back to our day jobs and tend to forget about things like this – until it is time to do it again. But…

We wanted to get a better understanding of the longer-term benefits gained by participating, 1st in the internal activity, 2nd by attending the Summit and 3rd what happened after that.

In March 2015 we did a follow-up survey to understand the level of ongoing activity that may have been generated by participating in the Summit.

So 8 months on

25 participants from 22 (originally 24) institutions responded.

2 institutions did not participate as the people involved had left their institution.

The survey contained 7 questions, plus demographic data to allow for the alignment with the previous study.

Each was asked to comment on:1. their experience and how useful it had been

personally & for their institution,2. what they had done within your institution since the

activity. 3. how useful the ACODE follow-up documents had

been.4. how ACODE will would now facilitate a formal

benchmarking activity every 2 years. 5. the usefulness of an online tool to assist them in the

collection and reporting their data.6. whether these data should be shareable with other

participating institutions, and 7. the value in hearing what other institutions were

doing across all of these areas.

So 8 months on… There was still a significantly positive

affirmation of the usefulness of the activity both at a personal and institutional level.

All 22 had continued their affiliation with the tool, using it to base-lining ongoing TEL activities, or to provide a solid platform to advocate from.

There had been some follow-up activities within institutions, while others would return to using the benchmarks again in the near future.

1/3 said this had served as a confirmatory activity, particularly in support of their current direction. The other 2/3 had found them useful in providing evidence to their institution, with a view to promote further growth.

It’s was not just the Summit that is helpful, but the activities they did to prepare for this, and aligning this with their ongoing strategic approach.

In 5 of 22 cases formal reports had been written and presented to senior management, Although, most had brought this to the attention of their

senior managers. More common was using the findings for internal

reporting and informing future practice, at a practical and stategic unit level.

Not surprisingly, some institutions were undergoing some form of restructure, so the value was more localised.

ACODE provided a formal report on the activity, containing a description of the activity, a list of who undertook what and provided anonymised data.

The vast majority found this ‘very helpful’ and where not concerned by the data being anonymous.

When mooted that ACODE would run this event every 2 years, all agreed this was an excellent idea.

More importantly, the proposed addition of an online tool to help gather and collate benchmarking data was overwhelmingly supported (95%).

In agreeing to this each institution was also very cautious about making their finding too transparent.

At the Summit all institutions sat in on all the discussions, regardless of their Benchmarks.

Some said that this was too much, proposing alternate models for more focused discussions.

It was important to see, in planing the next event, if there had been any changes to this sentiment.

77% agreed it was better to sit in on all the sessions, 15% prefered more focused sessions, the remaining 8% made alternate suggestions.

Feedback has provided a clear way forward. Not only did institutions benafit from this activity, but they have incorporated much of their learnings in creating posative outcome for their institution.

Conclusion Many of the issues we face can be remediated by

simply taking the time to self-assess against a set of quality performance indicators.

When we extend that self-reflection by sharing our practices with those in similar circumstances, this provides for a truly dynamic learning activity.

This study confirmes there are both short and longer term benefits to benchmarking around TEL.

This is important, as regulatory bodies develop formal measures for ensuring institutions are meeting their obligations in relation to TEL.

An activity, like the ACODE Benchmarking Summit, provides an opportunity for institutions to meet the unique challenges of building a strong TEL future.

www.acode.edu.au

27-29 June 2016University of Canberra

Email: secretariat@acode.edu.au to register your interest

2nd Inter-Institutional Benchmarking Summit