Post on 15-Jul-2015
transcript
Module Evaluations, Institutional
Responses: Using Data to Focus Improvements in
the Student Experience
Dr. Peter Bird Learning Technologies
Dr. Rachel Forsyth Centre for Learning and
Teaching
Newcastle | Feb 12th 2015
Session Format
• Institutional Context
• The EQAL transformation program
• Use of course and module data to measure
impact
• Group Exercise to develop further impact
measures
• Plenary Discussion
• Key Lessons
Context | An interesting place to work
• Large, popular, multi-campus university
– 36,000+ students | 1,000+ courses | consolidating from 7 sites
• Pursuing an ambitious transformation initiative
– New Buildings, New Curriculum, New Admin Processes & Systems,
New Learning & Mobile Tech, New Quality Processes…
Problems impacting students and
staff• Lack of standardization of course units
• Course units with too many assessments
• Timetabling issues
• Quality of Feedback
• Disconnect between library resources and course
needs
• Robust but complex effort intensive QA
processes
Coordinating our response
Improve satisfaction, retention & success
Great
Online Experience
(Seamless, personalised)
Great
Learning Spaces
Great
Teaching
Simplify
Unwieldy
Curriculum
£350M
• A coordinated strike for step-change improvement
EQAL
New Curriculum• designing new units, …
New Admin Systems & Processes• personal timetabling, …
New Virtual Learning Environment• Moodle & myMMU web/mobile, Talis Aspire…
New QA & QE Processes• facilitating curriculum transformation
In the current climate
Diminishing unit of resource
Everything depends on everything else
We are large and risk averse
MISSION:
868L3+L4modules
9
Wrapping the institution around the learner
Student ID
Timetable
This work is licensed under a Attribution-NonCommercial-ShareAlike 2.0 licence
Sync to
personal
device
Summary | we cleared a
path to the information our
learners needed…
Engaging our students
Providing business
intelligence
Automating our
processes
Are we in a better place now?
… we can now begin to
interpret some digital
footprints
Using the data
2. Identify relevant data
sources
3.Summarise individual
data sources
4 Join on common identifiers
5 Prep the data for analysis
6.Analyse & visualise
1.Appreciate the issue
Refining
understanding of a
problem space
Understanding student satisfaction
2. Identify relevant data
sources
3.Summarise individual
data sources
4 Join on common identifiers
5 Prep the data for analysis
6.Analyse & visualise
1.Appreciate the issue
Refining
understanding of
student satisfaction
Student
demographics
Entry profiles
Internal survey
responses
Join on
student ID
Random
Forests
analysis
Satisfaction
predictors
↗ Confidence
↘ Organisation
→Teaching
Sum entry
points &
qual type
Handle missing
values, collapse
categories…
Internal student survey
2011/12
Things students found useful…
“…At the beginning we talked through the assessment criteria in
tutorials - decoding it. This was very helpful! I now know what is
expected of me…” (MMU Internal Survey. Spring, best things, 2012).
“Clear teaching and materials are given out at the start of the unit and
assignment available early on so can be started on quickly.” (MMU Internal Survey. Spring, best things, 2012).
Good reasons for clear briefs…
“[Tutor name] has been off and we have been given a substitute lecturer who
although is very keen and clearly very intelligent …no body on the course had
no idea about the … project so we have had no help at all since week 1 and
subsequently our marks could be low - very disappointing since we pay for this!”
Things to improve, Winter 2011
“Again there was much inconsistency with marking between the three tutors
teaching this course and while I had an excellent tutor in *Teacher name A*
friends who were in other groups (with *Teacher name B* or *Teacher name C*)
were being given entirely different instructions for the same assignments and
even appeared to be being marked to different criteria.” Things to improve,
Spring 2012
Suggestions for improvement
“Set of notes provided plus slides uploaded onto moodle. Clear explanation
of what is expected of us for each assignment and assessment and from
what we will have an exam at the end.” Winter, improve 2011
“…Last term I went to one of them with my work so far and they said it was
fine and meeting the assessment criteria. However I only got 58%. how is
this mark fine?! and why didn’t they guide me in the right direction when I
went for help?...” Spring, improve 2012
Time to look at some data….
Are we in a better place now?
Getting better but further to go…
Change management lessons1. Needs Hands-on management to unblock the big issues
2. Communication/Coordination between linked sub-projects vital
3. Participative inclusive approach
4. Make ‘big asks’ as small as possible
5. Set-up strong links into faculties
6. Use data responsibly not as a big stick
7. Work with staff in workshops to interpret data, identify the root
problem(s) and jointly develop improvement plans.
http://tinyurl.com/jiscmmusrcCurriculum Design
Assessment & Feedback http://tinyurl.com/jiscmmutraffic