Date post: | 11-Apr-2017 |
Category: |
Education |
Upload: | association-of-university-administrators |
View: | 658 times |
Download: | 1 times |
www.ed.ac.uk
What do you hope to take away from this session?
www.ed.ac.uk
www.ed.ac.uk
www.ed.ac.uk
www.ed.ac.uk
www.ed.ac.uk
www.ed.ac.uk
•Transformative
•Step change •Culture change
www.ed.ac.uk
www.ed.ac.uk
www.ed.ac.uk
www.ed.ac.uk
www.ed.ac.uk
mainstreamed
• Pre-arrival and Induction
• Peer Learning and Support
• Student Surveys
• School Councils
• Applicant Experience
further temporary funding/activity
• Online and Innovation in Student Services – (Big White Wall)
• Healthy University
• Emergency Response
completed
• Student Community Engagement
• Programme and Course Information Management Project (PCIM)
funding ended
• Student Information Points
• Student Communication
Project strands www.ed.ac.uk/student-experience
www.ed.ac.uk
Key to our success
www.ed.ac.uk
From EUSA’s point of view, the project has been characterised by partnership working with the University with the shared goal of securing a positive and seamless experience for students as they navigate their way through their time here. The project has been unafraid to try bold and experimental approaches, and has entrusted EUSA as a full partner with resources to deliver new and exciting projects and events. Briana Pegado, EUSA President
SEP interim report 2014
www.ed.ac.uk
www.ed.ac.uk
Impact evaluation in practice
Terminology – what does impact really mean for us?
The Amosshe model
Asking the right questions
A framework of evaluation at Edinburgh
Have a go!
Impact Evaluation: What It Is; What It’s Not:
What It Is: What It’s Not:
Programme managers and policy makers as
focus on
“controlling and measuring the inputs and
immediate outputs of a program—how much
money is spent, how many textbooks are
distributed—rather than on assessing whether
programs have achieved their intended goals
of improving well-being”.
The Gertler. Et al (2011) World Bank
“Whether… the educational
experience/service is making a difference
to what they do and how.” (Amosshe 2011
p 9).
Clients can be satisfied with a
service, but this satisfaction
does not automatically reflect
that change or improvement
has taken place.
Impact Outcome
‘Harder to capture’
‘longitudinal’ ‘ big picture’.
‘Objective specific’ ‘shorter
timeline’ ‘adds to the big
picture’.
Forms Of Evaluation – What’s The Difference?
Forms of evaluation
Usage/count Satisfaction Impact
Examples of count data
sources:
Web analytics.
Attendance at session.
Attendance at events.
Numbers participating in
a project or initiative.
Numbers using a service.
Retention rates
Examples of satisfaction
source data:
Surveys (NSS; PTES; PRES)
Satisfaction based
questions in module
evaluation.
Examples of impact based
data sources:
Focus groups.
Impact based questions
in surveys.
Aligning impact questions
to key aims and
objectives.
Case studies. Quantitative Qualitative
Are you asking the
right questions?
Sample: ‘How To Measure a Student’s Engagement
With a New Student Newsletter?’
Usage/Counts:
How many hits were
there on the
newsletter website.
How many hits were
there to specific
articles.
Satisfaction:
On a scale of 1-10,
how satisfied were
you with the content
of the student
newsletter.
Impact:
As a result of reading the
student newsletter did you take
action on the following:
Make an appointment to
see your personal tutor.
Sign up for study skills
workshops.
How you
shape the
question can
gather the
evidence for
your intended
outcomes?
Impact
Satisfaction Usage/
attendance
What is impact evaluation and why should we adopt a logic modelling approach?
Benefits:
It ensures your objectives measurable and realistic at the start of your project.
Provides clarity about ‘what’ change you are trying to achieve, not just a focus on ‘how’ you are going to deliver your project.
Provide a complete, well rounded, robust and credible picture of what is being and has been achieved.
Communicating evaluation results can be very affirming and motivating for staff to see evidence of change and the outcomes they are achieving.
AMOSSHE Model:
Based on logic modelling.
Researched and created for Higher
Education.
Simple and Practical framework.
Can be adapted to any sort of project or
work stream.
Can be scaled up or down.
You Take Control – It’s Not Being Done To You
“Basically, a logic model is a systematic and visual way to present
and share your understanding of the relationships among the
resources you have to operate your program, the activities you
plan, and the changes or results you hope to achieve.”
Kellogg Foundation p.1 (2006)
Logic modelling
The Amosshe Framework:
OBJECTIVES: What are the intended achievements of the project?
OUTPUTS: What are the deliverables?
OUTCOMES: What change will take place?
IMPACT INDICATORS: How can you demonstrate that the objective has been met?
DATA: What data sources will be used to provide evidence
Worked example of an impact plan (AMOSSHE 2011 p 19)
Service/Activity: Supporting international students who are new to the country and University through an
orientation programme
Objective
Your intentions
To provide international students with the information and socialisation opportunities to
ensure a successful transition into University life
Outputs International students attend a programme
Outcome
What change?
International students are equipped with the sufficient information for a successful
transition into University life
Data indicators 1. International students have registered with a doctor;
2. Students are informed about and aware of a range of services, such as student
services, opening a bank account, visas, living in the UK;
3. International students are confident about where to go to find information;
4. Students are informed about and aware of socialisation opportunities at the university.
Data Sources 1. Online survey of those international students attending the orientation programme;
2. Focus group of international students attending the programme to test out how well
the programme has supported their transition into university life;
3. Focus group of international students who did not attend the programme to test out how well they have fitted into university life.
Exercise
In teams of 3, match the statements in this
hypothetical example to the headings using
the Amosshe framework
Discussion
www.ed.ac.uk
Decide on framework
Coach staff
Join the dots…
Think big….and segment
Support and initiate the
gathering of feedback
Report on results
Unified approach
Interrogate strategy, draw out outcomes & support planning
What data is available and how
can we influence the right questions?
What are we
evaluating?
Co-ordinate the
reporting
Who is the audience, how
should it be presented?
www.ed.ac.uk
Staff Feedback on Pre-Arrival and Induction
www.ed.ac.uk
Student Feedback More than 80% of
respondents say that they feel they belong to
the 'University of Edinburgh' after their
Welcome Week.
75% of students had joined a society, volunteering group or other activity by the end of the Welcome period.
Almost 60% of undergraduates felt motivated after Welcome Week to make their time at University
‘outstanding’.
95% students found the New
Student webpages
useful
95% of undergraduate students were
satisfied with their experience of the University so far. NEW STUDENT
SURVEY 2015
When asked most students said they felt either confident or excited after their first few weeks.
91% of postgraduate students were satisfied
with the Welcome Week experience.
99% of postgraduate researchers and 94%
of postgraduate taught students were
satisfied with their experience of the University so far. NEW STUDENT
SURVEY 2015
www.ed.ac.uk
Student Experience Project - Impact
Impact: www.uwp.is.ed.ac.uk/publish/php/pdf/php2tJHSB.pdf
www.ed.ac.uk
www.ed.ac.uk
“Your unhappy
customers are
your greatest
source of
learning”
Bill Gates
www.ed.ac.uk
Student Experience Project - Impact “They have provided extremely valuable insight into student interests, perception, anxieties, and concerns. Thinking about student learning, and facilitating discussion – has been useful in my own teaching.”
“… as academics we assume far too much and our expectations of them. . . importantly the SEP has made me more aware that being a student today has aspects of the experience that didn't exist when I first entered HE - some of these have introduced new stresses and pressures that need to be managed.”
“Better insight and knowledge regarding student issues, a trusted source of information and advice, a sense that the university is attempting to have a centralised pool of expertise, information and practice.”
www.ed.ac.uk
Student Experience Project - Impact
“ Because of the Project I can talk to prospective students with more pride and certainty about the experience we provide.”
“… students are benefiting from a more joined up and coherent student experience, and one which is focused on enhancement as well as managing the business as usual activity. I feel there is a collective impact of the project which is greater than the sum of its parts.”
BUT “I do not feel that they have supported my role. They have made it more difficult and demanding as the students now have much higher expectations - there seems to be no way of saying "NO" to a student in case it affects their experience and they maybe tell someone! “
“I'd like to make one point that I feel very strongly about, namely that, as an academic, I - and the School - do *not* deliver a service. I worry a great deal about a shift towards this type of service culture, as in my view it leads to a damaging and counter-productive model of the student experience. Our role is to help students learn, not to deliver a degree.”
www.ed.ac.uk
Student Experience Project - lessons learned
www.ed.ac.uk
Student Experience Project - lessons learned
www.ed.ac.uk
www.ed.ac.uk
Underpinning belief: (some of us) believe that learning and teaching is less important than research
New underpinning belief: We believe that learning and teaching is an unambiguous priority
www.ed.ac.uk
In pairs, consider how you might use this model in your own services and projects?
How might you use this in your own work setting?
QUESTIONS