What makes a good evaluation?
Write your answer on a post-it
Public Engagement Masterclass 30: Some
ideas for evaluating your public engagement activity
Fran Marshall, Research & Evaluation Manager Dr Helen Featherstone, University of Bath
27/01/2016 © The University of Sheffield
Outline of session
• Evaluation aims, planning and methods – Fran
• Some examples of evaluation - Helen
Introduction to the project and Public Engagement and Impact Team
• Introduction to the Public Engagement and Impact Team
• The masterclass series
• What we can do to help?
27/01/2016 © The University of Sheffield
Masterclass series Masterclass title Date & time
20: Some ideas about how to evaluate your public engagement event Thursday 22 January 2015
21: What to think about when organising your public engagement event Tuesday 24 February 2015
22: Working in partnership with others
Wednesday 25 March 2015
23: Coordinating large scale public engagement, e.g. faculty based events, film or lecture series etc
Wednesday 29 April 2015
24: Public Engagement or Widening Participation/ Outreach? Things to think
about when organising events for children and young people
Wednesday 27 May 2015
25: Grant writing and impact Wednesday 24 June 2015
26: Writing a risk assessment for your event/ event planning Wednesday 29 July 2015
27: Engaging the public as researchers and different types of engagement Wednesday 30 September 2015
28: How to communicate your event
Wednesday 28 October 2015
29: Storytelling in public engagement Wednesday 25 November 2015
30: Some ideas about how to evaluate your public engagement event Wednesday 16 December 2015
27/01/2016 © The University of Sheffield
• The University’s Good Research and Innovation Practices policy
www.sheffield.ac.uk/ris/grip
• Minimum accepted practices in public engagement
• Higher practices to aspire to
27/01/2016 © The University of Sheffield
What makes a good evaluation?
Write your answer on a post-it
Who is here?
A. Academic staff
B. Non-academic staff
C. Post-graduate student
D. other
27/01/2016 © The University of Sheffield
Academ
ic st
aff
Non-aca
demic
staff
Post-g
raduate
student
other
14% 14%
38%34%
Which faculty are you from?
A. Arts & Humanities
B. Engineering
C. Medicine Dentistry and Health
D. Science
E. Social Sciences
F. Cross faculty
27/01/2016 © The University of Sheffield
Arts &
Hum
anities
Engineerin
g
Medici
ne Dentis
try a
nd H...
Science
Social S
cience
s
Cross
facu
lty
10% 10%13%
37%
3%
27%
How much do you know about planning evaluation?
A. A lot
B. Quite a lot
C. A little
D. Don’t know anything
27/01/2016 © The University of Sheffield
A lot
Quite a
lot
A little
Don’t kn
ow a
nythin
g
0%
32%
55%
13%
How much do you know about evaluation methods and when to use them?
A. A lot
B. Quite a lot
C. A little
D. Don’t know anything
27/01/2016 © The University of Sheffield
A lot
Quite a
lot
A little
Don’t kn
ow a
nythin
g
9%
34%
53%
3%
How much do you know about different kinds of evaluation?
A. A lot
B. Quite a lot
C. A little
D. Don’t know anything
27/01/2016 © The University of Sheffield
A lot
Quite a
lot
A little
Don’t kn
ow a
nythin
g
6%
32%
55%
6%
What is Public Engagement?
• “Public engagement describes the myriad of ways in which the activity and benefits of higher education and research can be shared with the public. Engagement is by definition a two-way process, involving interaction and listening, with the goal of generating mutual benefit”
National Coordinating Centre for Public Engagement
What is impact?
In relation to the Research Excellence Framework and Pathways to Impact:
• A change created by your research outside of academia
• From a small change in one person’s attitude to complete revolution in the way of life of millions worldwide
• Undertaking evaluation can help you demonstrate the impact of your project
27/01/2016 © The University of Sheffield
Where to start with evaluation…
27/01/2016 © The University of Sheffield
What is evaluation? • About the audience – show of hands
• Who has organised a public event before?
• Who has devised an evaluation?
27/01/2016 © The University of Sheffield
Monitoring or evaluation?
• Monitoring is counting
• Audience numbers, demographic information
• Evaluation is about critical reflection
• Can be: formative, about a process or summative
27/01/2016 © The University of Sheffield
When do we evaluate?
• Ideally before, during and after an activity
• In reality often done at the end - OK for single event but not for longer piece of research
• Includes looking at the quality of content, delivery and impact on participants
27/01/2016 © The University of Sheffield
Types of evaluation
• Formative: to support your activity, e.g. identifying gaps or barriers, gathering information to identify the need for your activity. Testing ideas and concepts – what is going to be most effective? What works best?
• Process evaluation: gathering information about how you can do it better next time
• Summative: i.e. a summary – final impact of your activity
• Is it a single evaluation or do you want base line and comparative data?
27/01/2016 © The University of Sheffield
Evaluation Tree
27/01/2016 © The University of Sheffield
What do you want to know?
27/01/2016 © The University of Sheffield
Start at the bottom
• Aims and objectives of event
• What kind of evaluation is it?
• What do you want to know?
27/01/2016 © The University of Sheffield
What do we need to evaluate? • Simple answer is the
public engagement activity you are doing
• More complex answer is what can we evaluate?
• What do you want to know v what can the audience tell you?
27/01/2016 © The University of Sheffield
Developing an evaluation plan
27/01/2016 © The University of Sheffield
Evaluation plan • Who do you want to evaluate?
• Who are your public?
• What resources are available for evaluation?
• What type of event is it?
• What questions should you ask?
27/01/2016 © The University of Sheffield
Who do you want to evaluate? • Audience
• Event leaders (University staff)
• Partner
• Volunteers
27/01/2016 © The University of Sheffield
Who are your ‘public’?
• Who are you expecting to attend your event?
• Design the evaluation activity around your audience
27/01/2016 © The University of Sheffield
Resources • Who will do your evaluation on the day?
• How much time do you have to analyse the data?
• What do you want to do with the data?
• How much money do you have for evaluation?
27/01/2016 © The University of Sheffield
Example costs
• Postcards - £35 for 1000
• A1 poster - £17
• Software for tablet survey - $49 per month
• Comment book - £10
• Google forms – free
• Surveymonkey - £25 per month
• Photographer - £75 per hour
• Plus pens, post-its etc
27/01/2016 © The University of Sheffield
What type of event is it?
27/01/2016 © The University of Sheffield
• Keep it simple and clear
• Should link to what you want to know
• Have a bottom line
• Ask the right questions
• Remember the unexpected
• Reflect on what you found out
27/01/2016 © The University of Sheffield
What questions should you ask?
27/01/2016 © The University of Sheffield
Asking questions
• Nothing is perfect – decide what fits best
• Keep it relevant
• Make it fun
• Don’t make it too onerous
• Put yourself in the participants' shoes
Evaluation questions exercise
1. How much do you earn?
2. Would you come to this venue again?
3. Do you have any problems with your boss?
4. To what extent do you agree - the Earth is like a spaceship with very limited room and resources?
5. Do you think URMS is a important method for calculating the fEC of the PI’s time in an ESRC grant application?
27/01/2016 © The University of Sheffield
Methods of collecting data
27/01/2016 © The University of Sheffield
Choose the method that can answer your questions and fits with your event
27/01/2016 © The University of Sheffield
Voting
Comments
27/01/2016 © The University of Sheffield
Any comments on the comments cards?
27/01/2016 © The University of Sheffield
Some comments from FotM
• So much to see! The 83 project is a real eye opener.
• Thermal imaging was fun and carnival. Was an immersive floor
• Great – very inspirational
• Thanks you so much for a wonderful performance. Really enjoyed it. Was great to see the power of music and dance and drama in action and witness the talent in this group! Also great to meet the Professors and staff behind it x
BUT
• I no want it anymore
• Bu bu bu bang!!!!
• My kittens enjoyed the exhibits (I brought him in my bag)
• I like the glass bowl guitar recorder be … is good
• Poor chat
• Peas
• The postit notes
• Carrots?
• The king of the cats
• Phew it smells in this building
27/01/2016 © The University of Sheffield
Social media and media coverage
• Google analytics for website traffic – help available through CiCS
• Storify: to share curated social media stories
• Twitonomy for twitter analytics
• Media reach
27/01/2016 © The University of Sheffield
• Focus groups
• Interviews
27/01/2016 © The University of Sheffield
27/01/2016 © The University of Sheffield
http://www.youtube.com/watch?v=8RlE8hp4WdI
Films and videos
Feedback video from masterclass 1
27/01/2016 © The University of Sheffield
“As a participant, I think it did work and reinforced (in other ways) some of the key issues of that early workshop, so doing the reflection/evaluation in that format was a useful process for me too”.
27/01/2016 © The University of Sheffield
27/01/2016 © The University of Sheffield
Some creative techniques
27/01/2016 © The University of Sheffield
Some creative techniques
Surveys and questionnaires – paper or online
27/01/2016 © The University of Sheffield
Arts & Humanities 9 14%
Medicine, Dentistry & Health
10 15%
Science 11 17%
Social Science 21 32%
Professional Services 8 12%
Other 7 11%
Any comments on the surveys?
27/01/2016 © The University of Sheffield
Observation and photography
27/01/2016 © The University of Sheffield
• Try to record your event with photos
• Observe your audience’s reaction • Count number of visitors • See what kind of engagement is
going on
After your evaluation
27/01/2016 © The University of Sheffield
Data analysis
• How will you analyse the data?
• Anything written on paper takes time
• Online surveys can be good for automated data analysis
• Make sure you think about the analysis at the start
27/01/2016 © The University of Sheffield
Example of output from iPad survey Why did you decide to come the event today?
I thought it would be enjoyable 15
Just passing/ looked interesting 10
I came with a friend 9
I know someone who is involved 8
To hear the speaker 4
To learn something new 3
other 8
Grand Total 57
27/01/2016 © The University of Sheffield
Summary from postcard feedback How much new information did you learn at the event?
A lot of new information 416
A small amount of new information 130
No new information 14
(blank) 19
Grand Total 579
27/01/2016 © The University of Sheffield
Reports • Assistance with planning
• Provides evidence
• Demonstrates value for money
• Reporting to funders
• Learning that can be shared with others and inform future activities
• Influence research
27/01/2016 © The University of Sheffield
Example report • Background to event
• Comments about how it was curated
• Budget
• Audience numbers
• Marketing
• Media coverage
27/01/2016 © The University of Sheffield
Example report • Comment about types of events and which
departments are involved
• Feedback from audience
• Feedback from event leaders
• Comments about what worked and what could be improved
27/01/2016 © The University of Sheffield
Link your findings back to the beginning
27/01/2016 © The University of Sheffield
Example of evaluation
27/01/2016 © The University of Sheffield
Festival of the Mind 2014
• Visitor numbers 26,000
• Visitor comments
• Website stats: Page visit, unique visitors, page views, pages viewed per visit, average visit duration, visitors using a mobile/tablet, non-mobile visitors, new or returning visitors and language settings
• Feedback from academics and partners involved
27/01/2016 © The University of Sheffield
Festival of the Mind evaluation
Visitors
• 579 postcards
• 57 surveys completed using iPads
• Online survey for pre-booked events (335 responses)
• Comment books
• Post-it note comments
• Economic survey for 1 event by external company
27/01/2016 © The University of Sheffield
Festival of the Mind evaluation
Event leaders and partners
• Online survey
• Small amount of interviews and event observation undertaken by an external company
27/01/2016 © The University of Sheffield
Feedback on social media
• Twitter followers: 3127 for main FOTM account with an additional accounts related to projects
• Facebook 1348 likes
• Blogs – over twenty different blogs set up or featured FOTM in the week of the festival.
27/01/2016 © The University of Sheffield
Dr Helen Featherstone, University of Bath
27/01/2016 © The University of Sheffield
Where can I go for help?
• Public Engagement and Impact Team
• Digital Engagement Team – social media
• CiCS – audience response system training, technical help and creative media
27/01/2016 © The University of Sheffield
Guidance
http://www.publicengagement.ac.uk/how/guides/evaluation/resources
http://www.rcuk.ac.uk/Publications/policy/Pages/Evaluation.aspx
http://www.publicengagement.ac.uk/how/guides/evaluation/examples
https://www.sheffield.ac.uk/life - evaluation reports
27/01/2016 © The University of Sheffield
Evaluation
What have you learnt?
How much have you learnt about planning evaluation?
A. A lot of new information
B. A small amount of information
C. No new information
D. Don’t know
27/01/2016 © The University of Sheffield
A lot o
f new
info
rmatio
n
A small a
mount o
f info
r...
No new
info
rmatio
n
Don’t kn
ow
50%
0%0%
50%
How much have you learnt about evaluation tools and when to use them?
A. A lot of new information
B. A small amount of information
C. No new information
D. Don’t know 27/01/2016 © The University of Sheffield
A lot of n
ew inform
ation
A small amoun
t of infor...
No new in
form
ation
Don’t know
42%
0%
8%
50%
How much have you learnt about different kinds of evaluation?
A. A lot of new information
B. A small amount of information
C. No new information
D. Don’t know 27/01/2016 © The University of Sheffield
A lot o
f new
info
rmatio
n
A small a
mount o
f info
r...
No new
info
rmatio
n
Don’t kn
ow
57%
0%
9%
35%
Opportunities for engagement – what is coming up?
• Life festival (April 2016)
• Festival of the Mind (September 2016)
• Off the Shelf (October 2016)
• Festival of Social Science (November 2016)
• Doc/Fest (March and June 2016)
27/01/2016 © The University of Sheffield
For further information contact:
@UniShefEngage
#GetEngaged
27/01/2016 © The University of Sheffield
Questions
27/01/2016 © The University of Sheffield
To Discover And Understand.