Date post: | 11-Jan-2016 |
Category: |
Documents |
Upload: | leo-morris |
View: | 224 times |
Download: | 0 times |
v
MEASURING IMPACT: USING QUANTITATIVE RESEARCHEibhlín Ní Ógáin
London, November 2012
AGENDA
1 What is impact measurement
2 NPC’s approach
3 Quantitative approaches
4 Tips and things to remember
TRANSFORMING THE CHARITY SECTOR
Increasing the impact of charities
eg, outcomes frameworks
Charity
Funder Sector
We are a think tank and
consultancy working at the nexus between charities and
funders
Increasing the impact of funders
eg, effective commissioning
Strengthening the sector
eg, collaboration towards shared
goals
WHAT IS IMPACT MEASUREMENT?
5
Impact: The difference an organisation or activity makes to the people or problem it aims to help
Impact measurement: Planning, managing, tracking and reviewing how much difference you are making
NPC’S APPROACH TO MEASURING SOCIAL IMPACT
7
NPC’S APPROACH TO MEASURING SOCIAL IMPACT
Strategic vision / goals
A well-developed Theory of Change
Existing evidence Appropriate measurement tools
QUANTITATIVE RESEARCH
8
9
WHY USE QUANTITATIVE RESEARCH
• Qualitative: understand something in detail. Rich information and insight.
• Quantitative: how many people? How large an effect? How much impact and how robust?
• Use quantitative research when dealing with large numbers of people and when you want to show a consistent effect.
• E.g does new type of psychotherapy lead to improved mental well-being? Need consistent measure and need large sample.
• Depends on type of ‘intervention’ and questions you are trying to answer.
• Know a lot, measure a little, know a little, measure a lot.
10
DIFFERENT METHODS IN QUANTITATIVE RESEARCH
• Ranges from easy to measure to hard to measure
• Easier: Hard and objective outcomes: move into employment, gain a qualification, reduction in reoffending.
• Harder: Softer, difficult to observe outcomes: changes in feelings, personal growth, well-being.
• NPC’s preference is for previously tested scales and questionnaires.
• If there is a gap, develop own questions.
11
TYPES OF QUANTITATIVE TOOLS AND THEIR USES
11
Type of toolIdentify needs of group
Measures progress for individuals
Aggregate to show change
Robust measure of change
Explain why change has happened
Admin. data ? ? ? Case work tool (eg, Recovery Star) ? Clinical tools Scales (developed by researchers)
Single questions
Hard outcomes (eg, statutory data)
12
WHEN TO MEASURE
• Pre-post design: how outcomes change over time by collecting data before (pre) and after (post) the programme.
• When you think change happens influences your decision about:
– When to collect the before measure
– When to collect the after measure
12
Before measure
After measure
Programme / service
13
THINK ABOUT ATTRIBUTION
• Is the programme the only factor influencing any changes? What would have happened anyway?
• Control groups
• Compare before and after data for participants and non-participants, ideally matching on their key characteristics
• Holy grail: Randomised Control Trial, where potential participants randomly assigned to control group
13
14
THINK ABOUT SAMPLING
• Representativeness
– Does your sample represent your population?
• Response rates
– Is your response rate high enough to avoid bias?
• Statistical validity
– Are your sample sizes large enough for analysis?
• Oversampling (segmentation)
14
Biased/small sample => Avoid percentages
15
YOU DON’T NEED TO MEASURE EVERYTHING
16
WHAT: BEFORE YOU THINK ABOUT COLLECTING DATA ASK YOURSELF:
Has anyone already proved the causal link you want to make?
Yes
No
Is it really important you need data on this outcome?
Do you already collect this data?
Don’t collect unless you decide it is really important
Use research to limit your data collection
Yes
No
Yes
Can you access this data?
No
Develop your own data source
Use this data (if it is fit for purpose)
Does anyone else collect this data?
NoNo
Yes
Yes
17
PRIORITISE OUTCOMES THAT:
• you directly influence (rather than indirectly support)
• are important / material to your mission
• are not too costly to measure
• will produce credible data?
18
HOW ROBUST DOES EVIDENCE OF YOUR SOCIAL IMPACT NEED TO BE?
• What do your target stakeholders (investors?) think?
• What is possible, given your resources etc?
19
LEVELS OF EVIDENCE
Randomised control trial
Anecdotes / quotes
Before and after survey
Self-reported change
Case studies
Control groups
Credibility
Basic Advanced
Nesta level 1 Nesta level 2 Nesta levels 3, 4, 5
20
6 TIPS FOR MEASURING YOUR IMPACT
2. Keep it simple. Concentrate on what you really want to know and don’t try to be too ambitious in the scope of your evaluation.
1. Decide what’s important. Define your research question and be clear what you want to achieve.
3. Know your limits. You need to choose an approach that matches your resources.
4. Consider investing in training. If you are not confident, get training or seek external help.
5. Think about the future. Don’t build a system that will quickly go out of date and look at the year-on-year costs.
6. Use the information that you collect: If you don’t use it, don’t measure.
21
RESOURCES
• Review of evidence of ICT and learning: http://educationendowmentfoundation.org.uk/toolkit/ict
• A good website to look for measures and indicators: http://wilderdom.com/tools/ToolsSummaries.html
• How to communicate your results: Hedley, S et al (2010) Talking about results. New Philanthropy Capital.
• Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco.
• http://www.theoryofchange.org/• The innovation network’s logic model workbook:
http://www.innonet.org/client_docs/File/logic_model_workbook.pdf• The Centre for What works (http://www.whatworks.org/) has an outcome portal where
you can browse different outcomes and their corresponding indicators. • TRASI (http://trasi.foundationcenter.org/) has a database where you can search
different approaches to impact assessment. • NPC has a page on shared measurement where it outlines some indicators for different
sectors. http://www.philanthropycapital.org/publications/improving_the_sector/measuring_together.aspx
22
RESOURCES
• Survey question bank: A website where you can research widely used surveys and single questions by theme or using key word searches. http://surveynet.ac.uk/sqb/
• CES’s guide to theory of change: http://www.ces-vol.org.uk/downloads/makingconnectionsusingatheoryofchangetodevelopplan-800-808.pdf
• Forum for Youth Investment’s guide to measuring soft skills in youth programmes: http://www.forumfyi.org/files/Soft_Skills_Hard_Data.pdf
• A list of the most common psychological scales and questionnaires: http://www.ull.ac.uk/subjects/psychology/psycscales.shtml
• Harvard Family Research Project’s tools for out-of-school time programmes: http://www.hfrp.org/out-of-school-time/publications-resources/measurement-tools-for-evaluating-out-of-school-time-programs-an-evaluation-resource2
• Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco.
• Ritchie et al. (2003) Qualitative Research Practice – A Guide for Social Science Students and Researchers. Sage Publications Ltd
• Measurement and assessment tools for family and parenting interventions: http://www.actionforchildren.org.uk/media/3986553/outcome_tools_1.pdf
vEibhlín Ní Ógá[email protected]