Date post: | 14-Jan-2015 |
Category: |
Education |
Upload: | field-furthering-inclusive-learning-and-development |
View: | 745 times |
Download: | 0 times |
Are we making a difference?
Measuring the outcomes of disability related services and supports
Presenter: Dr Erin Wilson
School of Health and Social Development, Deakin University
Acknowledgements: Dr Nick Hagiliassis and
Dr Robert Campain, Scope
Background- what do I bring?
• Worked with Scope; Perth Home Care Services; WA Mental Health
Commission and others
• Worked with team of researchers to develop a range of data
collection and outcome measurement tools in a range of contexts:
– Individualised funding and planning
– Early Childhood Intervention Services
– Flexible funding in early childhood
– Therapy and psychology services
– Community based mental health services
– Assistive technology
– Range of disability service models
Selection of data collection and analysis tools developed and trialed
Scope Outcomes Framework (used for analysis) • A whole of life outcomes framework across seven life domains
Scope Outcomes and Impacts tool • A one page survey rating scale to measure impact (positive and negative) of service on life
domains of individual or family
Individualised Planning/Review Tool • A tool based on the categories of a person centred plan to rate achievement against goals
Measurement of Services and Supports (MOSS) tool • A pre and post measure of change resulting from individual goal setting and services based on
this goal
Family Stress and Coping tool AND Family Capacity tool • Two survey scales rating a family’s self perceived level of coping and capacity
Early Childhood Individualised Support Plan tool • Attachment to Individualised Support Plan to rate outcomes at end of service period, and
barriers and enablers to these
1 in 4 poll Social Inclusion Survey • A population level survey rating levels of social inclusion related to service exclusion, economic
exclusion and social exclusion
Overview
• Key terms
• WHAT outcomes might we use – selecting and defining outcomes
• HOW to measure outcomes – guiding principles
How do we know if we’re making a difference?
3 key ideas/terms
•Process
•Output
•Outcome
Terminology – key elements What are ‘outcomes’?
In broadest terms, ‘outcome’ means: what happened
BUT different things are focused on and there is confusion over
terms and their relationship:
1. PROCESS: did we receive or deliver a quality service? The ways in
which program services and goods are provided. Practices used.
2. OUTPUT: how much service did we receive or deliver, how many
people received it etc? The goods and services produced (quantity,
efficiency: e.g., number of people served, speed of response to
complaints).
3. OUTCOME: what happened (for the person)? The broader results
achieved through the provision of goods and services (Horsch, 2005)
• Under the proposed NDIS, it is possible that measurement will occur across
outcomes, but also outputs and processes.
– e.g., outputs - number of clients serviced, respite hours, equipment payments,
home and vehicle modification data;
– e.g., process – degree of service coordination, how disputes are managed.
• Measuring process or output is frequently used as a ‘proxy’ measure of
outcome
• This is based on the thinking (and sometimes research) that where a
given process is used (or level of output occurs) this leads to the
desired outcome (eg. Support to exercise leads to better health –
measure whether support to exercise provided)
• Department of Human Services Standards – largely oriented to
measure ‘good practice’ i.e process
• Output or Process measures ≠ Outcomes.
– i.e., They do not reveal what happened as a result of the service or
intervention or activity.
Steps in developing an outcomes measurement approach for human services
1. Decide the desired areas of change – i.e desired outcomes and for whom
2. Define these areas & unpack complex terms – eg. what do we mean by ‘communication competence’, ‘awareness’,
‘community inclusion’, ‘support’, ‘partnership’?
3. Identify the indicators of these changes – how will we know, what will we see if change occurs?
4. Decide how these indicators can be measured
– How to collect data. How to analyse data. Do measures exist? Can existing measures be modified? Do measures need to be developed?
5. Combine and present the resulting information in a clear and informative way. (Adapted from ABS, 2001, ch 1).
1. Decide the desired
areas
2. Define these areas & unpack
complex terms
3. Identify the
indicators
4. Design how to
measure
5. Present information
Steps of outcomes measurement Task
1. Decide the desired areas of change (what are the outcomes?)
Check if specific outcome areas are identified by funder, government policy, or by others such as individuals who set specific goals for services. Consult stakeholders (practitioners and service recipients)
2. Define these areas & unpack complex terms
Check published literature for concept definition and sub domains of concepts. Consult stakeholders
3. Identify the indicators of these changes Check published literature to identify indicators; consult with stakeholders
4. Decide how these indicators can be measured
Check published literature for existing data collection instruments and methods; consult stakeholders; design methods
5. Combine and present the resulting information
Report and publication writing
Step 1: decide the desired areas – YY
Which processes should we measure? Processes / practice approaches we might measure:
– Family centred practice • (Process measurement tool is Measure of Processes of Care (MPOC), King,
Rosenbaum & King, 1995)
– Person centred practice
– Individualised service delivery
– Strengths based practice
– Recovery based practice
– Etc, etc
Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.
An example: Family Centred Practice
1. Decide the desired
areas
2. Define these areas & unpack complex
terms
3. Identify the
indicators
Family Centred Practice
5 Domains: 1. Enabling and
Partnership 2. Providing general
information 3. Providing specific
information 4. Co-ordinated and
comprehensive care 5. Respectful and
supportive care
(King, Rosenbaum and King, 1995)
Enabling and Partnership e.g., Parents felt like partners in their child’s care
Providing general information e.g., Parents received adequate information about their child’s disability
Co-ordinated and comprehensive care e.g., Looking at the needs of the ‘whole’ child (e.g. Mental, emotional and social needs) instead of just physical needs
Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.
An example: Department of Human Services Standards
1. Decide the
desired areas
2. Define these areas & unpack complex
terms
3. Identify the
indicators
Department of Human Services
Standards (VIC)
4 Areas: 1. Empowerment 2. Access and
Engagement 3. Wellbeing 4. Participation
(DHS, 2012)
Empowerment e.g.,
Rights and responsibilities are developed and provided in an accessible format that facilitates understanding by all people
Access and Engagement e.g.,
Service-delivery hours are responsive to the needs of people accessing the service
Wellbeing e.g., People actively participate in an assessment of their strengths, risks, wants and needs
Participation e.g., The service provider supports people to develop and maintain their personal, gender, sexual, cultural, religious and spiritual identity
Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.
An example: Person Centred Practice
1. Decide the desired
areas
2. Define these areas & unpack complex
terms
3. Identify the
indicators
Person Centred Practice
Domains / major areas:
??
??
Step 1: decide the desired areas of change – yy
Which outcomes should we measure?
Outcomes for whom?
• Person with a disability
• Family /carer of person
• Service provider
• Service sector
• Community members/organisations
• Wider society and social structures
Frequently capacity building and social change outcomes
Step 1: decide the desired areas of change – yy
Which outcomes should we measure?
• What outcomes were seen as
important by panel?
• What others would you add?
• What are the most important?
Step 1: decide the desired areas of change -Which outcomes should we measure?
• Outcomes for persons with disability might include:
– Quality of life
– Social inclusion
– Participation
• (community participation, cultural participation, social
participation, political participation, economic participation….)
– Satisfaction (with services, with life…)
– Loneliness
– Autonomy, independence, choice….
– Etc, etc
What are the outcomes sought under the NDIS?
“An important component of the evidence base under the
NDIS will be the systematic collection of data on outcomes
of particular services or interventions for people with
disabilities. Data on outcomes in employment, education,
social participation, and capacity for self-care, and on the
measures that contributed to those outcomes, would help
to build an evidence base for analysing which interventions
or forms of assistance are more effective, and why”. p 576, Productivity Commission Report, Disability Care and Support, 2011
What are the outcomes sought in other important policy documents and Conventions?
Convention on the Rights of Persons with Disabilities (UN, 2006): • Full and effective participation of all people with disabilities in all areas of community
and economic life
• equality, autonomy, independence, opportunity, accessibility, identity, participation
and inclusion.
National Disability Strategy: • Inclusive and accessible communities
– Proportion of people using public transport; proportion participating in common cultural and
recreational activities
• Rights, protection, justice and legislation – Feelings of safety; participation in civic life
• Economic security – Average income; housing stress; participation in labour market
• Personal and community support – Access to and unmet demand for disability services, etc
• Learning and Skills
• Health and Wellbeing
Step 2: Define these areas & unpack complex terms – eg. what do we mean by ‘social inclusion’
Step 3. Identify the indicators of these changes – how will we know, what will we see if change
occurs?
2. Define these areas & unpack complex
terms
3. Identify the
indicators
Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.
An example: Social Inclusion (as used in 1 in 4 poll)
1. Decide the desired
areas of change
2. Define these areas & unpack complex
terms
3. Identify the
indicators
Social inclusion
Social exclusion Service exclusion
Economic exclusion
(Saunders et al 2007a, 2007b & 2008)
Social exclusion e.g., Having social contact with other people
Service exclusion e.g., Accessing medical services
Economic exclusion e.g., Having enough money to get by on
The 1 in 4 Poll - Social Inclusion Survey (example items from on line and paper based survey in Easy English)
4. Design how to
measure
Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.
An example: Quality of life
1. Decide the desired
areas of change
2. Define these areas & unpack complex
terms
3. Identify the
indicators
Quality of Life
1. Emotional wellbeing 2. Interpersonal
relations 3. Material wellbeing 4. Personal
development 5. Physical wellbeing 6. Self determination 7. Social inclusion 8. Rights
(Schalock, 2004)
Emotional wellbeing e.g., Contentment (moods, enjoyment); Self concept (eg self esteem)
Material wellbeing e.g., Financial status (income, benefits); Employment; Housing Physical wellbeing e.g., Health; Activities of daily living; leisure
Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.
An example: Outcomes Star (Services Connect)
1. Decide the desired
areas of change
2. Define these areas & unpack complex
terms
3. Identify the
indicators
Outcomes Star
Services Connect
1. Motivation and taking responsibility 2. Self care and living skills 3. Managing money and personal
administration 4. Social networks and relationships 5. Drug and alcohol misuse 6. Physical health 7. Emotional and mental health 8. Meaningful use of time 9. Managing tenancy and
accommodation 10. Offending (Triangle, n.d)
Reflections on WHAT outcomes to measure: Guiding principles
1. Outcomes selected must be meaningful
• Must capture what is important to people most
affected.
– Consider whole of life rather than focus on single aspects
– Consider broad outcomes list to include ALL that is
important to people
– Consider allowing individuals to name own outcomes (eg
goal focused) – able to be configured to the individualised
service and support context.
Example: A Whole Of Life, Whole Of Person Approach
Scope Outcomes Framework (Wilson, 2006)
• Encapsulates nine life areas:
Personal, Social, Economic, Political, Educational, Recreation and Leisure, Cultural, Spiritual, Environment
• Reflects
– multidimensional aspects of a person’s life
– move away from outcomes measurement approaches that have a narrower focus, e.g., those that focus on physical functioning exclusively.
– whole of life/person approach (not just the piece of it the service addresses).
• Drawn from a wide range of literature (and concepts) such as:
– Personal Outcomes Measures (Accreditation Ontario, 2000),
– National Core Indicators (Human Service Institute, 2003),
– Quality of life and subjective well being concepts (e.g., Cummins, 2007; Shalock,
2004; International Wellbeing Group,2005),
– wide range of other literature (happiness, friendship, social capital, etc),
– lived experiences of clients, families and communities involved in Scope services.
Scope Outcomes Framework / Life domains model (Wilson, 2006)
Personal and family wellbeing
That is, in the area of health and function, happiness, wellbeing, safety, sense of independence and choice.
Social life That is, in the area of friendship and relationship, community involvement and sense of belonging
Political life That is, in the area of having a say about things that affect you (eg in a local service or community group, about your area, school, funding etc
Cultural life That is, being involved in cultural activities (eg. arts, music, theatre, dance at any level). This might be through attending activities or playing an active part. Or being part of your own cultural group
Recreational and leisure life
That is, being involved in recreational or leisure activities at any level. This might be through attending activities, playing an active part or doing what you enjoy
Economic life That is in terms of your finances, employment or business.
Educational life That is, any aspect of your family’s education, training, personal or professional development.
Spiritual life That is, any aspect of your family’s religious or spiritual activities
Your environment
That is, your family’s access to and enjoyment of public spaces (eg parks, pools, theatres, shopping centres, public transport etc) or your own private space (eg. your home). Making environments you use more accessible and appropriate (eg playgroups, kindergartens, library, etc).
Outcomes across whole of life focus: Outcomes and Impacts tool based on the life domains model (Wilson, 2006)
Thinking about whole of life or breadth of outcome coverage Scope/Wilson life domains Personal Outcome Measures (Council on Quality Leadership, 2012)
Personal and family wellbeing
People are safe People have the best possible health People exercise rights People are treated fairly People are free from abuse and neglect People experience continuity and security People decide when to share personal information People choose where and with whom they live People choose services People choose personal goals People realize personal goals
Social life People are connected to natural support networks People have intimate relationships People interact with other members of the community People perform different social roles People participate in the life of the community People have friends People are respected
Political life
Cultural life
Recreational and leisure life
Economic life People choose where they work
Educational life
Spiritual life
Your environment People use their environments People live in integrated environments
Reflections on WHAT outcomes to measure: Guiding principles continued
2. Information collected should be able to guide
improvement
– “The point is not to collect information - it is to learn enough
about how things are going so we can keep going or change
direction with confidence. We want monitoring which leads to
action to improved… [practice], and to have a positive impact on
people’s lives” (Ritchie et al, 2001: 177)
– Identification of factors affecting outcomes: enablers, barriers.
2. What worked for you? What helped you achieve your goal? • People, activities, equipment/aids/resources, finances, environment, policies, time. What else? • Which of these was the biggest help? Comments from person with a disability Comments from others (please specify role, eg. paid support worker, family): Practitioner comments:
3. What stopped you from achieving your goal? What made achieving your goal difficult? • People, activities, equipment/aids/resources, finances, environment, policies, time. What else? • Which of these was the biggest barrier? Comments from person with a disability: Comments from others (please specify role, eg. paid support worker, family): Practitioner comments:
Example from MOSS: Enablers and Barriers focus
Step 4: Design HOW these indicators can be measured
Guiding principles for HOW to measure:
1. Self report
2. Achievable
3. Accessible
4. Effective
See also Andresen, 2000
4. Design how to measure
HOW to measure: Guiding principles
1. Self report of person with disability (or those most
affected, e.g. carer)
– Capture the view point of a person with a disability
– Create alternative ways a person can give information directly
about themselves
• E.g. Use of multi media
• Provide a scribe to assist write, type or enter data
– If must use proxy to report, give explicit instruction to report
from the viewpoint of person they represent
– Be clear who is providing data
• Identify different respondent views – distinguish between person
with disability (service recipient), staff, and others
– Have a mechanism for informed consent of the person
Examples of self report
• Ensuring the perspective of the person with a disability is captured – explicit instruction to proxies;
– 1 in 4 poll:
67.4% of respondents completed the survey independently;
15% had carers complete the survey on their behalf.
\
The survey is being completed by:
A person with disability without assistance
A person with a disability with assistance
A carer of person with disability (someone speaking
on behalf of a person with a disability who cannot
express their views themselves)
Where support is provided, the survey should be completed from the person with a disability’s view point
Examples of self report
• Ensuring the perspective
of the person with a
disability is captured;
– MOSS:
offers three rating scales in
different formats to enable
individuals to select which
one is meaningful to them.
Accessible for some people
with intellectual disability
\
2. What worked for you? What helped you achieve your goal? • People, activities, equipment/aids/resources, finances, environment, policies, time. What else? • Which of these was the biggest help? Comments from person with a disability Comments from others (please specify role, eg. paid support worker, family): Practitioner comments:
3. What stopped you from achieving your goal? What made achieving your goal difficult? • People, activities, equipment/aids/resources, finances, environment, policies, time. What else? • Which of these was the biggest barrier? Comments from person with a disability: Comments from others (please specify role, eg. paid support worker, family): Practitioner comments:
Example from MOSS: Different viewpoints
HOW to measure: Guiding principles
2. Achievable
– Brief and easy to use
– Avoids administrative burden
– Considers time, workload, training, data analysis allocations
– Considers BOTH how data will be collected AND how data will analysed
HOW to measure: Guiding principles
3. Accessible
– Instrument can be used (with and without support by others) by
people with a wide range of disabilities including people with
intellectual disabilities;
– Has alternative formats:
• Easy English format and/or visual supports;
• simple visual scaling;
• physical administration demands low;
• allows for simple verbal responses (e.g., ‘yes’ or ‘no’);
– Considerations to item/instrument bias; reliability, validity vs
internal inconsistency; acquiescence;
– Culture/ language adaptations.
Examples of accessibility
• Value in engaging Easy English experts to
translate content
• Checked by people with intellectual disability
E.g. 1 in 4 poll:
High ‘user friendliness’ ratings (mean rating 8.5)
Examples of accessibility MOSS tool
• offer simplified choices and rating scales
HOW to measure: Guiding principles
4. Effective
– Items well match the outcome concepts identified (concept capture
and validity)
– Indices of change/measurement are responsive – sensitive to and
able to capture small changes
– Considers timing of data collection – when will outcomes be
achieved and apparent? (at moment of intervention, long term)
– Has credibility: indicators provide sufficient information about a
condition or result to convince both ‘supporters’ and ‘sceptics’ (Andresen, 2000)
– Allows for aggregating up for generating comparative data, e.g.,
comparison to general population, benchmarking data
• Data can be analysed at the individual, service provider, disability system and
societal level.
Deciding the ‘when’ of outcomes: outcomes at what moment?
– e.g., outcomes for the person with a disability
Example: Deciding the ‘when’
Outcomes for the person with complex communication
needs
– at the moment of intervention (e.g. use of communication
board in a shop)
• can communicate effectively
• able to make an effective transaction
• increased confidence
– beyond the moment of intervention - ie on person’s life as a
whole
• increased ability to shop independently (decrease support costs)
• increased choice and range of places can participate in
Issues we’ve confronted
• Selecting the ‘right’ indices of measurement (e.g.
performance, achievement, satisfaction, impact, difficulty,
etc)
• Aggregating data – moving from a focus on an individual’s
outcomes (highly individualised) to whole of service /
population outcomes
• Data analysis – difficulties in quantifiers, thematising etc –
time consuming and often highly subjective
• Administration compliance – getting staff to administer
data collection tools (correctly)
• And more…..
Data collection and analysis tools developed and trialed
Scope Outcomes Framework (used for analysis) • A whole of life outcomes framework across seven life domains
Scope Outcomes and Impacts tool • A one page survey rating scale to measure impact (positive and negative) of service on life
domains of individual or family
Individualised Planning/Review Tool • A tool based on the categories of a person centred plan to rate achievement against goals
Measurement of Services and Supports (MOSS) tool • A pre and post measure of change resulting from individual goal setting and services based on
this goal
Family Stress and Coping tool AND Family Capacity tool • Two survey scales rating a family’s self perceived level of coping and capacity
Early Childhood Individualised Support Plan tool • Attachment to Individualised Support Plan to rate outcomes at end of service period, and
barriers and enablers to these
1 in 4 poll Social Inclusion Survey • A population level survey rating levels of social inclusion related to service exclusion, economic
exclusion and social exclusion
Caution
Outcome measurement is a not an exact science – it tells us some
things but not everything, only part of the story
Not everything that can be counted counts,
and not everything that counts can be counted. (Albert Einstein)
It is very difficult to establish what CAUSED outcomes – that is, to attribute
the change to your service/support. Be cautious in your conclusions.
Be careful not to spend more time in outcome measurement and its
complexities than in taking action to improve the lives of people
Relevant publications by research team Campain, R.& Wilson, E. (under review) Life is what happens while you’re busy making plans: issues in
the development of outcome measurement for individualised plans for people with disability. Quilliam, C. & Wilson, E. (2011). Literature Review - Outcomes measurement in disability services: a
review of policy contexts, measurement approaches and selected measurement tools. Melbourne: Deakin University.
Hagiliassis, N., Nicola-Richmond, K., Mackay, A., & Wilson, E. (2011). Administration Manual - Measuring Outcomes in Services and Supports (MOSS) tool: a means to assess the outcomes of person-directed goals. Melbourne: Scope.
Nicola-Richmond, K., Wilson, E., Hagiliassis, N., & Mackay, A. (2011). Summary Sheet - Measuring Outcomes in Services and Supports (MOSS) tool. Melbourne: Scope.
Quilliam, C., Wilson, E., Hagiliassis, N. & Nicola-Richmond, K. (2010). Measurement of Services and Supports (MOSS): Project Overview and Findings. Melbourne: Deakin University.
Wilson, E. & Campain, R. (2011), Above and Beyond: Exploring outcomes and practices of Scope Southern Region Early Childhood Intervention Service for children with disability, Melbourne: Scope (Vic).
Wilson, E. & Campain, R. (2008). Problem solving, people skills, and personalised arrangements. Early results of research into individualised approaches for young adults receiving ‘Futures’ funding at Scope. C.Bigby & C. Fyffe (Eds.). Achieving their own lives. Proceeding from the Third Intellectual Disability Policy Roundtable, La Trobe University, October
Wilson, E. (2006). Defining and measuring the outcomes of inclusive community for people with disability, their families and the communities with whom they engage. C. Bigby; C. Fyffe & J. Mansell (Eds). From Ideology to Reality: Current issues in implementation of intellectual disability policy. Proceedings of the Roundtable on Intellectual Disability Policy. Bundoora: La Trobe University
Wilson, E.; Jenkin, E. & Campain, R. (2011). Outcome Measurement of Community Based Mental Health Services in Western Australia: Literature and Concept Summary. Melbourne: Inclusion Matters
PLUS many conferences and workshops and seminars
References ABS (2001) 4160.0 Measuring Wellbeing: Frameworks for Australian Social Statistics. Viewed at:
http://www.abs.gov.au/Ausstats/[email protected]/0/0120F9081822AAEBCA256B5F00804216?Open, on 31 October 2005.
Andreson E (2000), Tools of disability outcomes research: criteria for assessing the tools of disability outcomes research, Archives of physical medicine and rehabilitation, Vol. 81, No. 12, pp. S1-S4.
Commonwealth of Australia (2011) . 2010-2020 National Disability Strategy. An initiative of the Council of Australian Governments. Canberra. http://www.fahcsia.gov.au/sites/default/files/documents/05_2012/national_disability_strategy_2010_2020.pdf
Department of Human Services (2012) Department of Human Services Standards self-assessment report and quality improvement plan. http://www.dhs.vic.gov.au/about-the-department/documents-and-resources/reports-publications/department-of-human-services-standards-self-assessment-report-and-quality-improvement-plan
Horsch, K. (2005). Indicators: Definition and use in a Results-Based Accountability System. Harvard Family Research Project viewed at http://www.gse.harvard.edu/hfrp/pubs/onlinepubs/rrb/indicators.html on 31 Oct 2005.
King, S.; Rosenbaum, P. & King, G (1995). The Measure of Processes of Care: A means to assess family-centred behaviours of health care providers, McMaster University, Neurodevelopmental Clinical Research, Hamilton, Canada.
Schalock, R.L. (2004). The concept of quality of life: what we know and do not know, Journal of intellectual disability research, 48, (3): 203-216.
Saunders, P., Naidoo, Y. & Griffiths, M. (2008). Left out and missing out: Disability and disadvantage. Social Policy Research Centre, Sydney, Australia.
Saunders, P., Naidoo, Y. & Griffiths, M. (2007a). Deprivation & social exclusion in Australia. Social Policy Research Centre, Sydney, Australia.
Saunders, P., Naidoo, Y. & Griffiths, M. (2007b). Towards new indicators of disadvantage: Deprivation and social exclusion in Australia. Social Policy Research Centre, Sydney, Australia.
Triangle (n.d.) Outcomes Star. http://www.outcomesstar.org.uk/
Contact
Dr Erin Wilson
Senior Lecturer, School of Health & Social Development
Deakin University
Dr Nick Hagiliassis
Research Coordinator
Scope