+ All Categories
Home > Documents > Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit –...

Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit –...

Date post: 29-Jul-2020
Category:
Upload: others
View: 4 times
Download: 0 times
Share this document with a friend
36
Health Outcomes Monitoring Toolkit: pilot results March 2014
Transcript
Page 1: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Health Outcomes Monitoring Toolkit: pilot results

March 2014

Page 2: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Executive Summary

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 1

Contents

Executive Summary 2

Introduction 6

Two-stage survey approach 11

Single survey approach 21

Appendix: summary data tables 28

Page 3: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Executive Summary

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 2

Executive Summary

Background

The Health Outcomes Monitoring toolkit

This document presents the results of a pilot of a health outcomes monitoring toolkit developed by Citizens Advice.

The purpose of this toolkit is to enable advice agencies to measure the impact of advice on health and well-being and the social determinants of health.

The toolkit includes guidance on the research process, questionnaires and an Excel tool for storing and analysing the data. The areas covered by the toolkit are:

• Client profile – including health and disability status and demographic characteristics such as ethnicity

• Mental well-being – measured using the shorter Warwick-Edinburgh Mental Well-being scale (WEMWBS) scale1

• Determinants of health – including debt, finances, housing and employment

Following consultation with bureaux, two versions of the toolkit were tested in the pilot:

• Single-stage approach: this involves conducting a one-off survey with clients between four to six weeks after advice

• Two-stage approach: this involves conducting an initial survey at the time of advice and a second survey four and six weeks after advice

Details of the pilot study

The toolkit was piloted by eight Citizens Advice Bureaux (CAB) from September to December 2013. The aim of the pilot was to test whether the toolkit is effective and practical, and to inform the development of a generic toolkit for the wider advice sector.

The breakdown of the survey approached used, and samples achieved, is set out below.

1 The Warwick-Edinburgh Mental Well-being Scale was funded by the Scottish Executive National

Programme for improving mental health and well-being, commissioned by NHS Health Scotland, developed

by the University of Warwick and the University of Edinburgh, and is jointly owned by NHS Health Scotland,

the University of Warwick and the University of Edinburgh.

Page 4: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Executive Summary

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 3

• Single-stage approach: piloted by 5 CAB and achieving a sample of 214 clients

• Two-stage approach: piloted by 3 CAB and achieving a sample of 143 clients

Key messages

Overview: CAB services promote mental well-being and help to tackle some of the causes of health inequality

• Before advice, clients had lower than average mental well-being – but after receiving help from CAB, levels of mental well-being improved significantly

• After receiving advice from CAB, many clients reported positive improvements in their debt situation

• Some clients also reported improvements in their financial circumstances and

housing situation

CAB clients experience health problems and poverty

• Many CAB clients are either disabled people, people with learning difficulties, or people with a long-term condition or mental health problems

• Most also had a monthly household income of less than £800

• 2 in every 3 clients experienced fuel poverty before advice

Clients’ mental well-being improved after receiving advice from CAB

• The results from the two-staged approach offer the best evidence for the impact of advice on mental well-being

• On a scale ranging from 7 to 35, clients’ mean WEMWBS score before receiving advice was 19.15 – a full six points below the UK population average of 25.18

• Four to six weeks after receiving help from CAB, clients’ mean WEMWBS score

increased significantly to 24.67 – just short of the UK population mean of 25.18

Page 5: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Executive Summary

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 4

Clients’ debt situations improved after receiving advice from CAB

• Almost 1 in 2 clients reported an improvement in their debt situation after advice

• More than 1 in 4 clients achieved a tangible debt outcome, such as debt rescheduled or debt written off

• More than 1 in 4 clients felt their understanding of managing their finances had improved after advice

Some clients improved their financial circumstances and housing situation after receiving advice from CAB

• Some clients reported improvements in their income and expenditure after receiving advice from CAB

• A small group of vulnerable clients who were either homeless or at risk of becoming homeless also reported that their housing situation had improved

• There was little evidence of a general effect on unemployment or risk of unemployment; it should be noted however that supporting transitions to employment is not a direct objective of the service

• The relationship between advice interventions and some of these broader social determinants of health requires further research

Conclusions and recommendations

The toolkit is an effective resource

• The results presented in this report suggest that the toolkit is effective in providing evidence of the impact of advice on health and poverty

• It is worth noting, however, that achieving suitable sample sizes is a considerable challenge for advice agencies with stretched resources

The two-staged approach is recommended

• The two-staged approach provides the most compelling evidence because it enables comparison of clients’ mental well-being before and after advice

Page 6: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Executive Summary

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5

• Although this requires more resources to be invested upfront, it is likely to provide a

more useful set of results for the agency

Evidence can help build a theory of change

• These results show a much clearer impact on mental well-being and debt problems than broader determinants of health such as employment circumstances

• This may have been influenced by the measures used – for example, the focus of the toolkit on measuring changes in employment status may have meant that more nuanced changes in employability were not captured

• These results may also reflect genuine differences in the impact that an advice

agency has on different social determinants of health – for example, an advice agency has scope to improve clients’ financial capability but little leverage over the energy costs that are a major factor in fuel poverty

• In addition to providing evidence of the impact of the advice, the toolkit can therefore

help agencies to reflect on the degree of change they can expect to bring about across different aspects of clients’ lives

Research can help us to understand change

• Although the two-stage approach helps to measure change before and after advice, further research is required to assess the causes and process of change

• The first priority is to develop the evidence base through conducting further research among a larger sample of clients – this would enable more robust conclusions to be drawn, and a firmer basis for comparative analysis across client groups and impacts

• A second priority is to use qualitative methods to gain some insight into the process

of change from clients’ perspectives, and to identify the reasons why change may occur in some instances and not others

• Funders could play a role in providing resources and support for advice agencies to

conduct further research in both of these areas

Page 7: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Introduction

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 6

Introduction

About this document

This document presents the results of a pilot of a health outcomes monitoring toolkit developed by Citizens Advice. The purpose of this toolkit is to enable advice agencies to measure the impact of advice on clients’ health and well-being and the social determinants of health. This in turns provides evidence of the role for advice agencies in tackling health inequalities.

As this report focuses on the results recorded, it demonstrates the type of evidence that the toolkit can be used to collect. For a technical evaluation of the toolkit, please visit: www.citizensadvice.org.uk.

The development of this resource was part of a broader project led by Citizens Advice to enable Health and Well-being Boards and new Clinical Commissioning Groups to make use of evidence from voluntary sector advice agencies. As part of this project, Citizens Advice developed a Health and Poverty report that enables each Citizens Advice Bureau to profile the social determinants of health in their local area.

The health outcomes monitoring toolkit can provide evidence of the impact of a bureau’s service on most of these determinants of health, although some areas (such as domestic violence) are not covered. For further information about this project, or to access a generic version of the Health and Poverty report, please visit: www.citizensadvice.org.uk.

The health outcomes monitoring toolkit

The health outcomes monitoring toolkit provides a set of resources to help advice agencies collect this evidence through a basic client survey. This includes guidance on the research process, questionnaires and an Excel tool for storing and analysing the data. The areas covered by the toolkit are discussed below.

Client profile The toolkit covers a range of client profile criteria relevant to health and poverty. This includes disability and health status and demographic characteristics such as ethnicity. Mental well-being:

Subjective well-being is now widely considered an important component of health policy.2 Improved well-being can have a positive impact on health, for example by increasing longevity, improving recovery from illness and supporting healthy behaviours. As discussed further below, the health outcomes monitoring toolkit uses the Warwick-Edinburgh Mental Well-being scale (WEMWBS) to measure clients’ mental well-being.

2https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/277566/Narrative__January

_2014_.pdf

Page 8: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Introduction

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 7

Determinants of health: before and after advice The toolkit covers a range of factors that can affect clients’ health and well-being before they receive advice, including:

• Finances: income profile (poverty) including detail on energy costs (fuel poverty) and number of dependent children (child poverty)

• Debt: whether clients are struggling to repay debt • Housing: whether clients are homeless or at risk of homelessness • Employment: whether clients are unemployed or at risk of becoming unemployed

The toolkit includes a corresponding set of questions to assess any changes in these factors after clients have received advice, for example:

Finances – whether clients have improved their understanding of managing their money, increased their income or decreased their expenditure

Debt – whether clients have managed to achieve a debt outcome, such as having debt written off or repayments rescheduled

Housing and employment – whether clients who were in a disadvantaged position before advice have managed to improve their situation

For full details on the topics covered, please refer to the questionnaires in the generic version of the toolkit at: www.citizensadvice.org.uk/

The WEMWBS scale of mental well-being

The toolkit uses the Warwick-Edinburgh Mental Well-being scale (WEMWBS) to measure clients’ mental well-being.3 WEMWBS is a validated scale for assessing mental well-being in the adult population used by health researchers across the UK.

The health outcomes monitoring toolkit uses the 7-item shorter WEMWBS scale rather than the full 14-item version.4 This consists of a set of 7 positive statements about mental well-being (e.g. “I’ve been feeling useful”, “I’ve been feeling relaxed”) that research participants respond to using an option on a graduated Likert scale (e.g. “Rarely”, “Often”). Each option on the scale is assigned a value.

As shown in the example below, the responses given by research participants to each statement can be used to derive total value. This can range from a minimum of 7 to a maximum of 35.

3 For further information, see: http://www.healthscotland.com/scotlands-health/population/Measuring-

positive-mental-health.aspx 4 For further information, see: http://www.healthscotland.com/documents/5238.aspx

Page 9: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Introduction

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 8

The first wave of Understanding Society, the UK household longitudinal study, provides a population average for the shorter WEMWBS scale. Based on a sample of over 38,000 observations, this study recorded a mean value of 25.18.5 This provides a useful comparison for the mean scores recorded using the health outcomes monitoring toolkit.

The relationship between mean WEMWBS scores and mental well-being can be illustrated using hypothetical responses. The graph below shows the mean score that would be achieved if clients gave an identical response to each of the 7 items on the shorter WEMWBS scale.

For example:

Minimum: a score of 7 for a client responding ‘None of the time’ to each item

Maximum: a score of 35 for a client responding ‘All the time’ to each item

On this basis, a client with an average level of mental well-being would give a response between ‘Some of the time’ and ‘Often’ for each item. In practice, clients can give different responses for different items, but this method can be used to illustrate mean values.

5 https://www.understandingsociety.ac.uk/documentation/mainstage/dataset-

documentation/wave/1/datafile/a_indresp/variable/a_swemwbs_dv

WEMWBS statements Response Value

I’ve been feeling optimistic about the future None of the time 1

I’ve been feeling useful Rarely 2

I’ve been feeling relaxed Some of the time 3

I’ve been dealing with problems well Often 4

I’ve been thinking clearly All of the time 5

I’ve been feeling close to people None of the time 1

I’ve been able to make my mind up about things Rarely 2

Total value 18

Page 10: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Introduction

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 9

Single and two-stage approaches

Following consultation with Citizens Advice Bureaux participating in the pilot, two versions of the health outcomes toolkit were developed.

The single-stage approach involves conducting a one-off survey with clients between four to six weeks after advice. To allow agencies to make a comparison between before and after advice, clients are asked to give both retrospective and current information about determinants of health.

The two-stage approach involves conducting two related surveys on separate occasions: one at the time of advice and a second during a later follow-up interview between four and six weeks after advice. This allows comparison of two sets of information about determinants of health from before and after advice.

It was acknowledged at the outset that there are strengths and weaknesses with both approaches. The single-stage approach is less resource-intensive for advice agencies to implement, but is limited because it relies on client recall of their situation before advice and includes a single measure of mental well-being. By contrast, the two-stage approach includes two data points for each client, enabling firmer comparison of the change from before to after advice, but is more resource-intensive to implement.

7 12 17 22 27 32

None of the time

Rarely

Some of the time

Population mean

Often

All of the time

Mean WEMWBS score

Mean WEMWBS scores and mental well-being

Page 11: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 10

Pilot details

The aim of the pilot project was to:

• test whether the toolkit offers an effective and practical way to assess the impact of advice on health and poverty

• identify the strengths and weaknesses of the toolkit in order to inform the development of a generic toolkit for the wider advice sector6

The toolkit was piloted by eight Citizens Advice Bureaux from September to December 2013. The breakdown of the survey approached used, and samples achieved, is set out below.

Approach Number of bureaux Total sample achieved

Single stage approach 5 bureaux 214 clients surveyed

Two stage approach 3 bureaux 143 clients surveyed

The majority of bureaux adopted the single stage approach, which suggests that limitations on resource for client research are an important factor.

In addition, the modest response rates reflect feedback from bureaux that conducting follow-up research with clients can be challenging and resource intensive.

6 To access this toolkit, see: www.citizensadvice.org.uk/

Page 12: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 11

Two-stage survey approach

Sample and response rate

Three of the eight Citizens Advice Bureaux piloting the toolkit opted for the two-stage approach. Across these three bureaux, 216 clients were surveyed at stage one, with 143 continuing on to stage 2. This gives a sample of 143 clients for analysis.

The two-stage survey covered 34 core questions. The average response rate across all of these questions was 89%.

The results below are aggregated from the 3 bureaux piloting the two stage approach.

Respondent profile

Disability and health status was recorded for 143 clients. 72 clients in the sample (50%) were either disabled people, people with learning difficulties, or people with a long-term condition or mental health problems.

This is significant proportion considering that these groups make up around 38% of Citizens Advice clients across the service, and around 21% of the population of England and Wales.

Black, Asian and Minority Ethnic (BAME) clients made up around 6% of the sample. This is slightly lower than the proportion of BAME clients across the Citizens Advice service as a whole.

7%

1%

25%

17%

50%

Disability and health status - would you describe yourself as...?

Having a disability

Having a learning disability

Having a long-term health condition

Having a mental health condition

Not disabled/no long-term healthconditions

Page 13: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 12

Mental well-being

Key message: clients’ mental well-being improves significantly after they have

received advice from CAB

The shortened version of WEMWBS was used to score individuals across the seven items related to mental well-being. For the two-stage approach, clients were scored twice: before advice and during a follow-up survey four to six weeks after advice.

On a scale ranging from a minimum 7 to a maximum of 35, WEMWBS scores were collected for 138 clients at both stages of the survey.

Before CAB advice: low than average levels of mental well-being

Before advice, clients recorded a mean score of 19.15 – a full six points below the UK population average of 25.18.

Matched WEMWBS scores for before advice (n=138)

Mean 19.15

Median 18.00

Mode 21.00

Std. Deviation 6.87

Confidence Level (95%) 1.16

7 12 17 22 27 32

Before CAB advice

Population mean

Mean WEMWBS score

Mean WEMBS scores before advice

Page 14: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 13

After CAB advice: mental well-being improves after advice

After advice, clients recorded a mean score of 24.67. This is an improvement of over 5 points from the mean value recorded before advice, and brings the average level of mental well-being close to the UK population mean of 25.18.

Matched WEMWBS scores for after advice (n=138)

Mean 24.67

Median 25.00

Mode 21.00

Std. Deviation 7.10

Confidence Level (95%) 1.19

A paired samples t-test was conducted to compare the WEMWBS scores of these 138 clients at each stage. The test showed that there was a significant difference in scores recorded at stage one and those recorded at stage two.

The specific causes of this improvement in mental well-being were not explored in this research. However, in response to a question on subjective attribution, 91% of clients said that they did not think that things would have got better if they had not sought advice from CAB.

7 12 17 22 27 32

Before CAB advice

After CAB advice

Population mean

Mean WEMWBS score

Mean WEMWBS scores: before and after CAB advice

Page 15: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 14

Improved WEMWBS scores – what difference does it make?

As discussed above, we can illustrate the relationship between mean WEMWBS scores and mental well-being using hypothetical responses.

The graph below shows the mean score that would be achieved if clients gave identical responses to each of the 7 items on the shorter WEMWBS scale. In this case, we have added the mean values recorded before and after advice in the two-stage results.

In this scenario:

• Before advice: a hypothetical client would be most likely to say ‘Rarely’ or ‘Some of the time’ in response to positive statements about mental well-being

• After advice: a hypothetical client would have be most likely to say ‘Some of the time’ or ‘Often’ in response to positive statements about mental well-being

In practice, clients gave different responses for different items, but this method can be used illustrate the change in mean WEMWBS scores before and after CAB advice.

7 12 17 22 27 32

None of the time

Rarely

Before CAB advice

Some of the time

After CAB advice

Often

All of the time

Mean WEMWBS score

Mean WEMWBS scores: before and after advice

Page 16: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 15

Case study: improved mental well-being

Mandy, divorced and with a five year old daughter,

had recently been diagnosed with bipolar disorder,

which had been affecting her ability to work

resulting in her unemployment. She was living on

means tested benefits and had multiple debts,

including a £37,000 mortgage shortfall and council

tax arrears.

Mandy’s mental health declined and, in the face of

her unmanageable debt, she found it impossible to

focus on getting better.

When Mandy contacted CAB, they helped her to

apply for bankruptcy, raising money from a trust

fund to cover the cost of the application. They also

advised Mandy to approach her community mental

health team to help her with her condition, and

provided a direct link to the bureau’s dedicated

mental health programme.

She is now able to manage her finances so her

outgoings don’t exceed her income. She has been

shown how to access further support, including

specialist advisers from the CAB mental health

team. Mandy says that she is now sleeping better.

Page 17: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 16

Determinants of health: before and after advice

For the two-stage approach, information on determinants of health was collected at two different points. An initial survey was conducted before advice, followed by a second survey conducted four to six weeks after advice.

Debt

Key message: the proportion of clients struggling with debt reduced after they

received help from CAB

Debt status was recorded for 141 clients before advice. Of these, 86 clients (61%) were struggling with debt.

Debt status after advice was recorded for 103 clients. Of the 86 clients struggling with debt before advice:

• Debt problems resolved: 37 clients (43%) said they were no longer struggling with debt after advice

• Debt outcomes achieved: 46 clients (53%) said they had had a debt rescheduled These positive outcomes are reflected in a shift in overall responses:

• Reduced prevalence of debt problems: the overall proportion of clients struggling with debt reduced from 61% before advice to 41% after advice

0

10

20

30

40

50

60

70

80

90

100

Before advice After advice

% o

f cl

ien

ts s

tru

gglin

g w

ith

de

bt

Proportion of clients struggling with debt

Page 18: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 17

Household income and finances

Key message: many clients improved their financial capability after advice, but

changing income and expenditure was more difficult to achieve

Monthly household income was recorded for 137 clients before advice. A majority of 77 clients (56%) belonged to the low income category of under £800. Of these, 48 clients (26%) had one or more dependent children.

Of the 143 clients in the sample, the following changes were reported after advice.

• Financial capability: 69 clients (48%) reported that their understanding of management of finances had improved

• Income and expenditure: 16 clients (11%) said they had managed to reduce their expenditure and 7 clients (5%) had increased their monthly income

0

10

20

30

40

50

60

70

80

90

100

Income increased Expenditure reduced Understanding ofmanagement of finances

improved

% o

f cl

ien

ts r

ep

ort

ing

eac

h o

utc

om

e

Financial capability - comparison of outcomes

Page 19: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 18

Case study: debt and finances

Kelly

Kelly contacted CAB about a council tax bill that was

being enforced by bailiffs. The debt started because

Kelly hadn’t been budgeting well, and hadn’t

responded to requests for contact from her local

authority. When she received notice of bailiff

enforcement, Kelly was 6 months pregnant.

Kelly felt very anxious and overwhelmed by the threat

of bailiff action and nervous about them entering her

property. She felt unable to manage the demands of

work and trying to maintain her home, whilst feeling

the impact of pregnancy on her physical and mental

well-being.

CAB identified that Kelly was not in receipt of single

occupancy discount for council tax, which she was

eligible for, and helped her to apply for a backdated

payment. CAB also arranged for a six week hold on

bailiff action while they advised Kelly on her options

for repayment. CAB then arranged for her to pay off

the debt in monthly instalments.

When asked how she felt about her health and

finances following CAB intervention, Kelly said she

felt “like a huge weight has been lifted off my

shoulders”.

Page 20: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 19

Energy costs and fuel poverty

Key message: fuel poverty was common among CAB clients, but few managed

to address this area of expenditure after advice

The two-stage approach captured details on energy costs in addition to the fuel poverty measure used in the single-stage approach. The cost of energy bills was recorded for 128 clients. They were asked to consider gas, electricity, coal and oil bills.

• A significant proportion of 47 clients (37%) said they spent between £100-199 per month

• The most commonly reported monthly cost was between £50-£99 per month, among 44 clients (34%)

Fuel poverty status before advice was recorded for 131 clients. The definition of fuel poverty used was whether clients were spending more than 10% of their household income on fuel bills.7

The results show that a majority of 86 clients (66%) were experiencing fuel poverty before advice. Of these, the majority did not report an increase in income or decrease in expenditure after advice.

7 Please note that the definition of fuel poverty has been revised since the toolkit was first designed.

22%

34% 18%

19%

4%

1% 1%

Total monthly cost of your energy bills

£0 - £49

£50 - £99

£100 - £149

£150 - £199

£200 - £249

£250 +

Not known

Page 21: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Two-stage survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 20

Housing and homelessness

Key message: whilst some clients improve their housing situation after advice,

there was no evidence of a general improvement in this sample

Housing status was recorded for 143 clients before advice. Of these 4 considered themselves homeless, and a further 14 felt that they were at risking of becoming homeless.

Of the clients who described themselves as either ‘homeless’ or ‘at risk of becoming homeless’ before advice, 3 stated that their housing situation had improved after advice, 11 that it had stayed the same, and 4 gave no response.

Employment

Key message: in this sample, there was no evidence of a general improvement

in employment in status after advice

Employment status was recorded for 143 clients before advice. Of these, 59 clients stated that they were unemployed, and seven that they were concerned about keeping their current employment.

Although improving clients’ employment situation is not a direct objective of Citizens Advice services, the toolkit was used to record any changes in employment that did occur.

Of the clients who described themselves as either ‘unemployed’ or ‘concerned about keeping their current employment’ before advice, 41 said that their employment situation had stayed the same, 2 that it had got worse and 23 gave no response.

Page 22: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Single survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 21

Single survey approach

Sample and response rate

Of the eight Citizens Advice Bureaux piloting the toolkit, five opted for the single-stage approach. Across these five bureaux, 214 clients were surveyed.

The single survey covered 19 core questions. The average response rate to core questions was 91%. This ranged from 76% for the question on fuel poverty before advice to 98% for questions on dependent children and disabilities.

The results below are the aggregated from the five bureaux piloting the single survey approach.

Respondent profile

Disability and health status was recorded for 209 clients. In total, 135 clients (65%) in the sample were disabled people, people with learning difficulties, or people with a long-term condition or mental health problems.

This is significant proportion considering that these groups make up around 38% of Citizens Advice clients across the service and around 21% of the general population of England and Wales.

The proportion of Black, Asian and Minority Ethnic (BAME) clients was around 16%, which is similar to the client profile across the Citizens Advice service and the population of England and Wales.

8%

2%

39%

16%

35%

Disability and health status

Having a disability

Having a learning disability

Having a long-term health condition

Having a mental health condition

Not disabled/no long-term healthconditions

Page 23: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Single survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 22

Mental well-being

Key message: after advice, levels of mental well-being were two points from the

population mean, but it is challenging to identify change with this approach

The single-stage approach measures mental well-being scores after advice using the shortened version of the WEMWBS scale. Responses across the all 7 items in the WEMWBS scale were recorded for 162 clients.

On a scale ranging from a minimum 7 to a maximum of 35, the mean mental well-being score recorded four to six weeks after advice was 23.11. This is two points from the population mean of 25.18.

WEMWBS scores for after advice (n=162)

Mean 23.11

Median 24.00

Mode 28.00

Std. Deviation 6.43

Confidence Level (95%) 1.00

As noted, one limitation of the single-stage approach is that it does not include a baseline score for clients’ mental well-being before advice. This makes it difficult to assess any change in clients’ mental well-being after receiving advice.

7 12 17 22 27 32

After CAB advice

Population mean

Mean WEMWBS score

Mean WEMWBS score after advice

Page 24: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Single survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 23

This limitation is significant given that results from the two-stage approach show that, before receiving advice, clients in that sample were significantly below the UK population mean of 25.18. These comparisons are summarised in the table below.

Approach Mean score before advice

Mean score after advice

Single-stage N/A 23.11

Two-stage 19.15 24.67

Determinants of health: before and after advice

For the single survey approach, all information on determinants of health was collected at the same point. Client surveys were conducted during one-off follow-up survey four to six weeks after advice provision. At this point, clients were asked about their circumstances before advice, and then whether this had changed after advice.

Debt

Key message: over half of clients who had been struggling with debt before

advice reported an improvement to their debt situation after advice.

Debt status before advice was recorded for 191 clients. Of these, 85 clients (45%) said they were struggling to repay a debt. Of those struggling with a debt before advice:

• Debt problems resolved: 52 clients (61%) said that their situation had improved after advice

• Debt outcomes achieved: 27 clients (32%) had debt rescheduled and 11 clients (13%) had debt written off

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Debt written off Debt rescheduled Debt situationimproved

% o

f clien

ts r

ep

ort

ing

each

ou

tco

me

Debt - comparison of outcomes

Page 25: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Single survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 24

Case study: debt and finances

Jack

Jack, 56, is a single client in receipt of Jobseeker’s

Allowance, Housing Benefits and Council Tax

Benefits. Jack was a victim of identity fraud,

leading to debts of £2,000. He reported the matter

to the police, but was still awaiting follow-up from

them when he contacted CAB. He had been

physically threatened by three men from the

companies to whom he supposedly owed money.

CAB advised Jack to report the identity theft to

Action Fraud. Action Fraud began investigating all

involved parties. CAB helped Jack with a letter to

the creditors explaining the situation and including

his Action Fraud reference number.

As a result of the letter, Jack’s debts were written

off, which was a great relief to him. He commented:

“CAB resolved my problem with a high degree of

professionalism and great success. I am very

happy with the service I received.”

Page 26: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Single survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 25

Household income and finances

Key message: more than 1 in 4 clients reported an increase in income and

improvement in financial capability after advice

Household income before advice was recorded for 190 clients. A majority of 134 clients (71%) of clients estimated that, after housing costs, belonged to the low income category of under £800 per month. Of these, 27% had one or more dependent children under 14 living with them.

Of the 214 clients in the sample, the following changes were reported after advice.

• Financial capability: 65 clients (30%) said that their understanding of how to manage their finances had improved

• Income and expenditure: 60 (28%) said that their monthly income had increased and 43 (20%) said that their monthly expenditure had decreased

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Expenditure reduced Income increased Understanding ofmanagement of finances

% o

f cl

ien

ts r

ep

ort

ing

eac

h o

utc

om

e

Financial capability - comparison of outcomes

Page 27: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Single survey approach

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 26

Fuel poverty

Key message: some clients moved out of fuel poverty after advice, but there

was no evidence of a general improvement in this sample

Fuel poverty status before advice was recorded for 163 clients. The definition of fuel poverty used was whether clients were spending more than 10% of their household income on fuel bills.8

Of 163 clients who responded to this question, a majority of 119 clients (73%) estimated that their fuel bills had been costing them at least 10% of their household income.

Of those in fuel poverty before advice, the following results were returned after advice:

• 11 clients (9%) were no longer spending more than 10% of their income on energy costs

• the majority of 106 clients (89%) still remained within the fuel poverty bracket

Housing and homelessness

Key message: among a small group of vulnerable clients, over half reported an

improvement in their housing situation after advice.

Housing status before advice was recorded for 201 clients. Of these, 7 were homeless and a further 22 felt that they were at risking of becoming homeless.

Of these 29 clients who were either homeless or at risk of becoming homeless before advice, housing circumstances after advice were recorded in 26 cases: 17 said that their housing status had improved after advice, 7 that it had stayed the same, and 2 that it had got worse.

8 Please note: the definition of fuel poverty has now changed.

Page 28: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 27

Employment

Key message: among clients unemployed or at risk of unemployment before

advice, almost 1 in 5 improved their employment situation after advice

Employment status before advice was recorded for 200 clients. Of these, 62 (31%) were unemployed and 19 clients (10%) were concerned about keeping their current employment.

Although improving clients’ employment situation is not a direct objective of Citizens Advice services, the toolkit was used to record any changes in employment that did occur.

Of the 81 clients who were either unemployed or concerned about employment before advice, 14 clients (17%) said that their situation had improved after advice, 53 clients (65%) said that it had stayed the same, and 4 clients (5%) said that it had gotten worse.

0

2

4

6

8

10

12

14

16

18

Got worse Stayed the same Improved

Nu

mb

er

of

clien

ts

At risk clients - housing status after advice

Page 29: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Appendix: summary data tables

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 28

Appendix: summary data tables

Two stage approach

Client profile – disability and health status

Response Number of clients (n=143)

Having a disability 10

Having a learning disability 1

Having a long-term health condition 36

Having a mental health condition 25

Not disabled / no long-term health conditions

71

Client profile – ethnicity

Response Number of clients (n=143)

Asian or Asian British 3

Black or Black British 3

Mixed 2

White 132

Other 3

Client profile – dependent children under 14

Response Number of clients (n=143)

Yes 94

No 48

No response 1

WEMWBS scores

Matched WEMWBS scores for before advice (n=138)

Mean 19.15

Median 18.00

Mode 21.00

Std. Deviation 6.87

Confidence Level (95%) 1.16

Page 30: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Appendix: summary data tables

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 29

Matched WEMWBS scores for after advice (n=138)

Mean 24.67

Median 25.00

Mode 21.00

Std. Deviation 7.10

Confidence Level (95%) 1.19

Debt – client struggling with debt

Response Before advice (n=143) After advice (n=143)

Yes 86 42

No 55 61

No response 2 40

Analysis: changes after advice among clients struggling to repay debt before advice (n=86).

Response Struggling to repay debt after advice

Debt rescheduled

Yes 40 46

No 37 25

No response 9 15

Household income and finances

Response Number of clients (n=143)

Under £800 77

£800 to £1350 41

Over £1350 19

No response 6

Analysis: changes in financial behaviour after advice among all clients (n=143).

Response Understanding of management of finances improved

Expenditure reduced

Income increased

Yes 69 16 7

No 12 69 86

No response 62 58 50

Page 31: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Appendix: summary data tables

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 30

Monthly energy costs

Response Number of clients (n=143)

£0-£49 28

£50-£99 44

£100-£149 23

£150-£199 24

£200-£249 5

£250+ 2

Not known 2

No response 15

Fuel Poverty

Response Number of clients (n=143)

More than 10% of income on fuel bills 86

Less than 10% of income on fuel bills 45

No response 12

Housing and homelessness

Response Number of clients (n=143)

Homeless (including B&B tenant) 4

At risk of becoming homeless 14

Neither of the above 125

Analysis: changes after advice among clients at homeless or at risk before advice (n=18).

Response Number of clients

Got worse 0

Stayed the same 11

Improved 3

No response 4

Employment

Response Number of clients (n=143)

Unemployed 59

Concerned about keeping your current employment

7

Neither of the above 77

Page 32: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Appendix: summary data tables

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 31

Analysis: changes after advice among clients unemployed or at risk before advice (n=66).

Response Number of clients (n=66)

Got worse 2

Stayed the same 41

Improved 0

No response 23

Single-stage approach

Client profile – disability and health status

Response Number of clients (n=214)

Having a disability 16

Having a learning disability 3

Having a long-term health condition 82

Having a mental health condition 34

Not disabled / no long-term health conditions

74

No response 5

Client profile – ethnicity

Response Number of clients (n=214)

Asian or Asian British 21

Black or Black British 8

Mixed 4

White 166

Other / unknown 3

No response 12

Client profile – dependent children under 14

Response Number of clients (n=214)

Yes 56

No 153

No response 5

Page 33: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Appendix: summary data tables

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 32

WEMWBS

WEMWBS scores for after advice (n=162)

Mean 23.11

Median 24.00

Mode 28.00

Std. Deviation 6.43

Confidence Level (95%) 1.00

Debt – client struggling with debt

Response Before advice (n= 214)

Yes 85

No 106

No response 23

Analysis: changes after advice among clients struggling to repay debt before advice (n=85).

Response Number of clients

Got worse 4

Stayed the same 29

Improved 52

Analysis: debt outcomes among those among client struggling to repay debt before advice (n=85).

Response Debt written off Debt rescheduled

Yes 11 27

No 37 25

No response 37 33

Page 34: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Appendix: summary data tables

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 33

Household income and finances

Response Number of clients (n=214)

Under £800 134

£800 to £1350 36

Over £1350 20

No response 24

Analysis: changes in financial behaviour after advice among all clients (n=214).

Response Understanding of management of finances improved

Expenditure reduced

Income increased

Yes 65 43 60

No 22 67 83

No response 127 104 71

Fuel poverty

Response Before advice (n=214)

More than 10% of income on fuel bills 119

Less than 10% of income on fuel bills 44

No response 51

Analysis: changes after advice among clients in fuel poverty before advice (n=119).

Response After advice

More than 10% of income on fuel bills 106

Less than 10% of income on fuel bills 11

No response 2

Housing and homelessness

Response Number of clients (n=214)

Homeless (including B&B tenant) 7

At risk of becoming homeless 22

Neither of the above 172

No response 13

Page 35: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Appendix: summary data tables

Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 34

Analysis: changes after advice among clients at homeless or at risk before advice (n=29).

Response Number of clients

Got worse 2

Stayed the same 7

Improved 17

No response 3

Employment

Response Number of clients (n=214)

Unemployed 62

Concerned about keeping your current employment

19

Neither of the above 119

No response 14

Analysis: changes after advice among clients unemployed or at risk before advice (n=81).

Response Number of clients

Got worse 4

Stayed the same 53

Improved 14

No response 10

Page 36: Health Outcomes Monitoring Toolkit: pilot results€¦ · Health Outcomes Monitoring Toolkit – Pilot Results March 2014 | 5 • Although this requires more resources to be invested

Aims and principles

The Citizens Advice service provides free, independent, confidential and impartial advice to everyone on their rights and responsibilities. It values diversity, promotes equality and challenges discrimination.

The service aims: • to provide the advice people need for the problems they face • to improve the policies and practices that affect people’s lives.

Citizens Advice 115-123 Pentonville Road London N1 9LZ

Telephone: 020 7833 2181

www.citizensadvice.org.uk www.adviceguide.org.uk

Citizens Advice is an operating name of The National Association of Citizens Advice Bureaux. Registered charity number 279057.


Recommended