Rural Community Value:Assessing the Impact of the Work of Rural Community Councils
DRAFT FINAL REPORT TO DEFRA
Malcolm Moseley, Stephen Owen and Phil Johnson University of Gloucestershire
Gary Craig, Sally McNamee and Mick WilkinsonUniversity of Hull
2006
Contact addresses: [email protected] [email protected]
Table of contents
Table of contents
Acknowledgements
Executive Summary
1: The context for the study 1.1 The purpose and focus of the research
1.2 The structure of the report 1.3 Rural communities: current debates 1.4 The voluntary and community sectors and local governance
2: The evaluation of rural developmental work: issues and concepts
2.1 The impact of the voluntary and community sectors2.2 Conceptual and methodological issues pertaining to impact measurement2.3 Measuring the Impacts of Community Development in rural areas2.4 Measuring the Economic Outcomes of local economic development initiatives2.5 Current debates within RCCs
3: Developing the Tool – the research and development work 3.1 Introduction3.2 The Intensive Case Study Phase3.3 The Creation of the Draft Tool3.4 Validating the Draft Tool3.5 Making Final Revisions
4: The Economic Outcomes Tool: A Good Practice Guide for RCCs4.1 Preamble4.2 An Overview of the Tool4.3 The Tool - Stage by Stage4.4 Annexes to the Tool
5: Conclusions and recommendations
References
Appendix 1: Websites and other useful resourcesAppendix 2: The Scoping Survey E mail QuestionnaireAppendix 3: A Listing of Some RCC activities
Acknowledgements
This study, completed in a relatively compressed timetable and either side of the disruption of a General Election, depended on the cooperation and support of a number of people and organisations and we would like to acknowledge, with thanks, the help they gave.
The project was supported by a Steering Group, chaired with a helpfully light touch by Janet Gawn, Head of Social Research, Rural Economies and Strategy Division in Defra. Those who took part in the work of the Steering Group at various points in the study were:
Janet GawnTony PooleHolly YatesIain SouthallMax BoelhoffEmma BevisChris Atheyall of Defra
Lorna Sambrook (East of England Development Agency)John Hazelwood (Gloucestershire Rural Community Council)Sylvia Brown (ACRE)David Atkinson (Commission for Rural Communities).
All Rural Community Councils (RCCs) took part in an email survey and provided documentation; Sylvia Brown of ACRE was most helpful with wise advice and access to ACRE’s records; and six RCCs acted as case studies for developing and trialling the analytical tool which is described in this report. The counties served by the RCCs were:
BedfordshireCumbriaDorsetDurhamGloucestershireNottinghamshire.
We are grateful for the time they gave at short notice to help with this study. Representatives of most RCCs and some other key partner agencies took part in two workshops, in Loughborough and York. We hope the results justify the commitment of all those who participated in this study.
ii
Executive Summary
Preamble
The Impact of RCCs research project, to which this report relates, ran for much of 2005. Its aim was to develop a practical tool that would enable England’s 38 RCCs (Rural Community Councils) to better assess and articulate the impact of their work – especially any work which might have local economic outcomes. The research was undertaken by the Universities of Gloucestershire and Hull in close association with the RCCs – six of which were actively involved in detailed case study work – and with ACRE, their national association.
The research arose from a growing appreciation that voluntary organisations such as RCCs need to demonstrate more clearly what are the ultimate outcomes of the work they undertake. This will enable a closer focusing on ‘outcomes’ in the management of their work and a greater ability to demonstrate to partners, funders and other stakeholders what outcomes have occurred as a result of particular activities. It is accepted that the RCCs’ missions typically embrace a range of social, cultural and environmental aims – as well as some relating specifically to the local economy – but often economic outcomes can and do flow from a host of activities not overtly focused on the economy.
The ‘tool of analysis’
Thus the main product of this research has been a detailed tool of analysis to be used by the RCCs themselves. The tool is designed for application to specific RCC activities, which may be discrete projects (for example establishing and running a ‘wheels to work’ scheme) or broader programme of ongoing work (such as the provision of advice and support to those who manage village halls within a specific county).
The proposal is that in relation to specific activities to be addressed one at a time, RCCs should work systematically through seven stages of analysis culminating in a short report of about four pages which might be termed ‘A Report on the Economic Outcomes of Activity X.’ This report would include quantified information where appropriate data are both available and meaningful but would be largely qualitative in character. It is estimated that about three person-days of staff time, plus time contributed by external ‘validators’, would typically be needed per activity evaluated, though this commitment of staff time could be reduced if appropriate monitoring procedures were put in place at the outset of the various activities to be scrutinised.
Briefly the seven stages of the tool are as follows.
1 Preparation. This stage involves defining the specific RCC activity in question (e.g. managing the Wheels to Work scheme in County X) and the precise period of its operation to be placed under scrutiny (e.g. the calendar year 2005) albeit accepting that related outcomes may in part have occurred subsequently. Also to be defined at this stage is the target zone (e.g. the rural parts of County X) within which any
outcomes will be deemed relevant. Preparation also involves the early clarification of data sources and resolving ‘who will do what and how?’ 2 Elaboration of the Activity. The next stage is to set out concisely just what the RCC did with regard to the chosen activity during the period under scrutiny. What support was given, meetings addressed, newsletters written, representations made etc? What did other agencies do that might have complicated the picture from an analytical point of view?
3 Outlining the Outputs. This stage involves attempting a comprehensive listing, with some degree of quantification where possible, of all the principal outputs of the activity elaborated in the previous stage. Some of these outputs may be economic; many will not be. But all should be listed as a prelude to focusing on the economic outcomes in the next stage, below. At this ‘outputs stage,’ an eclectic range of source material will need to be gathered and appraised.
4 Assessing Economic Outcomes. The task here is to estimate what longer term benefits of an economic nature have accrued within the target zone as a result of the activities and outputs defined earlier. Most of these economic outcomes will fall into one or more of five categories; increased employment / reduced non-employment; increased business activity; increased / improved ‘capital’ of various kinds; the avoidance or delay of expenditure by the state; the influencing of other agencies with regard to their support of local economic activity.
5 Checking for Over- or Under- Statement. Here an attempt is made to correct for five possible sources of error that often undermine analyses of this sort. Three of them commonly lead to an exaggeration of the real economic outcomes that have occurred; these relate to ‘attribution’, ‘deadweight’ and ‘displacement’. Two relate to typical sources of understatement, namely ‘multiplier effects’ and ‘delayed or protracted outcomes’. Advice is given on how these sources of error might be addressed, albeit pragmatically and imperfectly.
6 External Validation. It is now time to seek the validation of a draft ‘Report of the Economic Outcomes of Activity X’ based on the above research. Up to three well-placed external commentators should be recruited for this purpose; each will know the activity well and be able to give a reasonably objective view in a face-to-face interview. The validators are asked to comment in turn on sections of the draft report, addressing in turn the activity, the claimed outputs, the claimed outcomes and the suggested summary of over- and under-statements.
7 Conclusion and Final Edit. It remains to revise the draft report and present it to the board of the RCC as an aid to management, including staff management, and to the relevant external partners and funders of the RCC. A template is set out to help RCCs frame these reports.
II
Developing the above tool
In order to devise and refine the above tool of analysis the research team had undertaken a series of exercises which are written up at some length in the report but may be briefly summarised as follows.
1 An analysis of relevant literature pertaining to existing evidence on the impact of the voluntary and community sectors in
general – an analysis that established the considerable scale and complexity of the impact of these sectors in England
a range of conceptual and methodological issues pertaining to impact measurement – dealing with the treatment of such concepts as outcomes, attribution, deadweight, capital etc, and with the temptation to deem important that which is measurable rather than to attempt to measure that which is important.
the assessment of impact specifically in the domain of rural community development – a review that established inter alia that although there is now a body of experience pertaining to the evaluation of rural community development projects, there is very little guidance on how to evaluate the impact of rural community development per se
measuring the economic outcomes of local development initiatives. There is now a considerable literature on this, emanating from a host of national and local agencies, giving useful pointers on how to deal with the familiar range of issues around outputs, outcomes, deadweight, displacement, double-counting etc
2 A review of current RCC experience in this domain An analysis of a sample of RCC documentation (e.g. business plans and annual reports), supplemented by an e mail survey of RCCs, indicated both a growing interest in the general subject of evaluation and impact assessment, and also a wide range of RCC activity that would appear to produce economic outcomes of some kind. This review also revealed, however, that there is as yet no commonly agreed methodology for this work – a state of affairs often attributed in large part by the RCCs to the very different expectations and requirements of the wide range of agencies and organisations that purchase services from them.
3 A programme of Research and Development extending from June to November 2005 and undertaken by the research team in close association with six RCCs. This R&D involved
intensive case study work (involving the RCCs which serve Bedfordshire, Cumbria, Dorset and Nottinghamshire) in which various links between selected RCC activities (18 in all) and their outputs and apparent outcomes were hypothesised, explored with reference to available information and in large measure validated with external involvement.
And then, after the subsequent formulation of a draft tool of analysis,
III
the ‘road-testing’ of this draft tool by Durham and Gloucestershire RCCs – each RCC having identified two of their own recent activities (4 in all) for careful scrutiny. Again this validation and further development of the tool was undertaken with the involvement of local external validators.
two workshops involving about 30 RCCs plus various external agencies with an interest in RCCs as ‘delivery agents’, to receive and critique what by then had become a strong proposal regarding the form of the tool.
Conclusions
The main product of this research has been the tool– summarised above and set out at length in the body of the report. It is also being published as a freestanding Good Practice Guide.
But looking at the whole research exercise retrospectively a number of specific conclusions are drawn:
concerning the context – that RCCs need, and are generally ready, to demonstrate more convincingly the outcomes of their work both economic and non-economic;
concerning economic outcomes – that this is a difficult concept to grapple with both in theory and in practice and that it is a delusion to imagine that such outcomes can be easily expressed in simple arithmetic or cost terms;
concerning the nature of the proposed tool - that it has been found in this research to be user-friendly, relevant, useful and timely and that RCCs generally, and many of their stakeholders, have expressed their support for its application;
concerning its use by the RCCs – that there are important issues to be addressed by them regarding their information gathering and storing, the involvement of appropriate personnel within and outside the organisation, the exigencies of validation, and the embedding of the process into normal work practices. There are also important issues pertaining to the cost of staff time to be expended in undertaking the work.
concerning the transferability of the tool – that it should be capable of modification for use (a) in relation to the more social or community outcomes that characterise much of RCCs’ work, and (b) by other voluntary or community organisations facing broadly similar challenges.
IV
V
1: Introduction: the context for the study
1.1The Purpose and Focus of the Research
This research project was commissioned by Defra early in 2005, with the aim of developing a practical tool for assessing the impact of England’s 38 Rural Community Councils.
The brief for the research stated that:
‘ Defra is interested in providing support to the RCC network to demonstrate their impact….we want the research to advise on how impact information can be captured as well as what information to capture. Defra therefore wishes to commission research
to identify the impact of RCC activity (with special account of the economic impact) and
to develop indicators to assess the outcomes from the funding they receive.’
It was agreed at the outset that the work would focus primarily on the economic outcomes of RCC work, while recognising that such outcomes tend to flow from a wide range of RCC work including that of a more explicitly social, environmental or community nature. It was also agreed that in no sense would the research be an evaluation of RCCs; rather it would attempt to develop a tool for the RCCs themselves to apply whenever a rigorous appraisal of their outcomes might be useful.
There have been four main phases of work: a preparatory phase, involving a literature review and a scoping survey of
RCCs, a case study phase to explore in depth the work of just four RCCs and its
apparent outputs and economic outcomes, the subsequent drafting of an ‘outcomes tracing tool’, the testing of that tool by two further RCCs – and its subsequent modification,
Thus the RCCs and their national association, ACRE, were active and enthusiastic collaborators in each phase of the work but responsibility for this final report rests with the research team alone.
1.2 The structure of the report
The research has involved bringing together three key dimensions to current policy debate. The first is the nature of rural communities in the UK and, in particular, the issues surrounding deprivation on the one hand and the social and economic revival of these communities on the other. Secondly, it examines the work of a particular set of agencies, which privilege the values of community development – of local involvement in the identification and meeting of social and economic needs. Thirdly,
6
it is situated in the growing debate about ways in which the impact of work to deliver ‘human services’ – broadly defined – can be evaluated.
In this introductory chapter, we look at the first two of these dimensions; Chapter 2 will examine current debates on evaluation of public services with a particular emphasis on services that are set within a community development paradigm. In Chapter 3, we describe the methodology of the study, which led to the development of an analytical tool to be used by Rural Community Councils (RCCs) themselves as a means of looking at the impacts of their own work. The tool was focused on economic impacts – again broadly defined – but although most respondents to the study suggested that it was possible to separate out economic and social impacts of the work of RCCs, and indeed the brief given us by Defra implied this very strongly, there was also a strong view that economic and social impacts are very closely related and one may lead to or be associated with the other. Chapter 4 outlines the tool itself. Although this is focused very much on the work of RCCs, comments by other partners in this field of work have suggested that much of the structure of the tool may be applicable to other similar areas of work.
As we note in our discussion of methodology, it is important that any process of evaluation is closely related to the type of activity and process being evaluated. Thus, in evaluating work broadly within the community development paradigm, it is important that any methodology is as participatory as possible, offering opportunities for both partners and end-users to be involved in the process of evaluation. We have tried to stay close to this principle in this work by developing a participatory tool, by building our work on the participation of RCCs themselves, by developing a tool which requires input from partners and users, and, in the course of this study, by feeding back our developing ideas to the RCC constituency.1
In Chapter 5 we identify the main conclusions and recommendations from this study. Appendix 1 provides a listing of some useful resources we identified during the course of this study, including both relevant literature and websites, which provide further related information.
1.3 Rural communities: current debates
There remain differing understandings of the meaning of rurality within current policy debates. Discussion about the characteristics of rurality has continued within the UK for some years with, for example, a distinction drawn between ‘accessible rural’ (or peri-urban) areas on the one hand and remote areas on the other, together with an exploration of the relationship between urban areas and their more immediate rural hinterlands. There has also been a policy debate, which has attempted to separate out the distinctive characteristics of poverty and exclusion as experienced in rural areas as opposed to urban areas (see e.g. Chapman et al., 1998; Dunn et al., 1998; Shucksmith, 2000; Cornish et al., 2002), a debate which has been mirrored in the development of official poverty indicators by government and local government. This dimension to
1 We did this largely through sending regular Bulletins, four in all, to all RCCs and, later on, through two workshops in which a draft tool was the subject of discussion. Copies of the Bulletins can be obtained from [email protected] or from [email protected]
7
the debate is significant since community development tends to focus on areas of deprivation.
Alongside these debates about the meaning of rurality (and, indeed, the new definition of rural areas2), there has been a growing recognition that the nature of deprivation in rural areas differs in some important ways from deprivation in urban areas. As a result, the Index of Multiple Deprivation has increasingly moved towards identifying indicators of deprivation which are particularly pertinent to those living in rural communities, particularly in relation to the question of access to key services such as affordable transport, a meeting place, a GP surgery and a shop. In the varying classifications of rurality in the UK, many areas are characterised as mixed urban and rural which is a further complicating factor (see for example Cloke and Edwards, 1986; Craig and Manthorpe, 2000). It is not open to us to engage in a probably fruitless attempt to define the nature of rurality here but it is important to recall that the network of Rural Community Councils in England serves communities which cover both rural and remote areas and the context of work whose impact is being evaluated is an important factor in thinking about the nature of that work and the ways in which it might be evaluated.
Rural Community Councils are county-based charitable bodies which seek to promote the social and economic well-being of rural communities and alleviate rural disadvantage, often through a broadly developmental approach, which privileges the important role of community involvement in identifying and meeting local social, economic and environmental needs, but also through direct service provision, and through partnership working with other key local and regional agencies. Not least amongst these are England’s parish and town councils with which all RCCs have a close working relationship. This relationship takes many forms; it generally involves the RCCs in providing technical support and advice to the councils but as important is their joint working on projects to improve quality of life at this very local, parish, level.
There is a network of 38 RCCs in England, linked through a central body, ACRE (Action with Communities in Rural England) (www.acre.org.uk) which coordinates their work at a national level, draws together policy recommendations for government and other national bodies, and provides infrastructural support to individual RCCs. RCCs collectively receive several million pounds of core funding from central government, linked to service agreements, and (although variable from one area to another) significant sums of money in the form of grant aid for other projects and activities. They are thus major delivery agents for central and local government policy relating to rural regeneration. The significance of the work of RCCs is also that they are the single voluntary sector body within each county responsible for the promotion of a ‘community’ perspective on local needs and this significance has grown since the implementation of the Haskins Review through Defra’s Modernising Rural Delivery programme.3
2 For information on the new definition of rural areas, see www.defra.gov.uk/rural/strategy/annex_a.htm3 For further details of the Haskins Review see www.defra.gov.uk/rural/ruraldelivery/default.htm Defra is currently publishing a monthly newsletter (Modernising Rural Delivery) reporting progress with the programme.
8
This Review provides an enhanced role for Regional Development Agencies – key partners for RCCs both in terms of policy and service development and as a source of funds for their work and the work of other voluntary and community sector agencies in rural areas. The Review has led to ‘radical reforms to the delivery of the government’s rural policies … aimed at delivering services in a more streamlined customer-focused way, by a smaller number of organisations with devolved responsibility with clearer roles and working in partnership within an overarching sustainable development framework.’(Defra website) One key facet of the reforms has been the rationalisation of funding streams. The key goals of rural policy – within which the work of RCCs is to be situated are to do with ‘improving local delivery, tackling disadvantage and social exclusion, helping underperforming economies to begin to catch up with the best, and ensuring that the natural heritage is protected, all within locally agreed outcomes.’ (Ibid.)
As part of this developing role for RDAs and their voluntary and community sector partners, it has been increasingly recognised that it is important to be able to assess the impact of this work – including the work of RCCs - using measures which are relevant and practicable. Government is, in effect, moving steadily towards a relationship with the voluntary and community sectors where it ‘purchases outcomes’ from its local partners. As the original brief for the work described here commented, ‘it is important therefore that the contribution made by the RCC network to economic and social outcomes can be measured and properly acknowledged’.
1.4 The voluntary and community sectors and local governance
Linked to change in the voluntary and community sectors (VCS), the local governance context has also been changing. So has the direction of government policy. This provides a critically important context for the work of RCCs, alongside the specifically rural dimensions arising from the Haskins Review and summarised above.
There is not space here to debate the nature of the voluntary and community sectors as a whole. A comment made in the course of another study, however, gives some sense of the dimensions of the VCS 4 (Craig et al., 2005), a comment which – given variations from area to area - would be accepted by RCCs as covering the broad scope of their work:
The VCS helps towards a prosperous economy by providing flexible, local training facilities; it raises enormous sums of money for specific projects such as hospices; it undertakes a range of service provision; it acts as champions of people unable to articulate their own concerns; it promotes self-help, volunteering and citizenship; it provides speedy responses to need; it is innovative; it acts as an advocate; it provides a direct link to the grassroots; it helps to meet environmental targets; it promotes good local governance by facilitating local action planning processes; and it provides a range of leisure, arts and cultural facilities’.
4 We distinguish between the voluntary sector, associated with state funding, paid employees and, to some degree, service delivery and the community sector, which typically consists of representative bodies with few if any paid employees and little funding.
9
The present study reflects a growing interest in the nature of the relationship between government – at national, regional and local levels – and the VCS. In part this reflects a growing focus on partnership working but it also indicates a growing recognition of the increasing professionalism of the VCS and its ability to identify and continually respond to new needs as they emerge. The growing emphasis on partnership working over recent years has led to an increasingly high profile for the VCS. Associated historically with choice, flexibility and the capacity to release new resources (donations, volunteers, mutual aid, self-help), the sectors have also been seen as a way of reaching more marginalised groups in society and giving them a public voice. Through extending citizen involvement in both services and policy development, a healthy VCS offers the potential for a variety of provision, catering for a diversity of need and bringing a wider range of resources to bear on the policy process. Government has recognised the strengths of the VCS but, through a series of policy developments – often backed with substantial funding streams – has attempted to knit the development of the VCS more closely to its own policy and service agendas.
Government has increasingly, since 1997, sketched out its expectations of the VCS and – at least to some extent, discussed below – provided the policy framework and resources to support its aspirations. For example, government’s commitment to the Compact and to the VCS as a whole was reaffirmed with the publication by HM Treasury of a cross-cutting review of the VCS in service delivery (HM Treasury, 2002) and of a consultation document on the further development of Compacts (Home Office, 2005a); this latter saw the Compact as underpinning the expansion of the sector’s role. From 1998 onwards, debate about the Compact, at national, regional and local levels, has increasingly focused down on the kinds of relationship which voluntary and community organisations could expect to have with central, regional and local government and other public sector bodies such as Health Authorities, Primary Care Trusts, Learning and Skills Councils. Compact development is, as yet, rather uneven and, in general, lagging behind overall development in rural areas, a result of the difficulties of managing processes of consultation over long distances and with dispersed populations and weaker VS infrastructure. (Craig et al., 2004)
Government funding is now, in general, critical in determining the shape and size of the sectors. Beyond the political commitment to the development and use of the Compact at local and national levels, central government has now begun to make two major funding investments. One, the Future Builders programme, is ‘a £125M investment fund developed jointly with the sector, supporting schemes which increase the scale and scope of sector involvement in service delivery through a mixture of grants and different kinds of loans, focussing on development and acquisition of physical and intangible assets. The £125M initial investment by government has now been enhanced by an additional £90M for the financial years 2006/07 and 2007/8’. (Home Office, 2005a)
The second funding investment is the ChangeUp programme. ChangeUp is ‘a capacity-building framework for the voluntary and community sector, launched in June 2004 after fairly extensive consultation with the sector. Government is making an initial £80M investment to support the framework, with over 60% of these funds initially earmarked for delivery at a regional or local level.’ (Home Office 2005b) The Government is making an additional £70M available in the period 2006-8.
10
The Government’s stated intention in establishing ChangeUp was that the VCS and other stakeholders, at national and regional levels should ‘come together to agree the shape and structure of provision which best suits the needs of frontline organisations and how it should be funded.’ Specialist needs identified in the NCVO’s review of the ChangeUp scheme included urban/rural, generic/specific, mainstream/BME, voluntary/ community and volunteering support/sector support. Much of the VCS infrastructure to support these specific needs does not yet exist in a robust form in many parts of the country. Within rural areas, RCCs provide the major source of infrastructural support for the VCS but it is acknowledged widely that VCS infrastructure is generally weaker in rural areas than elsewhere, in part because of the difficulties of maintaining organisational structures across long distances and in areas of dispersed populations.
The ChangeUp programme is, in any case, a means to an end. Government has set out a vision for the VCS, originally rehearsed in the Treasury CrossCutting Review of the Sector (HM Treasury, 2002), which argued for a ‘coherent shared strategy to underpin capacity in the sector’ (NCVO 2003) in which the VCS is to have a very significantly enhanced role in service delivery in particular. This role was to be underpinned by the further development of the Compact at national and local levels and by strengthening the VCS at local and regional levels through the ChangeUp programme, that is, by the development of services to the sector to enable it to perform more effectively as agents of public service delivery. Explanations outside government of the government’s enthusiasm for greater VCS involvement in public service delivery have focused in particular on questions of cost; on the local ‘marketisation’ of public services; and on the government’s uncertainty about the role of local government as a means for effective and accountable service delivery, described by one commentator as ‘picking up the pieces of public service failure’. (‘Charities to pick up the pieces?’, Community Care, 3 March 2005: 18).
Within the VCS, there is of course greater enthusiasm for becoming more involved in public service delivery amongst those organisations (particularly large national organisations with networks of branches across the country), which have the capacity to engage in this work.5 The National Council for Voluntary Organisations (NCVO) has, however, strongly promoted the message that ‘those who wanted to deliver public services should be allowed to do so, without losing independence, and those who do not should not suffer.’ (HM Treasury, 2002) In general, direct service delivery is a fairly small area of activity for most RCCs, which emphasise their developmental role. Government itself has argued that the VCS can bring specific advantages to the delivery of public services, including ‘a strong focus on the needs of service users, knowledge and expertise to meet complex personal needs and tackle difficult social issues, an ability to be flexible and offer joined-up service delivery, the capacity to build users’ trust and the experience and independence to innovate’ (HM Treasury, 2004), characteristics which the VCS itself would recognise.
One further strand in current debates about the VCS relates to the government’s enthusiasm for the notion of ‘social enterprise’, linking to its wider agendas on economic development. As the government notes (Home Office, 2002), social
5 See for example Stephen Bubb, Chief Executive of ACEVO:’ The third sector’s immense potential in public service delivery remains largely unfulfilled’, Regeneration, 22 April 2005: ACEVO represents precisely this part of the VCS, that is, large voluntary sector organisations engaged in service delivery.
11
enterprises are essentially businesses but businesses trading to support a social goal or mission. The surpluses which are generated from trading – whether as local community enterprises, credit unions, development trusts, Fair Trade organisations, social firms, cooperatives and other forms of mutual enterprises – are reinvested either in the business or in the community or both. Some social enterprises aim to meet not only financial and social goals but environmental ones also. There are, as another recent report on the subject noted, in the context of a growing social enterprise culture ‘increasing expectations that the voluntary and community sectors both would and should develop social enterprise activities’. (Pharoah et al., 2004) This view reflects an understanding that the social mission and the community-based nature of many VCS organisations positions them uniquely to develop as social enterprises, combining the social mission of the VCS and the more business-like approach of the private sector, particularly in the context of a growing impatience within the sectors themselves at continuing grant dependency and the annual acute anxiety about renewing contracts and grants or seeking new sources of funding.
Social or community businesses also have to be seen very much within their local context and this is perhaps particularly the case for rural community businesses. A review of rural community businesses in the Yorkshire and Humber region identified 90 such businesses, mainly focusing on the delivery of local services, transport, social services and the provision of local goods. Much of this work was effectively substituting for the withdrawal of statutory or private sector bodies that had found provision of services too costly or difficult. (Gore et al., 2003) The report suggests the need for funding streams to be clarified and aligned so that support for social enterprise is clearly set out and investment sustained. Community economic development has been shown however to have a key role in ‘both tackling social exclusion and “reconnecting” disadvantaged communities to the economic mainstream.’(Hill and Kumi-Ampofo, 2004)
One further piece in the local governance jigsaw within which RCCs find themselves, is the emergence of Local Area Agreements. Local Area Agreements (LAAs) are intended to ‘provide the basis for strengthened partnership working and greater co-ordination between central government, local authorities and their partners’ (ODPM 2004) – including the voluntary and community sectors, working through the Local Strategic Partnerships. There has indeed been a suggestion by government that LAAs could not go forward without the endorsement of the VCS. One of the Mandatory Outcomes for LAAs is a requirement to ‘empower local people to have a greater voice and influence over local decision-making and the delivery of services’.
The VCS – and particularly RCCs in rural areas - can be critical actors in this process as well as protecting their own interests as ‘LAAs should include a statement of the involvement of the VCS in the design and delivery of the agreement.’ The intention also is that LAAs will ‘simplify funding streams from central government going into an area, helping to join-up public services and allowing local authorities and their partners greater flexibility for local solutions for local circumstances.’ (Ibid.) If they were effectively to be incorporated into this process, the VCS may be seen to have come of age within the arena of local governance and the emergence of Local Area Agreements may offer the opportunity for the VCS to take its proper place in partnership bodies responsible for governance at a local level.
12
There are, of course, dangers for the VCS as always with incorporation into the process of local governance. The most obvious one is that a consensus might emerge amongst key local actors around a definition of needs in each area which would come to be set in concrete: the ‘usual suspects’ would come to be funded on an ongoing basis and those outside the consensus – as well as those who cannot get to grips with the complexities of the LAA - would be excluded from having a voice. This issue has already come to the fore in partnership working in both rural and urban areas where sectors of the VCS – particularly representing the smaller and more marginalised groups – have felt excluded from key local partnership groups. This is an issue for the RCCs which, despite their own, often fragile, funding arrangements, are seen by the remainder of the VCS in rural areas as relatively powerful actors. One critical issue for Local Area Agreements is therefore that they leave scope for innovation, development and risk-taking.
Having briefly reviewed the local governance arena within which RCCs are increasingly situated, we turn now to examining the evidence on ways in which the impact of rural developmental work by a major VCS player might be evaluated.
13
2: The evaluation of rural developmental work: issues and concepts
As part of the scoping work to inform the development of the analytical tool, we reviewed the literature relating to three key interlocking areas: debates about evaluation within the voluntary and community sectors in general, about the evaluation of community development more particularly, and about the evaluation of local economic development. In addition, we also undertook an email survey of all RCCs, part of the intention of which was to ascertain RCCs’ own views on the nature of impact measurement. This chapter summarises the findings of this work, to provide a context for the following chapters in which we discuss the methodology of this study and the development of the tool itself.6
2.1 The Impact of the Voluntary and Community Sectors
One way of considering the impact of the voluntary and community sector (the VCS) is to focus on what its contribution is to the economic, social and environmental life of an area, and on the changes which result from its work. This contribution can be assessed in two differing ways, one rather narrow and the other rather wider in its scope. The first is to estimate its economic contribution to rural life within a defined area, both in direct and indirect ways but using a narrow conception of ‘economic’; this was one starting point for this study but we found in feedback from RCCs that they generally took a wider view. The second approach is to attempt to assess its broader impact, i.e. all of the changes which are brought about as a result of the diversity of investment in the VCS, and recognising that what may initially appear to be projects with a social orientation may well lead also to economic impacts.
The reviews we undertook suggested that the literature specifically on economic impact measurement in the voluntary sector is rather sparse; there is a limited amount of material, which defines particular economic impacts in quantifiable ways; and there is rather more on qualitative measurement. However, in relation to the wider goals of the voluntary sector (reflecting its mission of social inclusion and, often, its methods of community development) the literature becomes correspondingly more substantial but also more general in its treatment of impacts.
We were able to draw on a review of more than 30 reports undertaken by a range of bodies across the UK which attempted to assess the (narrow) economic contribution of the voluntary and community sectors to different regions. Interestingly, although the rigour of the methodology varied between those studies, the range of figures for regional or sub-regional contribution to regional Gross Domestic Product was, in the event, fairly narrow: this range was generally around 3%-4%.
For an individual ‘average’ local authority area, the total ‘GDP’ would be about £3bn and the contribution of the VCS to that therefore at least £100m. The most recent national government estimate, from HM Treasury (2004), suggests that ‘the 6 A more detailed version of this account – prepared for a meeting of the Project Steering Group - can be obtained from the co-directors of the study: [email protected] or [email protected]
14
[voluntary and community] sector contributes at least £7bn to the UK economy, a massive increase of over 250% since 1991. Furthermore, the sector also provides outputs that are difficult to measure, such as the value of volunteering. If the value of volunteering is included, the worth of clubs, groups and societies could be worth an additional £25bn’. This again suggests a contribution to each regional economy of between £2bn and £3bn. However, whilst these figures seem reasonably robust at a regional level, there is clearly some differentiation at the sub-regional level and, given the generally lower level of investment in rural VCSs compared with their urban counterparts, it might be expected that the contribution of a rural VCS is somewhat lower at somewhere between 2 and 3% of GDP. This is still a substantial figure.
As the Treasury notes, estimating the value of volunteering - particularly significant in some sub-regional contexts and probably more so within rural areas - is particularly difficult and the Treasury estimate may only count the more formal end of the volunteering spectrum. The Institute of Volunteering Research estimates that for every £1 spent by the public sector supporting VCOs, it receives an economic payback of £30 – the total savings to the public sector that the paid and unpaid labour of VCOs and their services to clients produce. Research on a sample of organisations in the East Midlands suggests that there was a return of between £2 and £8 for each pound invested by most organisations in their volunteers (based on national average wage levels) (Gaskin and Dobson, 1997).
A 1995 study by the National Centre for Volunteering estimated that the contribution of volunteers, both formal and informal, to the UK economy was £41bn, which would have made it the third highest component of the British Gross Domestic Product, a rather higher figure than that suggested by the Treasury above. Given the difficulties in assembling accurate data in both these other studies and the present one, the narrower computation of the economic contribution of the VCS has to be based on some rough and ready estimates but given that most of those made elsewhere converge around a relatively small range, taking the value of volunteering into account, it seems reasonable to assume that the overall contribution of the VCS to a rural economy would be 3-4% of GDP. In some rural areas, volunteering makes a greater economic contribution measured in certain ways than long-established industrial sectors. About eight volunteers are generally reckoned to be equivalent to one full-time paid job.
Volunteering, of course, also has another direct economic contribution in that it provides a route, through training and work experience, for those wishing to enter the labour market but who are not sufficiently skilled or confident to do so without some sort of transition. Expanding opportunities for volunteering – which can only be done by expanding those able to support, mentor and train them i.e. paid voluntary sector staff - can increase the employability skills of the wider workforce. The input of volunteers of course varies between different types of activity: in sports for example, volunteering probably has a much larger proportional input than in more professionalised areas of activity such as advice and advocacy.
A number of reasonable claims might be made about the added value of the VCS, as follows. (Bolton, 2003) These elements of value appear largely to be oriented towards the social dimensions of impact: but many may lead on to economic gains.
i. It builds or creates social capital
15
ii. It is good at meeting special needs because it has significant expertiseiii. VCOs provide an independent voiceiv. Voluntary organisations fully involve their stakeholdersv. The voluntary sector is flexiblevi. The voluntary sector is innovativevii. The voluntary sector can work across public sector dividesviii. The voluntary sector operates as the last resort of the powerlessix. The voluntary sector offers a qualitatively different experience to staff and
usersx. The voluntary sector identifies new needs
This framework offers the basis for voluntary and community organisations (VCOs) to think about the impact of their work and for developing a more robust set of measures for impact assessment. This is certainly an urgent task across the VCS as a whole, and not just for RCCs.
2.2 Conceptual and Methodological Issues Pertaining to Impact Measurement
Moving on to the challenge of assessing the impact of specific VCOs (e.g. RCCs) in specific contexts, we encounter a number of conceptual and terminological problems. The language of performance measurement has been dominated in the past twenty years by Government requirements to measure outputs, in a more strictly quantifiable manner, and to ‘hit targets’. These concepts are now being increasingly challenged from within local government and the health service as, on the one hand skewing objectives (a criticism particularly made of targets, whereby ‘hitting targets’ becomes the main rationale for an organisation’s activities) and, on the other hand, often being meaningless and telling little about the long-term performance of an organisation (Sanderson et al., 1998; Alcock et al., 2001).
Increasing the number of people receiving a service tells you little about the quality of the service offered or received, what difference the service made to the person in receipt of it, or whether the people receiving the service were those most in need of it. A realisation of this has led the language and practice of public policy evaluation to focus increasingly on the question of impact or outcome and, particularly for services and activities delivered to or with people, on the process of engaging with people.
This language is particularly appropriate for the voluntary and community sectors and most of all for those parts of it – community development, capacity-building, developing involvement and participation – where the dimension of process is critical to long-term outcomes. In short, the language of targets, inputs and outputs, summed up in the phrase ‘performance measurement’, is about the efficiency of an organisation (how many outputs per input) and not about its effectiveness in achieving its wider goals (particularly that of social inclusion) and how that goal is achieved.
It is appropriate here to make a clear distinction between the terms input, output, process and outcome as the language of evaluation has become substantially corrupted in the past few years. Starting from a baseline position (which will generally have involved the identification of a need – that is, a need which has yet to
16
be addressed), the inputs are the totality of resources committed specifically to a programme or activity by one or more partners. These resources might be financial, human or some other kind of input (in the case of volunteers, most notably the input of time or, say, the use of a car). This commitment of resources then brings outputs, that is certain new services, activities or products: these might be community groups developed (in the case of capacity-building or community development work), nursery places created (in the case of a childcare programme), the number of benefits claimed (as a result of advice work), jobs created through a training programme, or people attending a day centre or lunch club.
How things are done – i.e. the process – is critical for services or activities where trust, confidence in the organisation, social inclusion and the engagement of people (and perhaps the so-called ‘hard-to-reach’) are important considerations. Different aspects of process may be critical at different times: for example, there is no point in establishing a day centre if the transport available to get disabled people to the centre is not available or is provided by a surly or dangerous driver who makes people feel uncomfortable enough eventually not to want to use the service. Building a democratic and locally-accountable community group is pointless if, at the first meeting, a chair is appointed from outside (as has often happened in government-initiated neighbourhood programmes) or one is elected who turns out to be ruthlessly autocratic and suppresses dissent or the discussion of alternative ideas.
Finally, the outcome is the wider or longer-term impact of the intervention and should be clearly distinguished from outputs. It may be that a particular training scheme generates a certain number of jobs: the outcome, however, that needs to be assessed concerns whether the jobs that are created are sustainable, accessible to all regardless of gender or ‘race’ for example, and directly contribute to a reduction in long-term unemployment amongst specifically targeted groups. VCOs may typically have more than one outcome for specific projects or groups of users.
The focus on outcome rather than on outputs is also critical because it helps to move the debate on impact away from questions merely of cost. Funders making a judgement on which services or activities to fund merely on the basis of their cost are falling into a long-honoured trap. Every local authority in the country, and presumably every government department also, would be able to cite examples of ‘cheap’ tenders which have been awarded where the outcome has literally been disastrous – housing contracts, for example, where contractors have been called back time and again (at considerable cost to the authority) to repair or replace defective work. In this work, we were not fundamentally interested in RCC ‘efficiency’ (broadly, the ratio of outputs to their inputs), just in their ‘effectiveness’ (broadly, the validity and scale of their outputs and outcomes). The issue of inputs however is obviously important in determining what scale of outcomes might be possible.
The measurement of impact or outcomes brings with it a number of methodological difficulties. (Alcock et al., 2001) In terms of impact, we need to distinguish between the direct outcomes of an organisation’s work and indirect or intermediate measures (services provided such as training, community development, new skills, substitution for more expensive services which would have to be met through public expenditure).
17
Apart from the question of unanticipated impacts, which cannot by definition be planned for, another major methodological problem with establishing outcomes lies in attribution: is it clear that this activity led to that outcome, or was there something else going on which was responsible for the changes observed? And what would have happened had the intervention not taken place (the issue of deadweight)? This is a continuing debate within the field of the evaluation of human services but it is set within a recognition that it is inappropriate to deliver human services within an experimental context; it is not possible – or at least not ethically desirable - to deliver services to one group of people whilst refusing them to other equally needy groups simply to be able to replicate the scientific conditions of a laboratory. The only additional point to be made here is therefore that there can never be absolute certainty about outcomes but that the assessment of impact or outcomes can be enhanced by the careful collection of appropriate data i.e. by good monitoring systems. 7
Another serious difficulty relates to the timescale of impact; many consequences of actions by VCOs accrue in the long-term and we are clear from our review of the work of RCC that this is the case here. It becomes difficult, if not impossible, to assess many impacts if the timescales for their assessment are short. It is important that funders take this on board: having accepted that a focus on outputs is inappropriate, it is then quite unrealistic to be interrogating VCOs about impact within an annual timeframe. What is needed at least is a rolling programme of impact assessment over, say, a three-year period and this indeed could be made a condition of funding.
The longer timeframe necessary for impact assessment is also critical given the VCS’ focus on social inclusion and user involvement: consulting with users effectively (and not in a cursory or dismissive manner, and in a way which takes proper account of issues such as gender, ethnicity and disability or – in the context of RCCs - remoteness) also takes time. Although some VCOs are experimenting with new forms of consultation, proper consultation which offers effective engagement to more marginalised groups in particular cannot be done within the often very tight timeframe often offered by government for major programmes such as regeneration.
A further issue to be considered in relation to impact is that of quality and quality assurance. Organisations, including many within the VCS, are increasingly paying attention to issues of quality assurance - that is to assessing the quality of what is being delivered in terms of differing measures. (Waterhouse, 2001) Again we found relatively little work in this area but issues of quality are self-evidently important. A lunch club for older people, to take an example used elsewhere, would be of little value if the quality of food provided was poor (leading probably to a negative health impact) or if the club were organised in such a way that the opportunities for social interaction were minimised.
Focusing specifically on the economic impact of the work of VCOs, a recent review (UKVSRG, 2003) of impact assessment within these sectors points to a number of dimensions of economic impact and particularly to impact upon an area’s ‘capital’.
7 In common with most commentators in this field, we refer to assessment rather than measurement of impacts: measurement usually refers to narrow quantitative measures, which are often of dubious value.
18
The position of RCCs can be examined in relation to each of these as appropriate.8 These include:
i. Financial capital: this is a well-understood concept and is not elaborated here.
ii. Physical capital (land, buildings and fixed assets). We are not aware of any general survey that has attempted to review the physical capital of VCOs. A review of one medium-sized city’s voluntary and community sector suggests that the fixed assets within the sector as a whole totalled about £131m. Factored up, this might suggest VCS fixed assets in a typical large rural shire county to be about £0.3bn. In a rural context a host of community assets deserve special mention. Such facilities as village/community halls, playing fields, allotments, burial grounds and recreation facilities are commonly owned or managed by the local community – either by local community organisations or by the parish council. These are important and sometimes under-used resources, which the RCC can indirectly help to ‘valorise’ as real resources for local economic development. Indeed, to the extent that the devolution of assets to communities becomes a significant part of government policy (as foreshadowed for example by ODPM’s Neighbourhood Project), then this approach to economic development is likely to increase in importance.
iii. Human capital: this is the skills, knowledge and personal qualities – self-esteem, confidence etc - which individuals gain as a result of engagement in voluntary sector activity. Any voluntary or community organisation would be able to give examples of people who, having become involved in local activity, have developed important aspects of human capital and, as a result, have often contributed to local economic development. One typical example is that of a young mother who became involved in a nursery centre run on her estate; she came first as a mother, stayed on after a while as a helper, left the group to gain a training qualification and returned as a paid leader. Her confidence and skills and, presumably, self-esteem had grown; a job was created; and, potentially, more parents were freed (through the provision of childcare) to become involved in the labour market.
iv. Social capital: this is a concept which has had a considerable airing in policy circles in recent years but at its simplest it refers to the development of ‘social networks, norms and trust’ (Putnam, 1993), one of the key goals of community development or capacity-building.
The survey undertaken by the UKVSRG cited above identified a wide range of ‘human and social capital’ impacts from the 161 organisations consulted in their work. These included:
Quality of lifeConfidence and esteemSkills, general and personalImproved health measuresParticipation in service provisionEmpowermentImproved standards of livingChanges in organisationPublic awareness of issues
8 The following typology draws on this report.
19
Reduced demand on public servicesEnhanced cultural lifeImproved local environmentPride in the local area
This suggests that in thinking about impacts it is important again to differentiate between direct and indirect impacts and between:
a. impacts on other organisations, including partners (for example, a greater willingness to listen to VCOs, clearer consultation mechanisms);
b. impacts on ‘users’ – both those providing the service and those using it - of the organisation’s activities or services (skills, opportunities, confidence etc); and
c. impacts on the local community or area (e.g. cleaner streets, fewer derelict houses, improved levels of organisation, greater pride).
Again, the balance between these impacts will vary as between different organisations and different contexts, quite apart from the differing goals that organisations may have.
Another way of looking at the impact of the VCS is in terms of the wider value that its activities and services add to the area, in the case of RCCs, the rural areas of a shire county. Bolton’s 2003 study suggests that these incorporate not only outcome value in the ways described above, but also activity value and excellence value. Activity value is another way of referring to the element of process, which is also discussed above and is critical in the delivery of voluntary sector activities and services, in relation to the goals of social inclusion and equity. As Bolton notes, this chimes precisely with the government’s growing emphasis on ‘active citizenship’: active citizens and active communities both lead on to more effective local governance as the system of democratic checks and balances is strengthened by wider participation in political structures and mechanisms.
Excellence value is a more obscure concept except perhaps in the field of the arts and sport where funding for these sectors is quite explicitly aimed at promoting excellence, whether at shot-putting, marathon running, opera or garage music. As Bolton notes, transferring this concept to the general field of social welfare is more problematic but essentially it is about the (hopefully – but not necessarily - positive) impact that an organisation has upon its users, for example in terms of the kinds of social and human capital development discussed earlier.
In terms of what the VCS adds to an area, the element of financial value described above is particularly significant since, even in the case of social enterprises, VCOs are enabled to reinvest any surplus (whether financial or other – e.g. time) into their activities, services or the community more generally.
Bolton (2003) also argues that VCOs can take risks in a way that local government – for financial or political reasons – cannot, and that this attribute should also be seen as adding value since, as most successful entrepreneurs will confirm, without risk there is no innovation. Adding an element of risk into local provision might be seen as one of the more intriguing aspects of voluntary sector activity.
20
One major economic impact of the activities of the voluntary sector lies in its capacity to generate employment. This may happen both directly and indirectly, though the more indirect impacts will be more difficult to determine with any accuracy. As for ‘direct’ effects, national figures suggest that the level of employment in the VCS is about 2% of the total UK workforce, or just under 600,000 paid staff.(NCVO, 2004) This figure – which is regarded by many as an underestimate – is in any case a substantially lower proportion than estimates derived in some regions. It is possible to deduce again a figure for a particular region, sub-region or local authority area on a pro rata basis but that is not particularly useful for the present study other than as an order of magnitude figure. Nevertheless, the generation of employment both directly and indirectly is an important aspect of the work of RCCs. Overall, then, the VCS makes a significant economic impact both in terms of its contribution to a rural area’s GDP – from 2% (modest estimate) to 3% (more generous estimate) – and in terms of jobs created directly (some estimates available) and indirectly (no general estimates available), in terms of volunteer time and in other direct and indirect ways.
What about the wider impacts or outcomes9 of the work of the VCS? A general view of the VCS suggests that it can promote a prosperous economy by providing flexible, local training facilities; raising substantial sums of money for specific projects such as hospices (and about one-fifth at least of local VCO income is from donations) (HM Treasury 2004); undertaking a range of service provision; acting as champions of people unable to articulate their own concerns; promoting self-help, volunteering and citizenship; providing speedy responses to need; being innovative; acting as an advocate; providing a direct link to the grassroots and promoting participation; helping to meet environmental targets; promoting good local governance by facilitating local action planning processes; and providing a range of leisure, arts and cultural facilities.
There is some limited experience of the assessment of wider impacts, which can be built on. For example, the Big Lottery has been working for some years (in its previous incarnations) on the issue of outcomes, to help applicants be clearer about the way they expect to turn Lottery funding into measurable outcomes or, as they put it to applicants, to determine ‘what difference will your project make?’ Its priorities have included, in particular regions, identifying priority beneficiary groups and priority areas such as work on minority ethnic groups and work in the most rural areas to build capacity and voice.
In that regard, the Lottery has developed a series of outcome measures for the VCS in Yorkshire and the Humber. (Community Fund, 2004) These are:
the promotion of positive relationships – making links between communities; improving personal independence – developing skills and confidence,
providing new opportunities and improving access to facilities and services;
9 For simplicity, we are using the terms ‘impact’ and ‘outcome’ interchangeably here to refer to medium or long-term changes arising as a result of some voluntary or community sector activity. A more economic term is ‘value added’. Impact is sometimes taken to be a wider concept than outcomes with an outcome being defined by an organisation and represented within its aims and the impact being all changes – intended and unintended – resulting from the work of an organisation.
21
increasing voluntary and community group capacity by supporting the growth and development of small community-led groups;
reducing isolation by involving/including those who are excluded.
These are fairly general, or ‘high level’ outcomes and are each operationalised in more detail. For example, making links between communities translates into reducing damaging behaviour, greater participation in civic life, increased volunteering or mentoring, promoting good relationships (between parts of the community) and increased knowledge about improving community relationships. The point is not that these outcomes are ones which should be adopted by all funders – they are specific to the Big Lottery – but that it is possible to devise a series of high level outcomes, to operationalise them in more detail and, as the Big Lottery does, to help applicants develop their own thinking, through the application process, about what sort of outcomes they want and expect to achieve. (Community Fund, 2004)
The Lottery scheme is supported by an effective monitoring system which allows it to determine the pattern of grants in any given geographical area, in terms of type of activity, organisation and beneficiary group, and to adjust its own funding priorities accordingly (albeit within parameters set by the Department of Culture, Media and Sport). In a number of instances, the Lottery has had fruitful discussions with other funders such as individual local authorities to ensure that funding patterns are more sensitively aligned.
In similar vein, the NCVO and Countryside Agency undertook a study of the impact assessment of VCOs operating in rural areas, including organisations concerned with environmental conservation, community arts and support services for people with profound sensory disabilities. This also distinguished between impacts at different levels: individual, organisational, neighbourhood or community; and policy levels. All of these impacts are material to the work of RCCs. Critically, this work noted, on the basis of its own research and a review of other studies, that the ‘process [of assessing impact] is different for every community and every project.’ Amongst the kinds of impact identified in this study were reducing social isolation; developing skills and increasing self-esteem (at an individual level); encouraging bridging social capital, raising the profile of villages and breaking down barriers in rural communities (all at community level); and promoting arts amongst other agencies, helping creative industries become more sustainable, and facilitating networking (at the organisational level).
2.3 Measuring the Impacts of Community Development in Rural Areas
Most RCCs have, in their mission statements, an orientation in part towards social inclusion pursued through some form of community development work. This is an even more difficult area in which to devise impact measures since community development privileges process – i.e. how things are done – as much as it privileges outcomes i.e. what changes as a result of an intervention. We do not define community development here although a useful recent definition has been drawn up
22
in the Budapest Declaration. 10 This definition suggests again a potentially very wide range of impacts; and the need to look at the different kinds of impacts which might be observed on organisations, communities, partners and users, as outlined above.
A general review of community development work11 suggests also that there is relatively little experience of detailed evaluation of its impacts, and particularly in rural areas. This review indicates, in summary, that:
there is very little work done in this area; much of what is written strongly supports the case for an increase in the level
and role of community development in rural areas; most of what is done under the banner of evaluation (including the vast
majority of developing country experience) is in fact evaluation of projects or programmes (although sometimes there is a brief discussion of community development goals such as capacity-building);
the evaluation of community development sometimes gets a brief mention within projects and programmes which are related to community development or use it as a tool, but are not focused on it, for example work on social inclusion and rural regeneration 12. Such projects argue the need for promoting community development work in rural areas to strengthen the voice of disadvantaged people in relation to policy development and service provision.
where an evaluation of the impact of community development in rural areas does take place (which is very rarely), it tends to focus on the quantitative measures (outputs), still often required by government departments and other funders, rather than on outcomes.
In addition there are of course many rural projects which have the word ‘community’ in their title and whose work includes perhaps some community development but much else besides.
There is also a very limited literature on the evaluation of community development, either in this country or from abroad. Much of this literature was summarised in a recent article. (Craig, 2003) This review was not directed specifically at rural contexts and doubtless specifically rural indicators would also need to be identified. The article identified the relatively sparse literature on the evaluation of community development in the UK. The paper concluded that there were key elements to be incorporated specifically into the evaluation of community development, in summary:
evaluation itself must be participative: communities must be involved in developing measures
community development requires largely qualitative measures: the difficult task is to find measures for what is valuable rather than simply value what is measurable
there needs to be an emphasis on process and outcomes much more than on short-term outputs
evaluation needs to be framed as a learning experience for communities so that its benefits can be sustainable
10 See www.iacdglobal.org11 Some of this work was done for a separate investigation undertaken by Craig and his associates for the Carnegie Commission on Rural Community Development.12 See for example www.jrf.org.uk/knowldge/findings/foundations/760.asp; and www.jrf.org.uk/knowldge/ findings/housing/n12.asp
23
the evaluation of community development requires a multi-method approach which can capture differences of interest and emphasis within communities
It was realised that in the work of RCCs, it would be important to identify those activities that claim specifically to have a community development component. The literature appeared, however, to us to contain very little work attempting to develop specifically rural measures; this is a mirror of the debates about rural poverty alluded to above. Moreover, such literature as existed seemed to be project or programme specific, often driven by funders’ requirements and based on a limited range of economically-related data, including narrow performance indicators, rather than on a wider and more appropriate mix of data, including, for example, qualitative data (e.g. interviews with end-users and partners) and documentary analysis. Additionally, very little of the evaluative literature in rural development work focused on participatory modes of evaluation. This raised an issue pertaining to the whole of the community development sector. The evaluation of community development has to be guided by a value system that reflects that of community development itself: i.e. it must be participative. It is particularly the case that community development needs a longer timeframe than is generally allowed for within project establishment and evaluation but in practice this often gets squeezed out by the structure of funding arrangements as community development gains may be quite marginal after the first year or two of a project’s life.
It was clear from this review of the VCS as a whole, and specifically that part of it concerned with community development approaches, that it has struggled for some time, with the challenge of rigorously establishing and measuring the impact of its varied activities. It has also had difficulty in maintaining control over the nature of the measures used. Most of all, evaluation has often been driven by the requirements of funding agencies, many of which are short-term, narrowly economic and often inappropriate to the modes of work being considered. What might we learn then from the literature on local economic development?
2.4 Measuring the Economic Outcomes of local economic development initiatives
References on the Internet to measuring economic outcomes are legion and a selection of the more relevant sources is given in Appendix 1 to this report. Much that is written about measuring economic outcomes applies to strategic as well as local economic development. We have identified a number of difficulties involved in measuring the economic outcomes of voluntary and community sector organisations, including:
the inherent problem of measuring outcomes relating to multi-faceted development,
problems of cause and effect; did this funding produce that effect? the heterogeneity of the missions of VCS organisations, with some
championing certain issues; the plethora of interventions being used;
24
the local cost and distraction of collecting and interpreting data that is often felt to ‘miss the point’, or is used for monitoring purposes unrelated to wider evaluative goals.
These difficulties – or more accurately complexities - are echoed in much of the material analysed here.
Many sources emphasise the importance of setting economic outcomes in the context of assessing much broader quality of life outcomes; this is a feature particularly of the debate on the ‘Sustainable Communities’ initiative of ODPM. 13 The relevance to the present study is that, whilst it focuses on economic outcomes, these might result from initiatives pursuing mainly social, environmental, cultural and/or other objectives.
We distinguished above between outcomes and outputs. Many of the sources examined do not make such a distinction, but some do. A typical example of the distinction made is that by ODPM. In its Neighbourhood Renewal Glossary ODPM 14 makes a simple, pragmatic distinction thus: outputs are directly produced by a specific programme, whereas outcomes measure longer-term economic changes. However, in its advice note on local evaluation for regeneration partnerships, ODPM develops the discussion further. Outputs are the apparent direct results of scheme activity; examples include jobs created and number of pupils benefiting from projects designed to enhance/improve attainment. Outcomes are changes in the socio-economic or physical conditions in the target area, which are due to the scheme; examples include a reduction in unemployment and raising educational attainment. The outputs being achieved may translate into the desired outcomes, or they may not. The reason for discrepancies between outputs and outcomes is that socio-economic conditions are affected by many factors (the ‘deadweight’ phenomenon again) other than the activities of a regeneration scheme. ODPM uses a helpful example, which has embedded within it some of the methodological issues addressed in section 4 below.
The desired outcome of creating jobs locally is to reduce unemployment and raise incomes in the target areas, which suffer from exceptionally high levels of unemployment and poverty. By creating 10 jobs in the target area, local unemployment may be reduced by nothing, by 10 (though this is very unlikely), or by any number in between. One possible reason for the difference between new jobs and reduced unemployment is that the new jobs may be displaced [our italics] within the same labour catchment area, making no difference to the total employment opportunities available to local residents. Alternatively, even if the jobs are net additions to the area, they may be filled by commuters from elsewhere, or by workers who move into the area specially.
Evaluation must therefore go beyond outputs to assess outcomes. This in no way detracts from the importance of defining outputs carefully. Achieving outputs is a necessary condition of achieving outcomes. If outputs are non-existent or insufficient, outcomes will be too. This analysis introduces the issues of deadweight (or conversely, additional net benefit) and displacement. Another key issue raised by the
13 See http://www.neighbourhood.gov.uk/glossary.aspo#O.
14 See http://www.odpm.gov.uk/stellent/groups/odpm_urbanpolicy/documents/page/odpm_urbpol. 608055-01hcsp
25
ODPM guidance is that of double-counting, that is noting outputs which are properly resultant from another project or programme.
With reference to deadweight, evaluation should divide relevant changes into three categories:
those that (almost) certainly would not have happened without the scheme; those that (almost) certainly would; those where there is uncertainty.
In relation to displacement, local programmes can displace economic activity in a number of ways, for example:
encouraging an economic activity to locate in one place rather than another;
supporting start-ups or expansions that compete with other activities in the local area so that employment growth is offset by job losses elsewhere.
Finally, regarding double-counting, evaluation should acknowledge the contribution of other programmes to specified economic outcomes (and where appropriate show each contributor’s share of the costs), and the assessment of economic outcomes should take care to avoid aggregating the same benefits from different programmes.
The Treasury Green Book 15 does address the relationship between outputs and outcomes more pointedly, but places outcomes and outputs explicitly within a wider hierarchy that should be clearly set out in appraisals of economic programmes. This hierarchy runs from objectives through outcomes, then outputs to targets:
objectives state what proposals are intended to achieve and may be expressed in general terms so that a range of options to meet them can be considered;
outcomes are the eventual benefits to society that proposals are intended to achieve. They may not always be directly measurable in which case it will be necessary to specify outputs as intermediate steps along the way;
outputs are the results of activities that can be clearly stated or measured , which relate to the outcomes desired;
targets can help progress in terms of producing outputs, delivering outcomes and meeting objectives. These targets should be SMART - specific, measurable, achievable, relevant and time-bound.
The issue of the sustainability of outcomes is also raised in the ODPM guidance note but as with community development impacts, the difficulty is acknowledged that the necessary evidence will not become apparent until some years later. This potential difficulty is raised in a number of sources.
The Audit Commission has been leading an inter-agency group16 to develop a draft set of key indicators mainly for use by local authorities to measure progress towards
15 Source: http://greenbook.treasury.gov.uk/Chapter04.htm
16 This group includes inter alia ODPM: the Local Government Association, the Improvement and Development Agency, the Institute for Economic Development, with the involvement of central government and the Regional Development Agencies. see: http://www.local-pi-library.gov.uk/ documents/EconRegenFeedbackPaper.pdf
26
economic regeneration (and also to link through to broader Quality of Life Indicators with which they can be used in a wider regeneration context – social and environmental as well as economic). No explicit distinction is made, however, between economic outcomes and outputs, although the indicators seem more attuned to measuring outputs than outcomes. Whilst the purposes of the indicators are not exactly coterminous with the purposes of the present project, some methodological issues are relevant. Two of the caveats offered reflect the discussion above:
partnership working as the norm makes it difficult to attribute outcomes to any one organisation;
because of time-lags in the slow process of economic regeneration, the impact of interventions will not be apparent or measurable for some time.
The Audit Commission sets out examples of indicators for economic regeneration. These include for example: the proportion of people of working age in employment, the proportion of unemployed people claiming benefit who have been out of work for more than one year, and the percentage of local jobs by sector that are full-time. The Commission however suggests that it is important to use a number of indicators together ‘to help tell a story’. This is very much in line with the approach of this study. For example, the Commission asks, if local jobs are increasing but unemployment is constant, what is going on? Individual indicators do not unambiguously reflect good outcomes - and the ‘story’ will require some indicators to be collected in more detail than others.
Following up this discussion of indicators, the Improvement and Development Agency (IDEA) (2004), in reviewing the role of local authorities as facilitators in holistic regeneration initiatives in rural areas suggests that some of the best approaches to evaluation have included:
the development of locally applicable indicators to measure change; the use of qualitative as well as quantitative data by talking to customers and
stakeholders the active development of benchmarking to enable comparative assessment.
The Agency emphasises that measuring outcomes at the local level is difficult and that ‘traditional’ indicators such as unemployment rates, indexes of deprivation etc. can give a distorted picture. For example, negligible unemployment rates can mask a very low wage and seasonal economy.
In a useful text on the measurement of the economic impact of projects and programmes Weisbrod and Weisbrod (1997) point to the ‘seven deadly sins of economic impact studies’; these relate closely to some of the methodological difficulties outlined above:
confusing the economic role of a project with its net impact on the economy of an area;
adding together different measures of the same economic change; confusing study areas (e.g. neighbourhood and citywide); confusing time periods (e.g. immediate and eventual); assuming that a facility’s capacity and its actual level of activity are the same; applying multipliers where they do not apply; ignoring market effects on wages and land/building costs, which can also
affect economic competitiveness.
27
Below we provide two examples of major evaluations that have faced the issues outlined above.
28
_________________________________________________________________________Example A: Evaluating the Single Regeneration Budget
The National Evaluation Team from Cambridge University undertook for ODPM a national evaluation of the Single Regeneration Budget. Their study points to conceptual problems in previous evaluation work that had prevented systematic evaluation of regeneration outcomes, suggesting that a new approach using social survey techniques might be more appropriate. They claim that the SRB initiative changes outcomes directly, but only, through the net outputs created and that ‘net outputs affect gross outcomes that sum to produce the net outcome change observed’.
The team identified an approach to assessing the impact of the initiative on regeneration outcomes comprising five elements:
ask the residents themselves about the impact; identify the gross and net outcome changes – benchmark the net outcome changes to
comparators; identify the gross and net output changes – deadweight, displacement and leakage assessed
through beneficiary surveys; consider the net output changes in relation to gross and net outcome changes; estimate the balance of the impact on residents – and within that, the number who have
remained in the area throughout relative to those that have moved away or moved into the area during the period.
The team claimed to take into account deadweight, displacement, ‘leakage’ and long-term effects, but acknowledge that conceptual and measurement problems still abound.
Net outputs are assessed as, for example, number of jobs created, number of residents accessing employment through training or advice, number of business start-ups, amount of new improved building floor space. In this, the approach is almost identical with other attempts to assess outputs. Examples of outcomes achieved in specific areas studied include the following, which might be of interest methodologically to the present study:
a shift away from low income households a statistically significant increase in households earning over £300 per week a statistically significant increase in those employed full-time the employment rate increased at a rate slightly above the national average the proportion of those unemployed fell at a slightly sharper rate than the nation, bringing the
overall figure down to 6% by 2001.17
___________________________________________________________________________
___________________________________________________________________________
Example B: Establishing a Baseline for Evaluation
The Nottinghamshire Research Observatory proposed the development of a baseline position for evaluating the outcomes of the Single Regeneration Budget, recognising that there are multiple benefits associated with the SRB. This study addressed in detail a number of methodological issues to do with measuring outputs and distinguishes between two methods of assessing the effectiveness of an SRB programme:
measuring net outputs – based on gross outputs suitably weighted to take account of deadweight, displacement and other multiplier effects;
to assess the impact of the programme – requiring a measurement of the outcomes, comparing a locality after programme completion with what it would have been like without the programme (partially overcoming the difficulties involved by comparative analysis over time, looking at changes in time series, and with some comparator area). The study acknowledges the difficulties in measuring what the situation would have been like if the programme had not run, and suggests that the only feasible option is to compare the locality with some comparators and to do so over a period of time.
17source:http://www.odpm.gov.uk/stellent/groups/odpm_urbanpolicy/documents/page/odpm_urbpol_608123.hcsp
29
As part of the evaluation the Observatory pointed to the need to distinguish between gross and net outputs, advising that the gross outputs should be identified first and then be reduced by looking at the extent to which they would have happened in the absence of the programme. They advise that an adjustment for deadweight should precede an adjustment for displacement, which in turn should be followed by an adjustment for multiplier effects. In calculating net output they advise on the use of weightings to convert gross output to net output, but counsel the need to take into account ‘softer’ outputs that are not easily reduced to a numerical output measure - but they do not offer any examples of how this might be done.
The Observatory argued that it is important to measure overall outcomes, but that this involves acquiring a considerable body of evidence. It also demands the identification of a baseline point that establishes the economic conditions at a time immediately prior to the beginning of the regeneration programme (but also an indication of whether those conditions were deteriorating, constant or improving up to that point). Somewhat contrary to their assertion about ‘softer’ outputs, the Observatory states that the indicators used to measure baseline conditions should be quantitative even where qualitative change is being assessed. 18
_____________________________________________________________________
In conclusion, the various agencies cited above indicate a very broad range of potential outcomes. The following list indicates the diversity and the lack of tangibility that RCCs might have to deal with when assessing the economic impact of their work:
increasing employment, encouraging business growth, tackling economic disadvantage, promoting the economic viability of localities (Audit Commission),
the value of extra human capital, improvement in the efficiency of job search (Treasury Green Book),
the expansion or contraction of (the area’s) economy resulting from changes in a facility, project or programme (Weisbrod and Weisbrod, 1997),
reducing unemployment and raising incomes in the target areas, which suffer from exceptionally high levels of unemployment and poverty (ODPM)
2.5 Current debates within RCCs
In order to get a sense of the current thinking of RCCs on this outcomes topic, we examined a number of documents held by ACRE, including RCC annual reports, and a small sample of RCC Business Plans. We also undertook an email survey of all RCCs, which yielded a response rate of more than 50%.19
The review of RCC documentation identified a wide range of RCC activities that claimed to lead to economic outcomes, although in terms of the definitions above, it would seem that many of these might more appropriately be regarded as outputs. These included:
Transport schemes: such as wheels to work schemes with identified outcomes including increased number of mopeds into scheme, mopeds purchased locally, number of users who have secured employment/or obtained their own transport; and rural bus schemes where no specific outcomes were identified.
18source:http://www.theobservatory.org.uk/publications/SRB%20Baseline%20Document%20March%2003.pdf
19 We expected a higher return, given the salience of the issue but unfortunately the survey had to take place either side of the 2005 General Election.
30
Community support schemes: mainly community enterprises (encouraging local communities to provide their own facilities and services), with outcomes listed such as: the establishment of a workers co-operative for after-school club, co-operative marketing of garden produce from recycled timber, providing business advice to 9 local groups in one financial year, the establishment of childcare schemes, where outcomes were noted in terms of numbers of childcare places created, the creation of online schemes with 1500 computers into homes in pilot area, and the provision of IT training assistance to 20 small businesses.
Business support schemes: such as support for rural retail schemes through surveys and advice where outcomes included a new post office satellite operation in five villages; four pubs providing a PO service; two county training days provided for village retailers; and 15 business plans produced.
Rural regeneration schemes: several of the RCCs are partners in the distribution of grant aid and provision of support services through the EU Social Fund, Objective 1 and 2, and SRB funding. Here outcomes identified took the form of generalised claims of support, but the most specific examples were of a direct social nature. In one area, a partnership with Business Link resulted in a new farmers’ market
In the context of this study, the key findings from this analysis of RCC evidence are that:
there is no common template against which each RCC’s activities are reported. Although many RCCs undertake seemingly similar initiatives there is very little commonality of titling of schemes or presentation of action and outcomes. In these circumstances it is extremely difficult to create an overall picture of action across all RCCs;
the very nature of annual reports produces a generalised description of activities often with insufficient depth to enable a meaningful analysis of economic outcomes;
in the majority of cases there is no attempt in the material to make any clear link between the activity and its outcome. General expressions of ‘value to a sector or community’ are claimed, or in some cases a figure of ‘people helped’ or ‘money spent’ is indicated, but no clear linkage to outcomes achieved through such action;
as a broad generality the majority of activities reported by RCCs have no claimed economic outcomes. In the main, the outcomes claimed are of a social, community-building nature. Only in those activities related directly to the local economy or employment creation are economic outcomes mentioned.
However the single most significant finding for this research is that the vast majority of activities ascribed to RCCs in their annual reports are of a partnership nature. Very few activities are undertaken exclusively by an individual RCC. Consequently, it is impossible from these documentary sources, to identify an RCC’s actual input into an activity, and therefore, an RCC’s impact.
An illustration of this can be found in the material from many of the RCCs under the heading of ‘Public Transport’. Here the activities are principally funded and delivered
31
via Rural Transport Partnerships (RTPs). In some cases the RTP officers are centred within the RCC, but this does not easily translate into accepting this as an RCC activity. The RCCs are a partner in the RTP, but it is not possible to define to what extent that partnership extends.
The review of a small random sample of RCC Business Plans suggested to us both the emphasis now increasingly being placed by funders on determining outcomes (and the attempts by RCCs to begin to think in a more focused way about these), but also, again, the very wide variety of activity which RCCs were engaged in at a local level.
At the time of this research RCCs were examining a national Framework for Rural Delivery. This was being undertaken through the vehicle of the RCC Network. The purpose of the framework was set out in a paper to the November 2004 RCC Network Policy Conference, ‘to produce:
commonality on how support to rural communities might be packaged, costed and funded;
a means of demonstrating quality standards in delivery on the ground; a means of identifying appropriate outcomes and impact from each area of
work, to enable monitoring of work funded by Defra, voluntary and community sector (VCS) programmes and those of other funders.’
The paper sets out a draft structure of ‘functions of rural support’ (a generic list of RCC activities) and concludes with the following statement on ‘outcomes reporting’:
‘It is impossible to provide a definitive list of outcomes from each of the functional activities reported above. The complexity of RCC work needs a longer developmental process for this to emerge. However, certain principles in the way outcomes are measured are fundamental to any local or national evaluation of investment: social objective interventions often generate a mix of economic, social and
environmental outcomes. Identifying each of these is crucial in an effective reporting system. It will also be central in securing mainstreamed funding for support initiatives.
measuring outcomes or producing evidence of the value of investment should be an integral part of any internal systems of project and programme management and reporting mechanisms. This has implications for internal RCC administration.
outcomes cannot always be tied to specific interventions, but might be the result of interactions between several contributory activities within the community. A better awareness of social auditing and impact assessment techniques will be vital.
outcome measures require data collection systems that may need investment to implement’.
In conclusion, this preliminary review of RCC activities and debates about outcomes suggested that there was considerable – and growing – interest in the subject. This interest was confirmed subsequently at the workshops that we organised for RCC staff and partner organisations towards the end of the study.
However, at present, the literature produced by RCCs does not reflect this interest in terms of the language used or the focus given to it within their annual reports,
32
business plans and so on. In this sense, RCC literature may not be doing justice to the work of RCCs. It also seems to us that relatively little material had been drawn on by RCCs from outside the RCC sector although there are several projects current or recently concluded – such as work on quality standards - which have helped to develop thinking around the measurement of outcomes. In general, the focus in the literature appeared to be mainly on social rather than economic or environmental outcomes, reflecting both the historical development and current focus of RCC activity.
However, it was clear to us that much of this work – transport activities being a clear example - actually could be said to have economic outcomes but that these had not been highlighted in descriptive accounts. The fact that much RCC work is delivered through partnership working makes the issue of attribution particularly problematic and this has to be acknowledged both in any methodology but also by funders interested in defining outcomes.
A further key point strongly stressed by RCCs in discussion with us was that different funders, or funding frameworks (such as those required by Regional Development Agencies or Local Area Agreements) often demand differing measures of outcomes and that it would be valuable if funders agreed a common approach to outcomes measures, albeit allowing an appropriate degree of flexibility.
Finally, in relationship to funding requirements, it was stressed by respondents that evaluative work of this kind has to be an ongoing process, built in to activity from its inception, and also to be costed into activities from their inception
3 Developing the Tool – the Research and Development work
3.1 Introduction.
33
The previous two chapters have examined the need for RCCs to develop and apply an evaluative tool, in relation to their work, such that the outcomes – particularly the economic outcomes- of their work may be better established, assessed and recorded. Those introductory chapters have reviewed the many complexities that are inevitably involved in such work and have examined a good deal of recent experience in the development and application of tools of analysis broadly in this domain. Some of that work has been attempted by the RCCs themselves but much more has been the work of other agencies.
We now move on to describe, in this chapter, just how the research team has set about building on this earlier work to develop a ‘bespoke tool’ for application by the RCCs, and then – in the following chapter – to set out just what we believe that tool should comprise.
As noted earlier, the empirical research fell into four phases, undertaken in sequence over the period March to November 2005:
1. a literature review (already summarised in chapters 1 and 2 above) and an e mail survey of all 38 RCCs (referred to in Section 2.5 above – and see also Appendix 2 below);
2. an intensive ‘case study phase’ involving four RCCs;3. the subsequent creation of the ‘draft tool’;4. the validation and revision of the draft tool by means of development work
involving two further RCCs as well as two workshops bringing together most RCCs and a range of other key actors with an interest in the subject.
The present chapter outlines the second, third and fourth of those stages though it should be noted first that the e mail survey of RCCs generated not only a good deal of useful evidence and ideas regarding current RCC practice in this domain, but also some baseline evidence to guide the selection of RCCs that might assist with the case study and validation phases.
3.2 The Intensive Case Study Phase
The objectives of this stage of the work were: to explore, clarify and validate some apparent links between some activities
undertaken by four RCCs and their apparent economic outcomes; to explore with the four RCCs and some associated stakeholders just how
the tracing of such economic outcomes might in practice be undertaken; thereby to generate one (or more) preferred outcome tracing
methodology/ies for subsequent ‘road-testing’ with two further RCCs.
Four RCCs were invited to collaborate in this work, and they subsequently worked enthusiastically and conscientiously with the research team to make it a success. These RCCs, serving the counties of Bedfordshire, Cumbria, Dorset and Nottinghamshire, were located in four different regions of England, were of different sizes in terms of staffing and turnover and had had differing degrees of prior acquaintance with formal techniques of output and outcome assessment.
34
The University of Hull team worked with two of the RCCs and the University of Gloucestershire team with the other two. In this Phase it was the researchers who undertook most of the detailed work – albeit very fully supported and assisted by RCC personnel. (In the later validation stage, when the object was to see how two RCCs might get on implementing a proposed methodology, the responsibility was necessarily reversed)
About 10 person days of research team time was devoted to each case study – spread largely over the months of July and August 2005.
We will not in this chapter go into detail regarding how the inevitable practical difficulties of the research were addressed in this phase of work - regarding for example the tracing of outcomes and dealing with deadweight, displacement effects and multipliers– since this would involve unnecessary duplication with what is to follow in the next chapter which fully addresses such matters. Rather the aim will be to outline the overall shape of the work.
The first step involved meeting the Chief Executive of the RCC and senior colleagues to outline what would be involved and to secure their cooperation (which was readily given). It was also at this stage that four (in one case six) specific activities undertaken by each RCC were agreed upon for detailed study. These were chosen to provide a mix of discrete RCC projects and broader ongoing areas of RCC work. With regard to each activity it was agreed to focus on what each RCC had done in the 12 month period from April 2003 to March 2004 – though of course attempts would be made to establish outputs and outcomes arising within that period and since. The point was also made (though it was not always easy fully to respect it) that the aim would be to assess the economic outcomes of RCC involvement in those activities rather than of the activities per se.20
The RCC activities under scrutiny were as follows.
BEDFORDSHIRE village halls support managing an E-Rural Bus project Community Transport Brokerage managing a Village Care scheme.
CUMBRIA village halls support – two in particular initiating and supporting a community shop an Internet project relating to local communities running a Rural Women’s Network
DORSET facilitating Parish Plans the work of the Rural Transport Partnership Officer the work of the RCC Village Halls Officer
20 Thus the question would be, for example, “what have been the outcomes of RCC support for this Wheels to Work project?”, rather than “what have been the outcomes of this Wheels to Work project?”
35
the creation and management of a small town community resource centre
NOTTINGHAMSHIRE (which in fact provided six activities for scrutiny) support for the county’s village halls managing a Healthy Living Centre running a Community Engagement programme running a ‘shop in a box’ project running a ‘wheels to work’ project running a village-level arts and crafts project
(Note that the RCCs’ village halls support service was deliberately made common to all four exercises, to see if common lessons might emerge.)
The second step was generally to meet with the most appropriate RCC staff member(s) to establish what they had done in the year in question in relation to the specific activity; the perceived consequences of their action, any reservations they might have about cause and effect, the role of other stakeholders, and some suggestions for validators to be interviewed later on. This work often involved the scrutiny of diaries, worksheets, periodic reports etc – all relating to the 12-month period in question.
The third step involved the researcher in scrutinising a wide range of relevant documentation in RCC files, and sometimes elsewhere, on the same set of matters noted above, as a basis for producing a first draft of a report on the apparent outcomes of RCC involvement in the selected activities.
The fourth step comprised two or three ‘validation interviews’ with people from outside organisations who had been involved in, or were otherwise knowledgeable about, the activities in question. They comprised a mixture of partners, funders and project beneficiaries. In most cases a draft report, as referred to above, was sent to them in advance to give scope for prior thought and a focus to the interview. Without exception the validators proved helpful and provided a good measure of corroboration, occasional correction and extra evidence or opinion.
The final step was to revise the draft reports and discuss them with the appropriate RCC officers for further comments and in some cases minor amendment.
These 18 reports (one per activity) plus overarching county-by-county reflections on the whole exercise by the researcher, then provided a basis for the subsequent drafting by the research team of a model validation tool for ‘road testing’ by two further RCCs.
36
3.3 The Creation of the Draft Tool.
Once the reports of the Case Study Phase were assembled it was quite clear that the nature of RCC work and the highly varied, diffuse and often elusive character of much of the product of that work, were such that any attempt to create an Outcome Tracing Tool that would be largely quantitative was simply out of the question.
The research team therefore turned its mind to clarifying a rigorous process, based on the case study experience, that would culminate in the preparation by the RCC of a report setting out the main economic outcomes of one of its activities – and it was decided to entitle this report ‘Economic Outcomes of RCC Activity’
To draft this tool the research team began by clarifying a list of 18 criteria that any such tool would have to meet if it were to be respected and useful. (This list of criteria is not reproduced here – rather it is included as Annex 1 to the tool, in Chapter 4.)
There followed a creative process, which worked both from the evidence of the Case Study Phase and the list of criteria to produce a first draft of the tool, which was then subject to constructive criticism by the research project Steering Group.
3.4 Validating the Draft Tool
Validating and revising the draft tool had two elements. The first and more important involved the draft tool being ‘road tested’ by two RCCs not involved in the Case Study Phase. The second involved two workshops at which reactions were sought from a range of stakeholders who had had chance to study the draft in advance.
As for the ‘road-testing’ the two RCCs – those serving Durham and Gloucestershire - were selected with reference to the interest they had expressed in the earlier email survey of RCCs and their readiness to devote staff time to this exercise – over a four week period in October / November 2005. It was understood that this time it would necessarily be RCC personnel who would have to apply the tool – albeit carefully supported members of the research team.
DURHAM focused attention on two of its activities, namely: running and managing a retail fruit and vegetable co-op project, providing a combined insurance scheme for community buildings in the
county.The former was a short, fixed-term project, and the latter a longer-term, continuing programme.
GLOUCESTERSHIRE did likewise in relation to: Equity Release Events for Older People across the whole county, the SMART Project: Basic Skills in one extensive part of the county.
In the case of Gloucestershire the work proceeded as follows. The assessor chosen by the RCC had not been involved directly in either of the two projects so that she would be able to apply the tool for both activities with a certain degree of detachment. Both pilot studies were completed up to the draft report stage within two weeks. They
37
were undertaken by the assessor working alone and consulting with relevant actors within and outside the RCC. The assessor conducted validation interviews with five people during the subsequent week, two for the Equity Release project and three for the Basic Skills programme. One researcher attended each of these interviews as an observer. The researchers conducted a final exit interview with the assessor reflecting on the whole process of the pilot study.
The main findings of the pilot were that the RCC found the whole exercise most worthwhile, relevant, timely and useful; it would be a valuable process to undertake as part of their desire to demonstrate value for money, to reveal the economic outcomes of their work and to confirm the achievement of their own goals. The tool itself proved relatively straightforward to implement and produced tangible assessments of economic outcomes. There were few difficulties during the process. The main difficulty centred on assessing economic outcomes. It seems likely that economic outcomes – as opposed to outputs – only become clear after an interval longer than that allowed for in either of the two activities. Validation interviews proved to be an essential component of the process and led in one case to a substantively amended final report.
In the case of Durham the work proceeded in a manner very similar to that for Gloucestershire. The assessor had not had direct involvement with either of the two projects. The draft reports were completed within one week, with the validation interviews being conducted by the assessor some two weeks later. One of the researchers conducted an exit interview with the assessor reflecting on the whole process. Lessons learned from the pilot were very similar to those learned from RCC E, albeit the validators experienced some difficulty with the technical nature of the draft report and, indeed, there were some difficulties identifying suitable validators.
As for the two validation workshops the first involved about 60 RCC Chairs and Chief Executives fortuitously convening in Loughborough for their annual conference. All were sent in advance the draft tool and a brief explanation of the purpose of the research. Those assembled were placed in small discussion groups to consider certain issues pertaining to the tool, and notes were taken at the subsequent plenary discussion.
The second, convened in York, brought together representatives from RCCs and other stakeholders including, a Government Office and two Regional Development Agencies. Representatives attending the workshop were set two tasks. The first task was to assess the strengths and weaknesses of the Tool, while the second explored the means by which the Tool could be incorporated into the funding relationships between RCCs and major funders, identifying any likely difficulties.
3.5 Making Final Revisions
There remained the task of revising the draft tool, in the light of these validation exercises, especially the road testing that had been undertaken by Durham and Gloucestershire. Happily this round of revision did not need to involve major changes – though several very valuable amendments were made. In response to the two pilot studies, amendments were made to the draft tool, mainly by enhancing its clarity for
38
use by RCCs and validators, but also by indicating a greater number and wider range of examples of economic outcomes.
The revised tool was duly presented to the Defra Steering Group in late December 2005 as part of a draft of this Final Report and is set out in Chapter 4 below.
To conclude, the Tool is the product of: initial ideas about teasing out the economic outcomes of RCC activity, set
out in an internal research protocol to guide the ‘case study phase’ involving four RCCs
write-ups of the 18 exercises of ‘outcome tracing’ undertaken with those case study RCCs
revisions to those write-ups, following local validation exercises undertaken in relation to each one
clarifying a long list of criteria which an outcome tracing tool must respect subsequently drafting a comprehensive and detailed ‘draft tool’ the validation of this draft tool principally by means of ‘road testing’
undertaken by two further RCCs subsequent revision.
It is presented in the following chapter in a form that allows its subsequent publication and application.
39
4: The Economic Outcomes Tool: A Good Practice Guide For RCCs
This chapter of the Final report comprises the ‘tool’ that has been developed by the research team, to help RCCs better assess their economic outcomes. It builds on the groundwork reported in chapters 1 and 2, and the research work summarised in Chapter 3. Thus the text that follows is ‘the proposed tool’ in its final version. It has been written such that it can be extracted and reproduced as a free-standing document.
4.1 Preamble
In recent years there has been a growing appreciation that voluntary organisations such as RCCs need to demonstrate more clearly what are the ultimate outcomes of the work that they undertake. And, while accepting that the RCC mission typically embraces a range of social, cultural and environmental aims, as well as some of an economic nature, both the RCCs themselves and several of their funding partners recognise a need to set out more clearly what benefits their work brings specifically to the local economy.
In consequence this tool has been designed to help RCCs identify and assess the economic outcomes of their activities such that:
they can better manage their work to ensure that it is focused on outcomes (whether economic or not);
they can better indicate to their partners, funders and other stakeholders what economic outcomes have occurred as a result of particular activities.21
The tool is the product of the ‘Impact of RCCs’ research project, which ran for much of 2005, funded by Defra and involving researchers from the Universities of Gloucestershire and Hull. Key elements of the research were:
a phase (July / August 2005) of working with four RCCs serving Bedfordshire, Cumbria, Dorset and Nottinghamshire to develop the tool with reference to 18 selected, specific activities undertaken by those RCCs;
a phase (October / November 2005) of working with two further RCCs, those serving Durham and Gloucestershire, to ‘road-test’ the draft tool emerging from the previous phase.
The generous and positive assistance of those six RCCs is gratefully acknowledged.
This Guidance Note sets out the tool concisely. It is not to be followed slavishly, and further improvements are both needed and possible. But we believe that as it stands it offers the prospect of bringing some rigour and precision to a task that is often felt to be difficult, if not impossible, given the elusiveness of much of the product of RCC work and the complexity of the world in which they operate.
21 It is likely that, increasingly, funding agencies will require information about outcomes as a condition of their funding. The ability to provide this kind of information is likely to strengthen the hands of RCCs in bidding for funds for programmes and projects.
40
4.2 An Overview of the Tool
In essence the proposal is that, for each of a number of specific activities that they have recently undertaken, RCCs undertake a series of short analytical exercises, summarising the product of that work in a set of ‘worksheets’ broadly similar to those exemplified in the text boxes set out below.
These worksheets should correspond to the stages of work also set out below, first drawing on evidence held by the RCC itself and then on the contributions of external validators. The subsequent product of this exercise is not some single statistic, or even a document that is predominantly couched in quantitative terms. Rather it should be a validated report of the economic outcomes of RCC activity.
This concise document (typically no more than 4 or pages long for each selected activity and written in a straightforward style) is then a resource both for the RCC’s internal management purposes and for presentation to various external stakeholders and interested parties, notably the funders, partners and beneficiaries of the work in question. It might, for example, help demonstrate the relevance of RCC work to broader sub-regional or regional economic strategies, or help applications for the funding of follow-on work.
Our estimate is that when assessment of economic outcomes has become a normal part of an RCC’s operation the whole exercise for each activity should take no more than two or three person-days of RCC time, albeit spread necessarily over a period of some weeks to allow for validator input. The exercise would be most effectively undertaken if a programme of monitoring essential indicators had been put in train from the beginning of each activity, with the appropriate information being recorded as the activity progressed, possibly relating to key ‘milestones’ over the period.
Seven stages of work are involved in applying the tool and these are dealt with below in this order:
Stage 1: Preparation
Stage 2: Elaboration of the Activity
Stage 3: Outlining the Outputs
Stage 4: Assessing the Economic Outcomes
Stage 5: Checking for Over- or Under- Statement
Stage 6: External Validation of a Draft Report
Stage 7: Conclusion and Final Edit
For each of the first five stages a text box is set out. Each box provides a worked example of the responses that an RCC might make for a specified activity – the example used here is ‘providing the Village Hall Support and Advice Service for the
41
county of Shire A’. The text box for Stage 7 (Box G) suggests a template for the Final report of the economic outcomes of the RCC’s activity.
Before outlining these stages of work, there is a need to define some key terms that are used below. This ‘glossary’ is best set out here, for early scrutiny, as it underpins the whole argument. Note that this is how we are using these terms in the present context; other writers often use them differently. When first used in the text, these words are asterisked.
‘Activity’ - that part of the work of the RCC that is under scrutiny; it could be a discrete project or a broader programme of work
‘Additional Net Benefit’ - the positive consequences accruing locally from RCC work, minus any negative consequences of that work
‘Attribution’ - the ascribing of a consequence to a specific activity or agency ‘Capital’ - a resource of whatever kind that is potentially useful for economic endeavour
(for example, financial, human, social, cultural, physical, environmental or intellectual capital)
‘Deadweight’ - an apparent outcome that is likely to have occurred anyway, without the activity in question having taken place
‘Displacement’ - an apparent outcome that has merely involved the relocation of something within the target zone rather than its creation or enhancement there
‘Economy’ / ‘Economic’ - pertaining to the production, distribution or exchange of goods or services whether marketed or not
‘Local’ - pertaining to all or any of the target zone ‘Multiplier effect’ - any second-round or subsequent beneficial outcomes of the initial
outcome(s) of an activity ‘Output’ - a direct result of an activity, which can be clearly stated or measured ‘Outcome’ - an eventual benefit or disbenefit to society arising from an activity’s outputs –
sometimes referred to as ’impact’ by other commentators ‘Period of the Activity’ - the specific period of the activity under scrutiny in this study (e.g.
April 2003 to March 2004 inclusive) even if the activity began before and continued afterwards
‘Target Zone’ - the geographical area within which outputs and outcomes of the activity are deemed relevant (e.g. all of county X minus town Y)
‘Validation’ - an exercise to confirm, modify or refute claims madeBox A: glossary of terms
4.3 The Tool Stage by Stage
Stage 1: Preparation
Six fairly straightforward preparatory tasks are involved. Normally the RCC Chief Executive would undertake these in consultation with any appropriate colleagues, trustees and other stakeholders, but not with potential validators at this stage.
1. Define the activity*. It may be sensible to choose just two or three activities for scrutiny in any one year. Normally the definition of the activity should be based upon a verb or verbs indicating what the RCC actually did (e.g. ‘launching and managing the such-and-such venture’ rather than ‘the such and such venture’ itself), as this reflects the underlying purpose of seeking the outcomes* of RCC work. If there are any stated aims of the activity, they should be recorded here.
42
2. Define the period of the activity*. This may be the whole duration of a specific project, or it may be just one ‘slice’ of an ongoing RCC programme or activity. The point is that it should be possible unambiguously to define just what the RCC did, and what therefore might be the basis for discrete outputs* and outcomes (though there is no suggestion that the latter will have fallen only into the period of activity. Many will have occurred subsequently). The assessment of outcomes is likely to be best undertaken either in the year following completion of the project / programme or even after a full year has elapsed.
3. Define the target zone*. The mission of most RCCs is confined to a single county, or possibly the rural component of a specific county. That area may be an appropriate basis for defining the target zone, though other definitions may be preferred. This becomes a crucial concept when it comes to considering displacement* and multiplier effects*. Thus for example, an RCC activity that attracts a firm into the target zone creates a genuine output; one that effectively moves a business within the target zone does not.
The RCC activity under scrutinyProviding the Village Hall Support and Advice Service (VHSAS) for the county of Shire A, to include any stated aims
Period-of-the-activityFrom April 2003 to March 2004 inclusive
The Target ZoneThe rural areas of Shire A – defined as the whole county minus the towns of X,Y and Z
Who will do the work? The RCC’s deputy Chief Executive will take the lead in doing the work. She will undertake the validation interviews and write the final report on economic outcomes.
Method to be usedWe will follow the approach advocated in the Defra ‘Good Practice Guide’ (2005)
Main data sources to be used - The quarterly reports of the VHSAS, to the RCC Executive Committee – i.e. those produced in the
period-of-the-activity and subsequent 6 months - The papers for / minutes of the county Capital Grants Committee over the same period - The VHSAS’s files of correspondence with individual halls and with outside stakeholders. - An unsuccessful funding application made by the RCC to the Big Lottery in December 2004. It
sought funding for an Assistant Village Halls advisory officer, and reviewed in some detail the previous two-year’s work.
Box B: illustrating a worked response to Stage 1 - Preparation
4. List the main data sources likely to be useful, including any monitoring reports that have been produced - this is dealt with in more detail below. All pointers from the research emphasise the importance of building in the information requirements from the beginning of the process rather than bolting them on at the end. Include here suggestions for validators (see Stage 6 below). It would be useful even at this early stage to set up interviews with proposed validators, but these should not take place until Stages 1 to 5 have been completed.
43
5. Decide who will do the work and over what period. Decide how much staff time may be devoted to it. Two or three person-days per activity, spread over a few weeks, may often be appropriate if the pre-existing database is reasonably good. As for who will do the work –the assessor - It should be someone other than the person responsible for the activity so that a degree of detachment and objectivity can be achieved.
6. Decide the method to be used. The method, in effect, is that which follows in this Guidance Note. But there may be some reason to depart from, or elaborate upon, it. In any event, talking it through with the person deputed to do the work would be useful at this stage.
All of that having been done, it is time to move on to the exercise itself.
Stage 2: Elaboration of the Activity
Here the task is to set out concisely what the RCC actually did during the period of the activity, making brief reference also to relevant contributions by other actors or agencies. As far as possible the latter (i.e. the work of others) should not be considered as outputs of the activity, but contributory to it or undertaken in parallel; in short, what contribution did partners make?
Activity: Village Hall Support and Advice Service
The activity has comprised a variety of pro-active and re-active tasks carried out by the VHSAS, with clerical support from RCC support staff and occasional support in relation to specific matters / events from other RCC staff and members, notably x, y and z
The main tasks undertaken by the VHSAS over the year were: - addressing, by invitation, 24 meetings in village halls on village hall matters - undertaking 12 ‘village hall audits’ re access for disabled people - giving one-to-one advice / information to village hall trustees, by phone, email, letter and personal
visit. The most popular topics were: the implications of the Disability Discrimination Act, the new licensing legislation, capital funding, and charity legislation and the responsibilities of trustees. In all there was correspondence with 77 hall committees
- producing and distributing to 213 halls and other local stakeholders, four issues of the Shire A Quarterly Village Halls Bulletin
- casework, involving personal visits, with 13 hall committees involving various local disputes and issues
- helping 26 hall committees with specific applications for capital funding (in each case seeing draft applications)
- providing administrative support for the county Joint Capital Grants Committee (4 meetings over the year) – involving in particular bringing forward applications for funding from individual halls, commenting on their merits and on priorities, helping successful applicants to get their projects underway, etc
- organising and helping to run five training events, relating to ‘Licensing and You’; ‘Insurance and Risk Assessment’ ‘Future of Lottery Funding’, ‘Equal Opportunities and Village Halls’, Responsibilities of Trustees’
- liaison with various agencies outside the county – notably national Village Halls Forum and ACRE
Box C: illustrating a worked response to Stage 2 – Elaboration of the Activity
This is the place to record: meetings addressed, newsletters written and circulated, projects launched, support given, etc. This may have been the work of one individual,
44
or of several, depending on how the activity has been defined. Both pro-active and reactive work should be recorded. Obviously this task will be straightforward if good records have been kept - possibly in the form of monthly reports to line-manager, quarterly reports to funder etc. Some measure of quantification and of naming places and groups (village x, parish y, group z) will be useful when it comes to later stages in the work.
Stage 3: Outlining the Outputs
The next stage is to attempt a comprehensive listing (with some measure of quantification where possible) of all the principal outputs of the activity in question. (Some examples of outputs that emerged in the development and piloting of this tool are included in Annex 2 to the Tool). They need not be, and probably many will not be, overtly economic* in character; focusing down onto the economic will come in Stage 4. Also, it should be appreciated that many outputs will have occurred after the period of the activity - and some may have occurred beyond the target zone - in such cases a note should be made of this also.
The sources of evidence to be used should be very largely internal - various documentation assembled as the activity proceeded, including minutes, correspondence, newsletters, reports to sponsors etc. Outside agencies might be contacted for specific pieces of information - but ‘validation*’ will come later. Sources should be cited concisely in footnotes (e.g. ‘postholder’s quarterly reports to Chief Executive’; or ‘sample survey of passengers undertaken in October 1994’).
Activity: Village Hall Support and Advice Service
In this case it has been decided to record as outputs a variety of achievements and results in which the RCC activity (notably some aspect of the VHSAS’s work as outlined above) may reasonably be considered to have been crucial. ‘Crucial’, here, means that the output is unlikely to have happened without the relevant RCC activity
All of these outputs occurred during the period-of-the-activity or shortly afterwards:
- capital sums secured for village halls in Shire A, for the purpose of their (re)construction, refurbishment, improvement or maintenance. Some £240,000 comprising £x from sources within the county (notably the local authorities but also…) and £y from sources outside the county (notably £z from ‘Awards for All’, £a from national charities…etc). Village halls in the target-zone receiving over £5,000 from such sources for such purposes were (a full listing of village halls receiving capital grants would be helpful here)
- village halls (re) constructed / refurbished or significantly improved - Upper Hamlet: new disabled access, re-roofing and other works - Lower Village: structural repairs and addition of a committee room / surgery (a full listing of village halls reconstructed would be helpful here) - the Shire A Village Halls Association established with officers and committee structure in place and
73 halls in membership by end of March 2004. Other partnerships established with Regional Funding Advisors and the Midshire District Council Community Building Funding Scheme
- a total of 84 people attended the five training events. Feedback forms distributed two months later indicated that 66% of those attending were by then making use of ‘at least one thing that I learned’.
Box D: illustrating elements of a worked response to Stage 3 – Outlining the Outputs
45
The assessor should check the accuracy of any figures used in the report (e.g. numbers of beneficiaries, amounts of money transacted)22. When recording figures it is crucial to focus only on those figures relating to the activity that have been influenced by the work of the RCC. It is important also to clarify the census date for the figures used and to acknowledge the implications of any different census dates.
There is no magic formula for defining what are or are not outputs - the key idea is that they are tangible and direct results of an activity. These results may well have depended also in some measure on other activities by other actors - but the point is that as outputs they would not have happened without the RCC activity taking place. Thus the listing of activities (Stage 2) and of outputs (Stage 3) should be complementary even if a clear read-across from the former to the latter is unrealistic.
A note should be made of any doubts or caveats regarding cause and effect; otherwise the implication is that these outputs have flowed directly from the activities listed earlier.
Be as comprehensive as possible - and do not worry if some measure of overlap occurs, with two or more outputs capturing aspects of the same phenomenon.
22 The accuracy of figures should be checked where appropriate with validators in Stage 6 of the process.
46
Stage 4: Assessing Economic Outcomes
Note the earlier definitions of ‘outcomes’ and of ‘economic’. The task here is to try to define what longer-term benefits of an economic nature have accrued in the target zone as a result of the activities and outputs considered earlier. Again, this should first be attempted (here in Stage 4) on the basis of internal information sources, which should be cited in footnotes. Much of this will be assertion or conjecture at first - but in each case corroborating evidence should be sought and included as far as possible. As a general rule: be specific; give concise examples where appropriate; quantify where it is possible, but do not ignore an outcome where this is not possible. See Annex 3 to the Tool for a list of some possible economic outputs / outcomes used to provoke discussion in the early development and piloting of the tool.
The listing of economic outcomes will probably comprise a series of statements (possibly grouped under the five headings suggested below) each with a brief elaboration / justification. Of key importance is a brief indication of why it is believed that the outcome has flowed from the activity / output.
Most, if not all, economic outcomes of RCC activity are likely to be captured under one or more of the following headings, which allow for some repetition, with certain outcomes being expressed in more than one way. This does not matter in what will ultimately be an essentially qualitative report.
1. Increased employment / reduced non-employment in the target zone There may be reference here to jobs created / available/ taken up. The kind of questions to be considered include: ‘What employment has been created?’, ‘Where did this happen?’, ‘How many jobs?’, ‘Were they created directly, or indirectly?’ Are they permanent or seasonal and well paid or poorly paid by local standards? ‘What sort of people have taken up these jobs?’
2. Increased business activity in the target zoneBusiness activity embraces both commercial and social enterprises (i.e. any entity that sells goods or services). The increase may have come about in various ways, especially by means of an increase in local purchasing (by any of the actors and beneficiaries involved in or influenced by the activity), or by means of support given directly or indirectly to a business as a result of the activity. In the case of increased business activity resulting from increased local purchasing, a note should be included on, in broad terms: ‘Who has purchased what from whom?’23 If an order of magnitude estimate can be made concerning the amount of resultant increased business activity, then that should be included. Any improvement (however construed) of business activity should also be noted even if an increase (e.g. in turnover or employment) is not apparent. Note also any decrease or curtailment of local business activity that may have resulted from the RCC activity.
3. Increased / Improved ‘capital’* in the target zone.
23 The text should clearly state that the effect of RCC expenditure itself should be taken into account in assessing economic outcomes.
47
Here capital is used to denote a resource for development, in addition to the increased financial capital indicated under the two headings above. Three kinds of capital are particularly relevant in this context:
human capital - meaning people (notably residents of the target zone) helped to become more able to engage in economic activity. Thus human capital may have been increased or improved by the activity - perhaps by training or educational activity, by the provision of childcare or of transport, or by the engagement of people in some form of community development exercise from which their own personal development ensues;
social capital - meaning networks, organisations and groups and the trust and reciprocity that can come from social interaction. Again, much RCC activity is geared to the production or improvement of social capital; the task here is to establish what social capital has been produced and how it might have, or is already having, economic consequences;
physical capital - meaning ‘things’ that have been constructed or installed as a resource for possible economic activity - such as buildings, vehicles or IT hardware. Helping to improve the stock of community buildings in a parish, e.g. village halls, or land assets such as allotments or playing fields, is also relevant here if local potential for economic development is thereby enhanced.
4. The avoidance or delay of expenditure by the stateHere the focus is on activity that effectively reduces the burden of expenditure falling, or likely to fall upon, local authorities, health authorities, police service etc. Examples might include frail, older people being able to remain living at home for longer, thanks to a village care scheme or to a demand-responsive transport scheme established as part of, or as a consequence of, the RCC activity. Again the challenge is to link cause and effect and to offer reasonably firm evidence of the scale of the saving.
5. Influencing other agencies with regard to their own support of local economic activity This may be loosely characterised as the ‘partnership effect’ of RCC work. An example might be RCC support of Parish Plan exercises, with the subsequent Parish Plans being picked up by Local Strategic Partnerships and with individual partner agencies deciding, say, to give more support themselves to local demand-responsive transport schemes in the target zone.
Note that in Stage 4 of the report there should be comments on all of the five sub-headings 1-5 even if only to report that there are no outcomes for a particular sub-heading. In Box E below an example is set out of a comment that might be included for sub-heading 2: Increased Business Activity in the Target Zone.
Activity: Village Hall Support and Advice Service
The following may be suggested as economic outcomes flowing from the work of the VHSAS and more specifically from the outputs indicated in Stage 3 above. The work of the RCC is regarded as crucial to the achievement of these outcomes as it is unlikely that the improvements to the village halls would have occurred without the RCCs contribution.
Increased Business Activity in the Target Zone
48
Capital expenditure on rebuild/ refurbishment / renovation, as indicated above in Box D, has clearly supported the county’s building and maintenance sector to the value of £x during the year, with work commissioned by village halls and undertaken by local architects and builders
In similar vein may be added the effects of village halls’ purchasing of goods and materials, to the estimated value of £y, consequent on any increase in activity following refurbishment etc – e.g. kitchenware, cleaning materials, furniture, alcohol.
In addition the county’s halls are now better able to provide an appropriate venue for some business activity. This effect is limited by charity law and by the level of demand locally for premises of the kind on offer. But there were (x) examples of health-related surgeries, (y) community or part time village-shops, and (z) markets of various kinds (e.g. farmers markets, car boot sales and bazaars at several halls in the county). Each may be seen as an economic activity in which goods or services are marketed on, or adjacent to, village hall premises.
The improvement of village hall facilities has also increased their attractiveness for various functions, e.g. 27 weddings have used local businesses to provide local catering, a bar, taxis, entertainment etc. In one case a new village hall car park has enabled its hiring out to a caravan club which has in turn created more business for the local pub.
Box E: illustrating elements of a worked response to Stage 4 – Assessing the Economic Outcomes
Stage 5: Checking for Over- or Under-Statement
The whole point of this assessment exercise is to try to establish the additional net benefit* of RCC activity. Unfortunately, however, there is a possibility that the above estimation of outputs and outcomes may incorporate significant error - serving either to exaggerate or to under-represent the consequences for local economic development of RCC activity.
The literature, reinforced by case studies undertaken as part of the research that generated this Good Practice Guide, suggests that five such sources of error warrant systematic attention in the present exercise. Three of them (relating to attribution*, deadweight* and displacement*) involve a potential inadvertent exaggeration of the outcomes of RCC activity. The other two (relating to what are termed ‘multiplier’ or ‘knock-on effects’ and ‘delayed or protracted outcomes’) involve a potential inadvertent under-recording of the true outcomes of RCC activity. The possibility of each of these errors having occurred in identifying outputs and outcomes should be carefully considered.
1. Attribution (sometimes called ‘double counting’)Is there a danger that the RCC is falsely ascribing to its own activity certain outcomes that are more properly the product of other factors or other agencies? In practice this is often a difficult question to answer since so much that RCCs do is undertaken in partnership. The acid test appears to be this; can it be safely argued that the outcome would not have happened had the RCC activity not been undertaken? That does not mean that the outcome has to be the consequence solely of RCC activity; merely that the RCC activity was a crucial ingredient. This potential source of error needs to be acknowledged in relation to each outcome recorded in Stage 4, and appropriate evidence marshalled if causation is in doubt. The contribution of other agencies to achieving outputs and outcomes should be acknowledged and recorded.
49
2. Deadweight*Would the apparent outcome have happened anyway – or differently - even if the RCC activity had not taken place? For example, an RCC might operate a ‘wheels to work’ scheme to help isolated young people get to work or to training. If, of 20 participants in the scheme, three would have managed to get to work or training by some other means of transport, then there is 15 percent ‘deadweight.’ Again the evidence of outputs / outcomes needs to be re-examined, or further evidence assembled to check for deadweight.
3. Displacement* Is the RCC activity in fact not producing additional net economic benefits in the target zone, but only redistributing resources or economic activity that was already to be found in that zone? Two particular types of such displacement, or local redistribution, might be suggested. They are unfortunately quite common:
1) supporting a venture in one place within the target zone only for a parallel activity elsewhere in the zone to be eroded (e.g. helping certain village shopkeepers or postmasters to thrive by taking business from others also in the zone). Hence the need to define the ‘zone’ quite carefully –taking business from an urban supermarket would be quite admissible as an outcome if the target zone excluded that town;
2) mobilising volunteer effort for one economic initiative - e.g. running a community shop - when those volunteers have in consequence to reduce their commitment to other worthwhile causes in the zone.
With regard to both deadweight and displacement, the only rigorous way of estimating their incidence and possible severity is via a survey of beneficiaries and /or other actors. The question, put simply, is: ‘What would you have been doing if the activity had not taken place?’ At the very least the RCC should ponder the possibility of deadweight and displacement with regard to each of the apparent economic outcomes previously identified - and give some sort of reasoned argument for discounting or accepting its existence.
Activity: Village Hall Support and Advice Service
AttributionThe RCC’s village hall support service is delivered through partnerships – working alongside the local authorities and other agencies. But it should be noted that there is no other dedicated village hall support service in the county and that without it village hall trustees would largely be left to their own devices in keeping up to date with possible funding sources, the technicalities of framing persuasive applications, etc. That said, not all of the benefits of the capital expenditure on Shire A’s village halls can be fairly attributed to the RCC; some must surely have accrued to the donors of the funding themselves. By common consent, though, the RCC officer played such a central role in that process that few of the subsequent benefits could be reasonably assumed to have come about without her contribution.
DeadweightIn the absence of the support and commitment of the RCC’s Village halls adviser it is believed that only a small percentage of the money channelled into the county’s village halls in 2003 –04 would have been achieved. It is impossible to say how much – though the increasing sophistication of the fund raising process and requirements of the funding bodies strongly suggests that much of the incoming funding would not have been acquired or would have been channelled to less appropriate locations or activities.
50
DisplacementThis is a potentially serious problem; in some villages and small towns there were other venues that might have proved suitable for the activities taking place in refurbished village halls (e.g. in extended schools) But it may be pointed out that in every successful application for capital funding (as listed earlier) village hall trustees had to demonstrate that possible alternative venues were unavailable or unsuitable.
Multiplier or knock-on effectsA local building firm won a £150,000 contract to help build a new village hall made possible by capital funding secured with the help of the RCC village halls officer. Most of the firm’s workforce lives within the target zone and most of the necessary materials to do the work were also bought locally. To the extent that the wages of the workforce and the receipts of the builders’ merchants were then spent locally, a second-round multiplier effect ensued, which may reasonably be credited to the initial RCC activity. Third and subsequent rounds were also quite likely to occur though the sums involved probably reduced quite quickly to the extent that ‘leakage’ from the target zone occurred.
Delayed or protracted outcomesIn assessing outcomes to March 2004 it is important to note that some of these are likely to have resulted from RCC funding advice and support given up to five years earlier and that such advice and support given subsequently is likely to have outcomes that will take effect up to five years into the future.
Box F: illustrating elements of a worked response to Stage 5 – Recording any Over- or Under-Statement
4. Multiplier* or ‘knock –on’ effects This term is used here to embrace a variety of subsequent or ‘second-round’ effects that may reasonably be attributed to the initial activity.
a) One category of these involves the local economic multiplier, which occurs when local expenditure generates further local expenditure by its recipients, such as when a local building firm employing local labour has won a contract to help build a new village hall and buys building materials locally (see the example cited more fully in Box F above). There will be some further, or third round knock-on effects, but in practice it is not feasible for the RCC to chase all streams of money and to estimate their effects on employment and business (the first two categories of outcomes listed under Stage 4). But some informed speculation on this might be possible.
b) A second type of knock-on effect involves other instances in which ‘one thing leads to another.’ Thus, if a newly refurbished village hall becomes able to offer after-school child care, this may enable a number of local mothers to take up full-time employment, with effects upon themselves (as human capital) and on the businesses / places where they spend their newly gained wages. It may also of course reduce their ability to do voluntary work locally - an example of a possible negative outcome. In practice all that can realistically be attempted is to allude to such knock-on effects when they seem likely to be irrefutable and significant in scale. Evidence is again needed.
5. Delayed or protracted outcomes RCCs frequently and reasonably claim that their work, being often of a community development nature, typically bears fruit many years afterwards. If this is ignored the RCC may seriously sell itself short. One way of partially dealing with this is to delay the whole assessment until a year or two have elapsed after the period of the activity.
51
Even that may be too soon - but delaying much more may allow the evidence trail to go cold, as well as proving of little value for any follow-up action.
In practice the RCC should attempt some sort of qualitative forward projection of likely effects, setting out the assumptions that are being made in so doing. For example, 18 months after a sustained programme of promoting Parish Plans, there might still be expressions of interest coming through with a view to at last launching a Parish Plan exercise. Once begun, such delayed exercises may take another year or more to undertake, and the subsequent projects championed in the plans may then take years to be implemented. The result is that the year of intensive work promoting Parish Plans may quite reasonably be bearing fruit five or even ten years later. Again, the RCC may usefully allude to this, making use of whatever evidence may be available to suggest a protracted flow of outcomes.
To conclude, these five possible sources of over- or under- statement are important. They should not be ignored even if all that can be done is to provide a carefully reasoned exposition of their likely incidence and magnitude.
Stage 6: External Validation of a Draft Report
Stages 1 to 5 should culminate in a draft report on the economic outcomes of a certain RCC activity broadly following the structure set out above. This report should be concise (a few pages), in part quantitative but largely qualitative, carefully written and rigorously argued. As indicated earlier in Stage 1, it is sensible to identify validators at the beginning of the process and to plan the process of validation well in advance; this might include setting the dates for the interviews.
It is now time to seek the report’s validation by a small number of well-placed, external commentators. The RCC will benefit from having, and being able to point to, an externally ratified assessment of the claimed outputs and outcomes of its activities from validators whose objectivity will be respected both within and beyond the RCC. Each validator should know the activity well, but without owing allegiance to the RCC. Some potential validators may have been partners in the venture. Others may be funders who followed events closely, or possibly beneficiaries of some kind, though care should be taken to avoid uncritical enthusiasts.
It is suggested that up to three such validators be sought for each activity and that face-to-face interviews should be undertaken with each, although telephone interviews, and/or written consultation may be possible if the former proves particularly difficult. It is also suggested that the interviews are not undertaken by any RCC officer closely involved with the activity as that might constrain the frankness of the validator. Indeed the RCC might wish to appoint an external person for this purpose.
52
What about end-users as validators? The case for participative assessment is a strong one – i.e. for fully involving the people who have benefited from the activity in question, such as passengers on the minibus or the customers in the community pub. But for two reasons the practical value and cost-effectiveness of this - undertaken as part of the retrospective validation exercise - is likely to be small:
as beneficiaries they may be unlikely to be critical, or to see the bigger picture of the project set in context;
a large survey, rather than reliance on anecdotal comment, would take too long and divert the limited resource of staff time.
It would be more useful if some form of participative assessment were built in during the operation of the activity in question as part of the normal management practice of the RCC rather than as part of the retrospective validation.
Consultation must be based around the draft report, which must have been sent to the validator in advance. When the Draft Report of Economic Outcomes is sent to validators, the terms ‘attribution’, ‘deadweight’, ‘displacement’, ‘multiplier effect’ and ‘delayed outcome’ should be explained clearly and concisely. A proposed format is set out in Annex 4 to the Tool.
The validators should be asked if they are able to verify (or refute) what has been written - and to elaborate on the ‘story’ wherever it seems incomplete. Assurances should be given about the confidentiality of their contributions. (They should be asked to agree to being cited as validators in the final report outcomes, but without their being ascribed any responsibility for the report and without any statements or views being specifically attributed to them)
In interview each validator should be asked to comment in turn on the sections of the draft report -
the activity, its outputs, its outcomes, and the summary of possible under- and over- statements. (The views of the
validators on attribution, deadweight, displacement, knock-on effects and delayed outcomes should be carefully sought and given due weight.)
A note should be made of any reservations and elaborations expressed.
Where the validators’ comments are material to the assessment of economic outcomes, they should be woven into the relevant sections of the final report rather than standing as a separate section, but nevertheless they should be explicitly recognizable in the text, perhaps through the use of italics.
Stage 7: Conclusion and Final Edit
After the validation, it remains to revise the draft report and present it as a concise and validated report entitled ‘Economic Outcomes of RCC Activity’. A suggested template, rather than a worked example, for this is set out in Box G.
This report should be: presented to the Board or Executive Committee of the RCC, which - over
time - will therefore receive a series of reports spanning all or most of the
53
RCC’s activities. The report may be used in framing subsequent work programmes, in the preparation of RCC publicity material, and for staff / trustee training events designed to explore an ‘outcome-oriented’ approach to RCC work;
circulated for comment to a range of partners of the RCC and to the principal funder(s) of the activity in question.
Economic Outcomes of RCC Activity – Final Report
Assumed readership – the RCC’s trustees, various local stakeholders and outside funding bodies, including, for example, Regional Development Agencies. Therefore the final report will need to be clear, concise and self-contained, without cross-referencing to other documents. Technical terms will need to be defined, drawing upon the glossary in this Guidance Note.
A document of some 2,000 to 2,500 words, perhaps four pages, is suggested. Note that the contributions / reservations of the validators may best be woven into the text at appropriate places – perhaps in italicised text thus…”One of the validators added that…” Or “one of the validators felt that there was some exaggeration here…”
1 Introduction Why this report has been produced. How it will be used. How it was created. Method used to produce the report. It was produced by…The validators were …(express anonymously). Brief description of the validation process used, including the way that the validators’ comments contributed to the assessment of economic outcomes. Main sources of information used were… Glossary or 3 or 4 main terms.
2 The activity What it consisted of during the period in question. Where the final report is to be used externally, a brief description of the context for the activity should be included that will allow the reader to understand the purposes of the activity and the involvement of other organisations.
3 Outputs
4 Main economic outcomes
5 Possible under / overstatement
6 Conclusions covering e.g. the overall findings of the value and extent of RCC contributions to the economic outcomes, problems encountered in the process, arrangements for continued monitoring of outcomes, lessons learned for improving the process, and the implications for future work by the RCC
Box G: illustrating a possible template for the final report: ‘Economic Outcomes of RCC Activity’
54
4.4 Annexes to the Tool24
Annex 1 to the Tool: Key Criteria: what should the Economic Outcomes Tool recognise / achieve?
It may be useful to RCCs to know the key criteria that were used to guide the formulation of the tool. Initial criteria emerged from discussion within the research team; these were then tested with various research stakeholders, including those involved in the project’s Steering Group, and subsequently refined; they were then reviewed in the light of the pilot work undertaken with the four RCCs indicated above and finalised within the research team. There are 18 such criteria.
The tool should:
1. Provide RCCs with the means to achieve a reasonably accurate and reasonably comprehensive assessment of the impact upon the local economy of some or all elements of their work
2. Be useful to RCCsa) as part of their normal management processesb) in their dealings with various clients, funders, partners, beneficiaries and end-
users
3. Acknowledge that most RCCs see their social or community (rather than economic) objectives to be central to their mission, while also recognising that economic consequences may flow from all kinds of RCC work, be it essentially social, economic, cultural or environmental in purpose and character
4. Be concise and easily comprehensible and relatively straightforward to apply
5. Recognise a) the ‘enabling’ nature of much RCC workb) that much RCC work is undertaken in partnership with other agencies
6. Be applicable to both limited duration RCC projects and to broad ranging and ongoing programmes of RCC work of very different kinds
7. Require the assembly and analysis by RCCs of firm evidence, whether quantitative or qualitative, rather than speculation alone – while accepting that a ‘summative calculation’ of the economic impact of an RCC, using a common unit of measurement, is impossible
8. Require only evidence that is readily available to the RCC or capable of being assembled without excessive cost or effort
9. Incorporate guidance to RCCs on just what constitute economic consequences and on how they might usefully be categorised
10. Advise on the type of evidence likely to be appropriate for assessing different types of economic outcome
24 As distinct from annexes to the whole research report, which we will call ‘appendices’
55
11. Be able to recognise and assess, with some degree of credibility, any part of the apparent impact which would have happened anyway without RCC action, or involved merely relocating what was already happening (or would have happened) elsewhere in the rural parts of the county
12. Be capable of relating effect to cause with some confidence
13. Be capable of estimating, with some degree of credibility, any ‘knock-on’ economic outcomes of the activity in question, over and above direct consequences
14. Advise on any continuing and systematic monitoring of relevant information that may be needed as the RCC work proceeds
15. Place a good deal of weight on the evidence, assessment and opinions of other stakeholders or ‘third parties’ (i.e. people and agencies outside the RCC itself – such as clients, funders, partners, competitors and beneficiaries/ end-users)
16. Be capable of identifying any negative as well as positive outcomes
17. Recognise that many economic outcomes of RCC activity will occur over an extended period of time, probably several years
18. Encourage an awareness of any major distributional consequences as they might impact upon social exclusion
56
Annex 2 to the Tool: Some examples of outputs from RCC activities
Examples of outputs that emerged in the development and piloting of this tool included:
Parish Plans launched
Parish Plans completed
village halls refurbished or improved
village activities launched
groups / organisations launched,
training courses provided
trainees attending
qualifications achieved
accommodation provided for events
jobs directly created
drivers recruited
people / groups using the facility
newsletters / briefings written and circulated by beneficiaries
passengers carried / trips made
several other outputs were of an explicitly financial nature such as the number (and total value) of grants secured for village halls (or Parish Plans), or the co-finance forthcoming to match those grants
bookkeeping system established
database prepared
business plans completed
membership increased
57
Annex 3 to the Tool: Examples of some possible economic outcomes that were discussed at various stages in the development of this tool
The following might be a useful rule of thumb as a checklist of positive economic outcomes (and here we are admittedly blurring outputs and outcomes to some extent).
1 Pertaining to the performance of the local economy / local businesses businesses created, attracted, sustained greater profitability of local businesses greater efficiency of local businesses greater output / turnover of local businesses greater ‘exporting’ i.e. increased sales beyond the target zone as defined above (to
include sales to visitors and tourists coming in from outside the area) more sustainable business processes and practice greater business confidence more innovation in business processes and products.
2 Pertaining to the local multiplier reduced leakage (i.e. increased purchasing from within the local area, by local
businesses, organisations and households).
3 Pertaining to the working of the local labour market reduction of the number (or percentage) of unemployed people increase in the number (or percentage) of employed people increase in the activity rate (% of residents of working age who are in employment or
actively seeking work) increase in ‘women returners’ to the labour market creation, safeguarding and attraction of jobs increase in full-time and all-year employment greater employment of disadvantaged groups e.g. from BMEs, disabled, long term
unemployed increased recruitment of staff from within, rather than beyond, the local area.
4 Pertaining to an increase / improvement of available human capital more / better / more appropriate training and education more / better / more appropriate transport to work more / better / more appropriate childcare more / better / more appropriate job placement
5 Pertaining to ‘social capital / institutional environment’ as an economic resource more / better institutional support (e.g. development trusts…chambers of
commerce…business networks etc).
6 More / better physical infrastructure or ‘manufactured’ capital (e.g. serviced industrial land, workspace, ICT provision, transport infrastructure).
58
Annex 4 to the Tool: Explanation of terms for validators
‘Attribution’ - the ascribing of a consequence to a specific activity or agency. Is there a danger that the RCC is falsely ascribing to its own activity certain outcomes that are more properly the product of other factors or other agencies?
‘Deadweight’ - an apparent outcome that would however have occurred anyway without the activity in question having taken place. Would the apparent outcome have happened anyway – or differently - even if the RCC activity had not taken place?
‘Displacement’ - an apparent outcome that has merely involved the relocation of something within the target zone rather than its creation or enhancement there. Is the RCC activity in fact not producing additional net economic benefits in the target zone, but only redistributing resources or economic activity that was already to be found in that zone?
‘Multiplier – or knock-on effect’ - any second-round or subsequent beneficial outcomes of the initial outcome(s) of an activity. Are there any secondary economic effects such as where income generated as a result of RCC activity is then spent on local goods and services?
‘Delayed Outcome’ - an eventual benefit or disbenefit to society arising from an activity’s output. Is it possible that some of the economic outcomes from the RCC activity under scrutiny will not be noticeable for some time into the future?
59
Annex 5 to the Tool: A note on monitoring, resourcing and adapting the tool
The monitoring that should be set in train while the activity is underway should be linked to the RCC’s internal processes of activity / staff management. The purpose of the monitoring is to capture much of the necessary information on:
the activity itself - a record of tasks undertaken during the period of the activity by the RCC officer(s) in question – meetings attended, assistance given, etc;
the main outputs of the activity as they unfold (e.g. training courses, money levered in etc);
the beneficiaries - one or more brief user surveys may be appropriate (e.g. asking ‘wheels to work’ beneficiaries what use they are making of the motor scooters, whether they have secured employment, etc). This may be done via brief and simple questionnaires or by telephone interviews. Other stakeholders may help with this monitoring work. The key things are to keep it simple and to ensure confidentiality. A ‘longitudinal’ element may usefully be built in to help chart the picture as it develops over time including after the period of the activity has elapsed.
With reference to the resourcing of this work, and anticipating that RCCs’ reactions may well be that ‘we are not adequately resourced to do this work’:
they should keep the nature of the above exercise simple; they should consider the possibility of building in a ‘premium’ for such work in
tenders, service level agreements, etc. That said, any staff member spending two or three days on this work, as suggested, would experience some measure of staff development as a welcome by-product. This learning consequence is typically part and parcel of any structured self-assessment exercise.
One final point; the tool has been designed so that it could be adapted for use by RCCs in a more comprehensive way - i.e. to capture the much wider range of social, community and other outcomes of RCC work.
60
Chapter 5 Conclusions
1. Concerning the context
1.A Given the changing arrangements for delivering services to rural communities, RCCs have the opportunity to take on wider responsibilities, probably with commensurate funding. In this context RCCs need to be able to demonstrate more convincingly the outcomes of their activities, and it is clear from this study that generally they recognise this.
1.B To date they have been inhibited from making serious progress in this respect by the inherent difficulties of the task, the added complexity of different funders having different reporting, monitoring and evaluation requirements and the resource implications of doing a thorough job.
1.C Economic outcomes are only one kind of outcome resulting from RCC activity but they can and do flow from a wide range of RCC work, much of it not overtly ‘economic’. Other RCC outcomes are mainly social in nature but these outcomes can also be cultural or environmental. All the various kinds of outcome need to be assessed and considered together to fully reflect the nature and benefits of RCC work.
1.D RCCs seem to be enthusiastic about being able to assess and demonstrate the economic outcomes of their activities and they recognise the improvements to their internal management that could flow from this assessment. Thus, assessing economic (and other) outcomes will not only help RCCs better to inform their funders of the value of their work; it will also help them better focus, prioritise and evaluate their work.
1.E RCCs are beginning to recognise that evaluation needs to be embedded in the work of the organisation; this means that it needs to be recognised by funders as a core cost. This study has indicated the order of magnitude of costs involved in undertaking this kind of evaluative work. More generally, feedback from funders suggested that a figure in the order of 3-5% of total annual turnover might appropriately be assigned to evaluation costs.
2. Concerning economic outcomes
2.A An economic outcome is quite a sophisticated concept that can be difficult for some people to understand clearly. So, too, are related concepts such as deadweight, displacement, double-counting, attribution and multipliers. However, many RCCs already appreciate the concepts and their relevance to better management – at least in outline. Some still have difficulties, though, in conceptualising and then assessing the tangible economic outcomes that might result from individual activities. There may therefore be training requirements for RCC staff to become better familiarised with the use and meaning of such concepts.
2.B It should be noted that:
61
economic outcomes may not become manifest or recognisable for some time after an activity has been completed, possibly several years;
some negative economic outcomes might result from RCC activity, though we have found few examples of this.
2.C Economic outcomes cannot and should not be stated in simple arithmetic or cost terms. They are much more complex than that. More valuable than an essentially fruitless attempt to quantify everything is the application of careful and critical thought within a clear framework – and the associated work of data assembly and recording, rigorous analysis, validation and communication. Funders and other sponsors need to recognise this and to appreciate that the assessment of economic outcomes is not just a target-driven means of evaluating their ‘investment’ in an RCC.
3. Concerning the design of the Tool
3.A The Tool devised from the research builds on the above principles and it has been shown to be capable of tracing tangible economic outcomes. It has been designed with the full involvement of a range of RCCs and their national association ACRE. Our consultation exercises demonstrated a good level of support for the Tool, amongst both the RCCs and a number of their funders and stakeholders.
3.B Feedback from our study suggests that RCCs find the Tool relevant, useful and timely; they are able to understand and apply it without too much difficulty, given the caveats set out above. But in the initial period during which RCCs start to adopt and apply the Tool, additional support might need to be given to demonstrate to them how they can identify the economic outcomes of their contribution to particular activities.
4. Concerning the use of the Tool by RCCs
4.A Although the final assessment of economic outcomes is performed retrospectively, RCCs need to plan ahead for the assessment and collect appropriate information during the period that the activities are undertaken. This will make the process of assessment more efficient and less problematic. It therefore means that the process of assessment needs to be embedded into the activity from its design and inception.
4.B The person undertaking the assessment, while being employed by the RCC, should nevertheless be someone who was not directly involved in the original conduct of the activity; a degree of detachment and objectivity is essential.
4.C It is essential to base the assessment of economic outcomes on evidence where this is possible rather than on conjecture. The tool has been set out in some detail together with a worked example; but RCCs should be encouraged to avoid a ‘tick the box’ approach to the exercise. The Tool should therefore be seen as a flexible framework rather than as a list of actions to be ticked off.
62
4.D Assessment is probably best undertaken at least six months, possibly up to two years, after the activity has been completed in order to capture most of the economic outcomes.
4.E Validation is an extremely important part of the assessment process. It refines and corrects the report prepared by the RCC’s assessor and provides a further degree of distance and objectivity that enhances the validity of the process. There are no benefits to the RCC from using an over-sympathetic validator. Selecting validators requires care – balancing a need for detachment and objectivity against a need for a good knowledge of the activity being scrutinised – in this sense there is no such thing as an absolutely independent validator. In some cases validators may be drawn from partner organisations or from funding bodies (though not to validate activities which the funder is responsible for funding).
4.F To get the best value out of validation, terminology should be used as directly and simply as possible. Where it is necessary to use technical terms such as ‘deadweight’ or ‘displacement’, these should be clearly defined.
4.G The final report on the assessment of economic outcomes should be as succinct and to the point as possible to ensure that it is readily understood and useful both for external stakeholders and the internal purposes of the RCC itself.
4.H Assessment will become easier as RCCs get used to applying the Tool. They should not be disheartened if their first efforts seem to be time-consuming. After their initial experience applying the Tool - and if information is readily at hand - they should be able to assess each activity with no more than two or three person-days of work.
4.I To ensure that the process is embedded in the normal working practices of the RCC, the Board should be involved as a matter of course in receiving and commenting on the results of the assessment. This ‘embedding’ is important for political as well as management purposes. We envisage a rolling programme of ‘activity assessments’ along the lines suggested becoming embedded in the ongoing work of the RCCs.
5. Concerning the transferability of the Tool
5.A The Tool, with appropriate amendments, should be transferable to other kinds of outcome from RCC activities, notably social outcomes.
5.B The Tool is probably also transferable to the economic outcomes of organisations other than RCCs, again with relevant modifications.
6. Concerning the research process
6.A The process used during the research was robust. It combined desk research, a survey of all 38 RCCs in England, intensive work with four RCCs, consultation with a
63
range of stakeholders, testing the initial model with two RCCs and with groups of national and regional actors, and finally reflection by the research team on the whole process. There was a marked degree of consistency in the responses of the various stakeholders who took part in the research and, consequently, there is a high level of confidence in the research findings.
6.B In order to test the Tool more fully and possibly amend it, further research should be undertaken to monitor the economic outcomes of selected RCC activities for a period of up to two years after completion. Defra might also consider undertaking broader research in two or three years time to evaluate the overall progress that RCCs have made in applying the Tool.
7. Concerning its wider use
We would encourage Defra to initiate discussions with other government departments, Government Regional Offices and Regional Development Agencies in particular to explore ways in which this Tool, or appropriate variants of it, can be adapted in their partnerships with voluntary sector bodies. This might help reduce the confusing evaluation regimes with which such bodies have to cope.
64
References
Alcock, P., Craig, G. et al. (2001), What counts? What works? Evaluating anti-poverty work and social inclusion in local government, London: Improvement and Development Agency.
Bolton, M. (2003) Voluntary sector added value: a discussion paper, London: National Council for Voluntary Organisations.
Chapman, P., Phimister, E., Shucksmith, M., Upward, R. and Vera-Toscano, E. (1998) Poverty and exclusion in rural Britain, York: Joseph Rowntree Foundation.
Cloke, P. and Edwards, G. (1986) ‘Rurality in England and Wales 1981: a replication of the 1972 Index’, Regional Studies, Vol. 20, No. 4: 289-306.
Community Fund (2004) List of outcomes, Leeds: Community Fund. Mimeo; see also Culpitt, S.D. and Ellis, J. (2003) Your project and its outcomes, London: Community Fund.
Cornish, S. et al. (2002) Under the stones: hidden need in rural Cumbria, Penrith: Northern Fells Rural Group.
Craig, G. (2002) ‘Measuring empowerment: towards the evaluation of community development’, Journal of the Community Development Society, Vol. 33, No. 1.
Craig, G. and Manthorpe, J. (2000) Fresh fields: rural social care, research, policy and practice agendas, York: Joseph Rowntree Foundation.
Craig, G., Taylor, M., Carlton, N., Garbutt, R., Kimberlee, R. Lepine, E. and Syed, A., (2004) The paradox of compacts: monitoring the impact of compacts, London: Home Office.
DunnCraig, G., McNamee, S., Peake, R. and Wilkinson, R. (2005) A21CVCS4Y+H: the future
contributions of the voluntary and community sectors to the Yorkshire and Humber region, Leeds: Yorkshire Forward.
Gaskin, K. and Dobson, B. (1997) The economic equation of volunteering, York: Joseph Rowntree Foundation.
Gore, T., Powell, R. and Wells, P. (2003), The contribution of community businesses to the rural economy of Yorkshire and the Humber, Sheffield: Sheffield Hallam University/ Leeds Metropolitan University.
Hill, C. and Kumi-Ampofo, K. (2004) ‘Supporting community-led economic and social renewal through Structural Funds programmes’ Regional Review, Spring: 12-14.
HM Treasury (2002) Cross cutting review of the voluntary and community sectors, London: HM Treasury.
HM Treasury (2004) Exploring the role of the third sector in public service delivery and reform: a discussion document, London: HM Treasury.
Home Office (2002) Social enterprise: a strategy for success, London: Home Office.Home Office (2005a) Strengthening partnerships: next steps for Compacts, London: Home
Office.Home Office (2005b) Developing capacity: Next steps for ChangeUp, London: Home
Office.IDEA (2004) Facilitating the development of the rural economy: the role of local
authorities in providing effective facilitation, London: Improvement and Development Agency.
NCVO (2003) Voluntary sector infrastructure: a discussion paper, London: National Council for Voluntary Organisations.
NCVO (2004) 16th Annual Voluntary Sector Salary Survey 2004, London: National Council for Voluntary Organisations.
ODPM (2004) Local area agreements: a prospectus, London: Office of the Deputy Prime Minister.
Pharoah, C., Scott, D. and Fisher, A. (2004) Social enterprise in the balance: challenges for the voluntary sector, West Malling: Charities Aid Foundation.
Putnam, R. (1993) Making democracy work, Princeton: Princeton University Press.
65
Sanderson, I., Bovaird, T., Davis, P.,Martin, S. and Foreman, A. (1998) Made to measure: evaluation in practice in local government, London: Local Government Management Board; Martin, S. and Davis, H. ‘What works and for whom: the competing rationalities of best practice’, Policy and Politics, 29 (4): 465-476
Shucksmith, M. (2000) Exclusive countryside?, York: Joseph Rowntree Foundation.UKVSRG (2003) Measuring impact: case studies of impact assessment in small and
medium-sized voluntary organisations, London: UK Voluntary Sector Research Group.Waterhouse, G. (2001) Evaluating quality in the voluntary sector, London: National
Council for Voluntary Organisations.Weisbrod and Weisbrod (1997) accessible at http://www.edrgroup.com/edr1/library/
lib_guides_basic/measuring-the-economic-im.shtml
66
Appendix 1: Websites and other useful resources
Websites
http://greenbook.treasury.gov.uk/annex01.htm
http://greenbook.treasury.gov.uk/Chapter04.htm
http://odpm.gov.uk/stellent/groups/odpm_urbanpolicy/documents/page/odpm_urbpol_608123.hcsp
http://www.acre.org.uk/rcclinks.htm
http://www.ces-vol.org.uk
http://www.countryside.gov.uk/VitalVillages/social_capital/rcc.asp
http://www.defra.gov.uk/rural/factsheet/community.htm
http://www.edrgroup.com/edr1/library/lib_guides_basic/measuring-the-economic-im.shtml
http://www.instituteforphilanthropy.org.uk
http://www.local-pi-library.gov.uk/documents/EconRegenFeedbackPaper.pdf
http://www.neighbourhood.gov.uk/glossary.aspo#O
http://www.neweconomics.org/gen/
http://www.odpm.gov.uk/stellent/groups/odpm_urbanpolicy/documents/page/odpm_urbpol.608055-01hcsp
http://www.theobservatory.org.uk/publications/SRB%20Baseline%20Document%20March%2003.pdf
http://www2.dfpni.gov.uk/economic_appraisal_guidance/evtech.htm
www.iacdglobal.org
www.jrf.org.uk/knowldge/ findings/housing/n12.asp
www.jrf.org.uk/knowldge/findings/foundations/760.asp
www.defra.gov.uk/rural/strategy/annex_a.htm
67
Reports, monographs and books
16th Annual Voluntary Sector Salary Survey 2004, London: National Council for Voluntary Organisations
Alcock, P., Craig, G. et al. (2001), What counts? What works? Evaluating anti-poverty work and social inclusion in local government, London: Improvement and Development Agency.
Bolton, M. (2003) Voluntary sector added value: a discussion paper, London: National Council for Voluntary Organisations
Brown, S., 2003, The Role of the Community Building, ACRE, Gloucs, ISBN 187115765X
Charities Evaluation Services discussion papers 1-8Cornish, S., et al Under the Stones: Hidden Need in Rural Cumbria Northern Fells
Group, Wigton, 2002 Craig, G. (2002) ‘Measuring empowerment: towards the evaluation of community
development’, Journal of the Community Development Society, Vol. 33, No. 1Craig, G., Dornan, P., Bradshaw, J. et al. (2003) Underwriting citizenship for older
people, National Audit Office/Universities of Hull and York.Culpitt, SD. And Ellis, J. (2003) Your project and its outcomes, London:
Community FundDTZ Pieda Consulting (2005) Faith in England’s NorthWest: Economic Impact
Assessment, Manchester: North West Regional Assembly.Gaskin, K. and Dobson, B. (1997) The economic equation of volunteering, York:
Joseph Rowntree Foundation.Godwin, E., and Macdonald, T., Rural services and social exclusion on behalf of
Voluntary Action Cumbria, Home Office/Defra 2005 Gore, T., Powell, R. and Wells, P. (2003), The contribution of community
businesses to the rural economy of Yorkshire and the Humber, Sheffield: Sheffield Hallam University/ Leeds Metropolitan University
Green, G., Grimsley, M. and Stafford, B. et al. (2001) Capital accounting for neighbourhood sustainability, Sheffield and York: Sheffield Hallam University and York Partnership.
HM Treasury (2003) Exploring the Third Sector in public service delivery and reform: a discussion document, London: HM Treasury.
Improvement and Development Agency (2004) Facilitating the development of the rural economy: the role of local authorities in providing effective facilitation (I&DEA, London).
Johnson, M. (2005) ‘Commissioning for change’ Children Now, 20 April: 22-23Martin, S. and Davis, H. ‘What works and for whom: the competing rationalities of
best practice’, Policy and Politics, 29 (4) 465-476Plugging the Leaks: making the most of every pound that enters your local
economy, Bernie Ward and Julie Lewis, New Economics FoundationThe money trail: measuring your impact on the local economy using LM3, JustinSacks, New Economics Foundation and The Countryside AgencySustainable Futures: investing in community based organisations, Stephen Thake,London Metropolitan University, New Economics FoundationSocial return on investment: valuing what matters. findings and recommendationsfrom a pilot study, New Economics FoundationThe need for strategic change in the voluntary sector, Les Hems, Institute forPhilanthropy
68
Putnam, R. (1993) Making democracy work, Princeton: Princeton University PressRegional Economic Strategies for the East of England, East Midlands, North East
and South West.Sanderson, I., Bovaird, T., Davis, P.,Martin, S. and Foreman, A. (1998) Made to
measure: evaluation in practice in local government, London: Local Government Management Board
Taylor, M., Craig, G. et al. (2003) Willing partners? the role of the voluntary and community sectors in promoting democracy, Final report to the Economic and Social Research Council.
UKVSRG (2003) Measuring impact: case studies of impact assessment in small and medium-sized voluntary organisations, London: UK Voluntary Sector Research Group
Village Halls Survey: A report into the condition and use of village halls in Cumbria Voluntary Action Cumbria/Cumbria County Council, 2004
Warwickshire and Worcestershire Rural Partnership Vital Villages SRB Scheme, Final Evaluation Report, Janet Yates Associates, March 2005
Waterhouse, G. (2001) Evaluating quality in the voluntary sector, London: National Council for Voluntary Organisations.
69
Appendix 2: The ‘Scoping Survey Email Questionnaire’
THE IMPACT OF RURAL COMMUNITY COUNCILSEmail survey
Friday 27th May 2005
Dear RCC Chief Officer
We hope that by now Defra have written to you about the study we have been commissioned to undertake by Defra, with the support of ACRE, into the economic impact of Rural Community Council activities.
This brief preliminary survey will help us understand the kinds of work that RCCs are doing in this area and your own thoughts about ‘outcomes’ and related matters. The plan is to then follow up the survey with more intensive work with a few RCCs.
Our remit in this study is not to evaluate Rural Community Councils individually or collectively. It is to develop a tool which will provide support to the RCC network to demonstrate more effectively the economic outcomes – broadly defined - of their activities, and the overall impact of their work. It is intended that this will assist Defra, RDAs and RCCs to a better mutual understanding of the handling of outcomes and outcome measures.
What constitutes an ‘economic outcome’ is certainly open to interpretation, and is something we hope to be in a better position to define upon completion of this preliminary study. As such we would welcome any contributions which interpret ‘economic outcomes’ in the broadest sense. We should add that our research covers ALL types of work undertaken by RCCs with potential economic outcomes, whatever the funding source.
* A short bulletin is attached which provides further information about the study.
We would be very grateful if you would kindly complete this short questionnaire as fully as possible and return it by Friday 10th June to Dr Mick Wilkinson at: ([email protected])
Yours sincerely
Professor Gary CraigCentre for the Study of Social Inclusion and Social JusticeUniversity of Hull
Professor Malcolm MoseleyCountryside and Community Research UnitUniversity of Gloucestershire
A word of explanation about our terminology: we are interested in the economic outcomes or impact of the work of your organisation rather than its outputs: by this we mean longer-term sustainable changes resulting from your work rather than short-term (often numerical) measures. For example, restoring the fabric of a village hall as a result of a grant might be seen as an output of RCC work but having within it an IT centre running training courses, a
70
crèche enabling parents to work, or a credit union, community shop, etc. would clearly be contributing to economic outcomes. There are other possible interpretations of these two terms - ‘economic’ and ‘outcomes’. Please indicate your own thoughts on this below.
We recognise that all kinds of RCC activity – social, economic, environmental, cultural, leisure – can lead to economic outcomes and also that activities may have multiple outcomes. It is also fully accepted that many very valuable RCC activities are not designed to have, and may not have, economic outcomes at all.
1. Please tell us about no more than six types of activity, typical of your RCC’s work programme (whatever their funding source)
which are likely to have had economic outcomes; the economic outcomes which you believe they have led to or may lead to; and whether / how you have tried to measure or assess these outcomes.
It would be helpful to our research if you could, in addition, list under each of the activities, any social or environmental outcomes which may also emanate from them.
2. Please also tell us about:
a. any thoughts you may have on the meaning of the terms ‘outcomes’ and ‘economic’ outcomes.
b. any useful experience, consultancy or project work in which your RCC has been involved which has helped you think about the economic outcomes of your work or how to measure them.
c. any literature (books/articles/project reports/evaluations) which you have found particularly helpful in this respect (if you have copies you can share with us, that would be helpful: send to Dr M Wilkinson, Centre for Social Inclusion and Social Justice, CASS, University of Hull, Hull HU6 7RX).
d. any useful analysis/assessment reports you may have written which our study should consider. This may help us to identify suitable case study RCCs (see q.3 below).
3. In the next phase of this research, we will want to work with a few RCCs for a week or so to explore some of these questions in greater depth and to develop, with their help, a tool to assist them (and RCCs generally) in their work. Although this might perhaps be seen as a bit of ‘organisational consultancy’, courtesy of Defra, we are exploring whether a modest facility fee might be available.
Please tick the box below if you would prefer not to work with us in this way. (Not ticking the box in no way commits you of course; it simply gives us an indication of where we might seek research partners, in due course).
□ I would prefer not to be a ‘case study RCC’.__________________________________________________________________Please print:
Name of this RCCContact personContact email or phone
Thank you. Please return this form by Friday 10th June to Dr Mick Wilkinson at: ([email protected]) Please direct any queries to him.
71
Appendix 3: A listing of some RCC activities
There is a wide range of activities undertaken by RCCs that do not make specific claims to produce economic outcomes. This does not mean that such outcomes do not ensue, but it does suggest that an assessment of their economic value is not felt to be particularly relevant. The following is a summary list of these activities drawn from a representative sample of RCC Annual Reports. It is included to provide an indication of the range of RCC activities for those less familiar with the work of RCCs
RCC 1Community Structure Village Halls Parish Councils Rural Housing Consultation responses RepresentationCommunity Planning Parish Plans Local Strategic Partnerships Market and Coastal Town initiatives CCD consultancy SW network of RCCsCommunity Regeneration Social Inclusion CK Fund Social Entrepreneurship Carnegie UK Trust Community Outreach in rural areas
RCC 2 Rural Transport Partnership
Voluntary Car SchemeCommunity TransportCountryside AccessNew Project grants of £728,000 in 2003/04
Rural Social Inclusion Project Patient and Public Involvement in Health Forums Traveller and Gypsy Sites playbus Community development Projects Rural Housing Project – including 13 housing need surveys Village halls Advisory Service Rural team Conference Rural Enterprise Project
RCC 3Transport A Coast Community Rail Partnership Community Transport Access to Health Project An Integrated Transport ProjectCommunity Development Social Inclusion Supporting Communities
72
The Learning network Building Our own Capacity Social Auditing Local Network Fund for Children and Young People A District Healthy Living Project Village Halls funding and advisory service
RCC 4 Rural Housing Village and Community Halls Rural Transport Partnership Social Inclusion Village of the Year Community Assessments
RCC 5Rural Regeneration Parish Plans Market Town Initiative Community Development Co-ordinationSupporting Voluntary Groups Equipment Loan Scheme Organisational Health Checks Short Course Training Programme Support to Village HallsHealth and Social Care Care groups for the Elderly Safety Events EXTEND – movement to music for those with limited mobilityCommunity Transport Voluntary Car Scheme A Rural Transport Partnership
RCC 6 Identifying Community Needs – e.g. participating in 25 Parish Plans Affordable Housing – 10 housing needs surveys; network of Rural Housing Enablers Village Halls and Community Buildings – including £500,000 in grants Social Inclusion Transport – including 14 parish need surveys Voluntary Sector Network
RCC 7Community and Voluntary Support Local Network Fund for children and young people Rural Stress and Sport Insurance Scheme Resource CentreCommunity Regeneration Community Development Projects Special Needs DevelopmentRural Services Rural Service Advice Rural Housing Enabler project Community Support Co-operation
73
Village Hall Advisory ServiceRural Transport Partnership
RCC 8 Homes for Village People – assessing local housing need Rural Stress Initiative Community Buildings Voluntary Action and Community Care Renewable Energy at the Local Level
RCC 9Rural Regeneration A Rural Enabling Body and Rural Target Fund - £1.4m in 2002/03 on 58 projects Parish Plans Vital Villages A Renewable Energy Network Market Towns InitiativeRural Transport Partnership and Parish Transport GrantsSupporting Voluntary Action CVS network A Small Grant Project Fund
74
