LECTURE 8.EVALUATION IN AGRICULTURE AND RURAL DEVELOPMENT
MSc Economic Policy StudiesEC8011: Welfare Economics, Project and Programme Appraisal and EvaluationAlan [email protected] Nov 2011
Outline
The policy context for EU and Irish agricultural policy
Evaluation experiences Evaluation specificities Case studies
Arterial drainage, agricultural research, rural development
Forestry
THE POLICY CONTEXT FOR AGRICULTURE AND RURAL DEVELOPMENT
Agricultural and rural development policy
Multiple objectives Ensuring food security and food at
reasonable prices Regulating food safety Supporting farm incomes Stabilising prices Promoting sustainable agriculture Achieving greater territorial cohesion Encouraging growth in agricultural output
through provision of public goods
The policy framework
Market measures and direct payments provided through Pillar 1 of the CAP
Rural development measures (improving agricultural competitiveness, sustainable land management and territorial cohesion) promoted through Pillar 2 of the CAP
Significant state expenditure through co-financing EU schemes as well as independent schemes and regulatory agencies
Specificities of evaluation in the agricultural and rural development context Covers both policies and programmes Address both socio-economic and
environmental objectives Complexity
Objectives are often very general in nature Difficulties in identifying impacts
attributable to specific measures in the context of multiple intervening factors
The challenges of establishing the counterfactual of what would have happened in the absence of the programme
EVALUATION EXPERIENCIES
Role of the Office of Public Works
Began CB analysis of arterial drainage schemes in 1970
Methodology evaluated by Bruton and Convery (1982) “The Office of Public Works stands alone among
public agencies in Ireland in its willingness to publish its investment appraisal procedures. We have accepted the invitation implicit in this act and have critically analysed the procedures so described. In doing so we are reviewing techniques of analysis which in many – perhaps almost all – cases are no doubt better than those employment elsewhere in government in Ireland. We salute the members of the Office of Public Works who have had the courage and confidence to subject their work to public scrutiny: this volume is dedicated to them.”
Department of Agriculture and Food
Analysis and Evaluation Unit in the 1990s Supported by EU Structural Funds
Value for Money Reviews
Department of Agriculture, Fisheries and Food Beef Classification Scheme, 2005 Compensatory Allowance Scheme, 2006 BSE eradication, 2006 Laboratory Testing for Plant and Animal Diseases,
2007 Farm Waste Management Scheme, 2007 Dairy Hygiene Scheme, 2008 Food Industry Research Measure, 2008 Processing investment aids, 2008 Bovine TB eradication, 2008 Young Farmers Installation Scheme, 2009 Forest Roads Scheme, 2010
Agricultural policy covered by EU standard guidelines for evaluation and IAs Evaluation plan Impact assessment of proposed policy
changes e.g. CAP reform post-2013 Public quality statement on the evaluation
produced ex-post by DG Agri DG Agriculture and Rural Development
web page In addition, the European Court of Auditors
(ECA) periodically undertakes evaluations of particular programmes
EU rural development programmes
In the EU, a formal system of evaluation is in place.
EU member states are obliged by EU legislation to assess the performance of their Rural Development Programmes (RDP).
The EU Rural Development Regulations of 1999 and 2005 require that each RDP undergoes ex ante, mid-term and ex post evaluations according to a specified timetable.
Innovations in evaluation 2007-2013 A more strategic approach concerning the definition
of rural development programmes; A common framework for the monitoring and
evaluation of all measures funded through the European Agricultural Fund for Rural Development;
The establishment of an “ongoing evaluation system” in order to: i) better linking monitoring activities with evaluation needs in terms of data collection; ii) establishing and quantifying baseline indicators and target levels in a timely manner; and regularly assessing the progress of the programmes in achieving quantified goals against baselines;
Setting up of a European Evaluation Network to help establishing good practice and capacity building in the evaluation of Rural Development Programmes.
EU rural development evaluation requirements
Annual MS progress report containing summary of on-going evaluation activities.
In 2010, ongoing evaluation will be in the form of a separate mid-term evaluation report.
In 2015 it will be in the form of a separate ex-post evaluation report.
Also, in 2010 and each year thereafter, each member state will be required to submit to the European Commission a summary progress report implementing its national strategy plan and objectives, and strategic guidelines.
A summary of the midterm and ex post evaluation reports prepared by the member states will be made, under the responsibility of the European Commission, by end-2016.
The evaluation system16
Period 2007-2013
“Ongoing evaluation”
Programming
Ex-ante evaluation
Mid-term evaluation
Policy implementation
Ex-post evaluation
Evaluation network for RD HelpdeskEvaluation expert committee
CMEF
17
1. Community strategic guidelines define EU priorities of each of the axes
2. National Strategy Plans translate EU priorities into strategies in light of national specificities
3. National or regional programmes of measures on the basis of needs assessments and the national strategies
4. Implementation of programmes: monitoring and ongoing evaluation based on a Common Monitoring and Evaluation Framework
EU strategic approach for rural development programming 2007-2013
CMEF indicators
Common baseline indicators (59)
Objective-related indicators + targets Context-related indicators
Common monitoring indicators (95)
Financial indicators Output indicators Result indicators
Common impact indicators (7)
Additional programme-specific indicators
Handbook on the CMEF (Guidance document with 4 annexes consisting of 15 chapters)http://ec.europa.eu/agriculture/rurdev/eval/index_en.htm
ROLE OF MODELS IN AGRICULTURAL POLICY EVALUATION
Model simulations - applications
Implications of moving to decoupled payments
Implications of removing milk quotas Implications of concluding a Doha Round
agreement Implications of meeting climate change
targets
Models as experimental laboratories
Types of economic models Partial equilibrium commodity models
Single-commodity models (food safety regulation, labelling..)
Multi-commodity models (FAPRI-Ireland….) Calibrated or econometric
Partial equilibrium programming models Computable general equilibrium models Microsimulation models (tax-benefit..)
Model characteristics
Time series econometric vs. market equilibrium models
Static vs. dynamic Level of aggregation Establishing the baseline Defining scenarios
Pros and cons of models
Validation Credibility gained through repeated use
(HERMES model…) Expensive to construct and maintain More complex models can become ‘black
boxes’ – difficult to communicate to policy makers (SEAMLESS…)
EXAMPLEARTERIAL DRAINAGE INVESTMENTBRUTON AND CONVERY 1982
Arterial drainage case study
Background Arterial drainage – artificial widening and
deepening of main rivers to increase their effectiveness at draining their catchment areas
Field drainage – the activities necessary to remove surplus water from fields
Interdependent Long history in Ireland Undertaken by the Office of Public Works
Arterial drainage case study
Ireland suffers from drainage problem due to its topography
By preventing flooding, main benefit is to increase the productivity of affected lands
Potential environmental implications for: Water supply Fisheries Wildlife (drainage of wetlands) Aesthetic and amenity considerations
Value of investment ensured only through regular maintainance
Why public involvement?
Arterial drainage is a public good Free-rider problem Transactions costs Role of externalities Income redistribution (using tax revenues to
enhance the productivity of land belonging to more deprived members of the community)
Rationale for involvement in field drainage Less clear cut? Higher discount rates of
elderly farmers
Analytical framework
To assess the net impacts of drainage, the prospective impacts if the land were not drained should be deducted from the prospective impacts with drainage (with-without approach)
Analysis of alternatives should proceed marginally Analysis should examine alternative means of
achieving the same objective (expanding net farm income)
Test validity of assumptions using ex post analysis Adjust flows of funds over time using appropriate
rate of interest
Select project with highest NPV(for mutually exclusive alternatives)
Year A B C
0 -10 -40 -30
10 40 100 60
Results
Present net worth 14.6 21.4 6.8
Internal rate of return (%)
14.9 9.6 7.2
Benefit/cost ratio 2.46 1.53 1.23Choosing the alternative which maximises either the IRR or the B/C ratio can mean foregoing increments at the margin which return more than they cost
Analytical framework
Shadow pricing Estimating the net additional income
accruing to the landholders Cost of labour (if otherwise unemployed)?
Valuation of environmental effects Assigning market values not practical Marginal analysis can help identify threshold
effects Secondary impacts
Multiplier effects – not valid
1945 Arterial Drainage Act
Based on recommendations of a Commission set up to study how the government might best intervene on arterial drainage Should focus on draining entire catchments Govt should bear all the cost of construction Beneficiaries should pay 70% of the assessed
improved annual value of the lands affected towards the cost of maintenance
Significant government subsidy justified on grounds of spin-off effects of greater agricultural prosperity, the positive impacts in making field drainage possible, avoiding remedial costs of deterioriating drainage conditions, indirect benefits to transport, public health, urban areas, sewage outfalls
Arterial draining is an “essential service” and a matter of “national pride”
The brave Mr Hanna (DoF, minority report) Did not accept that the (unestimated) benefits were
sufficient to justify a programme whose cost per acre was almost twice the prevailing price of land
Better agricultural returns could be had by improving existing dry land
The programme offered a “palliative of very limited efficacy for unemployment”
Dismissed the argument of indirect benefits, pointing out that investments in housing and primary education would be far more valuable
Concluded that adoption of the programme would involve “a dissipation rather than a creation of national wealth”.
Catchments should only be drained where returns commensurate with the cost of construction and maintenance could be obtained, and state should never contribute more than 60% of construction cost.
Government decision
Government adopted an arterial drainage programme covering twice the area recommended by the Commission (1.2 million acres rather than 600,000 acres)
28 major catchments, ranked by severity of flooding
Thirty years later, Department of Finance Appraisal Team (1968) noted that, often, cost of drainage exceeded full post-drainage value of affected lands and recommended halt to programme until full cost-benefit appraisal of future schemes
Hence introduction of CB analysis for arterial drainage in 1970
The OPW methodology
Primary benefit: increase in landholder income Secondary benefits: spin-off employment and
incidental benefits to non-farm community Direct costs of arterial drainage Cost of follow-up field drainage and investment in
livestock and buildings associated with the output targets
Environmental impacts not costed but reported in EIA
Fifty year time horizon and discount rate of 3.5% used
Difficulties with OPW methodology
Alternative measures of gains to landholder income Improved vs unimproved value of land NPV of additional income Requires assumption (90%) re proportion of
farmers that will adopt the follow up field drainage
Either estimate very sensitive to base year given volatility in prices/incomes
Treatment of secondary benefits Implications of cost overruns
Bruton and Convery recommendations Found evidence of diminishing returns – the
more cost-effective schemes already undertaken
Undertake more marginal analysis of schemes Introduce some cost-sharing to reduce demand
for what is perceived as a free public good Drop estimation of secondary benefits
(assuming zero shadow price of labour) Do more ex-post analysis Equalise rate of grant among competing
Department schemes
EXAMPLEEVALUATION OF AGRICULTURAL RESEARCH EXPENDITURE
Benefits from public sector R&D
Investment in agricultural research => changes in technology =>increases in agricultural productivity
Research as a public good Measuring the benefits from research
Estimating consumers and producer benefits using a supply and demand model of a commodity market
Using regression analysis to estimate parameters of an aggregate production function model
Both ex-ante and ex-post applications
Issues in measuring R&D benefits
Assessing the relationship between the size of investment in R&D and output or productivity
Identifying the relationships between increased productivity flows and economic benefits
Accounting for the timing of streams of benefits and costs, given lengthy lag times
The basic supply-and-demand model of research benefits
S0
S1
D
P
Q
P0
P1
A B
C
F
GH
Issues
Applies equally to demand-enhancing as to supply-increasing technological change
Clearly identifies distributional implications Measurement issues (slopes of curves…) Taking account of market distortions (price
support, environmental externalities) Potential for understating costs (cost of public
funds…) Potential for overstating benefits (ignoring other
sources of productivity gains, only evaluating ‘winners’,…)
EXAMPLEINDECON MID-TERM EVALUATION OF IRISH RURAL DEVELOPMENT PROGRAMME 2010
Pillar 2 Rural development
As set out in the Rural Development National Strategy, the programme is designed to contribute to the following overarching objectives:
Improving the competitiveness of agriculture by supporting restructuring, development and innovation (Axis 1);
Improving the environment and the countryside by supporting land management (Axis 2);
Improving the quality of life in rural areas and encouraging diversification of economic activity (Axis 3).
LEADER Action Groups (Axis 4)
Terms of reference
Assess the degree of utilisation of resources, the effectiveness and efficiency of the programming of the EAFRD, its socio-economic impact and its impact on the Community priorities;
Answer the Common Evaluation Questions (including the horizontal evaluation questions) required as part of the CMEF guidelines set out by the European Commission;
Assess measures with respect to their balance within the programme;
Judge on the degree to which the Rural Development Programme Ireland contributes to achieving the objectives set out in the national and Community strategy;
Identify the factors that contribute to the success or failure of the programme’s implementation, including as regards sustainability, and the identification of best practice;
Propose measures to improve the quality of the programme and its implementation;
Review the goals of the programme and aim to draw lessons concerning rural development policy;
Economic context required changes in RDP At EU level, the CAP reforms (including the CAP Health Check and the
European Economic Recovery Plan (EERP)) have combined with national budgetary decisions to require significant adaptation of Ireland’s RDP.
Closure of the Farm Improvement Scheme (Measure 121) to new applicants from October 2007;
Suspension of the Young Farmers Installation Scheme (Measure 112) and the related Early Retirement Scheme (Measure 113) to new applicants from October 2008;
Reduction in the maximum hectarage limit supported under the Less Favoured Area scheme (Measure 212) in October 2008; and
Reduction in the rate of the Rural Environment Protection Scheme (REPS) (Measure 214) in 2009 and the subsequent closure of the scheme to new entrants from July 2009.
In 2009, additional funds of €146.3 million were made available under the CAP Health Check modulation agreement and under the EERP. New monies allocated to a number of new schemes including a new investment scheme (the Targeted Agricultural Modernisation Scheme (TAMS)), a new Agri-Environment Options Scheme (AEOS), a new Natura 2000 scheme and a Rural Broadband scheme.
These modifications to the programme were finally agreed in March 2010 while adhering to the requirement to maintain a 10:80:10 ratio of EAFRD funding between Axes 1, 2 and 3-4.
Primary survey research
Survey of farmers supported under the Young Farmers Installation Scheme and under the Early Retirement Scheme – beneficiaries under these two schemes were surveyed together owing to the relationship between the two schemes;
Survey of farmers supported under the Farm Improvement Scheme;
Survey of farmers supported under the Less Favoured Area scheme, the Rural Environment Protection Scheme (REPS), Natura 2000 payments and Vocational Training for REPS;
Survey of Leader Local Action Groups (LAGs) – 36 groups in total.
Overview of expenditure progression
Axis 1 of the RDP aims to improve the competitiveness of agriculture in Ireland through support for farm restructuring, development and innovation
Measures Achieving the transfer of land to young trained
farmers better able to meet the new challenges facing Irish agriculture;
Providing training to beneficiaries under the NATURA 2000 and Agri-environment measures under Axis 2 to optimise delivery of these measures;
Encouraging the transfer of holdings from older farmers to young farmers (setting up or already established);
Supporting capital improvements in farm structures to ensure that primary agriculture becomes competitive and market-oriented.
Axis 1 monitoring indicators
Measure 111 indicators
Measure 111 – Views of participants
Measure 112 – Young Farmers’ Installation Scheme
The objectives of this measure are to: Increase the amount of land transferred to
young, trained farmers better able to meet the new challenges facing Irish agriculture,
Off-set the costs faced by young people when setting up their farm, and
Provide assistance for the investments required on such holdings.
Indicators
Evaluation
While 92.1% indicated that the Scheme was very significant or significant in its contribution to them setting up as a young farmer, only 30% of respondents indicated that they would not have embarked on a career in farming in the absence of YFIS financial supports.
DAFF VFM review of scheme Cost of scheme amounted to €1 for every €3
distributed to young farmers Limited impact in encouraging agricultural
qualifications (other schemes more important) Scheme was discontinued
Measure 113 Early retirement scheme
The objectives of this measure are to complement the YFIS and encourage land mobility. Older farmers are provided with an early pension to encourage the transfer of their land to younger trained farmers.
Indicators
Evaluation
Survey results suggest low deadweight
Axis 2
Axis 2 of the Rural Development Programme 2007-2013 is designed to ensure continued agricultural land use, promotion of environmentally friendly farming practices and preservation of the farmed landscape.
80% of expenditure under the RDP comes under Axis 2.
Measure 212 – Payments to Farmers in Areas with Handicaps: Less Favoured Areas This scheme aims to avoid land
abandonment and the disintegration of farming communities in less advantaged areas by ensuring the continued use of the agricultural land in an environmentally sustainable manner.
Environmental impacts
Although little data exists to prove the level of contribution of this scheme to maintaining the countryside and improving the environment, there are clear benefits to the environment from a farmer retaining their holding, farming in compliance with environmental protection requirements consistent with their statutory obligations and complying with a minimum stocking level.
Due to this measure 3.28 million hectares are now deemed to be successfully contributing to: Biodiversity Water Quality Climate change Improvement in soil quality Avoidance of marginalisation and land abandonment.
Evaluation of LFA scheme
Land abandonment - what is happening to land prices?
Measure 214 – Agri-environmental Payments: REPS scheme
The REPS 4 scheme was designed to promote the following: Techniques for using agricultural land which protect and
improve the environment, biodiversity, the landscape and its features, climate change, natural resources, water quality and the soil;
Environmentally friendly farming processes; The conservation of farmed environments which are of
high nature value; The upkeep of historical features on agricultural land; The use of environmental planning in farming practice; The continued use of agricultural lands (even in
disadvantaged areas); The continued vitality and viability of rural
communities; The conversion to organic production standards.
Evidence of REPS success?
Difference-in-difference approach
Hynes et al (2007) examined six variables over a decade (1995-2005).
This study assumed that input use on REPS farms would develop in a similar pattern to input use on extensive Non-REPS farms in the absence of REPS.
An extensive Non-REPS farm index was developed for the reference periods for the six variables. The extensive Non-REPS farm index was then applied to the base year figures for REPS farms to establish a benchmark for activities in a subsequent year.
The proportionate difference between the benchmark figure in the chosen year and the actual figure was then attributed to the presence of the REPS programme.
Hynes et al (2007) results
REPS and biodiversity
The REPS scheme also aims to improve and promote farming practices that protect and increase biodiversity (wildlife habitats and endangered species of flora and fauna).
Copland and O’ Halloran study aimed to determine whether REPS farms had greater numbers and more diversity of birds than Non-REPS farms. Bird and habitat data were collected from 61 REPS farms and 61 Non-REPS farms from three regions in Ireland over three bird breeding seasons.
The study found that overall there was no increased diversity among REPS farms compared to Non-REPS farms and concluded that REPS had little or no impact on bird diversity and numbers.
“The lack of any significant differences in species diversity between REPS and non-REPS farms is perhaps not too surprising given the “shallow” nature of the scheme. The basic eleven measures within REPS are likely to have limited impacts on bird diversity, with many of the measures requiring habitats to be “retained” or “maintained”, rather than managed, increased, created or improved.”
Further references
DAFF Value for Money Reviews Forestry
Bacon Report 2004, A Review and Appraisal of Ireland’s Forestry Development Strategy
Clinch, P. 1999, Economics of Irish forestry: evaluating the returns to economy and society, COFORD