Post on 27-Dec-2015
transcript
Contents Why do we need a common approach to M&E of
common agriculture policy? Who is responsible? How does the policy and the legal framework look like? How is the M&E system built up and what guidance is
provided? Evaluation questions Indicators Evaluation approach What lessons can be drawn from the assessment of
impacts in rural development of the current programming period?
2
When?
Who is responsible?
4/5 June 2009
Challenges of the CMEF & Ongoing Evaluation
3
One policy = one evaluation→ CAP common impacts
Pillar I
European Commission
Pillar II
Member States
Need for coordination and collaboration
Who?How?
Monitoring & Evaluation
For 2007-2013 the CMEF (Common Monitoring and Evaluation Framework) relates only to rural development.
For 2014-2020 there will be one monitoring and evaluation framemork for the whole CAP (Art. 110 of CAP Horizontal Regulation proposal).
Important components will be (among others) indicators and common EQ
4
Legal framework for the evaluation
Objectives of monitoring and evaluation of the CAP
5
RDR, Art.75: Monitoring and evaluation system aims at:
Improving the quality of the design and the implementation programmes;
Demonstrating the progress and achievements of rural development policy;
Assessing the impact, effectiveness, efficiency and relevance of the RD policy interventions.
Technical Handbook of the M&E framework of the CAP
Provides information on:
Objectives and purpose of M&E,
Framework for M&E of the CAP (intervention logic and indicators),
Actors and responsibilities in M&E (incl. Expert group of M&E the CAP),
Data sources,
Rural development specificities for monitoring (MC, AIR, PF) and evaluation (EP, ex ante, CEQ, ex post), including links to guidance documents already published,
Use of M&E information,
Future development,
Annexes: pillar I and II indicators fiches.
6
CMES as part of CMEF
7
CAP Common Monitoring and Evaluation Framework (1306/2013, Art. 110)
Common elementsPillar I specific elements Pillar II specific elements
Responsibility for evaluation: DG Agri
Common Monitoring and Evaluation System (1305/2013, Art. 67 and 808/2014. Art.14)
Common impact indicators
Data sources
Common contextCommon context indicators
Intervention logic: Overall CAP policy objectivesCommon impacts
Pillar I specific objectivesPillar I instruments
Pillar I result indicators Pillar I output indicators
RD priorities/specific objectivesPillar II measures
Pillar II result /target indicators Pillar II output indicators Performance framework
Operation database
Responsibility for evaluation: MAs
Common evaluation questions
Elements of the CMES (Reg.808/2014, Art. 14.1)
a) Intervention logic showing interactions between priorities, focus areas and measures,
b) Common context (including RD related impact indicators), result/target, output indicators a performance review indicators,
c) Common evaluation questions,
8
Fiches for common context, impact, result/target and output indicatorGuidance on use and establishment of proxy indicators Guidance on the Indicator PlanGuidance on performance review and reserve
CEQ related to: →focus areas of RD priorities→other aspect of RDP (RDP synergies and TA)→Union level objectives
Elements of the CMES (Reg.808/2014, Art. 14.1)
d) Data collection, storage and transmission
e) Reporting on monitoring and evaluation activities
f) Evaluation plan
g) Ex ante, AIR 2017 and 2019 and ex post evaluation
h) Support to actors in M&E to fulfil their obligations
9
Common data sources: FADN, Eurostat, others (updated tables for CCI), II, RI.
Operations database and electronic transmission of monitoring data (OI)
Guidance on monitoring covering elements included in the AIR
Guidance on the Evaluation Plan preparation and implementation
Guidance on the ex ante evaluation of RDP 2014-2020
Other guidance documents 2014-2020
Intervention logic
Starting point for evaluation (ex ante, evaluation plan for evaluation during the programming period, ex post)
Composed of hierarchy of objectives (CAP, RD priority, focus areas, measures, operations)
Linked to financial allocations (inputs)
Linked to hierarchy of indicators
10
11
Europe 2020Europe 2020
CAP generalCAP generalobjectivesobjectives
Pillar I specificPillar I specificobjectivesobjectives
InstrumentsInstruments
Smart,
Resource efficiency flagship
sustainable and inclusive growth
CAP
Viable food production Sustainable management of natural resources and climate
action
Balanced territorial development
Contribute to farm incomes and limit
farm income variability in a
minimally trade distorting manner
Improve competitiveness of agricultural
sector and enhance share in food chain
Maintain market stability
Provide public goods (mostly
environmental) and pursue climate
change mitigation and adaptation
Foster resource efficiency through
innovation
Maintain a diverse
agriculture across the
EU
Intervention logic for Pillar I
Innovation Union flagship
Meet consumer
expectations
Direct payments Single CMO Horizontal and other instruments
• Basic payment
• Green payment
• Young farmers scheme
• Small farmers scheme
• Coupled support
• Support in areas facing natural constraints
• Market measures, including exceptional measures
• Producer organizations / Interbranch organisations
• School milk and fruit scheme
• Wine national support program and regulatory measures
• Cross compliance
• Quality policy
• Organic farming
• Promotion policy
• Research / EIP / FAS
Pillar I Intervention logic
12
Europe 2020
CAP generalObjectives
Pillar II specificObjectives (Priotities)
Smart, sustainable and inclusive growth
CAP
2. Competitivenessof all types of
agriculture and farm viability
3. Food chain organisation and risk management
4. Restoring, preserving and
enhancing ecosystems
5. Resource efficiencyand shift towards alow carbon and climate resilient economy
6. Social inclusion, poverty reduction
and economic development in rural areas
Intervention logic for Pillar II
1. Knowledge transfer and Innovation
Viable food production
Balanced territorial development
Sustainable management of natural resources and
climate action
Pillar II Intervention logic
1. Fostering knowledge transfer and innovation in agriculture, forestry and rural areas
RD Focus areas
1A Fostering innovation,
cooperation, and the development of the knowledge base in
rural areas
Relevant measures
Art. 36 C
o-operation
Art. 15 K
nowledge transfer
and information actions
Art. 16 A
dvisory services, farm
managem
ent and farm
relief services
1B Strengthening the links between agriculture,
food production and forestry and research
and innovation, including for the purpose of
improved environmental management and
performance
1C Fostering lifelong learning and
vocational training in the agricultural and
forestry sectors
RDP Intervention Logic
An Intervention logic will be drawn up for each RD priority showing their contributions to selected Focus Areas (including potential contribution of particular measure to several focus areas).
A basic intervention logic is proposed by the EC covering the most commonly expected combinations ( see following slides)
MAs have the flexibility to develop a specific intervention logic appropriate to their territory and its needs.
14
Evaluation
Ex ante evaluation (Art. 8.1 (a) and Art.77)
Evaluation plan (Art. 8.1 (a) )Common evaluation questions
Evaluation Questions (Pillar II)
Evaluation
questions
Policy objectiv
es
Indicators
Common Evaluation Questions: Horizontal and focus area evaluation questions:
define the focus of evaluations in relation to policy objectives, and;
help to demonstrate the progress, impact, achievements, effectiveness, efficiency and relevance of rural development policy.
Programme-specific evaluation questions
Common evaluation questions – 30
Focus area related – 18 questions Other aspects of RDP:
Operational performance - 1
Technical assistance - 1
National rural networks - 1
Horizontal evaluation questions EU 2020 - 5
CAP objectives - 4
4/5 June 2009
Challenges of the CMEF & Ongoing Evaluation
17
“Horizontal” Evaluation Questions
Linked to the overall policy objectives and other aspects of RDPs in order to demonstrate the achievements EU 2020 objectives
CAP objectives
RD cross-cutting priorities (environment, CC, innovation)
National Rural Networks
Technical Assistance
Operational performance (synergies)
18
“Horizontal” Evaluation Questions
Capture the contribution of the programme towards the overall policy objectives in terms of impacts.
Answered with the means of common impact indicators, common context indicators and complementary result indicators (and additional information when necessary)
Reporting in the AIR in 2019 and in the ex post evaluation.
19
Horizontal Evaluation QuestionsExample
20
To what extent has the RDP contributed to the CAP objective of fostering the competitiveness of agriculture?
JUDGEMENT CRITERIA • The agricultural
entrepreneurial income has increased
• The agricultural factor income has increased
• Agricultural productivity has increased
COMMON RD INDICATOR
• Agricultural entrepreneurial income
• Agricultural factor income
• Agricultural productivity
CAP objectiveFostering the competitiveness of agriculture
FA-related Evaluation Questions
Linked to the objectives of the Focus Areas in order to demonstrate the achievements towards the policy objectives
Capture the contribution of the interventions under each FA in terms of programme results
Answered with the means of result indicators (and additional information when necessary)
Reporting in the AIRs in 2017 and 2019 and in the ex post evaluation
21
FA-related Evaluation QuestionsExample22
Focus Area 4c: Preventing soil erosion and improving soil management
To what extent have RDP interventions supported the prevention of soil erosion and improvement of soil management?
JUDGEMENT CRITERIA
• Soil erosion has been prevented
• Soil management has improved
COMMON RD INDICATOR
• % of Agricultural land under management contracts to improve soil management (Result indicator)
• % of forestry land under management contracts to improve soil management (Result indicator)
ADDITIONAL INFORMATION
• Additional information on soil erosion of the land under management contracts
Common indicators 23
Common context indicators (45) refer to RDP context and include impact indicators
Financial (input) indicators refer to resources allocated to measures
Output indicators – PI (22) and PII (27) measure activities implemented within RDP/policy
Result indicators – PI (15) and PII complementary results (6)
measure direct and immediate effects of RDP/policy
Impact indicators (16) refer to benefits both at the level of measure or RDP/policy
Pro
gra
mm
e-sp
ecif
ic i
nd
icat
ors
(R
D)
Context indicators The Common Context Indicators (+ common impact
indicators) of 2014-2020: compulsory minimal set for the SWOT analysis, the RDP strategy and assessment of impacts 45 common context indicators
13 of 16 common impact indicators Socio economic; Sectorial; Environmental
Used in territorial description and SWOT, evaluations
Included in RDP via structured table (all required!) - the use of the common context indicators will be taken into account for the approval of RDPs by the EC
Base on the availble data, sources from EUROSTAT and other EU-level data sources, OR from national/regional sources. (Proxies may be used at regional level)
24
Context (Pillar II) and impact indicators
4/5 June 2009
Challenges of the CMEF & Ongoing Evaluation
25
Pillar I
EU price variability
Agriculture trade balance
Consumer price evolution of food products
Impact indicators
Cover both Pillar 1 and Pillar 2
Some are more relevant for Pillar 1 (e.g. trade related)
Some are more relevant for Pillar 2 (e.g. territorial development)
As far as possible existing datasets (EUROSTAT, Farm Structure Survey, FADN etc.) available at EU, national and/or regional level will be used for quantification.
26
Results indicators
25 result indicators
Show direct achievements of policy
Linked to Focus Areas of P2-5
Mix of:
targets (19) - monitoring
complementary result indicators (6) - evaluation
Requirement to flag operations contributing to Focus Areas with complementary result indicators
4/5 June 2009
Challenges of the CMEF & Ongoing Evaluation
27
Target indicatorsAt least one quantifiable target indicator is required for each Focus Area.
Mostly based on result indicators (19),
Some are closer to output indicators – monitoring (e.g. Priority 1).
Show direct achievements of policy
Linked to Focus Areas of P2-5
The target indicators will be reported on annually in the AIRs using:
Direct monitoring data,
Estimates based on coefficients supplied in the guidance (e.g. to estimate the production of renewable energy from new investments).
In few cases the calculation of the target indicators will be conducted by the evaluator when assessing the achievements of RDP (e.g. water and energy savings)
28
Output indicators
26 indicators at measure level:
Output indicators - for all measures (total public expenditures) Output indicators linked to several measures (area under…) Output indicators linked to specific measures (support for
Leader start up)• Only a selection in the Indicator plan (Planned Outputs to be
quantified)• New indicators for NRN, Risk Management….
29
Programme Specific Indicators
(Article 50(2) 1303/2013; draft RD IA Annex I Part 1 Point 4(a)(i))
Design in SMART way,
Shall:
provide additional info specific to RDP territory e.g.
Definition of rural area
Forestry sector
describe issues where common data lacking e.g.
Innovation
Short supply chains
Local markets
support and justify particular interventions e.g.
Thematic sub-programmes
Programme specific Focus Areas
Methodological approach (PII)31
Setting up the M&E system
- Evaluation Plan- Governance - ToR- Prepare EQ and
indicators
Preparing and Structuring- Intervention logic- Establish methodology- Identify indicators
Observing - Collection of data and qualitative information
Analysing- Process and synthetise data and information- Calculating Net effects
Judging- Conduct
Assessment of impacts
- Answer EQ- Conc&Rec
Reporting and
disseminating
“Standard” AIR – every year from 2016
(Art 75 1305/2013 and 50 1303/2013)
Key information on implementation of the programme and its priorities (Financial and monitoring)
Issues which affect the performance of the programme and the measures taken
Steps taken to implement technical assistance and programme publicity requirements
Actions taken to fulfil ex ante conditionalities (in 2017 and in 2016 where relevant)
+ Annex template on implementation of the financial instruments
Content of "standard" AIR (reporting on evaluation activities)
EP modifications
Evaluation activities undertaken
Activities related to provision and management of data
List of completed evaluations
Summary of completed evaluations
Communication activities
Follow-up of evaluation results
Enhanced AIR 2017
Information from evaluation activities on: Reporting and quantification of programme
achievements, in particular through assessment of the complementary result indicators, and relevant evaluation questions.
Description of implementation of sub-programmes Implementation of actions to take into account the
principles set out in art 6, 7 and 8 CPR (Art. 50 CPR)a) Promotion of equality between men and women and non-discrimination b) Sustainable development (Art. 8 CPR)c) The role of the partners referred to in Article 5 CPR in the implementation of the programme and preparation of the progress report
Enhanced AIR 2019
Information from evaluation activities on:
Reporting and quantification of programme achievements, in particular through assessment of the complementary result indicators, and relevant evaluation questions.
Reporting on interim impact of the RDP: contribution to programme and EU strategy and objectives, in particular through assessment of the programme's net contribution to changes in CAP impact indicator values, and relevant evaluation questions.
Progress made in ensuring integrated approach