EuropeAid

ENPI CROSS BORDER COOPERATION

SEMINAR ON MONITORING METHODOLOGY

2. The EC ROM system: BASIC CONCEPTS

BRUXELLES, 9-10 SEPTEMBER 2010

EuropeAidOrigin of ROM

• In May 1999 the Council of the European Union invited the European Commission to establish a performance-based monitoring system with the objective to strengthen Monitoring, Evaluation and Transparency•Monitoring is considered crucial for timely adjustments in

programming. • Each Directorate General (DG) of the EC is called upon to

“design monitoring arrangements that ensure that information on outcomes and use of resources is regularly collected”. • EuropeAid responded to this call with the establishment of

the Results-Oriented Monitoring system (ROM).

2

EuropeAid

What is ROM?

A “snapshot” project by project

According to a standardized methodology

It essentially assesses results and outcomes and not just activities and use of resources

EuropeAidWhat ROM does?

•….provides the EC with a wide range of quantitative and qualitative data on the performance of the funded actions•…..serves as a tool for day-to-day project management by

informing stakeholders about the performance of a specific project, but it also contributes to general EuropeAid policy articulation, implementation and review. •…..thanks to the monitoring data collected in EuropeAid

database, provides an overview of the performance of the overall EC portfolio. •…thanks to data collected by independent experts through

regular onsite missions through a consistent, highly structured methodology, it ensures the quality and the comparability of the collected data.

4

EuropeAidROM in the Project Cycle

5

EuropeAidPurpose of ROM

• ROM system gives an overview how the project portfolio is progressing towards results, helping the answers to the main questions:

- are projects and programmes (likely) to deliver the intended improvements in the lives of people? - is the project, with all due flexibility pursuing its planned course and will it attain its objectives? Is it on track?

ROM can serve as a useful tool for identifying trends which should then be complemented by more in-depth analysis such as evaluations.

6

EuropeAidMain features of ROM system

• There are 5 main features characterizing the ROM system:

The adoption of the accepted DAC evaluation criteriaThe use of external consultants to ensure

independency and transparencyThe application of a common standard methodology

throughout all missionsThe capacity to consolidate data and information to

offer larger views on portfolio performanceThe utilization of standard PCM / LFA approach

7

EuropeAid1. ROM assessment criteria

• The DAC evaluation criteria are the basic points to assess the project performance.• Relevance describes how well a project addresses a real

problem of the beneficiaries and how well it matches the EC development policies strategic objectives.• Efficiency stands for how well the inputs are transformed

into output and outcomes.• Effectiveness measures the degree to which the project’s

outputs have provided benefits and contributed to the project purpose. • Impact describes how and to which degree the project has

contributed to the solution of the problem and to the achievement of the overall objective. • Sustainability measures to the likelihood of a continuation of

the benefits produced by the project after the period of external support has ended. 8

EuropeAidDAC concepts - 21 Assessment criteria

•Relevance & Quality of designRelevance & Quality of design

•1.1 Present level of relevance

•1.2 Present design

•EfficiencyEfficiency

•2.1 Availability of means / inputs

•2.2 Implementation of activities

•2.3 Results achievement

•2.4 Partner contribution

•Effectiveness Effectiveness

•3.1 Access to project results / services

•3.2 Use of project results / services

•3.3 Achievement of Project Purpose

•3.4 Follow up on recommendations

•Impact ProspectsImpact Prospects•4.1 Wider planned impact

•4.2 Monitoring on wider impact

•4.3 Follow up on recommendations

•Potential SustainabilityPotential Sustainability•5.1 Financial/ economic viability

•5.2 Ownership

•5.3 Policy Support

•5.4 Capacity development

•5.5 Socio-cultural adequacy

•5.6 Gender equality

•5.7 Technological adequacy

•5.8 Environmental adequacy

9

EuropeAid2. Who are the monitors?

• ROM Monitors are external independent experts recruited by the ROM contractors (private consulting companies) after approval by the EC. • Externality is required to avoid conflict of interest and

ensure independence of judgment• Selection criteria:Sector/thematic experienceRegional/country knowledgeMix of Senior/Medium/Junior/LocalAvailabilityNo previous direct involvement in the project

• Risks: costs, quality not always adequate, availability, dependence on each expert commitment

10

EuropeAid3. Basics elements of the standard methodology

• The use of a common standardized methodology is one of the most decisive features of the ROM system

• The standard methodology is based on four main instruments:

A) common criteria to select the actions to be monitored

B) common procedures to implement the missions

C) use of common formats for collecting the information through standardized question and grading criteria

D) Common outputs

11

EuropeAid3a. Selection of actions to be monitored

• All projects with budgets exceeding 1 million €

• All projects that have been in execution for at least six months (as of signature of the Financial Agreement etc.)

• All projects that have at least six months remaining before closure

Start Implementation End

- Monitoring visits are avoided when a mid-term review is

taking place (period of six months before and after) - When AIDCO or Delegations specifically request it,

projects below 1M€ can be included (special flexibility with thematic / NGO / SA budget lines)

12

6 months 6 months

€ 1 million

Monitoring period

EuropeAid3b. Procedures to implement the missions

Time allocation for ROM missions:• The monitoring exercise is to be speedy and quick. • Five working days are usually allocated for the field visit

of an ongoing project. •Monitors have one day before the mission to get

acquainted with the project and study the files and two more days after the mission to complete the reports.• So total working days per monitoring report are 8 plus

travel (1 or 2 more)• Ex-post ROM incl. an adequate preparation time to identify

interlocutors can require more days (up to 10 working days).• A ROM mission usually assigns two operations per

monitor.13

EuropeAid

14

EuropeAid3c. Common formats

• 3 special formats:

Project Synopsis provides a concise overview on project background, logic of intervention and administrative data.

Background Conclusion Sheet is the key methodological instrument for ROM providing the standard instrument through a set of questions for monitoring and ensure objectivity, consistency and comprehensiveness.

Monitoring Report is the main document to present findings of the monitoring mission. It includes general and financial information on the project, grades for 5 ROM criteria (obtained via the embedded calculation in the BCS) and a summary of conclusions.

15

EuropeAid3c. Common scoring system

• The grading of a project/programme for each of the five evaluation criteria mentioned gives a quick overview of its performance. In addition, it allows for a comparison between different operations and for the aggregation of overall portfolio performance data. • The grade is meant to be a summary of the answer, rather

than the answer being a justification for the grade• However, grades strongly reduce the complexity of the

issues at hand and their context. They risk overshadowing the actual findings.• In the BCS, monitors should tick the performance

conclusion checkboxes next to the questions after having given an answer in writing.

16

EuropeAid3c. Common scoring system

17

EuropeAid3c. Using the scoring system

18

EuropeAid3.d Standard ROM Outputs

1. A two page monitoring reportmonitoring report: it provides, short explanations and recommendations on quality improvement + important complementary information to internal monitoring

2. In order to guarantee uniformity and consistency it is supported by the ‘Background Conclusion Sheets’‘Background Conclusion Sheets’, which ensure that all monitors address the same questions and issues. Each sub criteria has to be graded and the overall scores for each criterion are automatically calculated and then used in the MR.

3.3. Project synopsisProject synopsis 4. A responseresponse from the field on quality and usefulness of

the report is also uploaded

19

EuropeAid

20

EuropeAid

21

EuropeAidOverview of EU ROM 10 years after

• ROM: a useful instrument at “micro” level. To inform stakeholders on project performance and help project managers “to think in result oriented terms”. ROM’s main objective

• ROM: provides statistics on overall portfolio performance. An overview for management decisions in AIDCO. ROM added value (quantitative analysis)

• ROM reports (stored in CRIS) are one of the sources to inform the programming cycle. Lessons learnt and experiences: e.g. to the Quality Support Groups. ROM in the learning cycle

22

EuropeAidWho benefits from ROM ?

• EuropeAid HQ – broad overview of project performance and identification of patterns of problems that need to be analyzed and supported.

• EC Delegations and Task Managers – timely, independent review of portfolio of projects with identification of principal weaknesses and corresponding recommendations.

• Project Management – provision of independent advice from appropriately qualified experts. Opportunity to assess progress and make necessary adjustments.

• Final beneficiaries - opportunity to discuss progress and improved delivery of project services and outputs

23

EuropeAidROM results after 10 years

ROM on ongoing projects:•More than 13,000 monitoring reports produced on 5500

project •More than 180 countries visited• In 2009 1,548 monitoring reports were produced• The total amount of investments monitored was €8.3 billion. • ROM covered 28% of the expenditures managed by

EuropeAid and 21% of the projects and programmes ongoing in 2009

ROM Ex post:• 700 projects/reports in 100 countriesROM on SPSP :• 34 missions in all regions from 2007 until the mid 2010

24

EuropeAidROM resultsLarge majority of projects are performing well,

Slight improvement in 2009

EuropeAid

2

ROM resultsPerformance by evaluation criteria 2009

EuropeAidUsing ROM data:Factors for good and bad performance

ROM data, especially those collected Ex-Post, can provide valuable lessons learned for future programming and implementation.

A Study on “Causes underlying Effectiveness and Impact of EC Development Projects” identified 6 factors each for good and bad performance:

27

EuropeAidFactors for poor performance:

I. Weak Analysis Stage/Identification Phase

II. Strategic relevance (for EC or partner government) that does not correspond to real relevance for the target groups and final beneficiaries

III. Mismatch between allocated resources and planned objectives

IV. Inadequate risk management

V. Non-existent or very poor internal monitoring systems

VI. Weak Adaptation capacity and Communication among partners

28

EuropeAidFactors for good performance:

I. Proactive managers and inclusive management approaches

II. Implementing partners with strong organisational / technical capacity

III. Strong analysis stage to target real problems/priorities with well-suited strategies

IV. Relevance for beneficiaries addressed throughout the project’s lifetime

V. Demand-driven interventions resulting in widespread access and use of services and high levels of ownership

VI. Strong adaptation capacity as a result of good communication among project partners. 29

EuropeAidSome final points for your consideration

• ROM system is very effective in the production of large amount of information. However to be fully exploited it needs to be “re-processed” and “re-analyzed” in order to extract sound and credible lessons learnt for the future• ROM is a complex system based on a sophisticated

methodology that needs highly qualified and professional human resources at every level (from general management to field missions)• This entails that the system could be relatively expensive: a

single report cost is estimated now around 9/10,000 euros, raising then the question of the “value added” of the investment in case of project of minor size• Giving the cost and the needed overall structure, the

frequency of the missions on same project (re-monitoring) should then be decided with caution

30

EuropeAidActivity B – Identify theory / reality gaps

• Now you have a basic idea of what Monitoring is, how is implemented , what can produce.• Try now to identify theory/reality ‘gaps with the

environment of the ENPI CBC programme you are working in• Are there potential changes required and decision making

implications for ENPI CBC management?• Think in terms of expected results, capacity building,

availability of resources, human resources, inter-country agreement, etc.

31