FAO–GEF project monitoring tool ANNEXES

FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS

Rome, 2020

VERSION 1.0, January 2020

FAO–GEF project monitoring tool

ANNEXES

Annex 1 – UNEG Ethical Guidelines and primary principles underpinning evaluations and reviews in FAO1

1. UNEG Ethical Guidelines

Respect for dignity and diversity Evaluators shall: a. Respect differences in culture, local customs, religious beliefs and practices, personal interaction, gender roles, disability, age and ethnicity, and be mindful of the potential implications of these differences when planning, carrying out and reporting on evaluations, using evaluation instruments appropriate to the cultural setting. b. Keep disruption to a minimum while needed information is obtained, providing the maximum notice to individuals or institutions they wish to engage in the evaluation, optimizing demands on their time, and respecting people’s right to privacy. Rights and fair representation In including individuals or groups in the evaluation, evaluators shall ensure: a. Right to Self-Determination. Prospective participants should be treated as autonomous agents and must be given the time and information to decide whether or not they wish to participate and be able to make an independent decision without any pressure or fear of penalty for not participating. b. Fair Representation. Evaluators shall select participants fairly in relation to the aims of the evaluation, not simply because of their availability, or because it is relatively easy to secure their participation. Care shall be taken to ensure that relatively powerless, ‘hidden’, or otherwise excluded groups are represented. Compliance with codes for vulnerable groups Where the evaluation involves the participation of members of vulnerable groups, evaluators must be aware of and comply with legal codes (whether international or national) governing, for example, interviewing children and young people.

Redress

Stakeholders receive sufficient information to know a) how to seek redress for any perceived disadvantage suffered from the evaluation or any projects it covers, and b) how to register a complaint concerning the conduct of an Implementing or Executing Agency.

Confidentiality

Evaluators shall respect people’s right to provide information in confidence and make participants aware of the scope and limits of confidentiality. Evaluators must ensure that sensitive information cannot be traced to its source so that the relevant individuals are protected from reprisals.

Avoidance of harm

Evaluations can have a negative effect on their objects or those who participate in them. Therefore evaluators shall seek to: minimize risks to, and burdens on, those participating in the evaluation; and seek to maximize the benefits and reduce any unnecessary harms that might occur from negative or critical evaluation, without compromising the integrity of the evaluation.

1 Charter for the FAO Office of Evaluation, 2010.

http://www.fao.org/docrep/meeting/018/k7774e.pdf

2. The primary principles underpinning evaluation and reviews in FAO

Independence Independence should be protected throughout the evaluation process: policy, institutional framework, management of the evaluation function, conduct of evaluations and follow-up. The evaluation function must be located in the Organization outside the line management that it is mandated to evaluate, and have a direct line of reporting to the governing bodies and the Director-General. In this way, it remains separate from those responsible for the design and implementation of the policies and operations that are evaluated2. It must be free from undue influence by management through independent control of the financial and human resources allocated to evaluation, including independent performance assessment of evaluation staff. It must have freedom to design and conduct evaluations according to professional quality standards.

Impartiality Evaluation must be free from bias. This means that evaluators must demonstrate professional and

personal integrity and conflicts of interest must be avoided. Independence and quality of evaluation

design are additional pre-requisites for impartiality. Evaluations must value the input of the main

stakeholders, demonstrating a degree of empathy while at the same time maintaining intellectual

rigour. Because no individual is totally impartial, evaluation teams must balance different perspectives

and backgrounds.

Credibility Evaluations must command a high degree of credibility, both from the governing bodies and from

managers who must make and implement decisions. Besides impartiality and independence, the

credibility of evaluation also requires that the team of evaluators has proven technical competence in

the area under evaluation and its context as well as demonstrated competence in evaluation.

Independent peer review of evaluation reports also reinforces their credibility.

Transparency Evaluation reports and management responses are in the public domain. Evaluations follow a

consultative process, whereby evaluators and evaluation managers engage in dialogue to the maximum

extent possible with main stakeholders throughout the evaluation process.

Usefulness Usefulness should always be a prime consideration for selection of a topic for evaluation. Evaluations

will be most useful when addressing key areas of concern for the governing bodies and/or FAO

management, especially when there are perceived to be problems, priorities are changing or if there

are new opportunities. Evaluation should be timed to fit into the management decision-making process.

2 For FAO-GEF projects, the Budget Holder (BH) is responsible for the MTR. However, the BH has overall responsibility for the

project/programme, so is not independent of its line management. Consequently, to ensure a level of independence, the appointed MTR team comprises consultants who have not been involved in any stage of the project/programme being assessed. In addition, a dual reporting role,

whereby the MTR team reports to both the BH and the GEF Coordination Unit Focal Point in Rome, has been introduced to improve independence and reduce the potential for conflicts of interest.

1

Annex 2 – Key stakeholder roles and responsibilities in MTRs of FAO‒GEF projects This annex gives further detail on the roles and responsibilities of the main individuals and groups involved in planning and implementing an mid-term review (MTR) for a FAO‒GEF project. 1. Budget holder or MTR manager The FAO project budget holder (BH) – the FAO Representative in the case of a single-country project – is responsible for project execution, managing project results, timely implementation and reporting and for ensuring appropriate use of resources, including sound financial management (or oversight of finances under the Operational Partners Implementation Modality (OPIM – for more, see Annex 3). The FAO project BH is, thus, responsible for ensuring that the MTR is planned in time with adequate provisions, but receives guidance and support from the FAO GEF Coordination Unit (FAO GEF CU) programme officer (focal point). The BH, in coordination with the Project Management Unit (PMU), facilitates communication with government counterparts, partners and the GEF Operational Focal Point (OFP) to schedule the MTR, including the organization of briefing and debriefing sessions or workshops in the country. The BH is responsible for preparing and advertising the terms of reference and recruiting the external consultants for the MTR in consultation with the FAO GEF CU, lead technical officer (LTO) and other relevant Project Task Force (PTF) members. While the BH has overall responsibility for the MTR, if the FAO GEF CU agrees, he/she may appoint an MTR manager (RM) from an FAO decentralized office or divisional staff to manage and coordinate day-to-day MTR activities, including the contracting of independent MTR consultants and arranging mission logistics. However, to ensure the independence of the process, the RM should not have been involved in the GEF project up for review (at any stage of the project design, implementation or backstopping). The RM and FAO GEF CU focal point supervise the MTR lead consultant and overall MTR team and jointly approve final payment for the MTR report. Budget holder (BH)/MTR manager (RM) Main responsibilities: Coordinate and oversee the overall MTR process, including consultant contracting and designating and managing the RM (in the case of the BH)

PHASE RESPONSIBILITIES Phase 1 ‒ Plan

1. Contact the FAO GEF CU six months before the project’s mid-term point (halfway between the entry-on-duty (EOD) and not-to-exceed (NTE) dates per FAO’s Field Programme Management Information System (FPMIS) to initiate the MTR.

2. Ensure that sufficient funds have been allocated for the MTR, in accordance with the monitoring and evaluation (M&E) arrangements set out in the project document and the MTR Guide and in discussions with the FAO GEF CU.

3. If the project is jointly implemented with another GEF agency, ensure that the responsibilities for managing and delivering the MTR are clear among the agencies.

4. If necessary, appoint an RM to act in place of the BH, with full responsibility for managing and delivering the MTR.

5. Compile and prepare a list of key project-related documentation for the MTR (the “project information package”). This includes the collection and collation of co-financing data (and sending the co-financing table to each of the co-financers) well ahead of the MTR mission, monitoring data, progress reports, back-to-the-office reports (BTORs) and any associated GEF tracking tools, etc., supported by the PMU. All relevant documents must be made available and uploaded to FPMIS.

2

Phase 2 ‒ Design and prepare

1. Together with the PMU and PTF members, prepare the first draft of the overall terms of reference of the MTR, including the scope/objectives, its utility and feasibility, timing and budget, an initial stakeholder analysis, the main progress made to mid-term and the main challenges/issues faced, as well as the terms of reference for the independent MTR consultants (see Annex 6 for template).

2. Revise the terms of reference of the overall MTR taking into account feedback from the FAO GEF CU and PTF members. Share the revised terms with government counterparts for their review, then finalize and advertise them.

3. Prepare the first draft of the individual terms of reference for the MTR consultants and send them to the FAO GEF CU for review and comment; finalize the terms of reference.

4. Issue a call for interest for qualified consultants. Select and recruit the MTR team, in consultation with the FAO GEF CU and prepare individual contracts.

5. Organize a briefing for the MTR team with the FAO GEF CU to ensure a common understanding of the MTR terms of reference, relevant GEF and FAO policies and requirements.

6. Provide the MTR team with the key project documents needed for the MTR (project information package – see Box 1 in the main Guide for a detailed list).

7. Ensure that the PMU prepares the GEF tracking tools (for GEF-5 projects, and for GEF-6 and GEF-7 BD projects with protected-area elements) in advance of the MTR and/or data on the GEF-7 core indicators compiled (for GEF-6 and GEF-7 projects).

8. Liaise with the PMU to identify stakeholders to be interviewed by the MTR team and send formal requests for interviews with government authorities and other partners for the MTR mission, as necessary.

9. Be available for a (Skype) interview/briefing with the MTR team before the MTR mission as part of the inception process, if required.

10. Review the MTR inception report, including the questions matrix and the list of interviews, before the mission.

11. Share the MTR inception report with relevant PTF members (including the LTO and the FAO GEF CU FLO and FAO GEF CU MTR focal point) for review and comment and provide the MTR team with feedback before the mission starts.

12. Upon receipt of an acceptable MTR inception report, formally approve the MTR mission.

Phase 3 ‒ Collect, analyse and report

1. Facilitate MTR team meetings and field missions, including organizing interviews and field missions as required.

2. Be available for Skype interviews/briefings with the MTR team during the MTR mission. 3. Organize debriefing sessions/workshops with MTR team for the PMU, government

counterpart(s) and key partners (including the GEF OFP). 4. Participate in the wrap-up/debriefing meeting in which the MTR team presents its initial

findings. 5. Lead and coordinate the review process for the draft MTR report with relevant

stakeholders for: (1) the internal FAO review of the draft report, sharing drafts with the BH, LTO, PMU, FAO GEF CU FLO and FAO GEF CU MTR focal point, the PTF (allow one week), followed by (2) external review by the project counterparts (through the BH or FAO Representative) and the wider group of relevant stakeholders.

6. Coordinate, collect, collate and synthesize feedback on the draft MTR report (for complex projects) and send feedback (in the comments matrix) to the MTR team for incorporation into the revised version.

7. Contribute to reviews of further drafts of the MTR report as required to ensure the overall quality of the report.

8. Approve the final MTR report together with the FAO GEF CU focal point.

Phase 4 ‒ Disseminate and follow up

1. Ensure the executive summary of the MTR report is translated if not written in English. 2. Approve final payment to the MTR team following clearance by the FAO GEF CU

programme officer. 3. Lead the development of a follow-up Management Response with the PMU, PTF and

FAO GEF CU (within four weeks), including a meeting of the Project Steering Committee (PSC) to discuss the recommendations of the MTR. Ensure the inputs are included in the Management Response and approved by the PSC.

4. Ensure that MTR recommendations are properly reflected in the subsequent annual workplan and budget and reported on in six-monthly FAO project progress reports (PPRs) and the annual GEF project implementation review (PIR).

3

2. National Project Director The National Project Director (NPD) designated by the national lead agency assumes overall responsibility on behalf of the government in question for the successful execution of the project. The NPD serves as a coordinating focal point for the project with other government agencies. He/she ensures that all government inputs committed to the project (including government co-financing) are made available and oversees/supports the work of the project coordinator. The NPD also participates in the PSC. The National Project Director Main responsibilities: Facilitate the participation of government partners in the MTR process and support the PMU in ensuring good communication of the MTR and its results across government


1. Inform government partners, including the GEF OFP, about the MTR process, request their participation in the MTR process and confirm their availability during the proposed period of the MTR.


1. Review the draft terms of reference and provide comments to the BH/RM.


1. Ensure that all relevant government partners provide timely inputs (including on co-financing) to the review.

2. Be available for interview with the MTR team during the MTR mission. 3. Attend the wrap-up/debriefing meeting in which the MTR team presents its initial findings. 4. Collect feedback from government partners on the draft MTR report within the specified

timeframe (one week). 5. Review the draft MTR report and provide comments to the BH/RM


1. Contribute to the MTR Management Response and ensure that MTR recommendations are properly implemented.

3. Project Steering Committee A multi-stakeholder PSC is established for every FAO‒GEF project to guide and oversee the execution of the project. It provides guidance to ensure that project execution is in accordance with the project document, and reviews and approves any proposed revisions, including any changes to the project’s results matrix and implementation arrangements. The PSC also reviews and endorses all annual workplans and budgets; reviews project progress and the achievement of planned results, as presented in the six-monthly PPRs, PIRs and financial reports; advises on issues and problems arising from project implementation, submitted for consideration by the PMU or stakeholders; and facilitates cooperation between project partners.

Project Steering Committee

Main responsibilities: Facilitate the participation of government partners and other key stakeholders in the MTR process PHASE RESPONSIBILITIES Phase 1 ‒ Plan

1. Committee members should keep themselves informed of MTR dates and process. 2. Confirm the availability of partners and stakeholders during the proposed dates.


1. Committee members should keep themselves informed about the terms of reference and MTR process.


1. Be available for interview with the MTR team during the MTR mission. 2. Review the draft MTR report and provide comments to the BH/RM.


1. Contribute to the MTR Management Response and ensure that MTR recommendations are discussed, approved and properly implemented.

4

4. Lead technical officer One FAO LTO is assigned to each FAO‒GEF project and programme. He/she coordinates and ensures the flow of technical expertise, backstopping and oversight to FAO projects and programmes throughout the project cycle, in cooperation with other relevant technical officers in the FAO PTF. As FAO LTOs participate in supervision missions and provide technical oversight, their BTORs and other technical reports are important sources of information for the MTR. FAO LTOs support project BHs by advising on the technical expertise required for the MTR (and final evaluation) and commenting on MTR team composition and the content of the MTR terms of reference. The LTOs are responsible for a project’s environmental and social screening and sign off on the risk classification form. They oversee the implementation of risk mitigation measures, as reported in the PPRs and PIRs for projects with a moderate or high risk rating, in accordance with the FAO Environmental and Social Management Guidelines.1

Lead technical officer

Main responsibilities: Support the process to design the MTR and the review of MTR report

PHASE RESPONSIBILITIES

Phase 1 – Plan

1. The LTO should keep him/herself informed on the MTR dates and process 2. Confirm his/her availability to be interviewed by the MTR team

Phase 2 – Design and prepare

1. Ensure that the MTR terms of reference prepared by the BH/RM are technically sound and provide feedback to improve the terms of reference, as necessary

2. Support the PMU with the completion of the GEF Tracking Tools (for GEF-5 projects, and for GEF-6 and GEF-7 BD projects with protected-area elements) in advance of the MTR and/or data on GEF-7 core indicators compiled (for GEF-6 and GEF-7 projects)

3. Assist the BH/RM and FAO GEF CU with reviewing the MTR consultant(s) qualifications and CV, as required

4. Be available for an initial Skype interview/briefing with MTR team before the MTR mission to provide own views on the project that the LTO has been backstopping

Phase 3 – Collect, analyse and report

1. Be available for a Skype interview/briefing with MTR team during the MTR mission if needed

2. If possible, participate in the debriefing session with the MTR team at the end of the field mission to discuss preliminary findings, conclusions and possible recommendations

3. Review the draft MTR report from the technical point of view and provide feedback

Phase 4 – Disseminate and follow up

1. Assist the BH in drafting the Management Response

5. FAO GEF Coordination Unit The FAO GEF CU, located in FAO’s Climate Change and Environment Division (CBC), is the funding liaison unit (FLU) responsible for maintaining corporate relations with the GEF throughout the project cycle. It manages the FAO‒GEF project portfolio, supervises and monitors the projects, and reports to the GEF Secretariat on their delivery, including the submission of MTR reports. The FAO GEF CU provides guidance and support to the project BHs with MTRs. It also lends coordination support to the FAO Evaluation Office (OED) on terminal evaluations. The FAO GEF CU reviews and comments on the terms of reference of the MTR and participates in the selection of the external consultants to be hired by the BH. It also provides backstopping for the planning and the implementation of an MTR, undertakes quality assurance on the inception report, the draft and final MTR reports, and clears the final MTR report for BHs to make the final payment to MTR consultants. A specific FAO GEF CU MTR focal point is appointed as the point of contact for an MTR, usually a FAO GEF CU programme officer.

1 FAO (2015: 19) Diagram 1: Environmental and Social risk management in the FAO project cycle and responsibilities

5

In addition, the FAO GEF CU funding liaison officer (FLO) monitors the results and budget of the project, including supervising and reporting to GEF through the annual GEF PIRs.

FAO GEF Coordination Unit Main responsibilities: Support the MTR process and provide quality assurance of draft reports and follow-up action


1. Plan to ensure the MTR will take place in a timely manner (begin planning 4-6 months before the due date, which is halfway between the EOD and NTE dates).

2. Provide guidance to the BH/RM on the MTR process and on GEF requirements 3. Share the MTR Guide for FAO‒GEF projects and its annexes.


1. Review and comment (both the FLO and FAO GEF CU MTR focal point) on the MTR terms of reference to ensure that they align with the minimum terms-of-reference standards, as outlined in the MTR Guide and provide feedback to improve them, as required.

2. Assist the BH/RM with the identification and selection of the independent consultants for the MTR team, providing joint approval (with the BH/RM) on the final selection.

3. Provide guidance to the MTR team on the MTR process and GEF requirements, providing a copy of the MTR Guide for FAO‒GEF projects and relevant GEF policies and guidelines.

4. The FLO liaises with the BH/RM and LTO to ensure that the GEF tracking tools are revised and updated at mid-term (for GEF-5 projects and GEF-6 and GEF-7 (biodiversity projects with protected-area elements) and/or that data on GEF-7 core indicators are compiled (for GEF-6 and GEF-7 projects) and sent to the MTR team on time.

5. Together with the PMU, the FLO briefs the MTR team on the project. 6. Review and approve the MTR inception report (FAO GEF CU MTR focal point) with the

BH/RM, including revision of the MTR plan and deliverables (mission agenda, MTR matrix, etc.)

7. Be available for a Skype briefing (FAO GEF CU MTR focal point) with the MTR team before the MTR mission and for interview by the MTR team (the FLO, as part of the PTF needs to be interviewed by the MTR team).


1. Provide guidance to the MTR team, BH/RM and PMU on issues relating to GEF MTR and policy requirements.

2. The FLO should be available for a Skype interview/briefing with the MTR team during the MTR mission.

3. Take part in a Skype debriefing session with the MTR team at the end of the field mission.

4. Provide review and feedback at the end of the MTR field mission on initial findings and conclusions.

5. Review and comment on the first draft MTR report for quality assurance and provide feedback to the MTR team (FLO and CGU MTR focal point).


1. Score and comment on the final version of the MTR report 2. Post the completed MTR report, the Management Response and tracking tools to FPMIS

and translate the Management Response into the official national language(s) if not English.

3. The FLO should ensure that the relevant section of the third PIR is revised in line with the findings and recommendations of the MTR report ‒ for example, if changes to the project’s theory of change and logframe/results matrix are approved at mid-term and there are changes to the risks and their mitigation (this is the specific responsibility of the FAO GEF CU FLO).

4. Submit the MTR report to the GEF Secretariat (through the GEF portal) together with the Management Response (all in English) and the revised tracking tools for GEF-5 projects and contributions to GEF-7 core indicators for GEF-6 and GEF-7 projects.

6

6. Project Management Unit The PMU is responsible for the day-to-day operations, monitoring and reporting of the project in line with the project’s annual workplan, budget, and M&E plan. On larger projects, the team typically includes a chief technical advisor and/or project manager/coordinator, a national coordinator and an M&E specialist, as well as other specialists and experts (the number and roles will vary from project to project). For national projects, the PMU is typically led by a national project manager (sometimes termed project coordinator) recruited specifically for the project. Some of the PMU positions may be funded or provided by the government in question as part of its co-financing contribution. The PMU monitors project progress by regularly collecting data on output and outcome indicators. The PMU’s data collection, therefore, is among the most important sources of information for the MTR. The PMU members must make themselves available and responsive to the MTR team throughout the entire MTR process. The PMU and the BH then implement the measures and actions recommended by the MTR, as agreed in the Management Response, track and document progress and report on the measures implemented through the annual GEF PIR and six-monthly FAO PPR.

Project Management Unit

Main responsibilities: Provide the MTR team with project information and assist with MTR logistics PHASE RESPONSIBILITIES Phase 1 ‒ Plan

1. Together with the BH/RM, prepare the first draft terms of reference for the MTR, including its scope/objectives, usefulness and feasibility, timing and budget, the main progress made up to mid-term and the main challenges/issues faced, as well as the terms of reference for the independent MTR consultants.

2. Review and update the project GEF tracking tools at mid-term (for GEF-5 projects and for GEF-6 and GEF-7 biodiversity projects with protected-area elements) and/or compile data on GEF-7 core indicators (for GEF-6 and GEF-7 projects).

3. Assist the BH/RM in compiling the key project-related documents (project information package).


1. Prepare the logistics for MTR interviews and missions, including internal travel and arrangements for meetings and interviews (make sure itineraries are agreed and stakeholders are given sufficient notice).

2. Be available for a Skype interview with the MTR team before the MTR mission.


1. Assist with the logistics of the MTR mission. 2. Support the organization of MTR interviews and meetings. 3. Be available for face-to-face and/or Skype interviews with the MTR team during the MTR

mission; engage with the MTR team to review the GEF tracking tools (for GEF-5 projects) or GEF-7 indicators (for GEF-6 and GEF-7 projects) and finalize them, with assistance from LTO and FLO if necessary.

4. Participate in the wrap-up/debriefing meeting in which the MTR team presents its initial findings and recommendations.

5. Review the draft MTR report, check for errors and information gaps and provide feedback to the BH/RM for incorporation into the comments matrix.


1. Support the BH/RM in drafting the Management Response, in consultation with the PSC members and the PTF.

2. Integrate MTR recommendations into subsequent annual workplans. 3. Implement the MTR recommendations and actions, as agreed in the Management

Response. 4. Provide inputs to the BH for the PIR or PPR on the implementation of MTR

recommendations as agreed in the Management Response.

7

7. MTR team The MTR team comprises (usually two) independent consultants: (1) a team leader, who is typically an international or regional consultant, but can also be a national consultant with experience of GEF project reviews and evaluations and with a technical/evaluation background appropriate to the project, and (2) a national consultant, with a relevant technical background and good understanding of and insights into the context of the project. The two consultants will not have been involved in, or benefited from, any aspect of the GEF project. When reporting on progress or requesting information, advice or guidance, the MTR team should address correspondence to both the BH/RM and the FAO GEF CU MTR focal point.

MTR team

Main responsibilities: Uphold contractual obligations outlined in the MTR terms of reference

PHASE RESPONSIBILITIES Phase 2 ‒ Design and prepare

1. Review the terms of reference with the BH/RM and FAO GEF CU to ensure a feasible and effective MTR.

2. Review the project information package, including GEF tracking tools from the CEO endorsement and mid-term stages (for GEF-5 projects, and for GEF-6 and GEF-7 biodiversity projects with protected-area elements) and/or data on GEF-7 core indicators (for GEF-6 and GEF-7 projects).

3. Liaise with the BH/RM and PMU to ensure appropriate timing of the review mission. 4. Hold initial pre-mission discussions/Skype interviews with the BH/RM, LTO, FAO GEF

CU (FLO and MTR focal point) and PMU, as required. 5. Liaise with the BH/RM and PMU (project manager/coordinator and rest of the project

team) to define the MTR methodology. 6. Prepare MTR inception report and send it to BH/RM and FAO GEF CU and incorporate

feedback (the inception report should ideally be approved two weeks before planned MTR mission(s) to allow time for MTR interviews and other logistics to be arranged).


1. Undertake the MTR mission(s). 2. Organize a mission wrap-up meeting with the BH/RM, PMU and project counterpart

to present initial findings and conclusions and discuss preliminary possible recommendations.

3. Lead a Skype debriefing session with the FAO GEF CU and other staff in headquarters. 4. Complete and submit the first draft of the MTR report to the BH/RM and FAO GEF CU

within three weeks of completing the field mission(s). 5. Review and address the two rounds of stakeholder feedback (internal review involving

FAO staff and external reviews with project counterparts and other partners) on the draft MTR report, incorporate corrections and relevant comments, and provide an audit trail by completing a comments matrix, attached as an annex to the MTR report.

6. Provide a revised, final MTR report to BH/RM and FAO GEF CU within one week of receiving stakeholder feedback.

7. Phase 4 ‒ Disseminate and follow up

1. Participate in a stakeholder workshop to discuss findings, conclusions and recommendations, if appropriate and available (participation through Skype is acceptable alternative if unable to attend in person).

8. Other FAO staff involved in the MTR a. FAO Project Task Force

The PTF is a management and consultative body established for each FAO project. The PTF consists of designated FAO staff with the appropriate authority and skills mix to ensure effective technical, operational and administrative project management throughout the project cycle. The PTF is formally established during project formulation. PTF members of FAO‒GEF projects include at a minimum: the project BH as PTF Chair, supported by an operational team based in the relevant Country, Sub-regional or Regional Office for single-

8

country, sub-regional or regional projects or at headquarters for interregional or global projects; an LTO at sub-regional, regional or headquarters level; a headquarters technical officer (not applicable to projects with LTOs at FAO headquarters in Rome); and the FAO GEF CU as the FLU. Other potential members of the PTF include a Chief Technical Adviser and other technical and operational staff where relevant. b. FAO Office of Evaluation

Since January 2019, when the FAO Office of Evaluation (OED) decided to decentralize the mid-term evaluation of projects, all MTRs of FAO–GEF projects are managed by the project’s BH, overseen by the FAO GEF Coordination Unit (FAO GEF CU). However, if needed, the FAO GEF CU can request that OED oversee a project MTR. In exceptional cases (to be agreed on a case-by-case basis), where a larger MTE is deemed necessary (such as for complex projects or projects with challenging issues), MTRs may still be managed by OED. 9. Other key responsible parties a. The GEF Secretariat is responsible for monitoring the overall GEF portfolio, receiving and keeping records of the MTR and MTE reports submitted by FAO and other GEF agencies, and providing comments where applicable. In future, the GEF will release part of the Implementing Agency’s fee (30 percent) upon delivery of the MTR. b. The GEF Independent Evaluation Office (IEO), which is independent of the GEF Secretariat, sets the minimum standards for GEF project evaluation,2 ensures the independent evaluation function within the GEF and is responsible for undertaking independent evaluations that involve a set of projects from more than one GEF agency, typically on a strategic level, on focal areas or on cross-cutting themes. c. The GEF Operational Focal Point (OFP)3 is a government official, nominated by a country to act as the principal point of contact for GEF activities in that country. According to the GEF Evaluation Policy 2019 (GEF IEO, 2019), Minimum Requirement 4 (Engagement of Operational Focal Points), “the OPF will be informed of midterm reviews and terminal evaluations and will, where applicable and feasible, be briefed and debriefed at the start and at the end of evaluation missions. They will receive a draft report for comment, will be invited to contribute to the Management Response (where applicable), and will receive the final evaluation report within 12 months of project or programme completion.” It goes on to say that “the GEF OFPs play a key role in facilitating access to staff members of government institutions involved in GEF projects during evaluations. They may promote the use of, follow-up to, and action on evaluation recommendations related to GEF matters and directed at the regional, national, and project levels. They also play an important role in keeping national stakeholders (including the civil society organizations involved in GEF activities) fully consulted with, informed on, and involved in the plans, conduct, and results of country-related GEF evaluation activities.” Consequently, the GEF OFP should be kept informed of, and involved in, the MTR process, in line with GEF guidance.

2 Two different GEF policies were approved in 2019: one on monitoring, prepared by the GEF Secretariat (GEF Secretariat, 2019), and one on evaluation policy by the GEF’s Independent Evaluation Office (IEO) (GEF IEO, 2019). 3 The list of country GEF focal points is available on the GEF website: https://www.thegef.org/focal_points_list.

https://www.thegef.org/focal_points_list

1

Annex 3 – FAO’s roles and responsibilities as a GEF agency 1. Under the Direct Execution (DEX) Modality

Under the Direct Execution (DEX) Modality, FAO implements and executes projects and provides most of the services to national institutions under the guidance of the Project Steering Committee (PSC), chaired by the lead ministry or main national executing partner. FAO can delegate part of the services to be provided to non-profit entities (such as public authorities and non-government organizations) by signing letters of agreement or to commercial entities through service contracts. Under DEX, FAO is accountable from both a technical and fiduciary standpoint for the achievement of all expected project results. The separation of the implementation and execution functions, an important aspect of the GEF Minimum Fiduciary Standards, is ensured by maintaining the following allocation of roles and responsibilities. The day-to-day management of an FAO‒GEF project is the responsibility of the appointed FAO budget holder (BH) and the project management unit (PMU) (execution function). Technical oversight, project supervision and evaluation are the responsibilities of the FAO technical officers assigned to the project, the FAO GEF Coordination Unit (FAO GEF CU) as funding liaison unit and the FAO Office of Evaluation (OED) in an implementation function. Under the Direct Execution modality, the BH, in consultation with the PSC, PMU, FAO Project Task Force (PTF) members and the FAO GEF CU, plans an appropriate budget and timeframe for a mid-term review (MTR) (and works with the OED on the subsequent terminal evaluation), usually during the project formulation phase. 2. Under the Operational Partners Implementation Modality (OPIM)

Under the Operational Partners Implementation Modality (OPIM),1 FAO is responsible for the implementation of the GEF project, which “entails oversight of project execution to ensure that the project is being carried out in accordance with agreed standards and requirements”.2 FAO, as a GEF implementing agency, has overall accountability and responsibility to the GEF Council for the delivery of results. FAO provides monitoring and oversight of project implementation and technical support to ensure that the project is being carried out in accordance with agreed standards and requirements. FAO is also responsible for ensuring that project monitoring, reporting and evaluation milestones are met, in line with GEF’s monitoring and evaluation policies, including an annual GEF project implementation review (PIR), an MTR and a terminal evaluation, as well as six-monthly FAO project progress reports (PPRs). The executing entity ‒ the operational partner (OP) in FAO terminology ‒ is required to implement the agreed results of the project in line with FAO and other fiduciary requirements and performance standards. The OP is responsible for the day-to-day management of the project results entrusted to it, in full compliance with all terms and conditions of the Operational Partners Agreement (OPA). The OP is responsible and accountable to FAO for the timely and quality implementation of the project results, the operational oversight of implementation activities, timely reporting and the effective use of GEF resources for their intended purposes. The OP bears full fiduciary and programmatic risk and is administratively and technically responsible to FAO for the implementation of the project results, as well as monitoring and financial management, in accordance with the OPA. Such responsibility extends to all funds disbursed by the OP to any entity under contract with it. Under the OPIM, during the project formulation phase, the project formulator (the person responsible for designing the project, usually the FAO Representative of the country in question) ensures that costs for an

1 FAO Manual Section 701 on Operational Partners Implementation Modality (OPIM) (available on the FAO Intranet at http://intranet.fao.org/faohandbook/fao_manual/chapter_vii_operational_modalities/701_operational_partners_implementation_modality/701_index/; non-FAO staff and consultants can request this document from the MTR Manager). 2 GEF (2007), Standards, Section A.2 (g)

http://intranet.fao.org/faohandbook/fao_manual/chapter_vii_operational_modalities/701_operational_partners_implementation_modality/701_index/

http://intranet.fao.org/faohandbook/fao_manual/chapter_vii_operational_modalities/701_operational_partners_implementation_modality/701_index/

2

MTR and a terminal evaluation are included in the FAO-administered portion of the budget, so that FAO can select and recruit external consultants and manage the independent review/evaluation processes in a timely manner. The project formulator must make sure that the arrangements for the MTR and the terminal evaluation are explicitly set out in the relevant sections of the project document. 3. Under joint implementation with other GEF agencies

Where GEF-financed projects are jointly implemented by FAO and other GEF agencies, arrangements for reporting to the GEF and for managing MTRs (and terminal evaluations) will be discussed and agreed during the project design phase by the agencies involved and stipulated in the project document. The participating GEF agencies may choose to delegate responsibility for managing the MTR (and terminal evaluation) to one GEF agency, termed the “lead agency”, with the active participation and/or support of the other agencies. The MTR will follow the protocols of the lead agency, while ensuring that MTR plans and results are communicated in a timely manner with all of the agencies involved and the GEF Secretariat. The PSC, PMU, BH, FAO lead technical officer and FAO GEF CU are advised to plan ahead and clarify in advance whether FAO or another GEF agency will manage the MTR of jointly implemented projects. 4. Under the programmatic or integrated approaches

Under the pilot “programmatic approach” or “integrated approach”, various projects are grouped into a “programme”, often including a smaller-scale “parent” project as a coordination mechanism and several “child” projects containing the main project activities under the same programmatic umbrella in a number of regions and/or countries. “Child” projects typically require independent MTR reports (either as individual project reviews or case studies as part of a synthesis review or programme evaluation), in addition to a summary analysis of the programme as a whole. It is highly advisable to ensure coordination and harmonization of “child” and “parent” projects to ensure comparability of mid-term assessment results across the group of projects and adherence to the requirements of the participating GEF agencies.

i

Annex 4 – Annotated terms-of-reference template for FAO‒GEF project and programme MTRs Please use this document as a template for developing the terms of reference for the MTR of your project. It will need to be adapted to the specific conditions of your project.

Terms of reference for the mid-term review of

[project name]


[Month Year]

ii

Contents

Acronyms and abbreviations............................................................................................... iii

Introduction ........................................................................................................................... 1

1 Project/programme background and context ............................................................ 2 1.1 Description of the project, project objectives and components ............................................... 2 1.2 Project stakeholders and their role ......................................................................................... 2 1.3 Theory of change ................................................................................................................... 3 1.4 Implementation progress and main challenges to date ........................................................... 4

2 MTR purpose and scope ................................................................................................ 5

3 MTR objectives and key questions ............................................................................... 6 3.1 MTR objectives ...................................................................................................................... 6 3.2 MTR questions ....................................................................................................................... 7

4 Methodology ................................................................................................................ 10

5 Roles and responsibilities ............................................................................................ 11

6 MTR team composition and profile ........................................................................... 13

7 MTR products (deliverables) ....................................................................................... 14

8 MTR timeframe ............................................................................................................ 15

Annexes ................................................................................................................................ 16

iii

Acronyms and abbreviations

List your acronyms and abbreviations here. Check the whole document and list all the acronyms and

abbreviations you have used. Do not include acronyms and abbreviations that are not in the document. When

an abbreviation is used for the first time in the text, it should be explained in full; it should be included in

the list of acronyms when it is used repeatedly in the report. Typical standard acronyms and abbreviations

include the following:

BH Budget holder

CBIT Capacity-Building Initiative for Transparency

CO Country Office

EOI Expression of interest

FAO Food and Agriculture Organization of the United Nations

FLO Funding liaison officer

FPMIS Field Project Management Information System

LDCF Least Developed Countries Fund

LTO Lead technical officer

LTU Lead technical unit

MTE Mid-term evaluation

MTR Mid-term review

NPD National Project Director

OED FAO Office of Evaluation

PMU Project management unit

PSC Project Steering Committee

PTF Project Task Force

RM Mid-term review manager

RO Regional Office

SCCF Special Climate Change Fund

SO FAO Strategic Objective

SRO Sub-regional Office

1

Introduction

This document provides the template for the terms of reference for mid-term reviews (MTRs) of

FAO projects and programmes funded through the GEF, via the Least Developed Countries Fund

(LDCF), the Special Climate Change Fund (SCCF) and the Capacity-Building Initiative for

Transparency (CBIT). It provides specific guidance on how to complete each section of the terms

of reference.

The template provides examples and questions that can be used to develop and adapt the MTR

terms of reference. All guidance information should be removed from the final terms of reference.

This document will mainly be used by FAO staff, in particular, the budget holder (BH) or his/her

designated MTR manager (RM) (see Annex 2 on roles and responsibilities), who is responsible for

initiating and managing the MTR, and staff from the FAO Project Task Force (PTF) and FAO GEF

Coordination Unit (FAO GEF CU). For more information on the process of conducting an MTR,

please see the main MTR Guide.

This template is a living document that needs to be adapted to the context of a given MTR, as well

as to improve the efficiency and effectiveness of the MTR process.

Please note the following formatting requirements:

• Paragraphs should be numbered

• Font: Segoe UI

• Size: 10.5

2

1 Project/programme background and context

1. This section should provide the MTR team with an overall and understanding of the

project/programme and the developmental context in which it is implemented.

2. The brief background and context section makes clear what is being assessed and identifies the

critical social, environmental, economic, political, geographic and demographic factors within which

the project or programme operates and which will have a direct bearing on the MTR. This description

should be focused and concise (2‒3 pages maximum) highlighting only those issues most pertinent

to the MTR. It should not be a simple “cut and paste” of background information from the project

document, but should summarize the project.

1.1 Description of the project, project objectives and components

3. This section should provide the key background and context descriptors and should include the

following information:

• The name/title of the project/programme being reviewed and basic GEF and FAO

information, including GEF Project ID, project name, GEF agency, GEF financing and co-

financing (including contributions from FAO, other donors, other funding, promised and

material co-financing, linked to the co-financing table on Annex 2), name of the project’s

executing entity, whether the project is linked to a GEF programme, and the project

duration (this information may be presented as a table);

• Description of the context of and rationale for the project:

- Area where the project is taking place (region, country, landscape and target area(s),

if possible), with geo-referenced locations and a map;

- Rationale, problems, challenges and barriers the project seeks to address;

• Description of the project or programme:

- When and how the project/program was initiated;

- Brief description of the project’s key objectives and different project components;

- Target group/beneficiaries the project is aiming to reach (for the project as a whole

and by component);

- Project set-up, including management arrangements, human resources and budget;

- Project implementation status and key dates;

• How the subject fits into the national priorities, FAO Country Programming Framework,

FAO regional priorities and initiatives, FAO Strategic Objectives, GEF priorities and

Sustainable Development Goals as appropriate (in summary only). It should also include

a brief description of how the MTR fits within the context of other ongoing and previous

assessments and evaluations of the same project and other relevant initiatives.

1.2 Project stakeholders and their role

4. The terms of reference should list the key partners and stakeholders involved in the project, including

the executing agencies and partners, local groups and beneficiaries. It should outline the role each

plays in the project, as well as the RM and PMU’s views on why they should be included in the MTR.

The initial stakeholder analysis should be captured in Table A4.1.

3

Table A4.1. Stakeholder analysis matrix template

Key stakeholders

(disaggregated as

appropriate)1

What is their role

in the project?

What is the reason for their

inclusion in or exclusion from

the MTR?

Priority for MTR (1-3)2

How and when should

they be involved in the

MTR?

1. Active stakeholders with direct responsibility for the project, e.g. FAO, executing partners

Name Stakeholder group 1


Etc.

2. Active stakeholders with authority to make decisions on the project, e.g. members of the PSC



Etc.

3. Secondary stakeholders (only indirectly or temporarily affected)



Etc.

4. Stakeholders at grassroots level who benefit directly or indirectly from the intervention (gender disaggregated where

possible)



Etc.

5. Stakeholders at grassroots level who do not benefit from the intervention (gender disaggregated where possible)



Etc.

6. Other interest groups that are not participating directly in the intervention, e.g. development agencies working in the area,

civil-society organizations



Etc.

5. This initial list of key stakeholders is important to help identify potential groups and individuals to be

consulted and interviewed as part of the MTR process. The initial list is likely to be modified by MTR

team members once they become engaged in the MTR and will be updated as part of the MTR

inception report.

1.3 Theory of change

6. A theory of change helps to identify relevant objectives and questions for the MTR. Some FAO‒GEF

projects have an explicit theory of change developed during the project formulation phase, which is

included in the project document. If available, the project’s theory of change should be included

1 Include the names of relevant individuals, if known, and be as specific as possible

2 1 = essential; 2 = desirable; 3 = if time and resources allow

4

here. If it is only implicit, the MTR team should reconstruct a preliminary theory of change after the

fact as part of the inception report, based on the project’s logframe and review of other project

documents (see Annex 10 for more background on the theory of change).

1.4 Implementation progress and main challenges to date

7. This sub-section should briefly highlight the key results achieved and main challenges faced since

project implementation began. Avoid listing activities and choose instead to focus on what has been

delivered (following the theory of change, if available) and what the key barriers to delivery have

been. The purpose of this section is to indicate to the MTR team the current level of progress on

project implementation and highlight project-specific issues that should be considered by the MTR.

5

2 MTR purpose and scope

8. The “purpose” of the MTR is the reason for conducting the MTR, including the choice of timing. It

should answer the question: “Why are we doing this MTR?”

9. The main purpose of the MTR is to:

• provide accountability – to respond to the information needs and interests of policymakers and

other actors with decision-making power, for example, FAO management and the FAO GEF CU;

• improve the project/programme – project/programme improvement and organizational

development provide valuable information to managers and others responsible for regular

project/programme operations (for example, the PMU, PTF, FAO GEF CU and PSC); and

• contribute to knowledge – in-depth understanding and contextualization of the

project/programme and its practices, of particular benefit to the FAO GEF CU, FAO staff and future

developers and implementers.

10. It may be possible to address several purposes and uses in a single MTR design. However, it is usually

necessary to decide during the MTR design phase where most resources will be focused.

11. This section should identify the primary intended users of the project MTR, the stakeholders that will

use its findings to effect change. This can include the BH and designated RM, the Project

Management Unit (PMU), the national project counterpart, the PTF (including the funding liaison

officer (FLO) and the lead technical officer (LTO) and other FAO technical staff at headquarters), PSC

members, the GEF and other stakeholders. The BH/RM, the PMU and the PTF should seek to identify

the purpose of the MTR in a consultative manner, in collaboration with all primary intended users,

and pinpoint how they intend to use the MTR results. However, it should be noted that use of the

MTR and related products may vary from stakeholder to stakeholder, so the BH/RM and relevant PTF

members will need to negotiate and reach overall agreement between primary intended users on

the main purpose(s) of the MTR and its intended use.

12. The scope of the MTR states precisely what is evaluated (the objective of the MTR) and defines the

parameters of the MTR: the time period; the segments of the target population to be included; the

geographic area the MTR should focus on; stakeholders to be involved; and whether all components

of the project/programme are to be addressed. To the extent possible, an explanation should be

provided to justify the choices made, for example, a focus on selected project components or specific

geographic areas.

13. More detail on the purpose, use and scope of an MTR can be found in the MTR Guide.

6

3 MTR objectives and key questions

3.1 MTR objectives

14. The MTR objectives describe precisely what it should achieve and what it should examine in relation

to the GEF evaluation criteria. It will address and rate the following:

Relevance – the extent to which the intervention’s design and intended results are consistent with

local, national, sub-regional and regional environmental and development priorities and policies and

to GEF and FAO strategic priorities and objectives; its complementarity with existing interventions and

relevance to project stakeholders and beneficiaries; its suitability to the context of the intervention over

time.

Effectiveness – the degree to which the intervention has achieved or expects to achieve results (project

outputs, outcomes, objectives and impacts, including Global Environmental Benefits) (GEF, 2019c)

taking into account key factors influencing the results, including an assessment of whether sufficient

capacity has been built to ensure the delivery of results by the end of project and beyond and the

likelihood of mid- and longer-term impacts.

Efficiency – the cost-effectiveness of the project and timeliness of activities; the extent to which the

intervention has achieved value for resources by converting inputs (funds, personnel, expertise,

equipment, etc.) into results in the timeliest and least costly way compared with alternatives.

Sustainability – the (likely) continuation of positive effects from the intervention after it has ended

and the potential for scale-up and/or replication; any financial, socio-political, institutional and

governance, or environmental risks to sustainability of project results and benefits; any evidence of

replication or catalysis of project results.

Factors affecting performance – the main factors to be considered are:

• project design and readiness for implementation (e.g. sufficient partner capacity to begin

operations, changes in context between formulation and operational start);

• project execution, including project management (execution modality as well as the involvement

of counterparts and different stakeholders);

• project implementation, including supervision by FAO (BH, LTO and FLO), backstopping, and

general PTF input;

• financial management and mobilization of expected co-financing;

• project partnerships and stakeholder involvement (including the degree of ownership of project

results by stakeholders), political support from government, institutional support from operating

partners (such as regional branches of agricultural extension services or forestry authorities);

• communication, public awareness and knowledge management; and

• application of an M&E system, including M&E design, implementation and budget.

Cross-cutting dimensions – considerations such as gender, indigenous-peoples and minority-group

concerns and human rights; the environmental and social safeguards applied to a project require,

among other things, a review of the Environmental and Social Safeguards (ESS) risk classification and

risk-mitigation provisions identified at the project’s formulation stage.3

3 FAO applies an online screening system during the project design phase. This is mandatory, even if the project was approved before FAO adopted the GEF Policy on Agency Minimum Standards on Environmental and Social Safeguards (GEF, 2011) in February 2015, as FAO had already applied the Environmental Impact Assessment Guidelines in 2011 (FAO, 2012a) to screen and rate the risks of every FAO project. Consequent ly, the MTR team should review and confirm the ESS assessments and risk status at mid-term and any changes suggested, if needed. The most recent GEF guidance can be found in GEF (2019b). A GEF project should not cause any harm to the environment or to any stakeholder and, where applicable, will take measures to prevent and/or mitigate any adverse effects.

7

3.2 MTR questions

15. MTR questions should be included in this section, corresponding to one or more GEF evaluation

criteria (the MTR gathers evidence by posing questions to assess its degree of compliance with the

GEF criteria). MTR questions should be based on project objectives and draw on the project’s theory

of change. They should be sufficiently broad, but still help focus the MTR and address project-specific

issues, as agreed by the BH/RM and principal stakeholders. They will be refined later in consultation

with the MTR team and documented in the inception report.

16. Depending on the size and complexity of the project (and, thus, the MTR), each question can be

divided into sub-questions, creating an MTR matrix.4 Example questions for each of the criteria listed

in paragraph 13 can be found in Box A4.1. Please note that the questions need to be phrased in the

context of the project's theory of change.

Box A4.1. Examples of MTR questions (to be adapted for each project)

1. Relevance

(rating required)

Are the project outcomes congruent with country priorities, GEF focal areas/operational

programme strategies, the FAO Country Programming Framework and the needs and priorities

of targeted beneficiaries (local communities, men and women, and indigenous peoples, if

relevant)?

Has there been any change in the relevance of the project since its formulation, such as the

adoption of new national policies, plans or programmes that affect the relevance of the project's

objectives and goals? If so, are there any changes that need to be made to the project to make

it more relevant?

2. Effectiveness of

project results

(rating required)

(Delivery of results) To what extent has the project delivered on its outputs, outcomes and

objectives? What broader results (if any) has the project had at regional and global level to date?

Were there any unintended consequences? Is there any evidence of environmental stress

reduction (for example, in direct threats to biodiversity) or environmental status change (such as

an improvement in the populations of target species), reflecting global environmental benefits

or any change in policy, legal or regulatory frameworks? To what extent can the achievement of

results be attributed to the GEF-funded component?

(Likelihood of impact) Are there any barriers or other risks that may prevent future progress

towards and the achievement of the project’s longer-term objectives? What can be done to

increase the likelihood of positive impacts from the project? To what extent can the progress

towards long-term impacts be attributed to the project?

(For programme assessments) (Coherence) How coherent is the programme with its child

projects’ theories of change, indicators and expected/achieved results? What is the added value

of bringing the different interventions together under one programme (compared with the same

level of investment made through comparable alternatives)?

4 See Annex 9 of the MTR Guide for an MTR matrix template.

8

3. Efficiency

(rating required)

To what extent has the project been implemented efficiently and cost effectively? To what

extent has project management been able to adapt to any changing conditions to improve

the efficiency of project implementation?

To what extent has the project built on existing agreements, initiatives, data sources,

synergies and complementarities with other projects, partnerships, etc. and avoided

duplication of similar activities by other groups and initiatives?

If the project is executed under the OPIM modality, add relevant OPIM questions, for

example, whether the execution agreement was followed efficiently. An additional set of

questions is suggested for projects with an OPIM component in Annex 12 of the MTR Guide.

4. Sustainability

(rating required)

(Sustainability) What is the likelihood that the project results will be useful or persist after

the end of the project? What are the key risks that may affect the sustainability of the project

results and its benefits (consider financial, socioeconomic, institutional and governance, and

environmental aspects)?

(Replication and catalysis) What project results, lessons or experiences have been replicated

(in different geographic areas) or scaled up (in the same geographic area, but on a much

larger scale and funded by other sources)? What results, lessons or experiences are likely to

be replicated or scaled up in the near future?

If the project is executed under the OPIM modality, add relevant OPIM questions (see list in

the OPIM toolkit).

5. Factors affecting

progress

(ratings required)

(Project design) Is the project design suited to delivering the expected outcomes? Is the

project’s causal logic (per its theory of change) coherent and clear? To what extent are the

project’s objectives and components clear, practical and feasible within the timeframe

allowed? To what extent was gender integrated into the project's objectives and results

framework? Were other actors – civil society, indigenous peoples or private sector – involved

in project design or implementation and what was the effect on project results?

(Project execution and management) To what extent did the executing agency effectively

discharge its role and responsibilities in managing and administering the project? What have

been the main challenges in terms of project management and administration? How well

have risks been identified and managed? What changes are needed to improve delivery in

the latter half of the project?

If the project is executed under the OPIM modality, add relevant OPIM questions (see list in

the OPIM toolkit).

(Financial management and co-financing) What have been the financial-management

challenges of the project? To what extent has pledged co-financing been delivered? Has any

additional leveraged co-financing been provided since implementation? How has any

shortfall in co-financing or unexpected additional funding affected project results?

(Project oversight, implementation role) To what extent has FAO delivered oversight and

supervision and backstopping (technical, administrative and operational) during project

identification, formulation, approval, start-up and execution?

(Partnerships and stakeholder engagement) To what extent have stakeholders, such as

government agencies, civil society, indigenous populations, disadvantaged and vulnerable

groups, people with disabilities and the private sector, been involved in project formulation

and implementation? What has been the effect of their involvement or non-involvement on

project results? How do the various stakeholder groups see their own engagement with the

project? What are the mechanisms of their involvement and how could these be improved?

What are the strengths and challenges of the project’s partnerships? Has the stakeholder

engagement plan been adhered to and documented? Have all stakeholders been made

aware of the ESS plan and the grievance complaint mechanism?

9

(Communication and knowledge management) How effective has the project been in

communicating and promoting its key messages and results to partners, stakeholders and a

general audience? How can this be improved? How is the project assessing, documenting

and sharing its results and lessons learned and experiences? To what extent are

communication products and activities likely to support the sustainability and scaling up of

project results?

(M&E design) Is the project’s M&E system practical and sufficient? How has stakeholder

engagement and gender assessment been integrated into the M&E system? How could this

be improved?

(M&E implementation) Does the M&E system operate per the M&E plan? Has information

been gathered in a systematic manner, using appropriate methodologies? To what extent

has information generated by the M&E system during project implementation been used to

adapt and improve project planning and execution, achieve outcomes and ensure

sustainability? Are there gender-disaggregated targets and indicators? How can the M&E

system be improved?

6. Cross-cutting

priorities

(Gender and minority groups, including indigenous peoples, disadvantaged, vulnerable and

people with disabilities) To what extent were gender considerations taken into account in

designing and implementing the project? Has the project been designed and implemented

in a manner that ensures gender-equitable participation and benefits? Was a gender analysis

done?

(ESS) To what extent were environmental and social concerns taken into consideration in the

design and implementation of the project? Has the project been implemented in a manner

that ensures the ESS Mitigation Plan (if one exists) has been adhered to?

17. It should be noted that GEF is placing increased emphasis on gender concerns and how its

programmes and projects contribute to gender equality and women’s empowerment (GEF, 2017a;

2017b; 2018a; 2018b). Consequently, the MTR should, as much as possible, collect and report sex-

disaggregated and gender-sensitive indicators and results (further questions for assessing gender

concerns are suggested in Annex 12 of the MTR Guide). GEF is also paying more attention to

stakeholder engagement and development, the use of knowledge products and the identification of

good practices. All of these areas require specific reporting when the MTR report is uploaded to the

GEF Portal webpage.

18. A programme assessment should include specific questions to examine the programme’s coherence

with “child project” theories of change, indicators and expected/achieved results. It should also

measure and demonstrate the added value of the programmatic approach over the same level of

investment made through comparable alternatives (GEF IEO, 2019).

10

4 Methodology

19. The MTR should adhere to the UNEG Norms & Standards (UNEG, 2016) and align with the FAO–GEF

MTR Guide and annexes detailing methodological guidelines and practices. The MTR will adopt a

consultative and transparent approach, keeping internal and external stakeholders informed

throughout the MTR process. The evidence and information gathered will be triangulated to

underpin its validity and analysis and to support its conclusions and recommendations.

20. The terms of reference should suggest the overall approach to and method of conducting the MTR,

including how to address the GEF evaluation criteria, the main general and project-specific questions,

and the data sources and choice of tools that are likely to yield the most reliable and valid answers

within resource limits.

21. The methodology described in the terms of reference, as well as the theory of change, should be

based on an initial assessment conducted by the BH/RM. For complex projects and programmes, and

in all cases where an inception mission is foreseen (see below), the theory of change and the

methodology may be further developed by the BH/RM and MTR team and presented in an inception

report.

22. Final decisions about the specific design and methodology for the MTR should emerge from

consultations between the project team, the MTR consultants and key stakeholders on what is

appropriate and feasible in order to meet the MTR’s purpose and objectives and answer the MTR’s

questions.

11

5 Roles and responsibilities

23. This section briefly describes the different roles that key stakeholders play in the design and

implementation of the MTR.

24. The BH is accountable for the MTR process and report and is responsible for the initiation,

management and finalization of the MTR process. Depending on availability and commitments, the

BH may designate another individual, the RM, to act on their behalf.

25. With the assistance of the project’s LTO and the FAO GEF CU, FLO and MTR focal point, and

guidance from this document and the main MTR Guide, the BH/RM is responsible for the drafting

and finalizing the terms of reference and providing input to the background and context section. The

terms of reference should be based on a document review, discussions with the PTF and, if possible,

a face-to-face or Skype meeting with the LTO to get a good understanding of the project. The BH/RM

is also responsible for identifying and recruiting the MTR team members, in consultation with the

FAO GEF CU and the LTO. In collaboration with the FAO GEF CU, the BH/RM also briefs the MTR team

on the MTR methodology and process and leads the organization of MTR missions. The BH/RM and

the FAO GEF CU’s MTR focal point review the draft and final MTR reports to assure their quality in

terms of presentation, compliance with the terms of reference, timely delivery, quality, clarity and

soundness of evidence and analysis supporting the conclusions and recommendations. The BH is

also responsible for leading and coordinating the preparation of the FAO Management Response

and the associated follow-up report, supported by the LTO and other members of the PTF. Further

details on the Management Response can be found in the MTR Guide.

26. The FAO GEF CU will appoint a focal point to provide technical backstopping throughout the MTR

process, including guidance and punctual support to the BH/RM and MTR team on technical issues

related to the GEF and the MTR. This includes support in identifying potential MTR team members,5

reviewing candidate qualifications and participating in the selection of consultants, as well as briefing

the MTR team on the MTR process, relevant methodology and tools. The FAO GEF CU also follows

up with the BH to ensure the timely preparation of the Management Response.

27. PTF members, including the BH, are required to participate in meetings with the MTR team, make all

necessary information and documentation available and comment on the terms of reference and

MTR report. However, their level of involvement will depend on team members’ individual roles and

level of participation in the project.

28. The National Project Director (NPD) facilitates the participation of government partners in the MTR

process and supports the PMU in ensuring good communication across government. The Project

Steering Committee (PSC) facilitates government and other partner and stakeholder participation

in the MTR process.

29. The MTR team is responsible for developing and applying the MTR methodology, producing a brief

MTR inception report, conducting the MTR and producing the MTR report. All team members will

participate in briefing and debriefing meetings, discussions and field visits. They will contribute

written inputs to the draft and final versions of the MTR report, which may not reflect the views of

the government or of FAO. The MTR team leader will guide and coordinate the MTR team members

in their specific tasks and lead the preparation of the draft and final reports. The team leader will

consolidate team inputs with his/her own and will have overall responsibility for delivering the MTR

5 The BH/RM should be responsible for the administrative procedures associated with the recruitment of the MTR

consultants.

12

report. The MTR team will agree with the FAO GEF CU MTR focal point on the outline of the report

early in the MTR process, based on the template provided in Annex 12 of the MTR Guide. The MTR

team is free to expand the scope, criteria, questions and issues listed above, and develop its own

MTR tools and framework, within the timeframe and resources available and based on discussions

with the BH/RM and PTF. Although an MTR report is not subject to technical clearance by FAO, the

BH/RM and FAO GEF CU do provide quality assurance checks of all MTR reports.

30. The relevant GEF Operational Focal Point (OFP) must be involved in any GEF project or programme

evaluation process, in accordance with the GEF Evaluation Policy (2019). The BH should inform the

OFP of the MTR process and the MTR team is encouraged to consult with him/her during the review

process. The team should also keep the OFP informed of progress and send him/her a copy of the

draft and final MTR reports.

31. More detailed guidance on the roles and responsibilities of the key individuals and groups involved

in the MTR can be found in Annexes 2 and 3 of the MTR Guide.

13

6 MTR team composition and profile

32. The skills, competencies and characteristics needed in the MTR team are specific to the MTR. The likely

structure and composition of the MTR team, including the roles and responsibilities of its members,

should be set out in the terms of reference for individual consultants.

33. The lead international MTR consultant should have the following minimum technical requirements:

• an advanced university degree in evaluation, agriculture, natural-resource management, social

and economic development, or a related field (to be adapted to each MTR);

• five years of relevant experience in supporting, designing, planning and/or conducting

development evaluations;

• knowledge of FAO and GEF work/procedures, or other UN agencies, would be an asset as would

appropriate language skills.

34. The MTR consultants should be independent of any organizations that have been involved in

designing, executing or advising on any aspect of the project being evaluated in the MTR and should

not have been involved in any aspect of the project previously.

35. The national consultant should have the following experience:

• a university degree in evaluation, agriculture, social and economic development, or a related field

(to be adapted to each MTR);

• three years of experience in a relevant technical area and a good understanding of the national

and/or local context, as appropriate;

• ideally, experience in supporting, designing, planning and/or conducting development

evaluations; and

• knowledge of FAO and GEF work/procedures, or other UN agencies, would be an asset as would

appropriate language skills.

36. Both consultants are expected to demonstrate the following competencies:

• results focus

• teamwork

• excellent communication skills (both written and oral) in English

• building effective relationships

• knowledge sharing and continuous improvement

14

7 MTR products (deliverables)

37. This section describes the key deliverables the MTR team is expected to produce. At a minimum,

these products should include the following:

• The MTR inception report. The MTR team should prepare an inception report before

beginning data collection. This should detail the MTR team’s understanding of what is being

assessed and why, and their understanding of the project and its aims (set out in a theory of

change). It serves as a map and reference for planning and conducting an MTR and as a useful

tool for summarizing and visually presenting the MTR design and methodology in discussions

with stakeholders. The inception report details the GEF evaluation criteria, the questions the

MTR seeks to answer (in the form of an MTR matrix), the data sources and data collection

methods, analysis tools or methods appropriate for each data source and data collection

method, and the standard or measure by which each question will be evaluated. The inception

report should include a proposed schedule of tasks, activities and deliverables, designating a

team member with lead responsibility for each task or product (as appropriate).

• The draft MTR report(s). The project team, BH/RM, FAO GEF CU and key stakeholders in the

MTR should review the draft MTR report to ensure its accuracy and quality in two review rounds:

(a) a first review, taking around 10 working days, by the project team and FAO (BH, LTO, FLO

and FAO GEF CU MTR focal point), then a second review, also taking around 10 working days,

by the government counterpart(s), key external partners and stakeholders.

• The final MTR report. This should include an executive summary and be written in an official

language of the country where the project is taking place (English is preferred if there is a choice

and if the project involves more than one country with no common official language). It is

important that the executive summary is presented in both the official national language and

in English. Supporting data and analysis should be annexed to the report, if deemed important,

to complement the main report. Translations into other official UN languages, if required, will

be FAO’s responsibility. The executive summary should include the following paragraphs in

order to update the GEF Portal: (1) information on progress, challenges and outcomes on

stakeholder engagement; (2) information on progress on gender-responsive measures; and (3)

information on knowledge activities and products. The template for the MTR report can be

found in Annex 11 and guidance on writing the report in Annex 12 of the MTR Guide.

• A two-page summary of key findings, lessons, recommendations and messages from the MTR

report, produced by the RM and PMU, in consultation with the MTR team, that can be

disseminated to the wider public for general information on the project’s results and

performance to date. This can be posted as a briefing paper on the project’s website but more

creative and innovative multimedia approaches, such as video, photos, sound recordings, social

media, short stories (for suitable cases or country studies), infographics or even comic or

cartoon format, may be more effective depending on the circumstances.

• Participation in knowledge-sharing events, such as stakeholder debriefings, as needed.

15

8 MTR timeframe

38. This section lists the due date or timeframe of the MTR and describes all tasks and deliverables (such

as briefings, the draft report and final report), as well as the associated roles and responsibilities of

the key MTR individuals and groups.

Table A4.2 Suggested MTR timeline

Task When/duration (recommended) Responsibility

Terms of reference preparation 3 months before the MTR field

mission

BH/RM, LTO, FLO and FAO GEF CU

MTR focal point

Terms of reference finalization 2 months before the MTR field

mission

BH/RM

Team identification 2 months before the MTR field

mission

BH/RM, LTO, FLO and FAO GEF CU

MTR focal point

Team recruitment 1 month before the MTR field

mission

BH with input from the FAO GEF CU

for international and national

consultants

Travel arrangements and

organization of the agenda and

travel itinerary in country for the

field mission

4‒6 weeks before the MTR field

mission6

BH/RM, project team and MTR

team

Reading background

documentation

2‒3 weeks before the MTR field

mission

MTR team in preparation for the

MTR

Briefing of MTR team 2‒3 weeks before the MTR field

mission

BH/RM, supported by PTF and FAO

GEF CU as necessary

MTR inception report 2 weeks before the MTR field

mission

MTR team

Quality assurance and clearance

of the MTR inception report

1 week before the MTR field

mission

BH/RM and the FAO GEF CU MTR

focal point

MTR missions – confirmation of

interviews, meetings and visits

1‒3 weeks for the MTR field

mission

MTR team with the support of the

PMU

Production of first draft report for

circulation

No more than 3 weeks after the

field mission

MTR team

Circulation and review of first draft

MTR report

5‒10 working days for review BH/RM, PMU, FAO GEF CU MTR

focal point, LTO for comments and

quality control (organized by

BH/RM)

Production of second draft MTR

report

1 week for the inclusion of

feedback (recommended; could

be less if consultants are available)

MTR team

Circulation of second draft MTR

report

5‒10 working days for review BH/RM and key external

stakeholders (organized by BH/RM)

Production of final MTR report 1 week for the inclusion of final

feedback (recommended; could

be less if consultants are available)

MTR team

Management Response 1 month after the final report is

issued BH

Follow-up reporting in FAO PPR or

GEF PIR

Maximum 6 months after the MR

is issued BH

6 Note that FAO rules require all travel authorisation to be approved at least 15 days before travel.

16

Annexes

39. Annexes to the MTR terms of reference can be used to provide additional detail about the

background to the MTR and requirements to facilitate the work of MTR consultants. Some examples

include:

• Project results framework and theory of change – This provides additional information on the

structure and causal logic of the project being assessed.

• FAO‒GEF project MTR report outline, including the GEF rating table – This is available in Annex 11

in the MTR Guide.

• Documents to be consulted – This is a list of important documents and web pages the MTR team

can consult at the outset, before finalizing the MTR’s design and inception report. A list of key

documents to be included in the “project information package” can be found in Box A4.2.

Box A4.2. Documents to be provided to the MTR team (“project information package”)

1. GEF PIF with technical clearance 2. Comments from the GEF Secretariat, the GEF Scientific and Technical Advisory Panel (STAP)

and GEF Council members on project design, plus FAO responses 3. FAO concept note and FAO Project Review Committee report 4. Request for GEF CEO endorsement 5. FAO–GEF project preparation grant document 6. GEF-approved project document and any updated approved document following the

inception workshop, with latest budgets showing budget revisions 7. Project inception report 8. Six-monthly FAO PPRs 9. Annual workplans and budgets (including budget revisions) 10. All annual GEF PIR reports 11. All other monitoring reports prepared by the project 12. Documentation detailing any changes to the project framework or components, such as

changes to originally designed outcomes and outputs 13. List of stakeholders 14. List of project sites and site location maps (for planning mission itineraries and fieldwork) 15. Execution agreements under OPIM and letters of agreement 16. Relevant technical, backstopping and project-supervision mission reports, including back-to-

the-office reports by relevant project and FAO staff, including any reports on technical support provided by FAO headquarters or regional office staff

17. Minutes of the meetings of the PSC, FAO PTF and other relevant groups 18. Any ESS analysis and mitigation plans produced during the project design period and online

records on FPMIS 19. Any awareness-raising and communications materials produced by the project, such as

brochures, leaflets, presentations for meetings, project web address, etc. 20. FAO policy documents in relation to topics such as FAO Strategic Objectives and gender 21. Finalized GEF focal-area tracking tools at CEO endorsement, as well as updated tracking tools

at mid-term for GEF-5 projects (and for GEF-6 and GEF-7 projects with Biodiversity Focal Area (BD) Objective 2 and management of protected areas) and/or review of contribution to GEF-7 core indicators (retrofitted) for GEF-6 projects, and GEF-7 core indicators for GEF-7-approved projects, as defined in the Core Indicators Worksheet (GEF, 2019a)

22. Financial management information, including an up-to-date co-financing table, a summary report on the project’s financial management and expenditures to date, a summary of any financial revisions made to the project and their purpose, and copies of any completed audits for comment (as appropriate)

17

23. The GEF Gender Policy (GEF, 2017), GEF Gender Implementation Strategy (GEF, 2018a), GEF Guidance on Gender Equality (GEF, 2018b) and the GEF Guide to Advance Gender Equality in GEF Projects and Programmes (GEF, 2018c)

The following documents should also be made available to the MTR team on request or as required:

24. FAO Country Programme Framework documents, the FAO Guide to the Project Cycle (FAO, 2012b), FAO Environment and Social Management Guidelines (FAO, 2015), FAO Policy on Gender Equity, the Guide to Mainstreaming Gender in FAO’s Project Cycle (FAO, 2017a) and the Free, Prior and Informed Consent Manual (FAO, 2016)

For programmes:

25. CEO endorsement/approval of child projects under the programme

26. Programme framework documents and child project titles or concepts

Annex 5 – Illustrative MTR budget lines for an FAO-GEF project

Table A5.1 Illustrative MTR budget lines for an FAO–GEF project – single country with a maximum of two field site visits (based on a 13-day mission with international travel)

Number Units (days) Unit cost

(USD) Total (USD)

Mission: travel and DSA

Team leader – airfare, visa, travel expenses, etc.

Daily subsistence allowance – team leader1

Daily subsistence allowance – national consultant

National transport – field visits, interviews, data collection

Subtotal

Honorariums

Team leader

National consultant

Subtotal

Costs for stakeholder participation in MTR interviews (if required)

Travel costs for interviewees2

Venue and miscellaneous (e.g. catering)

Subtotal

Other related costs (if required)

MTR report translation to English/local language

Subtotal

Contingency

Contingency – 5% of total

Total

1 The number of days for the team leader should take international travel into account. Also, the number of days should assume that consultants will work a six-day week. This takes into account the fact that most government officials do not work weekends, so there are restrictions on interviews during these periods, and allows consultants the option of a rest (non-work) day once a week. 2 Particular consideration should be given to the costs of facilitating local-community stakeholder participation; such participants may need to take time off work (and so lose income) to attend interviews.

ANNEX 6 ‒ Consultant job descriptions for FAO-GEF MTR team leaders and members


Terms of reference for Choose an item. *

Name: [Name of consultant]

Job Title: Senior MTR Specialist/Team Leader

Division/Department: [Name of division/department in charge of the MTR, for example, the country/regional office or the lead technical unit]

Programme/Project Number: Mid-Term Review of “[Name of Project” (Code)]”

Location: Home-based with missions to headquarters and the project countries ([list countries])

Expected Start Date of Assignment: [EOD of consultant] Duration: [Number] days

Reports to: Name: [Name of supervisors – BH and FAO GEF CU MTR focal point]

Title: [Job titles of supervisors]

* Please note: If this TOR is for Consultant / PSA.SBS contract, the minimum relevant experience required for the assignment is as follows:

1 year for a category C 5 years for a category B 12 years for a category A

** Please enter a short title (max 25 chars) for this assignment.

GENERAL DESCRIPTION OF TASK(S) AND OBJECTIVES TO BE ACHIEVED

[Overall project description – one paragraph] [Project objectives – one paragraph]

Under the direct supervision of the project’s budget holder (BH), [name and full job title, e.g. FAO Representative] and of the FAO-GEF Coordination Unit (FAO GEF CU) MTR focal point, [name of individual], the Team Leader is responsible

for coordinating the MTR team’s contribution to the MTR of “[Name of Project” (Code)]” and has ultimate responsibility for ensuring the delivery of the MTR report. His/her mandate is derived from and must fully comply with the overall terms of reference of the MTR. He/She reports to the BH and to the FAO GEF CU MTR focal point.

The Team Leader will coordinate the MTR team’s activities, ensure that the methodology foreseen in the terms of reference is implemented and organize the division of labour within the MTR team according to areas of expertise.

[Include if relevant:] He/she will also cover in detail the analysis of the [insert as appropriate] aspects of the project and the following sections of the MTR report: [insert as appropriate]. Specific responsibilities include:

• Review relevant background documentation made available by [budget holder or designated MTR manager (RM) and project team], including the project document, the project’s logframe, progress and final reports, workshop and technical reports, among others as listed in the MTR terms of reference;

• Review and complement (expand as appropriate) the methodology described in the terms of reference, contribute to the preparation of data-collection tools, including questionnaires, checklists and interview protocols

as appropriate;

• Coordinate preparation, drafting and finalization of the MTR inception report, including an MTR matrix, theory of change and stakeholder table, and participate in the finalization of the team’s work programme;

• Lead and coordinate the collection of primary data by the MTR team through interviews and meetings (face-to-face or virtual) with relevant FAO officers in headquarters and/or [list as appropriate: project coordination unit,

FAO Decentralized Offices], among other things, and during country visits with key stakeholders including the government, FAO Representative, external partners, project teams, international organizations, the private sector, civil society, academia, research institutes and ultimate beneficiaries, as appropriate, as described in the

terms of reference of the MTR;

• Coordinate the collection of primary data by the MTR team during country visits and relevant secondary data, according to the methodology presented in the MTR terms of reference and detailed in the inception report;

• Lead the analysis and discussion of evidence collected within the MTR team to identify key findings and preliminary conclusions that respond to the MTR’s issues and questions, and formulate preliminary recommendations in line with the findings and conclusions;

• Ensure that all the findings are sufficiently triangulated and validated;

• Present the preliminary MTR findings to key stakeholders, as required;

• Lead, coordinate and prepare all deliverables planned in the MTR terms of reference, according to specifications provided in the terms of reference and detailed in the FAO–GEF MTR Guide and annexes;

• Coordinate the participation and contribution of team members in all deliverables, as required;

• Lead the preparation of the first and second drafts of the MTR report, integrate comments received, as appropriate, from the BH, FAO GEF CU, other FAO and government agency staff, and other relevant

stakeholders, as appropriate;

• Lead the finalization of the MTR report and coordinate the inputs of other members of the MTR team into the final version, as needed.

In terms of reporting, or if information, advice or guidance is required from FAO by the consultant, he/she should address requests to both the BH/RM and the FAO GEF CU focal point.

KEY PERFORMANCE INDICATORS

Expected outputs:

-

Required completion date:

[List as appropriate, for example:] • Inception report, including MTR questions

• Briefing on preliminary findings of the MTR following the field mission(s) • First draft of the report

• Second draft of the report • Final MTR report, including comments matrix/audit trail

Partial payments of the honorarium will be authorized on completion of fieldwork and completion of the above deliverables, as agreed with the BH/RM and FAO GEF CU

focal point.

[Target completion date]

[Target completion date] [Target completion date] [Target completion date] [Target completion date]


Terms of Reference for Choose an item. *

Name: [Name of consultant]

Job Title: [Technical expertise of consultant, for example – forestry expert or NRM expert]/team member

Division/Department: [Name of division/department in charge of the MTR, for example, the country/regional office or the lead technical unit]

Programme/Project Number: Mid-Term Review of “[Name of project” (Code)]”

Location: Home-based with missions to headquarters and the project countries ([list countries])

Expected Start Date of Assignment: [EOD of consultant] Duration: [Number] days

Reports to: Name: [Names of supervisors] BH and MTR Team Leader Title: [Job titles of supervisors]

* Please note: If this terms of reference is for a consultant / PSA.SBS contract, the minimum relevant experience required for the assignment is as follows:

1 year for a category C 5 years for a category B 12 years for a category A

** Please enter a short title (max 25 chars) for this assignment.

GENERAL DESCRIPTION OF TASK(S) AND OBJECTIVES TO BE ACHIEVED

[Overall project description – one paragraph] [Project objectives – one paragraph]

Under the administrative supervision of the project’s budget holder (BH), [full job title, e.g. FAO Representative] and MTR Team Leader, the Team Member will contribute to the MTR for his/her area of specialization and be responsible for

supporting the MTR Team Leader. His/Her mandate is derived from and must fully comply with the overall terms of reference for the MTR. He/She will provide inputs as agreed with the Team Leader and MTR Manager (RM). The Team Member reports to the BH and the MTR Team Leader.

He/she will also cover in detail the analysis of the [insert as appropriate] aspects of the project and the following sections of the MTR report: [insert as appropriate].

Specific responsibilities include:

• Review the relevant background documentation made available by [the budget holder or designated MTR manager and project team]: the project document, the project’s logframe, progress and final reports,

workshop and technical reports, among other things;

• In accordance with the methodological approach described in the terms of reference, contribute to the preparation of data-collection tools, including questionnaires, check-lists and interview protocols, as

appropriate; insofar as possible, participate in the finalization of the MTR team’s work programme;

• Participate actively in the collection of primary data, as required by and identified in the MTR matrix, under the leadership of the Team Leader during the country visits, among other things, through interviews and

meetings (face-to-face or virtual) with key stakeholders, such as government entities, project partners ([include name of project partners]), the FAO Representative, external partners, project beneficiaries, international organizations, the private sector, civil society, academia and research institutions;

• Provide specialised expertise in [complete as appropriate], assessing outputs, deliverables and activities carried out under his/her specific area – (outputs/outcomes could be listed – if necessary);

• Contribute to the consolidation of all evidence collected during the MTR process in a structured format following the key MTR questions and criteria;

• Ensure that all primary and secondary data collected are recorded during the MTR process and organized in a structured format following the key MTR questions, based on which all findings will be triangulated;

• Participate in the analysis and discussion of evidence collected during the MTR within the MTR team to identify key findings that respond to MTR issues and questions, and formulate preliminary conclusions and recommendations in his/her specific field of expertise, as discussed and agreed with the Team Leader and

the RM;

• Deliver inputs (including written, as required by the MTR Team Leader) on the specific areas of assessment, per the overall terms of reference, to all deliverables, including: the inception report, debriefing sessions

following field missions, the draft MTR report according to the specifications provided in the terms of reference, and participate in addressing and integrating any feedback on the draft report from stakeholder reviewers (e.g. the budget holder, FAO GEF CU, government counterparts, local-community stakeholders), as appropriate, and support the MTR Team Leader in finalizing the report, as required.

• [Other responsibilities as stated in the MTR terms of reference. For example, this could include conducting a technical assessment of the project outputs and/or activities.]

In terms of reporting, or if information, advice or guidance is required from FAO by the consultant, he/she should address requests to both the BH/RM and the FAO GEF CU focal point and the MTR Team Leader.

KEY PERFORMANCE INDICATORS

Expected outputs: Required completion date:

[List key expected outputs/milestones of the MTR team member’s work, for

example:] • Inception report, including MTR matrix • Field reports (as appropriate)

• Briefing on preliminary findings of the MTR following the field mission(s) • First draft of the report

• Final MTR report

Partial payments of the honorarium will be authorized upon completion of the above deliverables, as agreed with the BH/RM and FAO GEF CU focal point.

[Target completion date] [Target completion date] [Target completion date]

[Target completion date] [Target completion date]

1

Annex 7 – Notes on drafting the MTR inception report This annex provides some guidance on preparing the inception report for an FAO–GEF project mid-term review (MTR). The inception report complements the MTR terms of reference and clarifies the MTR team’s understanding of and approach to the MTR. It helps to establish clarity and mutual understanding between the parties involved in the MTR – principally the MTR team, the budget holder (BH) or their designated MTR manager (RM) and GEF Coordination Unit (FAO GEF CU) – including how the work will be organized, who will do what and when. It is an internal FAO document and is, therefore, not published. The MTR team leader is responsible for preparing the inception report, with input from the other members of the MTR team as needed. The inception report, the terms of reference and the MTR Guide and annexes are the main MTR reference tools. The inception report should be concise, with a suggested length of 7‒10 pages, and be informed by an analysis of the documentation provided to the MTR team and the technical briefings/inception meetings held in phase 2 (design and preparation). It should contain five sections:

1. the background to the project and a theory of change1 (graphic and narrative); 2. a brief stakeholder analysis and a preliminary list of potential interviewees (building on the initial

stakeholder analysis presented in the terms of reference); 3. detailed information on the proposed approaches to be used in the review and the methods

selected for data collection and analysis; 4. an MTR matrix that sets out the key questions and sub-questions the MTR team will use as the

basis for collecting information; and 5. a short section on the limitations and risks, timeline and deliverables of the MTR.

1. Project background and theory of change This section should give a brief narrative on the context, aims and components of the project (1‒2 pages maximum, as details are already included in the project document). It should outline the MTR team’s understanding of the project and its aims, including a statement of the project’s objectives and planned results, as well as a description of the project’s causal logic. It should be set out a theory of change, presented both as a brief narrative and in graphic format, as an annex to report. The section should not be a cut-and-paste of the background text in the project document, but a summary of the essential project elements that need to be understood in order to set the scene for the MTR. For most FAO‒GEF projects approved during GEF-5 and GEF-6, the causal logic of the intervention is often partially represented in a logframe, rather than a more complete theory-of-change narrative and graphic. If a theory of change has not been produced (at the formulation stage or thereafter) or is incomplete, the MTR team will need to reconstruct one during the inception stage using the project document and other sources, to identify whether the project’s strategy is coherent, proving effective in achieving the desired results and likely to contribute to longer-term impacts. If initial analysis suggests not, the MTR should identify the changes required to the project’s strategy and framework to better deliver the desired project outcomes and objectives. The central question here is whether the project, as designed, will actually achieve the desired results and make an impact and, if not, whether it can be redesigned to do so. The amount of theory-of-change “reconstruction” needed will depend on the quality of the initial theory of change (if there is one), the time that has lapsed since the project’s design and the extent of any changes made to the design during implementation (for example, to project outputs and outcomes during the inception period). Project objectives or goals should be incorporated into the theory of change, usually as

1 The theory of change of a project depicts the causal pathways from project outputs (goods and services delivered by the project) through outcomes (changes resulting from the use made by key stakeholders of project outputs) to impact (long-term changes in environmental benefits and socioeconomic conditions). In the case of GEF, the focus is on delivering longer-term Global Environmental Benefits (GEB), as measured by core indicators under GEF-7.

2

intermediate states or long-term impacts. The initial (re)constructed theory of change should then be discussed and validated during phase 3 of the MTR, during the field missions and in meetings with the project team and stakeholders. It is important that the theory of change be at least provisionally agreed before most of the data collection has been done (review of progress reports, in-depth interviews, surveys, etc.), as it will help to define which direct project outcomes, drivers and assumptions need to be assessed and measured to enable the MTR team to evaluate project effectiveness, as well as the likelihood of impact and sustainability. For more guidance on developing a theory of change, see Annex 10 of the MTR Guide. 2. Stakeholder analysis and list of interviewees A stakeholder analysis with a list of potential key interviewees should be presented, based on a review of the preliminary stakeholder analysis in the MTR’s terms of reference and initial discussions with the BH/RM, LTO, FLO and PMU staff. The stakeholder analysis should identify the different groups to be involved in the MTR, citing specific agencies, non-governmental organizations (NGOs), civil-society organizations, universities, institutions and individuals. It should detail the rationale and criteria for their selection (their role in project, etc.) and how they will be involved in the review. The analysis should consider the most appropriate methods of ensuring the effective participation of each group, the importance of the MTR results to them and their likely use. (See Annex 8 of the MTR Guide for more information on stakeholder analysis and a template.) FAO individuals to be interviewed should include (but not necessarily be limited to) the BH, LTO, FAO GEF CU FLO, staff from other FAO technical divisions and other relevant members of the project task force (PTF). Non-FAO interviewees may include individuals from the beneficiary communities and governments (national and local, including the GEF operational focal point, if available), the institutional project counterpart, the other implementing and executing agencies, the co-financing partner(s), NGOs, the private sector, academia, civil society and other agencies and development agents working in similar areas. Local stakeholders often have different views and concerns to national-level stakeholders, such as government or state agency staff. It is important that these groups have a voice in the MTR. Consequently, the design of the MTR needs to ensure that local groups are given sufficient opportunity to be included. Meetings and interviews with these groups (or their representatives) should be treated as a priority in planning MTR field missions. 3. Methodology This section of the MTR inception report should present details of the conceptual and methodological approach and the data-collection methods and tools that will be used by the MTR team to collect credible and useful data (specifying qualitative and quantitative methods, as well as any participatory or mixed-method approaches). The choice of data-collection and analysis tools will depend on the fieldwork design and will be tailored to the project in question, with a view to answering specific review questions. The inception report should explain how the validity and reliability of data and analysis will be ensured, including methods of triangulation. The MTR is an opportunity to assess implementation progress on gender-related activities, targets and results as well as assessing gender-related design features. Consequently, this section should also detail how the methodology and data collection for the MTR will reflect good practices in assessing gender equality as well as the involvement of marginalized groups in GEF projects (see GEF, 2017a; 2017b; 2018a and UN CC: Learn, n.d.), including any-data collection methods used to ensure the full participation of women. This section of

the inception report should also document how the key stakeholders to be interviewed by the MTR were selected, differentiating between male and female stakeholders, and how their participation was ensured (literacy levels, language skills, logistical constraints, etc.). The proposed tools should be explicitly linked to the corresponding question(s) in the MTR matrix and briefly listed/documented in the main text of the inception report. The choice of tools should be informed by

3

stakeholder analysis, as well as by an analysis of the completeness, availability and reliability of the data collected during the project’s design and inception/early implementation phases. The MTR should take into account secondary data ‒ information referenced by interviewees or documented but not directly accessed by the MTR team – M&E information and any previous reviews or evaluations, etc. The tools most frequently used include semi-structured interviews, focus groups, checklists, desk studies, short stories, direct observations from field visits, online surveys and project or country case studies (for regional or global reviews and evaluations). The RM and project management unit (PMU) should advise the MTR team on the feasibility of the proposed tools and fieldwork design (for example, in relation to the available budget). 4. The MTR matrix and sampling This section presents the set of questions and sub-questions the MTR team will use as the basis for its data-collection interviews. These are set out in the MTR matrix (see Annex 9 of the MTR Guide for an MTR matrix template). The MTR matrix should specify the relevant indicators and data sources and how the data will be collected (desk reviews, online surveys, semi-structured interviews, focus groups, short stories, etc.) for each review question/sub-question under the various criteria. The geographic coverage of the intervention and field locations to be visited (with a map showing the field sites, where possible), as well as the criteria for the selection, should be presented in the inception report. The groups and numbers of people to be interviewed at each site should be detailed wherever possible, including information on the involvement of women and minority groups. 5. Limitations, risks, timeline and deliverables This section of the MTR inception report should detail any limitations and risks that could affect the MTR, such as the availability of data, the timing of field visits, the choice of MTR tools and the gender-responsiveness of the chosen approach. There should be an explanation as to how the MTR team will manage and mitigate these limitations and risks and/or their implications for the MTR process and evidence gathering. The inception report should present a proposed workplan for the MTR team, including a schedule of tasks, activities and deliverables with a preliminary itinerary for the field visits. The agenda should include a final 2‒4 hour briefing with the FAO Country Office, key executing partners and other stakeholders. GEF stresses the need to devote sufficient time for meetings with local communities and groups that have either participated in the project or been impacted by it (positively or negatively). Consequently, the workplan should allocate a significant proportion of the field mission to assessing local stakeholder views and issues of relevance to these groups.

1

Annex 8 – MTR participatory stakeholder engagement and analysis In line with internationally agreed evaluation standards, including United Nations Evaluation Group (UNEG) Norms and Standards 2016, Standard 4.61 (UNEG, 2016) and the GEF Policy on Stakeholder Engagement (GEF, 2018e), it is essential to involve those stakeholders (from implementers to partners and beneficiaries) directly affected by the intervention or subject matter from the earliest stages of the MTR process. Engaging stakeholders encourages ownership of the MTR and enhances the relevance and use of its results. For instance, consulting with the direct implementers of an intervention creates an opportunity to find out what they think the objectives of the MTR should be and the key issues it should address. Stakeholder groups identified in the Project Preparation Grant (PPG) phase participatory stakeholder analysis should be considered in the MTR stakeholder analysis, along with those involved in the implementation stage. Participatory stakeholder analysis is the process whereby the MTR manager (RM) decides whom to involve in the MTR process, when and in what way, based on a clear understanding of each stakeholder’s interests, interactions and experiences (both positive and negative) with the project. To understand how best to engage the various stakeholders, the RM must identify the key project stakeholders early on in the planning phase, then analyse their likely interest and stake in participating in the MTR process. A variety of mechanisms can be used to consult with a broad range of stakeholders (for example, consultation meetings on evaluation design, validation workshops on preliminary findings and post-evaluation learning workshops). The degree and level of stakeholder participation in an MTR process varies depending on budgetary and time constraints and the RM will need to weigh the level of stakeholder participation against the benefits and limitations. Useful guidance can be found in the UNEG handbook, “Integrating Human Rights and Gender Equality in Evaluation” (UNEG, 2011) (Box 1).

Box A8.1. Determining the degree of stakeholder participation

The following questions should be considered when deciding the degree of stakeholder participation in an MTR.

• Beneficiaries, implementers, rights-holders and duty‐bearers can be involved in the MTR process to varying degrees. What will the implications be in terms of effort, timeline and budget?

• Should stakeholders be consulted together or separately? If consulted together, would this pose risks and dangers for any members of the groups involved? Would this cause some groups to be excluded or less heard, for example, women, elderly, people with disabilities or ethnic minorities? What will the process be for ensuring that all perspectives are fairly heard, avoiding bias, as some may be more reticent than others for various reasons (power differences, literacy levels, confidence levels, etc.)? How will the team mediate differences between stakeholder groups and individuals, build agreement and take decisions when differences cannot be reconciled?

• How can the envisaged level of participation in the MTR process be ensured, even if the reality is that the intervention to be assessed has limited participation to date? How can the MTR help the intervention to overcome participation challenges?

1 This states that, “Inclusive and diverse stakeholder engagement in the planning, design, conduct and follow-up of evaluations is critical to ensure ownership, relevance, credibility and the use of evaluation. Reference groups and other stakeholder engagement mechanisms should be designed for this purpose.” It further states that, “Processes should be in place to secure the participation of individuals or parties who may be affected by the evaluation, are able to influence the implementation of recommendations or who would be affected in the long term. Stakeholders should be consulted in the planning, design, conduct and follow-up of evaluations”.

2

• Is there a clear communication strategy with MTR stakeholders in terms of who will participate, who will be consulted and how, and who will make decisions when there are differences of opinion?

• Do MTR team members have facilitation skills and experience commensurate with the level of participation decided?

• Has sufficient consideration been given to participation to ensure the credibility of MTR results?

Source: Based on UNEG (2011) The RM and MTR team should consult the stakeholder engagement plan or equivalent mapping/assessment exercise undertaken in the project design phase, or any updated stakeholder analysis undertaken since project implementation. Local stakeholder participation and participatory approaches in evaluation are particularly necessary in projects and programmes that affect the incomes and livelihoods of local groups, especially disadvantaged populations in and around project sites (for example, indigenous and other local communities, women and poor households). Consequently, particular attention should be paid to identifying disadvantaged and vulnerable groups or individuals that may be affected by the project, as well as civil society groups involved or with an interest in the project (these groups should be identified in the stakeholder engagement plan and in the risk matrix or environmental and social management plan), and to ensuring that they have the opportunity to participate in the MTR process. A matrix is the most common tool used in stakeholder analysis (see Table 1). It helps guide the RM and MTR

team in the collection and analysis of information on stakeholders by considering key aspects of stakeholder

interests and how these might be managed throughout the MTR process, as well as the type and level of

stakeholder engagement in various phases of the MTR. Table A8.1 contains a template for a stakeholder analysis

matrix that can be used during the MTR process.

Table A8.1. Participatory stakeholder analysis matrix template

Key stakeholders

(disaggregated as

appropriate)2

What is their

role in the

project?


inclusion in or exclusion

from the MTR?

Priority for

MTR (1-3)3

How and when should


MTR?

1. Active stakeholders with direct responsibility for the project, e.g. FAO, executing partners

Name

Stakeholder group 1

Name

Stakeholder group 2

Etc.

2. Active stakeholders with authority to make decisions on the project, e.g. members of the PSC

Name

Stakeholder group 1

Name

Stakeholder group 2

Etc.

2 Include the names of relevant individuals, if known, and be as specific as possible

3 1 = essential; 2 = desirable; 3 = if time and resources allow

3

Key stakeholders

(disaggregated as

appropriate)2

What is their

role in the

project?


inclusion in or exclusion

from the MTR?

Priority for

MTR (1-3)3

How and when should


MTR?

3. Secondary stakeholders (only indirectly or temporarily affected)

Name

Stakeholder group 1

Name

Stakeholder group 2

Etc.

4. Stakeholders at grassroots level who benefit directly or indirectly from the intervention (gender disaggregated

where possible)

Name

Stakeholder group 1

Name

Stakeholder group 2

Etc.

5. Stakeholders at grassroots level who do not benefit from the intervention (gender disaggregated where

possible)

Name

Stakeholder group 1

Name

Stakeholder group 2

Etc.

6. Other interest groups that are not participating directly in the intervention, e.g. development agencies working

in the area, civil-society organizations

Name

Stakeholder group 1

Name

Stakeholder group 2

Etc.

1

Annex 9 – MTR matrix template

Evaluative questions Indicators Sources Methodology List evaluative question(s) and sub-questions here

Relationships established, coherence of project design and implementation approach, specific activities conducted, quality of risk mitigation strategies, etc.

Project documents, national policies or strategies, websites, project staff, project partners, data collected throughout the MTR mission, etc.

Document analysis, data analysis, interviews with project staff, interviews with stakeholders, etc.

1. Strategic relevance

2. Effectiveness – progress towards results

3. Efficiency

4. Factors affecting performance

5. Sustainability of project results

6. Cross-cutting issues – equity issues (e.g. gender, youth, vulnerable groups) and environmental and social safeguards (ESS)

1

Annex 10 – Theory of change for MTRs and final evaluations of FAO-GEF projects

1. What is a theory of change? The theory-of-change approach has been used more and more in recent years to design and evaluate development interventions, including for GEF-financed interventions. A theory of change is a description of how and why a desired change is expected to happen in a particular context through an intervention (project or programme). It describes the causal pathways between results, from the “outputs” to direct “outcomes” of the intervention, through “medium-term and longer-term outcomes” or “intermediate states” that require additional inputs and involve other actors, towards desired “impacts”. In the case of GEF projects, these latter stages in the causal logic usually relate to focal-area goals, strategic priorities and objectives. A theory of change also outlines the drivers of and assumptions on the contextual or environmental factors that will support or hinder progress toward the realization of outcomes and impacts along the pathway of change, for example, the political situation or climate change. The theory of change is generally presented as a graphic with an accompanying narrative.

A theory of change attempts to answer the question, “How will the project or programme contribute

to the desired long-term change?”

Figure A10.1 Example of a simple, generic theory-of-change diagram

A key difference between a logframe and a theory of change is that although a logframe can articulate the goals and resources of a project, it focuses less on the complex social, economic, political and institutional processes that underline environmental and social change and generally does not capture many of the medium- and longer-term outcomes and states along the causal chain.

2

2. Why is a theory of change useful for MTRs and final evaluations? Analysis of an intervention’s theory of change helps to identify the strategies and approaches a project/programme needs to ensure the delivery of its stated objectives and long-term goals. When a theory of change is developed or analysed for an MTR (or final evaluation), it allows the project’s design logic to be tested. For instance, it might show that a project contains sets of activities and outputs that will not deliver any meaningful contribution to outcomes or impact. Alternatively, theory-of-change analysis may identify activities that are still needed to deliver the project’s outcomes and, so, are essential to achieving the impact. Developing and analysing an intervention’s theory of change also helps to identify effective strategies and approaches that the project or programme could use for more successful delivery of its stated objectives and long-term goal. This can be particularly useful at the mid-term point, when there is still time to modify and strengthen a project. A theory of change can also help define the external factors that influence change along the “outcome pathways”, which may be drivers (over which the project or its partners have a certain level of control or influence) or assumptions (where the project has little to no control). As a process, the development or revision of a theory of change helps in a couple of ways.

• When undertaken as a group exercise (highly recommended), developing a theory of change helps to enhance ownership of and commitment to the project or programme in question and increases the credibility and acceptance of the MTR or final evaluation results.

• Dedicating time to working jointly with stakeholders helps build mutual trust and respect and increases the probability that stakeholders will actually use the MTR or final evaluation results. It is also helps to manage expectations on both sides.

As a product, the theory of change is useful in:

• facilitating a common understanding of the project/programme between evaluators and stakeholders. A theory of change, when discussed and agreed with stakeholders, reflects a common understanding or interpretation of the project’s intervention logic. Often, it is necessary to articulate the “missing middle” in the intervention’s logic, that is, the different levels of outcomes that need to happen for an impact to occur based on the proposed intervention. Simplifying often complex realities and the intended causal pathways of an intervention into a visual representation helps to make the implicit explicit and ensures that everyone shares the same understanding.

• underpinning and strengthening the design of the MTR or final evaluation. A theory of change clarifies cause-and-effect relationships and the evidential basis for identifying what to evaluate – in other words, what to measure to judge that the actual results of an intervention have been delivered. This will, in turn, determine the approach of the MTR or final evaluation.

• better defining the scope of the MTR or final evaluation. A theory of change shows where and how the project or programme intends to have influence and helps the MTR or final evaluation team to develop more targeted and relevant evaluation questions and identify the data needed to make an informed assessment.

The theory of change can also be used to identify and communicate the role and contributions of key actors and stakeholders, opportunities for replication and catalysis of project results, and issues where sustainability needs to be considered (this is more important for final evaluations, but can provide useful pointers at the mid-term stage). The theory of change is generally presented as a graphic with an accompanying narrative. Sometimes, RMs or final evaluation managers develop the theory of change during the preparatory phase (phase 1), alone or with stakeholders, to help define the scope, questions and even the approach of the review. At other times, managers may prefer to engage the MTR or final evaluation team in the theory-of-change development process.

3

3. How to develop a theory of change To develop a theory of change, one must first identify the desired long-term goals and then work backwards to identify all the conditions (outcomes) that need to be in place for these goals to be met. The process creates an “outcome pathway” of short-, medium- and longer-term outcomes, showing each outcome in logical (causal) relation to all of the others, as well as the chronological flow. This exercise provides the basis for identifying the types of activity or intervention that will be needed to deliver the outcomes and longer-term impacts. Where a theory of change already exists for a project, the MTR team should review and tests its logic, ideally through a group exercise with the project team, revising the structure, components and linkages where necessary. Where there is no theory of change, the MTR team will need to reconstruct one by identifying and analysing project elements from the principal project documents and from responses during interviews with key individuals involved in the project and a group exercise, as set out below.

3.1 Step-by-step guide to creating a theory of change 1. Identify the key stakeholders and individuals to be involved in developing and analysing the theory of

change. This should include the project team, relevant FAO and executing partner staff, and other key stakeholders. Theories of change are best developed and refined as facilitated group exercises, although an initial working draft can be produced by the MTR or final evaluation team to encourage engagement and discussion.

2. As a group, identify the long-term goal(s)/impacts the project seeks to achieve (for example, to restore fish stocks to levels that can produce maximum sustainable yields).

3. Map backwards from the long-term goal by working out the preconditions to or requirements for achieving that goal (initially identified as outcomes or states) and then work back further to identify the required outputs and activities, linking these elements using arrows.

4. Identify any assumptions along the causal chain without which the hypothesis will not work (for example, the overcapacity of fishing fleets must be reduced to sustainable levels).

5. Examine and articulate the rationale behind the connection of each outcome with another (the linking arrows), why actions associated with one outcome will trigger the achievement of another (identify the mechanisms) and ensure the order of outcomes is arranged in the correct (time) sequence (Outcome A needs to happen for Outcome B to be achieved, etc.).

6. Once the causal chain has been established, members of the team/group should identify the most strategic/important interventions required to bring about the desired change (some outcome pathways have more impact than others). These can be illustrated by colouring the boxes and arrows of the outcome pathway or making them bigger than others.

7. Identify a “line of accountability” – the point beyond which a project cannot be held solely accountable for delivering the identified outcome (this should apply to medium-term outcomes and beyond and will require inputs from other non-project actors and interventions). It is generally represented as a dotted line in the graphic depicting the theory of change.

8. Undertake a quality review at the end of the process, answering three main questions:

• Is the theory plausible? Does the logic make sense? Are outcomes in the right order? Are there big gaps in the logic?

• Is the theory “doable”? Is the project realistic? Can it be completed within the timeframe? Are there sufficient resources? Is there a need for additional partners?

• Is the theory testable? Are there solid measurable (SMART) indicators to track progress?

9. As well as capturing the theory of change graphically, it is important to write a narrative to explain the different elements, their connections and logic of the project. This is best done directly during the aforementioned steps. It will require a dedicated note taker to capture all of the discussions, both from group work and plenary discussions.

4

Figure A10.2 Creating a theory of change

Credit: Sidney Harris

4. Tips for developing a compelling theory of change • Project goals, objectives and outcomes should be specific and stated as conditions or states, not

actions or interventions. Objectives and outcomes should be given as simple, short statements that spell out what is to be achieved, for example, “sustainable management of the high seas”. They should avoid additional words that indicate how the objective or outcome is to be achieved (for example, “based on ecosystem system-based management”) or who should be involved (for example, “by engaging all relevant stakeholders through appropriate mechanisms”).

• The theory of change should be kept as simple as possible. Although there is no rule for how many elements should be shown in a theory-of-change diagram, for a complex project, a maximum of 10‒20 would be expected.

• Focus on the cause-and-effect relationship between the elements of the theory of change. One difference between a theory of change and a logframe is that the theory of change acknowledges the breadth of vertical and horizontal relationships that occur in an intervention’s logic. It makes explicit the intended causal relationship between an intervention’s outputs and its ultimate desired impact, thereby embracing the complex reality the project aims to change.

• Identify the evidence that can indicate the effective achievement of each outcome. The outcomes in a theory of change need to be coupled with SMART indicators that guide and facilitate the measurement of progress towards achieving those outcomes and the project’s long-term goals. Consequently, where existing projects are being assessed, indicators should be reviewed and, if necessary, new SMART indicators identified.

• Identify and test the assumptions underlying the cause-and-effect relationships to make them explicit. A theory of change should spell out the assumptions underlying the cause-and-effect relationships that are expected to occur. The MTR or final evaluation should identify, examine and test any key assumptions about the change process to determine their validity. It may be possible to identify additional project activities that could address or mitigate weak assumptions, particularly in relation to

5

the links between a project’s direct activities, outputs and expected outcomes. If assumptions are identified that cannot be addressed, then at least the limits of the intervention will be better known and understood.

5. Theory of change terminology

Impact is a fundamental and durable change in environmental state and human living conditions brought about directly or indirectly by the project, or long-term changes that deliver (or will lead to) GEBs and social benefits. Impacts can be intended or unintended. Outcomes are changes in individual or institutional behaviour, capacity or state to which a project contributes. Outcomes can act over a short (project outcome) or medium-term (medium-term outcome) period beyond the lifetime of the project. Alternatively, outcomes can be classified as: (1) first- order outcomes, which relate largely to the achievement of enabling conditions, such as improved knowledge or skills, new policy agreements or the availability of financing; (2) second-order outcomes, which are often expressed as changes in behaviour, such as changes in agriculture, land management or fishing practices; (3) third-order outcomes, which relate to changes in environmental and/or societal conditions (sometimes referred to as “intermediate states”), including the achievement of Global Environmental Benefits (GEBs); and (4) fourth-order outcomes, which refer to sustainability and ecosystem resilience, such as supporting the delivery of the SDGs. Outputs are the services and products delivered directly by the project in order to achieve the desired project outcomes, such as trained individuals or local pilot projects. Activities are the practical, time-bound actions the project undertakes in order to achieve the desired project outputs, such as training workshops, communication products and research activities. The project should have complete control over activities and outputs. Assumptions are the significant external factors or conditions that influence the ultimate realization of a project’s outcomes and impacts, but are beyond the immediate influence of the project, for example, the turnover of government officials, the global financial situation or severe weather. Drivers are significant external factors, which, if present, can contribute to the ultimate realization of project outcomes and impacts and over which the project, or its stakeholders/partners, has (or could have) some degree of control or influence, for example, public pressure on decision-makers or market demand for a sustainable product.

iii

Annex 11 – FAO–GEF project mid-term review report outline This document provides an annotated template for GEF project mid-term review (MTR) reports, suggesting a structure and offering guidance on content. The template should be used by the MTR team in conjunction with the MTR Guide, particularly Annex 12, which provides a more detailed explanation of some of the terms used in this template and further guidance on completing specific sections. The MTR team can adjust the structure to suit the specific needs of an MTR, as long as the report logic is maintained and the flow of information and analysis is coherent and clear. The section headings of this template should be maintained. To reduce repetition, the authors should use numbered paragraphs and make use of cross-references (hyperlinks) where possible. This page should be removed.

iv

Mid-term review of [Project Title, FAO symbol of GEF project and GEF ID number]

[Month Year]

v

Contents

List of boxes, figures and tables ................................................................................................. vi

Acknowledgements ................................................................................................................... vii

Formatting and presentation of the report ............................................................................... vii

Acronyms and abbreviations .................................................................................................... viii

Executive summary ..................................................................................................................... 9

1. Introduction ...................................................................................................................... 10

1.1. Purpose and scope of the MTR ...................................................................... 10

1.2. Objective of the MTR ..................................................................................... 10

1.3. Intended users ............................................................................................... 10

1.4. Methodology ................................................................................................. 11

1.5. Limitations ..................................................................................................... 11

2. Project background and context ....................................................................................... 12

3. Theory of change............................................................................................................... 12

4. Key findings and MTR questions ....................................................................................... 12

5. Conclusions and recommendations .................................................................................. 14

5.1. Conclusions .................................................................................................... 14

5.2. Recommendations ......................................................................................... 15

6. Lessons learned ................................................................................................................. 16

7. Appendices ....................................................................................................................... 17

Appendix 1. Terms of reference for the MTR .......................................................... 17

Appendix 2. MTR itinerary, including field missions (agenda) ................................ 17

Appendix 3. Stakeholders interviewed during the MTR .......................................... 17

Appendix 4. MTR matrix (review questions and sub-questions) ............................. 17

Appendix 5. List of documents consulted (“Reference list”) ................................... 17

Appendix 6. Results matrix showing achievements at mid-term and MTR observations 18

Appendix 7. Co-financing table ................................................................................ 19

Appendix 8. GEF evaluation criteria rating table and rating scheme ...................... 19

vi

List of boxes, figures and tables

Boxes Box 1: Box title ............................................................................................................................... 2 Figures Figure 1: Figure title ...................................................................................................................... 5 Tables Table 1: Table title ......................................................................................................................... 4

vii

Acknowledgements

[Insert your acknowledgements here. Name of MTR manager, team leader and team member(s) should be included here. See example below.] The MTR team comprised [X] independent international consultant(s), with [first name, surname] as lead consultant covering [give technical area of expertise, e.g. rural development and extension, fisheries and aquaculture, project evaluation], and [X] national consultant(s), including [first name, surname], who covered [give technical area of expertise, e.g. sustainable agriculture, forestry, fisheries and aquaculture, etc.]. The MTR was carried out with the invaluable assistance of the FAO staff of the [xxx] Country Office/Sub-regional or Regional Office. Their insight, knowledge, advice and comments made this MTR possible. The MTR team would like to thank all those who contributed to this MTR, led by [MTR Manager in the decentralized office] and supported by [XX] in the FAO GEF Coordination Office (FAO GEF CU). The MTR benefited from the inputs of many other stakeholders, including government officers, non-governmental organizations, local communities, research institutions, staff at other United Nations and international donor agencies, private-sector representatives and project beneficiaries. Their contributions were critical to the MTR team’s work and are deeply appreciated. MTR team - Name lead consultant(s) - Name of national consultant(s) FAO GEF CU staff - Name of FAO GEF CU staff focal point for the MTR

Formatting and presentation of the report

Please note the following formatting requirements:

• Paragraphs should be numbered

• Font: Segoe UI

• Size: 10.5

It is recommended that, where appropriate, tables, graphics, infographics, boxes, pictures and maps are used to provide a more interesting, accessible and effective report.

viii

Acronyms and abbreviations

Insert acronyms and abbreviations here. Check through the document and list all the acronyms and abbreviations that have been used. Do not include acronyms and abbreviations that are not in the report. As a general rule, try to avoid their overuse and only use them when: (a) the subject is commonly known by an acronym or abbreviation, for example, UNICEF or HIV/AIDS; and (b) acronymic key words are in frequent use, for example, “DRR programme” in a report on disaster risk reduction programmes. When an abbreviation is used for the first time in the text, it should be explained in full. It will be included in the list of acronyms if it is used repeatedly in a report. Possible acronyms and abbreviations to be mentioned in the report could include the following:

BH Budget holder

OED FAO Office of Evaluation

FAO Food and Agriculture Organization of the United Nations

FLO Funding liaison officer (FAO)

FPMIS Field Project Management Information System

LTO Lead technical officer

MR Management Response

MTR Mid-term review

PMU Project Management Unit

PTF Project Task Force

RM Mid-term review manager

SO FAO Strategic Objective

Mid-term review of [Name of GEF Project]

9

Executive summary

1. The executive summary should be approximately 10–15 percent of the length of the main report, excluding annexes. Paragraphs should be numbered.

2. The executive summary structure should be as follows:

Introduction

3. Provide key information on the purpose and use of the MTR, main MTR questions, scope and methodology, and any relevant dates, including field missions. Detailed information on the background of the project is not necessary in the executive summary, as it is presented in the MTR inception report and project document.

Main findings

4. The main findings of the MTR are presented here, grouped by MTR criterion and question (relevance, effectiveness, efficiency, sustainability, factors affecting performance and cross-cutting dimensions).

5. To facilitate reporting of the MTR through the GEF online portal, the executive summary (and main text) should include a rating on:

• progress towards achieving the project’s development objective(s) • overall progress on implementation • an overall risk rating

6. In addition, summary paragraphs (with headings and a suggested maximum of 200 words each) should be provided on:

• Progress, challenges and outcomes of stakeholder engagement (based on the description of the stakeholder engagement plan included at CEO endorsement/approval, if present), as well as environmental and social safeguards (ESS);

• Progress on gender-responsive measures, indicators and intermediate results as documented at CEO endorsement/approval in the gender action plan or equivalent (if present); and

• Knowledge activities/products (based on the knowledge-management approach approved at CEO endorsement/approval) and lessons learned (if available).

Conclusions

Illustrate conclusions in bold.

Recommendations List all recommendations in bold GEF rating table Include table here (see Appendix 1)


10

1. Introduction

The introductory section of the MTR report should outline the purpose, scope and objectives of the MTR and give details of the MTR process. It should explain the approach and methodology, including data-collection methods, making explicit the underlying assumptions, challenges, strengths and limitations of the methods and the approach employed during the MTR. Some of this information will have been provided in the pre-mission inception report. This section should not be more than 1-2 pages. 1.1. Purpose and scope of the MTR

The purpose of the MTR states why the MTR is being conducted and at this point in time (refer to the MTR terms of reference). This relates to the GEF and FAO requirements for mid-term reviews. The scope of the MTR states precisely what is being evaluated – this may be a project, a programme or a cluster of projects. It defines the time period covered by the MTR; the target population and beneficiaries (including any specific target groups, such as female-headed households or indigenous populations); the geographic area of focus; the target stakeholders; and which components of the project are to be addressed (if only a subset is to be addressed). 1.2. Objective of the MTR

The objective(s) of the MTR describe(s) what the MTR aims to achieve. This involves setting out the main MTR questions that need to be answered. Objectives summarize what the MTR needs to examine and questions indicate the information that needs to be gathered. The list of MTR questions (as set out in the MTR terms of reference) should be included here. Sub-questions related to each of the main MTR questions are shown in the MTR matrix (presented in Annex 9 of the MTR Guide). Box 1: Main review questions

Sample text for effectiveness – delivery of results • To what extent has the project delivered on its outputs, outcomes and objectives? • What broader results, if any, has the project had at regional and global level to date? • Were there any unintended results? • Is there any evidence of environmental stress reduction (for example, direct threats to

biodiversity) or environmental status change (such as improvement in the populations of target species), reflecting Global Environmental Benefits or any change in policy, legal or regulatory frameworks?

• To what extent can the achievement of results be attributed to the GEF-funded component?

1.3. Intended users

This section should identify the primary audience or users of the MTR, what they want to learn from the MTR, and why and how they are expected to use the results of the MTR. The section should also reference the stakeholder analysis and selection of interviewees identified in the inception report.


11

1.4. Methodology

This section should describe in detail the selected methodological approaches used in the MTR and the rationale for their selection, as well as any limitations faced by the MTR, such as time and budget constraints. This should help the users of the report judge the applicability and merits of the methods used to collect data and the credibility and reliability of the findings, conclusions and recommendations. Evidence collected during the MTR (for example, from documents and MTR interviews with stakeholders) should be analysed and findings presented by MTR question and sub-question. Each main MTR question should incorporate one or more GEF evaluation criteria. Cross-cutting dimensions, such as gender and equity, human rights, indigenous peoples and environmental and social safeguard issues, should also be addressed through the MTR questions. Please note that the questions need to be contextualized to the project’s theory of change. This section should include: • The overall methodological approach(es) used for the different MTR questions

• Sample and sampling frame – if a sample strategy was used, information should be provided on the sample size and characteristics, the process for selecting the sample (i.e. selection criteria, random or targeted) and the extent to which the sample is representative of the entire target population. The report should discuss the limitations of the sampling strategy in terms of ability to generalize results (for example, a limited number of interviews due to time and budget constraints, or an inability to access some groups due to their remote location).

• Data-collection methods and sources

- Data-collection methods, tools – the methods and tools used to collect data, along with evidence of the appropriateness, reliability and validity of these methods in relation to the MTR questions. Mention should be made of the use of the theory of change to analyse the structure and causal logic of the project and help identify MTR questions and potential stakeholders and interviewees.

- Data sources – the sources of information (documents reviewed and stakeholders consulted or interviewed) and how the information obtained addressed the MTR questions (given in the MTR matrix). It should also include a list of key informants interviewed, a list of potential stakeholders (individuals and institutions) and the rationale for their selection.

• Stakeholder engagement — the level of stakeholder engagement in the MTR (which is related

to their relationship to the project) and how their involvement contributed to the credibility of the MTR and its results. It should also consider how the key stakeholders were selected for the MTR, how their participation was ensured, their literacy levels, language skills and logistical constraints, etc.

• Composition of the MTR team – brief details on the main experience of the MTR team members relevant to the needs of the MTR.

1.5. Limitations

This section should identify and discuss the major limitations to the MTR, such as security issues, access to information and data, and the unavailability of key stakeholders, along with steps to mitigate those limitations.


12

2. Project background and context

This section should provide users with a brief (maximum 2–3 pages) and broad understanding of the project/programme, its objective(s) and expected results, and the development context in which it is being implemented. It should include the following information:

a. The name/title of the project being evaluated;

b. A description of the context:

• Context (region, country, landscape and, where relevant, challenges) • A summary of the threats and barriers being addressed by the project

c. A description of the project:

• When and how the project was initiated, project duration and start/end dates • GEF Project ID, GEF agency(ies), level of GEF financing, name of the executing entity(ies), GEF

focal areas • The project countries, geographic location, and target or pilot sites (with map, if possible) • The project’s strategy and expected results, including a brief description of the project objectives

and components, as listed in the project document • The groups and beneficiaries the project is aiming to reach (for the project as a whole and by

component) • The key partners involved in the project, including the implementing agencies and partners, and

other key stakeholders • Human and financial resources, including contributions from GEF and FAO, other donors and

funding, presented by component, both pledged and materialized co-financing (linked to the co-financing table in Appendix 2)

• Any significant political, socioeconomic and environmental changes since the beginning of project implementation and/or any other major external contributing factors that have affected it (both positively and negatively), or are likely to affect future project performance

• Any changes made to the project’s design, timeline or budget since GEF CEO endorsement and an explanation as to why these changes were made

• The project’s implementation status to date

3. Theory of change

This section presents the project’s causal logic and results chain: how the project activities and outputs link to project outcomes, leading to longer-term outcomes, intermediate states and the ultimate desired impacts. It should include a presentation and analysis of the theory of change of the project, as a graphic, with a narrative (2–3 pages). The theory of change (be it part of the original project design or drafted after the fact) reviewed or reconstructed by the MTR team for the inception report should have been discussed and updated with stakeholder feedback gathered during the MTR field missions, including feedback from interviews with the project team, RM and Project Task Force (PTF). More information on developing, reviewing and using a theory of change for a MTR can be found in Annex 10 of the MTR Guide.

4. Key findings and MTR questions

The MTR report should be internally consistent. The evidence presented needs to be complete and convincing. The main MTR questions should address issues related to the following six key MTR criteria and their sub-categories.


13

Relevance

- alignment with regional, sub-regional and national environmental and development priorities, as well as GEF strategic priorities

- FAO Strategic Objectives and higher goals (for example, the Sustainable Development Goals) - complementarity with existing interventions

Effectiveness

- achievement of project outputs; progress towards project outcomes - This should be summarised in the “progress-towards-results” table in Appendix 6 of this document, which presents achievements at the mid-term stage and the observations of the MTR team for each output and outcome (more information on completing this table can be found in Annex 12 of the MTR Guide)

- degree of attainment of project objectives and higher-level results, including an assessment of the likelihood of longer-term impacts (using, for instance, a “review of outcomes to impacts” analysis approach)

Efficiency

- cost-effectiveness of the project - timeliness of activities

Sustainability

- socio-political, financial, institutional and governance, and environmental risks to sustainability - any evidence of replication or catalysis of project results

Factors affecting performance

- project design and readiness - quality of project execution and management arrangements (including assessment of risks) - project oversight by FAO as the GEF Agency and national partners - financial management and co-financing - project partnerships and stakeholder engagement - communication, visibility, knowledge management and knowledge products - monitoring and evaluation (M&E), including M&E design, implementation and budget

Cross-cutting dimensions

- equity issues related to inclusion of vulnerable and indigenous groups, including gender analysis, human rights, disadvantaged/vulnerable/disabled peoples and decent labour (as relevant)

- environmental and social standards (including stakeholder engagement; further guidance available in Annex 12 of the MTR Guide)

As part of the process to assess effectiveness, the MTR team should compare and analyse the GEF tracking tool at the baseline with the one completed by the Project Management Unit (PMU) just before the MTR for GEF-5 projects and for GEF-6 and GEF-7 projects with GEF Biodiversity Focal Area Objective 2 with protected-area elements. For all GEF-6 and GEF-7 projects, the MTR team should also assess the degree to which relevant GEF-7 core indicators have been achieved. These should have been identified by the project team, with the support of the funding liaison office (FLO) or lead technical officer (LTO), with baselines retrofitted prior to the MTR if they did not exist (now required under the updated Results Architecture for GEF-7).1

1 GEF/C.54/11/Rev.01, Updated Results Architecture for GEF-7. June 4, 2018. For projects approved in GEF-6 that have not yet been completed, a shift to core indicators and sub- indicators is required at the next available opportunity in the project cycle (CEO Endorsement/ Approval, mid-term or completion).


14

For each of the six MTR questions, findings should be developed as follows (sample text):

MTR question 2 – Effectiveness Finding 1. The project has produced some good outputs as of the MTR point, particularly in relation to Component 2. These include the completion of community seed banks and the development and capacity building for selling traditional agrobiodiversity products. Overall, however, the delivery of the expected outputs has been mixed, with some deliverables behind schedule (for example, some of the policy work under Component 1).

Finding 2, etc.

5. Conclusions and recommendations

5.1. Conclusions

The MTR report should include a section setting out the conclusions of the MTR, based on its findings. The conclusions should respond to each of the key MTR questions per the terms of reference. They should consolidate the assessment of various aspects to judge the value of the project on specific criteria, such as relevance or effectiveness. Conclusions should highlight the main strengths and weaknesses of the project, provide insights into the identification of and/or solutions to important problems or issues pertinent to project stakeholders, including FAO and GEF. Conclusions should start with the positive achievements and a short explanation of how these were achieved, then move to the less successful aspects of the project with explanations as to why they occurred. Conclusions should be well substantiated by the evidence gathered by the MTR, clearly connected to the findings presented in the MTR report and provide a clear basis for the recommendations that follow. The conclusions section should provide a statement on: (a) progress towards achievement of the project’s development objective(s); (b) overall progress on implementation; (c) an overall risk rating for the project; and (d) a statement on the MTR’s overall assessment of the project. An additional section with general conclusions (not linked to a specific MTR question) may be added if these emerged from the MTR. Conclusions should be numbered and highlighted in bold and cross-referenced to the main text of the report (referring to specific paragraph numbers or through hyperlinks), as in the following example.

Conclusion 1 – Effectiveness (sample text). The project has made some important contributions to strengthening policy that should support agrobiodiversity conservation , notably, its input to an amendment to the Seed Act. However, progress on most of the other policy-development and mainstreaming work has been slow, in part because the Project Management Coordination Unit has lacked sufficient capacity in this area. While the appointment of two part-time policy consultants should speed up delivery, it is clear that the project’s policy development work is unlikely to achieve current targets or many meaningful results in the short period of the project that remains (11–12 months), as policy-related work generally requires significant time to achieve results. In the MTR’s views, a short extension to the project’s official end date is necessary for additional technical support, awareness-raising and capacity building to allow concrete results to be delivered in this regard.

Conclusion 2, etc.


15

5.2. Recommendations

The MTR report should provide recommendations (actionable proposals) directed at FAO, the PMU, project partners and other relevant stakeholders, in particular, those directly involved in project execution, on specific actions to be taken by identified individuals/position holders to resolve concrete problems affecting the project’s execution and performance and to enhance the delivery, sustainability and eventual impact of project results. Recommendations at the mid-term point should identify actions that can be taken within the available time and resources (including local capacities) of the project to ensure maximum delivery of the planned results by the end of the project. Recommendations should explain the issues that need to be addressed and may include:

• measures to improve the project design and framework, management and administration, partnership arrangements, M&E of the project – such as the need for a revised set of SMART (specific, measurable, achievable, realistic and timebound) indicators for the project’s results framework – and other operational issues related to factors that have affected performance;

• more strategic issues to strengthen the project’s relevance, effectiveness and efficiency and to reinforce the initial benefits of the project, as well as additional actions to improve the likelihood of impact or mitigate risks to sustainability; and

• proposals for future actions to further the project’s objectives and/or scale up results (to produce wider impact).

Each recommendation should start with the rationale for its consideration and be anchored in the findings and conclusions of the report, with paragraph cross-referencing where possible. They should be followed by suggested actions to implement the recommendations. It should be noted that not all conclusions will lead to a recommendation and not every GEF criterion will necessarily need a recommendation, while some may need many. Recommendations should be succinct and SMART and tackle one set of issues at a time. This is especially important when different levels of decision-making and/or responsibility for action are involved. Recommendations should be practical and focused on the most critical interventions. Importantly, they should clearly identify who should be responsible for their implementation. A suggested timeframe for the implementation of each recommendation should be given. Each recommendation should present measurable performance targets, so that the BH, FAO GEF CU, FLO and PMU can monitor and assess compliance with their implementation. In some cases, it may be useful to propose options and briefly analyse the pros and cons of each one. It is recommended that the MTR team use the table below (Table A11.1) to capture and summarise the MTR recommendations. Table A11.1 Recommendations table

Rec. no. Rationale for recommendation Recommendation Responsibility

Timing/dates for actions

Strategic relevance

A.1 Recommendation: A.2 Recommendation: Etc.

Effectiveness

B.1 Recommendation: B.2 Recommendation: Etc.


16

The MTR team should focus on identifying recommendations for the most critical issues that will, in its view, contribute most to improving the operation, delivery, results, sustainability and possible impact of the project over its remaining timeframe and beyond. Although there is no correct number, in general, MTR reports should include no more that 8–10 recommendations. However, the associated explanatory text on each can be expanded to include “sub-recommendations” in cases where recommendations are complex (for example, involving many stakeholders).

Although not mandatory, it is strongly recommended that the MTR team discuss prioritization, timing and programming of draft recommendations with the PMU, BH, FLO, LTO, government counterparts and other relevant FAO staff and stakeholders as part of the post-mission MTR briefings. This will support the development of more tailored, useful and implementable recommendations and help speed up the report drafting and reviewing process, and aid the future Management Response. Where possible, the draft findings, conclusions and recommendations of the review should be presented to the Project Steering Committee for discussion and endorsement.

6. Lessons learned

“Lessons learned” are the knowledge generated from an analysis of real project results and experiences, which have the potential to improve future actions and have wider value and potential for broader application, replication and use. They are based on a review of successes and good practices, or are derived from an analysis of challenges encountered and mistakes made which should be avoided in the future. Like recommendations, lessons learned should be anchored in the findings and conclusions of the MTR, with cross-referencing to appropriate paragraphs in the MTR report wherever possible. Lessons learned should briefly describe the context from which they are derived and the contexts in which they may be useful. Lessons and good practices can be identified for substantive, methodological or procedural issues, which may be relevant to the design, implementation (management, partnerships, M&E, etc.) of similar projects and programmes. Specific lessons on how human rights, gender and other equity concerns have been successfully integrated into project delivery and/or how they could have could have been taken into consideration should be highlighted. As the project is only partly complete at the mid-term, MTRs may not generate many (if any) lessons; it may be simply too early in the project’s implementation.

Efficiency

C.1 Recommendation: C.2 Recommendation: Etc.

Sustainability and catalysis/replication

D.1 Recommendation: D.2 Recommendation: Etc.

Factors affecting performance

E.1 Recommendation: E.2 Recommendation: Etc.

Cross-cutting dimensions

F.1 Recommendation: F.2 Recommendation:

Etc. Recommendation:


17

7. Appendices

The appendices should include any material that is essential to understanding the main report and must be directly referenced in the main report.

Appendix 1. Terms of reference for the MTR

Appendix 2. MTR itinerary, including field missions (agenda)

Appendix 3. Stakeholders interviewed during the MTR

First name Last name Position Organization/location

1

2

3

Appendix 4. MTR matrix (review questions and sub-questions)

Appendix 5. List of documents consulted (“Reference list”)

The list of documents consulted (“References”) should include published, unpublished “grey literature” and internal project-related documents. Published documents should be written following the standard conventions for peer-reviewed journals, per the FAO style guide (FAO, 2017b). Examples are given below. Brunelli, C. & Vivani, S. 2014. Exploring gender-based disparities with the FAO Food Insecurity Experience Scale. New York, United Nations Statistics Division (UNSTATS). (Also available at https://unstats.un.org/unsd/gender/mexico_nov2014/Session%201%20FAO%20paper.pdf). Castell, G. et al. 2015. Escalas de evaluación de la inseguridad alimentaria en el hogar. Madrid, Revista Española de Nutrición Comunitaria. (Also available at http://www.renc.es/imagenes/auxiliar/files/RENC2015supl1INSEGURALIMENT.pdf CFS. 2012. Global Strategic Framework for Food Security and Nutrition. First Version. Rome, FAO. (http://www.fao.org/docrep/meeting/026/ME498E.pdf). FAO. 2006. Policy Brief: Food Security. Rome. (Also available at http://www.fao.org/forestry/13128-0e6f36f27e0091055bec28ebe830f46b3.pdf). FAO. 2008. An introduction to the Basic Concepts of Food Security. Rome. (Also available at https://unstats.un.org/sdgs/indicators/database/?area=PRY).

https://unstats.un.org/unsd/gender/mexico_nov2014/Session%201%20FAO%20paper.pdf

http://www.renc.es/imagenes/auxiliar/files/RENC2015supl1INSEGURALIMENT.pdf

http://www.fao.org/docrep/meeting/026/ME498E.pdf

http://www.fao.org/forestry/13128-0e6f36f27e0091055bec28ebe830f46b3.pdf

http://www.fao.org/forestry/13128-0e6f36f27e0091055bec28ebe830f46b3.pdf

https://unstats.un.org/sdgs/indicators/database/?area=PRY


18

Appendix 6. Results matrix showing achievements at mid-term and MTR observations

As part of the assessment of the delivery of project outcomes and outputs, the MTR should assess progress made towards the mid-term project targets using the matrix in Table A11.2. Where mid-term targets are not given in the project logframe, assessment can be made against end-of-project targets. Assessment of progress should be colour-coded using a “traffic-light system”, with a rating assigned to progress on each outcome (but not outputs) using the standard GEF six-point rating scale. Recommendations should be made for those areas marked as “not on target to be achieved” (red).

Table A11.2 Progress-towards-results matrix showing the degree of achievement of project outcomes and outputs (against mid-term targets)*

Project strategy

Indicator Baseline level

Level at first PIR

(self- reported)

Mid-term

target2

End-of- project target

Mid-term level & assessment

(colour-coded red, yellow or green)

Achievement rating3

Justification for rating

Objective Indicator 1 Indicator 2 Etc.

Outcomes

Outcome 1 Indicator 1 Indicator 2 Etc.

Outcome 2 Indicator 1 Indicator 2 Etc

Etc.

Etc.

Outputs4

Output 1.1 Indicator 1 Indicator 2 Etc.

Not applicable Not applicable



Output 2.1 Indicator 1 Indicator 2 Etc

Etc.




Etc

Indicator assessment key

Green = Achieved Yellow = On target to be achieved Red = Not on target to be achieved

* As presented in the results framework in the original project document or subsequently updated by the Project Steering Committee (PSC) at project inception

2 If available 3 Use the six-point progress-towards-results rating scale: HS, S, MS, MU, U, HU 4 Outputs may not have indicators but should have targets and milestones e.g. two training workshops delivered by end of year 2.


19

Appendix 7. Co-financing table

Sources of

co-

financing5

Name

of co-

financer

Type of

co-

financing6

Amount

confirmed at CEO

endorsement/

approval7

Actual amount

materialized as of

(date of MTR)

Actual amount

materialized at mid-

term or closure

(confirmed by the

review/evaluation

team)

Expected

total

disbursement

by the end of

the project

Cash In kind Cash In kind

TOTAL

Appendix 8. GEF evaluation criteria rating table and rating scheme

GEF evaluation criteria rating table The MTR team is required to rate the aforementioned MTR criteria for the purposes of reporting to GEF and FAO on progress to date. Ratings need to be well substantiated, based on evidence gathered from the MTR, and a summary description of this evidence should be presented in the MTR Ratings & Achievements Summary Table (Table A11.3). The MTR team should compare their (independently derived) ratings with those of the most recent GEF project implementation review (PIR) and describe any significant discrepancies. Most criteria will be rated on a six-point scale, as follows: highly satisfactory (HS); satisfactory (S); moderately satisfactory (MS); moderately unsatisfactory (MU); unsatisfactory (U); highly unsatisfactory (HU). Sustainability and the likelihood of impact are rated from likely (L) down to highly unlikely (HU). Explanations as to how to rate the criteria of effectiveness, sustainability and factors affecting performance can be found in the corresponding sections in Table A11.3.

Table A11.3 MTR ratings and achievements summary table

GEF criteria/sub-criteria Rating8 Summary comments9

A. STRATEGIC RELEVANCE

A1. Overall strategic relevance HS→HU

A1.1. Alignment with GEF and FAO strategic priorities HS→HU

A1.2. Relevance to national, regional and global priorities and beneficiary needs HS→HU

A1.3. Complementarity with existing interventions HS→HU

5 Sources of Co-financing may include: Bilateral Aid Agency(ies), Foundation, GEF Agency, Local Government, National Government, Civil Society Organization, Other Multi-lateral Agency(ies), Private Sector, Beneficiaries, Other. 6 Grants, loans, equity participation by beneficiaries (individuals) in the form of cash, guarantees, in kind or material contributions and other (please explain). 7 The type of co-financing whether cash or in-kind should be indicated separately 8 See rating scheme at the end of the document. 9 Include reference to the relevant sections in the report.


20

B. EFFECTIVENESS

B1. Overall assessment of project results HS→HU

B1.1 Delivery of project outputs HS→HU

B1.2 Progress towards outcomes10 and project objectives HS→HU

- Outcome 1 HS→HU

- Outcome 2 HS→HU

- Etc. HS→HU

- Overall rating of progress towards achieving objectives/ outcomes HS→HU

B1.3 Likelihood of impact Not rated at MTR

C. EFFICIENCY

C1. Efficiency11 HS→HU

D. SUSTAINABILITY OF PROJECT OUTCOMES

D1. Overall likelihood of risks to sustainability L→HU

D1.1. Financial risks L→HU

D1.2. Sociopolitical risks L→HU

D1.3. Institutional and governance risks L→HU

D1.4. Environmental risks L→HU

D2. Catalysis and replication HS→HU

E. FACTORS AFFECTING PERFORMANCE

E1. Project design and readiness12 HS→HU

E2. Quality of project implementation HS→HU

E2.1 Quality of project implementation by FAO (BH, LTO, PTF, etc.) HS→HU

E2.1 Project oversight (PSC, project working group, etc.) HS→HU

E3. Quality of project execution HS→HU

E3.1 Project execution and management (PMU and executing partner performance, administration, staffing, etc.)

HS→HU

E4. Financial management and co-financing HS→HU

E5. Project partnerships and stakeholder engagement HS→HU

E6. Communication, knowledge management and knowledge products HS→HU

E7. Overall quality of M&E HS→HU

E7.1 M&E design HS→HU

E7.2 M&E plan implementation (including financial and human resources) HS→HU

E8. Overall assessment of factors affecting performance HS→HU

F. CROSS-CUTTING CONCERNS

F1. Gender and other equity dimensions HS→HU

F2. Human rights issues HS→HU

F2. Environmental and social safeguards HS→HU

Overall project rating HS→HU

10 Assessment and ratings by individual outcomes may be undertaken if there is added value. 11 Includes cost efficiency and timeliness. 12 This refers to factors affecting the project’s ability to start as expected, such as the presence of sufficient capacity among executing partners at project launch.


21

Rating scheme Additional explanation on how to assess ratings for specific criteria, for example, whether they are highly satisfactory or moderately satisfactory, can be found in Tables A11.4 to A11.7.13 Overall outcome ratings MTRs should use mid-term targets per the project’s logframe to assess outcome delivery. If no mid-term indicator targets are available, the MTR should base outcome ratings on an assessment of the delivery of results to date against milestones in workplans and delivery compared with end-of-project targets. Table A11.4 How to assess ratings for specific criteria

Rating Description

Highly satisfactory (HS) Level of outcomes achieved clearly exceeds expectations and/or there were no shortcomings

Satisfactory (S) Level of outcomes achieved was as expected and/or there were no or minor shortcomings

Moderately satisfactory (MS) Level of outcomes achieved more or less as expected and/or there were moderate shortcomings

Moderately unsatisfactory (MU) Level of outcomes achieved somewhat lower than expected and/or there were significant shortcomings

Unsatisfactory (U) Level of outcomes achieved substantially lower than expected and/or there were major shortcomings

Highly unsatisfactory (HU) Only a negligible level of outcomes achieved and/or there were severe shortcomings

Unable to assess (UA) The available information does not allow an assessment of the level of outcome achievements

Source: GEF (2017c)

In line with similar guidance on the assessment of ratings for GEF terminal evaluations (GEF, 2017c),

the overall rating of the outcomes of the project should be based on performance on the criteria of relevance, effectiveness and efficiency. The calculation of the overall outcome rating will consider all three criteria, of which relevance and effectiveness are critical. The relevance rating will determine whether the overall outcome rating is in the unsatisfactory range (MU to HU = unsatisfactory range). If the relevance rating is unsatisfactory, the overall outcome will be unsatisfactory as well. However, where the relevance rating is satisfactory (HS to MS), the overall outcome rating could, depending on its effectiveness and efficiency rating, be either satisfactory or unsatisfactory.

Table A11.5 Factors affecting performance (assess each element separately; M&E is treated differently)

Rating Description

Highly satisfactory (HS) There were no shortcomings and quality of design and readiness/project implementation/project execution/co-financing/partnerships and stakeholder engagement/communication and knowledge management and results exceeded expectations.

Satisfactory (S) There were no or minor shortcomings and quality of design and readiness/project implementation/project execution/co-financing/partnerships and stakeholder engagement/communication and knowledge management and results meet expectations.

Moderately satisfactory (MS)

There were some shortcomings and quality of design and readiness/project implementation/project execution/co-financing/partnerships and stakeholder engagement/communication and knowledge management and results more or less meet expectations.

13 See further information on GEF rating scales in Annex 2: Rating scales in GEF (2017c).


22

Moderately unsatisfactory (MU)

There were significant shortcomings and quality of design and readiness/project implementation/project execution/co-financing/partnerships and stakeholder engagement/communication and knowledge management and results were somewhat lower than expected.

Unsatisfactory (U) There were major shortcomings and quality of design and readiness/project implementation/project execution/co-financing/partnerships and stakeholder engagement/communication and knowledge management and results were substantially lower than expected.

Highly unsatisfactory (HU)

There were severe shortcomings in quality of design and readiness/project implementation/project execution/co-financing/partnerships and stakeholder engagement/communication and knowledge management.

Unable to assess (UA) The available information does not allow an assessment of the quality of design and readiness/project implementation/project execution/co-financing/partnerships and stakeholder engagement/communication and knowledge management.

Table A11.6 Monitoring and evaluation design or implementation ratings (Overall M&E design, design and implementation assessed separately)

Rating Description

Highly satisfactory (HS) There were no shortcomings and quality of M&E design or M&E implementation exceeded expectations.

Satisfactory (S) There were no or minor shortcomings and quality of M&E design or M&E implementation meets expectations.

Moderately satisfactory (MS)

There were some shortcomings and quality of M&E design or M&E implementation more or less meets expectations.

Moderately unsatisfactory (MU)

There were significant shortcomings and quality of M&E design or M&E implementation somewhat lower than expected.

Unsatisfactory (U) There were major shortcomings and quality of M&E design or M&E implementation substantially lower than expected.

Highly unsatisfactory (HU)

There were severe shortcomings in M&E design or M&E implementation.

Unable to assess (UA) The available information does not allow an assessment of the quality of M&E design or M&E implementation.

Table A11.7 Sustainability

Rating Description

Likely (L) There is little or no risk to sustainability.

Moderately likely (ML) There are moderate risks to sustainability.

Moderately unlikely (MU)

There are significant risks to sustainability.

Unlikely (U) There are severe risks to sustainability. Unable to assess (UA) Unable to assess the expected incidence and magnitude of risks to sustainability.

1

Annex 12 – Supplementary notes on drafting of the MTR report This annex provides supplementary guidance to the notes in the MTR report template in Annex 11.

1. Key findings The findings should be presented in relation to the following six GEF criteria, which mostly follow the standard Organisation for Economic Co-operation and Development (OECD)/Development Assistance Committee (DAC) evaluation criteria (OECD/DAC, 2002)1 outlined in the MTR terms-of-reference template (Annex 4): (a) relevance (b) effectiveness, (c) efficiency, (d) sustainability of project results, (e) factors affecting performance (including project implementation and execution) and (f) cross-cutting issues (including environmental and social safeguards, gender and equity issues). The various elements in the “factors affecting performance” section are often included in analysis of the project’s relevance, effectiveness and efficiency, but are separated out and highlighted as a specific category in the FAO–GEF MTR report, as they cover many issues that generally need specific attention at the mid-term review point of a GEF project. Cross-cutting issues are also treated separately to highlight any gender and equity concerns or issues related to environmental and social safeguards, including the degree of stakeholder engagement and the implementation of any stakeholder engagement or environmental and social management plan (ESMP). Both GEF and FAO are paying increased attention to these aspects in project design and implementation and both require topic-specific reporting under GEF-7. Guidance on how to interpret and report on the specific criteria can be found in the following sections.

2. Relevance Relevance is generally defined as the extent to which an intervention is in line with the priorities and policies of the target group, recipient and donor (in this case, GEF and FAO). The MTR report should include a brief assessment of the project’s relevance at formulation. Specific areas for assessment should include:

• whether a project is in line with specific GEF focal areas (biodiversity, land degradation, climate change, international waters, chemicals and waste) strategic priorities and operational programme(s);

• its relevance to FAO’s mandate and its alignment with FAO’s policies and strategies; and • the extent to which the intervention responds to the stated environmental and development

concerns, needs and priorities of the target countries, sub-regions or regions (for example, national or sub-national development plans, poverty-reduction strategies, national biodiversity strategies and actions plans, national adaptation plans) and the target beneficiaries.

Complementarity with existing interventions that address similar needs among the same target groups should also be considered in this section. The MTR report should also examine whether the project still addresses the above priorities, remains country-driven and continues to tackle the barriers to change the project seeks to address, as identified in the project document. It should explore the extent to which these barriers have changed since the project was designed and their relevance. The key questions are: “Is the project still relevant to the target country(ies), beneficiaries and donor(s)?” and “Is it still addressing the right barriers to change?” If not, the MTR report should identify additional issues that need to be tackled to ensure that the project will achieve

1 OECD/DAC (2002) definition of evaluation: “The systematic and objective assessment of an on-going or completed project, programme or policy its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors.”

2

its long-term aims. Analysis of its causal logic, as part of the process of analysing the project’s theory of change (see Annex 10 of MTR Guide), should help to determine whether there are additional barriers to achieving project impact and whether the original logic of the project holds true.

3. Effectiveness – progress towards results The effectiveness of the project should be assessed along three dimensions: (a) the achievement of outputs, (b) the achievement of direct project outcomes and objectives and (c) the likelihood of long-term changes and impacts. Where possible, data and analysis should be disaggregated by gender and stakeholder group to allow the development of more nuanced findings and conclusions. 3.1. Achievement of outputs The MTR should assess, for each component, the project’s success in producing the planned outputs (products and services delivered) using the table shown in Appendix 6 of the MTR report template (see Annex 11 of the MTR Guide) and the achievement of the milestones as presented in the project’s logical framework (logframe) and formal workplan. The MTR report’s narrative should assess both the quantity and quality of the outputs, as well as their usefulness. The MTR should briefly explain the reasons for the success in delivering the project’s various outputs and meeting quality standards, or its failure to do so. This should be cross-referenced with more detailed explanations of the causes, per the section on “factors affecting performance” (see section 6 of this Annex). The GEF has simplified its results architecture for the GEF-7 period (July 2018 to June 2022) with the requirement for projects to report on the delivery of “core indicator” targets set at the project design stage. The GEF also requires this new architecture to be applied to GEF-6 projects (July 2014 to June 2018). However, all GEF-5 (July 2010 to June 2014) projects should continue to use and submit the changes in scores in GEF tracking tools. This information is then reported in this section of the MTR report. However, for GEF-5, GEF-6 and GEF-7 projects with protected-area elements, the MTR team should also compare the GEF management effectiveness tracking tool (METT) compiled for the MTR, with the tracking tool submitted to GEF for CEO endorsement, and discuss any changes with the project management unit (PMU) during the MTR mission. The MTR should check that the entries are filled in correctly and discuss the appropriateness of the tracking-tool criteria and any difficulties in completing them with the project team. Any trends should be identified and analysed and documented in the MTR report. FAO–GEF projects approved during the GEF-6 period (July 2014 to June 2018) are required to shift to GEF-7 core indicators and sub-indicators targets at the next available opportunity in the project cycle (the MTR or final evaluation) and are no longer required to submit tracking tools2 unless, as mentioned, they have a protected-area element under Biodiversity (BD) Focal Area Objective 2. Therefore, project teams (with the support of the lead technical officer, or LTO) for FAO–GEF projects approved during GEF-6 (and GEF-7) are required to identify and agree appropriate core indicators/sub-indicators. For GEF-6 projects, this means that the baselines for the selected GEF-7 indicators will also need to be reconstructed, which is likely to require time, resources and discussion. Consequently, for these projects, the process of integrating the GEF-7 core indicators and associated targets into the project’s monitoring and evaluation (M&E) system needs to begin in the early planning stage of the MTR (Phase 1) and should not be left until the MTR team starts to collect data. 3.2. Progress towards outcomes The MTR should assess the extent to which the project’s stated outcomes and objectives (as shown in the project’s logframe) were effectively achieved, or are expected to be achieved by the end of the project. The key question here is, “To what extent have the expected outcomes and objectives of the project been achieved to date?” The overall project objective is frequently a higher-level result in the project’s causal logic, which the project alone cannot achieve, whereas the project outcomes should be largely or completely

2 GEF/C.54/11/Rev.01 June 4, 2018. Updated Results Architecture for GEF-7.

3

achievable). Therefore, the MTR report should describe the actual or likely contribution of the project to its objective. MTRs should assess progress against the mid-term targets set out in the project’s logframe, based on the findings of the MTR mission(s) and interviews with the project stakeholders, supplemented by data from annual Project Implementation Reviews (PIRs) and the GEF tracking tools or GEF-7 core indicators. Where mid-term targets are lacking (for example, where they were not identified at the project formulation stage or added at project inception), the MTR team should assess progress against the end-of-project targets (which all projects should show) and compare the delivery of results to date with the milestones in workplans. Team members should use their professional judgment based on experience of similar projects. However, it is acknowledged that assessments without mid-term targets are likely to be more speculative. Importantly, all GEF projects and programmes should report on the GEF’s “additionality” – defined as the additional effects (environmental and otherwise) that can be directly associated with a GEF-supported project or programme – using the evaluative approach provided by the GEF Independent Evaluation Office (IEO) (GEF IEO, 2018). Along with the above narrative text, the MTR report should summarize the progress towards the mid-term targets for each outcome and the project objective and where these targets are included in the logframe (not all projects give objective indicators and targets although they should do). This should be captured in the “Progress-towards-results matrix showing the degree of achievement of project outcomes and outputs” table included in Annex 11 – the MTR report template – of the MTR Guide (see Table A11.2). The “baseline level”, “mid-term target” and “end-of-project target” columns should be populated with information from the results framework, inception report, PIRs and project document. Using that data, the MTR team should complete the column “mid-term level & assessment” (colour code this column only) and conclude whether the mid-term targets: (a) have already been achieved (colour the “mid-term level & assessment” table cell green); (b) have been partially achieved or is on target to be achieved by the end of the project (colour yellow); or (c) are at high risk of not being achieved by the end of the project and needs attention (colour red). The MTR team should review the indicator-level progress and ratings for the outcomes (outputs are not rated in the PIRs) presented in the most recent PIR and any deviations from the results should be noted and explained in the “justification for rating” column. The “achievement rating” column should be completed by assigning ratings to the project objective and outcomes (but, again, not the outputs), based on achievement of the mid-term targets under the relevant indicators. To assess the overall achievement of project outcomes, the standard six-point rating scale (highly satisfactory to highly unsatisfactory) should be used. The MTR should report on the extent to which the original indicators and associated targets were achieved, even where they were deemed poorly chosen or formulated at the design stage. If needed, the MTR should make recommendations to strengthen or replace indicators and targets, and this should be discussed in the project design section of the MTR report (see section 6.1 of this Annex for more). The “justification for rating” column should be completed with a brief explanation as to why each rating was assigned, by comparing the “mid-term level & assessment” column with the “mid-term target” (or “end-of-project target” if no mid-term targets are given) column, using the criteria in the appropriate rating scale. For those indicators marked “not on target to be achieved” (red), the MTR report may recommend actions to be taken, which should be summarized in the report’s recommendations table. The MTR team should discuss the project’s theory of change (from the inception report) with stakeholders during the MTR missions and/or interviews to validate its outcomes, intermediate states, causal pathways, impact drivers and assumptions. However, formal reporting on any revised project outputs or outcomes recommended by the MTR should begin after their endorsement by the PSC, budget holder (BH) and FAO–GEF Coordination Unit (FAO GEF CU). It is recognized that at the MTR stage, any suggested new or reformulated outcomes will lack indicators and targets and these will need to be retrofitted (with appropriate

4

baselines) following the Management Response to the MTR report and acceptance and endorsement of the revised project framework by the PSC. Where possible, the MTR should suggest revised indicators and targets for consideration by the project team and FAO. Particular attention should be paid to assess a project’s capacity to deliver and sustain results. The GEF places strategic emphasis on enhancing a country’s capacity to achieve more impactful results (GEF, 2019d). Similarly, FAO considers system-wide capacity development as essential to achieving more sustainable, country-driven and transformational results at scale (FAO, 2018). FAO’s corporate and GEF-contextualized capacity-development approach is fully aligned with, and addresses the project justification requirement of the FAO–GEF project document, on “innovativeness, potential for scaling and sustainability through system-wide capacity development”. The MTR should focus on assessing the project’s capacity-building efforts to strengthen (1) individual capacities (such as knowledge, skills and competencies), (2) organizational and institutional capacities (for example, the performance of organizations, cross-sectoral, multi-stakeholder, vertical and horizontal coordination and collaboration mechanisms at national and sub-national levels) and (3) the enabling environment (such as the formulation and implementation of regulatory and policy frameworks, institutional political economy and enhanced political commitment and will). Specific questions may be formulated based on the OED Capacity Development Evaluation Framework recently published by the FAO Office of Evaluation (FAO OED, 2019). 3.3 Likelihood of impact Few GEF projects are likely to achieve immediate sweeping change and significant impact by the mid-term stage; this may take many years. Limited data and resources often constrain the ability of an MTR team to measure any impact achieved by a project. However, the MTR should assess the likelihood that the project will make a substantive contribution to the higher level, longer-term intended changes and impacts presented in the revised/reconstructed theory of change. In summary, this involves the following steps: • assessment of the internal logic of the project. Are outputs logically connected (from cause to effect)

to intended outcomes and are intended outcomes logically connected to the project’s desired impact? Have all essential outputs and outcomes been taken into account in the project design? For this assessment, it is often useful to compare the formal logframe of the intervention with the reconstructed theory of change.

• assessment of effectiveness. To what extent have the outcomes presented in the reconstructed theory of change been achieved?

• verification of drivers and assumptions. Have all drivers and critical assumptions been identified and adequately considered? Has the project taken into account whether the assumptions and drivers identified in the reconstructed theory of change have held/will hold and whether there are any likely unintended positive and negative effects? Has the project made all possible efforts to engage potential drivers and made the necessary adjustments in case certain critical assumptions prove invalid?

• assessment of the likelihood of impact. Based on the previous steps, the MTR team should be able to identify how likely the project is to contribute to the desired long-term impact and what that contribution will be. If the internal logic of the project is strong, outcomes have been achieved and all drivers and assumptions are in place, it can be said that the intervention will contribute to the desired impact (the Global Environmental Benefits) (GEF, 2019c). On the other hand, if there are flaws in the internal logic of the project (for instance, if some outputs do not properly link to outcomes), if certain key outcomes have not been achieved or certain drivers or assumptions are not in place, the likelihood will be lower that the project will contribute to impact.

The GEF IEO has produced guidance on impact assessment, which promotes a “review of outcomes to impacts” (ROtI) approach to assess the likelihood of longer-term outcomes and impact (GEF IEO, 2009).

5

4. Efficiency of project implementation Efficiency refers to the cost-effectiveness and timeliness of project execution. Cost-effectiveness is the extent to which an intervention has achieved, or is expected to achieve, its results (in terms of outputs) at the lowest possible cost; timeliness refers to whether planned activities were delivered in the expected timeframes and in line with workplans, and whether events were well organized and efficiently sequenced. The key questions are, “To what extent has the project been implemented efficiently and cost effectively?” and “Has management been able to adapt to any changes in conditions and improve the efficiency of project implementation?” The MTR report should briefly describe project efforts to make use of or build upon existing institutions, agreements, partnerships, data sources, synergies and complementarities with other initiatives, programmes and projects that have helped to increase project efficiency, as well as any cost- or time-saving measures put in place to maximize results. The MTR team should examine what, if any, activities have been achieved for little effort or cost and, conversely, identify those that have been particularly wasteful, for example, where duplication of effort has occurred. The MTR should identify any negative impacts on project execution, costs and effectiveness caused by project delays or extensions and suggest ways to mitigate these. Where possible, the MTR should consider whether the project was implemented in the most efficient way compared with similar interventions or approaches. The discussion of specific risks to efficiency should be cross-referenced with a more detailed presentation of factors affecting performance in Section 6.

5. Sustainability Sustainability can be defined as the probability of project-derived results and impacts continuing in the long term after the end of the intervention and once GEF financing has finished. The MTR should identify and assess the key conditions or factors that are likely to contribute to or undermine the continuation of project outcomes. This requires an examination of the risks to continued results. Risk assessment and management are often a challenge for project management teams. MTR interviews and discussions with the PMU and project partners present an opportunity to highlight and strengthen risk management in project planning and execution at the mid-term point if it has been weak or lacking until then. The key question here is, “To what extent are there financial, institutional and governance, sociopolitical and/or environmental risks to sustaining project results in the long-term?” Some of the factors affecting sustainability may stem directly from the project, while others will relate to specific circumstances or developments that are beyond the project’s control, for example, the political environment. The MTR should assess whether the project has developed an appropriate exit strategy detailing how project results will be sustained and enhanced over time and present measures to mitigate risks to sustainability once the project ends. It is particularly important to assess the extent to which the project has achieved stakeholder (including government) consensus on courses of action after the project ends. At the mid-term point, sustainability issues may not have received close attention, particularly if there has been a slow start or delays to project implementation. Nevertheless, the MTR should document project activities relating to sustainability and highlight activities needed to improve the likelihood of sustainability over the remainder of the project. Again, the discussion of specific risks to sustainability should be cross-referenced with a more detailed presentation of factors affecting performance in Section 6. The MTR team should validate whether the risks identified in the project document, FAO project performance reviews (PPR) and GEF PIR are still valid. Most importantly, they should address whether the risk ratings used are appropriate and up to date and whether suggested risk mitigation measures are effective (or likely to be effective). Additional risks to the sustainability of a project’s results (not given in the project document) may have been identified during the process of reviewing or reconstructing the theory of change, as the drivers and assumptions required to achieve higher-level results are often related to factors affecting sustainability. Any such risks may need to be added to the FAO and GEF reporting forms. Sustainability should be assessed and rated for four main aspects of sustainability – financial, sociopolitical, institutional and governance, and environmental.

6

5.1 Financial risks to sustainability Some direct project outcomes do not require further financial input once achieved, for example, the adoption and integration of revised policy into an official government policy framework. However, other project results may depend on additional direct financing to maintain them, for instance, continued capacity building to implement a new resource-management approach, such as climate-smart agriculture practices. Consequently, the MTR should assess the extent to which each project outcome is dependent on prolonged post-project funding to ensure the benefits are sustained. It should also consider potential new financing sources, including the public and private sectors and income-generating activities, to meet this need. 5.2 Sociopolitical risks to sustainability The MTR should assess the extent to which social or political factors, such as increased movement of rural populations to urban centres following crop failure or a change of government, are likely to influence the sustainability of project results and progress towards impacts (either positively or negatively). This should include government-agency, institutional and other stakeholder ownership, awareness and support given to project execution, including the degree of cooperation received from the various public institutions involved in the project (this will affect the longevity of the impacts and the sustainability of the project results). The MTR should assess whether there is sufficient commitment and incentives to sustain the project results and, if not, how this could be improved during the remainder of the project. Particular attention should be paid to any involvement and buy-in of key individuals whose support will help to ensure that project results become embedded in target policies, plans, institutions and offices. This is especially important for the success of any mainstreaming elements of an FAO–GEF project. A key area the MTR should examine is the extent to which project activities have already promoted positive, sustainable changes in attitude, behaviour and power relationships among the various stakeholders supporting the project’s aims and objectives. It should also assess the extent to which the project’s integration of gender issues and human rights is likely to increase the sustainability of results. 5.3 Institutional and governance risks to sustainability The MTR report should assess the degree to which institutional and governance issues impact the likely sustainability of project results. Specifically, it should examine whether project results – such as institutional, policy and legal frameworks and capacity, as well as governance structures and processes – are likely to be robust enough to support continued results after the project ends. It should pay particular attention to whether the project’s institutional capacity-building efforts (systems, structures, staff, expertise, etc.) will be self-sufficient after project end. 5.4 Environmental risks to sustainability Although GEF-funded projects aim to deliver Global Environmental Benefits, it is important to consider whether there are any adverse environmental factors that could undermine and reverse the project’s results and outcomes or prevent project results from being scaled up in the future. Chief among these are potential climate-change impacts, which should be touched upon in the MTR report. Environmental risks should be considered during preparation of the environmental and social management plans and may need to be considered as part of the review of the project’s Environmental and Social Safeguards (ESS) plan.

7

5.5 Ratings for sustainability Based on its assessments, the MTR team should assign a rating to each category based on a four-point scale: likely (L), moderately likely (ML), moderately unlikely (MU) and unlikely (U) (see Annex 11 of the MTR Guide for more details). In addition, the MTR should give an overall rating for the likelihood of sustainability of project results and outcomes at the MTR stage.

5.6 Catalysis and replication The MTR should assess any incidental changes in behaviour and practice that the project has helped to catalyse.3 The MTR report should assess whether the project design presents strategies to promote and support upscaling, replication and/or catalytic action. It should identify how and to what extent the project is providing incentives (social, economic or market-based, for instance) to spur changes in stakeholder behaviour and contribute to policy and institutional change, such as the uptake of project-demonstrated technologies, practices or management approaches that could be duplicated elsewhere. In addition, the MTR should detail any project contributions that are helping to create sustained follow-on financing (catalytic financing) from governments, the private sector, donors, etc. The MTR should also examine the extent to which the project has created opportunities for particular individuals or institutions to catalyse change – so-called “champions” – particularly individuals who support the project’s aims and results in government and civil society, without which the project would not have achieved all of its results and who could potentially help to further promote the upscaling and sustainability of project outcomes. The MTR should describe and assess the approach adopted by the project to promote direct replication4 of projects results and effects and determine to what extent replication has already occurred or is likely to occur in the near future. The MTR should attempt to identify the factors that may positively influence catalysis and replication and the scaling up of project results, experiences and lessons and which factors could be promoted over the remainder of the project (for example, improving linkages with global networks).

6. Factors affecting performance The MTR team should review key factors that can influence, both positively and negatively, the performance and delivery of a project. Each of the following should be assessed and rated separately:

• project design and readiness • project execution and management • project implementation and oversight • financial management and co-financing • project partnerships and stakeholder engagement • communication, awareness-raising and knowledge management • M&E

6.1 Project design and readiness The MTR should assess the project’s design at the mid-point and consider any formal modifications or revisions made to the project’s outputs, outcomes and components and the associated indicators, targets and milestones during project inception and/or early implementation. Where the project outputs are

3 The catalytic role of GEF interventions is embodied in their approach of supporting the creation of enabling environments and of investing in pilot activities, including supporting activities that upscale new approaches to a national, regional or global level. 4 Replication is defined as results, experiences and lessons from the project that are either applied in a different geographic area or scaled up in the same geographic area but funded from other sources.

8

considered to be inappropriately or inaccurately stated in the project document, for transparency, a table should be provided showing the original formulation and the MTR’s amended version. The MTR should undertake a critical analysis of the project’s results framework (captured in its logframe) and the updated or reconstructed theory of change to determine whether the project’s causal logic is sound. Specifically, the MTR should assess whether the project’s objectives and components are clear, practicable and feasible within the allotted timeframe and whether any elements (activities, outputs or outcomes) need to be strengthened or replaced. The MTR should pay attention to the extent to which lessons from other projects were incorporated into the project design and identify any obvious omissions, to what degree the views of stakeholders were taken into account in decision-making during the formulation process and the extent to which equity issues (gender, different ages and marginalized groups) were identified and incorporated into the project’s design. In terms of the project’s “readiness” for implementation after its approval, the MTR should focus on whether appropriate measures were taken to address any weaknesses in project design or to respond to changes made between project approval, the securing of funds and project mobilization (prior to project inception). The MTR should consider, in particular, whether the capacities of the project’s executing agencies were properly taken into account when the project was designed and whether adequate project-management arrangements (initial staffing and financing) were in place when the project began.

6.2 Project execution and management This sub-section of the MTR report should assess the performance of the project management or coordination unit and the project executing partner(s) in managing and delivering the project, including the operational and administrative arrangements. The MTR team should compare current project-management mechanisms with those set out in the project document and the inception report (if one was prepared), documenting where there have been significant changes and how effective these have been in terms of project milestones, outputs and outcomes. The MTR should assess whether management responsibilities and reporting lines are clear and the extent to which decision-making has been transparent and timely. The MTR should identify any operational, institutional or political problems or constraints hindering the effective implementation of the project and show how the project has tried to overcome them. Specifically, the MTR should assess the effectiveness of project management with regard to:

• the project execution and administration framework and arrangements (at all levels, including national agencies), taking into account staffing levels and recruitment, the procurement of goods and services (including consultants) and the preparation and negotiation of cooperation agreements, to the extent that these might have influenced project performance;

• maintenance and performance of executing partner relationships; • communication and collaboration with FAO staff (in Country Offices, Regional Offices and

headquarters) and other projects; • the extent to which work-planning processes are results based and, where they are not, how these

can be re-orientated to focus on results; and • a quality assessment of the identification, mitigation and management of project risks, including

the mitigation and management of environmental and social safeguards and risks as identified in the project’s ESS screening prior to project approval, should also be presented in this section, along with any evidence of how well project management was able to adapt to changing conditions and risks (adaptive management response).

The MTR should pay particular attention to identifying any issues that could impact project duration, such as lengthy delays to project commencement and execution and how these have affected the achievement of project results and outcomes. Where there have been significant delays, the MTR should comment on whether the official length of the project needs to be reconsidered for (at least) some activities (a “no-cost extension” recommendation).

9

Where FAO is both the GEF implementing agency and executing agency for the project, special attention should be paid to the internal execution arrangements, in particular, whether there is adequate separation of duties and responsibilities (an effective “firewall”) between FAO’s execution and implementation functions. Where there is an Operational Partners Implementation Modality (OPIM) component to the project, the MTR team should assess its effectiveness and sustainability. The following sets of questions are suggested for inclusion in the MTR matrix to assess these aspects.

• Were the specific features associated with the OPIM project component taken into consideration during project preparation and design (for example, operational procedures and the capacity of the operational partner(s) (OP))?

• Was the OP selected based on complementarity of skills, capacity and expertise to achieve results?

• To what extent was the OP involved in the formulation of the project and has this engagement contributed to OP ownership during implementation?

• Based on the OP’s capacity assessment and other factors, was the OP expected to be an effective and efficient executing partner? Did the first half of project implementation confirm this?

• To what extent does the project governance structure facilitate project execution and its contribution to project objectives?

• To what extent has delegation of project execution to the OP(s) facilitated or hampered project execution and contributed to project objectives?

• Has the delegation of project execution to the OP(s) contributed to greater national, sub-regional and/or regional ownership (supporting the sustainability of project results)?

• Did the delegation of project execution to the OP(s) bolster the capacities of regional, sub-regional and/or national entities?

• Have there been any advantages to the involvement of the OP(s) in project execution in terms of the achievement and sustainability of results?

• Sub-question: Does the OP provide quality financial reports, PPRs and PIRs and present calls for funds in a timely manner, per the terms and conditions of the signed Operational Partners Agreement?

• Sub-question: How is FAO monitoring the progress and technical quality of the work of the OP? Is the budget holder/project team equipped to do so? Does it get support if needed? Has this led to the implementation of corrective actions and the improvement of identified weaknesses? Does FAO review and approve reports and arrange for the transfer of funds in a timely manner? Does FAO respond to queries and provide technical assistance in a timely manner?

• Sub-question: Is FAO effectively carrying out its role of implementing agency (monitoring, oversight and guidance)? Is the FAO office adequately equipped to do so (from an operational, financial and budgetary perspective)? Does it have sufficient access to (and, where necessary, influence over) the OP to identify operational weaknesses and have improvements implemented? Are improvement plans and audit/spot-check observations effectively followed up?

6.3. Financial management and co-financing The MTR should assess the quality, effectiveness and timeliness of the project’s financial planning, the control of financial resources and financial reporting and whether sufficient financial resources have been made available to the project and its partners. It should also compare actual project costs with the budget set out in the project document and report on variances (see Table A12.2), at outcome level at a minimum. The output level may be too fine-scale for reporting purposes, but it is recommended that projects report at this level where possible.

10

Table A12.2 Project expenditure by outcome and output at the mid-term stage

Component/outcome/output Estimated cost at design (USD)

Actual cost/ expenditure (USD)

Expenditure ratio – actual/planned

Component 1

Outcome 1

Output 1.1

Outcome 2

Output 2.1

etc Component 2

Outcome 2

Output 2.1

Output 2.2

etc

Overall project total

The MTR should examine the level of actual versus planned disbursement for every year up to the mid-point and comment on any changes or trends. Any financial management issues that have affected the performance, timeliness or quality of project delivery should be highlighted. The MTR should comment on the appropriateness of any budget revisions. Financial information should be collected by the BH and PMU prior to the start of the MTR. As mentioned, this should be at least to outcome level and, if possible, to output level. The MTR team members are not accountants and are not expected to undertake an audit of the project’s finances. However, aspects of financial management that should be considered, including:

• contact/communication between the BH, PMU and FAO GEF CU fund liaison officer (FLO) • PMU and FLO knowledge of the project financials, including knowledge of project progress/status

when disbursements are undertaken • degree of attention paid to compliance with procurement rules and regulations • PMU and FLO responsiveness to addressing and resolving financial issues • any revisions to budgets and any issues with disbursement, including proof of transfers • any relevant legal agreements, such as letters of agreement

The MTR report should include the standard GEF co-financing table (planned and actual co-financing commitments), which is presented in Appendix 7 of the MTR report template (Annex 11 of the MTR Guide). This table should be completed by the BH/MTR manager (RM), with support from the PMU and FAO GEF CU FLO, before the MTR mission. They will need to contact each of the co-financing parties to get a full and up-to-date account of co-financing early in the MTR process. The MTR team should then check that this information has been provided and that the co-financing table has been completed. Note, however, that while the MTR should comment on the figures in the co-financing table, the team is not required to audit partner contributions. What it can do is give an opinion on whether the co-financer is really engaged in the project, (attending meetings, allocating staff time to the project, providing logistical support, etc.). The MTR report should identify and describe briefly any additional financial resources (cash and in-kind)5 the project has attracted since project approval and indicate how these resources are contributing to achievement of the project’s results (leveraged co-financing is often under-reported but shows additional commitment and interest in a project so it is important that it is fully reported). Again, this additional co-financing should be identified by the PMU well before the MTR begins and should be reported separately to the original pledged co-financing (at output level, if possible).

5 These resources are additional resources, beyond those committed to the project itself at the time of project approval, that are mobilized after CEO endorsement as a direct result of the project. These resources can be financial or in kind and they can be from other donors, NGOs, foundations, governments, communities or the private sector, not listed in the project document.

11

6.4. Project oversight - supervision, guidance and technical backstopping

This sub-section of the MTR report should:

• assess whether the project’s governance and supervision model is comprehensive, clear and appropriate;

• comment on the effectiveness of supervision, guidance, operational and technical support provided by FAO (BH, LTO and FLO), the PSC and other supervising/supporting bodies, including the adequacy of project supervision plans, inputs and processes;

• comment on whether oversight responsibilities and reporting lines are clear; and

• consider whether decision-making has been transparent and undertaken in a timely manner. The MTR should pay particular attention to the quality, timeliness and responsiveness of FAO’s supervision and technical backstopping support to the project in its role as the GEF implementation agency. This should include assessment of the inputs of the LTO(s), BH, FLO and other possible members of the project task force (PTF) and the effectiveness of technical support collaboration and coordination between the various FAO functional units involved. The MTR should examine the project’s broader governance arrangements, particularly the role and effectiveness of the PSC, and assess the extent to which project management has responded to direction and guidance provided by the PSC and PTF. The MTR should also comment on the effectiveness and value of the PTF, the contribution it has made to the project and whether this could be improved. 6.5. Project partnerships and stakeholder engagement The MTR report should detail the level and quality of stakeholder engagement and the project’s partnership arrangements, both at the design stage and during implementation. The MTR should examine three related (often overlapping) processes: (1) the active engagement of stakeholders in project design, implementation of project activities and decision-making; (2) consultations with and between stakeholders; and (3) the dissemination of project-related information to and between stakeholders. The PMU and RM should produce an initial stakeholder analysis as part of the “project information pack” (see the main MTR Guide for more) provided to the MTR consultants, which should then be reviewed, revised and expanded as necessary by the MTR team and included in the MTR inception report. This stakeholder analysis provides an overview of the individuals and organizations actively involved in the FAO–GEF project and those whose interests may be affected (positively or negatively) by the project. More guidance on stakeholder analysis can be found in Annex 8 of the MTR Guide. GEF is paying more and more attention to stakeholder engagement in and ownership of its projects and programmes (GEF, 2017d; see also GEF 2017e; 2017f; 2018e), so this needs to be a particular focus for the MTR team. This is illustrated by a section of the new GEF online project-reporting portal that asks for “information on progress, challenges and outcomes on stakeholder engagement (based on the description of the stakeholder engagement plan included at CEO endorsement/approval)”. This requires specific reporting on stakeholder engagement on an annual basis and when MTR reports are submitted. FAO’s environmental and social risk assessment (see Box A12.1) requires a stakeholder engagement plan6. This means stakeholder consultations must be documented during project design and implementation. Stakeholders should be involved in, made aware of, and continuously updated on the ESMF/ESMP. The MTR should comment on the comprehensiveness and effectiveness of any partnership strategy developed by the project. A strengths, weaknesses, opportunities and threats analysis (SWOT analysis) of the project’s partnership approach and its implementation may help to identify key issues. It is particularly

6 Moderate risk projects must prepare an ESMP. This can be as simple as a matrix identifying concrete mitigation actions for each identified risk, the roles and responsibilities of relevant actors and the indicators/means of verification to provide evidence of action taken and demonstrate that the risk has actually been mitigated. For projects classified as high risk, FAO policy requires a full independent assessment and ESMP, the risk mitigation actions of which need to be monitored at mid-term. This is also the time to confirm the risk classification and to identify/acknowledge any new risks.

12

important to examine the roles, responsibilities and capacities of key partners. The theory of change should also be used to assist the MTR team in identifying the key stakeholders and their roles, capabilities and motivations at each stage along the project’s causal pathways. The MTR should assess the degree and effectiveness of the key partner engagement in project design, planning, decision-making and activity implementation, particularly those involved in direct project execution and those participating in the PSC and PTF. Consideration should be given to whether local and national government/public-sector stakeholders were fully supportive of the project objectives at the design stage and to what extent they have an active role in project decision-making during project implementation (such as representation on the PSC). It should also assess any support given to maximize collaboration and avoid duplication with partner efforts, mentioning any cost-sharing or other efforts to pool resources, as well as any learning and exchange of expertise, such as with other FAO and GEF projects and programmes. The MTR should give special consideration to the role and effectiveness of the project team’s communication activities in helping to serve and maintain the project’s partnerships and any linkages between its communication strategy/plan and partnership approach (or strategy/plan, if formalized). The MTR should review internal project communication with key partners and project stakeholders, including whether communication has been regular and effective, whether existing communication channels and networks are being used effectively and whether there are adequate mechanisms allowing partners to give feedback on project activities. 6.6. Communication, awareness-raising and knowledge management The MTR should assess how effectively project aims, progress, results and key messages are communicated, along with any structured lesson-learning and experience-sharing between project partners and interested groups. The MTR should comment on the design, coverage and effectiveness of the project’s communication and knowledge-management strategy and plans (if available) for the dissemination of results and lesson sharing. It should take into account the resources available for communication and knowledge-management activities (both financial and in terms of specialized technical communication expertise) and assess whether communication approaches and activities support sustainable project results. The MTR should review the means and effectiveness of external project communication, including whether appropriate and adequate channels, networks and methods for communication and results dissemination have been established and are effective (for example, the quality of web presence, outreach and public awareness campaigns). The MTR should, where appropriate and possible, assess the success of external awareness-raising activities aimed at influencing attitudes or shaping behaviour among the wider public. GEF seeks to ensure that it makes full and effective use of the knowledge and learning it has accumulated from past investments, applying this to current and future projects. It views knowledge management as integral to achieving results and transformational change. Knowledge management is the systematic management of an organization’s cumulative knowledge and experience – its knowledge assets (Box A12.1). According to the GEF Evaluation Policy (GEF IEO, 2019), knowledge management is the process by which GEF partner organizations generate value and improve performance from their intellectual and knowledge-based assets. Effective knowledge management is seen as helping to create and transform knowledge into action, innovation and change. It is also deemed essential to scaling up project results to larger areas and wider landscapes and seascapes (horizontal scaling), other agencies and organizations (vertical upscaling) and related situations (replication and extrapolation). It is closely linked to performance enhancement.

13

Box A12.1 Knowledge management

Knowledge management: the systematic processes, or range of practices, used by organizations to identify, capture, store, create, update, represent and distribute knowledge for use, awareness and learning across and beyond the organization. Knowledge management systems: any kind of IT system that stores and retrieves knowledge, improves collaboration, locates knowledge sources, mines repositories for hidden knowledge, captures and uses knowledge, or in some other way enhances the knowledge-management process. Knowledge products and services: outputs such as databases, publications, visual materials, maps (products) and outcomes such as awareness-raising, information-sharing and capacity-building (services). Knowledge assets: are the accumulated intellectual resources of an organization in the form of information, ideas, learning, understanding, memory, insights, cognitive and technical skill and capabilities.

Source: GEF (2015); Baldrige (2003); Stocking et al. (2018)

It should be noted that the new GEF online portal requires specific reporting on knowledge management on an annual basis and at mid-term. Specifically, GEF asks for “Knowledge activities/products (based on the knowledge management approach approved at CEO Endorsement/Approval and lessons learned (if available)”. The MTR should comment on the design, effectiveness and impact of the project’s knowledge-management approach, activities and products. Lastly, based on responses gathered during interviews and the overall assessment of the project’s results and performance, the MTR report should suggest project areas that might provide particularly significant or effective communications materials. 6.7. M&E The quality of project M&E should be assessed in terms of its two main elements: M&E design and budgeting, and M&E implementation, including project reporting. 6.7.1. M&E design and budgeting Each project should be supported by a coherent M&E plan that is designed to track progress on the achievement of project outputs and direct outcomes, longer-term outcomes and objectives. The M&E plan should include the collection and analysis of data disaggregated by gender or minority group, with M&E roles and responsibilities clearly defined. The MTR should assess the design, quality and effectiveness of the project’s M&E plan and associated tools. The MTR should include an assessment of whether the project’s results framework (captured in the project’s logical framework, both original and approved updates) is effective as a planning and monitoring instrument. The MTR report should present a critical analysis of the project’s objective and outcome indicators – how SMART they are – and the extent to which the mid-term and end-of-project targets are realistic (or not), suggesting amendments/revisions to the indicators or targets as required. It should comment on whether there was sufficient baseline information available for the indicators at the design stage to allow analysis of changes and trends resulting from the project and whether gender-disaggregated indicators were adequately included in the project logframe and M&E framework. Any suggested revisions to the results framework, such as a reformulation of outcome indicators to make them ‘SMARTer’, should be included as an annex to the MTR report, with justification for each change.

The MTR should note the extent to which the project’s M&E system uses inclusive, innovative and

14

participatory approaches, such as community based monitoring, citizen science groups or natural-resource user groups, such as farmers or fishers. 6.7.2. M&E implementation The MTR should assess whether the monitoring system is fully operational, enabling the timely tracking of results and progress towards the project’s objectives, and whether emphasis has been placed on outcome monitoring (results-based project management). The MTR should examine how information generated by the M&E system has been used to adapt and improve project management, execution and the achievement of results, and to support efforts to promote the sustainability of results (adaptive management). The MTR report should also identify whether sufficient human and financial resources have been allocated to M&E and whether these resources are being used effectively and in a timely manner. The MTR should also assess the quality, utility and timeliness of PMU and partner reporting to both GEF and FAO, including its realism and candour. Projects funded by GEF require annual reporting through PIRs, the appropriate tracking tool (for GEF-5) or contributions to GEF-7 core indicator targets for GEF-6 and GEF-7 projects. The MTR team should verify in the MTR report that the annual PIRs have been submitted and that the tracking tools and core indicators have been updated.

In addition, the MTR should consider how lessons from the adaptive management process have been identified, captured, documented, shared and incorporated into project implementation and assess their quality and usefulness. The MTR should also comment on the utility of the mechanisms used to capture lessons learned (for example, external facilitators, annual project retreats or stakeholder-led workshops).

7. Cross-cutting dimensions 7.1 Gender and equity The MTR should pay attention to whether gender and marginalized groups are being adequately represented (for example, representation on and engagement in the PSC) and the extent to which their needs and interests are being addressed by the project. The MTR should be guided in its assessment of any gender concerns by the GEF Gender Policy (GEF, 2017a), GEF Gender Implementation Strategy (GEF, 2018a), GEF Guidelines on Gender Equality (GEF, 2017b), GEF Guidance to Advance Gender Equality in GEF Projects and Programs (GEF, 2018c), FAO’s Guidelines for the assessment of gender mainstreaming (FAO, 2017c), the FAO Policy on Gender Equality (FAO, 2013) and FAO’s Gender Equality Objectives.7 It should also be noted that the new GEF online portal requires specific reporting on gender concerns as part of overall reporting on an annual basis and at mid-term. Specifically, it asks for “information on progress on gender-responsive measures, indicators and intermediate results as documented at CEO endorsement/approval in the gender action plan or equivalent”. Key areas the MTR needs to detail include the extent to which the project’s intended results are contributing to the realization of international gender equality norms and agreements, as reflected in the GEF and FAO gender policies, objectives and targets, as well as to regional, national and local strategies to advance gender equality. While GEF programmes and projects contribute to gender equality and women’s empowerment in different ways, GEF is placing increased emphasis on tracking its contribution to narrowing three key gender gaps: (1) access to and control of natural resources; (2) participation and decision-making; and (3) access to socioeconomic benefits and services. The review should provide a brief description of the extent to which the project’s design, implementation,

7 These are: (1) the equal participation of women as decision-makers in rural institutions and in shaping laws; (2) equal access to and control over decent employment and income, land and other productive resources; (3) equal access to goods and services for agricultural development and to markets; and (4) a reduction in women’s workload.

15

monitoring and reporting have taken gender and equity concerns into account (for example, gender, youth, vulnerable, marginalized and indigenous groups) and recommend adjustments to project activities and targets to improve implementation, outreach and results for women and other groups in terms of participation, access to resources and benefits. Other issues for consideration could include specific vulnerabilities of women to environmental degradation or disasters and women’s level of engagement in agricultural, fisheries or forestry activities and livelihoods. The MTR should take into account the various activities, capacities, access to resources, roles, needs and priorities of both men and women, as well as any previous gender-specific analysis or assessment that has identified potential impacts of the project intervention on women and men. Gender assessment questions in an MTR might include:

• Is the project reaching both women and men? How does the project engage with women and girls?

• What impact is the project having on women and men, as well as on their gendered power dynamics?

• What are the wider impacts/changes of the project in relation to women and men?

• Are there any legal, cultural or religious constraints on women’s participation in the project?

• To what extent do women fully participate in project decision-making processes and frameworks? To what extent are their voices heard and do decisions reflect their concerns?

• How can the project’s results framework be more gender responsive?

• Does the monitoring and evaluation strategy consider women and men separately?

• Have indicators been developed to measure how women and men are impacted by the activities and results? Are there gender-sensitive indicators in the project results frameworks?

• Are partner organizations aware of and trained to address gender inequalities among beneficiaries?

• What adaptive management measures are recommended to improve gender mainstreaming and the project’s work to advance gender equality and women’s empowerment (for example, mid-term gender analysis)? Is additional gender expertise needed?

• To what extent has the project addressed gender equality concerns in its design and contributed to the empowerment of women throughout its implementation?

• What contribution has the project made to meeting GEF and FAO’s gender equality objectives? It should be noted here that in the GEF-7 results framework, core indicator 118 on the “number of direct beneficiaries disaggregated by gender” is intended to allow a basic level of systematic capture and aggregation on the gender dimension of GEF projects across the GEF portfolio. MTR teams should use it to capture both expected and achieved results for GEF-7 and GEF-6 projects, where relevant and possible. Similarly, the MTR should pay similar attention to any issues associated with age, such as the vulnerability of children to environmental degradation, fewer livelihood opportunities (through the project) for youths, constraints on the elderly population, etc. 7.2 Indigenous peoples If relevant, the MTR should assess a project’s human rights-based approach and how it is addressing, for example, the needs and concerns of indigenous peoples, the extent to which it has applied the United Nations Common Understanding on a Human Rights-based Approach (UN HRBA Portal, 2003), whether it is in line with the United Nations Declaration on the Rights of Indigenous People (UN, 2007) and whether it has followed the concept of Free, Prior and Informed Consent (FPIC) (FAO, 2016) in its design period. This should be treated separately to the section dealing with stakeholder engagement.

8 Core indicator 11 captures the number of individuals who receive targeted support, or direct assistance, from a GEF project. The indicator enables the GEF Secretariat to better identify those projects that are most likely to provide significant socioeconomic co-benefits.

16

7.3 Environmental and social safeguards

GEF’s Updated Policy on Environmental and Social Safeguards (GEF, 2018f)9 requires GEF agencies to assess environmental and social risk during project design and to develop an ESS plan where there is significant risk of adverse impacts, including environmental and social mitigation and management measures. Areas to be considered include:

• direct, indirect, cumulative and transboundary environmental and social risks and potential impacts, as well as the risks and impacts of associated facilities;

• physical, biological (including the introduction of potentially alien invasive non-indigenous species), socioeconomic (including livelihoods), other social (including risks to social organization, health and human safety) and cultural resources;

• applicable national and local laws, as well as directly relevant provisions of international treaties and agreements; and

• short- and long-term risks posed by climate change and other natural hazards. Special consideration is given to disadvantaged or vulnerable groups and individuals that are or may be affected by a project or programme, including people with disabilities, as well as the identification, assessment and prevention of adverse gender-related impacts, including gender-based violence and/or sexual exploitation and abuse. Agencies are required to report annually and at the mid-term and terminal evaluation points on any measures necessary to address these risks.

FAO also has its own policies, procedures, systems and capabilities to ensure that projects and programmes are classified and managed based on the level and magnitude of potential risks and impacts. It undertakes its own structured review of such risks, in line with GEF requirements, during the project preparation phase of an FAO–GEF project. Relevant information is captured in a screening checklist and risk classification certification form, which identifies the level and types of risk (ESS), which groups are likely to be most affected and the degree of stakeholder consultation (during project design). For projects identified with a risk rating of “moderate” or “high”, a risk mitigation plan – the ESMF/ESMP – is developed and usually attached as an annex to the project document. The MTR should assess the ESS assessment undertaken at the design stage and any required ESS plan implemented and validate the ESS risk assessment presented in the project’s annual PIR, suggesting revisions as necessary (see Box A12.2).

Box A12. 1 Assessing ESS at mid-term

• GEF is paying increased attention to ESS in its projects and programmes (GEF, 2018f) and these elements need to be assessed by the MTR team.

• The MTR will assess the environmental and social performance of the project, as well as its environmental and social risks, in accordance with the project’s environmental and social requirements, including its environmental and social instruments (environmental and social screening, environmental and social management framework, or ESMF), environmental and social impact assessment (ESIA) and environmental and social management plan (ESMP).

• The MTR will also review any environmental and social implications resulting from changes in the design of a project or project circumstances.

• If an ESS assessment was not undertaken at the project formulation stage (projects approved before 2016) then the MTR team should review the ESS checklist and judge whether a full ESS screening is needed and include this under the recommendations section of the MTR report.

If an ESS assessment was not undertaken at the project formulation stage, the MTR team should review the checklist and decide whether a full ESS screening is needed. If so, this should be included in the recommendations section of the MTR report as an ESS corrective measure.

9 The GEF Policy on Environmental and Social Safeguards came into effect on 1 July 2019.

Annex 13 – Comments matrix for stakeholder feedback on draft MTR reports and MTR team responses

The FAO GEF Coordination Unit (FAO GEF CU) aims for high levels of transparency and consultation throughout the MTR process. The MTR report – in early and advanced drafts – is shared with internal FAO and external stakeholders, as appropriate and feasible, for review, comments and suggestions. The template in Table A13.1 aims to show how the feedback received on the draft MTR report has (or has not) been incorporated into the final MTR report. This audit trail should be included as an annex to the final MTR report. For transparency, it is important that the comments are available to all concerned and that the MTR team is able to expresses its own views on and response to each comment, be it acceptance and integration or rejection. The MTR manager (RM) should send the matrix to each reviewer in tandem with the draft MTR report. Reviewers are requested to indicate the number of the section/paragraph associated with any comments/feedback they submit and, if necessary, to cut and paste the text from the draft MTR report for clarity. The MTR team will provide its own response, as appropriate.

Table A13. 1 Comments received on [insert date] from the mid-term review of [project name]

Reviewer Comment number

Section / paragraph

number

Comment/feedback on the draft MTR report (cut and paste relevant text from draft report, as appropriate)

MTR team’s response and action taken

Annex 14 – Peer-review guidance

FAO GEF Coordination Office (FAO GEF CU) quality assurance criteria for MTR terms of reference and reports

The criteria checklists are meant as broad guidelines to ensure that peer reviewers take a consistent approach to assessing terms of reference and reports. The peer reviewer may raise issues that are not listed in this annex, as long as they also cover the standard criteria. Terms of reference 1. The national/regional context, the internal FAO context and the programme to be evaluated are described

succinctly, providing sufficient and relevant background to inform and contextualize the MTR questions.

2. The purpose of the MTR is clear and its timing sufficient to inform subsequent planning and programming.

3. The scope of the MTR (timeframe, country/region, interventions included and/or excluded) is clearly laid out and reasonably focused given resources and time constraints.

4. The main MTR questions are structured logically, useful to the purpose of the MTR, adequately

contextualized in detailed methodological questions and can realistically be answered given resources and time constraints.

5. The MTR questions explore the extent to which relevant equity concerns (such as gender, youth, vulnerable groups) were addressed in project/programme design and implementation and examine the project/programme’s contribution to FAO’s Gender Equality Objectives.1

6. The methodology clearly explains how data will be collected in response to all MTR questions, including efforts to collect both quantitative and qualitative data and through ample consultation with beneficiaries and other key stakeholders.

7. The terms of reference describe the expertise required by the MTR team to collect evidence, engage

stakeholders and answer all MTR questions in a competent and credible manner, including gender and other equity issues.

8. The roles and responsibilities of the project/programme implementers and executing agencies, FAO GEF CU,

MTR team members and other stakeholders are clearly described, are sufficient to ensure the usefulness of the MTR and consistent with FAO GEF CU guidance (including on who has the responsibility for finalizing and issuing the report).

9. The terms of reference present a realistic timetable, setting out the main steps in the MTR process, including milestones for consultation with key stakeholders and sufficient time for report finalization.

1 These are: (1) the equal participation of women as decision-makers in rural institutions and in shaping laws; (2) equal access to and control over decent employment and income, land and other productive resources; (3) equal access to goods and services for agricultural development and to markets; and (4) a reduction in women’s workload.

MTR reports

1. The structure of the report is clear, follows the logic of the MTR, adequately covers the main MTR questions, minimizes repetition of information and uses the template found in Annex 11.

2. The methodology section includes an overview of the contextualized MTR questions and a description of how these were answered by data-collection efforts. It presents a balanced discussion of methodological limitations.

3. Findings are well substantiated by evidence from relevant and triangulated sources, including both qualitative and quantitative data.

4. The report paints a convincing picture of results achieved based on significant beneficiary feedback.

5. The report uses clear and appropriate language, minimizing ambiguity, subjectivity and bias and distinguishing between fact and opinions, each being properly referenced.

6. The report includes a cogent, well-articulated and fact-based examination of relevant equity issues (including gender mainstreaming, attention to youth, inclusion of vulnerable groups).

7. The conclusions build on the review’s findings in a coherent way, respond to the MTR’s main questions and purpose, build on a careful examination of underlying performance factors and reflect reasonable performance expectations.

8. Recommendations address the main findings and conclusions, target specific actors and cover gender equality and other equity issues. They are realistic, actionable, important to the success of the enterprise, with suggested timeframes and not too numerous.2

9. The appendices include lists of the people and documents consulted.

2 It is difficult to set a precise limit, as it would depend on the complexity of the project or programme being evaluated, but as a rule of thumb, there should be no more than a dozen recommendations.

1

Annex 15 – Glossary: Definitions of key GEF terms1 Agency fee: Financing provided to a GEF partner agency in connection with a GEF project or programme CEO approval: The GEF CEO’s approval of a fully developed medium-sized project or enabling activity CEO endorsement: The GEF CEO’s endorsement of a fully developed full-sized project Child project: A project that forms part of a programme, as set out in a programme framework document Co-financing: Financing in addition to GEF project financing, which supports the implementation of a GEF- financed project or programme and the achievement of its objectives Evaluation: The systematic and impartial assessment of planned, ongoing or completed activities, projects, programmes in specific focal areas or sectors, policies, strategies and their implementation, or other topics relevant to the GEF partnership and organization2 Full-sized project: A project with GEF project financing exceeding USD 2 million GEF additionality: Additional effects (both environmental and otherwise) that can be directly associated with a GEF-supported project or programme GEF agency: An agency eligible to request and receive GEF resources directly for the design, implementation and supervision of GEF projects and programmes GEF Instrument: Instrument for the Establishment of the Restructured Global Environment Facility, effective7 July 1994, as amended – the document that established the GEF after an initial pilot phase Gender-sensitive indicator: An indicator that can be used at various levels to monitor and report on socioeconomic and gender-sensitive changes over time GEF-financed activity (or intervention): Any programmatic approach, full-sized project, medium-sized project or enabling activity financed from any GEF-managed trust fund, as well as regional and national outreach activities GEF operational focal point: A recipient-country nominee, who ensures that GEF proposals and activities in the country are consistent with country priorities and commitments under global environmental conventions; identifies project ideas to meet country priorities; endorses project proposals; facilitates broad based in-country consultations on GEF operational matters; and provides feedback on GEF activities, including project implementation3 Global Environmental Benefits: Benefits on which GEF investments are predicated and which GEF project activities are mandated to deliver, on biodiversity, climate-change mitigation, international waters, land degradation and forests, and chemicals and waste Goal: A higher-order objective to which a GEF-financed project or programme should contribute Knowledge management: The process by which organizations within the GEF partnership generate value and improve performance from their intellectual and knowledge-based assets

1 Based on GEF IEO (2019) 2 Adapted from OECD/DAC (2002) 3 Based on GEF (1996)

2

Impact: The positive and negative, primary and secondary long-term effects of a project or programme, be they direct or indirect, intended or unintended Indicator: A quantitative or qualitative factor or variable that provides a simple and reliable means of measuring achievement, to reflect the changes associated with a project or programme, or to help assess the performance of an organization Lead agency: The agency that coordinates all activities under a programme Medium-sized project: A project with GEF project financing of up to USD 2 million Mid-term review: An assessment of a project or programme’s performance and results, carried out for adaptive management purposes at the mid-point of its intended duration Monitoring: A continuous or periodic function, carried out by project or programme management, that uses a standardized and systematic process of collecting and analysing data on specific indicators to provide decision-makers and management of a GEF-financed activity with information on progress in the achievement of objectives and the use of allocated funds Outcome: An intended or achieved short- or medium-term effect of a project or programme’s outputs Output: A product or service that results from the completion of activities implemented as part of a project or programme Portfolio: A subset of projects focusing on a specific theme, GEF focal area, geographic region, country or GEF agency Programme: A coherent set of interventions designed to achieve specific global, regional, country or sector objectives, consisting of a variable number of child projects Programme’s added value: The additional results brought about by a GEF-funded programme compared with a pre-existing or hypothetical set of standalone full- and/or medium-sized projects or comparable alternatives Programme framework document: The document that sets out the concept of a programme proposed for GEF financing Result: Discernible and measurable intervention outputs, outcomes and progress toward longer-term impacts, including Global Environmental Benefits Stakeholder: An individual or group with an interest in the outcome of a GEF project or programme, or which is likely to be affected by it, such as local communities, indigenous peoples, civil-society organizations and private-sector entities; stakeholders may include national project or programme executing agencies, or groups contracted to conduct activities at various stages of the project or programme

Date post:	15-Oct-2021
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times