+ All Categories
Home > Documents > Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities...

Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities...

Date post: 27-Jun-2020
Category:
Upload: others
View: 3 times
Download: 0 times
Share this document with a friend
16
Part 1 Evaluation in JICA
Transcript
Page 1: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

Part 1Evaluation in JICA

Page 2: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

1-1 JICA's Evaluation Activities(1) Objectives of Evaluation

JICA's project evaluation is carried out at each stage ofthe project cycle in order to assess the relevance and effec-tiveness of a project as objectively as possible, and furtherimprove it. More precisely, the evaluation is conducted forthe following three purposes: (1) feeding back evaluationresults to the decision-making process for use in projectmanagement, (2) utilizing the lessons learned from evalua-tion results to assist the learning process of the aid organi-zations concerned, and (3) disclosing information relatedto the effectiveness and processes of JICA's cooperationprojects both domestically and internationally to secureproject transparency and accountability. By utilizing theevaluation results and gaining public understanding andsupport for its projects, JICA is committed to providingmore effective and efficient cooperation.

(2) Types of EvaluationHere, JICA's project evaluations are explained as being

categorized in terms of “what to evaluate” (the evaluationfocus), “when to evaluate” (the evaluation implementationstage), and “who will evaluate” (the evaluator).

1) Evaluation FocusODA evaluations can be classified into “project-level,”

“program-level,” and “policy-level” evaluations from theperspective of “what to evaluate.” Among these evaluations,JICA conducts project-level and program-level evaluations(Figure 1-1).

The project-level evaluation covers individual projects.The results are fed back for formulating or reviewing pro-jects and deciding whether a project should be completedor continued, or utilized as lessons for similar projects.JICA is making efforts to secure project transparency andaccountability by disclosing evaluation results.

Chapter 1

10 Annual Evaluation Report 2007

Figure 1-1 ODA System and JICA's Evaluation

ODA Charter

Medium-term Policy on ODA

Country A's NationalDevelopment Plan

Development issues 2Country A’s development issues 1

Country A’s development issues 1a Country A’s development issues 1b

Country Assistance ProgramSection-specific Initiatives

Partner country's development plan, etc.

JICA's evaluation focus

Policy

level

Program

level

CountryA's

inde-pendentproject

Cooperation

byotherdonors

Projectlevel

JICA Country Program (for Country A)

Development issues 1

JICA program* JICA program*

JICA

project

JICA

project

CountryA's

inde-pendentproject

Cooperation

byotherdonors

JICA

project

JICA

project

JICA's basic policies• Mid-term Plan• Thematic guidelines, etc.

* JICA program is defined as astrategic framework to support the achievement of mid- and long-term development goals in a developing country.

JICA's Evaluation Activities and Effortsfor Expanding and Enhancing Evaluation

Page 3: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

EvaluationinJICAPart

1

Program-level evaluation evaluates a set of projects relat-ed to particular countries or a development issue in a com-prehensive and cross-sectional manner, for generalizingitems common to the target countries and development issueas much as possible, and drawing out lessons that allow easi-er feedback. Its evaluation themes include what effects areachieved by JICA's cooperation in particular countries or theconsequence of JICA cooperation approaches to specificdevelopment areas. Moreover, it is directed at specific coop-eration schemes including the Volunteer Program and Disas-ter Relief Program. These evaluations are conducted by theOffice of Evaluation of the Planning and CoordinationDepartment of JICA or other JICA project implementationdepartments involved with the project. The results of pro-gram-level evaluation are not only fed back to the planningand implementation of individual projects, but also utilizedfor improving cooperation approaches of JICA, for formula-tion, improvement and effective implementation of JICACountry Programs and thematic guidelines, as part of a fur-ther comprehensive effort. Furthermore, given the recentintensified program approaches by JICA, the program-levelevaluation implements evaluation on JICA program, com-bining a number of projects organically and closely, with aclear scenario for achieving the goal of a particular program.In fiscal 2006, JICA conducted evaluation focusing on fourprograms implemented around the world. The results aredescribed in Part III.

2) Evaluation within the Project CycleProject-level evaluations are classified into four types

from the perspective of when to evaluate: ex-ante, mid-term,

terminal, and ex-post evaluations, which correspond to fourstages in the project cycle (Figure 1-2).

a. Ex-ante evaluationThe ex-ante evaluation is conducted prior to the imple-

mentation of a project to check conformity with develop-ment policies of the partner country, Japan's aid policy, andneeds of the partner country, as well as to clarify the projectcontent and expected cooperation effects for the purpose ofcomprehensively evaluating the necessity of the project andthe relevance of the cooperation scheme. Evaluation indica-tors of a project set at the ex-ante stage will be used to mea-sure the progress and effect of cooperation in subsequentmonitoring and evaluations at stages from mid-term to ex-post evaluations.

b. Mid-term evaluationThe mid-term evaluation is conducted at the midpoint of a

project in order to evaluate it for smooth operation leading tooutcome. It aims to clarify the achievements and implement-ing process, and examine whether plans of the project arerelevant, focusing on effectiveness, efficiency, and otheraspects. Results of the mid-term evaluation are utilized torevise the original plan or improve the operation structure.

c. Terminal evaluationThe terminal evaluation is conducted to examine whether

the project will achieve the outcome as planned prior to ter-mination of a project. It comprehensively analyzes theachievement level of the project purposes, efficiency, andprospective sustainability of a project. Based on the result, it

Chapter 1 JICA's Evaluation Activities and Efforts for Expanding and Enhancing Evaluation

11Annual Evaluation Report 2007

Figure 1-2 Position of Evaluation within JICA's Project Cycle

Plan

PlanEx-post

evaluation

Ex-postevaluation

Ex-anteevaluation

Terminalevaluation

Mid-termevaluation

Accountability

Feedback

Monitoring

Feedback

ImplementationPost-implementation

Program cycle

Project cycle

Feedback

Feedback

Formulation of JICA Country Program and thematic guidelines

Page 4: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

is decided whether to complete the project according to theoriginal plan or whether follow-up is necessary.

d. Ex-post evaluationThe ex-post evaluation is conducted a few years after

completion of a project to verify the achievement level of theoverall goal, the presence of ripple effects, and sustainabilityof the effects at which the project was aimed. Results of ex-post evaluation serve as lessons learned for effective andefficient project implementation in formulating and imple-menting new projects and/or programs in the future.

The program-level evaluation evaluates and analyzeseffects resulting from cooperation among plural projects andtheir approaches, mainly from an ex-post evaluation stand-point and in a cross-sectional manner, after the projects arecompleted. However, the evaluation may be occasionallyconducted as an ex-ante or mid-term evaluation, to confirmthe relevance of programs or review plans.

3) Evaluation by Types of EvaluatorsIn terms of “who evaluates” (i.e. evaluator), JICA's eval-

uation is categorized as internal evaluation and externalevaluation. There is also a joint evaluation conducted byJICA and external institutions.

a. Evaluation by JICA (internal evaluation)It is conducted by JICA, which is responsible for project

management in cooperation with external experts, such asconsultants and academics. Since the internal evaluation isheaded by the parties concerned who are well versed inthe processes and methods of JICA's operations, there arethe advantages of smoothly accessible information neces-sary for project management and review, and the easy acqui-sition of specific, practical recommendations and lessons.JICA also consults third parties (academics, journalists,NGOs, etc.) with expertise in development assistance andfamiliarity with JICA's undertakings, and has them reviewinternal evaluation results (= secondary evaluation)1 in orderto assure transparency and objectivity of internal evaluation.

b. Evaluation by third parties (external evaluation)This is the evaluation entrusted to third parties who are not

involved in the planning and implementation of the evaluat-ed project, and who have high expertise in the fields evaluat-ed, specifically external specialists and institutions, such asuniversities, research institutes, academics and consultants.The external evaluation is intended to secure the quality,transparency, and objectivity of evaluation. External evalu-ation may be conducted by external experts and third-partyorganizations in the partner country in addition to those inJapan.

External evaluation in JICA includes cases where externalthird parties implement thematic evaluation as part of pro-gram-level evaluation, and secondary evaluation describedin a).

c. Joint evaluationThis evaluation is conducted jointly with the institutions

concerned in partner countries in which JICA implementsprojects and other aid agencies (donors). Joint evaluationwith partner countries allows JICA and the partner coun-tries to share an awareness of the effects and issues ofJICA's cooperation. Further, it also contributes to enhancingthe evaluation and monitoring capacities of partner countriesby allowing relevant parties in partner countries to share anevaluation method in the evaluation process. Since allJICA's projects are jointly implemented with partner coun-tries, the project-level evaluation is conducted jointly withpartner countries at all stages from project planning tocompletion. Consequently, these evaluations can be con-sidered to possess the aspects of the JICA-led internal eval-uation described in a) and those of joint evaluation con-ducted with developing countries. The program-level eval-uation is also conducted with the participation of relevantparties of the partner countries, and its evaluation resultsare fed back to those parties through seminars.

The joint evaluation with other donors is consideredimportant along with the advancement of aid coordinationin developing countries. It proves helpful in learning abouteach other's projects and evaluation methods through theevaluation process itself.

(3) Methods of EvaluationThe project-level evaluation framework is composed of

three stages: (1) studying and understanding the situationsurrounding the project; (2) assessing the value of the pro-ject by the five evaluation criteria; and (3) drawing recom-mendations and lessons, and feeding them back for improve-ment.2

When conducting an evaluation, it is important to under-stand how to utilize its results. This entails verifying thecurrent situation of a project in a systematic and objectivemanner, and making a convincing judgment about its val-ues based on the results. In addition, it could be said thatevaluation results are further utilized by precisely analyzingpromoting or inhibiting factors toward the enhancement ofeffects and smooth implementation of projects, and drawingout recommendations and lessons for future projects.

1) Studying and Understanding the Situation Sur-rounding the Project

The first step is to examine the project achievementsregarding what has been achieved in the project and to whatextent it has been achieved. The next step is to identify andanalyze the implementation process regarding what is hap-pening in the process of achievement and what kind ofeffects it has on the achievements. Furthermore, the causalrelations between the project and the effect, namely whetherconfirmed achievement has resulted from the project, isexamined.

12 Annual Evaluation Report 2007

1 Fiscal 2007 secondary evaluation results by the Advisory Committee on Evaluation are described in Part 4 of the report.2 JICA's project evaluation methods are explained in detail in the “JICA Evaluation Handbook: Practical Methods for Evaluation” (JICA, March 2004).

These guidelines are available on the Evaluation page on JICA's website (http://www.jica.go.jp/index-j.html).

Page 5: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

2) Assessing the Value of the Project by the FiveEvaluation Criteria

The next step is to make value judgments about the projectbased on information about the actual conditions of the pro-ject as obtained through the above-mentioned procedure. Forjudging the value of projects, JICA has adopted the fiveevaluation criteria (relevance, effectiveness, efficiency,impact, and sustainability) proposed in 1991 by the Devel-opment Assistance Committee (DAC) of the Organisationfor Economic Co-operation and Development (OECD)(Table 1-1).

3) Drawing Recommendations and Lessons andFeeding Them Back for Improvement

Based on the results of an evaluation study, recommenda-tions should be proposed on specific actions for the projectstakeholders, and lessons should also be formulated to pro-vide information for future projects. Evaluation results arereported to those involved in the project and disclosed pub-licly. Feedback of evaluation results to projects is importantin improving the project and enhancing its effectiveness. Inorder to make recommendations and lessons that are easilyfed back, it is necessary to clarify the contributing andinhibiting factors that have affected the production of effectsof a project. It is also necessary to specify the target of thefeedback.

(4) Evaluation SystemJICA's current evaluation system is composed of the Eval-

uation Study Committee, the Advisory Committee on Evalu-ation, Office of Evaluation, and the project implementationdepartments (headquarters and overseas offices). The majorroles and activities of each group are shown in Figure 1-3.

Chapter 1 JICA's Evaluation Activities and Efforts for Expanding and Enhancing Evaluation

13Annual Evaluation Report 2007

EvaluationinJICAPart

1

Table 1-1 Perspectives of Five Evaluation Criteria

“Relevance” questions integrity and necessity; whether the project purpose meets the needs of the intended beneficiaries; whether it is consistent with the partner country's policies and Japan's aid policies; and whether the project approach is appropriate.

Relevance

“Effectiveness” questions whether the project purpose has been achieved to benefit the beneficiaries and target societies.

Effectiveness

“Efficiency” questions whether input resources have been utilized effectively, mainly byfocusing on the relations between costs and outputs.

Efficiency

“Impact” questions long-term effects and ripple effects brought by the implementation of a project, including the achievement level of the overall goal and unintended positive and negative effects.

Impact

“Sustainability” questions whether the effects achieved in the project are sustained even after the completion of cooperation.

Sustainability

Figure 1-3 JICA’s Evaluation System

Advisory Committeeon Evaluation

EvaluationStudy Committee

Working Group

Consultation

Recommendation

Project ImplementationDepartments

Evaluation Chiefs

Board ofVice-Presidents

Study on evaluation policies and methodsStudy on methods to feed back evaluation results for future projects

Advice/Information Advice/Information

Conducting evaluation/using evaluation results

Overseas Offices

Evaluation Chiefs

Conducting evaluation/using evaluation results

Sharing/Exchanging information

Office of Evaluation

• Improving methods of JICA evaluation• Promoting feedback of evaluation results• Implementing program-level evaluation• Quality control for evaluation

The committee is made up of external experts (academics, UN agencies, NGO members, and the private sector) knowledge-able about issues concerning development aid and evaluation. The committee provides advice to the Evaluation StudyCommittee on evaluation systems and methods. It alsoreviews the results of internal evaluations to improve the objectivity of evaluations.

The committee is headed by the JICA Vice-President in charge of the Planning and Coordination Department and composed ofthe directors general of related departments. The committee examines and discusses JICA's basic evaluation policies as well as the methods of promoting evaluation feedback.

Page 6: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

1-2 Efforts for Expanding and Enhanc-ing Evaluation

Recently, the situation surrounding JICA activities hasbeen changing greatly as a result of ODA reform and theprogress of international aid coordination. New JICA willbe established in October 2008 to manage yen loans andgrant-aid cooperation in addition to technical cooperationin an integrated manner. Under such circumstances, JICA'sproject evaluation has made various efforts in order toimplement effective and efficient projects, as well as ensureaccountability.

In fiscal 2006, JICA implemented upgrading evaluationcapacity of its overseas offices to strengthen field-basedmanagement, solidifying evaluation on diverse aid schemesincluding the Disaster Relief Program, expanding evaluationof JICA programs, and practicing secondary evaluation byexternal specialists. This section introduces JICA's effortsto expand and strengthen its evaluation system based onthree perspectives: 1) consistent evaluation from ex-ante toex-post stages, 2) evaluation covering various activities,and 3) securing transparency and objectivity in evaluation.

(1) Consistent Evaluation from Ex-ante to Ex-post Stages

1) Upgrading the Evaluation SystemTo implement a more effective and efficient project, it is

necessary to ensure that the project is appropriatelyplanned/designed, examine the progress and its effects inthe implementation process, and strive to achieve the pro-ject goals, while reviewing its planning and implementa-tion status based on the examination results as needed.Even after the completion of projects, it is required to con-firm whether the activities keep developing, and lead tohigher and wider effects which the projects aimed at, basedon the issues assured upon completion of the projects.

JICA introduced the ex-ante evaluation system in fiscal2001 and the ex-post evaluation system by projects in fiscal2002, creating a system for enabling JICA to assess effectsof projects at each stage, from ex-ante, mid-term, terminal toex-post, by adding the new evaluation stages to previouslypracticed mid-term and terminal evaluations.

Based on establishing the consistent evaluation systemsfrom ex-ante through ex-post evaluations, JICA revised itsproject evaluation guideline in fiscal 2003 and released the“Evaluation Handbook: Practical Methods for Evaluation,”which clearly explains JICA's new evaluation systems. Asecondary evaluation for the terminal evaluation conductedby the Advisory Committee on Evaluation confirmed thatthe quality of JICA's evaluations has been improved afterrevision of the evaluation guideline. In fiscal 2006, JICAconducted the upgrading of survey items in each evaluationand review of report formats to raise consistency betweenterminal evaluation and ex-post evaluation.

2) Improving Evaluation CapacitySince JICA became an independent administrative insti-

tution, it has made ongoing efforts to strengthen field basedmanagement and reinforce the structure of its overseas

offices, as well as promote the delegation of authority andthe formulation and implementation of projects led by itsoverseas offices. With regard to the project evaluation,JICA has strived to strengthen its evaluation system so thatits overseas offices can independently plan and manageproject evaluations from ex-ante to ex-post evaluations,and compile the evaluation results. Specifically, JICAintroduced a system under which evaluation chiefs areassigned to its overseas offices, as are in the headquarters, toaccumulate information related to evaluation at the fieldlevel and to manage the quality of evaluation, and the Officeof Evaluation of the Planning and Coordination Departmentof JICA supports and supervises the evaluation chiefs. Fromfiscal 2006 to 2007, JICA conducted evaluation chief train-ing for 49 offices to enhance the evaluation capacity ofoverseas offices. Evaluation chief training was providedvia a teleconferencing system that connects the headquar-ters with overseas offices, in offering practical trainingbased on case studies that are applicable to actual projects.

Although the ex-post evaluation introduced in fiscal2002 is implemented in principle by the overseas offices,JICA has also attempted to improve the capacities of theofficials responsible for evaluation, including local offi-cials, by holding local seminars to familiarize them withthe evaluation methods. As a result, the number of coun-tries where JICA's overseas offices conducted ex-post eval-uations by fiscal 2006 has been increased to 48 countries.

3) Strengthening the Feedback of Evaluation ResultsIn addition to analyzing evaluation study results and

making objective value judgments based on those results, itis required in the project evaluation to feed back the evalu-ation results for planning and implementing future projectsmore effectively and efficiently. JICA has been mountingan effort to improve the feedback of evaluation results fromproject level to program level. More precisely, when con-ducting the ex-ante evaluation of a project, JICA encour-ages using lessons obtained from the evaluation results ofother similar projects and describing the results in the col-umn of “utilization of lessons learned from similar projectsin the past” in the ex-ante evaluation document. JICA hasalso introduced the “synthesis study of evaluation results,with which JICA expects to extract applicable lessons com-mon to specific sectors and issues by examining the evalua-tion results of projects related to specific sectors or issuesin a comprehensive, cross-cutting manner. Additionally, inthe process of preparing the “Thematic Guideline” thatcompiles JICA's cooperation policies for each develop-mental issue, the lessons obtained from evaluation resultsare to be reflected in the guideline.

(2) Evaluation Covering Various Activities1) Introduction of Evaluation to Various Activities

JICA has various cooperation modalities other than Tech-nical Cooperation Projects and Development Studies indeveloping countries. For example, the Volunteer Programsuch as the Japan Overseas Cooperation Volunteers aims topromote mutual understanding through public participationin international cooperation; the Disaster Relief Program

14 Annual Evaluation Report 2007

Page 7: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

provides personnel assistance and emergency relief suppliesin the wake of major natural disasters overseas. Due to dif-ferences in nature, it is difficult to apply the evaluationmethod for Technical Cooperation Projects to these modali-ties as it is. Accordingly, JICA has worked to introduce sys-tematic evaluation, including development of evaluationmethods that suit the nature of characteristics of the modali-ties.

The Volunteer Program is evaluated from three view-points as it has three objectives, namely, contributions tosocial and economic development and reconstruction indeveloping countries, promotion of friendly relations andmutual understanding between Japan and developing coun-tries, and sharing volunteer experiences with society back inJapan. Accordingly, projects are evaluated from these view-points. Based on this framework, evaluation was introducedto the Volunteer Program in fiscal 2004 and a comprehen-sive evaluation study was completed in June 2007. In fiscal2006, JICA took a new approach in taking up three caseswhere volunteers were dispatched in a group under a com-mon goal. JICA evaluated the processes and impacts ofthese cases from their formation stage to implementationstage.

With regard to the Japan Disaster Relief Teams, whichare dispatched as part of the Disaster Relief Program to res-cue and treat disaster victims, JICA conducts an evaluation

from a perspective that corresponds to the characteristics ofthe projects involved. The evaluation guideline of the pro-gram stipulates that the activities of medical teams compris-ing doctors and nurses, and rescue teams comprising rescueworkers of the National Police Agency, Fire DefenseAgency and Japan Coast Guard are evaluated in JICA's ownperspective of "STOP the pain" to respond to its urgency andmeet the needs. JICA implements evaluation setting a"LOCK the pain" viewpoint for expert teams who directemergency disaster measures and disaster relief activities,hoping for adequate responses during a preparation perioduntil they are dispatched and unerring operation while theyare dispatched. From 2006 to 2007, JICA summarized pluralevaluation results to have been implemented in line with theguideline, and conducted an ex-post on-site survey on thetwo past cases of dispatching the Japan Disaster ReliefTeam, to verify the effects on beneficiaries through inter-views with disaster victims and government officials inthose days. Based on these results, JICA drew lessons forJICA's future assistance and evaluations, and published areport (Box 1).

2) Examination of Methodology of Grass-roots-typeCooperation Evaluation

Under the concept of “human security,” grass-roots-typecooperation in the form of “cooperation directly reaching

Chapter 1 JICA's Evaluation Activities and Efforts for Expanding and Enhancing Evaluation

15Annual Evaluation Report 2007

EvaluationinJICAPart

1

In the emergency disaster reliefactivities conducted as part of “Disas-ter Relief Program,” to establish moreobjective evaluation methods that thepublic can easily understand, JICAattempted to establish its own evalua-tion guideline or emergency responsein accordance with the particulars ofthe program and the cooperationframework. The effort resulted in for-mulation of “Evaluation Guideline forthe Japan Disaster Relief Team(“STOP the pain”) in fiscal 2002 andan Evaluation Guideline for the JapanDisaster Relief Expert Team (“LOCKthe pain”) in fiscal 2003.

Since 2003, JICA has been evaluat-ing disaster relief activities based onthese Guidelines and, in fiscal 2006,compiled individual evaluation resultsrelated to seven disaster relief activi-ties, including the Iran earthquake,and a general overview of these activi-ties in the “Emergency Disaster ReliefActivities Evaluation Report.”

In fiscal 2007, which marked the20th anniversary of the Japan DisasterRelief Team, JICA conducted an eval-uation study of the dispatches of res-cue and medical teams for the earth-quake disaster in Pakistan, and themedical team for the Indonesian Cen-tral Java earthquake disaster in orderto examine an objective analysis ofDisaster Relief activities and the use-fulness of the Evaluation guidelines

themselves. In this study, a field sur-vey was conducted with the participa-tion of external experts to understandthe effects of activities by the JapanDisaster Relief Team exerted on thefinal recipients, including interviewswith local government officials andpeople in affected areas.

JICA was highly evaluated by thepeople concerned as a result of thisstudy based on four items of the Eval-uation Guideline for the Japan Disas-ter Relief Team: 1) speed, 2) targetgroups (meeting victims' needs), 3)operation (efficiency of activity), and4) presence (degree of acknowledg-ment).

In the interviews conducted withlocal residents during the field survey,kind words were expressed in particu-lar for the Japanese government andthe medical teams, which providedmedical care in affected areas. Thiswas a precious opportunity for thestudy team to examine the effects onrecipients. Moreover, a local hospitalin Indonesia that inherited the medicalteam's equipment established an orga-nization based on what it had learnedfrom the Japanese medical team, andprovides disaster-relief training andmedical support for victims of disas-ters in Indonesia. As noted above, thestudy team was also able to confirmthe positive ripple effects.

In view of recent comprehensive

evaluation results, JICA will continueto implement disaster relief activitesmore effectively and efficiently in thefuture.

Through the latest evaluation study,issues concerning the evaluation peri-od set in the Guideline, detailed evalu-ation standards, criteria of judgment,and method of collecting informationnecessary for evaluation have sur-faced. JICA will continue to improveits evaluation methods in line with thecharacteristics and actual conditionsof disaster relief operation in thefuture.

1 Introduction of Evaluation to Various Activities - Thematic Evaluation:the “Emergency Disaster Relief Activities”

Upper: Hospitaldamaged by anearthquake nowunder repairLeft: Victims beinginterviewed by thestudy team

Page 8: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

local people” has been increasing in JICA. There are manyexamples of grass-roots-type cooperation where regionalresidents participate in the processes of program planningand implementation, or the so-called community participa-tion approach. In view of this movement, JICA has beendeveloping an evaluation method for the projects that incor-porate the community participation approach. Under theframework of an NGO-JICA Evaluation Subcommittee(Table 1-2) formed with NGO staffers possessing a wealth ofknowledge about grass-roots-type cooperation, JICA hasrepeatedly discussed an ideal approach to proper communityparticipation and an evaluation method in line with thatapproach. In response to results of Thematic Evaluation“Community Participation,” the Subcommittee implemented“Community Participation (Phase 2)" to develop a morepractical project evaluation method from fiscal 2006 through2007. In this evaluation, the Subcommittee conducted evalu-ation focusing on both NGO projects and JICA projects, toanalyze and study existing or expected differences in thedegree of community participation depending on projects,and how the differences are measured and evaluated (Box2).

16 Annual Evaluation Report 2007

Following Thematic Evaluation“Community Participation (Phase 1)”conducted in fiscal 2005, the NGO-JICA Evaluation Subcommittee hasbeen conducting the “Community Par-ticipation (Phase 2)” from fiscal 2006.

1. Overview of EvaluationBased on the fact that despite the

projects incorporating the approach ofcommunity participation, the position-ing and targeted degree of communityparticipation differ from project toproject, this evaluation is aimed atanalyzing and examining to whatdegree of community participationis/was targeted by implementing aproject, and how the project imple-menters evaluate the degree ofachievement.

2. Evaluation Basis and Per-spective

To begin with, participation (resi-dents' involvement with the project)that becomes the key of evaluationwas classified into the following threetypes: 1) passive participation (partic-ipation of residents mobilized inresponse to a call by external parties,such as NGOs and donors, under strictcontrol by them), 2) cooperative andfunctional participation (in which resi-dents realize benefits of activities, res-idents and project implementers worktogether, and residents play a certain

role), and 3) autonomous participation(residents develop activities indepen-dently and positively).

Since participants (evaluation focus)from different units and areas areinvolved depending on the project, theywere classified as “individual unit,”“organizational unit,” or “community/society.” And since the evaluation cri-teria of Phase 1 mixed “active aspects”and “inner aspects,” they were alsoclassified. Based on the results, newevaluation criteria were establishedaccording to subjects, with indicatorsto measure a degree of participation.

3. Results of Case StudyJICA set up a joint study team with

an NGO and conducted a field surveyon four projects (two programs each ofthe NGO and JICA) in Central Ameri-ca (Panama and Honduras) and Africa(Ghana) either underway or complet-ed. The study team visited the sites ofeach project, gathering informationfrom regional residents, groups, andrelevant parties, while utilizing a par-ticipatory study approach. Based onthe information collected in the fieldsurvey, NGO and JICA conductedevaluation analysis on the degree ofparticipation and participants, in linewith the key and perspective describedin (2).

The analysis results revealed thatthe four projects were classified into

categories which were aimed at “coop-erative and functional participation”(two projects) and “autonomous par-ticipation” (two projects) according totheir initial goals. In projects thatachieved “autonomous participation,”factors that enhanced residents' inde-pendence and autonomy were ascribedto the fact that residents' groups andproject implementers kept theirpromises and built strong relations oftrust, and that the project imple-menters had a clear-cut exit strategyand shared it with the residents in theprocess of implementation.

The analysis results noted abovewill be released to the public afterbeing compiled in a report (Communi-ty Participation (Phase 2).

2 Cooperation with NGOs - Thematic Evaluation “Community Participation(Phase 2)”

Table 1-2 Members of the NGO-JICA Evaluation Subcommittee

NGO

JICA

Atsuko Isoda Japan International Volunteer Center/Kagawa Nutrition University

Toshio Shirahata SHAPLA NEER = Citizens' Committee in Japan for Overseas Support

Shunsuke Suzuki AMDA

Toyokazu Nakata i-i-network, Research and Action for Community Governance

Makoto Nagahata Kansai NGO Council

Kazushi Hojo Aspiring Citizens for Community Empowerment with Sunny Smile (ACCE)

Yoshie Muramatsu CARE International Japan

Hiroshi Tanaka The Institute for Himalayan Conservation

Kazunori Miura Office of Evaluation, Planning and Coordination Department

Akihisa Tanaka Office of Evaluation, Planning and Coordination Department

Rina Hirai Office of Evaluation, Planning and Coordination Department

Satoshi Kodakari Office of Evaluation, Planning and Coordination Department

Fumio Imai Partnership Promotion Group, Training Affairs and Citizen Participation Department

Yoshiharu Yoneyama Administration Team, Regional Department I (Southeast Asia)

Makiko Iwasaki Administration Team, Regional Department I (Southeast Asia) (until July 2007)

Residents participating in a gathering forhealthcare education (Ghana RegionalHealthcare Comprehensive ImprovementProject)

Page 9: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

3) Evaluation of JICA ProgramsIn order to implement JICA's cooperation for development

issues more effectively, JICA has been making efforts tointensify its approach to a program that combines a numberof cooperation projects. In concrete terms, to help develop-ing countries achieve the goals of their medium- to long-term development strategies, JICA has been striving to inte-grate a strategic framework with specific goals and coopera-tion scenarios, attaching importance to a combination oforganic inputs and collaboration with other aid agencies,into a “JICA program,” and thus strengthen the programmanagement system.

As part of its efforts, JICA has been developing a methodof evaluating JICA programs. The evaluation of JICA pro-grams is mainly conducted in the following three steps:a. In order to evaluate the relevance of cooperation as a

means to raise effects for solving problems, not only con-sistency of the partner country's strategy with JICA's pro-gram, but also the priority and positioning in the strategyof the partner country are examined.

b. In evaluating a JICA program, consistency and relationsamong constituent elements of the program are examinedwith a focus on its strategic characteristics as well asaccumulation of individual project implementation.

c. Evaluation is made using the concept of “contribution”based on cooperation and collaboration among the part-ner country, Japan, and other donor countries and agen-cies.

In fiscal 2005, JICA evaluated programs implemented inHonduras, Vietnam, and Malawi on a trial basis by using theabove method. Based on the results, JICA evaluated the“Regional Development Program of South Sulawesi” inIndonesia, the “Program for Water Supply in the PovertyArea” in Bolivia, the “Program for the Improvement ofHealth Status of People Living in Upper West Region” inGhana, and the “Healthcare Sector Program” inAfghanistan.3 JICA has also made continued efforts to intro-duce and strengthen its program evaluation in fiscal 2007.

4) Participation in Joint Evaluation with Other DonorCountries and Agencies

In recent years, international society has focused onachieving common development goals in collaboration withdonor countries and agencies, while respecting the owner-ship by developing countries as demonstrated in the UN Mil-lennium Development Goals (MDGs) and the PovertyReduction Strategy Paper (PRSP). Given this background,JICA has been advancing approaches to conducting its eval-uations together with other aid agencies, and in the past con-ducted joint evaluations with the Canadian InternationalDevelopment Agency (CIDA) and U.S. Agency for Interna-tional Development (USAID). From fiscal 2006 to 2007,JICA participated in a joint multi-donor evaluation to assessthe effectiveness of multi-donor cooperation in the Tanzan-ian healthcare sector. It has also participated in evaluation ofthe implementation of the Paris Declaration on Aid Effec-tiveness, an international effort for development, as a mem-ber of the Development Assistance Committee (DAC) of theOrganization for Economic Co-operation and Development(OECD) along with other aid agencies (Box 3). Participat-ing in these joint evaluations will make it possible to conductevaluations from a broad-based perspective, such as thepositioning and effectiveness of JICA's cooperation withinan international framework, as well as mutually sharing thelessons obtained from the evaluations and helping tostrengthen aid donor coordination in the future.

(3) Securing Transparency and Objectivity in Eval-uation

1) Establishment of the Advisory Committee onEvaluation

In fiscal 2002, JICA established the Advisory Committeeon Evaluation, which included external experts from univer-sities, NGOs, and international organizations. The commit-tee has provided JICA with a broad range of recommenda-tions and proposals to enhance evaluation systems, evaluatenew target schemes, and improve methods for feeding backand disclosing evaluation results.

Chapter 1 JICA's Evaluation Activities and Efforts for Expanding and Enhancing Evaluation

17Annual Evaluation Report 2007

EvaluationinJICAPart

1

3 The evaluation results of the four programs conducted in fiscal 2006 are described in Part 3.

The Paris Declaration on Aid Effec-tiveness (hereinafter the Paris Decla-ration) was signed in February 2005by over 100 organizations (such asdonor countries, developing countries,international institutions, and NGOs).The Paris Declaration proposed 12indicators and 56 commitments thatare to be achieved by 2010, based onthe principles of “Ownership,”“Alignment,”* “Harmony,” “Manag-ing for Results,” and “MutualAccountability.”

Since 2006, the OECD-DAC Net-work on Development Evaluation(EVALUNET) has been considered to

conduct evaluation to follow up theimplementation of the Paris Declara-tion together with the DAC WorkingGroup on Aid Effectiveness. At theregular meeting of EVALUNET heldin November 2006, it was decided toimplement case studies on the evalua-tion, focusing on plural donor coun-tries and recipient countries whichwere interested in the activities. Japandecided to offer its support toBangladesh and the Philippinesamong the countries desiring a coun-try program evaluation.

JICA is cooperating in implementa-tion of the evaluation by participating

in the evaluation steering committeecomposed of the Bangladesh govern-ment, representatives of local donors,and a working committee. The evalua-tion focuses on three sectors of prima-ry education, energy and electricpower, and environment, and isintended to verify the progress ofefforts toward achieving the goals ofthe Paris Declaration, as well as thecontributing and inhibiting factors.

* To aid the countries in accordance withtheir administration systems.

3 Partnership with Other Aid Agencies in Evaluation - Follow-up to the ParisDeclaration on Aid Effectiveness

Page 10: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

The committee has also been conducting a secondary eval-uation every year since fiscal 2003 to secure the objectivityof the terminal evaluation carried out by JICA as an internalevaluation. The secondary evaluation results are released inthe Annual Evaluation Report published each fiscal year,

and the recommendations and lessons obtained from thoseresults are used to improve planning and management ofprojects, and evaluation implementation methods.4

As noted above, JICA has attempted to improve andexpand its evaluation system, while incorporating the rec-ommendations and opinions of external experts.

2) Promoting Evaluation by Third PartiesJICA promotes external experts' participation in its evalu-

ation not only to increase objectivity and transparency, butalso to improve the quality of evaluation through use of theirexpertise.

Although JICA conducts project-level evaluation as aninternal evaluation, it is also promoting an approach inwhich evaluation is conducted by external parties (sec-ondary evaluation), in order to secure objectivity and trans-parency of the evaluation results. The results of terminalevaluation conducted upon the completion of projects aresubjected to secondary evaluation by Advisory Committeeon Evaluation as described in 1). With respect to the ex-postevaluation of projects, JICA also requests external partiessuch as experts in developing countries to conduct secondaryevaluation, in order to secure objectivity and quality of itsevaluation (Box 4).

In the thematic evaluation, which serves as a program-level evaluation, JICA tries to secure expertise and objectiv-ity in almost all program evaluations by requesting the par-ticipation of external experts specializing in the fields to beevaluated as evaluation advisors in order to receive adviceon evaluation framework and value judgments. In addition,the evaluation reports are published along with commentsfrom external experts who do not take part in the evaluationprocess. Evaluation requiring particular expert knowledge isentrusted to universities and think tanks specializing in thetheme. Thus, JICA is promoting efforts to implement exter-nal evaluation by third parties.

18 Annual Evaluation Report 2007

Advisory Committee on Evaluation

Table 1-3 Members of the Advisory Committee on Evaluation

Chairperson:Hiromitsu Muta: Executive Vice President for Finance, Tokyo Institute of Technology

Committee Members:Atsuko Aoyama: Professor, Department of International Health, School of Medicine, Nagoya UniversityKiyoko Ikegami: Director, UNFPA Tokyo OfficeAtsuko Isoda: Vice-President, Japan International Volunteer Center; Professor, Faculty of Nutrition, Kagawa Nutrition UniversityTsuneo Sugishita: Professor, Faculty of Humanities, Ibaraki University, formerly employed at Yomiuri ShimbunMasafumi Nagao: Professor, Center for the Study of International Cooperation in Education, Hiroshima UniversityKaoru Hayashi: Professor, Faculty of International Studies, Bunkyo UniversityKanji Hayashi: Manager, International Cooperation Group, International Economic Affairs Bureau II, Nippon Keidanren (Japan Business Federation) Koichi Miyoshi: Professor, Graduate School of Asia Pacific Studies, Ritsumeikan Asia Pacific University

Regarding the ex-post evaluation ofprojects, JICA has had externalexperts of the partner country, who arenot directly involved in the projectsand evaluations, conduct a secondaryevaluation on the results of primaryevaluations by the overseas offices.Here we introduce the secondary eval-uation results of the ex-post evalua-tion of Water Supply TechnologyTraining Improvement Project inEgypt.

[External expert]Dr. Nader K. Wasif, consultant at

Upper Egypt Training Institute

[Summary of secondary evalua-tion result]

The evaluation report is largely sat-

isfactory with respect to the setup ofevaluation questions, evaluationresults, and structure of report. On theother hand, if more quantitative indi-cators had been included and mea-sured in impact evaluation, such asfrequency of water failure, quantity ofwater supply before and after imple-mentation of the project, and numberof technical problems occurring, clear-er answers would have been obtainedfor the evaluation questions. Althoughthese indicators were listed in the Pro-ject Design Matrix (PDM) in a design-ing stage of the project, it was difficultto obtain them under the circum-stances of Egypt. In the present evalu-ation, there was no way other than set-ting up alternative questions and indi-cators. We believe the alternative

questions and indicators were ade-quate under such constrained circum-stances.

In terms of effectiveness to finalbeneficiaries, we should have putemphasis on data concerning waterquality, which was greatly affected bywater sources and water supply.

Since the quality of water suppliedto beneficiaries is influenced more bythe water supply process than thewater purification process, a more pre-cise result of the project's effective-ness could have been obtained by mea-suring water quality at the purifyingstage and the water supply processseparately.

Despite the difficulties in obtainingdata, it is considered as a whole thatthe evaluation was conducted usingthe data effectively.

4 Secondary Evaluation by External Experts (Ex-post Evaluation of Projects)

4 The results of the secondary evaluation for fiscal 2007 are described in Part 4.

Page 11: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

3) Enhancing the Disclosure System of Evalu-ation Results

It is important for JICA to disclose the evaluation resultsof its international cooperation projects/programs in aspeedy and reliable manner in order to achieve accountabili-ty. JICA discloses all the evaluation reports and uploadsevaluation results in a timely manner on its website.5

JICA's website discloses project/program-level evalua-tions, such as summaries of evaluation results, thematicevaluations and evaluations of JICA programs, Annual Eval-uation Reports, and evaluation guidelines. The same infor-mation is available in English on its website. JICA is alsomaking efforts to improve its website by uploading an evalu-ation training material jointly developed with the WorldBank. In fiscal 2006, the website was revised to make it easi-er to browse, with the average number of accesses to theJapanese site registering 2,800 per month, up 300 over theprevious year, and 2,000 per month for the English site, up300 over the previous year.

The results of the Thematic Evaluation are also released tothe public by holding evaluation seminars. In fiscal 2006,JICA held a seminar to release the results of thematic evalua-tions regarding 'community participation' and 'capacitydevelopment of local administration.' A total of 154 partici-pants comprising a wide array of people such as persons con-cerned with development aid, consultants, universityresearchers, and students attended the seminar. In addition toJICA receiving various comments on the results of its evalu-ations from the participants, they actively exchanged opin-ions at this seminar. JICA continuously strives to discloseevaluation results in a fast and easily understandable man-ner.

1-3 Inauguration of New JICAIn October 2008, JICA and the ODA loan divisions of the

Japan Bank for International Cooperation are scheduled tobe integrated, inaugurating New JICA that will become anaid agency providing, in an integrated manner, assistanceunder three ODA schemes: technical cooperation, ODAloan, and grant aid. It is expected that New JICA will gener-ate synergistic effects in all aspects of project planning,implementation, monitoring, and evaluation. With regard toits project evaluation, JICA, along with JBIC, has beenexamining the establishment of coherent monitoring andevaluation systems, based on the project formulation andimplementation system of New JICA towards the integra-tion. When examining new systems, JICA clarifies commonand differing points in the project evaluation methods andevaluation implementing systems of both agencies, studiesproject evaluation systems by overseas aid agencies, andstudies project evaluation systems of other aid agencies andinternational institutions, as reference information, concern-ing the implementation systems and methods of evaluation.

According to the tentative results of the study, it is foundthat both agencies are likely to have common directions andefforts in terms of evaluation, such as establishment of anevaluation system from ex-ante and to ex-post evaluations,introduction of program evaluation, significant concernsover objectivity and transparency of evaluations, the easy-to-understand and prompt disclosure of evaluation results,strengthening of the evaluation feedback system, and utiliza-tion of the DAC's five evaluation criteria. Conversely, thereare differences in evaluation timing, evaluators (internal andexternal evaluations), and evaluation indicators due to thedifferent characteristics of their aid schemes.

Both JICA and JBIC continue to consider building coher-ent evaluation systems throughout the three aid schemes oftechnical cooperation, ODA loan, and grant aid, in consider-ation of evaluations by other aid agencies and results of thisstudy.

Chapter 1 JICA's Evaluation Activities and Efforts for Expanding and Enhancing Evaluation

19Annual Evaluation Report 2007

EvaluationinJICAPart

1

5 The “Evaluation” page on JICA's website is available at <http:www.jica.go.jp/english/evaluation/index.html>.

Page 12: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

The primary objectives of project evaluation conducted byJICA are to ensure accountability to the people, utilize eval-uation results as a tool for project management by feedingthem back into projects, and enhance learning among theparties concerned. With these objectives, JICA deems itimportant to share and accumulate good practices within theorganization using evaluation results in the course ofimproving projects through feedback.

Since 2004, JICA has extended an effort for studying andaccumulating good practices, in which evaluation resultswere utilized for improving the quality of JICA projects. Theevaluation results at the project or program level (recom-mendations/lessons learnt)1 are utilized as follows:a. Planning and operation of individual projectsb. Formulation of JICA cooperation policies by sector and

issuec. Systems for improving project implementation

d. Sharing and systemizing knowledge and experience forproject improvement

In 2007, JICA studied how to use the evaluation results(recommendations/lessons learnt) in the field by conductinga questionnaire survey at the offices in charge of formulatingand implementing technical cooperation projects and devel-opment studies, namely Regional Departments, Develop-ment Issue Departments, and Overseas Offices. In addition,how to use the lessons drawn from past similar projectsdescribed on the project ex-ante evaluation summary sheetsprepared after 2004 was also studied. Table 1-4 lists the pro-jects having applied the evaluation results, which wereextracted from the study.

The following are the good practices in the four sectors ofagriculture, environment, good governance and education,in which evaluation results from a single project or programlevel were utilized for improving the planning and manage-ment of other projects.

Improving JICA's Cooperation UsingEvaluation Results

20 Annual Evaluation Report 2007

Chapter 2

Table 1-4 Project Examples Using Evaluation Results

AsiaIndonesia Sulawesi Capacity Development ProjectCambodia Freshwater Aquaculture Improvement and Extension ProjectSri Lanka Project on Rural Livelihood Improvement in Hambantota District (South CAP)Thailand Project on Anti-Trafficking in Persons Bangladesh Strengthening Primary Teacher Training on Science and MathematicsPhilippines Sustainability Improvement of Renewable Energy Development in Village ElectrificationViet Nam Project of Human Resources Development for Water Sector in the Middle RegionMyanmar The Project on Rural Water Supply Technology in the Central Dry ZoneLaos Project for Improving Science and Mathematics Teacher Training

AfricaEthiopia Groundwater Development and Water Supply Training Center Phase 2Kenya Strengthening of Mathematics and Science in Secondary Education Project (Phase 2)Kenya Project for the Improvement of Health Service with a Focus on Safe Motherhood in the Kisii and Kericho DistrictsZambia Health Capital Investment Plan (HCIP) Support ProjectTanzania Technical Cooperation for Supporting Service Delivery Systems of Irrigated AgricultureTanzania Technical Cooperation in Capacity Development for Regional Referral Health ManagementMadagascar Project for Improvement of Maternal Newborn and Child Health ServiceMalawi The Project for Strengthening of Mathematics and Science in Secondary Education through In-service Training

Oceania Papua New Guinea Promotion of Smallholder Rice Production ProjectFiji In-service Training of Community Health Nurses

Middle East Saudi Arabia Management Plan for Conservation of Juniper WoodlandsTurkey The Project on Strengthening the Program of Expanding Industrial Automation Technologies Department

Latin America Argentine Organizing for the Poverty in Local AreaEl Salvador Enhancement of Technology for the Construction of Popular Earthquake Resistant Housing Costa Rica Project on Sustainable Fisheries Management for the Gulf of NicoyaNicaragua The Project for the Improvement on the Quality of Mathematics Teaching in Primary EducationBrazil The Healthy Municipality Project in Northeast BrazilBrazil The Technological Development Project for Sustainable Agriculture in Eastern AmazonPeru The Project for Strengthening of Educational Management in the Rural Education Networks of Canas and SuyoBolivia Project for Improvement of Health System at Community LevelMexico Assistance for Sustainable Rural Development in Soconuco Region, the State of Chiapas (PAPROSOC-2)

1. Recommendation: Proposal and/or advice extracted from one evaluation result given for a concrete action, for the evaluated project or implementa-tion of a relevant project. Lesson: An item that is generalized to some extent, derived from an evaluation result. It is reflected in projects other than a target project, develop-ment programs, and the formation of assistance strategies.

Page 13: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

Chapter 2 Improving JICA's Cooperation Using Evaluation Results

21Annual Evaluation Report 2007

EvaluationinJICAPart

1

2-1 AgricultureIn the agriculture sector, the evaluation results of the

“Training Services Enhancement Project for the Rural Liveli-hood Improvement in the Philippines (1996-2001)” were uti-lized in several similar projects (Table 1-5).

This project was intended to enable the Agricultural Train-ing Institute (ATI) in the Philippines to implement effectivetraining for rural life improvement2, reflecting their traditionsand needs of the local people. The project was implemented ina model village in Bohol province for the first three years,focusing on pilot activities for livelihood improvementincluding the necessary training for farmers and extensionofficers. Based on the pilot activities, a training manual forlivelihood improvement in rural areas was compiled for ATIofficials, and from 1999 the pilot activities and related train-ing were extended to three other locations in Bohol province.

The project was unique because the training program wassimultaneously improved with the implementation of pilotactivities. The improvement was very successful by reflectingexperiences from the pilot activities. The Terminal Evaluationsurvey conducted in February 2001, the experiences of thepilot activities concluded that input of a project must be decid-ed after clarifying the scope of burdens that beneficiaries couldbear. Thus, the lesson where “input into a project activity mustbe decided, considering the economic scale of beneficiariesand economic effect exerted by the pilot project before itsimplementation” was learned. Another lesson was learnedfrom the project. In the initial plan, ATI was the institutionresponsible for all activities of the project, but it turned out tobe difficult that ATI as a training institution kept shoulderingthe responsibility of livelihood improvement activities evenafter the completion of project cooperation. This drew the les-son where “a framework and concept of a project must bedecided, considering the mandate (scope of service) and per-sonnel of an implementation institution in the planning stage.”

Projects applying the lessons“The Development and Promotion of Location-specific

Integrated High-yielding Rice and Rice-based Technologiesin the Philippines (2004-2009)” is a project that follows aseries of assistance3 provided by the Government of Japan forthe Philippine Rice Institute (PhiRI) founded in 1985. As aresult of the series of cooperation, research and developmentof rice capacity made tremendous progress in the Philippines.And now, it is in the next stage to disseminate rice technolo-gies applicable to the conditions of local farmers in eachregion by modifying and verifying the developed rice technol-ogy. This project, succeeding the outcomes from the previoustwo technical cooperation projects determined the project tar-get as “increasing rice productivity among the target farmers,”particularly intending to practical application and dissemina-tion of the developed technology to the farmers' level.

In this project, technical packages comprising a combina-tion of farming machines, cultivers and farming technologysuitable to the target areas were developed, in collaborationwith local farmers on demonstration farms. The lesson "of

considering the economic scale of beneficiaries when a pro-ject is implemented" learned from the “Training ServicesEnhancement Project for Rural Livelihood Improvement” wastaken into the design of this project. Then, this project figuredout the economic scale of the beneficial farmers through abaseline survey and a technology development activity in theexperimental rice fields with the participation of farmers. As aresult, farming systems even affordable to small-scaled farm-ers were developed.

Another lesson "of considering the mandate and manpowerof the implementing institution" obtained in the project wastaken. This project was designed to be implemented by PhiRIin close cooperation with local governments in accordancewith their own mandate. PhiRI is in charge of developing therice cultivation system; local governments are responsible fordisseminating the developed system. It is expected to producea better outcome and secure sustainability, by allowingrespective institutions to cooperate in line with their primaryresponsibilities.

In September 2007, a mid-term evaluation of this projectwas conducted. There was a result where the productivity offarmers who had adopted “low-input and area adaptive farm-ing system” increased. It is expected that more farmers willapply the farming system in the future.

2-2 Environment In the environment sector, lessons learned from a program-

level evaluation known as a thematic evaluation "Environ-mental Center Approach: Development of Social Capacity forEnvironmental Management in Developing Countries andJapan's Environmental Cooperation" (2003) (hereafter called“ECAe”) are being applied to improve the planning andimplementation of the projects listed in Table 1-6.

Activity together with farmers in an experimental field in the Philippines

2. A concept of “Rural Livelihood Improvement” in the project; Qualitative perspectives, such as labor, nutrition and living environment, were added tothe ATI's conventional perspectives of “increase in agricultural productivity and income.”

3. The Grant Aid (assistance for facilities and equipment) totaling 2.26 billion yen during 1989 and 1991, and two technical cooperation projects imple-mented thereafter: Project of Rice Institutes (1992-1997) and Project for High-yielding Rice and Rice-based Technologies (1997-2002).

Philippines Development and Promotion of Location-Specific Integrated High-Yielding Rice TechnologiesPhilippines Rice-Based Farming Systems Training and Support Program for the ARMMTurkey Improvement of Livelihood for Small-Scale Farmers in Eastern Black Sea Region

Table 1-5 Examples of Projects Where the Evaluation Resultsof “Training Services Enhancement Project for Rural Life Improvement in the Philippines” Were Fed Back

Page 14: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

“ECAe” evaluated the outcomes of JICA EnvironmentalCentre projects in four countries (Indonesia, China, Thailandand Mexico) as is typical of JICA's environmental coopera-tion. “ECAe” was within the evaluation framework of con-tributing to the development of social capacity for environ-mental management4 with a variety of viewpoints.

“ECAe” proposed the following lessons, as a way of moreeffective and efficient environmental cooperation.

(1) Clear positioning of an environmental center in theenvironmental administration system so that the cen-ter may exert impact.

(2) Definitions of commencement period and completionperiod: optimum commencement period of a project iswhen environmental laws and environmental adminis-tration are established; desired completion period of aproject is when environmental pollution enters areduction cycle.

(3) Increased impact on major players in a system, bystrengthening ties with enterprises and citizens.

(4) Support for improving the environment managementcapacity of local sectors, in a trend toward decentral-ization.

Case Examples of Applying the LessonsThe lessons have been applied to "the Capacity Develop-

ment of Environmental Monitoring at Directorates for Envi-ronmental Affairs in Governorates in Syria" (2005-2008).Since the 1980s, Syria's environmental issues have beenaggravated along with its industrialization, and the impact hasbeen threatening citizens, such as the pollution of drinking

water and food caused by water contamination, and respirato-ry disorders by air pollution. In addressing these issues, theSyrian government introduced environmental laws since1991, such as the Basic Environment Law and the Environ-mental Protection Law, and established the Directorates forEnvironmental Affairs (DFEAs) in all 14 provinces as region-al environmental bureaus for environmental monitoring byJanuary 2004. However, DFEAs struggled to deal with theissues due to a shortage of technological capacity and equip-ment, particularly with regard to environmental monitoring,although exercising its responsibilities for environmentaladministration, environmental monitoring and awareness-building activities to citizens.

Given this background, the project was initiated with thepurpose of regular environmental monitoring, accumulationand management of analysis data, and capacity upgrade thatmade it possible to conduct citizens' awareness programsincluding the disclosure of measurement results, in accor-dance with the plan prepared by DFEAs themselves.

The project applied lessons (1) and (4) from the “ECAe” inits planning stage and implementation, in order to strengthenthe capacity of DFEAs at the provincial level. The effort con-forms to the environmental administration policy of the Syri-an government, and attempts to strengthen the environmentalmanagement capacity at the local level for alleviating nation-wide environmental issues. DFEAs were close to the citizensand in a good position to make them aware of the environmen-tal issues. The project activities to improve the capacity ofDFEAs had a positive impact in that: some requests from thecitizens were fulfilled utilizing the analysis data from the labs;the basis of water/air quality analysis was established; and cit-izens showed greater concerns about environmental issues.

By applying lesson (3), the project introduced collaborativeactivities with enterprises as major players in the system. Inthe project, four leading DFEAs held awareness seminars andworkshops targeting industrial sectors. It turned out in the ter-minal evaluation survey conducted in 2007 that the seminarsand workshops contributed to raising awareness, such thatseveral factories built a wastewater treatment facility, factoryowners understood observance of the Environmental Law andnecessity of environmental monitoring, and awareness aboutthe importance of environmental conservation was raisedamong enterprises taking part in the workshops. It is expectedthat this kind of collaboration with enterprises will be promot-ed, thus leading to awareness and environmental educationactivities for citizens in the future.

2-3 GovernanceIn the governance sector, the lessons learned from the evalu-

ation of “Strenthening Sulawesi Rural Community Develop-ment to Support Poverty Alleviation Programs” in Indonesia(1997-2002) (hereafter called “the Project”) are applied toplanning/implementation of a plurality of projects (Table 1-7).

The Project implemented various activities in four modelvillages in Taklar prefecture in South Sulawesi Province, withthe target of developing a participatory social developmentmodel applicable in South Sulawesi Province. The activitiesinclude development of the administration supporting system

22 Annual Evaluation Report 2007

Table 1-6 Examples of Projects Where the Results of “Thematic Evaluation: Environmental Center Approach” Were Fed Back

Philippines Capacity Development Project on Water Quality ManagementKenya Improvement of Environmental Management Capacity in Nakuru CitySyria Capacity Development of Environmental Monitoring at Directorates for Environmental Affairs Guatemala Water Environment Improvement in Metropolitan AreaMexico Strengthening of Air Monitoring Program

Counterparts receiving instructions from an expert (Syria)

4. The capacity of a society in coping with environmental issues on its own is called social capacity for environment management. The social capacityis defined as an operating capacity of a social environment management system, which is composed of governments, enterprises and citizens underthe national-local relationship.

Page 15: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

Chapter 2 Improving JICA's Cooperation Using Evaluation Results

23Annual Evaluation Report 2007

EvaluationinJICAPart

1

(SISDUK) as a support for participatory rural communitydevelopment based on the needs of the communities, develop-ment and implementation of a training program called “Partic-ipatory Local Social Development” (PLSD) aimed at foster-ing human resources who would be engaged in SISDUK, andpromotion of rural community development with the partici-pation of communities under SISDUK. The prefectural gov-ernment and its assembly recognized the effectiveness of themodel, under which local communities and administrationworked in collaboration, and thus enshrined SISDUK into lawas a prefectural ordinance before completion of the project. Asa result, SISDUK was planned to be implemented in 73 vil-lages across the prefecture, on the budget of the prefecturalgovernment when the Project was completed in 2002. Further,the Project exerted a positive effect on neighboring prefec-tures, and similar projects have been expanding accordingly.

The Project has received recognition with its high sustain-ability and impact. The reason lies in the fact that the Projecttook a plenty of time for building the collaborative model andthe mechanism, asking a wide range of stakeholders includingNGOs and local universities for cooperation, with establish-ment and dissemination of the model in mind from the verybeginning. The lessons learned from the terminal evaluationof the Project include items relating to social arrangements,involvement of a wide range of stakeholders and humanresource development, prefabrication of a mechanism thatenables establishment and dissemination of the model, pointsin selection of model villages, and efforts for sustainability inthe developed model. These lessons are applied to other pro-jects relating to governance, rural community developmentand poverty reduction.

Case examples of applying lessonsThe Project for Improvement of Public Administration for

Local Governments in Punjab in Pakistan (2004-2007) setfour outputs, aimed at the operation of administrative servicewith high transparency and efficiency, through operationalimprovement regarding the CCB project5 and capacity devel-opment of administrative officers. The four outputs were:comprehension of needs and problems facing local communi-ties, comprehension of improvement points in local adminis-trative management, formulation of the CCB project improve-ment plan, and acquisition of techniques and knowledge nec-essary for the project activities.

The CCB project was expected as a new administrative ser-vice that would encourage participation of community organi-zations for regional development. However, it did not func-tion as expected because a specific implementation procedureof the CCB project was not established at the local govern-ment level, there was serious miscommunication between thecommunities and administration regarding implementation ofthe projects, and execution of the CCB budget was delayeddue to inadequate procedures within the local government.

Since the CCB project and SISDUK were participatorydevelopment models, and had resemblance in the concept thatthe communities and administration work in collaboration, thelessons of Sulawesi were applied to the planning/designingand implementation of the project in Punjab.

The CCB project paid attention to the “social arrange-ments” and allocated relatively longer time for allowing theofficers to comprehend the situation, needs and problems ofthe communities during the first two-and-half-years. In theprocess, the officers deepened their understanding of the com-munities, and changed their attitude.

From a standpoint of the “involvement of a wide rage ofstakeholders and human resources development,” the projectheld a series of meetings with the local NGOs, private enter-prises, hospitals and schools to promote public awareness onthe CCB project and request their cooperation. The projectfostered local coordinators as a bridge between the communi-ties and the officers, and incorporated them on a trial basis.This attempt turned out to be effective for promotion of theproject, as the coordinators played a supportive role in com-prehension of the communities' needs, preparation of propos-als and others. In addition, their roles contributed tosmoothening the flow of information between both parties,and thus proved effective for improving coordination betweenthe local administration and the communities. The coordina-tors played a similar role as NGOs did in Sulawesi.

The project designated ten villages placed under differentsituations as model sites, with the lesson of “establishmentand dissemination of the model” in mind, when the communi-ties implemented the CCB project. This helped build a univer-sal model applicable across Punjab province, and led to raiseeffectiveness in dissemination of the CCB project.

5. CCB (Citizen Community Board) refers to the newly established system, based on the Local Administrative Law enacted August 2001, for realizingthe idea of development and policy-making in a bottom-up principle. The government provides financial aid for regional development activities (CCBactivities) conducted by local residents' organizations.

Local residents at the meeting of the Critizen Community Board(CCB) in Pakistan

Governance SectorIndia Conservation and Wise-use of Natural Resources of Chilika Lagoon through Community ParticipationIndonesia Sulawesi Capacity Development Project

Other than Governance SectorSri Lanka Technical Cooperation Project for Agricultural and Rural Development for Rehabilitation and Reconstruction through Community Approach Project in Trincomalee Sri Lanka The Project on Rural Livelihood Improvement in Hambantota District (South CAP) Pakistan Improvement of Public Administration for Local Government in PunjabMyanmar The Project on Rural Water Supply Technology in the Central Dry ZoneMyanmar The Eradication of Opium Poppy Cultivation and Poverty Reduction in Kokang Special Region No. 1

Table 1-7 Examples of Projects Where the Evaluation Results of the “Sulawesi Rural Community Development Project” Were Fed Back

Page 16: Part 1 Evaluation in JICA€¦ · Part 1 Evaluation in JICA. 1-1 JICA's Evaluation Activities (1)Objectives of Evaluation JICA's project evaluation is carried out at each stage of

2-4 EducationIn the educational sector, JICA introduces case examples

where lessons learned from “Synthesis Study6: Elementary/Secondary Education in Science-Math Sector” (2004) (here-after called “the Study”) were applied to other projects. Asshown in Table 1-8, the lessons learned from the Study wereapplied to designing and formulating a plurality of projects inthe education sector.

In the Study, JICA conducted cross-sectional analysis,focusing on evaluation results of 12 projects in eight coun-tries.7 The evaluation drew out lessons that were common toall of the projects, in view of (1) planning and designing, (2)dissemination means of output, (3) coordination, (4) institu-tionalization and (5) monitoring and evaluation, as fiveimportant elements that influenced success of a project in theelementary/secondary science-math sector. For example, (2)proposes establishment of a mechanism that allows develop-ment of teaching materials that correctly reflect school needs,involvement of a wide range of stakeholders aimed at ripplingoutputs, and transmission of information, (3) proposes posi-tioning relevant institutions for increasing coordinationeffects with donors and other institutions, and specification ofissues that must be dealt with, and (4) proposes improvement-oriented attitude which attaches importance on improvingexisting training systems rather than establishing new sys-tems, in view of feasibility and sustainability.

Case examples of applying lessonsThe lessons learned from the Study were applied to project

formulation of the “Teaching Methods Improvement Projecttowards Children's Development” in Mongolia (2006-2009).

With introduction of a new education standard in 2005 dueto a reform in the educational sector, new curriculums and sub-jects, such as integrated study and natural science (integratedscience), and a child-centered teaching method were intro-duced in Mongolia. However, many teachers were unable tostep out from a conventional rote teaching method, in whichteachers kept “teaching” one-sidedly, and had little under-standing of how to compose a class for new subjects such asintegrated study and how to teach them, as well as how todevelop and use teaching materials for science and math. Thisinduced confusion in the classroom.

Given the background, the project is implementing develop-ment of tutorial manuals for teachers, introduction of newteaching methods, development of monitoring methods, andactivities regarding capacity improvement of the project stake-holders (Teaching Method Development Center, municipal/prefectural supervisors for school education, and principalsand teachers in model schools). These activities are imple-mented with a project target of developing teaching methodsthat support children's development, in accordance with thenew standard for basic education, assigning the TeachingMethod Development Center as the implementation institutionestablished for each subject of elementary education, matheducation, IT education and science education.

In formulation of the project, JICA followed lesson (2)drawn out of the Study, and adopted a mechanism of develop-ing tutorial manuals that reflected the school needs correctly,by getting municipal/prefectural school supervisors andincumbent teachers involved as the parties most familiar withschool, and a mechanism of disseminating adequate teachingmethods to educators including managing teachers in modelschools through workshops. It also planned to implementexplanatory meetings on new teaching methods for parents,issue newsletters and hold open classes, as a mechanism ofinvolving local residents in order to establish new teachingmethods in the region. In application of lesson (3), JICAdefined the positions of a wide range of project stakeholdersand their roles by considering role-sharing with other majordonors including the Asia Development Bank (ADB) in theeducation sector.

As a result of the above-described efforts, tutorial manualsfor teachers were prepared for eight courses in four subjects,and distributed to throughout Mongolia with help from ADB.The project is scheduled to keep developing teaching methodsand tutorial manuals, and put emphasis on dissemination/establishment of the teaching methods developed with theMongolian Ministry of Science, Technology, Education andCulture in the latter half of the project. In conducting the dis-semination/ establishment practices, JICA refers to lesson (4),and purposes introducing/disseminating the tutorial manualsdeveloped by the central government into local governments,utilizing existing training organizations for incumbent teach-ers. Use of the existing organizations leads to secure owner-ship of educators in Mongolia in practicing the disseminationof tutorial manuals. It is expected that the efforts will be sus-tained after completion of the project.

24 Annual Evaluation Report 2007

Table 1-8 Examples of Projects Where the Evaluation Results of “Synthesis Study: Elementary/Secondary Education and Science-Math Sector” Were Fed Back

Laos Project for Improving Science and Mathematics Teacher TrainingBangladesh Strengthening Primary Teacher Training on Science and Mathematics Myanmar Strengthening Child Centered ApproachMongolia Teaching Methods Improvement Project towards Children's DevelopmentNiger Strengthening of Mathematics and Science in Secondary Education Malawi The Project for Strengthening of Mathematics and Science in Secondary Education through In-service TrainingHonduras Project for the Improvement of Teaching Method in Mathematics Phase 2Papua New Guinea Project for Strengthening Long Distance Education

(“SMASSE Niger”)

Experts and counterparts discussing teaching methods for integratedstudy class in Mongolia

6. The synthesis study is a study that JICA conducts as part of the thematic evaluation, for drawing out generalized lessons that are likely to be fedback to other JICA programs and projects. It is conducted by collecting evaluation results from a plurality of projects regarding specific issues or sub-sectors, and analyzing common trends and issues, and good practices obtained after comparing a plurality of projects.

7. Eight countries of the Philippines, Indonesia, Cambodia, Egypt, Kenya, South Africa, Ghana and Honduras.


Recommended