Ex-Post Project Evaluation 2010: Package III-3 (Mauritius, Tanzania)

December 2011

JAPAN INTERNATIONAL COOPERATION AGENCY

EVD JR 11-39

WASEDA UNIVERSITY

Preface

Ex-post evaluation of ODA projects has been in place since 1975 and since then the coverage of evaluation has expanded. Japan’s ODA charter revised in 2003 shows Japan’s commitment to ODA evaluation, clearly stating under the section “Enhancement of Evaluation” that in order to measure, analyze and objectively evaluate the outcome of ODA, third-party evaluations conducted by experts will be enhanced. This volume shows the results of the ex-post evaluation of ODA Loan projects that were mainly completed in fiscal year 2008, and Technical Cooperation projects and Grant Aid projects, most of which project cost exceeds 1 billion JPY, that were mainly completed in fiscal year 2007. The ex-post evaluation was entrusted to external evaluators to ensure objective analysis of the projects’ effects and to draw lessons and recommendations to be utilized in similar projects. The lessons and recommendations drawn from these evaluations will be shared with JICA’s stakeholders in order to improve the quality of ODA projects. Lastly, deep appreciation is given to those who have cooperated and supported the creation of this volume of evaluations.

December 2011

Masato Watanabe Vice President

Japan International Cooperation Agency (JICA)

Disclaimer

This volume of evaluations, the English translation of the original Japanese version, shows the result of objective ex-post evaluations made by external evaluators. The views and recommendations herein do not necessarily reflect the official views and opinions of JICA. JICA is not responsible for the accuracy of English translation, and the Japanese version shall prevail in the event of any inconsistency with the English version. Minor amendments may be made when the contents of this volume is posted on JICA’s website.

JICA’s comments may be added at the end of each report when the views held by the operations departments do not match those of the external evaluator. No part of this report may be copied or reprinted without the consent of JICA.

Republic of Mauritius

Environmental Sewerage And Sanitation Project

External Evaluator: Takeshi Daimon, Waseda University 0. Summary

This project has been extended with the objective of “preventing water pollution in the metropolitan area and protecting the oceanographic ecosystem.” The project indeed matches the development policies, development needs, and Japanese development policies, and is highly relevant. Additionally, efficiency is fair, because the project cost is within what was planned, but the project period far exceeded what was planned. BOD and other operating indicators meet the planned level and have contributed to the improvement of environmental and sanitation conditions; however, the effectiveness is fair, because the operating level is lower than the planned level by approximately 30%. Sustainability is an area of concern, because, although the overall system and technology are fine, there is a shortage of revenue in recovering Operations and Maintenance (O&M) costs and because some of the non-functioning facilities are left unrepaired for a long time.

In light of the above, this project is evaluated to be partially satisfactory.

1. Project Description

Project Location（Star）

Treatment Plant: Pond (Upper Right) and Trash Filter (Lower Right)

2

1.1 Background Since independence, and especially since the 1980s, Mauritius has experienced environmental

problems owing to economic development and demographic expansion. The textile industry has been expanding rapidly, and demographic concentration in Port Louis, the capital city, has been prominent. In addition, household and industrial wastewater is currently discharged generally without any proper treatment into the lagoons and oceans around the island, threatening the tourism industry (which is a major source of earnings in foreign currency) and the livelihoods of artisanal fishermen with oceanic pollution. The standard of living in terms of per capita income has increased, but the quality of life was adversely affected by increased public health and sanitation problems including intestinal and eye diseases.

The demand for wastewater treatment in Port Louis was expected to increase from 25 thousand m3/day (1997), through 48 thousand m3/day (2005), to 61 thousand m3/day (2007). This exceeds the existing capacity of wastewater treatment plants (17 thousand m3/day) in Fort Victoria and Pointe aux Sable, both of which were in a deteriorated condition and require immediate replacement with new treatment systems. Meanwhile, the Government of Mauritius requested the Japanese Government to help establish a new wastewater treatment system in the Montagne Jacquot area, which was located six km south-west of the city. Wastewater was to sent to this facility through compression pipes from new pumping stations, covering 1,340 ha (or a population of 118 thousand, as of 1997) in residential and commercial districts. 1.2 Project Outline

The project aims to prevent water pollution and to protect marine biodiversity by establishing a new sewerage system in southern part of the capital city Port Louis, thereby contributing to the improvement of living standards as well as to the development of tourist industry.

This project is co-financed by the World Bank.

Loan Approved Amount/ Disbursed Amount

4,538 million yen / 4,533 million yen

Exchange of Notes Date/ Loan Agreement Signing Date

September 1998 / September 1998

Terms and Conditions Interest Rate: 1.8％

Repayment Period: 25 years (Grace Period: 7 years)

Conditions for Procurement: General Untied

Borrower / Executing Agency(ies) Government of the Republic of Mauritius/ Wastewater Management Authority1

1 The executing agency was originally Waste Water Authority, Ministry of Public Utilities, at

3

Final Disbursement Date December 2008

Main Contractor (Over 1 billion yen) China International Water and Electric Corporation (Chinese national) Jan De Nul Dredging Limited (Belgian national)

Main Consultant (Over 100 million yen) Nippon Jogesuido Sekkei Co., Ltd.

Feasibility Studies, etc. M/P (AfDB, 1994) F/S, D/D (World Bank, 1997)

Related Projects (if any) Technical Assistance: Short Term Experts (3) Yen Loan: Grand Baie Sewerage Project (L/A signed in July 2010) International Organization: Environmental Sewerage and Sanitation Project (Accelerated Co-financing Facility)

2．Outline of the Evaluation Study 2.1 External Evaluator Takeshi Daimon, Waseda University 2.2 Duration of Evaluation Study Duration of the Study: December 2010 – December 2011 Duration of the Field Study: February 27 to March 12 and July 31 to August 6, 2011. 2.3 Constraints during the Evaluation Study (if any)

Regarding Operations and Maintenance (O&M), the executing agency of the Wastewater Management Authority (WMA) does not have a separate account of revenues and expenditures of treatment plants, making it difficult to conduct accurate financial analysis (re-calculation of FIRR) from the project finance perspective.

3．Results of the Evaluation (Overall Rating: C2) 3.1 Relevance (Rating: ③3) 3.1.1 Relevance with the Development Plan of Mauritius

the time of appraisal, and has been corporatized as WMA since August 2001. 2 A: Highly satisfactory, B: Satisfactory, C: Partially satisfactory, D: Unsatisfactory 3 ③: High, ② Fair, ① Low

4

In 1990, the Government of Mauritius (GOM) approved the National Environmental Action Plan (NEAP), resulting from the World Bank assistance since 1988, and established the National Environmental Committee. The wastewater sector, in particular, has been recognized as a high priority area, and the Master Plan (M/P) was drafted in 1994 that has actually been financed by the African Development Bank (AfDB). The M/P lists the major policy goals of the wastewater sector as “prevention of water pollution in island and oceanic areas,” “improvement of people’s health and public health,” and “establishment of technological, legal, institutional, and financial framework in order to achieve sustainable growth in the sector.”

The NEAP was modified to the NEAP2 (2000–2010 Action Plan) as a part of the National Environmental Strategies. The NEAP2 was taken over by the Maurice Ile Durable (MID) Plan (NEAP3 Action Plan), which was effective at the time of post-evaluation. The importance of the wastewater sector continues to be recognized from the perspective of benefits to the tourism and fishery sectors through the prevention of seashore pollution.4

Hence, the aim of “preventing water contamination in metropolitan areas and promoting the oceanographic ecosystem in surrounding areas” has been overall relevant. 3.1.2 Relevance with the Development Needs of Mauritius

Until the time of appraisal (November 1997), owing to the textile-based industrialization and demographic concentration in Port Louis since independence, household and industrial wastewater was discharged into the lagoons and in the ocean without proper treatment. The consequent ocean pollution has threatened tourist resources and has adversely affected the health and sanitation situation in nearby areas, as evidenced by an increase in intestine and eye diseases. The demand for wastewater treatment in Port Louis was expected to increase from 25 thousand m3/day (1997), through 48 thousand m3/day (2005), to 61 thousand m3/day (2007), exceeding the existing capacity of wastewater treatment plants (17 thousand m3/day). All existing plants were in a deteriorated condition and need immediate replacement with new treatment systems.

As discussed later in the Effectiveness section, at the time of post-evaluation, the wastewater treated was 31 thousand m3/day (measured in 2009), lower than the demand originally planned. However, the demand for treatment still exists from the development perspective. It is pointed out that the reason to fall below the demand forecasting is a deceleration of growth in the textile industry. The surplus treatment capacity is being matched by the WMA’s efforts to increase the number of network connections. 3.1.3 Relevance with Japan’s ODA Policy

The appraisal document makes no mention about the consistency of Japanese development policies and this project. Hence, this can only be estimated through the circumstantial evidence from

4 Interviews with the Ministers of Environment and of Fishery as well as the “Maurice Ile Durable” Green Paper.

5

that time. For example, the ODA White Paper (FY 1999) stressed the importance of achieving diplomatic goals such as “strengthening of bilateral friendship,” “securing of support and cooperation for Japan in multi-lateral diplomacy,” and “contribution to African issues (promotion of socio-economic development, settlement of conflicts, and assistance for humanitarian aids).” The report also mentions that Mauritius has a relatively high income and that assistance must be extended “in order to support economic stabilization.”

This project matches the overall diplomatic goals of “bilateral friendship” as well as “African issues (especially environmental problems) in Japanese assistance policies for Africa.” Additionally, the project is consistent with Japanese assistance policy in the sense that this assistance has been extended to “support economic stabilization.”

This project has been highly relevant with the country’s development plan, development needs,

as well as Japan’s ODA policy, therefore its relevance is high.

3.2 Efficiency (Rating: ②) 3.2.1 Project Outputs

As the table below shows, there was no difference between the planned and actual outputs for either the physical component (the target of the ODA loan being a consulting service and the wastewater treatment system without the compression pipeline) and technical assistance (which is not covered by the ODA loan but by the World Bank loan).

Table 1 Comparison of Original and Actual Outputs Item Original Actual

Pumping Station Force Main (*) Waste Water Treatment Plant Sea Outfall Consulting Services Technical Assistance (*)

2 6 km

Treatment Capacity 48,000 m3/day 645 m long, 30 m deep

Procurement Preparation and Supervision

Management and Staff Trainig, etc.

As planned As planned As planned As planned As planned

As planned

(Source: PCR, hearing during post-evaluation) * World Bank-financed portion (non-ODA Loan)

The Environmental Impacts Assessments (EIA) conducted in 1997 and 2001 decided that the outflow of treated water would not be disposed of through an underground borehole (digging boreholes for water outfall), but through “sea outfall” (645 m offshore, 30 m depth undersea), and that the outflow would be disinfected (for bacteria, etc.) through chlorination in addition to the originally planned primary treatment (elimination of suspended particles and metals). These modifications and addition will be discussed in “3.2.2.2 Project Period” and “3.4.2 Other Positive and Negative Impacts.”

6

3.2.2 Project Inputs 3.2.2.1 Project Cost

The project cost increased (to 126% of the original cost at the time of the appraisal)5 for the treatment plant, pumping station, and outfall. This increase can be attributed to the price escalation of steel and other materials owing to the delay in construction.

On the other hand, the costs of the consulting service and the World Bank’s technical assistance have decreased by 58% from that quoted at the time of the appraisal6. This is due to a reduction in the contract amount for technical assistance in the ODA loan and the World Bank’s financing.

Further, Mauritian expenditure in domestic currency, such as spending on the pre-treatment plant and land acquisition, has decreased by 66% (in USD equivalent)7 from the time of the appraisal. This is partially because government transfers were not made for big clients, such as factories (these facilities financed their own construction), but mostly because there has been a 45% drop in the value of the local currency against the dollar and a 65% drop against the Japanese yen.

As a result, the total project cost was realized as 61.0 million USD (or 7,015 million JPY at the average exchange rate during the lending period), as compared to the appraised 63.7 million USD (or 7,708 million JPY with the exchange rate at the time of appraisal). In USD, the cost was only 96% (or 91% in JPY) of the appraised amount.

Table 2 Comparison of Original and Actual Project Costs Item Original Actual

Amount paid in Foreign currency

5,155 million yen

4,658 million yen

Amount paid in Local currency

2,553 million yen (442 million MUR)

2,367 million yen (628 million MUR)

Total 7,708 million yen 7,015 million yen Japanese ODA loan portion 4,538 million yen 4,533 million yen Exchange rate 1 USD = 121 yen

1 USD = 21.1 MUR (As of December 1997)

1 USD = 115 yen 1 USD = 30.5 MUR

(Average between January 2005 and December 2007)

(Source: PCR) (Exchange rate used for post-evaluation is taken from average during lending period). 3.2.2.2 Project Period

Although the construction period had been planned from October 1999 to December 2002 (39 months), it actually started in March 2005, after a five-year delay; the construction period ended in January 2007 (23 months). Despite the construction period having been shortened by 16 months, the delay in the start date resulted in 99 months8 for the project period, or 194% of the planned 51 months from the signing of the loan agreement. 5 39.1 mil. USD (at appraisal), as compared to 49.1 mil. USD (at completion). 6 14.3 mil. USD (at appraisal), as compared to 9.5 mil. USD (at completion). 7 4.2 mil. USD (at appraisal), as compared to 2.4 mil. USD (at completion). 8 This has resulted in the extension of final disbursement date from December 2004 to December 2008.

7

One of the reasons for this delay is the WMA’s move to modify the appraised scope of the project and introduce an alternative plan (namely, sea outfall instead of disposal of the wastewater through boreholes). This was partially owing to the frequent protests and demonstrations by coastal fisheries’ fishermen. The alternative plan was rejected by the Ministry of Environment in January 2001; however, it delayed the entire process of the approval (December 1998), modification (October and December 2001), and evaluation (July 2002) of P/Q documents by three and half years.

Another reason for the delay is the lawsuit against the treatment plant and pumping station. When the first bidder was negotiating a contract with the WMA, the second bidder, unsatisfied with the results, went to court to request the suspension of bidding.9 The second bidder was eventually offered the construction contract.10 This process resulted in the delayed approval of the bidding result in February 2004. The sea outfall went through a separate bidding process, with the announcement of the bid in February 2003, evaluation in May 2003, and approval of the evaluation (in which a Belgium company was offered the bid) in February 2004.

Although the project cost was within the plan, the project period was exceeded, therefore efficiency of the project is fair.

3.3 Effectiveness 11(Rating: ②) 3.3.1 Quantitative Effects 3.3.1.1 Results from Operation and Effect Indicators

As an operating indicator, a “daily average inflow” of 48,075 m3/day (thereafter, this design water flow was assumed to be maintained) 12was introduced. In reality, however, as the table below shows, in 2007, the daily average inflow (measured) was 32,714 m3/day (or 68% of that planned at the time of appraisal); by 2009, this ratio had not increased, and continued at the same level.

Additionally, the WMA considers the number of connections to the wastewater as an operating indicator, although it was not part of the operating indicators at the time of appraisal. As the table below shows, the higher number of connections does not match the stagnant amount of treated water, because, according to the WMA, Mauritius has suffered from a chronic shortage of water since 2007 and there has been a supply cut in potable water. In addition, there was not an expected increase of large clients, owing mainly to the external factors such as closure of textile factories in the sewerage covered area in light of the decelerated growth of textile industry.

9 Filing a lawsuit was the only way to disagree with the procurement procedure before Public Procurement Act of 2006 was promulgated, making it possible to appeal to Independent Review Panel. This made the procedure a lot shorter than the lawsuit. 10 This is based on the information obtained during the interview with the consultant (NJS) although no information was provided from WMA. 11 Rating takes into account of Impact as well. 12 “Maximum daily flow” by design is 187,500 m3/day.

8

Table 3 Comparison of Original and Actual Operating Indicators Original Actual*

2006 2007 2008 2009 Average Daily Flow (m3) 48,075 32,714 30,191 31,096 Number of Connection N/A 2,145 2,545 2,845

(Source: WMA) (* Actual figures are annual average. The sewerage system started operational in January 2007,

and there is no actual data in 2006.) For effectiveness indicators, on the other hand, Biochemical Oxygen Demand (BOD)13, Chemical

Oxygen Demand (COD)14, and Total Suspended Solids (TSS)15 have been used, with COD 580 mg/l,

BOD 340 mg/l, and TSS 400 mg/l（as of 2006）set as goals (measured at the outflow point). These figures are consistent with the “Environment Protection (Standards for effluent discharge into the ocean) Regulation 2003.”16

Table 4 Comparison of Original and Actual Effectiveness Indicators

Standard Original Actual*

2006 2007 2008 2009

COD 750 580 446 252 326

BOD 250 340 83 142 152

TSS 300 400 94 100 100

Source: WMA (*Actual figures are bi-monthly average).

3.3.1.2 Results of Calculations of Internal Rates of Return (IRR) (1) Economic Internal Rate of Return (EIRR): At the time of appraisal, it was decided that the EIRR would not be calculated “because, in the wastewater project, as in education, health, and other social infrastructure, it is difficult to quantify the benefits,” 1718making it impossible to conduct before-after comparisons. (2) Financial Internal Rate of Return (FIRR): Using post-project tariff revenues, actual O&M data (2007–2009)19, and the same assumptions20 made at the time of the appraisal, FIRR was re-calculated, 13 BOD is the amount of dissolved oxygen needed by aerobic biological organisms in a body of water to break down organic material present in water. 14 COD is the amount of dissolved oxygen needed to oxygenize organic compounds in water. 15 TSS is the non-filterable residue in water. 16 Environmental Protection (Standards for effluent discharge into the ocean) Regulation 2003, Government Notice No. 45 of 2003. 17 JICA appraisal documents. 18 The World Bank portion of the project calculates EIRR, taken tariff and tourist industry revenues as benefits, with 19.2% at the completion (ICR), as compared to 12.2% at appraisal. 19 As WMA does not have a separate accounting system per treatment plant, the FIRR has

9

resulting in a negative figure. As discussed in the Sustainability Section, this is because the tariff level and amount of wastewater are insufficient to recover O&M costs. In addition, with increased tariff revenue in 2011 and drastic management efforts21, including an O&M cost cut, the FIRR has barely been positive.

Table 5 Recalculated FIRR

Appraisal Post-Evaluation（base） With managerial efforts

7.4％ -10.4% 0.8%

Source: Recalculated by external evaluator

3.3.2 Qualitative Effects Assumptions at the time of appraisal include the beneficiaries—that is, 120 thousand residents and

approximately 150 factories (of which approximately 60% were related to the textile industry)—contributing to “the improvement of public health and sanitation,” “protection of ecosystem and promotion of tourism,” and “promotion of sustainable economic activities.”

As a part of the qualitative evaluation, a beneficiary survey was conducted for residential and commercial districts (sample size of 100, including 10 fishermen and 5 factories).22 Among those who responded with “very much satisfied” or “satisfied” (58), a great majority listed “improvement of sanitation” (32), or “improvement in coastal water” (16) as their reason of satisfaction. Considerably fewer number of people listed “protection of environment” (6), “decrease in wastewater pollution” (2), and “promotion of tourism industry” (1), and none listed “the decrease of water-borne diseases”23 or “promotion of economic activities.”24 40 responded with “unsatisfied” or “very much unsatisfied” because of the “overflow of water at the pumping station in rain”(13), “high level of tariff”(18), and “environmental damage near treatment plant and damage to fishery”(9). Out of 10 samples from fishery industry, 8 responded “unsatisfied” or “very much unsatisfied” for the reason of polluted water (and resulting damage to fishing). From the above, it was found that the beneficiaries recognized environmental effects more than economic effects.

not been calculated but approximated from the balance sheet of WMA and the share of daily inflow at Montagne Jacquot Treatment Plan over the aggregate amount of treatment (approximately 30%, measured in 2008). 20 Calculated based upon the assumption of “project life = 30 years including the construction period” (conditions at the time of appraisal). 21 Based upon the assumption of 40% improvement of cash flow (with increased revenue or decreased O&M costs). 22 At the time of post-evaluation, there were only 15 to 20 factories in the covered area, and the number of textile factories decreased, so the sample size of the commercial facilities was limited to 5 firms. 23 A separate survey on the water-borne diseases showed a slight improvement (21 cases before the project and 17 cases after the project). 24 Interviews with neighboring firms (5 sample firms) also showed that after the project the sales “have not changed” (1), “have decreased” (4), while the management “has not changed” (1) “has suffered from increased sewerage pre-treatment costs” (4).

10

This project has somewhat achieved its objectives, therefore its effectiveness is fair.

3.4 Impact 3.4.1 Intended Impacts

The expected impact at the time of appraisal was “to contribute to the upgrading of living standards and development of the tourism industry.” The standard of living is considered to have been upgraded to some extent, as shown in the number of beneficiaries who listed “improvement of sanitation conditions” as their major reason for satisfaction under the “qualitative effects” section. On the other hand, there is no evidence of benefits linked to the development of the tourism industry, even though there has been a steady increase in the tourism business and the number of foreign visitors to Mauritius since the project implementation. This is not enough to determine causality, however.

Table 6 Trends of Tourism Indicators

1995 2000 2005 2008 2009

Foreign Tourists (in 1,000) 422 656 761 930 871

Tourism Revenue (in million USD) 616 732 1,189 1,823 1,390

Tourism Revenue (% of total exports)

26.2 27.9 31.7 37.0 33.2

Source: WDI

In Mauritius, there are four wastewater treatment plants, including the plant in Montagne Jacquot, which accounts for approximately 30% of all water treated in the country by design and measurement. The construction of this plant has expanded the capacity of the nation’s wastewater treatment and has enhanced connectivity. At the time of the appraisal, there were approximately 24,000 connections (including individual and corporate contracts), with a coverage of 18% (70% in Port Louis only); at the time of post-evaluation, however, there were 64,700 connections (same definition), with a national coverage of 25%.

3.4.2 Other Impacts (1) Impacts on the natural environment

Mauritius has an Environmental Impact Assessment (EIA) system, and two EIAs were conducted in 1997 and 2001, before the project started. The reason for the 2001 EIA was, as explained in the “3.2.1 Output” section, that the “sea outfall” system was adopted rather than the “underground borehole injection” system, to change the original scope of work. Since 2008, the Ministry of Environment has commissioned an Independent Environmental Audit (IEA) for the WMA managed

11

wastewater treatment plants, whereas the Ministry of Fisheries has surveyed water quality in coastal areas, including the vicinity of this plant.

The environmental impacts of the treated sewerage water on coastal areas are related to the existence and quantity of BOD, COD and other organic pollutants, TSS and other suspended solids, arsenic and other chemical pollutants, and fecal coli and other bacteria. Suspended solids and chemical pollutants have been within the permissible levels, according to the surveys conducted by the WMA, IEA, and the Ministry of Fisheries. 25 In a survey conducted by the Ministry of Fisheries, however, bacteria have been detected in the vicinity of the plant.

Table 7 Trends of faecal coliform （Unit：CFU26/100ml）

Vicinity of Pointe aux Sables Pumping Station (max and min)

Vicinity of Montagne Jacquot Treatment Plant (max and min)

Albion Testing Point (average)

2000 10 - 2450 NA NA

2001 5 - 26900 NA NA

2002 4 -14500 NA NA

2003 5 - 95000 NA NA

2004 2 - 500 NA 6

2005 5 - 395 NA 10

2006 2 - 610 NA 41

2007 3 - 315 NA NA

2008 10 - 280 NA NA

2009 15 -1940 12 – 540 NA

2010 15 - 450 25 -1500 NA

Source: Ministry of Fisheries Note: Point aux Sable Pumping Station is located on 2 km north coast to the MJ Treatment Plant, and Albion is located on 1 km south coast.

There were no standards for faecal coliform in the “2003 Environmental Regulation”27 at the times

of appraisal and post-evaluation. However, the “Guidelines for Coastal Water Quality”28 (1999) stipulates that if a level of more than 200 CFU/100 ml of faecal coli29 is detected, swimming and

25 WMA conducts its own surveys (unpublished), apart from the official monitoring required by EIA. There is no record of periodical reporting of the monitoring results to JICA, however. 26 Colony-Forming Unit to measure the number of bacteria. 27 Environmental Protection Regulation 2003. 28 Guidelines for Coastal Water Quality (General Notice No. 620 of 1999). 29 US and EU standards are set at this level, and Japanese “water quality standards for swimming” (Ministry of Environment Ordinance) stipulates that 100 CFU/100ml or below is “suitable” and 400-1,000 CFU/100ml or below is “possible” for swimming.

12

fishing are banned; wastewater plants are expected to meet this standard, although it is not legally binding, as with other possible sources of contamination. The maximum level of faecal coli measured actually exceeds this standard, as shown in Table 7. According to the Ministry of Environment, this is most likely due to the wastewater plants.30 However, further studies are required to detect the various other possible causes of pollution more accurately.

The EIA (2001) required the treatment plant to disinfect sewerage through chlorination; however, this has been temporarily stopped since 2008. The WMA explained that this is because the EU, as part of its technical assistance, advised the WMA to “stop chlorination because it is not effective with ammonized sewerage (like in this system).”31 The WMA further explains that the resumption of chlorination depends on a re-examination of the technical aspects (that is, effectiveness), and cost effectiveness of chlorination. At the time of the post-evaluation, however, there was no concrete plan for such a re-examination

Furthermore, the sewerage-filtered sludge is compressed, de-watered, and dumped in the Mare Chicose Landfill (a trash dumping center with a capacity of 400–500 tons/day)32 without being burnt .

(2) Relocation and Expropriation There was no relocation, but land (10 ha, privately owned) was expropriated in November 1998.

This land was intended for the expansion of the treatment plant (for secondary treatment with microbiological disinfection). This land remains untouched up to now. There is no residential district in the vicinity; however, there is a prison and a farmyard.

(3) Other Positive and Negative Impacts The appraisal made no mention about these impacts, and there were no problems at the time of

post-evaluation. Hence, even it is at the effort level, there is some concern about the negative impacts of faecal coli.

3.5 Sustainability (Rating: ②) 3.5.1 Structural Aspects of Operation and Maintenance

At the time of the appraisal, O&M was expected to be contracted out to a private company.33 In

30 2010 data on the fecal coli in the vicinity of Montagne Jacquot Treatment show a significant variation of 25 – 1,000 CFU/100 ml. The Ministry of Fisheries explains that this is due to the fluctuation of sea current. It is, however, possible to improve the measurement to obtain more accurate data, so it appears to be necessary to conduct a separate survey for cross-checking purpose. 31 Memorandum (27th June 2008), Technical Assistance for the Mauritius Wastewater Sector Policy Support Programme (European Union). 32 Trash dumping center (40 ha) financed by the World Bank in 2000. World Bank (Mauritius - Environmental Solid Waste Management Project (Project Information Document). 33 WMA explains that the most of O&M operations are contracted out to private companies in

13

actuality, total 31 WMA staff are operating the O&M division. These staffs are divided into various time shifts (as the plant is supervised 24 hours a day) and some work for the two pumping stations. Therefore, the actual number of personnel working in one time shift is considered to be between 15 and 20.

There are 424 WMA staff members (as of January 2009),34 and the decision-making body is a board comprising of the Permanent Secretary of the supervising agency (Ministry of Public Utilities) and representatives of other relevant ministries. There is a chairperson of the board of the WMA, and an officer-in –charge with two deputy general managers (one in charge of technical issues and other in charge of administrative and financial issues). The overall O&M is supervised by the vice-president in charge of technical issues, and more routine works are supervised by the director and the O&M staff.

The wastewater quality is inspected by the WMA Laboratory. The inspection is conducted at the wastewater treatment plants and pumping stations, for compliance with environmental standards. Additional inspection is conducted at coastal areas with water outfall, firms, hotels, and hospitals with their own water quality treatment facilities. Inspection is conducted regularly35 by sampling water and inspecting it against various parameters (total solid, chemical hazards, etc.) in the laboratory.

Hence, the overall institutional framework of the O&M is good.

3.5.2 Technical Aspects of Operation and Maintenance Mauritius had accumulated experience in O&M techniques since the French had assisted Mauritius

with its sewerage system before this project. The overall initial conditions in human capital (education level, etc.) are favorable. Further, there is no problem in the current technology level. Given these relatively favorable initial conditions, it is not always easy to bring about additional technology transfer and institutional capacity building. However, the O&M techniques acquired through this project (sewerage system management and water quality inspection) have been used for other projects 36 , indicating that there has been some transfer of technology. In addition, an institutional building (World Bank financed soft component), three JICA short-term experts (O&M), and a one-year training by the contractor after the completion of project have been provided.

These inputs can be sustained only if there are “self-help efforts” by the implementing agency in the follow-up and sharing periods of the training experience. WMA explains that “capacity building has been top priority”37 at both the individual and institutional levels and that it has sent personnel to Japan, France, Egypt, and other countries for training.

3.5.3 Financial Aspects of Operation and Maintenance St. Martin Treatment Plan, while at the Montagne Jacquot, it was not contracted out because of “high cost” (WMA executive staff). 34 WMA Annual report (2009). 35 In most facilities, sampling surveys are conducted twice a month. 36 World Bank (2007) Implementation Completion and Results Report (Loan No. 42830). 37 WMA Annual Report.

14

At the time of the appraisal, WMA was just one division under the Ministry of Public Utilities, and investment and current costs were a part of the Ministry’s (general) budget. In fiscal year 200138, when the WMA was corporatized,39 it became financially autonomous from the general budget. Since the corporatization, the steady increase in revenue until 2005 resulted in a surplus; since 2007, however, owing to the increased O&M from this project, the balance sheet has been in deficit.

The “Expenditure/Tariff Revenue” ratio is below 100% if cost can be recovered and above 100% if the cost cannot be recovered. For 2008, the ratio was 151% (the cost recovery ratio is its reverse number, 67%).

Table 8 Trends of Financial Indicators

2003/04 2004/05 2005/06 2006/07 2007/08 2008/09Revenue(A) 227 236 250 N/A 301 311

Tariff Revenue 196 196 204 N/A 233 249Gov't Subsidies 31 40 41 N/A 68 62

Expenditure(B) 134 198 211 N/A 318 377O&M 56 73 84 N/A 213 238

Personnel N/A N/A N/A N/A 67 70Repayment N/A N/A N/A N/A 25 58

Others N/A N/A N/A N/A 121 110Balance(A-B) 93 38 39 N/A -17 -66Expenditure/Tariff Revenue 68% 101% 103% N/A 136% 151%

Source: WMA WMA endeavors to increase revenue by increasing the tariff (or decreasing the number of unpaid

contractors, estimated to be approximately 15%). However, the tariff40 falls exclusively within the purview of the Ministry of Public Utilities and cannot be changed through WMA’s managerial efforts alone. The Ministry of Public Utilities states that it is discussing a drastic reform plan to merge potable and sewerage systems in order to increase efficiency and profitability.

On the other hand, the balance sheet of the WMA is not recorded for each treatment plant separately and is only assessed as an agency. In the current difficult financial situation, the private sector would make each section a “cost center” and would introduce the cost accounting system to lower O&M costs as a part of the management efforts. The WMA has not institutionalized this system yet.

3.5.4 Current Status of Operation and Maintenance Facilities, as a whole, are operating to lower BOD to the minimum level; however, some facilities

have been out of operation for a long time. At the time of the post-evaluation, a de-watering/sludge holding tank (one out of two) remained unrepaired; ,a “drum screen” (trash filter) (one out of two) was

38 Mauritian Fiscal Year runs from July 1st to June 30th. 39 WMA Act 2000. 40 Since 2008, commercial tariff has been fixed to 20MUR/m3, while residential tariff has been fixed to 5.5MUR/m3 (below 10m3), 6.5MUR/m3 (11-20m3), 15.0MUR/m3 (21-30m3), and 34.0MUR/m3 (above 31n3).

15

not operational, a clarifier pond (one out of three) remained unused41, and polymer machines were not operational. The chlorine disinfection system has not been used, as mentioned earlier. With regular usage, these facilities cannot simultaneously break down so easily, thereby implying that there have been problems in maintenance methods.

WMA explains that its O&M budget is appropriated for repairs costing less than 2 million MUR (or approximately 5.5 million JPY)42, while a general budget must be secured from state budget (general budget) through the WMA board approval for repairs costing more, thus taking more time. All broken facilities left unrepaired at the time of post-evaluation costing more than 2 million MUR were awaiting the state budget.

The beneficiary survey indicates that industrial heavy oil has been illegally dumped into the sewage network, damaging the pumping stations and treatment plants. Additionally, during the rain, facilities frequently overflow with untreated water near the Pointe aux Sable station (the outfall of the previous system), as indicated by the residents in the area; WMA officials were also aware of the need to take some action.

Some problems have been observed in terms of financial aspect of O&M, therefore sustainability of the project effect is fair.

4. Conclusion, Lessons Learned and Recommendations 4.1 Conclusion

This project has been extended with the objective of “preventing water pollution in the metropolitan area and protecting the oceanographic ecosystem.” The project indeed matches the development policies, development needs, and Japanese development policies, and is highly relevant. Additionally, efficiency is fair, because the project cost is within what was planned, but the project period far exceeded what was planned. BOD and other operating indicators meet the planned level and have contributed to the improvement of environmental and sanitation conditions; however, the effectiveness is fair, because the operating level is lower than the planned level by approximately 30%. Sustainability is an area of concern, because, although the overall system and technology are fine, there is a shortage of revenue in recovering Operations and Maintenance (O&M) costs and because some of the non-functioning facilities are left unrepaired for a long time.

In light of the above, this project is evaluated to be partially satisfactory. 4.2 Recommendations

4.2.1 Recommendations to the Executing Agency

The substantive delay was caused by the protest of a bidder who lost during the procurement stage.

41 WMA explains that the clarifier pond (1) has not been in use because the inflow of sewer water remains below capacity for the purpose of “odor abatement”. 42 1MUR=2.75 JPY (exchange rate at the time of post-evaluation)

16

The only way to protest at that time was to file a formal lawsuit; however, a revision of the law (Public Procurement Act 2006) made it possible to file a claim at the Independent Review Panel designated by the Agency of Bidding Coordination. This is supposed to substantively simplify the procedure and shorten time; however, the implementation agency must make it even more efficient and transparent.

Data for treated wastewater quality inspection must be reviewed and reexamined; in particular, this must be done for the data for E-Coli, F-Coli, and other bacteria (maximum, minimum, and average) at the outfall and near the Montagne Jacquot Wastewater Treatment Plant. This is not covered by the current IEA survey. If necessary, an inspection will be conducted again by an institution designated by the Ministry of Environment to ensure that the water quality near the plant does not adversely affect fishing and tourism. Also, in the vicinity of the plant there exist various possible sources of contamination, with complex oceanographic settings including current, lagoon and bay, which would make it necessary to systematically monitor any causal relationship between the plant and contamination. In this regard, it is necessary for the Ministry of Environment to reach a consensus on the necessity to improve water quality data among the WMA, the Ministry of Fisheries, and all other stakeholders.

The post-evaluation has also revealed that the fecal coli observed at the Montagne Jacquot plant exceeded the “Coastal Water Quality Guideline” (1999); however, the guideline is not legally binding because there is no penalty for violating it. Additionally, there is no standard for fecal coliform in effluent water. At present, the Ministry of Environment is considering legalizing it; however, for now, the observance of guideline must be respected.

In this regard, the possibility of re-starting the chlorination or any alternative disinfection procedure, suspended since 2008 after EU recommendation, needs to be surveyed and discussed from the technical and cost-benefit perspectives.

Also, the beneficiary survey revealed the frequent overflow of water at two pumping stations especially in rainy days. A careful examination of this incidence shall be required in order to determine whether it is related to operation and maintenance or project design itself.

Also, it has been shown that the current tariff and fee collection system cannot allow full cost recovery for operation and maintenance. Therefore, a review of tariff, rationalization of personnel costs, and other drastic measures must be taken in order to strengthen financial viability.

4.2.2 Recommendations to JICA

In order not to weaken the project objective of “promoting the ocean ecosystem in coastal areas,” it is important for the WMA to examine and take necessary measures, if necessary, for the above-mentioned fecal coliform problems. However, the current financial constraints at the WMA may not allow such measures. Therefore, JICA could help them inspect water quality and take necessary action more promptly and effectively through additional surveys, such as Special Assistance for Project Sustainability (SAPS) or the dispatch of short-term sewage experts.

17

Montagne Jacquot has been designed as a primary treatment plant mainly to remove suspended particles. Originally, however, Mauritius had planned to build a secondary (microbiological) treatment plant after the construction of this project was completed. The level of ammonia may be reduced through nitrification by the secondary treatment thereby making a subsequent disinfection by chlorine more effective. 10 ha of land had been acquired for this; however, it remained unrealized owing to the troubled financial situation. As mentioned in the recommendation for Mauritius, if the necessity arises for secondary treatment, after a careful examination of the additional water quality inspection, JICA might be required to provide technical and financial support or advice.

This project is co-financed by the World Bank, which provided technical assistance for tariff structure, etc. Subsequently, the operational framework of the executing agency turned out to have adversely affected the physical component (Yen Loan portion), as evidenced in the malfunctioning O&M. This suggests that JICA needs to strengthen dialogues with the World Bank over the co-financing project even after the completion of the project for smoother project operation. 4.3 Lessons Learned

In countries with no JICA offices, mid-term monitoring mission and (short-term) expert dispatch would be useful in strengthening the quality and frequency of project monitoring. A project management consultant provided one year of technical assistance in O&M after the completion of the project, in keeping with the contract. This period of one year, however, was too short. Depending upon the capacity of the implementing agency, in other similar projects, three to five (or even more) years of technical assistance is commonly provided for O&M areas through BOT, etc.

Additionally, regarding the finding of concerns and risk of negative environmental impacts of fecal coliform, it is necessary to take joint action in the “Grand Baie Sewage Construction Project” to prevent similar problems in the new project.

18

Comparison of the Original and Actual Scope of the Project

Item Original Actual 1.Project Outputs Pumping Station Force Main (*) Waste Water Treatment Plant Sea Outfall Consulting Services Technical Assistance (*) *World Bank-Financed Portion

2

6 km Treatment Capacity 48,000 m3/day

645 m long, 30 m deep Procurement Preparation and

Supervision Management and Staff Trainig, etc.

As planned As planned As planned As planned As planned

As planned

2.Project Period

October 1998 – December 2002 (51months)

September 1998 – January 2007 (99 months)

3.Project Cost Amount paid in Foreign currency

5,155 million yen

4,658 million yen

Amount paid in Local currency

2,553 million yen (442 million MUR)

2,367 million yen (628 million MUR)

Total 7,708 million yen 7,015 million yen Japanese ODA loan portion 4,538 million yen 4,533 million yen Exchange rate 1 USD = 121 yen

1 USD = 21.1 MUR (As of December 1997)

1 USD = 115 yen 1 USD = 30.5 MUR

(Average between January 2005 and December 2007)

1

United Republic of Tanzania

Project for Rural Water Supply in Lindi and Mtwara Regions

External Evaluators: Takeshi Daimon and Rui Hiwatashi, Waseda University

０．Summary

This project is intended to support the water supply facilities in the Lindi and Mtwara regions, and thus increase water supply pervasion and supply safe water in the target areas continuously. Considering Tanzanian and Japanese development policies, the project is highly relevant. It is fairly efficient because the project cost remains within the original estimate, although the period has slightly increased. On the other hand, the population with access to safe water has increased, and there has been a decline in the incidence of waterborne diseases; thus, the effectiveness and impact of the project are high. O&M has had some problems in technical aspects, and the project is considered fairly sustainable.

In light of the above, this project is evaluated to be satisfactory

１．Project Description

Project Location: Lindi and Mtwara Regions A resident fetching water

1.1 Background The water supply situation in Tanzania is very critical situation, and about a half of the

population does not have access to safe and hygienic water, resulting in frequent waterborne diseases and—of particular concern—increased infant morbidity and mortality rates. In the absence of proper water supply facilities, the arduous job of fetching water falls on women and children. 1 1 Basic Design Report

2

In 1976, water supply facilities were constructed in the two southern regions covered by this project according to a “Water Sector Master Plan” with Finnish aid, and the water supply coverage improved to 75% by 1984. However, the conditions of the facilities rapidly deteriorated because of the lack of budget, with the end of Finnish aid in 1992, and a low level of awareness about maintenance among the people. Consequently, the water supply coverage had dropped drastically to about 35% at the planning stage of this project.

In order to overcome this situation, the Government of Tanzania (GoT) requested the Government of Japan (GoJ) to carry out a study for the improvement of water supply, which resulted in a “Southern Regions Water Supply Plan” formulated over almost two years, from February 2000 to December 2001. On the conclusion of this study, the GoT requested the GoJ for a grant-in-aid to provide a safe and stable water supply by constructing deep wells.

1.2 Project Outline This project is intended to support water supply facilities in the Lindi and Mtwara

Regions, thereby increasing water supply coverage and continuously supplying safe water to target areas.

Grant Limit / Actual Grant Amount 1,729 million yen / 1,671 million yen

Exchange of Notes Date Phase 1: September 2003 Phase2: June 2004 Phase3: June 2005

Implementing Agency Ministry of Water

Project Completion Date Phase 1: December 2004 Phase2: February 2006 Phase3: March 2007

Main Contractor(s) Mitsubishi Corporation（Phase 1）

Hazama Corporation (Phase 2) Koken Boring Machine, Co. Ltd.(Phase 3)

Main Consultant(s) Kokusai Kogyo Group

Basic Design From November 2002 to June 2003 (8.0 months)

3

Related Projects (if any) The Rural Water Supply and Sanitation Capacity Development Project（Technical Assistance）（June 2007 to August 2010）

Southern Regions Water Supply Plan (February 2000 to December 2001) Water Supply Project in Lindi and Mtwara Regions by CONCERN

2．Outline of the Evaluation Study

2.1 External Evaluators Takeshi Daimon and Rui Hiwatashi, Waseda University 2.2 Duration of Evaluation Study Duration of the Study: December 2010 – December 2011 Duration of the Field Study: February 14 to March 17 and July 19 to July 30, 2011.

2.3 Constraints during the Evaluation Study (if any)

The target areas include 174 facilities in 64 villages in Lindi and Mtwara Regions. Hence, 28 sample villages have been randomly selected (taking the coverage into consideration), and interviews have been conducted to obtain data on the impact and sustainability of the project. 2

3．Results of the Evaluation (Overall Rating：B 3） 3.1 Relevance (Rating：③ 4）

3.1.1 Relevance with the Development Plan of Mauritius The higher policy goal of improving the water sector is the “National Water Policy”

(NWP) of 1991, which was revised in 2002. The revised NWP took over the original focus of promoting the participation of the users of the water supply, cost sharing for operations and maintenance (O&M) by users, and the promotion of safe water and sanitation. This is in addition to a new emphasis on the nationwide construction of facilities for hygienic and safe water. However, as the water supply coverage in 2002 remained below 70% in urban areas and 50% in rural areas, the revised NWP redefined the policy goals to include the importance of the demarcation of roles and responsibilities

2 Surveys were conducted for water engineers in all 9 districts and 2 regions with jurisdiction over target villages. 3 A: Highly satisfactory, B: Satisfactory, C: Partially satisfactory, D: Unsatisfactory 4 ③: High, ② Fair, ① Low

4

among the water and sanitation sector personnel, as well as cost sharing of O&M by users. At post-evaluation, the National Water Sector Development Strategy (NWSD) 2006-2015 was drafted in 2006, based on the revised NWP, and the Water Sector Development Program (NSDP) was drafted in 2007, both supporting the water sector in the country. The NSDP also emphasizes the importance of rural water, and this project continues to promote a community-driven approach to strengthen O&M capacity.

Hence, this project has been consistent with the nation’s development policies before and after the project.

3.1.2 Relevance with the Development Needs of Tanzania

At the time of pre-evaluation, the drinking water situation in Tanzania was deteriorating, and almost half of the population had limited access to safe and hygienic water, causing waterborne diseases; especially worrisome was high infant morbidity and mortality. In addition, the lack of proper water facilities forced women and children to shoulder the burden of fetching water. The two southern regions covered by this project were also high-priority areas with as low as 35% water coverage in 2003, before evaluation. At post-evaluation, the improvement has not been significant because of the malfunction of facilities or possibly the lack of supply capacity to match the increasing population, 5 and the national coverage itself has improved slightly, remaining as low as about 60%.

Hence this project has been consistent with the nation’s development needs before and after the project. 3.1.3 Relevance with Japan’s ODA Policy

The Country Assistance Plan (CAP, 2000) states that continued assistance must be extended to (i) the basic infrastructure sector for poverty alleviation and the improvement of standards of living by achieving inclusive growth and (ii) inter- and intraregional infrastructure projects connecting city centers for the improvement of rural living conditions along with its byproduct of arresting the inflow of the rural population into the capital city (e.g., rehabilitation of major truck roads and water resource development in the southern region).

The revised CAP of 2008 still stresses the importance of building water-related infrastructure in areas with limited access to safe water as well as developing expertise in formulating and implementing water plans through training local staff in the sector.

Therefore, this project has been consistent with the Japanese development policy before and after the project. 5 Interview with the implementing agency

5

In light of above, this project is highly relevant because it is consistent the country’s

development plan, its development needs, as well as Japan’s ODA policy.

3.2 Efficiency (Rating：②） 3.2.1 Project Outputs The project has been implemented without change in scope of procured equipment and

facilities between plan (basic design) and realized (post-evaluation). (a) Well drilling machines and support equipment (Phase 1/Phase2)

Drills

Drills (Tools accessories ） High pressure compressor Electric Investigator Well logging equipment

Pumps for pumping test（for hand pumps） Pumps for pumping test（for motor pumps） V-notch weirs for pumping test（for hand pumps） V-notch weirs for pumping test（for motor pumps） Generator for pumping test（for hand pumps） Generator for pumping test（for motor pumps）

1 1 1 2 1 1 1 1 1 1 1

(They have been procured as planned. Drillers are on lent to other regions for drilling wells). Phase 2

Maintenance Equipment

Survey Equipment

Maintenance Trucks Motorbikes Well-washer compressor Cargo trucks Water gauge PCs Printers pH meters ORP meters EC meters Pickup Trucks

2 9 2 2 2 2 2 2 2 2 2

6

Current meters Survey machines

2 2

(b) Installation of hand pumps in 14 villages and collective pumping stations in 19

villages in Lindi Region Installation of hand pumps in 12 villages and collective pumping stations in 19 villages

in Mtwara Region

Facility Quantity

Construction of deep wells with hand pumps

Drilling new wells (75)(16 villages）(Phase 1) Drilling new wells (72)(16villages) (Phase 2) Installation of hand pumps (75) (Phase 1) Installation of hand pumps (72) (Phase 2)

Construction of deep wells with motor pumps

Drilling new wells (22) and existing wells (2)

（24villages）(Phase 1) Drilling new wells (12) (12villages)(Phase 2) High rising water tanks 20m3 (10), 30m3(3), 50m3

(10), and existing tank (1) (Phase 1) High rising water tanks 20m3 (9), 30m3(1), 40m3

(1), 50m3 (1) Pumping stations (96) (Phase 1) Pumping stations (36) (Phase 2)

Construction of deep wells with motor pumps for spring water

2 sites（2villages）(Phase 1) High rising water tanks 50m3 (1), 40m3 (1) (Phase

1) Pumping stations (10) (Phase 1)

(c) Technical Assistance for Operations and Maintenance (Soft Component) The following activities were carried out without any change since the plan.

Activities Phase 2 Phase 3

MM Target Areas MM Target Areas

Participatory workshop 4.7 4 districts in Lindi, Mtwara, Tandahimba, Kilwa

2 5 districts in Masasi, Newala, Nachingwea,

Village gathering 2

Socialization workshop 1

Hygienic education 2.1 2

7

O & M plan workshop 3.8 3 Ruangwa, Liwale Technical training for repair (DWE) 0.3 0.6

Technical training for repair (VWC) 1.6 1

Technical training for administration 6

Joint commission for stakeholders 0.6

Visiting training for O&A 5

3.2.2 Project Inputs

3.2.2.1 Project Cost

Table 1 Comparison of Original and Actual Project Costs Unit: Million Yen

Phase 1 Phase2 Phase 3 Total

Original 331 813 584 1729

Actual 331 756 584 1671

Source: Inspection Report The estimated period at the planning stage was 1,737 million yen (of which Japan’s

contribution was 1,729 million yen and Tanzania’s contribution was 8 million yen) and the actual cost was within the planned amount (94%).

3.2.2.2 Project Period

This project was implemented in three phases (Phase 1: 15 months from October 2003 to December 2004, Phase 2: 19 months from August 2004 to February 2006, Phase 3: 21 months from July 2005 to March 2007), of which the duration of procurement and construction periods (42 months in total) exceeded the plan (41 months in total) (or 102% of the plan). The executing agency (Ministry of Water) explains that the delay is due to changes in project sites in Phase 2 (11 sites in Mtwara and 4 sites in Lindi) due mainly to insufficient water.

Although the project cost was within the plan, the project period was slightly exceeded,

therefore efficiency of the project is fair.

8

3.3 Effectiveness 6(Rating：③） 3.3.1 Quantitative Effects

3.3.1.1 Results from Operation and Effect Indicators The original goals of the project were the following: (a) The water-served population should increase by about 200,000, raising the coverage

in the two target regions 7 from about 35% in 2002 to about 42% in 2007. (b) Per villager safe water must be 20 liters 8 per day.

The executing agency (Ministry of Water) reports that, as the table below shows, the access to safe water in the Lindi and Mtwara Regions has, overall, improved, achieving the 2007 goal of 42% coverage. However, since it peaked in 2008, the coverage has been declining because of the malfunction of facilities and a drastic increase of the inflowing population after 2009. This trend is particularly evident in Lindi.

Table 2 Access to Safe Water（％）

Lindi Mtwara

2004 45.2 53.6

2005 46.3 53.6

2006 47.3 57.0

2007 48.1 58.2

2008 63.4 72.8

2009 54.9 70.2

2010 40.1 67.0

Source: Ministry of Water

As explained before, during the pre-evaluation period there was no baseline indicator of water supply coverage to give a sense of (ratio of) the “population with access to safe water/population in the target areas”. Only statistics in the target regions (i.e., Tanzanian official statistics of “population with access to safe water/population in Lindi and Mtwara regions”) was used as an indicator of effectiveness.

The project originally assumed that the water supply facilities would be accessed by 100% of the target population (about 200,000), raising the coverage by 7% in the two southern regions. This may prove somewhat problematic.

Replies to the questionnaire from the water department in each Region show that nearly 6 Rating takes into account of Impact as well. 7 The Basic Document has no mention about coverage in targeted villages. 8 “National Water Plan” (GoT) (revised in 2002) defines the design standard as “within 400 meters from location of living place, 25 liters per person per site on annual basis, accessible to 250 users.”

9

100% of the target population in Lindi and 80% in Mtwara 9 have achieved their target goals, and the target for the amount of water per capita was achieved (23 liters in Lindi and 60 liters in Mtwara, both as of 2007). Furthermore, as the table below shows, a beneficiary survey with 150 sample households in 28 sample target villages shows that nearly all of the households (147 households, or 98%) who used surface water (spring, river, and rainwater) or an unsafe well before the project have now turned to the facilities provided by this project. The executing agency explains that areas without water supply facilities still have a low access to water, below 30%, and the regional disparity in water supply is a major issue.

Table 3 Sources of Drinking Water

Source: Beneficiary Survey 3.3.2 Qualitative Effects

At the pre-evaluation stage, the following qualitative effects were considered as targets. (a) Through on-the-job training (OJT), workers will acquire skills to construct deep

wells on their own in other places by fine-tuning the task details to local conditions. Interviews with the Department of Water Engineer (DWE) in all of the nine target

provinces confirm that the OJT was provided to staff in provincial water departments as planned and after the completion of the project the skills acquired and equipment

9 In Mtwara, 67,912 out of 84,890 residents in target villages, except for two villages with mal-functioning facilities, are defined as population with access to safe water. By the same definition, 67,392 residents are considered accessible to safe water.

Before After

Mineral Water 0 0

Unsafe Public Water

0 0

Safe Public Water 0 0

Protected Well 0 0

Unprotected Well 23 0

Safe Spring 6 0

Unsafe Spring 0 0

River 67 0

Rain 54 0

JICA Project Well ― 147

Others 0 3

10

procured are still used in a dozen water facilities. Therefore, this subgoal is considered to have been achieved.

(b) Drills and support materials will allow people to drill wells to a depth of more than

150 m, thereby improving drilling skills. The executing agency explains that actually those drills and support materials have

been used in a dozen places. Thus, this subgoal is also considered to have been achieved. (c) Maintenance equipment will allow workers to maintain and repair facilities

sustainably. A survey of village water committees (VWC) in the 28 sample villages shows that

prefectural water engineers conduct monitoring for maintenance and repair monthly in 25 villages (or 89% of all the villages). Hence, as planned, the maintenance equipment has been procured and used to achieve higher policy goals. However, the maintenance skills acquired through training were not always utilized in some districts and villages because courses were: “too general and did not teach how to make repairs,” or “too difficult for staff to comprehend at village or at the district level.” 10

(d) Survey equipment will allow them to conduct planning for the creation of deep

wells, evaluation of water quality, measurement design, and monitoring of maintenance. The executing agency explains that the survey equipment has been introduced and

utilized as planned, and actually a dozen deep wells have been planned, evaluated, designed, and monitored, which suggests that the higher goal has been mostly achieved.

(e) A community-based O&M system will be established and water facilities will be

maintained in a sustainable and appropriate manner. The VWC survey shows that 28 villages (100%) have successfully established an organizational framework (such as water committees at the village level and a maintenance mechanism including collection of fees) and O&M committees for water supply are fully functional. In the target districts, including four in Mtwara (Mtwara,

Masasi, Newala, Tandahimba）and five in Lindi (Liwale, Kilwa, Nachingwea,Ruangwa, Lindi), continued technical assistance (“Village Water Supply and O&M Planning Project”) has helped in strengthening people-centered O&M systems. Hence, subgoal (e) is considered to have been mostly achieved, and the effectiveness of aid through a soft component is evident. 10 Interviews with VWC.

11

Therefore, the above five qualitative effects are considered to have been achieved.

This project has achieved its objectives, therefore its effectiveness is high.

3.4 Impact

3.4.1 Intended Impacts

At the pre-evaluation phase, the following indirect effects (impacts) were assumed. (a) safe and hygienic water supply 11 (b) promotion of national water policy 12 (c) improvement of sanitary conditions 13 (d) education and promotion of maintenance and regional development 14

The beneficiary survey (among the 28 sample villages with 150 sample households)

has been used to analyze impact. First, (a) the question of whether “safe and clean drinking water has been secured for

villagers in order to improve the hygienic situation” seems to have been achieved, because, as explained in the “effectiveness” part, among users of surface water (spring-, river, and rainwater) and unprotected wells, almost all villagers (147 households, or 98% of the total) have access to the facilities provided by this project.

(b) The question of whether “the project has become a model case for rural water supply, in line with the national water plan, and with a huge influence over the promotion of a rural water supply plan in the future” also seems to have been achieved, because, according to the interview with the executing agency (Ministry of Water), the implementation of the project has indeed generated the expected impact and has been recognized as a model case for future plans. 11 “It is intended to secure safe and clean water for villagers, to improve hygienic environment, satisfying Basic Human Needs (BHN) of target 201,967 (estimated for 2005). In addition, after the completion of the project, it is expected to reduce labor for women and children to fetch water, thereby transferring its labor to agriculture and various other activities in the region.”（Basic Design） 12 “With the completion of this project, it is expected to serve as a model case for the rural water supply plan in line with national water plan, promoting significantly future rural water supply programs.”（Basic Design） 13 “It is expected, in the long run, to effectively help improve hygienic conditions, thereby decreasing the morbidity of waterborne diseases such as diarrhea, parasites, skin and eye diseases.” （Basic Design） 14 “Through this activity, people are expected to raise their awareness about maintenance on their own, by acquiring O&M skills necessary for water facilities. In this process, it is expected to help raise community identity, with possible spillover to productive activities in this region, thereby contributing to form rural communities.”（Basic Design）

12

(c) The improvement of hygienic conditions seems to have been achieved as well, as the morbidity of waterborne diseases such as diarrhea, parasites, skin disease, eye infections, has been improved since the implementation of the project (as shown in the table below).

Table4 Morbidity of Waterborne Diseases (Sample 150 households)

Are there household members with chronic waterborne diseases?

Before After

Yes 136 24

No 9 126

Others or No answer 5 0

Source: Beneficiary Survey Asked why they are “very satisfied” (105 households) or “satisfied” (39 households, or 96% of all respondents), the most cited reasons include “improvement of access to water” (58), “improvement of water quality” (28), “reduction of waterborne diseases” (28), and “improvement of sanitation and environment” (16). 15 In addition, when asked about time saving as an economic impact, nearly all households reported significant time saving, suggesting a “reduction in water-fetching labor by women and children,” as proposed in subgoal (a).

Table5 Time to Fetch Water

Before After

Less than 10 minutes

14 87

10 to 30 minutes

71 61

More than 30 minutes

65 2

Source: Beneficiary Survey

Furthermore, (d) the question of whether “people have an enhanced awareness of self-maintenance and have acquired sustainable skills for the O&M of water facility” is slightly problematic because, as explained in the “sustainability” section, once skills are

15 Other respondents include “increase in consumable water” (3), “time saving” (9) and “increase in

income” (9).

13

acquired through training and technical assistance, a lack of sufficient follow-up measures for additional training and monitoring makes it difficult to sustain the level of O&M skills or transfer them to others.

3.4.2 Other Impacts

This project has not involved any relocation of people or expropriation of land. No environmental impact from the construction and operation of water facilities has been mentioned in interviews or field surveys.

Hence, the project has generated the results planned and has had a high impact.

3.5 Sustainability (Rating：②） In the Lindi and Mtwara regions, 174 wells (with manual and electric pumping

systems) in the target villages (originally 64, but subsequently extended to include 69 villages of which 2 villages have unserviceable wells; thus, 67 de facto target villages) have been installed. The following sections describe the overall situation of Operations and Maintenance (O&M) based on an interview with the Ministry of Water, water engineers at the regional and prefectural levels, and the VWCs.

3.5.1 Structural Aspects of Operation and Maintenance The Ministry of Water, an executing agency of the project, delegates almost all its

O&M responsibilities to the prefectural level and daily O&M is conducted by LGA. Each village or facility has Community-Owned Water Supply Organizations (COWSOs), and they have the responsibility to provide water for people. However, COWSO has been institutionalized only since 2009 16 and, in many cases, the previous water supply

system (the VWC）continues to exist. On the other hand, the regional government plays an advisory role, and heavy machines, which are difficult to be purchased by an individual district, are also owned by the region to make them disposable at the district level when necessary.

Under the ongoing “Water Sector Development Program,” a “Water and Sanitation Team” is formed with water, regional development, health, and education divisions to strengthen the support systems for monitoring, repair of facilities and equipment, and other necessities at the village level.

As for daily O&M, the District Water Engineer (DWE) is considered to be operational

16 Water Supply and Sanitation Act No 12 of 2009.

14

with necessary timing and staff. 17 However, for more complex operations requiring a certain level of skills, engineers at the central level often fill in the gap and support repairs in the field.

The organization of the community, one of the activities implemented under this project, is highly appreciated at the regional level. The Regional Water Engineer (RWE) in Lindi evaluated this activity highly, saying “I have instructed that the measures taken under the project to enlighten communities shall be well coordinated between community development staff and water engineers”. These beliefs have been confirmed by facts. 3.5.2 Technical Aspects of Operation and Maintenance

As explained in the “qualitative” effect, core personnel in water supply, including prefectural water division staff, have acquired O&M skills and related equipment handling skills through OJT and other training. Hence, in many cases, technology is considered to have been transferred to villages, districts, and regions.

Interviews in villages reveal that manuals provided under the project for VWCs are not necessarily utilized. As a result, daily technical problems are not appropriately dealt with or, in some cases, acquired skills are not accumulated. For example, it has been pointed out by an RWE that the manuals were “not user friendly and could only be understood by DWE engineers, but not accessible to village- and community-level users due to a lack of feedback.”

This example illustrates a situation in which, in addition to the above-mentioned problem related to the manuals, the way training and technical advice was conducted was not effective enough (for them to be passed on to new staff, because the knowledge acquired remained at the individual level), nor was there any follow-up training or monitoring led by the district in order to fill in the skill gap, so that the O&M skills required for the wells have not been retained institutionally.

In order to overcome such problems, it is important to establish “capacity development,” for which the “institutionalization or organization” of O&M skills plays an important role. Interviews with VWC have reported some successful cases of organization, but the extent to which it has seen success remains difficult to determine. 3.5.3 Financial Aspects of Operation and Maintenance The annual budget for RWE in Lindi is about 9 million Tanzania shillings, which is barely sufficient for the monitoring and other minimum services. Mtwara has about 30 million

17 Interview with DWE

15

shillings. 18 The major responsibilities for water supply rest at the district level. Each district has had a stable or slightly declining budget over the past 3 to 5 years; budget size differs among individual districts, but is roughly 100 to 500 million shillings, of which 5 to 10 million shillings are allocated to the O&M of water sector. This budget size is considered sufficient to cover socialization and activities related to O&M at the village level.

On the other hand, collection of fees for O&M rests with villages: some villages charge for the amount of water used, while others charge fixed monthly or yearly fees per household or for all people above a certain age. The tariff for one bucket (20 liters) is about 20–30 shillings in the previous system. The target villages have received technical assistance as part of the soft component of the project, and they have regular bookkeeping practices for revenue and expenditure records. This also implies that the collected fees (about 100–200 thousand shillings) are enough to cover the O&M costs.

3.5.4 Current Status of Operation and Maintenance

The condition of O&M differs across facilities but out of 174 installed, about 80% are “fully” or “partially” operational, while 17% are under repair or going to be repaired and the remaining 3% are unrepaired. 19

Examples of out-of-order facilities include Newala (Mrunga and Okonboji) in Mtwara, where only one out of six is working, and five are found to be nonoperational due to malfunction or lack of a water source, as confirmed in the field survey. Finding more precise causes of the malfunction may require further technical inspection (according to DWE, this is because of missing length and the strength of “riser pipes”), but whatever the reasons, the people in these two villages must walk to the next village for over an hour to get safe water. 20

On the other hand, when the skills and knowledge necessary for planning, monitoring, and repair are not passed on to new staff, it is likely that those villages will have difficulty in conducting an effective O&M. In case of repair by DWE, there should be some cases requiring technical advice from the central government, but no such case has been reported in this survey.

Some problems have been observed in terms of technical aspects, therefore

sustainability of the project effect is fair.

18 RWE is a coordinating agency in charge of overall water supply and sanitation sectors and provides support to DWE, but it is not an implementing agency and has no investment budget on its own. 19 Interview with district water engineers in target areas. 20 Interview with district water engineers in target areas.

16

4．Conclusion, Lessons Learned and Recommendations 4.1 Conclusion

This project is intended to support the water supply facilities in the Lindi and Mtwara regions, and thus increase water supply pervasion and supply safe water in the target areas continuously. Considering Tanzanian and Japanese development policies, the project is highly relevant. It is fairly efficient because the project cost remains within the original estimate, although the period has slightly increased. On the other hand, the population with access to safe water has increased, and there has been a decline in the incidence of waterborne diseases; thus, the effectiveness and impact of the project are high. O&M has had some problems in technical aspects, and the project is considered fairly sustainable.

In light of the above, this project is evaluated to be satisfactory

4.2 Recommendations 4.2.1 Recommendations to the Executing Agency

It is recommended that training and monitoring be followed up by district (or region), targeting COWSO personnel in charge of O&M. This is intended to support self-help efforts (budgetary or personnel efforts) by target districts in Lindi and Mtwara after the project is over. 4.2.2 Recommendations to JICA

None 4.3 Lessons Learned

None

1

United Republic of Tanzania Integrated Malaria Control Project

External Evaluator: Rui Hiwatashi, Waseda University

0. Summary

This project was consistent with Tanzania’s policies in the health sector and specifically in malaria control, and matched the country’s development needs and Japan’s ODA policy. Thus, it is highly relevant. Although the Nursing Care Component came close to fully achieving its planned target, the Laboratory Diagnosis and Environmental-Management Components did not adequately achieve their planned targets. As for the achievement of the Overall Goal, although it is not possible to measure how the target values were met and to what extent the project contributed to them, the Nursing Care Component is presumed to have contributed somewhat to the Goal, and the others only to a limited extent. Hence, the effectiveness of the project is fair. While the project period of cooperation was within the plan, elements of inputs were partially inappropriate and the project cost exceeded the plan; therefore the efficiency of the project is fair. The effectiveness of the Nursing Care Component has achieved high sustainability with little concern about the financial aspects. However, in the Laboratory Diagnosis and Environmental-Management Components, some problems have been observed in the aspects of policy background, institutional and operation, technology, and finance, and therefore the sustainability of the project effects in general is fair.

In light of the above, this project is evaluated as partially satisfactory.

1. Project Description

Project Location (Map of Tanzania) 1 District-Based Training (Nursing Care)

1 16 municipalities/districts (now 17, under current administrative divisions) throughout the

2

1.1 Background In Tanzania, malaria has been causing socioeconomic distress among the people, along

with HIV/AIDS. Malaria accounted for 40% of all outpatient diagnoses. It was estimated that 14 to 18 million people were infected with malaria each year, among which 1.1 million was hospitalized due to severe conditions and 10,000 died. Most deaths due to malaria were among pregnant women and children under five. 2

To address these problems, the Ministry of Health (MOH 3) of the Government of Tanzania (GOT) has formulated and implemented a National Malaria Control Program (NMCP), which focuses on “Early Diagnosis and Treatment,” the methodological concept recommended by the Roll Back Malaria (RBM) partnership 4.

The Japanese International Cooperation Agency (JICA) continuously supported the efforts by the GOT: cooperation activities on anti-malaria issues mainly by grant aid from 1986; in-country training courses on early diagnosis using the “Acridine Orange” (AO) method; 5 and management of severe malaria patients and early treatment, from 1993 to 2002. This training resulted in certification of the AO method as one of the standards by MOH and the Medical Laboratory Scientists Association of Tanzania.

In addition to the above-mentioned achievements, the GOT requested that the Government of Japan provide another round of technical cooperation in the areas of strengthening malaria case management and environmental control of Anopheles (malaria-spreading mosquitoes) breeding sites in urban centers.

1.2 Project Outline

Overall Goal Malaria mortality is reduced

Project Objective 1. Evidence-based malaria case management with the focus on nursing care and diagnosis of patients is improved in

country were selected as actual project locations. 2 Ex-Ante Evaluation Report (2004). 3 Renamed Ministry of Health and Social Welfare (MOHSW) during the project. 4 The RBM partnership is the global framework to implement coordinated action against malaria, forged by the World Health Organization (WHO) and its member countries. 5 The AO method is one of the microscopic diagnoses for malaria, invented in 1991 by Professor Fumihiko Kawamoto, then of Nagoya University. The method is superior in terms of expeditiousness, accuracy, and simplicity compared to the traditional method, Gimsa, and entails the use of a microscope with a halogen lamp.

3

health facilities. 2. A sustainable environmental-management model to reduce

Anopheles breeding sites is established.

Outputs (Nursing Care Component) 1. District nurse trainers improve their training skills in

nursing care of malaria by training of trainers (TOT). 2. The knowledge and skills of the district nurse trainers are

adequately transferred to nurses by means of cascade training.

(Laboratory Diagnosis Component) 3. Laboratory technicians acquire skills and knowledge to

examine blood slides by means of the AO method, to conduct user maintenance of AO microscopes and to report laboratory practices to the Council Health Management Team (CHMT).

4. Health facilities can properly maintain AO microscopes and procure AO consumables.

(Environmental-Management Component) 5. Existing malaria drains 6 in Dar es Salaam are cleaned. 6. The communities in Dar es Salaam can properly manage

their environment to control Anopheles.

Inputs

Japanese Side: 1. Experts

One for long-term, six for short-term 2. Equipment 36,320,000 yen 7 3. Local Cost 122,290,000 yen 4. Other

Project Monitoring Mission (February 2007) Terminal Evaluation Mission (July 2007)

Tanzanian Side: 1. Two counterparts 2. Land and Facilities, Project Office, Utilities 3. Local Cost (spare parts and consumables for microscopes,

6 Drains with breeding sites of Anopheles. 7 Based on the exchange rate of 1 Tanzanian shilling (Tsh) to 0.11 Japanese yen, as of November 30, 2007.

4

etc.), Counterpart Salary, Seminars (other than those for which the Japanese side bore the costs)

Total Cost 290,000,000 yen

Period of Cooperation November, 2004–November, 2007

Implementing Agency Ministry of Health and Social Welfare (MOHSW) National Malaria Control Programme (NMCP)

Cooperating Agency in Japan

None

Related Projects (if any)

Projects related to malaria control implemented continuously from 1988 to 2002 (in-country training courses, etc.); International Parasite Control Project in Kenya; Urban Malaria Control Project (Swiss Tropical Institute [STI] and Bill & Melinda Gates Foundation); Pilot Project for Malaria Control (Global Fund to Fight AIDS, Tuberculosis and Malaria [GFATM], STI, and Princeton University); the common basket fund for the health sector; projects for bed nets (GFATM, United Nations Children's Fund [UNICEF], the Government of Switzerland, the British Embassy in Tanzania, the Netherlands Embassy in Tanzania, international NGOs, etc.); assistance for drafting of guidelines on and ensuring supply of curative drugs for malaria in Zanzibar (GFATM); and overall assistance for national malaria control (WHO’s RBM office).

1.3 Outline of the Terminal Evaluation

1.3.1 Achievement of Overall Goal As the latest data for malaria death rate available at the time of the Terminal

Evaluation were collected in 2005, it was not possible to measure the contribution of this project, which started in 2004.

1.3.2 Achievement of Project Objective The Nursing Care Component achieved its planned project objective. The Laboratory

Diagnosis Component mostly achieved the project objective—at more than 80% of health facilities, the provided AO microscopes were being operated and the overall system

5

(diagnosis record management, quality management, AO microscope maintenance, consumables procurement, etc.) established by the project was functioning at the time of the Terminal Evaluation. The Environmental-Management Component also achieved many of the planned objectives.

1.3.3 Recommendations Nursing Care Component: It was recommended that: 1) the training module be widely

shared through workshops or conferences among stakeholders in Tanzania as well as from overseas; 2) the training module be utilized through pre-service as well as in-service training in order to spread knowledge nationwide; 3) for further expansion of training especially on a regional level, the execution structure of cascade training may need to be strengthened by establishing a coordinating function on the regional level; and 4) the NMCP should have the training module reviewed and modified based on its new policy and on studies.

Laboratory Diagnosis Component: It was recommended that: 1) the role of the NMCP to supervise and provide technical guidance and support for laboratories on regional and district levels be strengthened; 2) the availability and distribution of quality supplies be ensured; 3) a resource directory, which may contain contact places for repair and other information, be prepared and distributed to health facilities, by the end of the project. It was also recommended that: 4) a concrete strategy of applying different diagnosis methods be prepared; 5) the role of microscopic diagnosis be clarified by the NMCP; and 6) MOHSW explore the possibility that the region might play a technical-backstopping role for district-level microscopic diagnosis after the termination of the project.

Environmental-Management Component: It was recommended that: 1) further analysis be made by the end of the project to determine the effectiveness of the work; 2) collaboration among communities, local government, and MOHSW be enhanced; and 3) these three actors continue disseminating knowledge on malaria prevention.

2. Outline of the Evaluation Study

2.1 External Evaluator Rui Hiwatashi, Waseda University

6

2.2 Duration of Evaluation Study Duration of the Study: December 2010–December 2011 Duration of the Field Study: February 14 to 27 and July 19 to August 1, 2011. 2.3 Constraints During the Evaluation Study (if any)

The target indicator levels for the objectives and outputs of this project were not set

from the designing phase until the termination of the project period. The selected indicators also turned out not to be very suitable for use in the evaluation, monitoring, and implementation of the project. The baseline data for the overall goal and project objectives were lacking as well. Additionally, it was not before 2008, the year in which the “NMCP Monitoring and Evaluation Plan 2008–2013” was developed, that state-of-the-art indicators and reliable data for malaria control were set and collected comprehensively in Tanzania. Hence, quantitative before-and-after comparison is conducted only to a limited degree in this report.

3. Results of the Evaluation (Overall Rating: C 8)

3.1 Relevance (Rating: ③ 9)

3.1.1 Relevance to the Development Plan of Tanzania During the project-design phase from November 2003 to February 2004, MOH was

implementing its “NMCP Medium-Term Strategic Plan 2002–2007” as the principal policy for malaria control. The medium-term plan focused on Early Diagnosis and Treatment, the basic principle under the RBM partnership, and set vector control as one of the main strategies. Hence, it is considered that the components of this project were selected in accordance with the country’s policy priorities.

The Nursing Care Component also conformed to the methodologies and measures listed in the strategic plan, including the improvement of guidelines and working references and the expansion of training courses. However, the strategic plan did not recommend any specific diagnostic methodology for the promotion of early malaria diagnosis. In fact, the characteristic of the AO method is expeditiousness, not only which

8 A: Highly satisfactory, B: Satisfactory, C: Partially satisfactory, D: Unsatisfactory 9 ③: High, ②: Fair, ①: Low

7

materializes early diagnosis. This was also pointed out during the project-design phase, emphasizing that the method would fully contribute to the promotion of Early Diagnosis only when it could be properly utilized at the health facility level.

With regard to vector control, while rehabilitation and maintenance of malaria drains was referred to in the strategic plan as a cost-effective measure, the approach of community-based cleaning and maintenance, on which this project focused, was not mentioned.

The strategic plan continued implementation without any major change until the time of the termination of the project in November 2007. However, as the malaria control effort in Tanzania is significantly affected by GFATM both technically and financially, MOHSW decided to focus attention on the dissemination of rapid diagnostic tests (RDTs), which was one of the GFATM’s recommendations, and to drastically scale down the component of microscopic diagnosis in the preparation of a GFATM grant proposal in 2007, resulting in approval from GFATM. In this way, the Laboratory Diagnosis Component of the project came to conform less to the country’s policy direction. The project’s approach of community-based cleaning and maintenance remained unmentioned in the mid-term plan for the following term, 2008–2013. While it might have been difficult to forecast the future mainstreaming of diagnostic methodologies in the project-design phase, the sustainability of the project’s achievement was affected, as described later, by this fade-out of conformity with policies at the level of specific methodologies or approaches.

3.1.2 Relevance to the Development Needs of Tanzania The latest statistics at the project-design phase 10 showed that the number of malaria

patients amount (as of 2000–03) annually to 1.4 to 1.8 million, accounting for 40% of all outpatients in Tanzania. Of them, 1.1 million have had to be hospitalized due to their severe conditions, and over 10,000 deaths occurring among them are attributable to malaria. Malaria was one of the major diseases in Tanzania, along with HIV/AIDS. Most deaths due to malaria were among pregnant women and children under five. Hence, there was a huge need for assistance of efforts in malaria control.

Additionally, it was stated in the 2003 report on the interim evaluation of the NMCP medium-term plan that efforts for increasing the use of bed nets have been sufficient and that there is no major funding gap at the present time. It was an appropriate approach for

10 The statistics in this paragraph come from the Ex-Ante Evaluation Report of the project and the NMCP Medium-Term Strategic Plan 2002–2007. They were originally cited from statistics for 2000 to 2003 compiled by the WHO and the GOT.

8

this project to select areas with technical and financial needs considering demarcation to other donors.

3.1.3 Relevance to Japan’s ODA Policy

Japan’s strategies for global malaria control are based on the Hashimoto Initiative for

Global Parasite Control, proposed in 1997, and the Okinawa Infectious Disease Initiative, announced in 2000.

The Country Assistance Program for Tanzania, formulated in 2000, which was effective during the project-design phase, stated that it would be important to continue the provision of medical equipment to key hospitals and malaria control projects, focusing especially on enhancement of medical care services in rural areas.

Therefore, this project has been consistent with Japan’s basic strategy for malaria control and with its assistance policy for Tanzania.

This project has been highly relevant to Tanzania’s development plan and development

needs as well as Japan’s ODA policy; therefore, its overall relevance is high.

3.2 Effectiveness and Impact (Rating: ②) In this project, there was no quantitative target level set in advance for indicators of

the overall goal, project objectives, and outputs for this project. Although indicators were defined on the project design matrix (PDM), the progress of the project was monitored only qualitatively. It was only when the Project Monitoring Mission was dispatched in February 2007 that the indicators were revised to be measurable quantitatively. Therefore, the below evaluation does not include quantitative comparison with planned targets.

3.2.1 Effectiveness

3.2.1.1 Project Output

[Nursing Care Component] Output 1: District nurse trainers improve their training skills in nursing care of

malaria by TOT. Pre– and post-test results on project TOT (Table 1) showed that average scores

9

increased by 29–40 points from pre-test to the post-test, with approximately 80% in score.

Table 1 Achievement of Output 1

Indicator* Achievement

1) Number of district nurse trainers who participated in training courses

34

2) Proportion of the participants who passed the post-test at each training session for district nurse trainers

Average scores: - for the training course held in January 2006

increased from 53% for pre-test to 82% for post-test.

- for November 2006, increased from 35% to 79%.

Source: Terminal Evaluation Report (Aug 2007) and Final Report (Jan 2008) of the project * No target value was set for indicators from the design phase to the termination of the project.

Interviews with and replies to the questionnaire from medical and nursing officers in

the 17 targeted districts/municipalities show that the nurse trainers from 16 districts/municipalities acquired the capacity to conduct district-based training (no response from the other district).

Furthermore, a beneficiary survey 11 shows that nearly all of the surveyed nurses (103 out of 104 samples) who participated in training by district nurse trainers were satisfied. The major satisfying factors were: focus on children under five years of age (answered by 80); focus on pregnant women (70); and facilitation of trainer (63).

Hence, Output 1 is considered to have been achieved, as the district trainers for nursing care of malaria improved their capacity enough to continuously conduct district-based training.

Output 2: The knowledge and skills of the district nurse trainers are adequately transferred to nurses by means of cascade training.

The comparative analysis conducted in the project (see Table 2) between the survey for

11 Survey questionnaires were sent to 128 field nurses who participated in district-based training courses supported by the project and were followed by in-person or telephone interviews when needed. A total of 104 answers were collected from all 16 targeted districts/municipalities.

10

baseline (176 samples) and for impact (92 samples) regarding district-based training showed that average scores increased by 30 points from baseline to impact, with approximately 70% in score.

Table 2 Achievement of Output 2

Indicator* Achievement

1) Number of nurses trained by district nurse trainers

558

2) Proportion of participants who passed the post-test at each cascade training session for nurses

Average scores for all of the training sessions conducted during the project period increased from 39.2% for pre-test to 72.3% for post-test (with the increase apparent in all of the targeted districts/municipalities).

Source: Terminal Evaluation Report (Aug 2007) and Final Report (Jan 2008) of the project * No target value was set for indicators from the design phase to the termination of the project.

This impact survey also showed, as indirect evidence, that patients and caretakers

leaving the hospital better understood information given by nurses after as compared to before the training. 12

Furthermore, the beneficiary survey shows that the district nurse trainers made good use of the curriculum and teaching materials developed through the project when conducting training sessions, which enjoyed a high reputation from participants. In this survey, most participant nurses answered that their self-efficacy was increased through the training sessions (99 nurses out of 104). Some concrete examples of skill improvement reported by the respondents were: improved skill in primary diagnosis; stronger confidence and more knowledge about cases that can be dealt with as prescribed in the manual; and better instructions about points such as drinking sufficient water.

It also indicated that the training content was adequate in terms of sustainability of the training effect and of ease of dissemination at the field level: 95 nurses out of 104 currently refer to the training materials more than once every few months; and 83 nurses have had the opportunity to conduct training for their colleague nurses who did not receive training in the project.

Hence, Output 2 is considered to have been achieved, as the knowledge and skills of

12 Fifty-five inpatients and their caretakers were asked six basic questions that needed to be understood correctly by them at the time of discharge from hospital, both before and after nurse training. The average score for all the questions improved after the nurses participated in the training session.

11

the district nurse trainers are adequately transferred to nurses, leading to the enhancement of their practical capacities.

This component was successful in encouraging initiative on the Tanzanian side because it adopted a hierarchical training system, called a “cascade approach,” 13 in light of Tanzania’s administrative structure in the health sector. In order to make this system run and maintain quality of sessions during implementation, the project consolidated training materials and protocols—the training package contained a comprehensive curriculum including monitoring activities as well as a standard budget model for reference in budgeting at a district level. Furthermore, the project utilized low-tech tools such as printed flip charts so that training sessions could be held in rural areas with no electricity and involved local human resources (both trainers and participants) from early on, with the aim of developing user-friendly materials for training. These various efforts are also considered to have contributed to its achievement.

[Laboratory Diagnosis Component] Output 3: Laboratory technicians acquire skills and knowledge to examine blood

slides by means of the AO method, to conduct user maintenance of AO microscopes and to report laboratory practices to the Council Health-Management Team (CHMT).

Pre and posttest results conducted during the training course for laboratory personnel (Table 3) showed that average scores increased by 10–30 points from pretest to posttest, with average scores higher than 85%.

Table 3 Achievement of Output 3

Indicator* Achievement

1) Number of laboratory technicians who participated in the training courses

95

2) Proportion of participants who passed the posttest at each training session for laboratory technicians

Average score

Pretest Posttest

September 2005

Sensitivity 75.5% 93.1%

Specificity 76.5% 86.3%

13 The training programme was implemented in three steps: (1) NMCP-trained national trainers; (2) TOT for district nurse trainers by the national trainers; and (3) district-based training.

12

January 2007 Sensitivity 58.0% 87.5%

Specificity 90.9% 89.8%

September 2007

Sensitivity 71.1% 91.1%

Specificity 80.0% 93.3%

Source: Terminal Evaluation Report (Aug 2007) and Final Report (Jan 2008) of the project * No target value was set for indicators from the design phase to the termination of the project.

Interviews with and replies to the questionnaire by medical officers and laboratory

technicians in the targeted districts/municipalities show that in the nearly all of these districts/municipalities (16 out of 17), the effect of the training courses was satisfactory.

Some concrete examples of acquired skills reported by the respondents were the AO diagnosis procedure; comprehension of proper combination with other methods; and techniques for daily maintenance and fine-tuning.

Hence, Output 3 is considered to have been achieved, as the most laboratory technicians who participated in the training courses learned how to conduct laboratory diagnosis of malaria by the AO method and acquired the knowledge, techniques, and procedures to operate and maintain AO microscopes.

Output 4: Health facilities can properly maintain AO microscopes and procure AO

consumables.

It had been repeatedly emphasized since the project-design phase that the key factor for

successful diffusion of AO diagnosis was to establish systems for maintenance of microscopes and procurement of related consumables and spare parts. According to an interview with the Japanese long-term expert, the project aimed to implement a variety of strategies to diffuse the AO method all over the country as one of the standard methodologies of laboratory diagnosis in Tanzania, knowing existence of many difficulties in building the systems. There was an expectation that once the method was diffused widely in the country, the distribution of consumables and spare parts would be secured on a commercial basis.

The survey on utilization of the AO microscopes conducted by the project in June 2007 (49 facilities surveyed of the 65 where the microscopes were supplied) 14 revealed that

14 When the external evaluator reviewed the original list for this survey, however, the number of surveyed facilities was 48.

13

while 41 microscopes were utilized, eight were not, 15 mainly due to absence of trained laboratory technicians.

However, it is observed at the time of this evaluation survey that the problematic situation has not yet been improved where a broken microscope is not properly repaired or spare parts and consumables are not available; 16 although this issue was already recognized during the implementation of the project.

When the original list for the survey on AO-microscope utilization was rechecked taking the above-mentioned background into account, about eight cases among hospitals/health facilities with the status “AO microscope being utilized with some problem” might have possibly remained unutilized after the time of this survey. Those problems were (1) the hospital’s policy did not put a strong focus on the AO method (2) human resources and skills were lacking due to the transfer, resignation, and/or study abroad of trained lab-technicians (3) broken microscopes had not been repaired and (4) consumables such as AO reagent and spare parts were not available. There is a possibility, according to this analysis, that up to 19 hospitals/health facilities among those surveyed might have already stopped utilizing their microscopes at the time of the survey.

The interviews with and replies to the questionnaire from the officers in the targeted districts/municipalities show that most of them (16 out of 17) cannot afford the needed consumables, as the AO method has not been mainstreamed and thus is not prioritized in budgeting. Other problems are also reported: for example, cases are seen where AO reagent is not available due to lack of distribution even in cases where the budget is allocated; or shorted-out microscopes remain unrepaired and unreplaced due to lack of human resources and skills of zonal workshops, and lack of distribution of necessary spare parts. 17

During the project-design phase, the Japanese side expected that it would be possible to secure a stock of relevant products by steady negotiation with the government agency responsible for medical procurement and that it would be feasible to establish a system for microscope maintenance when it was seen that the project would help organizational and individual capacity in the refinement centers. However, this expectation was not realized. The policy-level focus on the AO method was lost, and negotiation could not secure a stock or distribution of needed products for repairs.

15 The original list of the survey also showed that 11 microscopes, not eight, were unutilized. 16 The lack of appropriate repair availability may be attributed to lack of a system for local procurement of as many as 2,000 different spare parts for one microscope, as well as insufficient organizational/individual capacity of the country’s agency responsible for the repair of microscopes. Details are in Section 3.4.2 (Institutional and Operational Aspects of the Implementing Agency). 17 These problems are detailed later in Section 3.4 (Sustainability).

14

Hence, Output 4 is considered not to have been achieved, as systems for hospitals/health facilities to maintain AO microscopes and procure consumables were not established.

[Environmental-Management Component] The objective of this component was to establish a model for community-based

management of malaria drains. The project planned to pick up target drains and communities and facilitate community-based activities for cleaning and maintenance of drains that would lead to establishment of strong systems for environment management.

Output 5: Existing malaria drains in Dar es Salaam are cleaned.

The project selected two drains in Dar es Salaam containing Anopheles mosquito breeding sites and cleaned them as described in Table 4 below. The initial plan had been to have the drains cleaned by community people themselves, but as both the scale of the drains and the volume of dumped wastes far exceeded expectations, the project decided to contract a private company to clean the drains with heavy machines.

Table 4 Achievement of Output 5

Indicator* Achievement

The distance of drains in Dar es Salaam cleaned during the project period

1.7 km (Mtoni ward) 2.1 km (Magomeni ward)

Source: Terminal Evaluation Report (Aug 2007) and Final Report (Jan 2008) of the project * No target value was set for indicators from the design phase to the termination of the project.

It can be said that the output “to clean” drains itself was achieved. However, as this

was done by means of financial input from Japan and private contractors with heavy machines, the result was different from what had been originally expected—that the experience would be disseminated widely as a model of community-based environment management.

Output 6: The communities in Dar es Salaam can properly manage their environment to control Anopheles.

The project conducted education activities involving community leaders and members

15

as described in Table 5 below for the selected communities to establish systems for environment management.

Table 5 Achievement of Output 6

Indicator* Achievement

Proportion of Wajumbe (community leaders) who disseminate knowledge learned in seminars

- Community leaders’ seminars: 18 (550 leaders)

- Mass meetings: 10 (4,000 community members)

- Community education through seminars and household visits by Community Owned Resource Persons (CORPs): 109,164 community members

Source: Terminal Evaluation Report (Aug 2007) and Final Report (Jan 2008) of the project * No target value was set for indicators from the design phase to the termination of the project.

The “Best Practice” document prepared based on project activities showed that the

involvement of communities was successful and cleaning activities were implemented by community people on a large scale, along with massive cleaning by heavy machines.

However, as the Best Practice document and follow-up survey report pointed out and the field visit in the ex-post evaluation confirmed, the condition of one of the cleaned drains was soon as bad as before cleaning, because community people resumed dumping waste into the drain after the cleaning. According to an officer on Dar es Salaam city council, some of the primary causes were: the river into which the drain flows was so stagnant (due to lack of periodic massive maintenance by civil workers using heavy machines) that community cleaning only would not be visibly effective to improve the environment; thus, community people came to feel that community-based environment management would not work, and they did not have a proper place for dumping, as the public waste-management system was not functioning.

It must be pointed out that the project could not control all these constraints, but the establishment of a system for environment management cannot be observed outside of their context.

3.2.1.2 Achievement of Project Objectives

1) Project Objective 1 Evidence-based malaria case management with the focus on

nursing care and diagnosis of patients is improved in health

16

facilities. [Nursing Care Component]

Indicator 1. The knowledge, attitude and performance of nurses in

management of malaria patients are improved. Indicator 2. The satisfaction of patients with services provided is improved

in the following areas: - Nursing care of inpatients with malaria - Health education of inpatients and outpatients

Indicator 3. Project activities are institutionalized from the following

perspectives: - Official recognition of the training models by the

government of Tanzania - Integration of the budget for the project activities into

CCHP

* No target value was set for indicators from the design phase to the termination of the project.

As discussed earlier in the context of Outputs 1 and 2, it is considered that the

knowledge, attitude, and practical skills of nurses were improved and it is observed that patients and caretakers leaving the hospital better understood the information given by nurses compared to before the training.

The training materials developed by the project were adopted as MOHSW’s official documents to take up unique identification numbers of the Malaria Control Series of NMCP.

A survey conducted during the project period in the 46 districts that participated in the advocacy seminar showed that at least 21 districts had integrated their budgets to include training in nursing care for malaria patients into their council plans for 2007/2008. The questionnaire survey and interviews with NMCP officials conducted in the ex-post evaluation also indicate that the training model established by the project would expanded throughout the country in the latest Medium-Term Strategic Plan of NMCP, and that this plan has actually been implemented with funding from the donors like the United States and Tanzanian governments, using the same curriculum and training materials as the MOHSW/JICA project. Although the conditions of the training model’s continuous

17

implementation may vary among districts/municipalities, the questionnaire survey shows that since this model has been widely implemented when sufficiently financed by donor funds or other measures; the project’s activities have been almost institutionalized. This component was successful, as mentioned earlier, in encouraging initiative on the Tanzanian side in terms of financial and human resources, as the project adopted a cascade approach.

Hence, as far as the Nursing Care Component is concerned, Project Objective 1 has almost been achieved.

[Laboratory Diagnosis Component]

Indicator 4. The accuracy of blood-slide examination results for febrile

cases is increased. Indicator 5. The proportion of blood slides used for diagnosis of malaria is

increased at health facilities.

* No target value was set for indicators from the design phase to the termination of the project.

As discussed earlier with regard to Outputs 3 and 4, the effects of the training courses,

such as improvement in the accuracy of the blood-slide examination and the utilization of microscopes, were observed for a certain period after the implementation of the training. However, at least 10 to 20 microscopes provided to hospitals/health facilities later became un- or incompletely utilized owing to breakdowns and other problems before the project ended.

Hence, as far as the Laboratory Diagnosis Component is concerned, Project Objective 1 has not been achieved.

2) Project Objective 2 A sustainable environmental management model to reduce

Anopheles breeding sites is established.

[Environmental Management Component]

Indicator 6. The number of mosquito (Anopheles) breeding sites in the selected target areas of Dar es Salaam has decreased.

* No target value was set for indicators from the design phase to the termination of the project.

18

According to a report that compared the project with another one , which focused on

larvicide, to analyze the difference in malaria infection rate between these cases 18, the drain-cleaning approach adopted by the project had a statistically significant positive effect. The report also showed that the drain-cleaning intervention contributed to decreasing malaria infection rates when compared to the pre-intervention condition of the same area and to areas that had not received this intervention.

These findings suggest that the number of mosquito breeding sites in the selected target areas decreased for a certain period after the cleaning with heavy machines funded by financial inputs by the project. However, the interviews in the ex-post evaluation as well as this comparison analysis report indicate that the condition of one of the cleaned drains became as bad as it was before cleaning because the community residents resumed dumping waste into the drain soon after the cleaning. The interviews attributed this setback to the fact that the maintenance portion of the budget of the said area was limited and was allocated to the civil engineering division, not the public health division, resulting in inadequate implementation of environment maintenance for vector control of malaria.

The project aimed to establish a management model. The NMCP’s latest Medium-Term Strategic Plan (2008–2013) refers to the project activities as “Best Practice,” indicating that these will be adopted in other urban areas. Nevertheless, according to the interviews in the ex-post evaluation, community-based environmental management is only viable when it comes with civil engineering with respect to rivers and drains, and waste management systems, especially garbage collection. Several NMCP officials and the public health officer from Dar es Salaam City Council opined that the said model cannot be implemented in a sustainable manner in the absence of good interministerial or interdepartmental coordination between the public health and civil engineering divisions. Although this challenge had already been recognized during the project-design phase and the project was expected to address it, the result was not satisfactory.

Accordingly, the project should have also supported coordination with or advocacy to ministers and departments responsible for civil engineering and waste management.

Hence, Project Objective 2 has been achieved only partly in terms of the cleaning of the targeted drains, and not at all in terms of the establishment of the model.

To sum up, although the planned target (Indicators 1 to 3) of the Nursing Care

18 Marcia Caldas de Castro (Department of Global Health and Population, Harvard School of Public Health), “Environmental Management of Anopheles Breeding Sites in Dar es Salaam—Phase III: Evaluation of EM Activities—Final Report—October, 2008.”

19

Component was almost achieved, the planned target of the Laboratory Diagnosis Component (Indicator 4 and 5) and the Environmental-Management Component (Indicator 6) was not fully achieved. Hence, the objectives of this project were achieved to a certain degree; therefore its effectiveness is fair.

3.2.2 Impact

3.2.2.1 Achievement of Overall Goal, “Malaria Mortality is Reduced”

1) Indicator 1. Malaria mortality countrywide

* No target value was set for indicators from the design phase to the termination of the project.

The malaria mortality rate is considered very difficult to measure. 19 In Tanzania, this

rate varies depending on the information source. The countrywide malaria mortality rate is not adopted as indicator in the “NMCP Monitoring and Evaluation Plan 2008–2013,” which is being implemented at the time of the ex-post evaluation. As described in Table 7, according to WHO statistics, the absolute number of malaria deaths and their ratio to the total population of Tanzania have been declining.

Table 7 Number and Rate of Malaria Deaths in Tanzania

Year 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Population (thousand)

35,306

36,232

37,188

35,922

36,588

36,766

37,445

39,384

40,213

41,049

Malaria Mortality (reported)

- 1,228 8,15 15,25

1 19,85

9 18,32

2 20,96

2 12,59

3 12,43

4 840

Malaria Mortality Rate

(per 100 k) - 3 2 42 54 50 56 32 31 2

Source: WHO World Malaria Report 2010 and US Census Bureau However, it is difficult to measure the contribution of the project to the decline of the

malaria mortality rate. This was repeatedly pointed out by NMCP officials in the interviews. Malaria control in Tanzania has been extensively done in partnerships with 19 e.g. WHO, “Malaria deaths are the hardest to count,” Bulletin of the World Health Organization, Volume 84, Number 3, March 2006. http://www.who.int/bulletin/volumes/84/3/news10306/en/

20

many donors and using various approaches, including facilitation of the use of insecticide-treated nets (ITNs), indoor residual spraying (IRS), and artemisinin-based combination therapy (ACT), as well as the methods adopted by the project. RDTs became mainstreamed in the laboratory diagnosis of malaria during the project period, as mentioned earlier, and larviciding has been emphasized recently in environmental management for malaria control.

It is also extremely difficult to measure the contribution of nursing-care training in malaria treatment and the introduction of AO diagnosis to the nationwide results of project activities in only 17 target areas.

Having said that, the Nursing Care Component, which was continuously implemented after the termination of the project, is presumed to contribute to the decline of the malaria mortality rate in various ways. In particular, as one of the aspects of the component was to facilitate a steady expansion of ACT, it appears certain, generally speaking, that it has contributed to the decline of the malaria mortality rate through this expansion. It is impossible, however, to quantitatively evaluate the extent of the contribution, because the necessary conditions to quantify it, such as the other determinants of the malaria mortality rate, collection of baseline data of the targeted areas, or time-series data of uncontrolled areas for comparison, were not satisfied.

In addition, the project had not set a target value for the indicators, and it has been unclear how the project identified and understood the baseline situation and what the extent was of the expected effects. From this viewpoint as well, it is impossible to quantitatively evaluate the achievement of this Overall Goal of the project. As for the Laboratory Diagnosis and Environmental-Management Components, considering that their effects were not sustainable after the termination of the project, they contributed only to a limited extent to the achievement of this Overall Goal of the project.

2) Indicator 2 The malaria mortality in urban centers of Dar es Salaam

* No target value was set for indicators from the design phase to the termination of the

project.

During this ex-post evaluation, the time-series malaria mortality rates in Dar es Salaam

city were not available. According to an interview with an official on the city council, state-of-the-art, reliable statistics have begun being collected only recently, with the implementation of the NMCP Monitoring and Evaluation Plan 2008–2013.

21

To sum up the above discussion, this project has had some success in achieving its objectives; therefore its effectiveness is fair. Of the Project Objectives, while Indicators 4 to 6 were not fully achieved, Indicators 1 to 3 were achieved. As for the achievement of the Overall Goal, although it is not possible to measure how the target values were met and to what extent the project contributed, the Nursing Care Component is presumed to have notably contributed to the Goal, and the other components only to a limited degree.

3.3 Efficiency (Rating:②)

3.3.1 Inputs

Inputs Plan Actual Performance

(1) Experts One long-term Five for short-term

One long-term Six for short-term

(2)Trainees received N/A

N/A

(3)Third-Country Training Programs

N/A

N/A

(4) Equipment AO microscopes Solar generator

AO microscopes Solar generator

Total Project Cost 227,180,000 yen 290,340,000 yen

Total Local Cost Spare parts and consumables for microscopes Counterpart Salary Expenses for seminars other than those borne by the Japanese side (amount not available)

Spare parts and consumables for microscopes Counterpart Salary Expenses for seminars other than those borne by the Japanese side

(amount not available)

3.3.1.1 Elements of Inputs

According to a project document produced at the termination of the project, the Japanese experts were deployed mostly as planned. However, a project coordinator was also deployed, since the workload of the long-term expert was heavier than expected. According to the long-term expert, the workload related to logistics became heavier as the project progressed to cover a wide area. Although in the first year the situation had been addressed by local staff allocation, the project required an additional Japanese staff member to be deployed locally for the second year.

22

It had been difficult to estimate workload properly before launching the project, considering that it consisted of multiple components involving collaboration with multiple counterpart organizations and departments and covered target areas nationwide. However, as the long-term expert pointed out, it can be considered that normally the project coordinator is deployed from the beginning of the project, given that logistics issues increase naturally in a project involving capacity development, system building, and nation-wide expansion, and that it is generally difficult in African countries to secure local human resources that can play an appropriate middle-management role.

The project spent 36 million yen for equipment, which amounted to 23% of the total local cost of 159 million yen. Most of this amount was for AO microscopes and related equipment. Considering the current usage situation of the microscopes, this input is not considered to have been cost effective. It might have been better if more of the input had been allocated to enhancement of the system for procurement of spare parts and reagents and additional implementation of training at the zonal repair centers.

3.3.1.2 Project Cost

The actual project cost was higher than planned. It was 290,350,000 yen, 128% of the

planned 227,180,000 yen. 20 3.3.1.3 Period of Cooperation

The actual period of cooperation was 36 months, which was as planned. Hence, although the project period of cooperation was within the plan, elements of

inputs were partially inappropriate and project cost exceeded the plan, therefore efficiency of the project is fair.

3.4 Sustainability (Rating: ②) 3.4.1 Related Policy Surrounding the Project Tanzania’s ongoing Health Sector Strategic Plan III 2009–2015 keeps its emphasis on

the importance of malaria control, as before. However, it does not mention any concrete

20 The reason for the difference could not be determined, as the breakdown of the planned budget was not available at the time of this evaluation.

23

methodology for nursing care and laboratory diagnosis of malaria, but rather focuses on ITNs and IRS. It is planned to expand environmental management for vector control, but no detailed approach such as community-based drain cleaning is specified.

Nursing care and laboratory diagnosis of malaria remain a priority in the NMCP Medium-Term Strategic Plan 2008–2013. As for nursing care, it is planned to spread training for nurses on appropriate treatment of malaria patients to peripheral health facilities all over the country, for which, according to interviews with NMCP officials, the training package developed through the project has been utilized almost as it was during the project period.

As for methodologies of laboratory diagnosis of malaria, while the expansion of use of RDTs is emphasized, the AO method is unmentioned and the microscope in general is referred to as something to be utilized situationally. 21 NMCP officials, and medical officers and laboratory technicians in municipals or districts interviewed for this evaluation were of the opinion that AO diagnosis remains effective and necessary for Tanzania’s malaria-control efforts, even if RDTs has been mainstreamed. 22 However, the AO method has been less prioritized at the policy level, at least in the current Medium-Term Strategic Plan.

Environmental management has been considered in context of applying best practice in Dar es Salaam City to other urban areas. The health officer on the city council who formulated this policy for the Strategic Plan affirmed that “best practice” here implies the use of the approach adopted in the project. At the same time, however, it was also pointed out, as mentioned earlier, that the materialization of this policy would be possible only when budget allocation and planning coordination could be harmonized with those for civil engineering of rivers, drains, and waste-management infrastructure. The detailed approach focused on in environmental management is larviciding, which can be easily adopted elsewhere and produces a noticeable effect.

3.4.2 Institutional and Operational Aspects of the Implementing Agency

NMCP has been the key organization for the implementation of policies for malaria control in Tanzania. In NMCP’s organizational structure, in the same way under the current Medium-Term Strategic Plan as during the project period, the case-management

21 The interview showed that the AO method was not well discussed or emphasized compared to other methodologies in the process of drafting the Medium-Term Strategic Plan. 22 It was emphatically pointed out that the AO method is superior to others in expeditiousness and accuracy. It was also stated that although RDTs also enable rapid diagnosis, there is a necessity to combine both methodologies in an appropriate manner, such as the use of AO for the secondary, detailed examination based on an initial test by RDTs.

24

unit is one of the two major strategic units; the environmental-management team is under the other unit, for malaria prevention.

Several ways exist to expand the project’s approach to nursing care. For example, training material was revised once, after the project, from the viewpoint of medication and decision-making procedures, mainly by NMCP in collaboration with the training division of MOHSW and relevant players. Furthermore, the training itself has been incorporated in the programs of training agencies under the MOHSW administration. 23 The cascade approach, with consideration of the hierarchical health-administration system of the country, has contributed to securing the institutional sustainability of this training package.

The most serious problem for sustainability of the Laboratory Diagnosis Component is the fact that the procurement system for AO consumables and spare parts has not been established. It is true that in general and principle, basic healthcare services such as malaria control should be delivered with public funds, but it has also been recognized that the market distribution of microscopes and related products should have been addressed, since the project had intended to facilitate the AO method before the GoT substantially committed to adopt it. In Tanzania, the Medical Store Department (MSD) 24 has been in charge of securing stocks of consumables and spare parts for medical and healthcare equipment, and had been persuaded to stock these components for AO microscopes. However, for several reasons, such as the fact that the market for AO microscopes in Tanzania has been small and the current NMCP Medium-Term Strategic Plan has not put a priority on it, and also that MSD has been required to operate on a self-sustaining basis, it seems to have lacked any continuous incentive to stock AO-related products. Another challenge is how to allocate proper human resources to zonal repair centers to provide sufficient repair capacity for microscopes. Additionally, it is considered that the training system is not sufficient, as several cases has been reported in which the knowledge acquired by laboratory technicians through project training was not transferred to other staff members when the technicians moved, retired, or left for education. 25

As for environmental management in Dar es Salaam city, although it is necessary to achieve proper maintenance and renovation of rivers into which malaria drains flow, clean 23 According to the interview and questionnaire survey in the ex-post evaluation, the targeted districts/municipalities have made different levels of progress in implementation of training. Of 17 targeted districts/municipalities, 1/3 have implemented the training at an adequate frequency (once every one to two years), another 1/3 have implemented training but not enough, and the last 1/3 have not implemented training at all. For other areas than those covered by the project, the training is being provided under the US President's Malaria Initiative (PMI). 24 A department formally under MOHSW but operated independently, in charge of procurement, stock, and distribution of medical supplies, equipment, and consumables. 25 According to the interviews and questionnaire survey conducted in the ex-post evaluation.

25

drains with heavy machines, and establish a waste-management system, the collaboration process between ministries and departments in charge of these efforts has not been fully established. This lack has become an impediment to community-based environment management, and thus it is not clear that the community-based system for environmental management that the project developed remains effective at the present time.

3.4.3 Technical Aspects of the Implementing Agency

An appropriate technical level is maintained regarding nursing-care training. NMCP officials and relevant players have renewed by themselves training contents along with the advancement of methodologies and medicines. Thus, it is highly likely that they will maintain technical self-sustainability. The approach in which training materials developed by the project were incorporated into a series of GoT publications has also contributed to enhancing the situation with regard to maintaining relevant technologies within governmental operations.

As for AO diagnosis, it has a technical problem as well. It is highly likely that the necessary knowledge and skills, in terms of diagnosis and of maintenance and repair of microscopes, will not be shared properly when AO microscopes become viewed as something that should be utilized, because enough trainers will not exist and a training/diagnosis/repair system will not have been established, as mentioned earlier. The NMCP interviews revealed that the capacities of peripheral health facilities for deciding which diagnosis method to adopt according to the situation are not sufficient. NMCP guidelines still recommend microscopic diagnosis for patients who need detailed examination, while considering RDTs the basic diagnosis methodology for malaria. Lack of capacity at the field level is a major challenge for proper use of AO diagnosis in peripheral healthcare settings.

Environmental management is considered sustainable in its technical aspects. It was confirmed through this evaluation that the Dar es Salaam city council stores in an appropriate manner project deliverables such as the best practice document and pamphlets for information, education, and communication for community residents. The officials affirmed that these deliverables are always ready for reprint and distribution. For facilitation activities aiming to establish a community-based system for environmental management, these activities have usually been implemented by local counterpart organizations/individuals during the project implementation period; they are the main repository of techniques for further expansion of these activities. However, systems for integrated and effective utilization of these techniques, such as training systems, are not functioning at the present time.

26

3.4.4 Financial Aspects of the Implementing Agency Nursing care faces a financial gap between planned budget for NMCP training

implementation and actual GoT funding. However, this situation is normal for malaria control in Tanzania, and it has become routine to implement programmes with external funds from GFATM and other donors.

This trend, however, has become stronger recently. According to a report on financial gaps that the GoT developed in 2009 as part of a GFATM grant proposal, the annual budget of US$5.2 million for malaria control in the fiscal year 2006–2007 decreased to $2.8 million the next year and then $2 million in 2008–2009. At the same time, GFATM provided the GoT with $52.5 million for RDTs and ACT from 2008 to 2013, $59.8 million for voucher awareness-raising for pregnant women from 2008 to 2011, $113.3 million for long-lasting insecticidal nets (LLINs) and monitoring- and evaluation-capacity development from 2009 to 2014, and $173.6 million for ACT and home-based management of malaria. 26

It is forecasted that the malaria-control programme in Tanzania can be fully implemented with donor funds including those from GFATM, characteristically with a tendency to focus on areas on which GFATM puts a focus in terms of policy and technological support.

According to the NMCP interviews, a proposal to GFATM for financing nationwide expansion of the nursing-care training based on the model developed by the project has been submitted and is expected to be approved.

The problems in the situation of laboratory diagnosis and environmental management, as mentioned earlier, are that it is a lower policy priority and that there is insufficient coordination between ministries and departments; therefore, the financial aspect of its sustainability is also inadequately supported.

3.4.5 Continuity of Effectiveness/Impact As discussed above, while the effectiveness of the Nursing Care Component is

expected to be sustainable, the Laboratory Diagnosis Component has faced complicated problems and there is a concern that its effectiveness would fade away if the situation remains as it stands.

At the same time, however, given that one of the reasons for the lack of procurement

26 This information on budget and funds is based on the US PMI Malaria Operational Plan 2011.

27

systems and circumstances for AO products has been the small market for them in Tanzania, the possibility might be suggested that if the market expands on a commercial basis, which would mainly be among private medical facilities, the AO method would spread more widely. According to the local experts interviewed, due to the method’s effectiveness, expeditiousness, and accuracy, when an AO microscope that is durable, simple, and affordable enough to suit the circumstances of Tanzania is introduced, it seems likely that it will be distributed among private hospitals. 27 It is true that a careful feasibility study must be conducted beforehand, but it is expected that future efforts will be made by broader stakeholders, including those from the Japanese private sector, to realize this possibility, which is achievable with only small innovations.

The effectiveness of the Environmental-Management Component has not been sustained. Although there is a chance for organized community resources make it effective again in cases where the necessary conditions, such as improvement of rivers and drains and establishment of waste-management systems, are satisfied, it cannot be securely sustained in the long term without continuous coordination and an adequate budget.

To sum up the above discussion, although the effectiveness of the Nursing Care

Component is highly sustainable with few financial concerns, for the other two components some problems have been observed in the aspects of policy background, institutional and operation, technology, and finance, and therefore the sustainability of the project effects in general is fair. 28

27 According to the interviews with the officials in charge of laboratory diagnosis in MOHSW and NMCP, because the demand for introduction of the AO method had been recognized during the project-design phase and the advantages of this method in terms of expeditiousness and accuracy will contribute to improvement of the operational efficiency and reputation of a hospital, it is expected that the need from the management perspective of hospitals will be large. The officials also expressed the opinion that the major reasons for the lack of use of the AO method among private medical facilities are: cost and failure-proneness of the microscopes (due to unstable power supply); unpredictable availability of spare parts and consumables for the microscopes; and lack of institutional procedures for AO training. 28 JICA guidelines for ex-post evaluation do not provide a quantitative standard for sustainability but instead provide rough definitions of each rating: ②, “fair,” is appropriate for the conclusion that some problems have been observed in policy background or structural, technical, or financial conditions in the executing agency; and ①, “low,” when major problems have been observed. The evaluator concludes that while the sustainability of the effectiveness of the Laboratory Diagnosis and Environmental-Management Components is low, that of the Nursing Care Component is expected to be sufficiently high, and thus the overall project is considered to have partly succeeded in securing the sustainability of its effects.

28

4. Conclusion, Lessons Learned and Recommendations

4.1 Conclusion

This project was consistent with Tanzania’s policies in the health sector and

specifically in malaria control, and matched the country’s development needs and Japan’s ODA policy. Thus, it is highly relevant. Although the Nursing Care Component came close to fully achieving its planned target, the Laboratory Diagnosis and Environmental-Management Components did not adequately achieve their planned targets. As for the achievement of the Overall Goal, although it is not possible to measure how the target values were met and to what extent the project contributed to them, the Nursing Care Component is presumed to have contributed somewhat to the Goal, and the others only to a limited extent. Hence, the effectiveness of the project is fair. While the project period of cooperation was within the plan, elements of inputs were partially inappropriate and the project cost exceeded the plan; therefore the efficiency of the project is fair. The effectiveness of the Nursing Care Component has achieved high sustainability with little concern about the financial aspects. However, in the Laboratory Diagnosis and Environmental-Management Components, some problems have been observed in the aspects of policy background, institutional and operation, technology, and finance, and therefore the sustainability of the project effects in general is fair.

In light of the above, this project is evaluated as partially satisfactory.

4.2 Recommendations

4.2.1 Recommendations to the Executing Agency

4.2.1.1 Nursing Care Component

Although an appropriate organizational structure and technical level for expanding

training courses have been secured, progress on implementation of training varies by area. For example, some districts/municipalities have periodically implemented training, while others have achieved only insufficient implementation compared to annual intake. It is desirable to secure the continued provision of an adequate budget at national and district/municipal levels to spread the training across the country by leveraging the achievements of the project.

29

4.2.1.2 Laboratory Diagnosis Component

1) To approach MSD regarding the establishment of a procurement system for AO

consumables and spare parts

One of the major problems for this component is the lack of a stable procurement system for AO consumables and spare parts. Further negotiation with MSD is desired. In addition, it is expected that the introduction of AO microscopes to private hospitals will contribute indirectly to addressing this problem. Thus, it is desirable to seek to identify activities that the government can implement, such as conducting an AO-method dissemination seminar in which experts from the private sector can participate, to expand use of AO microscopes.

2) To provide thorough monitoring and supervision to health facilities to help achieve

appropriate decisions in selecting diagnosis methodology.

At the health-facility level, the method for diagnosis of malaria has often not been selected in an appropriate manner. It is recommended that thorough monitoring and supervision of health facilities be continually provided in order to materialize the strategic role of microscopic diagnosis, a use that can be properly combined with RDTs, at a field level.

3) To properly maintain and store the AO microscopes in preparation for a future

revival.

As described above, the reasons AO microscopes have not been utilized at a field level will not be easily addressed. However, the AO microscope still remains effective and necessary for malaria control, and will be a useful asset once conditions become right. It is desirable to ensure that in preparation for the future revival of use of AO microscopes, health facilities properly maintain and store the provided AO microscopes even if they are currently out of use.

4.2.1.3 Environmental-Management Component

Prerequisite conditions to sustain the effects of this component include proper

maintenance and renovation of rivers into which malaria drains flow, cleaning of drains with heavy machines, and establishment of a waste-management system. As mentioned

30

earlier, current circumstances do not satisfy these conditions, and limit the effect and sustainability of the project. It is recommended that efforts be continually made to realize good inter-ministerial and inter-departmental coordination between the public-health and civil-engineering areas on this issue.

4.2.2 Recommendations to JICA

4.2.2.1 Nursing Care Component

The malaria control effort in Tanzania is significantly affected by GFATM, both

technically and financially. Some NMCP officials and laboratory technicians recognize the value of AO diagnosis and have a desire to expand it. However, this desire is not expected to be fulfilled unless the method is better focused by GFATM. Since GFATM formulates policy in accordance with WHO recommendations, it is recommended to the Japanese officials lobbying WHO to recognize the value and necessity of the AO method of malaria diagnosis.

4.3 Lessons Learned

4.3.1 Setting and Management of Project Indicators with Consideration of Monitoring

and Evaluation

The indicators for the outputs and objectives of the project identified on the PDM were not practically suitable for monitoring and evaluation. In addition, baselines were not collected for them and targets were not set properly. In consequence, the project was neither managed nor monitored quantitatively, and this ex-post evaluation also had limitations in this regard.

This situation called attention to the necessity of selecting monitorable indicators, collecting baselines, and setting target values. Additionally, it is desirable to continue data collection and management during the project-implementation period by incorporating these actions into ordinary project activities. In a case where counterpart agencies already have or are developing capacity for monitoring and evaluation, the involvement of the relevant departments of those counterpart agencies in the design, management, and evaluation phases of the project would likely improve the appropriateness of the indicators and the quality of monitoring and evaluation.

31

4.3.2 Effectiveness of the Hierarchical System and Holistic Package of Training in Light of the Partner Country’s Administrative Structure.

The Nursing Care Component of this project can be rated high in terms of effectiveness and sustainability. This component was successful in encouraging initiative on the Tanzanian side, as it adopted a hierarchical training system called a cascade approach that was well suited to Tanzania’s administrative structure in the health sector.

In order to make this system run and maintain session quality during implementation, the project consolidated training materials and protocols: the training package containing a comprehensive curriculum including monitoring activities as well as a standard budget model for reference in planning budget at a district level; “low-tech” tools among the training materials; involvement of local human resources with the aim of developing user-friendly materials; and the training materials being adopted as official MOHSW documents. These various efforts are also considered to have contributed to the achievement of the project and the sustainability of its effects.

The efforts and ideas discussed above, as well as the whole system and package including the processes of development and dissemination of materials, can be seen as a case of best practice widely applicable beyond a single issue such as training on nursing care of malaria.

4.3.3 Consideration of Market and Maintainability

Establishment of systems for the repair of AO microscopes and procurement of spare

parts and consumables were major challenges for the project. Although in general and in principle, basic healthcare services such as malaria control should be delivered with public funds, the project had intended to facilitate the AO method before the GoT substantially committed to adopt the method. The result shows that the procurement of equipment, spare parts, and consumables would possibly be an issue in introducing new technologies and products under these conditions of a lack in local markets.

During the project-design phase, the Japanese side had the expectation that it would be possible to secure a stock of relevant products by means of steady efforts in negotiating with MSD, and also that it would be feasible to establish a system for microscope maintenance as the project helped the improvement of the organizational and individual capacity of the refinement centers. This expectation regarding procurement would only have been realized if the project succeeded in making MSD and private dealers give priority to securing stocks of AO spare parts and consumables, which is not a natural business operation under market principles. The other expectation regarding the

32

establishment of a maintenance system was also not so easy to realize—it seems to have required institutionalization of training in repairing the technology and enough volume of demand in society to cause this technology’s autonomic spread and sustainability. It had been also expected that the AO method would gain policy backup. None of these expectations have been realized so far. It can be pointed out that the project started and was in fact designed, in this difficult situation.

In any cooperation project intending to introduce a new product or service under a condition where sufficient commitment and support from the partner government are not yet promised, it must be carefully determined if there is a market that can sustain the distribution of relevant products and services and if the targeted society has sufficient capacity for their sustainable maintenance.