Synthesis Note No. 8 | September 2021
Technical Assistance Completion Report Validation: Observations and Emerging Lessons
EvaluationIndependent
Raising development impact through evaluation
Synthesis Paper No. 8 September 2021
Technical Assistance Completion Report Validation: Observations and Emerging Lessons This document is being disclosed to the public in accordance with ADB’s Access to Information Policy.
ABBREVIATIONS
ADB ASEAN DMC DMF IED PARD PRC PSM SDCC TA TASU TCR TCRV
– – – – – – – – – – – – –
Asian Development Bank Association of Southeast Asian Nations developing member country design and monitoring framework Independent Evaluation Department Pacific Department People’s Republic of China public sector management Sustainable Development and Climate Change Department technical assistance technical assistance supervising unit technical assistance completion report technical assistance completion report validation
NOTE
In this report, “$” refers to United States dollars.
Director General Marvin Taylor-Dormond, Independent Evaluation Department (IED) Director Nathan Subramaniam, Sector and Projects Division, IED
Team Leader Payton Deeks, Evaluation Specialist, IED Team Members Charity Gay Ramos-Galacgac, Evaluation Officer, IED Bryan Noel Lazaro, Evaluation Assistant, IED
The guidelines formally adopted by the Independent Evaluation Department on avoiding conflict of interest in its independent evaluations were observed in preparing this report. To the knowledge of the management of the Independent Evaluation Department, there were no conflicts of interest of the persons preparing, reviewing, or approving this report.
In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgments as to the legal or other status of any territory or area.
ACKNOWLEDGEMENTS This synthesis paper was prepared under the guidance of Director General Marvin Taylor-Dormond and Director Nathan Subramaniam, Sector and Project Division, Independent Evaluation Department (IED). The team was led by Payton Deeks, Evaluation Specialist. Team members were Charity Gay Ramos-Galacgac, Evaluation Officer, and Bryan Noel Lazaro, Evaluation Assistant.
The team is also grateful to Benjamin Graham, Advisor, IED for providing valuable comments. IED retains fully responsibility for this paper.
I. INTRODUCTION 1. In 2018, the Board of Directors of the Asian Development Bank (ADB) directed the Independent Evaluation Department (IED) to establish a technical assistance completion report validation (TCRV) system. The objective of the TCRV system was to systematically assess performance and capture insights on ADB’s technical assistance (TA) operations. It was expected to facilitate better learning and dissemination of knowledge from TA products and improve quality-at-entry of TA projects. As ADB developing member countries (DMCs) transition to middle and upper-middle income countries, their needs have evolved from simple capacity building for project preparation to more technical and/or specialized knowledge products aimed at regional priorities, policy and/or regulatory reforms. TAs have been one of ADB’s important instruments to address these evolving needs. From 2002 to 2018, ADB approved over 4,000 TA projects amounting to about $4.2 billion. However, during this period, ADB did not have a system to independently validate TA completion reports (TCRs). 2. IED adopted a phased approach to design and establish a new TCRV system starting with the establishment of an inter-departmental technical working group. The key tasks of the working group involved (i) a desk review of sample TCRs to assess their evaluability and quality, (ii) a comparative review of the TA evaluation processes ang general TA administration by seven comparator international financial institutions, (iii) drafting of new guidelines, and (iv) pilot testing of the new TCR validation guidelines in three departments.1 Following this, IED presented the TCRV process to the Development Effectiveness Committee of the Board of Directors in March 2019. In December 2019, IED issued the TCRV Guidelines, which was harmonized with the Project Administration Instructions 6.08 for TCR preparations. IED also held TCRV launch events to familiarize staff with the new guidelines. 3. The new TCRV system was mainstreamed in ADB in January 2020. The technical working group (composed of representatives from IED; Procurement, Portfolio and Financial Management Department; and Strategy, Policy and Partnerships Department) developed an approach to capture a purposeful sample of TCRs to be validated in 2020. The sample sought representation by sector, TA size, TA type, and funding type. A total of 62 TCRs were selected for validation in 2020.
4. This synthesis paper provides observations and emerging lessons from TCR validations. This paper does not intend to report on the performance ratings of all TCRs in ADB, but instead provides emerging lessons and illustrative examples based on the first year of mainstreaming the TCRV system. The portfolio distribution of the 62 TCRs are in Appendix 1. The complete list of the TCRs is in Appendix 2.
II. KEY FINDINGS FROM TECHNICAL ASSISTANCE VALIDATIONS 5. IED assesses TCRs based on three core criteria of relevance, effectiveness, and efficiency. The sub-ratings on each of these criteria are assigned a numerical value,2 and the overall TCRV rating is determined by a weighted average of the three criteria.3 IED also provides a qualitative assessment of the sustainability of the benefits from TA operations, but this is not included in the compounded overall rating. Lastly, IED assesses the quality of the TCRs prepared by the ADB departments and/or offices in terms of their coherence, quality of data, and quality of lessons learned. As a knowledge and learning product, the TCR’s lessons component is very important and is weighted 50% of the total quality rating.
1 Pilot departments were the Private Sector Operations Department, South Asia Department, and Sustainable
Development and Climate Change Department. 2 For example, highly relevant = 3, relevant = 2, less than relevant = 1, and irrelevant = 0. 3 In computing the compounded overall rating, relevance, and effectiveness each weighs 35% and efficiency weighs
30%. The calculated overall performance rating categories are: highly successful (overall weighted average greater than 2.30), successful (overall weighted average greater than or equal to 1.65 and less than or equal to 2.30), less than successful (overall weighted average greater than or equal to 0.75 and less than 1.65), unsuccessful (overall weighted average is less than 0.75).
2 | IED Synthesis Paper No. 8
6. The number of TCRs validated by IED in 2020 was too small to be statistically representative of the TCR population. However, the majority of validated TCRs were assessed as successful. TA projects with clear objectives, realistic focus, and well-defined indicators achieved good results. The following sectors had two or more TCRs included in the sample and had demonstrated good results: agriculture and natural resources and rural development, education, multisector, finance, transportation, and industry. A number of TA supervising units (TASUs) had more than one TCR in the sample and demonstrated good performance, namely Economic Research and Regional Cooperation Department, Pacific Department, Southeast Asia Department, and Sustainable Development and Climate Change Department (SDCC). In terms of TCR quality, more than one-third had a less than satisfactory rating, which is perhaps to be expected in the first year of mainstreaming, as TASUs are just starting to become familiar with the guidelines and receive feedback on the TCRs from IED. This low rating of TCR quality can be further explained by the weight that is placed on the quality of lessons, which often dictates the overall quality rating of a TCR.
A. Relevance 7. Most of validated TA projects are aligned well with country plans and ADB strategies, with clear rationale, but many suffered from flaws in project design. The relevance of TA projects is assessed based on the three dimensions of strategic alignment, appropriateness of design and results chain, and rationale and appropriateness of TA modality. Alignment of the TA projects to the DMC as well as ADB priorities was often well justified. TA type was usually appropriately selected. In terms of rationale, there was usually a good justification for TA projects, though it was not always succinctly presented. The design and appropriateness of results chains was the weakest dimension of the sampled TCRs. Several underlying factors contributed to such design weaknesses: the quality of risk identification; depth of consultation, and collection and preparation of baseline data. Well-designed TA projects tightly and directly linked outputs to the outcomes with clear indicators, and appropriate scope and timeframe. However, sometimes outcomes were too ambitious compared to a TA project’s limited scope, funding, or duration. For example, a TA project in the People’s Republic of China (PRC) aimed for the successful implementation of financial sector and regulatory reforms, which required many policy and regulatory changes.4 Other examples are a TA project in the finance sector in Mongolia that included a mixture of diverse activities in its design;5 and a TA project in Armenia which intended to improve industrial and innovation policies that involved overly complex and highly diverse activities.6 There are few TA projects whose original implementation period was designed at less than a year, which was not long enough to deliver the expected outputs.7 Most of the TA projects had the design and monitoring frameworks (DMFs) updated during implementation to address design deficiencies, but some TA projects failed to do this. For example, in a TA project supporting energy development in India, the DMF was not revised to reflect the changes in scope arising from change in the risk exposure of the implementing agency, and/or to tighten the originally crafted indicators which were found to have deficiencies.8 Only three TA projects in the 2020 sample were assessed as highly relevant: (i) Strengthening Monitoring and Enforcement in the Meghna River for Dhaka's Sustainable Water Supply, which involved local community watchdog groups
4 IED. 2020. Technical Assistance Completion Report Validation Report: Study on Local Financial Market
Development and Supervisory Framework in the People’s Republic of China. Manila: ADB. 5 ADB. 2020. Technical Assistance Completion Report: Supporting Financial Sector Development and Stability in
Mongolia. Manila. 6 IED. 2020. Technical Assistance Completion Report Validation Report: Export- and Innovation-Led Industrial
Development in Armenia. Manila: ADB. 7 For example, for Pakistan’s Education Sector Assessment, the original TA duration was five months; for Timor-
Leste’s Preliminary Assessment for Dili Airport Runway Upgrading Project, the original TA duration was eight months; for the Establishment of the Kazakhstan Knowledge Center on Integrated Water Resources Management, the original duration was nine months; and for Kazakhstan’s Fostering the Development of Renewable Energy, the original implementation period was for 11 months.
8 IED. 2021. Technical Assistance Completion Report Validation Report: Integrated Renewable Energy Development in India. Manila: ADB.
Technical Assistance Completion Report Validation: Observations and Emerging Lessons | 3
in a participatory pollution monitoring and reporting system, and piloted innovative cleaner production principles in industrial clusters in selected areas in Bangladesh;9 (ii) Action on Climate Change in South Asia, which introduced an automated web-based geographic information system that was used in climate change risks assessments;10 and (iii) a regional TA project which supported the establishment of regional financial market infrastructure linkages in the Association of Southeast Asian Nations (ASEAN)+3.11 B. Effectiveness
8. TA projects delivered a high number of capacity building activities and produced a number of knowledge products that were valuable to the DMCs, but in the majority of cases the results were not measured effectively. Of the 62 TCRs in the sample, almost 50% were capacity development TA projects, while 34% were policy and advisory in nature. However, a common observation across the TA projects offering capacity building is the lack of rigor in systematically measuring enhanced capacity through metrics such as pre- and post-tests or surveys. The few TA projects that provided measurements, include: (i) Action on Climate Change in South Asia, which explicitly included in its DMF indicators the conduct of pre- and post-training results; and (ii) Strengthening Monitoring and Enforcement in the Meghna River for Dhaka's Sustainable Water Supply, which provided evidence on learning outcomes during the interdepartmental commenting stage. The other TCRs simply indicated the number of training events and participants, without providing measurements for enhanced capacity.
9. For policy advice TAs, an important aspect was the extent of uptake of the TA recommendations as indicator of its contribution to broader policy reforms and knowledge generation. While the DMFs of several TA projects included a clear indicator on the uptake from the recipient DMC governments (e.g., endorsement or adoption of policy recommendations),12 the TCRs oftentimes lacked evidence to substantiate the uptake of recommendations or knowledge products. For example, in one TA project, which intended to provide policy recommendations on public service procurement, delivery and financing in the PRC, there was a lack of evidence of how the TA outputs and/or recommendations were adopted and how they improved the policy and institutional frameworks.13 Similarly, in the case of a regional TA project supporting the accountability mechanism framework for financial intermediaries, there was a lack of evidence to support the increase in the willingness of financial intermediaries to use the acquired knowledge on safeguard compliance supervision and accountability.14 One example of clear uptake was elaborated during interdepartmental commenting stage and not in the TCR: the gross ecosystem product guidelines on technical accounting produced in a TA project in the PRC are now being used by the central and local governments to measure their ecological progress for sustainable development.15 Another
9 ADB. 2020. Technical Assistance Completion Report: Strengthening Monitoring and Enforcement in the Meghna
River for Dhaka's Sustainable Water Supply in Bangladesh. Manila. 10 ADB. 2020. Technical Assistance Completion Report: Action on Climate Change in South Asia. Manila. 11 ADB. 2020. Technical Assistance Completion Report: Creating a Regional Settlement Intermediary in ASEAN+3:
Cross-Border Settlement Infrastructure Forum. Manila. 12 Examples of TA projects with clear indicator of uptake in the DMF were: Export- and Innovation-Led Industrial
Development (Armenia); Education Sector Assessment (Pakistan); Creating Enabling Environments for Gradual Rural–Urban Migration and Urban Settlements Development in Chuxiong Yi Autonomous Prefecture (PRC); Comprehensive Hub-Oriented Transportation Strategy for Urumqi Metropolitan Area (PRC); Improving Energy Efficiency, Emission Control, and Compliance Management of the Manufacturing Industry (PRC); Study on Local Financial Market Development and Supervisory Framework (PRC); and a regional TA on Establishment of the Pacific Region Infrastructure Facility Coordination Office.
13 IED. 2020. Technical Assistance Completion Report Validation Report: Improving Efficiency in Public Service Procurement, Delivery and Financing in the People’s Republic of China. Manila: ADB.
14 IED. 2020. Technical Assistance Completion Report Validation Report: Developing an Accountability Mechanism Framework for Financial Intermediaries. Manila: ADB.
15 IED. 2020. Technical Assistance Completion Report Validation Report: Developing Gross Ecosystem Product Accounting for Eco-compensation in the People’s Republic of China. Manila: ADB.
4 | IED Synthesis Paper No. 8
example was in Bangladesh, where the government adopted the Strategic Masterplan produced under the TA project, with work commencing on the projects recommended under the masterplan.16 C. Efficiency
10. In terms of efficiency, process issues affected the level of efficiency in delivering TA. The TCRV reports often highlighted process efficiency concerns. It was common for TA projects to request changes in scope, cost, implementation arrangement and/or extension in the completion period due to various implementation challenges. These challenges commonly arose from (i) TA design covering a combination of diverse activities, which could have been managed under separate TA projects; (ii) coordination challenges from too many players in a TA project (e.g., implementing agencies and counterparts for different components); (iii) consultant recruitment challenges, such as lack of enough qualified individuals, or poorly written Terms of Reference ; or (iv) underestimating implementation challenges in innovative or complex areas, such as multilateral collaboration, information and communication technology development, and climate financing. One example of a TA project that required changes during implementation was the support for transport and trade facilitation in the Greater Mekong Subregion whose midterm review recommended the narrowing of the TA scope commensurate with its resources and timeframe; the TA eventually had to request six extensions to be able to effectively respond to the scale and complexity of a cluster TA project.17 Another TA project supporting scaled-up climate finance had ambitious objectives that only became achievable after the provision of additional financing and extension of the completion period.18 Only one TA project, the Conservation of Forest Genetic Resources in Mongolia, was assessed to be highly efficient for delivering outputs on time and within budget, utilizing just over 84% of its funding, and generating cost savings through a training-of-trainers approach. D. Sustainability
11. Although sustainability is not a core criterion for TCRVs, it is still assessed at validation. Six TA projects in the sample had already secured follow-on funding or related projects and were assessed as highly likely sustainable.19 Other likely sustainable TA projects had their outputs—e.g., policy recommendations—adopted by governments. Those less likely or unlikely sustainable were often adversely affected by changing government priorities, or a lack of budgetary or human resources commitment. In one instance, while the knowledge products were produced, no outreach and capacity building event for disseminating this knowledge was conducted.20
16 IED. 2021. Technical Assistance Completion Report Validation Report: Strategic Master Plan for Chittagong Port in
Bangladesh. Manila: ADB. 17 IED. 2020. Technical Assistance Completion Report Validation Report: Support for Implementing the Action Plan
for Transport and Trade Facilitation in the Greater Mekong Subregion. Manila: ADB. 18 IED. 2020. Technical Assistance Completion Report Validation Report: Enhancing Readiness of ADB Developing
Member Countries for Scaled Up Climate Finance. Manila: ADB. 19 The TA projects that were validated to be highly likely sustainable were: Strengthening Monitoring and
Enforcement in the Meghna River for Dhaka's Sustainable Water Supply (Bangladesh); Developing Gross Ecosystem Product Accounting for Eco-Compensation (PRC); Preliminary Assessment for Dili Airport Runway Project Upgrading Project (Timor-Leste); Improving Operational Performance of the Water Supply Sector (Viet Nam); and two regional TA projects entitled Enhancing Road Safety for Central Asia Regional Economic Cooperation Member Countries, and Enhancing Association of Southeast Asian Nations Capital Market Integration.
20 IED. 2020. Technical Assistance Completion Report Validation Report: Improving Energy Efficiency, Emission Control, and Compliance Management of the Manufacturing Industry in the People’s Republic of China. Manila: ADB.
Technical Assistance Completion Report Validation: Observations and Emerging Lessons | 5
III. KEY LESSONS FROM TECHNICAL ASSISTANCE VALIDATIONS 12. As in the sovereign and nonsovereign operations, IED distills lessons from completed TA operations for institutional learning. The TCR validation format prompts lessons in eight categories for which TASUs are expected to identify learnings for wider application. 21 This section will summarize the emerging lessons based on the key observations from the validations of the 62 TCRs.
A. Design and/or Planning 13. A clear results chain and an appropriate scale in designing TA projects are crucial for successful implementation. TA projects that had deficiencies in their results chains, indicators, and targets had to adjust their design during implementation to facilitate a successful delivery of results. Changes in scope, cost, implementation arrangement, and/or completion period present opportunities to address deficiencies during the design stage. A few TA projects had their DMFs revised during implementation, making results chain clearer and easier to assess, for example: (i) the Information and Communication Technology for Development Initiative Facility in Asia and the Pacific, and (ii) the Establishment of a Pacific Region Infrastructure Facility Coordination Office. Similarly, the DMFs for TA projects which supported primary education development in Bangladesh22 and improvement of the water supply sector in Viet Nam23 were updated to capture the additional results concomitant to the additional funding. Careful selection and timely updates of the DMF indicators are crucial as this helps set a clear path ahead for ADB, the implementing/executing agency, and the stakeholders in successfully carrying out the TA deliverables. It also facilitates proper monitoring of TA progress during implementation and reporting of project performance at TA completion. Ensuring that there is documentation and approval of these changes is also important. 14. Baseline data or scoping missions and thorough risk identification are vital for appropriate indicator target setting at design. The conduct of adequate due diligence during the TA preparatory stage led to a more realistic project scope and timeframe. The TA projects that neglected to consider existing ADB work in a country or sector often encountered implementation challenges, as did those which failed to undertake proper due diligence. For example, in a TA project supporting agricultural programs in the Greater Mekong Subregion, the TCR noted that a thorough understanding of the quality of baseline data in the different pilot sites during the TA planning stage could have avoided additional work towards the end of TA project implementation.24 In the case of a TA project that aimed to improve energy efficiency in the manufacturing industry in the PRC, the TCR indicated that the lack of baseline information made the consultants tasked with developing a synthesis paper on policy lessons and recommendations ask the wrong questions in their interviews.25 In another energy project in Kazakhstan, the TCR recognized the need to allocate more time and resources to understanding the data during the TA preparatory phase (e.g., data availability, access, etc.) as that would have better guided the TA activities,26 while in the case of a TA project on enhancing poor women’s benefits from energy access, the TCR emphasized the
21 These are in the areas of (1) design and/or planning; (2) implementation and/or delivery, (3) management, (4)
knowledge building, (5) stakeholder participation, (6) partnership and cofinancing, (7) replication and scaling up, (8) post-TA financial resources. There is also an additional “other” lesson category to capture other lessons not covered by the eight main categories.
22 IED. 2021. Technical Assistance Completion Report Validation Report: Support for the Third Primary Education Development Project in Bangladesh. Manila: ADB.
23 IED. 2020. Technical Assistance Completion Report Validation Report: Improving Operational Performance of the Water Supply Sector in Viet Nam. Manila: ADB.
24 ADB. 2020. Technical Assistance Completion Report: Implementing the Greater Mekong Subregion Core Agriculture Support Program (Phase 2). Manila
25 ADB. 2020. Technical Assistance Completion Report: Improving Energy Efficiency, Emission Control, and Compliance Management of the Manufacturing Industry in the People’s Republic of China. Manila.
26 ADB. 2020. Technical Assistance Completion Report: Fostering the Development of Renewable Energy in Kazakhstan. Manila.
6 | IED Synthesis Paper No. 8
importance of conducting adequate due diligence on the feasibility of the TA scope during planning stage prior to inclusion of some outputs in the TA design.27 15. Including measures for quality of training in the DMF improves the evidence of strengthened capacity. While capacity building, particularly training, is a major focus of ADB’s TA work, it is not systematically measured across the portfolio. As noted above, it is common for TCRs to report attendance to trainings; however, they often do not include demonstrations of proficiency or other metrics. Including a qualitative measurement of enhanced capacity (e.g., through pre- and post-tests, surveys) in the DMF indicators would establish a clear and direct linkage of the trainings to outcomes. This would help guide further actions on making capacity building efforts more effective.
B. Implementation and/or Delivery 16. A well-focused lead agency with sufficient influence and authority and a clear implementation structure are crucial for TA success, especially in TA projects that involve many players. In the case of one TA project in Armenia that involved too many implementing agencies and counterparts for different components, a clearer coordination mechanism or implementation structure could have avoided delays.28 In another TA project in the PRC, the executing agency appeared to be not well positioned to engage the stakeholders, resulting in challenges during implementation. 29 In the case of a TA project that aimed to improve water supply in Viet Nam, a central ministry (instead of a subnational office) might have been more appropriate as implementing agency.30 The TA projects which had a focused and well-performing executing agency tended to have successful implementation such as TA projects supporting sustainable water supply in Bangladesh31 and public-private partnership in the PRC.32 As ADB continues its partnership with DMCs and expands its engagement in emerging areas and/or at subnational level, it faces the challenge of managing multiple stakeholders and navigating a complex process of coordination. Thus, an executing and/or implementing agency with appropriate mandate, authority, and a clear coordination mechanism could facilitate effective implementation.
C. Management of Staff and Consultant Resources
17. Careful planning and management of staffing and consultancy contributes to implementation efficiency and project success. There was no universal formula, and a variety of consultant team compositions and/or management strategies worked depending on the situation. A strong team leader and team continuity were among the factors in successfully carrying out the TA activities and deliverables. For example, in the case of the TA projects on Vocational Education-Enterprise Collaboration for Student Employment-Based Poverty Reduction in Gansu, and Poverty Reduction in Liupanshan, Ningxia Hui Autonomous Region, both in the PRC, strong team leaders were identified to be crucial in managing the teams and shepherding the TA activities. On the other hand, team continuity was highlighted in the TA on Action on Climate Change in South Asia, which identified a core group of consultants who kept the TA project moving through ADB staff turnover, and in the TA on Trade and Transport Facilitation in the Pacific wherein the utilization of the same consultant team all throughout the TA implementation period
27 ADB. 2020. Technical Assistance Completion Report: Enabling Poor Women's Benefits from Enhanced Access to
Energy in Hatiya Island in Bangladesh. Manila. 28 IED. 2020. Technical Assistance Completion Report Validation Report: Export- and Innovation-Led Industrial
Development in Armenia. Manila: ADB. 29 IED. 2020. Technical Assistance Completion Report Validation Report: Comprehensive Hub-oriented Transportation
Strategy for Urumqi Metropolitan Area in the People’s Republic of China. Manila: ADB. 30 IED. 2020. Technical Assistance Completion Report Validation Report: Improving Operational Performance of the
Water Supply Sector in Viet Nam. Manila: ADB. 31 ADB. 2020. Technical Assistance Completion Report: Strengthening Monitoring and Enforcement in the Meghna
River for Dhaka’s Sustainable Water Supply in Bangladesh. Manila. 32 ADB. 2020. Technical Assistance Completion Report: Financing Public-Private Partnerships in the People’s Republic
of China. Manila.
Technical Assistance Completion Report Validation: Observations and Emerging Lessons | 7
contributed to knowledge building.33 In other TA projects, identifying a back-up point of contact or arrangement was highlighted as important to ensuring smooth implementation.34 One TA project assessed that hiring a firm, rather than individual consultants would avoid coordination issues and lead to a better outcome,35 while other TA projects noted that using individual consultants allowed them to pinpoint expertise. 36 Some TCRs recognized well-written consultant Terms of Reference (including ones that were specific, but not too specific to limit candidacy) to be important.37 Timing was emphasized in other TA projects; specifically, recruitment of highly specialized or technical consultants at an early stage could avoid extensions later on during implementation. 38 D. Knowledge Building
18. Incorporating and budgeting awareness-raising measures in the TA design paves a pathway for a sustainable and long-term institutional knowledge building. Specifically, utilizing appropriate dissemination pathways, which may combine various delivery channels (workshops, print, TV, social media, moderated online discussions, toolkits)39 and are tailored to local needs,40 is an effective means of engaging local stakeholders for a more sustainable knowledge generation. Additionally, combining regional dissemination events with national and/or subnational activities facilitates a more effective dissemination.41 Budgeting for translation services of outputs, as necessary, could likewise increase uptake or facilitate deepening of knowledge.42 It is important that design allocates adequate time and resources for these knowledge dissemination activities, especially if it covers a broad development area,43 to ensure a targeted outreach.44
33 ADB. 2020. Technical Assistance Completion Report: Trade and Transport Facilitation in the Pacific. Manila. 34 This was noted by these TA projects: Fostering the Development of Renewable Energy (Kazakhstan); Reforming
the System of Personal Income Tax and Social Security Contributions (PRC); and a regional TA on Implementing ICT Tools to Improve Data Collection and Management of National Surveys in Support of the Sustainable Development Goals.
35 ADB. 2020. Technical Assistance Completion Report: Poverty Reduction in Liupanshan, Ningxia Hui Autonomous Region in the People’s Republic of China. Manila.
36 These TA projects were: Supporting Financial Sector Development and Stability (Mongolia); and Financing Public-Private Partnerships (PRC).
37 These TA projects were: Export- and Innovation-Led Industrial Development (Armenia); Vocational Education-Enterprise Collaboration for Student Employment-Based Poverty Reduction in Gansu (PRC); and the regional TA on Trade and Transport Facilitation in the Pacific.
38 IED. 2021. Technical Assistance Completion Report Validation Report: Integrated Renewable Energy Development in India. Manila: ADB.
39 Examples of TA projects that utilized various dissemination channels were: Gender-Responsive Sector and Local Development Policies and Actions (Mongolia); Poverty Reduction in Liupanshan, Ningxia Hui Autonomous Region (PRC); and regional TA projects such as Action on Climate Change in South Asia, and Coordinated Border Management for Results in Central Asia Regional Economic Cooperation.
40 Examples of TCRs that highlighted customizing activities to local needs were Gender-Responsive Sector and Local Development Policies and Actions (Mongolia), and a regional TA on Support for Implementing the Action Plan for Transport and Trade Facilitation in the Greater Mekong Subregion.
41 Examples of TA projects that have done this were: a regional TA on Strengthening Tax Policy and Administration Capacity to Mobilize Domestic Resources, and two TA projects in the PRC, namely the Promotion of Environmentally Sustainable Infrastructure Investment in Asia and the Pacific, and Comprehensive Hub-oriented Transportation Strategy for Urumqi Metropolitan Area.
42 Examples of TA projects that have done this were: Fostering the Development of Renewable Energy (Kazakhstan); Study on Local Financial Market Development, Supervisory Framework (PRC); and Poverty Reduction in Liupanshan, Ningxia Hui Autonomous Region (PRC), and a regional TA on Promoting Gender Equality and Women's Empowerment (Phase 2).
43 IED. 2020. Technical Assistance Completion Report Validation Report: Poverty Reduction in Liupanshan, Ningxia Hui Autonomous Region in the People’s Republic of China. Manila: ADB.
44 IED. 2020. Technical Assistance Completion Report Validation Report: Study on Local Financial Market Development and Supervisory Framework. Manila: ADB.
8 | IED Synthesis Paper No. 8
E. Stakeholder Participation 19. Regular engagement with a wide array of stakeholders is vital to ensuring ownership, broader support, and sustainability of TA initiatives. It is common among the TCRs to identify as best practice the conduct of regular and extensive consultations among key stakeholders (e.g., at inception, mid-term, towards the end of the completion date or at critical milestones of the TA project implementation). These workshops serve as a forum for better understanding the needs and priorities of the stakeholders, planning activities, and getting feedback on reports and other deliverables. Regular engagement could be instrumental in addressing implementation challenges such as input requirements from DMCs not being received on time45 or focal officials not being able to attend an important project event as it coincided with a major holiday in the DMC.46 By engaging key stakeholders regularly, the TA outputs and activities can be tailored to their needs, their ownership of the TA is enhanced, and continuity of the TA progress is ensured. F. Partnerships and Cofinancing
20. Partnerships and cofinancing could bring together much needed expertise and resources for a more effective action in delivering intended goals. Given the broad spectrum of the development area that needs assistance, leveraging strategic partnerships in selected areas where there are growing demands across DMCs can maximize outcomes. For example, ADB joined the Global Partnership Network on Identification. Under this partnership, the participating organizations agreed to support DMCs to establish and utilize legal identification systems for sustainable development in the form of funding and technical assistance. Based on the partnership, ADB has provided a TA project to support the Philippines and Cambodia to introduce national identification systems.47 In the Pacific, ADB’s partnership with the governments of Australia and New Zealand allowed larger and more transformative action to be undertaken contributing to regional dialogues, cross country knowledge sharing, among others. 48 In another TA project, ADB partnered with the 6-Banks group49 and jointly conducted dialogue with the government on international best practices to improve official development assistance performance, leading to a more leveraged action.50 Working with regional coordination bodies (e.g., South Asia Subregional Economic Cooperation, ASEAN+3) could likewise ensure continuity of action and decisions in advancing specific regional development agendas. 21. Application of the One ADB approach from concept stage throughout implementation can enhance the value of a TA project through complementary competencies and perspectives.51 For example, in a TA project for enhancing road safety for Central Asia Regional Cooperation member countries, the Central and West Asia Department effectively collaborated with East Asia Department and SDCC. The East Asia Department played a role in leading communication with stakeholders in Mongolia and the
45 IED. 2020. Technical Assistance Completion Report Validation Report: Fostering the Development of Renewable
Energy in Kazakhstan. Manila: ADB. 46 IED. 2020. Technical Assistance Completion Report Validation Report: Reforming the System of Personal Income
Tax and Social Security Contributions in the People’s Republic of China. Manila: ADB. 47 ADB. 2020. Technical Assistance Completion Report: Information and Communication Technology for
Development Initiative Facility in Asia and the Pacific. Manila. 48 IED. 2021. Technical Assistance Completion Report Validation Report: Establishment of the Pacific Region
Infrastructure Facility Coordination Office. Manila: ADB. 49 These banks were the following: Agence Française de Développement, Japan International Cooperation Agency,
Export-Import Bank of Korea, Eximbank, German development cooperation through KfW, and the World Bank Group.
50 IED. 2020. Technical Assistance Completion Report Validation Report: Improving Portfolio Performance and Aid Management in Viet Nam. Manila: ADB.
51 According to ADB Strategy 2030, One ADB refers to “bringing together knowledge and expertise across the organization to effectively implement Strategy 2030.” ADB. 2018. Strategy 2030: Achieving a Prosperous, Inclusive, Resilient, and Sustainable Asia and the Pacific. Manila.
Technical Assistance Completion Report Validation: Observations and Emerging Lessons | 9
PRC, while SDCC was instrumental in drafting and reviewing knowledge products.52 In a TA project aimed at promoting financial strengthening and modernization in Kazakhstan while preparing a nonsovereign loan, the TA team noted they should have obtained regular feedback from the Private Sector Operations Department during implementation to minimize unexpected developments in the preparation of the ensuing loan. No loan was eventually made.53 Similarly, the TCR for a TA project that supported ASEAN capital market integration emphasized the need for more coordination among ADB units to maximize synergies and reduce overlaps.54 Some TA projects (such as the South Asia Subregional Economic Cooperation Efficiency Initiative) received strong support from the resident missions, who also facilitated the engagement of national consultants in India and Nepal. Through these collaboration across the organization, knowledge, expertise, and resources are maximized to deliver more responsive TA results. G. Replication and Scaling-up 22. Pilot studies play an important role in assessing risks, scalability, and feasibility of operations. For example, the pilot studies in the TA that supported sustainable water supply in Bangladesh proved useful in determining the feasibility of a win-win scenario for the environment as well as the businesses.55 In the case of the TA on Action on Climate Change in South Asia, the successful demonstration of pilots provided a strong foundation for scaling-up efforts to mainstream climate adaptation and/or mitigation into national policies and programs. The use of pilots also proved instrumental in assessing: the readiness of the local supply chain and other supporting infrastructure in the TA that supported integrated renewable development in India;56 the design of lending operations in a TA project that promoted gender equality;57 and the replicability or scalability of activities in the case of a regional TA project supporting the national survey on the sustainable development goals.58 However, the pilot alone is not enough; supportive national institutional leadership and streamlined coordination are necessary to sustain outcomes from pilots. H. Post-Technical Assistance Financial Resources 23. Budgetary commitment for the subsequent activities ensures sustained benefits from the TA. In several TA projects, findings from studies provided opportunities for further investment by governments or by ADB to take forward the TA initiatives towards achieving broader development results. As such, the government’s commitment to provide budgetary allocation for the subsequent activities is crucial. In cases where future investments are consistent and linked to the country operations business plans, follow-up actions by ADB country offices could ensure continuity of efforts.59 Also, pooling resources and
52 ADB. 2020. Technical Assistance Completion Report: Enhancing Road Safety for Central Asia Regional Economic
Cooperation Member Countries. Manila. 53 ADB. 2020. Technical Assistance Completion Report: Preparing the Kazpost Financial Strengthening and
Modernization Project in Kazakhstan. Manila. 54 ADB. 2020. Technical Assistance Completion Report: Enhancing Association of Southeast Asian Nations Capital
Market Integration. Manila. 55 IED. 2020. Technical Assistance Completion Report Validation Report: Strengthening Monitoring and Enforcement
in the Meghna River for Dhaka's Sustainable Water Supply in Bangladesh. Manila: ADB. 56 IED. 2021. Technical Assistance Completion Report Validation Report: Integrated Renewable Energy Development
Project in India. Manila: ADB. 57 ADB. 2020. Technical Assistance Completion Report: Promoting Gender Equality and Women's Empowerment
(Phase 2). Manila. 58 ADB. 2020. Technical Assistance Completion Report: Implementing ICT Tools to Improve Data Collection and
Management of National Surveys in Support of the Sustainable Development Goals. Manila. 59 IED. 2020. Technical Assistance Completion Report Validation Report: Poverty Reduction in Liupanshan, Ningxia
Hui Autonomous Region in the People’s Republic of China. Manila: ADB.
10 | IED Synthesis Paper No. 8
activities with the other organizations could ensure that initiatives are strategically prioritized and synergized.60
IV. MOVING FORWARD 24. TCRVs are already contributing useful lessons for the design of future TA projects. ADB TASUs have reported using the TCRV feedback as inputs into upcoming TA project preparation. For example, during the interdepartmental review, departments managing several TA projects have indicated they would use findings from the TCRV as inputs into future TA work.61 In addition, the departments have expressed interest in improving TCR quality to contribute to a more extensive knowledge generation in the bank. 25. In July 2021, IED has started validating 100% of circulated and eligible TCRs. IED will further synthesize lessons to contribute to improved accountability, learning, and knowledge generation.
60 IED. 2020. Technical Assistance Completion Report Validation Report: Strengthening Tax Policy and Administration
Capacity to Mobilize Domestic Resources. Manila: ADB. 61 Examples of these are the TASUs of the following TA projects: Promoting Green Local Currency-Denominated
Bonds for Infrastructure Development in ASEAN+3, Enhancing Road Safety for Central Asia Regional Economic Cooperation Member Countries, and Support for Implementing the Action Plan for Transport and Trade Facilitation in the Greater Mekong Subregion.
Appendix 1: Portfolio Distribution of the 62 Validated Technical Assistance Completion Reports | 11
APPENDIX 1: PORTFOLIO DISTRIBUTION OF THE 62 VALIDATED TECHNICAL ASSISTANCE COMPLETION REPORTS
1. This section provides a snapshot of the 62 validated technical assistance completion reports (TCRs) by sector (Figures 1 and 2) and by department (Figures 3 and 4).
2. In terms of technical assistance (TA) count by sector (Figure 1), the public sector management (PSM) sector has the highest number of sampled TCRs with a share of 21%. Examples of TA projects in the PSM sector included those that supported the developing member countries (DMCs) in improving portfolio management and public sector administration-related matters (e.g., taxation, social security, compensation system, procurement). The energy sector comes next with a 15% share; followed by the agriculture and natural resources and rural development, finance, and transportation sectors having 13% share each. The transportation and the energy sectors, which together make up about half of ADB’s sovereign portfolio in 2020, had a combined share of 28% of the sample.1 The share of the other sectors in the sample are: industry (8%); education, multisector, and water and other urban infrastructure and services sectors with 5% each; and finally, information and communication technology, and health sectors with 2% each. By TA amount2 (Figure 2), the multisector and energy sectors have the largest share in the sample with 21% and 20% share, respectively. This is followed by the agriculture and natural resources and rural development sector (16%), then by the industry sector (11%) and transport sector (10%). While the PSM sector is the highest in terms of TA count, it accounts only for 8% in terms of TA amount, which suggests that TA projects in this sector have relatively small dollar amounts. Each of the rest of the sectors has a share of less than 10% of the sampled TA projects in terms of amount.
Figure 1. Technical Assistance Completion Report Validation Sample by Sector and by Technical Assistance Count
ANR = agriculture, natural resources and rural development, EDU = education, ENE = energy, FIN = finance, HEA = health, ICT = information and communication technology, IND = industry, MUL = multisector, PSM = public sector management, TRA = transport, WUS = water and other urban infrastructure and services. Note: Total percentage share may not add up due to rounding. Source: Asian Development Bank (Independent Evaluation Department).
1 ADB. 2021. Annual Portfolio Performance Report 2020. Manila. 2 TA amount refers to the latest approved amount of the TA as reflected in the TCRs.
ANR,13%(8 TA projects)
EDU, 5%(3 TA projects)
ENE, 15%(9 TA projects)
FIN, 13%(8 TA projects)
HEA, 2%(1 TA
project)ICT, 2%
(1 TA project)IND, 8%
(5 TA projects)
MUL, 5%(3 TA projects)
PSM, 21%(13 TA projects)
TRA, 13%(8 TA projects)
WUS, 5%(3 TA projects)
12 | IED Synthesis Paper No. 8
Figure 2. Technical Assistance Completion Report Validation Sample by Sector and by Technical
Assistance Amount
ANR = agriculture, natural resources and rural development, EDU = education, ENE = energy, FIN = finance, HEA = health, ICT = information and communication technology, IND = industry, MUL = multisector, PSM = public sector management, TRA = transport, WUS = water and other urban infrastructure and services. Source: Asian Development Bank (Independent Evaluation Department).
3. In terms of count by TA supervising unit, the East Asia Department comprised the largest share of TCRs in the sample (31%), followed by the South Asia Department (21%), the Southeast Asia Department with 16%, the Central and West Asia Department (11%), and the Pacific Department (PARD) with 3%. The small number of sampled TCRs from PARD may reflect the relatively small share of PARD in the overall ADB portfolio. In line with the emphasis of TA projects to generate knowledge, ten TCRs from the knowledge management and sustainable development group, i.e., Economic Research and Regional Cooperation Department, and Sustainable Development and Climate Change Department were included in the sample. In terms of TA amount, among the regional departments, Southeast Asia Department takes the largest share of the sampled TCRs at 44%, followed by the South Asia Department and PARD with 15% share each; East Asia Department with 11% share; and Central and West Asia Department with 4% share. While only two TCRs from PARD were sampled, these were regional in nature and covered a number of Pacific DMCs; thus, PARD’s relatively high share in terms of total TA amount.
ANR16%
EDU3%
ENE20%
FIN5%HEA
1%ICT2%
IND11%
MUL,21%
PSM8%
TRA10%
WUS3%
Appendix 1: Portfolio Distribution of the 62 Validated Technical Assistance Completion Reports | 13
Figure 3. Technical Assistance Completion Report Validation Sample by Regional Department or Technical Assistance Supervising Unit and by Technical Assistance Count
CWRD = Central and West Asia Department, EARD = East Asia Department, PARD = Pacific Department, SARD = South Asia Department, SERD = Southeast Asia Department. Note: Others in this figure comprise of the Economics Research and Regional Cooperation Department, Office of the Compliance Review Panel, and Sustainable Development and Climate Change Department. Source: Asian Development Bank (Independent Evaluation Department).
Figure 4. Technical Assistance Completion Report Validation Sample by Regional Department or
Technical Assistance Supervising Unit and by Technical Assistance Amount
CWRD = Central and West Asia Department, EARD = East Asia Department, PARD = Pacific Department, SARD = South Asia Department, SERD = Southeast Asia Department. Note: Others in this figure comprise of the Economics Research and Regional Cooperation Department, Office of the Compliance Review Panel, and Sustainable Development and Climate Change Department. Source: Asian Development Bank (Independent Evaluation Department).
CWRD, 11%(7 TA projects)
EARD, 31%(19 TA projects)
PARD,3%(2 TA projects)
SERD,16%(10 TA projects)
SARD, 21%(13 TA projects)
Others,18%(11 TA projects)
CWRD, 4%
EARD, 11%
PARD, 15%
SERD, 44%
SARD, 15%
Others, 11%
14 | IED Synthesis Paper No. 8
APPENDIX 2: LIST OF TECHNICAL ASSISTANCE COMPLETION REPORTS VALIDATED IN THE 2020 SAMPLE
TA No. Location Title
TCR Circulation
Date 4874 KAZ Preparing the Kazpost Financial Strengthening and Modernization
Project 17 Jun 2020
7099 IND Integrated Renewable Energy Development 2 Sep 2020
7979 BAN Strategic Master Plan for Chittagong Port 10 Jun 2020
8163 REG Implementing the Greater Mekong Subregion Core Agriculture Support Program (Phase 2)
17 Jun 2020
8271 SRI Southern Highways Investment Program 2 Jun 2020
8345 REG Establishment of the Pacific Region Infrastructure Facility Coordination Office
3 Sep 2020
8572 REG Action on Climate Change in South Asia 19 Jun 2020
8584 REG Coordinated Border Management for Results in Central Asia Regional Economic Cooperation
2 Jun 2020
8591 REG Macroeconomic Modeling for Improved Economic Assessment 7 Jul 2020
8606 REG Enhancing Readiness of ADB Developing Member Countries for Scaled Up Climate Finance
5 Jun 2020
8649 MON Preparing the Energy Efficiency and Urban Environment Improvement Project
3 Apr 2020
8674 REG Trade and Transport Facilitation in the Pacific 30 Jun 2020
8743 PRC Study on Local Financial Market Development and Supervisory Framework
11 May 2020
8797 REG Promoting Gender Equality and Women's Empowerment (Phase 2) 25 May 2020
8804 REG Enhancing Road Safety for Central Asia Regional Economic Cooperation Member Countries
17 Jun 2020
8806 VIE Improving Portfolio Performance and Aid Management 29 May 2020
8813 REG Information and Communication Technology for Development Initiative Facility in Asia and the Pacific
26 Mar 2020
8840 BAN Enhancing the Institutional Capacity of the Implementation Monitoring and Evaluation Division, Ministry of Planning
5 Oct 2020
8869 PRC Financing Public-Private Partnerships 9 Jul 2020
8895 PRC Promoting the Decisive Role of the Market in Resource Allocation 22 Sep 2020
8905 REG Enhancing Association of Southeast Asian Nations Capital Market Integration
11 Sep 2020
8916 PRC Reforms in the Public Sector Compensation System 22 Jun 2020
Appendix 2: List of Technical Assistance Completion Reports Validated in the 2020 Sample | 15
TA No. Location Title
TCR Circulation
Date 8927 BAN Enabling Poor Women's Benefits from Enhanced Access to Energy in
Hatiya Island 30 Jun 2020
9018 REG Implementing Information and Communication Technology Tools to Improve Data Collection and Management of National Surveys in Support of the Sustainable Development Goals
17 Jul 2020
9040 PRC Developing Gross Ecosystem Product Accounting for Eco-Compensation
25 May 2020
9043 MON Strengthening Capacity for Environmental-Economic Accounting 18 Mar 2020
9054 INO Enhancing the Regulatory Framework of Financial Sector Development and Oversight
8 Oct 2020
9095 REG Strengthening Integrated Water Resources Management in Mountainous River Basins
15 May 2020
9145 PRC Poverty Reduction in Liupanshan, Ningxia Hui Autonomous Region 27 Apr 2020
9180 MON Supporting Financial Sector Development and Stability 13 May 2020
9201 MON Gender-Responsive Sector and Local Development Policies and Actions
26 Mar 2020
9207 PAK Education Sector Assessment 10 Mar 2020
9228 PRC Vocational Education-Enterprise Collaboration for Student Employment-Based Poverty Reduction in Gansu
23 Apr 2020
9233 MON Conservation of Forest Genetic Resources 2 Jun 2020
9235 REG Strengthening Tax Policy and Administration Capacity to Mobilize Domestic Resources
2 Apr 2020
9241 PRC Creating Enabling Environments for Gradual Rural–Urban Migration and Urban Settlements Development in Chuxiong Yi Autonomous Prefecture
11 Mar 2020
9265 VIE Support to Strengthening Local Health Care Program 8 Jun 2020
9271 REG Facilitating Small and Medium-Sized Enterprises Foreign Direct Investment Flows: An ASEAN+6 Case Study
2 Apr 2020
9294 REG Promoting Green Local Currency-Denominated Bonds for Infrastructure Development in ASEAN+3
28 Feb 2020
9405 PRC Reforming the System of Personal Income Tax and Social Security Contributions
3 Jun 2020
9444 NEP Support for Value Chain Development under the Nepal Agriculture Development Strategy
30 Sep 2020
9477 KAZ Establishment of the Kazakhstan Knowledge Center on Integrated Water Resources Management
27 Aug 2020
9494 ARM Vertical Mobility and Safety Improvement 15 Apr 2020
16 | IED Synthesis Paper No. 8
TA No. Location Title
TCR Circulation
Date 9702 TIM Preliminary Assessment for Dili Airport Runway Project Upgrading
Project 29 Apr 2020
9718 REG Developing an Accountability Mechanism Framework for Financial Intermediaries
24 Mar 2020
8025 REG SASEC Subregional Energy Efficiency Initiative 9 Sep 2020
8085 BAN Support for the Third Primary Education Development Project 10 Sep 2020
8357 VIE Improving Operational Performance of the Water Supply Sector 18 Sep 2020
8426 BAN Capacity Building for Portfolio Management 2 Jul 2020
8619 REG South Asia Subregional Economic Cooperation Cross-Border Power Trade Development
21 Aug 2020
8728 PRC Improving Efficiency in Public Service Procurement, Delivery, and Financing
10 Jul 2020
8734 PRC Improving Energy Efficiency, Emission Control, and Compliance Management of the Manufacturing Industry
6 Aug 2020
8803 BAN Strengthening Monitoring and Enforcement in the Meghna River for Dhaka's Sustainable Water Supply
8 Sep 2020
8987 MYA Improving Road Network Management and Safety 23 Jul 2020
9039 PRC Promotion of Environmentally Sustainable Infrastructure Investment in Asia and the Pacific
21 Jul 2020
9045 ARM Export- and Innovation-Led Industrial Development 9 Jul 2020
9197 REG Creating a Regional Settlement Intermediary in ASEAN+3: Cross-Border Settlement Infrastructure Forum
16 Sep 2020
9301 KAZ Fostering the Development of Renewable Energy 17 Jul 2020
9379 PRC Comprehensive Hub-Oriented Transportation Strategy for Urumqi Metropolitan Area
1 Sep 2020
C0007 REG Support for Implementing the Action Plan for Transport and Trade Facilitation in the Greater Mekong Subregion
4 Sep 2020
C0010 REG South Asia Economic Integration Partnership 25 Sep 2020
C0013 INO Sustainable Infrastructure Assistance Program 15 Sept 2020
ADB = Asian Development Bank, ARM = Armenia, ASEAN = Association of Southeast Asian Nations, BAN = Bangladesh, DMC = developing member country, IND = India, INO = Indonesia, KAZ = Kazakhstan, MON = Mongolia, MYA = Myanmar, NEP = Nepal, PAK = Pakistan, PCR = project completion report, PHI = Philippines, PHI = Republic of the Philippines, PRC = People's Republic of China, REG = regional, SASEC = South Asia Subregional Economic Cooperation, SRI = Sri Lanka, TA = technical assistance, TCR = technical assistance completion report, TIM = Timor-Leste, VIE = Viet Nam. Source: Asian Development Bank (Independent Evaluation Department).
