Document of The World Bank Report No: 30232 IMPLEMENTATION COMPLETION REPORT (CPL-38050 SCL-3805A SCPD-3805S) ON A LOAN IN THE AMOUNT OF US$265 MILLION TO THE UNITED MEXICAN STATES FOR A TECHNICAL EDUCATION AND TRAINING MODERNIZATION PROJECT October 15, 2004 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized
CEA Accredited Testing Center (Organismo de Evaluación Acreditado)CECATI Training Center for Industrial Skills (Centro de Capacitación para el Trabajo
Industrial)CETMAR Upper Secondary Technical Ocean Centers (Centros de Estudios Tecnológicos del Mar)CGUT SEP's General Coordinating Entity for Technological Universities (Dirección General de
Universidades Tecnológicas)CIMO Multiple Support Service Program (Programa de Calidad Integral y Modernización)CONALEP National Professional Technical Education School (Colegio Nacional de Educación
Profesional Técnica)CONEVyT National Council on Lifelong Learning and Training (Consejo Nacional de Educación
para la Vida y el Trabajo)CONOCER Trust in Support of the System of Occupational Standards and Certification (Fideicomiso
de los Sistemas Normalizado de Competencias Laboral y Certificación de Competencia Laboral)
COSNET SEP's Council of the National System of Technological Education (Consejo del SistemaNacional de Educación Tecnólogica de la SEP)
GA Guarantee AgreementGEL General Education Law of 1993IDB Inter-American Development BankICATE SEP's Work Training Institute (Instituto de Capacitación para el Trabajo)IL Implementation LetterLD Legal Documents, including the LA and the GALA Loan AgreementNAFIN Nacional Financiera S.N.C.(the Borrower)NAFTA Free Trade AgreementOC Licensed Certification Agency (Organismo Certificador)PAC Training Support Program (Programa de Apoyo a la Capacitación) PAE Employment Support Program (Programa de Apoyo al Empleo)PAU Project Administrative UnitPROBECAT Labor Fellowship Retraining Program (Programa de Becas de Capacitación para el
Trabajo)PSC Project Steering Committee
SAR Staff Appraisal ReportSEIT SEP's Undersecretariat for Technical Education and Research (Subsecretaría de
Educación Tecnológica e Investigación)SEP Secretary of Education (Secretaría de Eduación)SHCP Secretary of Finance (Secretaría de Hacienda y Credito Público)SICAT For the Job Training System (Sistema de Capacitación para el Trabajo)SME Small and medium enterprisesSTPS Secretary of Labor and Public Welfare (Secretaría de Trabajo y Previsión Social)UEPC SEP's Undersecretariat for Educational Planning and Coordination (Subsecretaría de
Planeación y Coordinación Educativa)UTS Upper technical secondary
Vice President: David de FerrantiCountry Manager/Director: Isabel Guerrero
Sector Director: Ana-Maria Arriagada Task Team Leader/Task Manager: Juan Prawda/Mark V. Hagerstrom
MEXICOMX TECHNICAL EDUC/TRAINING
CONTENTS
Page No.1. Project Data2. Principal Performance Ratings3. Assessment of Development Objective and Design, and of Quality at Entry4. Achievement of Objective and Outputs5. Major Factors Affecting Implementation and Outcome6. Sustainability7. Bank and Borrower Performance8. Lessons Learned9. Partner Comments10. Additional InformationAnnex 1. Key Performance Indicators/Log Frame MatrixAnnex 2. Project Costs and FinancingAnnex 3. Economic Costs and BenefitsAnnex 4. Bank InputsAnnex 5. Ratings for Achievement of Objectives/Outputs of ComponentsAnnex 6. Ratings of Bank and Borrower PerformanceAnnex 7. List of Supporting Documents
Project ID: P034490 Project Name: MX TECHNICAL EDUC/TRAININGTeam Leader: Mark V. Hagerstrom TL Unit: LCSHDICR Type: Core ICR Report Date: October 29, 2004
SCPD-3805SCountry/Department: MEXICO Region: Latin America and the
Caribbean Region
Sector/subsector: Vocational training (92%); Central government administration (8%)Theme: Education for the knowledge economy (P); Social risk reduction (P);
Participation and civic engagement (S); Social analysis and monitoring (S)
KEY DATES Original Revised/ActualPCD: 11/10/1993 Effective: 03/31/1995 08/11/1995
STAFF Current At AppraisalVice President: David De Ferranti Shahid S. HusainCountry Director: Isabel Guerrero Edilberto L. SeguraSector Manager: Ana-Maria Arriagada Kye Woo LeeTeam Leader at ICR: Juan Prawda Mari MinowaICR Primary Author: Mark V. Hagerstrom
3. Assessment of Development Objective and Design, and of Quality at Entry
3.1 Original Objective:
The Technical Education and Training Modernization Project (Proyecto de Modernización de la Educación Técnica y Capacitación - PMETyC) aimed at improving the quality of technical education and training system in Mexico so as to meet the critical needs of the country's productive sector in a flexible manner.
PMETyC was designed and implemented with the intention of promoting the creation of a national system of competency-based standards to be used by employers and the formal technical education system. The creation of such a system had also the intended purpose of creating a transparent set of standards (basic, generic or transversal, and sector-specific) of expected performance in the workplace, placing an emphasis on the learning side of the teaching-learning equation, in contrast to the tendency to look at the supply-dominated formal education stock.
The project design responded to the three very important transformations that occurred in Mexico in the last decade, two in the education sector, and one in the economic sector of the country, which included: (i) the decentralization of the basic educational services to the states in May of 1992; (ii) the updating of the General Education Law (GEL) in 1993, whose Articles 44, 45 and 46 set the legal framework for parts of this project; and (iii) the enactment of the Free Trade Agreement (NAFTA) between Mexico, Canada and the United States on January 1st, 1994. In particular, Article 45 of the GEL recognizes the need for the training system provided by the education sector (Capacitación para el Trabajo) to conform to a system of competency-based standards.
3.2 Revised Objective:
The original Loan Agreement (LA), Guarantee Agreement (GA) and Letter of Implementation, hereby referred as the Legal Documents (LD), were amended ten times during the life of the project (see section 3.4), including partial cancellations of the Loan which modified Schedule 1 to the LA. Notwithstanding these amendments, the original project objectives remained unchanged throughout project implementation.
3.3 Original Components:
The following four components were included in the project description of Schedule 2 to the original LA: (a) establishment of a national system of competency standards and of a skills-testing and certification system; (b) modernization of training programs; (c) stimulation of demand for competency-based training and certification; and (d) project administration and studies.
Component A. Establishing a national system of competency standards and of a skills-testing and certification system included the following two sub-components:
Sub-component A1 to develop a national competency standards (basic, generic or transversal and sector-specific) system for individuals to be certified as competent to perform occupational functions and to strengthen the Trust in Support of the System of Occupational Standards and Certification (Fideicomiso de los Sistemas Normalizado de Competencia Laboral y de Certificación de Competencia Laboral - CONOCER).
Sub-component A2 to develop and operate, through a network of private sector institutions or enterprises validly licensed by the Trust (Organismos de Certificación - OC) and of individual or private sector institutions or enterprises validly accredited by an OC (Centro de Evaluación Acreditado - CEA), a national testing and certification program to measure and certify the attainment by individuals of the skill levels specified by the competency-based standards developed under sub-component A1 above.
- 2 -
Component B. Modernizing training programs to increase their flexibility and relevancy. This component included the following two sub-components:
Sub-component B1 to develop and implement on a pilot basis new competency-based training modular courses by the National Professional Technical Education School (Colegio Nacional de Educación Profesional Técnica - CONALEP), the Center for Technical Studies in Industry and Services (Centro de Estudios Tecnológicos Industrial y de Servicios - CETIS) and the Training Center for Industrial Occupations (Centro de Capacitación para el Trabajo Industrial - CECATI).
Sub-component B2 to develop and implement new training materials by public and private training institutions based on the competency-based modular courses described the sub-component B1 above.
Component C. Stimulating demand for competency-based training and certification to promote private sector initiative and participation in training design and implementation. This component included the following six sub-components:
Sub-component C1 to promote the competency standards developed under Component A.
Sub-component C2 to provide grants on a pilot basis to Mexican private sector enterprises in priority productive sectors to develop and implement competency-based training programs for their workers and to assess trainee attainment of competencies from such programs.
Sub-component C3 to provide grants to small and medium private enterprises (SMEs), under the framework of the Multiple Support Service Program (Programa de Calidad Integral y Modernización - CIMO), having as few as one and no more than 250 employees, to upgrade the skills of their labor-force under the competency-based standards developed under Component A.
Sub-component C4 to provide competency-based training developed in Component A under the Labor Retraining Fellowship Program (Programa de Becas de Capacitación para Trabajadores - PROBECAT).
Sub-component C5 to provide grants to cover a portion of the fees charged by the OCs and the CEAs for individuals taking the tests required for obtaining certification of their acquired competencies.
Sub-component C6 to provide: (i) equipment to participating public and private training institutions; and (ii) training and certification for instructors required for teaching of the skills necessary to meet the competency standards developed under Component A.
Component D. Project administration, information systems and studies included the following four sub-components:
Sub-component D1 to carry out studies to, inter alia: (i) determine the mechanisms for, and evaluate the probabilities of, eventual attainment of financial solvency by the Trust; (ii) identify and test (first in 1995, and thereafter as a comparative follow-up in 1999) the skills and competencies generally found across occupational clusters and economic sectors in the Mexican productive sector; and (iii) identify the combination of basic skills most likely to contribute to an individual's success in attainting competency certification.
- 3 -
Appraisal estimate total cost and financing (US$ million)
Components
Total IBRD Government Beneficiaries
Rating
Component A. Establishing a national system of competency standards and of a skills testing and certification system. Component B. Modernizing training programs. Component C. Stimulating demand for competency-based training and certification. Component D. Project administration, information systems and studies Total
33.1
152.8
208.0
18.1
412.0
17.8
115.0
118.4
13.8
265.0
15.3
37.8
60.9
4.3
118.3
0.0
0.0
28.7
0.0
28.7
Source: Annex 9 – Project Component by Financier – of the Staff Appraisal Report (SAR). The World Bank, Report 13416-ME, page 86, October 5, 1994
Sub-component D2 to carry out studies to, inter alia: (i) improve the Secretary of Education's (SEP) data base and information systems to identify training course enrollment trends; (ii) evaluate differences in labor market performance and employment success between trainees who participated in traditional vocational with those in competency-based courses; and (iii) evaluate training institutions' adaptation from traditional to competency-based training.
Sub-component D3 to carry out studies to, inter alia: (i) evaluate the effectiveness of competency-based training provided under CIMO to small and medium enterprises; (ii) evaluate the effectiveness of competency-based training provided under the PROBECAT scheme; (iii) evaluate, on the basis of case studies, the impact of competency-based training on selected private sector enterprises; and (iv) evaluate the competency-based skills certification process and related certification incentives program.
Sub-component D4 to manage through the Project Administrative Unit (PAU) and the Project Steering Committee (PSC) the operation of this project.
3.4 Revised Components:
The US$265 million Loan for this project was approved by the Board on October 27, 1994. The LDs were signed on February 7, 1995 and became effective on August 11, 1995. After Board approval, the LDs underwent ten amendments (including three partial cancellations of the Loan modifying Schedule 1 to the Loan Agreement), none of which substantially affected the original objectives (the dates of each amendment appear in Annex 7 of this report). The amendments included: (i) partial Loan cancelations; (ii) reallocation among the different expenditure categories of Schedule I to the LA; (iii) addition of new executing agencies; (iv) addition of new category of expenditures (construction); (v) modification of the PAU's structure and staff; (vi) extensions of deadlines in carrying out and completing, and in some cases deleting,
- 4 -
required evaluation studies; (vii) addressing emergency for situations caused by unforeseen natural causes; (viii) modification to cost-recovery formulas for Component C; (ix) modification to implementation schedules and project performance indicators; and (x) extension of the closing date.
At the request of the Borrower, this project underwent three partial Loan cancelations during project implementation. The first one for US$30 million (11% of the total Loan Amount), was carried out on July 14, 1995 under the context of the first amendment to the LDs. Accordingly, Schedule 1 to the LA was modified and the revised total Loan decreased to US$235 million. This cancellation was a direct consequence of the severe financial and economic crisis Mexico experienced at the end of 1994 where the Mexican peso suffered a 200% devaluation from 3 Mexican pesos to 9 per dollar. The second Loan cancelation of US$39,688,051.39 was carried out on September 7, 2001 under the context of the eight amendment to the LDs. Schedule 1 was again modified and the revised total Loan decreased to US$195,311,948.61. The Bank and the Borrower recommended this second cancellation following the supervision missions of May and August 2001 that identified significant unspent amounts of the Loan proceeds, mainly due to the budgetary restrictions and cuts made in the CY01 Government's Budget by the Secretaría de Hacienda y Crédito Público (SHCP). The third and final partial cancellation of US$14 million was carried out on August 1, 2002 under the ninth amendment to the LDs. Schedule 1 was modified and the total Loan decreased to US$181,311,948.61. This cancellation was requested by the Borrower as a result of a new review made by the Bank and the PAU in March 2002 on the undisbursed Loan and the Borrower's likely spending capacity in the remaining 15 months until the closing of the project.
The following amendments added several executing agencies to the project: (i) under the third amendment dated February 27, 1998, the Upper Secondary Technical Agricultural Centers (Centros de Bachillerato Tecnológico Agropecuario - CBTAs), the Upper Secondary Technical Forestry Centers (Centros de Bachillerato Tecnológico Forestal), the Upper Secondary Technical Continental Water Centers (Centro de Estudios Tecnológicos de Aguas Continentales) and the Upper Secondary Technical Ocean Centers (Centros de Estudios Tecnológicos del Mar - CETMAR); (ii) under the fifth amendment dated May 24, 1999, SEP's Upper Secondary Technological Industrial Centers (Centros de Bachillerato Tecnológico Industrial - CBTIs), SEP's General Coordinating Entity for Technological Universities (Dirección General de Universidades Tecnológicas - CGUT) and SEP's Work Training Institute (Instituto de Capacitación para el Trabajo - ICATE); (iii) under the sixth amendment dated October 22, 1999, the National Institute for Adult Education (Instituto Nacional de Educación para los Adultos); (iv) under the seventh amendment dated March 26, 2001, the Colegio de la Frontera Sur; and (v) under the tenth, and last amendment, authorized by the Bank on September 4, 2002, countersigned by the Borrower represented by Nacional Financiera (NAFIN) on September 5, 2003 and countersigned by the Guarantor represented by SHCP, on June 10, 2003, the Coordination of Upper General Secondary (Dirección General del Bachillerato).
The fourth and seventh amendments, dated respectively July 23, 1998 and March 26, 2001, created a new category of expenditure (construction) in Schedule I to the LA. The former amendment aimed at refurbishing the offices of CONOCER, while the latter, channeled a modest amount of project funds (under and emergency program launched by the Mexican Government) for the reconstruction of some project-related CONALEP school buildings damaged by a severe hurricane. The second amendment dated October 9, 1997, modified Schedule 1 to the LA.
The third amendment to the LDs of February 27, 1998 transformed the original structure of the PAU and increased its staff to better coordinate the execution of component B and sub-component D2 and serve as the project's linkage with other implementation agencies, namely CONOCER, CONALEP, INEA and the
- 5 -
Ministry of Labor and Public Welfare (STPS) for the preparation of reports and communication with the World Bank.
Under the context of this third amendment, the Bank also accepted extending the deadlines in carrying out and completing certain evaluation studies, specifically the one regarding the difference in labor market performance and employment success between trainees who participate in the competency-based courses provided by component B and those who participated in the traditional courses. It also accepted extending the deadline to submit to the Bank the project annual review. The fifth amendment dated May 24, 1999 extended the closing date from September 30, 2000, to June 30, 2003, modified the project's implementation schedule and agreed on the corresponding project performance indicators. The seventh amendment dated March 26, 2001 approved changes to the implementation schedule and the project performance indicators.
Finally, during the March 2002 supervision mission the following amendments were discussed with the Bank: (i) recognition of the replacement of PROBECAT and CIMO (which had disappeared) by the Training Support Program (Programa de Apoyo a la Capacitación - PAC) and the Employment Support Program (Programa de Apoyo al Empleo - PAE), respectively; (ii) cancellation of some tracer studies included in Part D of Schedule 2 to the LA; (iii) modification of the cost-sharing ratios of training provided by Component C of the project through STPS, increasing from 40% to 50% the contribution of SME employers and decreasing from 60% to 50% the Government contribution through the project; and (iv) modification of Schedules 1 and 2, as well as section 1.02 to the LA. On September 4, 2002 the Bank approved this (tenth) amendment to the LA, NAFIN (the Borrower) countersigned on September 5, 2002, SHCP (the Loan Guarantor) countersigned on June 10, 2003 (20 days before the Loan closing date) and the Mexican Federal Treasury ratified this amendment on October 29, 2003 (almost four months after the Loan closing date and two days before the expiration of the grace period to document all expenditures incurred by the project against the Loan before the closing date).
3.5 Quality at Entry:
Even though no formal Quality at Entry Criteria was available and required at the time this project was prepared and appraised in 1993-1994, for purposes of evaluating quality at entry, this ICR follows current Bank Quality Assurance Group (QAG) standards. Firstly, the project was consistent with the Mexico's Country Assistance Strategy (CAS) available at that time (Report No. 13008-ME of May 13, 1994) which emphasized as main goals poverty reduction, human resource development and private sector led growth (pages 10-12). Accordingly, the CAS highlighted the need to invest in skills training and to improve further the efficiency of resource allocation in technical training as well as the targeting of training for the unemployed. The CAS also recommended the progressive introduction of competency-based education and capacity building for small entrepreneurs (para. 44).
Notwithstanding that the preparation process had the political support of the Mexican Government (Ministries of Finance, Education and Labor) and was prepared by a qualified technical team, not all of the project components were fully owned by their respective designated stakeholders, particular SEP's Undersecretariat for Technical Education and Research (SEIT). Subsequently, the level of participation necessary to meet project objectives was never achieved.
At the time of project preparation and appraisal, there were no other Bank-financed competency-based experiences available to extract lessons learned during the design process of this innovative project. There were some Bank-financed traditional vocational and technical education operations, such as, inter alia the Mexico: Third Technical Training Project - CONALEP (Loan 3358-ME) and the Labor Market and
- 6 -
Productivity Project - PROBECAT (Loan 3542-ME), which differed substantially from the original ethos of PMETyC. Notwithstanding the lack of other related Bank-financed projects, the preparation team undertook a systematic and extensive effort to learn from other international competency-based experiences through study tours to Australia, New Zealand and the United Kingdom. The model operated by the UK, where labor and education are under one single Ministry, finally served as the basis for PMETyC's design. The resultant project design was very ambitious and complex involving two Ministries (SEP and STPS) in its envisaged implementation. The project took about three years to get off the ground after Board approval of the Loan. One project component (D), was only partially implemented.
A thorough review of the project preparation files reveals other quality at entrance shortcomings, both from the Bank's side as well as from the Borrower's. No implementation readiness assessment was ever carried out, despite that Annex 7 of the Staff Appraisal Report (SAR) includes a very detailed implementation plan, and Annex 2 of the SAR includes an evaluation of CONALEP and PROBECAT stemming out from the implementation of the Loans 3358-ME and 3542-ME. In addition, except for the economic evaluation undertaken for CONALEP and PROBECAT, no formal economic, cost-benefit or cost-effectiveness analyses were done for the remaining components of the project during the preparation phase. The project design that was appraised in June 21, 1994, did not include a first-year procurement plan, nor detailed terms of reference of the technical assistance and studies required during the implementation cycle. Despite that the project description included the carrying out of tracer and impact evaluation studies, the design did not incorporate: (i) a monitoring and evaluation system and methodology to update, from the onset of project implementation, the performance indicators; and (ii) the baseline data of project performance indicators. Satisfactory compliance with all of the above-mentioned output and impact oriented quality at entry criteria shortcomings are required nowadays by the Bank in the processing of any new operation before the negotiations phase.
Furthermore, as discussed further in section 4.1 of the ICR, it is clear that the project as defined in the Project Description of Schedule 2 to the LA was not addressed at the objectives it sought to achieve.
Finally, no social assessment, including employers and labor market surveys, were made during the preparation and appraisal phases concerning: (i) the demand for training; (ii) the likelihood of SME to participate and share the cost of training their employees; (iii) the entrepreneurs commitment to the idea of establishing a national system of competency-based standards and of a skill-testing and certification system; and (iv) the desirability of the upper secondary technical education system stakeholders to modernize their services through the introduction of a competency-based modular curricula.
Given the above observations, the ICR mission ranks the Quality at Entry of this project as unsatisfactory.
4. Achievement of Objective and Outputs
4.1 Outcome/achievement of objective:
There is little available evidence to indicate that the project was successful in achieving its stated developmental objectives as established in the Project Description of Schedule 2 to the LA, namely improving the quality of technical education and training system in Mexico so as to meet the critical needs of the country's productive sector in a flexible manner. Qualitative observations stemming out from the different Bank supervision missions seem to indicate that the project did not: (i) ensure wide participation of the private sector, especially SMEs, in using CONOCER's approved competency-based standards in the training and certification of their employees; (ii) increase the flexibility and relevance of the training
- 7 -
system; and (iii) ensure better labor market performance and employment success for the graduates of the competency-based curricula as compared with the graduates of the traditional vocational curricula.
The following are some of the key project performance and impact questions that needed to be answered (according to the studies required to be carried out under Component D before the tenth and last amendment) by the project at mid-term review and which were insufficiently addressed by the Borrower. Is CONOCER scheme really owned by entrepreneurs and their workforce, especially those from the private sector? Is CONOCER an efficient operation? What has been the impact of workers that have certified their skills in terms of their productivity, wages, career opportunities, advancement and professional mobility? Are there significant differences in productivity, labor relations and the organizational climate between firms that have benefited from CONOCER, especially SMEs, and those in similar economic sectors that have not? What have been the significant differences in job findings, labor market performance and employment success between trainees and/or students who participated in competency-based programs with respect to those graduating from the traditional programs? What was the impact of the training provided to unemployed workers under PROBECAT (currently PAC) and to workers of SMEs under CIMO (currently PAE) in terms of work opportunities and productivity? Some of these questions are partially addressed in section 4.2 of this ICR.
The ICR did gather sufficient evidence that the project achieved the following objectives, related tasks and processes: (a) establishment of a national system of competency standards and of a skills testing and certification system (CONOCER); (b) implementation of a competency-based education curriculum in some selected schools belonging to the formal technical training education sub-system; (c) provision of competency-based learning materials and equipment to participating upper secondary technical schools; (d) provision of competency-based training to unemployed workers and workers of some SMEs; and (e) establishment of supportive management information systems.
This ICR flags an inconsistency between the way the project objectives were stated in the Project Definition of Schedule 2 to the LA and the real ethos of the project. In effect, the project objectives were stated as "improving the quality of technical education and training system in Mexico so as to meet the critical needs of the country's productive sector in a flexible manner". Compared to the traditional Bank-financed operations leading to quality improvement, the project design did not include any of the following proven-effective activities, which are directly linked to improving the quality of technical education, including, inter alia: (i) the measurement of learning achievement outcomes (including baseline and project completion data) and efficiency-related indicators, like promotion, dropout and repetition rates (including baseline and project completion data); (ii) the pre and in-service teacher and school principal training aimed at improving the pedagogical practices in the classroom of the participating technical schools; (iii) the provision of learning inputs; (iv) the development of student orientation, labor practices and placement mechanisms and strategies; and (iv) the coordination and linkages with the labor market.
It is the strong view of this ICR that the real thrust of this investment as designed was to improve the pertinence of the upper secondary technical education and training provided in Mexico, to ensure a better fit with the general and sector-specific competency requirements of the employers (public and private) and the labor market (formal and non-formal). In short, the project successfully implemented some project tasks and processes as specified in the project's description and its components, and might have been considered as marginally satisfactory if judged relative to such a reformulation of the project objectives; however, its impact in attaining the originally stated project objectives is rated as unsatisfactory.
4.2 Outputs by components:
- 8 -
This section is based on the information: (i) collected by the Bank during the last supervision mission carried out in June 2003; (ii) provided by the Borrower's final report submitted to the Bank on April 30, 2004; and (iii) included in the project's file.
With respect to Component A (Establishing a national system of competency standards and of a skills-testing certification system), the project was successful in establishing the Council for Normalization and Certification Standards (a Trust known in México as the Fideicomiso de los Sistemas Normalizado de Competencia Laboral y de Certificacion de Competencia Laboral - CONOCER) for the development and certification of sector-specific national competency standards. At the time of the June 2003 mission, CONOCER had approved 613 norms for 5 levels of employment in 12 broad sectors of the economy (agriculture, mining, construction, mechanical/electrical/electronic, telecommunications, manufacturing, transportation, sales of goods and services, financial services, health and social protection, social communication and knowledge development). These 12 broad sectors represent about 25 sub-sectors of the Mexican economy. Out of this universe of approved norms, 62% were for level 2 of occupational qualification (corresponding to qualified workers) and 24% for level 3 (corresponding to qualified technicians). Sector-wide, 27% of the approved norms were in the manufacturing sector, 14% in the sales of goods and services sector, 14% in the mechanical, electrical and electronic sectors, and 12% in the agricultural sector. Out of these approved norms, almost 54% were active (being utilized by the employers), and the rest, 283 norms were dormant. This lack of norm "usability" by employers is an unwelcome sign that part of the activities of CONOCER, especially during the early stages of project implementation, were not demand-driven by interested employers. This is also partially explained by the very expensive copyrighted strategy used as inputs for the development of the Mexican standards. Approved norms have a validity of about two years, at which time they need to be revisited, and likely, revised and updated.
For the definition and approval of these norms, CONOCER established 66 Competency Standardization Committees (Comités de Normalización) and 10 Sub-committees, which included an ample spectrum of the sector-specific stakeholders, especially entrepreneurs, experts and, in some cases, representatives of the workers. To measure and evaluate the attainment by individuals of skill levels specified by the approved norms, CONOCER accredited 1,413 individuals, public and private sector institutions and enterprises known as CEAs (Centro de Evaluación Acreditado - CEA). About 46% of the CEAs were accredited public technical schools participating in Component B, 25% where private firms, 10% public firms, 12% associations and employer umbrella organizations, 4% were private schools, and one CEA was a workers union. To certify acquired skill levels specified by the approved norms and previously evaluated by the CEAs, CONOCER accredited and licensed 32 private and public sector institutions and enterprises known as OCs (Organismos de Certificación). Unfortunately, all these 32 OCs are located in Mexico City, thus increasing the already significantly higher, certification costs due to transportation factors. The project provided grants to OCs and CEAs through CONOCER, for their staff training and technical assistance needs, to support their start-up activities, to develop testing material and, in case of the OCs only, to help with their incremental operating costs.
CONOCER granted an equivalent of about 87,442 certificates to approximately 67,438 individuals through a total of 195,844 competency units over the life of the project. These results surpass the original targets recorded in the Implementation Letter. According to preliminary information provided by CONOCER to the June 2003 mission, the aggregated average unit cost of the entire certification process for all the 12 sectors of the economy and the five levels of occupational competency, including the definition, validation and approval of the norm, evaluation and certification, decreased in the period 2000-2002 from MX$1,814.28 to MX$848.74 (53% reduction). The Government's subsidy in this certification process has decreased 28% in this period. In fact, while in 2000, certification fees represented
- 9 -
only 14% of the entire process (MX$260.94 out of MX$1,814.28), in 2002 the cost-recovery represented 38% of the total process cost (about MX$321.59 out of MX$848.74). Obviously, this preliminary cost estimates vary by economic sector and the level of occupational competency corresponding to the norm. Improvements in the efficiency levels of CONOCER emerge from the lessons learned during the early stages of project implementation cycle that indicated the need to better target CONOCER's efforts to the second level of occupational competency and to the sectors of the economy where the certification process was most in-demand. The efforts of CONOCER to simplify and streamline the entire rather complex certification process, although still in its initial stages, resulted in greater efficiency, especially concerning the simplifying of the skill evidence portfolios. Given that the average unit cost of certification still exceeds the average unit cost of a 200 hour training course in Mexico, there is room for continued improvement of this component efficiency. This also indicates that the current priority CONOCER gives to the certification process instead of training has to be reversed to make this process even more efficient.
Further strengthening of CONOCER's alliances with the private employers is warranted to ensure a change of culture among employers regarding the need to train and certify their working force. These efforts should culminate in private and public employers largely financing the services provided by CONOCER, which would be a dramatic shift from current public subsidy. The current linkages of CONOCER with the technical education and training systems need strengthening. Likewise, CONOCER needs to work in better coordination with the National Institute of Adult Education, which is responsible for setting the basic skill standards of adult education curriculum. Also urgently needed is a more rational decentralization of the OCs in order to bring them closer to potential beneficiaries and decrease certification costs due to mobility. Furthermore, the ICR perceived that the certification of competencies standards is not fully aligned with the provider, thus individuals in the process of certification seldom have available to them modular courses required to achieve the prescribed requisites.
The lengthy process between the definition and approval of a given sector-specific labor competency-based norm is likely to be out of sync with the rhythms firms follow. By the time a norm is approved it is likely to be already outdated due to the rapid technological changes occurring in the productive sector. This explains in part the reason that 46.6% of the norms approved by CONOCER are dormant (unused). The ICR recommends that when an international norm is available, it ought to be adapted, if possible, to the Mexican sector-specific economic and productive reality.
Although there is room for further improvement, given that CONOCER is gradually improving its targeting efforts and internal efficiency, the ICR ranks the implementation of Component A of the project as satisfactory.
With respect to Component B (Modernizing training programs to increase their flexibility and relevancy), the project was successful in establishing a competency-based curriculum in 727 public upper technical secondary (UTS) schools (about 54.3% of the total universe of 1,339 UTS schools) benefiting 754,424 students during the life of the project. Out of these number of students, only 62,891 (8.3% of the ones enrolled in the new curriculum), certified their acquired competencies in the framework of Component A. The number of certified students is extremely small due to the relatively high cost of certification, which is not subsidized by the Government, and the complexities inherent in assembling the evidence portfolio of acquired skills. About 382 UTS schools (28.5% of the universe) were certified by CONOCER as competency-based evaluation centers. PMETyC provided competency-based equipment to 392 UTS schools. The project supported the training of 31,380 teachers and principals of UTS schools and financed the conversion of 1,096 teaching modules from the old (subject-based) to the new (competency-based) curriculum including learning material.
- 10 -
According to the information provided by SEIT during the June 2003 mission, the annual impact of PMETyC by the number of students benefiting from this investment in each UTS subsystem in 2002 as a percentage of the total enrollment in that year is very small (except for CONALEP): CBTI with 18,187 students (3.6%); CECATIs and ICATEs with 27,458 students (4.5%); CONALEP with 210,225 students (100%); CBTA with 2,044 students (1.6%); CETMAR with 489 students (1.9%); and Technical Universities with 510 students (1%). Adding all the subsystems, only 16% (258,400) of the total number of students enrolled in 2002 in the UTS subsystems benefited in 2002 from this new competency-based curriculum.
The results of this component are mixed. Some activities within the Component were considered by the ICR as satisfactory while others as unsatisfactory. At the end of the following analysis, an aggregate rating is provided for this component. On the one hand, there is some evidence of installed institutional capacity to develop and implement a competency-based curriculum in the UTS sub-system. In some participating UTS schools there is clear articulation with the regional private and productive sectors which demand competency-based training services from these schools for their employees. In other schools, this articulation is very weak at best, and demand for their training services by the regional productive sector, almost nonexistent. In those UTS schools where the articulation with the regional productive sector is working well, the supply of competency-based training provided by these UTS schools to interested regional firms is limited mainly due to the following reason. Public UTS schools are forbidden by Law to provide invoices for the services provided, and thus they cannot charge the real cost for these services to interested firms demanding competency-based training for their workers. As private employers require an invoice from these UTS schools for taxation rebate purposes, interested firms tend to send only a token of their workers to receive competency-based training from qualified UTS schools, and a vicious cycle is created. Firms will limit the demand for training, and UTS schools, functioning on an approved public budget, will also limit the supply of training they provide. To eliminate this vicious cycle, some UTS schools offering competency-based training have adopted the "Cooperative" legal and fiscal arrangement, where invoices can be provided and real costs be charged, including a small profit. The issue at hand is that only a few of the 727 UTS schools have adopted this "Cooperative" scheme (like the Centros de Adiestramiento y Servicios Tecnológicos (CAST) of CONALEP).
Furthermore, in the UTS schools where both curriculums, the competency-based as well as the traditional subject-based, are simultaneously being offered, several Bank supervision mission as well as the June 2003 mission detected unclear situations concerning the use of equipment, teacher training and training material, due, in part, to the lack of a clear strategy for the generalization of the competency-based curriculum in the given school.
The few tracer studies done for CONALEP and the Technical Universities show that graduates from this new curriculum see their likelihood for employment is slightly increased. Unfortunately, no tracer studies were conducted for the other UTS sub-systems to ascertain the advantages and disadvantages of the modular competency-based curriculum over the traditional subject-based curriculum with respect of the job-related performance for graduates of each UTS modality. The lack of these studies did not allow the June 2003 mission to assess, for example: (i) the time it takes graduates from the traditional vocational and competency-based modality in UTS sub-system to find a job; (ii) the percentage of graduates from each modality that find a job related to their training within 6 months after graduation; (iii) appropriateness of the job acquired by type of graduate; (iv) differences in initial salaries between the two sets of graduates; (v) difference in job-related performance as seen by the employer; (vi) characteristics of the short-to medium-term professional ladder for each kind of graduate during their employment tenure; and (vii) the percentage of graduates that continue further education.
- 11 -
On the positive side, this Component has developed very good competency-based training materials for SEIT's UTS subsystem, totaling about 183 didactic packages and about 603 assorted learning materials.
For all of the above-mentioned reasons, the ICR ranks the implementation of Component B corresponding to CONALEP, the Technological Universities and of some CBTI, CBTA and CECATI schools (those showing an effective linkage with the productive sector) as satisfactory, while for the rest of the SEIT subsystem as unsatisfactory. As a whole, in that only about 16 percent of the student were participating in the new curriculum, the ICR ranks this component as unsatisfactory.
According to the project-related closing report provided by STPS to the June 2003 mission, Component C (Stimulating demand for competency-based training and certification to promote private sector initiative and participation in training design and implementation) financed the training of about 80,600 unemployed or sub-employed workers during the 1995-2003 period through the Sistema de Capacitación para el Trabajo (SICAT) which replaced PROBECAT. This training was provided through about 664 courses based on 60 sector-specific approved labor competency norms by CONOCER. Out of this universe of unemployed or sub-employed workers benefiting from the project, a total of 34,077 certificates of acquired skills against approved norms were granted by the project. In addition, the program provided training to 1,215 instructors. During the 1995-2003 period, PMETyC financed the training of an additional 66,832 workers of approximately 13,609 firms, mostly SMEs, through the Programa de Apoyo a la Capacitación (PAC), before CIMO. Of this number of trained workers, 10,123 were certified based on approved norms. The program provided training to 2,300 instructors. A survey study conducted by STPS (whose methodology and findings could not be assessed by the June 2003 mission) with a sample of almost 1,400 workers and about 100 firms who benefited from this project Component indicated that out of 14,222 unemployed or sub-employed workers trained by PMETyC during 2002, 74% (10,520) found a job after completing their training. However, the study did not control for other factors, thus it is not possible to ascertain if the finding of a job was directly correlated with the competency-based training provided, or due to other factors. Furthermore, in the field visits carried out during the last two supervision missions it was clear that: (i) for many participants the incentive was the stipend provided by PROBECAT/SICAT rather than the training course itself; and (ii) a significant number of graduates of this training program financed by the project were out of a job at the time of the supervision missions. The ICR therefore does not have sufficient clarity to determine the impact and appropriateness of the training provided to the unemployed and sub-employed. The survey found that the interviewed employers perceived turnover of trained workers, as well as absenteeism, decreased. Two other positive impacts emerged from this Component. Firstly, STPS modified its policy to finance only sector-specific labor competency-based training with approved norms by CONOCER and disregard in the future the financing of the traditional training. Finally, employers are absorbing a larger share of the cost of training, evaluation, and certification, although this share is still very small.
On the other hand, the June 2003 mission identified several important remaining challenges. First, financing the certification phase of this process represents a big bottleneck, and thus a relevant hurdle for certifying a larger number of workers. Second, firms that value the competency-based training, by and large, are willing to have their workers evaluated and trained, but decline to fully absorb the cost of certification. Third, some employers fear that certified workers may demand higher wages. Fourth, the cost of certification also represents a financial burden for the employees, and with minor exceptions, they also refuse to assume the burden. Fifth, some workers do not see the immediate benefit of the certification but rather they perceive it as a limitation to find a job. Finally, the Government declines to subsidize the certification process.
- 12 -
The above-mentioned STPS report is consistent with the June 2003 mission perception that the impact of this Component in the universe of workers and firms in Mexico is still very small. Out of a universe of about 2.8 million registered firms in Mexico, of which 95.7% are SMEs (about 2.68 million), the project benefited a little less than 14,000 (about half of one percent) during the entire life of the project.
The STPS report points to the limited number of CEAs and OCs (the latter ones, all located in Mexico City), as contributing to the limited impact of this Component. The report identifies as factors affecting the timely implementation of this Component: (i) the delays in releasing authorized budget resources; and (ii) the frequent budget cuts, especially in 2000 and 2001. According to the budgetary figures released by this report, on the average, about 60.3% of the authorized budget was actually spent during the 1995-2003 period, ranging from a low 25.5% in 1995 to a high rate of 83.4% in 2002. The authorized budget for this Component registered a significant increase in nominal terms of 31.6% per year in the period 1995-2001, and a sharp decrease of almost 55% in 2001, a year that almost paralyzed the activities of this Component.
The STPS report confirms, moreover, the observation made above that the asynchronous rhythms of CONOCER in establishing, validating and approving a norm and the rhythms required at the level of the firm, result in norms being outdated by the time they are approved.
For all of the above-mentioned reasons, the ICR ranks the implementation of Component C as satisfactory with respect to CIMO/PAC, and unsatisfactory, with respect to PAE/PROBECAT. As a whole, the ICR rates this Component as unsatisfactory.
With respect to Component D (Project administration, information systems and studies), PMETyC financed a total of 44 studies. Of these, the ICR highlights the following studies: (i) an external mid-term review of PMETyC carried out by a qualified Spanish consulting firm (CIDEC); (ii) a diagnosis of training demand; (iii) a structure analysis of the sectors of the Mexican economy sectors as related to their skill requirements; (iv) an occupational analysis; (v) a costs and benefits of PMETyC; (vi) a study of the possibilities of self-financing the OCs; and (vii) tracer studies of graduates of the competency-based curriculum of CONALEP and the Technological Universities. In addition, PMETyC financed the development and implementation of the following management information systems: (a) the supply of competency-based technical education (Sistema de Información de la Oferta Educativa Basada en Competencias - SINOE) which includes nine management information sub-systems; (b) CONOCER's Information System (Sistema Integral de Información - SIII); (c) Occupational Catalogue (Sistema del Catálogo Nacional de Ocupaciones - SICNO); (d) a Job-search information system (Sistema de Informacion de Apoyo al Empleo - SISPAE, formerly known as the Sistema del Servicio Nacional de Empleo - SISNE); and (f) a Training Support Management and Budget Information System (Sistema Administrativo Presupuestario del Programa de Apoyo a la Capacitación - SAPAC). At the time of the ICR, these management information systems were at different levels of implementation (going from fine tuning the design to implementation readiness, and fully implemented).
Despite the above-mentioned achievements, the Project should have included provision for the carrying out of key studies to assess the achievement of the developmental objectives of this investment. These output-oriented studies include: (i) enrollment trends in technical education and the demand for training in specific areas; (ii) the institutional changes of the technical sub-systems that adopted competency-based curriculum; (iii) the impact of the PROBECAT and CIMO activities financed by PMETyC; (iv) possibilities of self-financing of CEAs; (v) evaluation of the certification incentives; and (vi) tracer studies of graduates of the UTS sub-system that adopted a competency-based curriculum, apart from CONALEP and the Technological Universities that did carry out such studies. In particular, this last proposed study
- 13 -
would have provided valuable information concerning the differences in labor market performance and employment success between trainees who participated in traditional vocational and competency-based courses. As mentioned-before, the Bank agreed in September 4, 2002, under the context of the last (tenth) amendment, to exclude these studies from the project.
Finally, as part of Component D, a PAU and a Project Steering Committee were established. As mentioned in Section 7.1 of the ICR, the Project Steering Committee was not effective in addressing key risks presented during project execution, such, as inter alia: (i) significant delays in project implementation emerging mainly from the project's complex design; (ii) the inadequate private sector involvement; and (iii) the resistance of different interest groups. The PAU was given an increasingly important role in coordination during the letter years of implementation, beyond its original administrative functions. However, it was not successful in overcoming weaknesses in design and preparation, and the lack of ownership of the participating agencies, especially STPS and SEIT.
For the above reasons, the ICR ranks the implementation of Component D, as unsatisfactory.
Based on the above observations and judgements for the various components, the ICR rates the outcome of this investment as unsatisfactory in meeting the stated project objectives.
4.3 Net Present Value/Economic rate of return:
Not applicable.
4.4 Financial rate of return:
Not applicable.
4.5 Institutional development impact:
Three institutional development impacts are derived from PMETyC. Firstly, a National System of Competency Standards operated by CONOCER was developed and is fully implemented. CONOCER operates, through a network of validly licensed private sector institutions or enterprises, a national testing and certification program to measure and certify the attainment by individuals of the skill levels according to approved norms. However, as further explained in Section 6.1 of the ICR, the Legal Status and composition of the Trust that created CONOCER under this project will be changed by the Mexican Government, somehow affecting the institutional capacity that has been developed.
Secondly, institutional capacity was developed in part of the UTS sub-system to define and implement a competency-based curriculum. CONALEP (with 262 schools, 212,000 students and about 13,900 professors) was the only sub-system in the UTS system to have changed completely its curriculum to a modular competency-based one. However, as indicated in Section 4.2 of the ICR, the competency-based curriculum capacity developed under in the UTS system benefits a small percentage of the UTS universe.
Thirdly, and as mentioned before, STPS modified its policy to finance only sector-specific labor competency-based training with approved norms. This institutional development impact generated by PMETyC, reverses the traditional approach of STPS to finance any training, including non-competency based one. However, as also indicated in Section 4.2 of the ICR, the number of SMEs and/or unemployed workers that have benefited from this project is extremely low, representing at best about half of one percent of the potential universe.
- 14 -
Finally, the ICR could not find analytical evidence of any relevant impact on: (i) the modernization of the UTS subsystem, especially the CECATIS; (ii) the strengthening of coordination mechanisms among training institutions; and (iii) the appropriation by a significant number of firms, especially SMEs, of the competency-based standards and certification processes as mean of hiring and promotion of workers.
For all of the above reasons, the ICR of the view that the institutional development impact of this project was modest.
5. Major Factors Affecting Implementation and Outcome
5.1 Factors outside the control of government or implementing agency:
One major factor outside the control of the Mexican Government had an adverse effect on project implementation - lack of confidence among foreign investors in continue to invest in Mexico during 1994. Foreign investors started, by and large, cashing in their profits and pulling out their investments from Mexico once their dollar-backed investment instruments came to maturity. These short-term market pressures where addressed by the Mexican Government using available reserves in the Bank of Mexico. However this strategy proved not to be sustainable to the point that when President's Zedillo administration took over on December 1, 2004 (barely four weeks after Board approval of PMETyC), the reserves were at an historic low, equivalent to less than one month of imports. The Zedillo Government was forced to float the peso in mid-December 1994 (two weeks after its inauguration), which promptly sank and nearly took the entire Mexican banking system down with it. The Mexican peso suffered a 200% devaluation (from 3 Mexican pesos to 9 per dollar in early 1995). This crisis led to a delay of three months in the signing of the Loan after Board approval, as well as a delay of 10 months in its effectiveness. Finally, the Mexican Government requested on July 14, 1995 a partial cancellation of the Loan on the amount US$30 million (or 11% of the total Loan Amount), reducing the Loan to US$235 million. This cancellation, seen by the Mexican Government as a currency adjustment to a banking transaction, was applied to practically the entire Mexican portfolio financed by both the Bank and the Inter-American Development Bank.
5.2 Factors generally subject to government control:
The ICR identified two major factors under the government's control which severely affected the speed of implementation. The first one concerns the consistent delays in authorizations to spend the approved project-related budget. These delays ranged from five to about seven months after the start of the corresponding fiscal year, thus affecting the entire implementation cycle. The second factor corresponds to the budget cuts and restrictions, including those for PMETyC, imposed over the course of several fiscal years of project life. These two factors impeded the PAU, and other implementing entities, especially STPS, from effectively carrying out its approved work-plan and budget.
5.3 Factors generally subject to implementing agency control:
Unfortunately, the expansion of PAU functions beyond the original design as an administrative unit toward a coordinating role was largely unsuccessful in enhancing the level of ownership by SEIT and STPS. It did have some success in promoting the competency-based approach in civil service, public enterprises and municipal governments, but these activities were outside the intent of the operation which was to focus on the private sector.
5.4 Costs and financing:
- 15 -
The total project cost was estimated at appraisal at US$412 million equivalent, of which US$265 were from the Bank loan (64%), US$118.3 million equivalent (29%), from counterpart funds, and US$28.7 million (7%) was to be the expected contribution of private enterprises and trainees participating in the in-service training and skill certification, and benefiting from PAC (formerly CIMO) and technical assistance under the project. For the reasons fully explained in section 3.4 of the ICR, there were three partial cancellations to the Loan amounting to US$83.69 million (32% of the original amount contracted). These three partial cancellations decreased the total of the contracted Loan to US$181,311,948.61.
The project disbursed and documented about US$259 million, of which almost US$165 million were from the Loan and US$94 million were from counterpart funding and contributions from private enterprises and trainees (67.2% of the total counterpart funding estimated at appraisal). The Borrower requested cancellation of the remaining undisbursed Loan amount of US$18.32 million (US$18.04 million cancelled on August 26, 2003 and US$0.28 million on November 13, 2003).
The underspending is explained mainly by: (i) the three cancellations described before; (ii) the significant yearly delays in releasing the authorized project budget; (iii) the several budget cuts and restrictions that took place over the life of the project; and (iv) the complexity of the project design that significantly delayed its full implementation by about three years after Loan approval by the Board. It took the Borrower about two years after Loan effectiveness to get the rather complex project design off the ground, and some project activities, such as those in Component C, exhibited more severe project implementation difficulties, resulting in significant shortfalls in executing the totality of the authorized budget.
Based on information provided by the Borrower during the last supervision mission of June 2003, PMETyC as a whole registered an average 81.2% effective expenditure capacity out of the authorized budgets during this period. The year 2000 was particular effective with 92% expenditure capacity rate from the authorized budget, while 1995, 2001 and 2002 registered an expenditure capacity below the 80% mark. Component A consistently displayed an effective pattern of about 100% expenditure of its authorized yearly budget during the entire project cycle. Component C, on the other hand, experienced a very ineffective expenditure capacity performance, ranging from a low 34% in 1995 and 44% in 1996 to a high of 90% in 2000, and averaging 54% overall. Component B presented a reasonably average capacity expenditure pattern of 85% during the life of the project.
The already complex project design was made more complex by having agreed during negotiations to 20 different categories of expenditures (which appear in Schedule I to the LA). This cumbersome arrangement derived from having the traditional categories of expenditures of goods and equipment, training materials, consultant services, incremental operating costs, stipends, certification support, training and technical assistance, civil works and publicity, broken down by different components of the project as defined in Schedule 2 to the LA. Some of these categories, like technical assistance to SMEs were reduced to zero during the fourth amendment to the LA in May 1999. Of the remaining undisbursed US$18.044 million of the Loan, about 38.3% corresponds to the training category that was supposed to be provided under Component B and C, 18% to goods that were mostly to be acquired for Component B, 27% to consulting services for Component D (studies that were never carried out) and 10% to incremental operating costs under Component A.
6. Sustainability
6.1 Rationale for sustainability rating:
The ICR rates the sustainability of what has been achieved to date by this investment as likely on balance,
- 16 -
but sustaining and expanding on its activities will hinge to some degree on the nature of support from the Inter-Amercian Development Bank (IDB).
It is the ICR's impression that CONOCER (Component A) will be financially, technically and institutionally sustained by the Mexican Government. At the request of the SHCP, the Legal Status of the Trust that created CONOCER will be changed from a Fideicomiso Público No Sectorizado to a Fideicomisado Sectorizado within either the Ministry of Education (SEP) or the National Council on Lifelong Learning and Training (Consejo Nacional de Educación para la Vida y el Trabajo - CONEVyT). This proposed change will remove some of the managerial and policy autonomy granted to CONOCER during the life of the project by its Board, making it subject to more policy and management control by SEP or CONEVyT in the future. This proposed legal change will further reduce the participation of STPS, and thus, to a large extent, from the private employers. If this change occurs, the original philosophy supported by the project as well as the international experience will be totally disregarded, making the sustainability weaker.
CONOCER financial self-sufficiency strategy was restricted to the Certification Bodies. These Certification Bodies, which are private entities, were subsidized by the project until mid-project implementation cycle, at which point the subsidy was suspended. The Certification Bodies, despite implementing several strategies to keep them financially afloat (like selling consulting services and seeking economies of scale by certifying norms and charging for the service) did not achieve full self-financial subsidy. This lack of financial self-sufficiency continues to be an issue, hindering a more effective completion of the entire certification cycle among workers benefiting from this process. On the other hand, the production and updating of norms by CONOCER will continue to be subsidized by public funds allocated to the CONOCER Trust.
With respect to Component B, it is the ICR team's view that only the CONALEP sub-system will continue using the modular competency-based curriculum that was developed under the project and adopted in all of its schools during the life of the project. With respect to the other technical education subsystems coordinated by SEIT, the ICR team believes that this new curricular scheme, tested in a fewer number of schools, will not be sustained in the long-run. With the closing of this Loan, the likelihood of SEIT continuing to expand and strengthen the competency-based curricular scheme in other UTS sub-system schools, exception made of the CONALEP subsystem, is very low. This prospect could change if the proposed financing of a sequel of PMETyC by the IDB materializes that addresses the limitations pointed out in the ICR (still under discussion in late October 2004).
With respect to Component C, STPS will continue providing training and support developed under PMETyC by means of a new mega three phase APL from IDB approved and signed in 2002 (each phase is a three year loan of US$600 million). This situation may not really address the issue of sustainability, but instead postpones the question until the new IDB Loan comes to a closure. From that point of view, the assessment of the ICR is that the Bank loan did not ensure the sustainability of this component at the closing of the Loan.
In the absence of an evaluation culture it seems unlikely that the impact evaluation studies that the Borrower were originally supposed to carry out (until the last amendment to the LA changed that agreement), will be undertaken after the closing of the Loan.
6.2 Transition arrangement to regular operations:
The project design ensured that mainstream institutions implemented project activities of the three main
- 17 -
components of PMETyC from the outset. In the case of Component A, the Trust CONOCER was established by the Mexican Government. Component B was mainly implemented by SEIT along all the participating UTS sub-systems. Component B also included other implementing agencies outside SEIT, like SEP's Dirección General del Bachillerato and the Colegio de la Frontera Sur. Component C was implemented by the STPS. The only parallel unit created by PMETyC, was the PAU, which in addition of coordinating the activities with the other implementing agencies, was mainly responsible for the implementation of Component D. Except for CONOCER, CONALEP and to some degree, STPS, no managerial capacity was built to continue the envisaged changes in the overall system.
7. Bank and Borrower Performance
Bank7.1 Lending:
The choice of a task team leader (newly arrived at the Bank, and thus, inexperienced in Bank policies and procedures, as well as in technical-vocational education and competency-based curriculum) underpinned many of the shortcoming in project preparation and during the early stages of implementation, especially when both the Borrower and Bank management were finding their way into uncharted territories. Prior to PMETyC, the Bank had no experience in financing: (i) the establishment of a competency framework for the labor market; nor (ii) the development of a competency-based curriculum in the UTS subsystem from which pertinent lessons could be derived. As a consequence, some mistakes were introduced from the outset in the project design.
As mentioned in section 4.1 of the ICR, the way the project objectives were described in the Project Description in Schedule 2 to the LA and the real thrust of the project were completely different. The project aims at improving the relevancy and pertinence of upper secondary technical education and training under the framework of sector-specific labor competency-based standards. As already mentioned in Section 4.1 of the ICR, the project does not include in its design any proven-effective intervention to improve the quality of technical education and training systems in Mexico to meet the critical needs of the country's productive sector in a flexible manner. The fact of changing a traditional vocational curriculum into a competency-based one does not constitute, by itself, a strategy to improve the quality of learning. At best, it may improve the pertinence (or relevancy) of technical education and training. The ICR also rates as unsatisfactory the performance of the internal peer reviewing process that failed to flag this inconsistency.
The design was too complex, ambitious, and crossed over several sectors (Education and Labor). It was contrary to the prevailing Bank policy recommendations at that time concerning ways to improve effective implementation of future projects (Wapenhans Report of October 2, 1992). The economic sectors and levels of occupational skills to be targeted with a competency standards framework were unfocused. The rather complex design took about three years after Board approval (and two years after effectiveness) to get off the ground.
As mentioned in Section 3.5 of the ICR, if current Quality at Entry Criteria would be applied to the design, the preparation process also lacked several key assessment tools, including: (i) a social assessment; (ii) an economic assessment; (iii) a procurement capacity assessment and first-year procurement plan; and (iv) an institutional capacity assessment. The Bank appraisal team flagged in the SAR the following three potential risks: (i) complexity leading to poor implementation; (ii) inadequate private sector involvement; and (iii) resistance of interest groups due to poor understanding of the project. These risks proved to take place during implementation, yet there was an inadequate response by both the Borrower (through the
- 18 -
project's Steering Committee) and the Bank in effectively addressing them, especially until early 1998. In addition, the performance indicators (which appear in Annex 1) agreed upon with the Borrower about two years after Loan effectiveness are mostly input, and in some cases, process indicators. No output and impact indicators were included in the design.
Schedule 1 included too many categories of expenditure (20), some of which were never utilized throughout the entire eight-year life of the project, indicating that at the time of preparation and appraisal, ambitious expectations prevailed over more realistic assessments of the implementation capacity of the Borrower, especially given Mexico’s deteriorating economy. The project definition in Schedule 2 of the LA is too splintered making the operational supervision of this investment difficult. The SAR presented the proposed project in a more aggregated fashion, but it underwent unnecessary splintering during negotiations as it was translated into legal language. The costing of the project design was carried out without using the prescribed Bank rules and procedures in an appropriate fashion, with the result, for example, of category 1a in Schedule 1 to the LA being estimated at US$200,000 and immediately being increased to US$2.5 million during project implementation.
For all of the above reasons, the ICR team grants an unsatisfactory rating to the Bank performance during the lending process.
7.2 Supervision:
A total of 23 PSRs are included in SAP, of which 13 indicate actual field supervision missions carried out between Loan effectiveness in August 1995 to the closing of the Loan in June 2003. Reviewing the project files, there is evidence of some missions (like the one in June 10, 1997 and another one in July 27-31, 1998) for which no PSR was prepared and archived. The number of recorded field missions fall short of the 16 required, given that the project lasted eight years, and that at least one supervision mission was supposed to be carried every six months. The Bank did not carry out any supervision mission during 1996, a critical year where the project was almost paralyzed by lack of disbursements, beyond the opening of the Special Account with US$12 million. Only one supervision took place in 1997, a year when the project started getting off the ground. A total of five task managers were involved in supervising this investment: (i) the first one was also responsible for the lending process in its entirety from 1995 to the mid-1997; (ii) the second one led only one supervision mission in February 1998; (iii) the third one led from May 1998 to September 2001; (iv) the fourth one from September 2001 to May 2002; and (iv) the last one from mid-2002 to the closing of the loan in June 30, 2003.
The ICR considers some of the ratings assigned in the PSRs, especially the ones for 1995 and 1997, unrealistic. Satisfactory ratings were provided in implementation progress and developmental objectives in a context of no implementation, limited Loan disbursements and no available evidence of progress towards achieving the developmental objectives. The supervision missions were also characterized by a tacit agreement not to question seriously the continuous request of the Borrower to expand the list of executing agencies interested in participating in Component B. For example, the Dirección de Colegios de Bachilleres, one of the last executing agencies to be included in the tenth amendment to the LA, indicated a policy decision to introduce a modular competency-based curricular reform in the Scientific-Humanistic in the first four semesters of the upper secondary sub-system (Colegios de Bachilleres), but this never materialized. The Bank's no-objection to the huge expansion in scope during project implementation, void of any analytical or operational evidence to justify it, was, in the view of the ICR, a mistake.
Components A and B are rated highly satisfactory in the PSR of February 1998, despite: (i) the obvious operating inefficiency of CONOCER, displayed by a certification cost of about 5 times the average cost of
- 19 -
a 200-hour training module and a record that left about 50% of its approved norms dormant; and (ii) a lack of impact evaluation evidence concerning the job performance of graduates of the competency-based curriculum of Component B as compared with the graduates from the traditional curriculum. Counterpart funding was rated consistently satisfactory despite the significant delays in releasing the authorized budget for the project every year in a timely fashion. The first flag of the project being a problem project was raised in February 1998, more than 30 months after effectiveness.
Since mid-1998, project supervision was characterized by a serious effort to improve the efficiency and effectiveness of project implementation. Several actions were taken, many of them as a result of the findings of the mid-term review carried out on September 1998, such as: (i) the project was restructured to reflect a more viable implementation reality; (ii) a three-year extension was granted to the Loan closing date (to June 30, 2003); and (iii) talks were initiated to explore a follow-up operation (PMETyC II). Project files shows that for the first time in the implementation cycle of this project, the Aide-Memoires were comprehensive in detailing real achievements and pointing at the relevant issues in need of prompt attention by the Borrower and the Bank. Nevertheless, the ICR is of the view that it was a Bank mistake to agree to the cancellation in 2002 of some relevant impact evaluation studies included in the tenth amendment. This view is strongly influenced by the current Bank policy to be more output-oriented in its current and forthcoming investments, as opposed to the more input and process-oriented projects of the past. This latter orientation by the various Bank teams during supervision underlies the satisfactory ratings given to development objectives during most of the implementation period and, hence, the disconnect with the ICR rating of unsatisfactory. On this basis, the overall supervision performance of the Bank would have to be rated as unsatisfactory, as well.
Finally, with the objective to guide the preparation of a possible follow-up project, the Bank conducted a series of studies financed by a PHRD Grant (TF-027790). To agree on the terms of reference with the Borrower and to launch these studies, the Bank carried out two missions in July 1999 and 2001. A thorough report entitled Mexico: Technical Education and Training: Issues and Options, in two volumes, was prepared and financed by this PHRD Grant. The preliminary version was distributed for review in mid-2003. This valuable report provides the only evidence (though limited and not conclusive) of: (i) the job-performance of a sample of graduates of the competency-based curriculum implemented by CONALEP and the Technical Universities that participated in PMETyC; and (ii) the impact of components A and C, based on a survey of a sample of 600 workers in the sectors of forestry (in the State of Durango), garments (in the State of Hidalgo) and tourism (in the metropolitan area of Mexico City) and managers of 30 firms, of which only 8 (out of 30) responded to the survey.
Although the Report was based on an insufficient number of observations, its findings were valuable in the Bank’s decision-making process. Based on the findings, compounded with a lack of more solid evidence of the project’s impact that should have been provided by the Borrower, a project implementation history full of shortcomings, and little willingness demonstrated by the coordinating unit to address the limitations of the ongoing operation, the Bank decided that it could add little value in supporting a follow-up operation and chose instead to support other priority areas in the education sector in Mexico more consistent with the direction of the new CAS (Report No. 28141-ME, March 18, 2004) focusing more directly on poverty alleviation.
7.3 Overall Bank performance:
Based on the above observations, the ICR team rates the overall Bank performance as unsatisfactory.
Borrower
- 20 -
7.4 Preparation:
Lacking experience in undertaking a project of this nature and appropriate guidance, the Mexican Government conducted an extensive survey of ongoing worldwide competency standards experiences available in the early nineties including: (i) the United Kingdom (UK) Qualifications and Curriculum Authority (QCA), the UK Sector Skills Development Agency (SSDA) and the UK National Reference Point for Vocational Qualifications; (ii) the Scottish Qualification Authority (SQA); (iii) the Australian National Training Authority (ANTA), the Australian Qualification Framework (AQF) and the National Centre for Vocational Education Research (NCVER); (iv) the New Zealand Qualifications Authority (NZQA) and the Skills New Zealand Competency Agency; and (v) the Spanish Instituto Nacional de Cualificaciones (NCUAL), the Instituto Nacional del Empleo and the Fundación Tripartita para la Formación y el Empleo. The Mexican Government decided to adapt the UK/Scottish model for PMETyC, although neither the project files, nor the SAR, include the reasons as to why this option was chosen.
Even though the original sketching of PMETyC in early 1993 was championed by SEIT which saw it as an opportunity to invest significantly in enhancing the quality and relevance of upper secondary technical education as well as updating the laboratories and workshops of the public technical school system, once the relationship with the Bank was formalized in the identification mission in mid-1993, SEP's Undersecretariat for Educational Planning and Coordination (UEPC) took full control of the preparation, and eventually, of the implementation of this project. UEPC's view of the proposed investment, different to SEIT's, was to adapt the UK/Scottish competency-based model to the Mexican case, assuming that by converting the traditional vocational curriculum into a modular competency-based one, the quality, as well as the relevancy, of technical education would improve. This change of view diminished SEIT ownership and they were never effectively brought back into the operation.
Overall, however, the ICR team is of the view that many of the shortcomings that occurred during the lending process were generated more from the Bank's team inexperience, inappropriate decisions and loose monitoring, than from the Borrower, although they did have some internal conflicts as suggested above. Thus the ICR team ranks the Borrower's preparation performance as satisfactory.
7.5 Government implementation performance:
Several issues can be highlighted regarding the Government’s implementation performance. First, as mentioned elsewhere in the ICR report, every year of the implementation cycle, the timely release of the authorized project-related budget was significantly delayed, causing a shortfall in the achievement of physical and financial targets. These delays were accompanied several times by budget reductions and expenditure restrictions. Secondly, as mentioned in section 6.1 of the ICR report, the new administration decided to change the legal status and composition of CONOCER, to which the Bank strongly objected, although no actions on this were forthcoming during the operation of the project. It is worth mentioning that the Bank unsuccessfully tried several times to open a discussion with the Mexican Government on improving the functioning of CONOCER and its eventual legal changes. For all of the above-mentioned observations, the ICR team ranks the Government implementation performance as unsatisfactory.
7.6 Implementing Agency:
PMETyC had several implementing agencies. The ICR team ranks as satisfactory the implementation performance of CONOCER (the implementing agency for Component A) and of CONALEP (one of several executing agencies of Component B). In the former case, the ranking is based on the fact that
- 21 -
CONOCER has been improving its efficiency and targeting, and as explained in section 5.4 of this report, it has had a consistent record of a high expenditure rate of its authorized budget and release of funds. In the latter case, CONALEP was one of the few executing agencies that supported the expansion of the modular competency-based curriculum to the totality of its enrollment based on comprehensive tracer studies on the job-performance of its graduates. CONALEP also registered a high expenditure rate of its authorized budget and release of funds.
The implementation performance of other agencies involved in Component B, most of them included in the UTS subsystem coordinated by SEIT, is mixed, ranging from satisfactory to unsatisfactory. The satisfactory rating goes to those UTS schools that achieved a real articulation with the regional productive sectors that demanded competency-based training services from these schools for their employees. The unsatisfactory rating goes to those schools that participated in PMETyC disconnected from the regional labor market that were supposed to be involved with.
The implementation performance of STPS in charge of Component C is ranked as unsatisfactory, mainly because the expenditure capacity performance averaged 54% during the entire life of the project, which is very ineffective.
With respect to Component D, and the general coordination of PMETyC, PAU's performance during initial years of implementation is considered adequate, despite the slow implementation of PMETyC as a whole (mostly due to a very complex design that took significant amount of time to get off the ground). Subsequently, however, during the latter years of implementation its coordination efforts were unsuccessful and it took on additional activities outside the scope of the operation; moreover its inability to undertake needed tracer studies undermined the ability to reach empirically-based conclusions regarding the impact of the program, and the overall performance is rated as unsatisfactory.
Finally, an in-depth random procurement review was carried out in May 2003. This review included a random sample of 10% of all the contracts for goods, services and civil works undertaken in 2001 and 2002 from all the project executing entities, amounting to 73 contracts. This review concluded that the Borrower followed the Bank guidelines and procedures in a satisfactory manner. A comprehensive report, with some minor recommendation, was prepared for the Borrower and Bank management and filed in IRIS3.
7.7 Overall Borrower performance:
Even though several components and the procurement procedures were satisfactory, the consistent delays in releasing authorized project-related budget funds, the fact that close to 50% of the approved norms by CONOCER are dormant after project completion, the inefficient implementation displayed by and large in Component C, the lack of effective articulation of many UTS schools participating in PMETyC with their respective regional productive sector, and unsuccessful coordination efforts leads the ICR team to rank the Borrower’s overall performance as unsatisfactory.
8. Lessons Learned
With respect to the lending process
(a) Experienced preparation teams are likely to generate better designed projects. Unexperienced Bank and Borrower preparation teams defining an innovative project in uncharted territory involving two
- 22 -
Ministries (Education and Labor) led to a very complex design that took over three years after Loan effectiveness to get off the ground.
(b) Ensure the ownership of all participating stakeholders. PMETyC suffered from the lack of clear ownership by key stakeholders like SEIT (at the outset of project preparation), and STPS (with the new administration), significantly undermining project implementation effectiveness and efficiency.
(c) Align investment timing with the political calendar. Investments that cross over different sectors as well as different elected administrations run the risk of disenfranchisement by the incoming administration in some of the participating sectors. An adaptive lending program is a more appropriate Bank lending instrument in this case, ensuring that the benchmarking separating the different phases of the APL are aligned with the political timing.
With respect to the supervision process
(d) Willingness and ability to build up and learn from experience. Successful projects are built on the foundation of previous, smaller scale interventions, slowly scaled up by intensive examination of "what worked" and "what didn't work and why". This type of reflective and flexible approach was, by and large, resisted.
(e) Ensure effective supervision. Effective supervisions, looking at both the substantive parts of the project as well as the fiduciary and governance aspects of implementation, are key instruments to continually improve the efficiency and effectiveness of project implementation. They also contribute to building a constructive and solid professional relationship between the client and the Bank.
With respect to Component A (Establishing a national system of competency standards and of a skills testing and certification system - CONOCER)
(f) Demand-driven and targeted approaches increase efficiency. The early stages of CONOCER (up to and including 2001) where highly inefficient and costly as a result of its supply-driven and untargeted approach, including too many sectors of the economy and five levels of occupational competency. As a consequence, 47% of the approved norms in the life of CONOCER remained dormant. Once CONOCER responded better to the demands from the employers and targeted its efforts to the lower two levels of occupational competency (qualified worker and technician), the efficiency and ownership of stakeholders greatly improved.
(g) Simplify processes to increase efficiency. The efforts of CONOCER to simplify and streamline the entire rather complex certification process (from the definition and validation of the norm to certification, including evaluation and training), are paying dividends in terms of increased efficiency, especially concerning the guidelines for the gathering of skill-evidence portfolios. This recommended simplification will also address the issue of the out-of-sync rhythms between a rather lengthy process including the definition and approval of a given sector-specific labor competency-based norm, and the requirements of the firms.
(h) Training should rank higher in priority than certification. The average unit cost of certification by CONOCER largely exceeds the average unit cost of a 200-hour training course in México, indicating that the current priority granted by CONOCER to certification over training needs to be reversed in order to make this process more efficient.
- 23 -
(i) Decentralize the certification of acquired skills. The average certification cost will decrease as a function of decentralizing the 32 licensed certification units, currently all located in Mexico City.
(j) Strengthen alliances with the private sector to increase CONOCER's stakeholder ownership. CONOCER needs to strengthen further its alliances with private sector employers to ensure a change of culture among these employers regarding the need to train and certify their working force, with the goal of gradually transferring the currently public subsidized cost of this process to private and public employers.
(k) Adapt international norms when possible. Adapting international norms to the Mexican context (when possible and applicable) instead of reinventing them will decrease costs.
With respect to Component B (Modernizing training programs to increase their flexibility and relevancy) and Component C (Stimulating demand for competency-based training and certification to promote private sector initiative and participation in training design and implementation)
(l) Certification is the bottleneck. As long as the issue of who pays for the certification of students enrolled in the competency-based curriculum and/or workers being trained, is not addressed satisfactorily, the number of certified students and/or trained workers will be extremely small due to the relatively high cost of certification and the complexities inherent in assembling the evidence portfolio of acquired skills.
(m) Effective articulation between the education and employers is vital. The competency-based scheme seems to work more effectively where there exists a clear articulation between the UTS schools and the regional productive sectors, especially the private ones.
(n) Tracer studies are necessary. These kinds of studies are necessary to ascertain the advantages and disadvantages of curricular innovations and to fine-tune project design and implementation strategies.
(o) Use the "Cooperative Legal Figure" as a way to expand competency-based training in UTS schools. The supply of competency-based training provided by those UTS schools where the articulation with the regional productive sector is working well could be greatly increased if these schools adopt a "Cooperative" legal and fiscal arrangement. This arrangement allows the schools to increase their revenue by providing invoices for their services, which can then be used by the employer for tax-rebate purposes. This additional revenue could be allocated by the school to: (i) acquire more equipment and adapt the physical spaces required; (ii) provide further training to their teachers; and (iii) incorporate new norms in their competency-based curriculum.
With respect to Component D (Project Administration)
(p) Inclusive and participative management which creates a conducive environment to strengthen stakeholder ownership and thus increases the likelihood for effective implementation, was largely lacking during critical years of implementation.
9. Partner Comments
(a) Borrower/implementing agency:
The Borrower provided the Bank on April 30, 2004, the Final Report of the project for the period 1995-2003. This report has been filed in IRIS.
- 24 -
Additionally, the following comments to the ICR report were received from the Borrower in September 2004:
CONTRIBUTION OF THE BORROWER
This note has as a purpose the following three points:
1. Express our disagreement with regard to the methodology utilized by staff in the preparation of the ICR.
2. Suggest that the project was successful when measured against what was agreed upon contractually with the Bank.
3. Identify the coincidence points with the Bank and of learned lessons.
Disagreement in the methodology utilized by the Staff of the Bank.
i) The document confuses the ratings that the Bank itself grants to the team that was responsible for the design and evaluation of the Project with the performance of the Borrower. The report points out that its preparation team did not adhere to the standards that are now current policies, but that did not exist at the time of the preparation and appraisal This situation results that in the whole ICR document there is an erroneous criteria of evaluation that respond to "Results of the project against what should have been" and not precisely to "Results of the project against what was designed and agreed upon with the Bank." The foregoing means that if the action of the Project was assessed in accordance with the provisions in the signed documents, the physical goals reached and observed by the different missions of supervision through the indicators agreed upon, surpass those established in the legal documents. The latter is corroborated with the aide memoire of the supervision mission of June 2003.
ii) The methodological perspective of the staff that prepared the ICR shows an ignorance of the origins of the project as documented at the time of appraisal. The Borrower in accordance with the Mexico Country Assistance Strategy (Report No. 13008-ME of 13 May 1994), tried to improve the quality (relevance) of the technical education and training with an approach to reform "from outside" and not "from inside" because it was conscious that the strong inertia of some educational executors would have impeded the good progress of the project. Initial reactions of nonacceptance of the project were expected. An example of this was the opposition of those executors to consider the certification of the competencies by agencies of third party.
ii) In addition, the Borrower’s team also wants to clarify that in the missions led by the last task manager of the Project, short field visits were carried out in only a small number of schools of Component B, which in our judgment suggest that it would be difficult to find evidence or solid elements for the evaluation and preparation of this ICR. This is also indicated by the Bank itself when he points out in the section 4.1 of the report that "The mission of June 2003 could not obtain sufficient analytical evidence in order to evaluate the impact of the project in the achievement of its development objectives, ….". The Borrower’s team considers that a larger sample, with sufficient time to identify evidence and with a plan of consistent research with the objectives of the component and with greater objectivity by the leader of the mission, would have
- 25 -
generated different results in the evaluation than those reported by the Bank staff in this ICR.
Results of the Project with regard to the commitments and goals considered in the legal documents agreed with the Bank.
i) It is necessary to point out before any other observation in this section, that the assertion by the Bank in the section 4.1 of the ICR is not correct, in the sense that there were questions of performance and impact not addressed by the Borrower in various studies required to be carried out under Component D before the tenth and last amendment. Specifically in the mid-term review we pointed out that all the studies were carried out with terms of reference approved by the Bank. We received no comments from the Bank disagreeing with these studies. As regards the assertion by the ICR that the Mission of June 2003 did not obtain the sufficient analytical information in order to evaluate the impact of the Project, it is important to mention that we agreed with the Bank that the June’s mission was of supervision, and that the closing mission would be carried out in September of that year, in which they would have the elements required for the evaluation; however the Bank decided not to have this mission.
ii) The staff confuses the scope of the Project. The nature of any educational project that means a change of paradigm in the construction as this one, has an impact that can only be observed in the long term. The observations that the ICR expresses in section 4.1 of the document are not pertinent; it shows on the other hand, an ignorance of the characteristics of the PMETyC and confuses it with traditional schemes for technical education and training. iii) The staff of the Bank does not take into account the innovative character of the PMETyC, nor the form of intervention of the government for its application. It does not take into account the scale of the participating educational institutions of the Project, nor characteristics of the design of the technical process. From the start, the PMETyC was an effort to reform technical education and training, and its construction implied a progressive process of improvement of the learning curve and its results should have been seen with regard to the experiences of the pilot studies and their demonstrative effect on the rest of the educational system and acceptance by the national productive apparatus. It is not possible to argue, as the Bank proposes, that after four years of operation of the project, (four more were occupied for the methodological construction, institutional, development of markets, among other aspects), significant changes can be observed in the quality of the educational offer and impact on the productivity of the certified workers. The proof of impact is given in the long run when the intervention of the project has managed to modify the social behavior and achieve the desired scenarios with sustainability. An example is that in the United Kingdom that only after more than 15 years of its application and in favorable cultural contexts in the application of the evaluation and certification by third parties, the results of impact on the application of the NVQs can be observed. The governmental intervention in the application of the PMETyC is based in actions that are sustained by the acceptance of the market and not by legal mandate, as in other countries such as Spain and Australia. The design of the PMETyC makes it unique for its design and construction; however its results have permitted an ever-growing use of its instruments. That is the case of institutions of the civil service, of national companies, public, and private universities. The PMETyC has also been a model for other Latin America countries and for Spain. The available examples are numerous, but apparently the ICR
- 26 -
has not considered them, despite the information given to the team members during the last missions.
iv) In consequence, we are in agreement with the ICR as shown in the final paragraph of section 4.1 where the project implemented successfully some originally established tasks, but the impact with regard to the established objectives of the project are difficult to assess; however the met targets surpassed the ones considered in the legal documents.
Points of coincidence with regard to the this report lessons learned.
· Improvement in the project design and in all the components is required, but mainly the application of interesting experiences of other countries and the exploration of national successful experiences has to be collected with better effectiveness.
· To strengthen the coordination entities within the components and among them is required, with transparent and consensus functions among the executors, and with better organized intervention of the government, mainly in the times of authorization of the budget and its execution.
· The sustainability in all the components is not homogeneous. Ups and downs are seen in the operation of some executors, which means that a greater commitment is required, particularly of component B.
· Changing the traditional vocational curriculum to a competency-based one does not constitute, by itself, a strategy to improve the quality of the learning. A more organized intervention complemented with other management quality tools such as ISO 9000 is required. This would also help minimize the perverse effects that occur when changes of the people responsible of the institutions are given. A greater uniformity in the design of the educational contents in regard to the standards of competition. is required.
· Improve the supervision missions, integrate it with skilled personnel who can contribute to a better performance of the project, and stay longer.
· Strengthen the alliances with the private sector to increase the demand for training, based on specific agreements of collaboration.
· The follow-up studies are necessary. This type of studies are necessary in order to determine the advantages and curriculum innovation disadvantages in order to refine strategies of design and implementation of the project.
· The enforcement of the model of competencies in the educational executors should be based on institutional reforms authorized by its respective authorities.
· The quality of the certifying agencies, of evaluation centers, of the eligibility and action of the worker trainers, should be strengthened.
- 27 -
· To promote the participation of the universities of the country both in the use of the competencies to reform its offers, as well as the service providers of research and needed studies for the project is required.
Target: Complete 310 qualification of competency standards
613 (46.6% of them are dormant and were never utilized)
Target: Create 24 Certification Bodies Recognized by the Private Sector
32
Output Indicators:
Indicator/Matrix
Projected in last PSR1
Actual/Latest Estimate
Number of certification units provided 126,168 195,844 (equivalent to 87,442 certificates provided or 67,438 individual fully certified an approved norm by CONOCER)
Number of courses in UTS schools converted to the modular competency-based curriculum
585 1,096
Number of teachers trained in competency-based curriculum in UTS schools
26,983 31,380
UTS schools participating in the project that received equipment
408 392*
Number of students in UTS schools enrolled in competency-based curriculum courses during the life of the project.
644,551 754,424
Number of UTS schools accredited by CONOCER as CEAs
412 382
Training fellowships provided by Component C
62,493 80,600
Certificates provided to unemployed or sub-employed trained workers by Component C
22,575 34,077
Pilot cases 54 54
Workers, mostly from SMEs, trained under Component C
42,757 66,832
SINOE Management Information System Operational
CONOCER's Management Information System
Operational
- 29 -
SISPAE Management Information System Operational
SAPAC Management Information System Operational
Occupational Catalogue Ready
Studies 10 44 **1 End of project
* The equipment provided by PMETyC for the participating UTS school relates only to the approved competency-based course (equipment was not provided for the entire UTS school).** These studies do not include the important and required impact evaluation studies mentioned in Part D2 of Schedule 2 to the Legal Agreement, before it was amended in September 2002.
- 30 -
Annex 2. Project Costs and Financing
Project Cost by Component (in US$ million equivalent)AppraisalEstimate
Actual/Latest Estimate
Percentage of Appraisal
Component US$ million US$ millionA. National System of Competency Standards, Skills Testing and Certification.
31.10 72.65 233.6
B. Modernization of Training Programs 141.70 115.53 81.53
C. Stimulation of Demand for Competency-Based Training and Certification
190.00 46.30 24.37
D. Information Systems, Studies and Project Administration D1. Information Systems 3.10 2.10 67.74 D2. Studies 8.50 3.86 45.41 D3. Project Administration 5.70 18.70 328.07
4. Traning Materials and printing and reproduction of training modules
13.00 14.40 38.40 0.00 65.80
(3.29) (3.71) (0.00) (6.29) (13.29)5. Stipends and Incentives 0.00
(0.00)0.00
(0.00)118.90(18.65)
0.00(0.00)
118.90(18.65)
6. Incremental Operating Costs
0.00(0.00)
0.00(0.00)
40.70(81.72)
0.00(0.00)
40.70(81.72)
Total 77.20 46.20 288.30 0.30 412.00(56.12) (16.26) (160.90) (6.29) (239.57)
For conversion purposes, an average rate of $8.72 mexican pesos to one US dolar was considered for the entireproject execution cycle (1995-2003).
1/ Figures in parenthesis are the amounts to be financed by the Bank Loan. All costs include contingencies.2/ Includes civil works and goods to be procured through national shopping, consulting services, services of contracted staff
of the project management office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units.
Project Financing by Component (in US$ million equivalent)
Component Appraisal Estimate Actual/Latest EstimatePercentage of Appraisal
Bank Govt. CoF. Bank Govt. CoF. Bank Govt. CoF.A. National System of Competency Standards, Skills Testing and Certification.
17.80 15.30 34.88 37.77 196.0 246.9
B. Modernization of Training Programs
115.00 37.80 84.39 31.14 73.4 82.4
C. Stimulation of Demand 118.40 60.90 28.70 30.34 15.96 25.6 26.2 0.0
- 32 -
for Competency-Based Training and CertificationD. Information Systems, Studies and Project Administration D1. Infromation Systems 2.80 0.50 1.84 0.26 65.7 52.0D2. Studies 7.50 1.20 3.39 0.47 45.2 39.2D3. Project Administration 3.50 2.60 10.01 8.69 286.0 334.2Total Project Cost 265.00 118.30 28.70 164.85 94.29 62.2 79.7 0.0
- 33 -
Annex 3. Economic Costs and Benefits
As mentioned elsewhere in the ICR, no economic cost-benefit analysis was carried out at the time of project appraisal. Furthermore, the Borrower carried out no impact evaluation studies after the mid-term review, as originally intended in the design of Component D (though the tenth and last amendment later relaxed compliance of this covenant to the LA). Therefore, no economic cost-benefit analysis was done at the time of the ICR either.
Nevertheless, in recent years the Bank undertook several related studies on the Mexican, Malaysia, Colombia and Taiwan (China) Technical Education sub-sectors, whose preliminary findings allow for some thought on the future challenges facing Mexican society in its attempt to modernize technical education. These studies, five of which are unpublished and two published, include: (i) Cost-Benefit Analysis of Technical Secondary Education in Mexico, by Gladys Lopez Acevedo, The World Bank, June 2002; (ii) Mexico Technical Education and Training: Issues and Options. Volumes I and II, by Indermit Gill and Anna Sant'Anna, The World Bank, May 31, 2003; (iii) Mexico: Training Assessment Study. The World Bank, 1997; (iv) Mexico: Enhancing Factor Productivity Growth. Country Economic Memorandum, The World Bank 1998; (v) Mexico: Technology, Wages and Employment, Report No. 22797-ME, The World Bank 2001; (vi) Enterprise, Training, Technology and Productivity in Malaysia. The World Bank Country Study, 1997; and (vii) Technology and Firm Size-Wage Differentials in Colombia, Mexico and Taiwan (China), by Hong Tan and Greeta Batra, The World Bank Economic Review, Vol 11, No. 1, pp.59-84, 1997.
The summary of the main finding of these studies is that effective and pertinent technical training is Mexico’s primary tool for reaching equilibrium in the market for skilled labor and increasing productivity.
According to Lopez Acevedo's study, secondary technical education may be an alternative for those individuals that face both high opportunity costs for continuing formal education and also need to acquire skills that enable them to participate in the job market. The fourth study mentioned above (World Bank 1998) showed that technical education after completing lower secondary school level increased earnings on average by 32%. The findings of the fifth, sixth and seventh studies suggest that investments in human capital, especially in training, in conjunction with technology adoption, increases productivity benefits. These findings also suggest that wage premiums are primarily the result of investment in research and development and in training.
However, the accomplishments of the technical secondary education system in Mexico are low in keeping the students in school and attaining appropriate skills needed in the labor market when they graduate. According to data provided by SEP, in the 2001-02 school year, the terminal efficiency rate at technical secondary school was only 51.7% as compared to 86.5% for the humanistic-scientific secondary subsystem. Dropout and repetition rates are unacceptably high, 23.1% and 21% respectively. Evidence on the low educational achievement in technical education is drawn from SEP's Council of the National System of Technological Education (COSNET) showing that, by and large, vocational and technical systems are not providing new entrants with appropriate skills.
Lopez Acevedo's study also provides a preliminary cost-benefit analysis of completing technical secondary education. The cost side of the equation includes investment (infrastructure and equipment) and operation (salaries of teachers and administrators, security services and utilities) costs. The benefit side includes annual average income for three types of sets of students, those with: (i) completed lower secondary but incomplete upper secondary technical education; (ii) completed lower secondary and upper secondary
- 34 -
technical education; and (iii) completed scientific-humanistic upper secondary education. The cost data refers to the three-year program, while the benefits were drawn from a Mincerian regression, based on the Mexican National Urban Employment Survey, controlling for self-selection. The annual costs and benefits are shown in the tables below.
Fixed and variable costs (CONALEP sub-system)
Total Costs Current 2001 pesosTotal costs 4 259 636.59Variable costs 71 723.25Fixed costs 4 331 359.84Number of students 1025Average CostsTotal average cost 4 225.72Variable average cost 4 155.54Fixed average costs 69.97
Source: Campos, Marcos. 2001. Estudio de Costos de los Ejercicios 1992 y 1994, Plantel CONALEP “Gustavo A. Madero
Benefits of secondary technical education by type of studentEducation Level Annual Average Income
(2001 pesos)Secondary plus incomplete technical education 30,030.6Secondary plus complete technical education 42,334.4Complete upper-secondary 38,936.6
Marginal Benefit of Complete Technical Secondary Education
12,304.8
Source: Own estimations based on the ENEU.
Assuming a discount rate of 12.5%, real earning differences remain constant in the years that follow, and the probability of finding a job is one year after graduating, the “break-even” year for complete technical secondary education, discounted present value of accumulated benefits equals economic costs, is 3 years after graduation.
Finally Lopez Acevedo's study also estimated the potential economic and fiscal impact of technical secondary education in Mexico based on a dynamic model that takes into account the flows and stock of human capital in technical secondary school and net benefit of completing technical secondary level. The preliminary findings show that as a result of a 10% decrease in dropout rates, and assuming that all graduates complete technical education in three years, this decrease in dropout rates translates into additional net benefits of close to $120 million Mexican pesos in 2001 and into additional fiscal resources of close to $36 million pesos in 2001.
One of the main conclusions of Lopez Acevedo's study is that increasing the quality and efficiency of general education levels is a necessary condition to increase the net benefits and decrease the costs, thus making the cost-benefit relationship even more attractive. As confirmed by the Gill and Sant'Anna's study (below): (i) employers’ decisions to train and the productivity outcomes of training depend on the stock of education and technical skills that individuals bring to the labor market; and (ii) it is more cost-effective to impart training to workers who are adept at learning (meaning, they have graduated from a relevant and high quality general education system).
- 35 -
G D P P e r W o r k e r , c o n s t a n t p e s o s
According to Gill and Sant'Anna study, informality and low productivity characterize the labor force of 40 million out of a prime-aged population of about 60 million in Mexico. This labor force has grown at an average rate of 3% between 1980-1999, and between 40 and 60% (depending on the measure used) are in the informal sector of the economy. These developments are consistent with low labor productivity growth. Indeed, productivity has stagnated in Mexico since the mid-1980s (see figure above), making only a slow recovery since 1995. This may be one of the most serious problems afflicting the Mexican economy. Possible culprits for stagnating labor productivity may be low general education levels (as evidenced in the Lopez Acevedo's study), poor use of technology, and/or poor technical education and training policies. Unfortunately, no tracer studies were undertaken by PMETyC to ascertain these hypotheses and assess the impact of graduating from UTS schools with a modular competency-based curriculum on job performance, wages and productivity as compared with graduating from the traditional vocational-technical education in UTS schools.
Mexico has made progress both in improving education and, more recently, in improving the opportunities for technology absorption. The average education level in Mexico for the 15-year-old or over population is now close to 7.6 years (still low compared with over 12 years in most of the OECD countries, over 9 years in Argentina and Chile, and over 10 years in Cuba). Mexico’s progress can be gauged by benchmarking it against South Korea, one of the most successful developers during the last two decades. Mean education levels in Korea tripled from 3.2 to 10.5 years between 1960 and 2000; mean education levels in Mexico also almost tripled, from 2.4 to 7.6 years during the same period.
But while long-term productivity growth in Korea has been high — averaging more than 1.5 percent annually between 1960 and 2000 — Mexico has struggled to raise productivity. Over the last four decades, annual productivity growth has been about 0.3 percent, and estimates indicate that this rate of increase dropped to 0.1 percent in the last decade. The data show that productivity growth has increased considerably since 1995, after Mexico joined the NAFTA, while productivity growth between 1990 and 1994 was less than 0.5 percent, it quadrupled to almost 2 percent in the late 1990s. Labor productivity has also risen since 1996, but more slowly. These developments provide some grounds for optimism, but also hint at other factors constraining labor productivity increases.
Mexico’s public spending on education as a share of GDP is about 5%, somewhat higher than that of Korea. But when private expenses are included, Korea spends almost 13.5% of GDP on education, almost certainly one of the highest percentages in the world. For training systems, the story is similar.
- 36 -
Government dominates Mexico’s provision of secondary and post-secondary vocational and technical education; in Korea, it is mainly private. While both countries have taken measures to encourage in-service training, Korea has relied more on policy instruments (levy-grants, mandates, tax incentives) to encourage training in private firms, while Mexico has, until recently, relied more on public provision.
Mexico may want to pay special attention to institutional reform to strengthen the links between industry and UTS schools, universities, research institutes and training providers. Technical secondary schools in Mexico were originally designed to complement the general secondary education with some vocational training, as done in some OECD countries like Spain (Ley Orgánica del Sistema Educativo) and the Netherlands. In practice, many of these technical secondary schools, far from their envisaged objective, have served to stream students into vocational-technical versus humanistic-scientific education as early as the seventh grade (first grade of the lower secondary education). In these cases, Mexico may want to reconsider postponing vocational education to post-secondary levels (after grade 9), as done by other more developed countries, as a more appropriate strategy, given that rapid technical changes necessitate a stronger general education base. Following this recommendation implies that the Mexican government consolidates the current lower secondary vocational education structure into a more universal basic education.
Gill and Sant'Anna's study also shows that, in contrast to vocational education (training for a job) and technical education, in-service training (training on the job) is mostly private (75% in the workplace or in private institutions). In-service training varies significantly in Mexico by firm and worker characteristics. Larger firms, firms with high intensity of research and development efforts, and firms in export and technology-sensitive sectors are more likely to provide in-service training. The bulk of firms, mostly SME, do not provide any kind of training, by and large, to their employees. Workers with higher education levels are much more likely to receive training than their less educated counterparts, and the age profile of trained workers exhibits an inverted “U”-shape pattern, cresting at ages 35-39. Training incidence does not change much in good economic years (1998-99) and bad (1994-95) years. Good training can be considered an effective link between education and technology, and a proven way to achieve a needed increase in productivity, especially in SME firms, which integrate the majority of the labor force.
CONOCER offers a way to strengthen the qualification systems by using a system of standards and assessments for vocational education and training. These standards can play various useful roles: (i) by improving the efficiency of public provision; (ii) through regulation of private providers of training; and (iii) by improving labor market information. It need not be only CONOCER which provides the framework for applying standards since, as mentioned elsewhere in the ICR, that has proven to be costly and to a certain degree inefficient and ineffective, but a national system could be developed which was flexible enough to incorporate, in addition to CONOCER, complementary institutional standards.
Finally, Gill and Sant'Anna study also advocates: (i) facilitating private sector participation in both institution-based and industry-based education and training as the public sector alone cannot provide all the resources needed in the different sectors of education and training; (ii) promoting the adoption of new technologies through improved access to them and stronger incentives to adopt them as an effective approach to training; (iii) promoting more effective training policies, especially to target those firms (SMEs) which are out of the training circuit and to continue programs to train poor, young jobseekers, especially programs that have employer participation; (iv) improving the timely dissemination and the pertinence of information concerning the medium and long-term benefits of training; and (v) reconsidering the current legal mandate to train (with which most of the Mexican firms do not comply), and replacing it with incentives to innovate and formalize training, especially among SME firms.
- 37 -
Annex 4. Bank Inputs
(a) Missions:Stage of Project Cycle Performance Rating No. of Persons and Specialty
Identification/Preparation10/1993 6 One program management
specialist, two training specialists, one educator and two economist
12/1993 6 Two economists, two training specialists and two educators
03/1993 5 Two economists, two training specialists and one educator
05/1994 7 Two economists, two training specialists, one educator, one procurement specialist and one operations analyst
Appraisal/Negotiation06/1994 8 Two economists, one
educator, one training specialist, one MIS specialist, one procurement specialist, two operations analysts
09/19/1994 5 Two economists, one operations analyst, one disbursement officer and one legal counsel
Supervision11/1994 1 One economist S S12/08/1995 4 Two program management
specialists, one training specialist, and one economist.
S S
01/31/1997 7 one MIS specialist, one procurement specialist, two economists, one training specialist; one information specialist; and one gender specialist
S S
02/20/1998 4 One education specialist, one training specialist, one conomist, one training specialist, and one MIS specialist
S S
09/14/1998 4 A task team leader, one economist, one accountant, and one educator.
S S
- 38 -
01/29/1999 5 A task team leader, two economists, one accountant, one Sr. educator.
S S
03/15/1999 2 A task team leader, and one economist.
S S
04/07/2000 1 A task team leader, one economist, and one educator consultant.
S U
11/30/2000 6 A task team leader, one financial management specialist, one MIS specialist, two economists, and one team assistant.
S S
08/24/2001 1 Task Manager U S
03/18/2002 4 A task team leader, one social sector specialist, one education specialist, and one task assistant.
S S
09/30/2002 3 A task team leader, one technical educator, and a social sector specialist
S S
06/09/2003 5 A task team leader and education specialist, project implementation specialist, social sector specialist, procurement specialist and team assistant
S S
ICRThe Borrower did not agree to convert the June 2003 as an ICR mission. Despite the Borrower's objection, the Bank team utilized the information collected in the June 2003 mission to prepare the ICR. No formal ICR mission took place afterwards.
(b) Staff:
Stage of Project Cycle Actual/Latest EstimateNo. Staff weeks US$ ('000)
(*) Estimated based on data provided by Project files, BW, Fact and SAP. Appraisal and Negotiations information included under Preparation
- 40 -
Annex 5. Ratings for Achievement of Objectives/Outputs of Components(H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable)
RatingMacro policies H SU M N NASector Policies H SU M N NAPhysical H SU M N NAFinancial H SU M N NAInstitutional Development H SU M N NAEnvironmental H SU M N NA
SocialPoverty Reduction H SU M N NAGender H SU M N NAOther (Please specify) H SU M N NA
Private sector development H SU M N NAPublic sector management H SU M N NAOther (Please specify) H SU M N NA
Lending HS S U HUSupervision HS S U HUOverall HS S U HU
6.2 Borrower performance Rating
Preparation HS S U HUGovernment implementation performance HS S U HUImplementation agency performance HS S U HUOverall HS S U HU
The ratings allowed by the ICR format are too general and do not allow for qualification (i.e. between satisfactory and marginally satisfactory). See Section 7 for further qualification explanation of the above ratings.
- 42 -
Annex 7. List of Supporting Documents
Bank preparation documents
Country Assistance Strategy (CAS), Document 13008-ME, May 13, 1994.
Initial Executive Project Summary. The World Bank. November 10, 1993
Pre-appraisal Mission Terms of Reference, Aide-Memoire, and Back-to-Office Report. The World Bank. May 16, 1994.
Appraisal Mission Terms of Reference, Aide-Memoire, and Back-to-Office Report. The World Bank. August 27, 1994
Staff Appraisal Report (SAR), Report No. 13416-ME. The World Bank. October 5, 1994.
Agreed Minutes of Negotiations. The World Bank. September 19, 1994.
Bank project implementation documents
Loan Agreement 3805-ME. The World Bank. Conformed copy. February 7, 1995.
Guarantee Agreement 3805-ME. The World Bank. Conformed copy. February 7, 1995.
Amendments to the Loan and Guarantee Agreements. First amendment of July 14, 1995 (first partial Loan cancelation). Second amendment of October 9, 1997. Third amendment of February 27, 1998. Fourth amendment of July 23, 1998. Fifth amendment of May 24, 1999. Sixth amendment of October 22, 1999. Seventh amendment of March 26, 2001. Eight amendment of November 7, 2001 (second partial Loan cancelation). Ninth amendment of August 1, 2002 (third partial Loan cancelation). Tenth amendment of September 4, 2002 countersigned by the Borrower (NAFIN) on September 5, 2002 and the Guarantor (SHCP) on June 10, 2003 and ratified by the Mexican Federal Treasury on October 29, 2003.
Statement of Mission Objectives, Aide-Memoires, Back-to-Office Reports, Management Letters and Project Status Reports of all the Supervision Missions. The World Bank. December 1995- June 2003.
Borrower project implementation documents
Análisis Presupuestal del PMETyC. Unidad Administradora del PMETyC. México. Junio 2003.
Aportes para la Preparación del Informe de Cierre del Proyecto para la Modernización de la Educación Técnica y Capacitación (Documento de Trabajo). Subsecretaría de Educación e Investigación Tecnológicas. SEP. Junio 2003.
Diagnóstico del PMETyC a Octubre 2002. Unidad Administradora del PMETyC. México. Junio 2003.
Informe de Cierre del PMETyC Respecto del Componente C (Estímulos a la Demanda de Capacitación y Certificación). Coordinación Técnica de Proyectos con Organismos Financieros Intermacionales. Secretaría del Trabajo y Previsión Social. Noviembre 2003.
- 43 -
Informe de la Coordinación General de Educación Basada en Competencias. Subsecretaría de Educación e Investigación Tecnológicas. SEP, Marzo 2003.
Memoria 1995-2002 (Documento preliminar) del Proyecto para la Modernización de la Educacion Técnica y la Capacitación (PMETyC). Unidad Administradora del PMETyC. México. Junio 2003.
Sistema Nacional de Educación Tecnológica 2000-2001 en Cifras. Subsecretaría de Educación e Investigación Tecnológicas. SEP (ISBN: 970-18-7419-6). Noviembre 2001.
Bank project-related documents
Cost-Benefit Analysis of Technical Secondary Education in Mexico, by Gladys Lopez Acevedo, The World Bank, June 2002.
Enterprise, Training, Technology and Productivity in Malaysia. The World Bank Country Study, 1997.
FY00 Quality of Supervision Assessment of the Mexico: Technical and Education Training Project. Quality Assurance Group. The World Bank. September 18, 2000.
Mexico: Training Assessment Study. The World Bank, 1997.
Mexico: Enhancing Factor Productivity Growth. Country Economic Memorandum, The World Bank 1998.
Mexico: Technology, Wages and Employment, Report No. 22797-ME, The World Bank 2001.
Mexico Technical Education and Training: Issues and Options. Volumes I and II, by Indermit Gill and Anna Sant'Anna, The World Bank, May 31, 2003.
Technology and Firm Size-Wage Differentials in Colombia, Mexico and Taiwan (China), by Hong Tan and Greeta Batra, The World Bank Economic Review, Vol 11, No. 1, pp.59-84, 1997.
![Environmental Assessment Report€¦ · Project Number: 30232 January 2008 NEP: Decentralized Rural Infrastructure and Livelihoods Project Prepared by [Author(s)] [Firm] [City, Country]](https://static.documents.pub/doc/80x56/5f9cebe95bc9790546210969/environmental-assessment-report-project-number-30232-january-2008-nep-decentralized.jpg)
