I PID

THE WORLD BANK GROUP AWorld Froe of PovorSy

$infoShopIlb. Wxffig lWak

Report No AB37

Updated Project Information Document (PID)

Project Name PAKISTAN-National Education Assessment System

Region South Asia Regional Office

Sector Primary education (100%)

Project ID P077288Supplemental ProjectBorrower(s) GOVERNMENT OF PAKISTAN

Implementing Agency CURRICULUM WING,MINISTRY OF EDU.,ISLAMABAD AND

Address PROV.DEPTSAddress Islamabad, Islamic Republic of PakistanContact Person Mr. Arif Majeed, Deputy Educational Adviser, Curriculum Wing,MoE, Islamabad.Tel 92 51 4448343 Fax: 92 51 9290141 Email

Bureau of Curriculum of Balochistan, Directorate of Curriculum and TeacherEducation, Nort West Frontier Province; University of Education, Punjab,Directorate of Research and Assessment, Sindh; Directorate of CurriculumResearch and Development in Azad Jammu and Kashmir, and Directorates ofEducation in Federally Administered Tribal Areas, Islamabad Capital Territory,and Northern Areas.Address: (1) Balochistan: Quetta, (2) North West Frontier Province Abbottabad,(3) Punjab. Lahore, (4) Sindh. Jamshoro; (5) Azad Jammu and KashmirMuzaffarabad, (6) Federally Administered Tribal Areas Peshawar, (7)Islamabad Capital Territory Islamabad; (8) Northern Areas Gilgit

Contact Person.Tel: Fax Email

Environment Category CDate PID Prepared May 8, 2003Auth Appr/Negs Date April 17, 2002

Bank Approval Date June 3, 2003

1. Country and Sector BackgroundAs a signatory to key international Declarations and Frameworks (Jomtien, 1990; Dakar, 2000), the

Government of Pakistan (GoP) is committed to improving the quality of education at all levels, and especially

at the elementary stage. The Jomtien Declaration underlines the need for ensuring that: "The focus of basic

education must be on actual learning acquisition and outcome, rather than exclusively upon enrolment"

(UNESCO, 1990). In the EFA Assessment 2000 document, student achievement scores are included as

Indicator No. 1 5.

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

2PID

The government's education policies (1992, 1998-2010) and the recently formulated Education SectorReforms (ESR) Action Plan (2001-05) recognize the interdependence of quality and quantity in education asan important contributory factor in national development. The National Education Assessment System(NEAS) is one of the key programs of the ESR. The Curriculum Wing (CW) in the Ministry of Education hasa legal mandate for the "maintenance of standards of education". Although the Ministry of Education hastesting mechanisms for assessing individual student performance used as the basis for determining promotionand placement, it has no system of measuring how well schools are doing in implementing the curricula andlearning objectives associated with them. Information about student achievement from traditional schoolexaminations, or from the external examinations conducted by the Boards of Intermediate and SecondaryEducation, is not comparable across the country because the marks are derived from different tests orexaminations. Consequently, the Ministry has no institutional mechanisms for assessing the quality of itseducation service delivery.

2. ObjectivesThe development objectives of the National Education Assessment System (NEAS) are: to design andadminister assessment mechanisms; to establish administrative infrastructure and capacity for assessmentadministration, analysis and report writing; and to increase stakeholder knowledge and acceptance ofassessment objectives and procedures.

Under those broad objectives, specific objectives of the assessment system to be piloted include developmentof an empirical understanding of the following:

* how well the curricula are being translated into knowledge and skills among students;* subject, geographic and gender-related areas of inequity in student performance;* the principal determinants of student performance including teacher, student background and

other causal variables;* how resource allocation (fiscal, material and human) might be redirected to improve student

performance;* how to assist teachers to use data to improve student performance; and* how to create a sustainable capacity to conduct quality surveys of student learning and to apply

findings to policy formation.

The NEAS is planned as a sample-based national assessment, to be conducted at grade 4 and grade 8 levels, infour subjects - Language, Mathematics, Science and Social Studies/lslamiyat/Life Skills. This project willprovide the basis for designing and pilot testing the administrative process, the measurement instruments andthe information dissemination strategy associated with assessment of learning outcomes. In addition, thisproject should enable the Government of Pakistan (GoP), Provinces/Areas to make assessment a permanentfeature of the education system in the country.

3. Rationale for Bank's InvolvementThis is a first project of its kind in the country that aims at establishing an institutional arrangement fornational education assessment. The government values the Bank's work on assessment. The Bank helpedgovernment initiate a process of awareness creation in 1997 and mobilized other donor partners to help thegovernment. This process has led to a realization among various stakeholders that establishment of a nationalassessment system is a major step towards improving the quality of education.

3PID

4. DescriptionBy its very design, this project is experimental/developmental in nature. Because of the specialized skills andthe paucity of them in Pakistan, a sizeable portion of the credit will be used for learning- in terms of both skillacquisition and applied learning through the pilot testing and experimental procedures of the assessmentdesign. As shown in the table below, the project would have three components: capacity building; pilottesting and implementation of the assessment activities; and information dissemination.

Capacity BuildingPilot testing and Implementation of Assessment ActivitiesInformation Dissemination

5. FinancingTotal ( US$m)

BORROWER $2.16IBRDIDA $3.63UK: BRITISH DEPARTMENT FOR INTERNATIONAL DEVELOPMENT (DFID) $1.14Total Project Cost $6.93

6. ImplementationThe project will be implemented over a period of five years (2003-08). The NEAS involves very closecollaboration between the federal Ministry of Education and the Provincial and Area Departments ofEducation (DOEs).

Overall, coordination is the responsibility of the Federal government under the umbrella of the broaderEducation System Reform initiative led by the Federal Secretary of Education. The various work groups thatwill be responsible for the design and implementation of the system and for its piloting are summarized in thechart which follows. See also Annex 13 for a detailed description of the responsibilities of various workgroups. Simply stated, the primary responsibilities are distributed as follows:

* Federal government (CW): Overall design and administration; consolidation of items for use ininstrument; leadership in pre-administration information campaign; national level reporting andpolicy formation; coordination of technical assistance and training;

* Partnering /Implementing institution(s) (Pakistan): item development, technical assistance,training, coordination of pilot testing and analysis; setting standards for administration and sampling,help in conducting assessment;

* Provincial/area governments: Item development for pooling at the national level, providing list ofschools and other data for sampling at national level; help in collecting background data of sampleschools/students etc.; test administration, consolidation of provincial data; analysis of provincialdata, dissemination of provincial findings.

4PID

Functions of Various Work Groups

NEAS Coordination through National PEACE Coordination throughPlanning & Coord Committee Prov Planning & CoordDevelop overall framework < * CommDtteeReview al Ivnstruments Coordinate provincial activinesFacilitate fieldwork Manage field workFaclstate report preparaton Supervise provincial reports

NEAS SubjatcGroups PEACE Subjacg GroupsCoordinate review of cup culum objestives / eReview mumpulum objestivesFnaeze assessment instruinents mnDras a Items for each subject to shareCompile results from l il pilot studies and redesign Drawith NEASCoInstruments rvi Pilot ItemsAnalyze cumculum achievement by subject and Propose Revisionsobjectives Prepare provincal repors on cumculum

v with|achievement

NEAS Bactground Data Group PEACE Bacground Data GroupCoordinate review of poScy h ssues PpIdentify major policy ssues to beDesign common background instuments addressed in study

l Coordinate analysis of pilot study eDraf background data Instrumentsl_Coordinate revision Pi C lot co mmon I ns trument

Analyze relations of background vatiables vfth Propose Revisions

F achievement a Analyze relations of backgroundConducts on-the-job training in instrument design, dvanables with achievement

|aNEAS Statistical Group c PEACE Analytic GroupReview Provincial School Censuses Prepare census of all pubiKt and p invatePnopose sample designs schools i n the provinceMonitor quanty of data et Manage province data GputConduct naronal analysis Clean data and send copy to NEAS

Overallmanad Coordinatinte analysis of province data

Technical Assistance

Provide tefcfiorcal assistangce/reviews foth all phases of hork at national anidprovincial levelsFacilitates acquisition of relevant literatureConducts on-thc-job trainig in instrument design, data filc linkages, data entryand cleaning, item analysis, and statistical analysis

The MoE constituted, through a notification dated March 17, 2003, with proper representation fromthe GOP, Provincial governments and the Azad Jammu and Kashincir government, the followingcommittees and groups with their specific TORs: (i) National Planning and Coordination Committee;

(ii) Assessment Instrument Developiment Groups; (iii) Background Data Working Group; and (iv)Statistical Working Group.

Overall management and Coordincrtion: The Curriculum Wing (CW) of the Ministry will provide theoverall coordination of the NEAS effort, drawing on the advice of the National Planning and

Coordination Commiittee composed of representatives from the Federal Ministry and each of theprovinces. This coordination, represented in the diagram above, includes overall coordination as well

5PID

as coordination of the assessment of achievement in four subject areas, examination of school andbackground factors that are believed to influence achievement, sampling design, analysis,report-writing, and dissemination of findings to all stakeholders. Provincial Educational AssessmentCenters (PEACE) will be set up in each of the provinces for the actual implementation of thefieldwork and for the preparation and analysis of the data collected in each province.

The Joint Education Advisor (JEA) of the CW will be ex-officio Project Director to be assisted by afull time Coordinator. The Bureau of Curriculum in Balochistan, Directorate of Curriculum andTeacher Education in North West Frontier Province, University of Education in Punjab, Directorateof Research and Assessment in Sindh, Directorate of Curriculum Research and Development in AzadJammu and Kashmir, and Directorates of Education in Federally Administered Tribal Areas,Islamabad Capital Territory, and Northern Areas will be the institutions responsible for ground levelimplementation of NEAS. The focal persons in these institutions have already been identified by theprovincial/area departments of education.

Prior to the conclusion of each fiscal year of the project, the National Planning and CoordinationCommittee will be expected to prepare for the Secretary of Education (in his capacity as head of theEducation System Reform Committee) an annual plan for the forthcoming year with a detailed timeline and a report of all work accomplished over the past year (including attachment of publishedreports). The Secretary, as head of the ESR implementation, will be responsible for monitoring theproject. The Coordinator of NEAS, with input from the Technical Advisor and the "partneringorganization" will also be required to make quarterly reports to the ESR, per agreed format and witha focus on implementation progress against benchmarks, and fiscal reporting.

The Bank will participate in National Planning and Coordination Committee meetings as appropriateand will assess the quality of learning embedded in the project through its semi-annual supervisionmissions and through regular contact with the Project Director and lead Technical Advisor.

7. SustainabilityThe sustainability of the NEAS will depend upon the quality of instruments developed; effectiveimplementation; capacity building; and the level of participation of various stakeholder groups. One factorwhich will help in institutionalizing the system is the fact that GOP, the Provincial governments, and thegovernment of Azad Jammu and Kashmir will fund the entire cost of staff salaries and allowances in theirrespective areas of responsibility. There is also a growing awareness of the importance of student assessment.However, ownership in the MoE and provincial/area departments of education is weak, as indicated by longdelays in meeting the basic requirements of the project, including appointment of staff in MoE. Strengtheningthis awareness and ownership would heavily depend upon the continued commitment of senior governmentofficials and policy-makers. The quality of TA, especially of the partnering organization that would behelping the government in conducting assessments and developing capacity, would also play an importantrole. To facilitate sustainability and to institutionalize training capacity, the project will help to establish anoverall certifying and accrediting body, which would award skill certificates in assessment and assure thequality of training provided by other agencies.

6PID

8. Lessons learned from past operations in the country/sectorA number of studies of learning achievement have been conducted in Pakistan from 1984 to date. Most ofthem are summarized in Annex 14 of the PAD. In addition, the Academy of Education Planning andManagement (AEPM) carried out a sample survey of achievement among Grade 4 students to meet thecountry requirements of EFA Assessment 2000 (AEPM, 2000). With the exception of UNESCO studies,assessment activities have tended to be part of individual development projects in various provinces / areasrather than a national level effort. All of these studies have weaknesses in, at least, five major areas whichlimit the conclusions that can be drawn and makes the comparison of student achievement over time difficult:

1. Different Grades at different times: These studies were conducted for different grades (1-3, 4-5, 3& 5, 5, 6, 7 or 8 only) at different times with little effort to relate findings to the results of previousstudies or grades.

2. Instrument development: The framework for constructing the items is rarely reported, so it isdifficult to evaluate curriculum coverage. A single form of a test paper is normally used, rather thanmultiple spiraled blocks. This increases the risk of copying, introduces order effects (items that comelate in the test tend to get less attention than those at the beginning) and limits the possible coverageof curriculum objectives.

3. Sample design: There is no generally-used sampling frame. The sampling frames used by thesestudies tend to cover Government schools well and others much less well. Community schools andprivate schools tend to be omitted. In many cases too many primary sampling units (PSUs) may becovered, in an effort to make assessment as participatory as possible. As a result, too few schools aretaken from any PSU to allow adequate representation of diverse school types. Weighting according tosize is not used in determining the probability of selection and in combining results.

4. Administration: Administration was frequently devolved to local educators, at the district or tehsillevel. This increases the risk of inconsistent or lax administration. Training and supervision ofadministrators is not normally reported in sufficient detail to judge its rigor.

5. Analysis and reporting: The emphasis in the reports is on total scores, normally expressed aspercentages. These provide little useful information. For example, does a mean score of 51% in mathin a North West Frontier Province test administered in 1995 relate to the curriculum goals masteredby the average student? Is it evidence of any change in standards since other (BRIDGES) testing,which produced a mean score of 26% in the same subject? It is impossible to say.

One possible exception to these limitations is the study carried out in 1995 by the Pakistan PsychologicalFoundation (Pervez, 1995). This study was sponsored by UNICEF and used the ABC (Assessment of BasicCompetencies) test, versions of which have also been used elsewhere in South-Asia. However, the ABC testis not based directly on Pakistani curriculum, and tests only very limited competencies. Therefore, althoughthe study is suggestive of low standards of achievements, it does not relate specifically to Pakistani curriculumgoals.

The work cited above is often interesting and informative. However, there is little national experience inconducting cross-provincial, curriculum based assessments and there is very limited technical capacity tocarry out this kind of studies in Pakistan.

The design of the project has benefited from the above lessons / findings. Although other projects also aimedat capacity development for assessment to some extent, it did not happen because assessment was one amongseveral components of the projects, and therefore it could not get the attention which it deserved. This time, alot of background work has been done over the last five years by the government to bring all stakeholders onboard in realizing the importance of student assessment/system evaluation. However, ownership of theproject remains weak and sustainability is also not guaranteed. The Bank is proceeding with this project with

7

PID

the understanding that it is a high risk-high reward activity.

9. Environment Aspects (including any public consultation)Issues: NA

10. List of factual technical documents:1. PIP for five years (June 2002 to June 2007)2. Detailed PIP for the period June 2002 to June 20033. Appraisal Mission Aide-Memoires4. Cost Tables5. Education Sector Reform Action Plan (2001-05)6.Reports of the Technical Group Meetings (TGMs)7. Federal, provincial/area PC-Is8. Government's Concept Clearance Paper9. GoP's request (through EAD) for Bank supportI 0. DFID consultant reports11. EFA Assessment 2000

11. Contact Point:

Task ManagerS. Ameer NaqviThe World Bank1818 H Street, NW

Washington D.C. 20433Telephone 9251-9090155Fax: 9251-2822396

12 For information on other project related documents contact:The InfoShopThe World Bank1818 H Street, NWWashington, D C. 20433Telephone (202) 458-5454Fax. (202) 522-1500Web http 11 www worldbank.org/infoshop

Note: This is information on an evolving project. Certain components may not be necessarilyincluded in the final project.

Tables, Charts, Graphs:

Processed by the InfoShop week ending: 05/07/2003

For a list of World Bank news releases on projects and reports, click here

8PID

I EARcwFEEDOACK-) , dSIhOn AAFSHOWCAME