National Learning Assessment in Ethiopia: sharing experiences and

lessons

HD Learning Week Girma Woldetsadik

AFTEE, The World Bank Feb 7, 2013

Outline

• National learning assessment • Challenges and opportunities

• Lessons

1. National learning assessment (NLA)

1.1. About NLA • System-level information on student learning

levels and related factors • A test administered nationally at different levels

in key subjects at four year intervals • First, baseline, NLA of grades 4 & 8: 1999/2000

– 2nd round in 2003/4 – 3rd in 2006/7 – 4th in 2010/11

• First baseline NLA of grades 10 and 12: 2008/09

1.2. Why was it important, in which context?

• Growing recognition of a link between educational attainment and economic development

• New Education and Training Policy in 1994 – improving quality of education - one of the focus areas – Recommended establishment of a national organization

for educational measurement and exams • Five year Education Sector Development Programs

since 1997 as part a 20-year education sector indicative plan – to improve educational quality, relevance, efficiency,

equity and expand access to education

• As a result of expansion in enrolment, concern about the quality of education: ‘are our children learning?’; and disparities across gender and regions

• Inform and guide policy reform and resource allocation to improve learning

• DPs push for assessing learning outcomes and readiness to support

1.3. What is the purpose?

• To provide information on student achievement levels in primary and general secondary education

• To identify factors that influence (enhance or retard) student achievement

• To analyze variations in student achievement by region, gender, location, and language of instruction

• To monitor changes in student achievement over time • To recommend appropriate remedial actions to

improve learning outcomes

1.4. What were the main features?

• To monitor the health of the education system • Grades: 4, 8, 10, 12

– Grade 4: English, mathematics, environmental science, reading comprehension in mother tongue

– Grades 8, 10 and 12: English, mathematics, biology, chemistry, and physics because of focus on science and technology in sec and HE

• Frequency of tests: testing every four years • Test population

– Sample schools and students (nationally representative)

Main features (cont’d)

• Achievement measures - Standardized achievement tests developed based on

minimum learning competencies at each grade level and subject

- Tests developed by curriculum experts, test development experts and teachers guided by assessment specialists

- Pilot study in selected schools; validation workshop involving test developers

– Tests in grades 4 (19 instructional languages) and 8 (4 languages); grades 10 & 12 in English

Main features (cont’d) • Type of testing

– Objective – Paper-based tests – Point-in-time test, under standardized conditions

• How results are reported - percentage scores: 0 to 100% - Standards: below basic, basic, proficient, advanced

• Questionnaires and focus group discussions on explanatory (school and non-school) factors - Sample students: about themselves, their family conditions, interest

and activities they carry out in spare time - Their teachers, school directors, and parents: about their students,

school environment, teaching-learning process

Main features (cont’d)

• How results are reported - percentage scores: 0 to 100%

• Questionnaires and focus group discussions on explanatory (school and non-school) factors - Sample students: about themselves, their family

conditions, interest and activities they carry out in spare time

- Their teachers, school directors, and parents: about their students, school environment, teaching-learning process

1.5. Who led the NLA from MOE?

• Led by staff of National Educational Assessment and Examinations Agency of the MOE (previously under higher education relevance and QA agency; general education QA and exam agency; national agency for exams)

• Supported by curriculum experts, teachers, consultants

• Financial support from development partners (USAID from start; and GEQIP, READ since 2009)

2. Challenges and opportunities

• Who were the project allies? Who opposed it? • What were the main challenges in designing and

implementing the project? What key decisions were taken? What were the main mistakes? Did it work as planned?

• How did the project address capacity building? • How was funding ensured? • What was done to ensure sustainability? • How was assessment information used (or not) to

inform the education system?

2.1. Who were the project allies? who opposed it?

• Government’s recognition of the importance of assessment – Assessment is included in the Education and Training Policy

(1994) – Directive by Council of Ministers to establish a National

Educational Assessment and Examinations Agency • Parents strongly support • DPs support: training, technical assistance, operational

costs • Opposition is not on its benefits, but capacity to do it,

and its consequences for education leaders

2.2. Main challenges in designing and implementation

• Unclear institutional arrangement until Jan 2012 • In 2012, assessment was institutionalized through

creation of the autonomous National Educational Assessment and Examination Agency – Ensured stable institution, funding, sustainability

• No policy document on NLA activity, but general understanding that NLA program would take place every 3-4 years – Policy framework and guidelines under preparation

• Irregular funding until recently: mainly from different donors

• Since Jan 2012 regular government budget + support from donors

• Inadequate competent staff to effectively carry out NLA activities – Reliance on technical assistance – DPs on-going support to build technical capacity

of the staff through training, and exposure visits

• System alignment: achievement tests are based on national curriculum using the minimum learning competencies

• But there is no formal or standard procedures for regular reviews (internal or independent) of NLA to ensure that it measures what it is intended to measure

• Tests are translated into the language of instruction for grades 4 and 8 – likely impact on comarisons

• But scores are not strictly comparable since the NLA exams are not yet completely standardized over time • Plans for item-bank development

• Unrealistic test score targets in sector plans

2.3. How was assessment information used (or not) to inform the education system?

2.3.1. On findings Overall achievement scores • National mean score of subjects for all grades

was less than 50% achievement level set by MOE • Grade 10 mean score of all subjects was 36%; and

grade 12: 47.8% • In grade 10, only 13.8% scored 50% and above;

and in grade 12: 34.9% • National learning results of grades 4 and 8 have

actually reduced slightly over time, esp. maths

On one hand: dramatic increase in primary enrolment

But low primary completion rates, %

44.9 44.7 43.6 47.8 49.4

40

50

60

70

80

90

100

2006 2007 2008 2009 2010 2011

And not accompanied by improvements in quality

48 49

41

40 40

36 35

0

10

20

30

40

50

60

1998 2000 2002 2004 2006 2008 2010 2012

Academic year [2000 stands for 1999/00]

NLA mean composite score in %

Grade 4 Grade 8

Grade 4 composite mean score in %

Subjects 2000 2004 2007 2011 Reading comprehension (mother tongue) 64.25 64.49 43.9 42.96 English 40.46 38.68 36.5 38.87 Maths 39.31 39.7 40.3 37.06 Environmental science 48.1 51.74 42.6 41.21 Composite 47.54 48.5 40.9 40.06

Grade 8 composite mean score in %

Subjects 2000 2004 2007 2011 Biology 47.16 41.34 38.3 42.1 Chemistry 40.27 40.1 34.7 36.42 English 38.74 41.07 38.4 36.86 Maths 38.23 40.93 34.1 25.53 Physics - 35.32 32.2 34.47 Composite 41.1 39.74 35.6 35.2

Grade 10 and 12 composite mean score in %

Subjects G10 G12 • English 37.4 43.4 • Maths 34.7 54.3 • Biology 40.3 55.5 • Chemistry 36.1 49.1 • Physics 31.2 36.6 Average 36 47.8

Comparisons across sub-groups • In all subjects in all grades, boys performed

better than girls • Wide disparities among regions – less in

under-served regions • Urban schools outperformed rural schools

• School variables: distance to school, frequency of home works given, and availability of textbooks

• Teachers qualification, experience in teaching, motivation, discussion with parents

• Non-salary recurrent expenditures • Leadership and management capacity of the

education bureaucracy • Home variables: number of meals a day, number

of times listening to the radio, time helping with family chores, parents education, language spoken at home and language of instruction

3.1.2. Explanatory factors

3.1.3. Dissemination and use of assessment information

• Generally delay in release and use of results • Reports are poorly disseminated

– few copies of the report available – not translated – no feedback to schools and teachers – limited awareness among general public – lack of accountability, i.e. no mechanisms to

monitor whether recommendations are properly implemented

• But results are to certain extent used by MOE and regional education bureaus – Teacher Development Program (2006-09) – General Education Quality Improvement Program (2009-13), (2013-2017)

• curriculum revision • textbooks development • teacher and leadership training

– Teacher licensing and re-licensing being considered • school improvement program and school grant • School inspection

• Poor result in EGRA in mother tongue in 2009/11 led to decision to revise curriculum and develop teaching/learning materials

• Improving learning achievements still remains challenging, but there is hope

• No concrete plans to participate in regional or international large-scale assessment: not possible to compare with other countries

Primary Education Quality in Decline A study conducted by the Institute of Education Research of Addis Ababa University in selected schools of four regional states revealed that the quality of primary education is declining. The research, presented by the Ethiopian Academy of Sciences, on December 27, 2012 at Semen Hotel stated that the majority of primary students score below the minimum 50pc expected for all subjects. The decline in quality of education is expressed in the research in terms of the students’ inability to attain the basic literacy and numeracy skills expected at different grade levels, according to the study. The research also identified there is a mismatch between the students’ capacity to learn with their grade level. Some students were not even able to properly comprehend classroom instructions, especially those that were given in English. PUBLISHED ON DEC 30, 2012 [ VOL 13 , NO 661]

5. Lessons

• The need for a stable organization responsible for assessment – Assessment has been under different organizations

until Jan 2012 – National Education Assessment and Examinations

Agency legally established in Jan 2012 • The need to establish a clear policy framework

for assessment – Defines roles and responsibilities – Ensures that it operates on a regular basis – Ensures regular funding

• Adequate and trained staff to manage assessment – NLA relies heavily on technical assistance – Currently staff being trained to ensure sustainability

• Ensuring effective use of assessment results

– Assessment not an end by itself – More effective use of data for design of interventions that

make a difference – Wide dissemination of results to stakeholders, including

teachers and general public – Mechanism to monitor the implementation of NLA

recommendations

Thank you