ACR - 2019 Regional Early Grade Reading Assessment ...2020/04/21 · mother-tongue-based...

Submission Date: April 21, 2020 AID-OAA-TO-16-00017 Number: REQ-ASIA-16-00017 TOCOR: Mitch Kirby Submitted by: RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709-0155 Tel: (919) 541-6000 Authored by: Kellie Betts, Maitri Punjabi, Sarah Pouezevara, and Chris Cummiskey 3040 East Cornwallis Road Research Triangle Park, NC 27709-0155 Tel: (919) 541-6000 This document was produced for review by the United States Agency for International Development.

All Children Reading–Philippines 2019 Regional Early Grade Reading Assessment (EGRA): Bahasa Sug, Chavacano, Magindanawn, and Mëranaw

Findings Report

ii 2019 Philippines Regional EGRA: Bahasa Sug, Chavacano, Magindanawn, and Mëranaw

Table of Contents Page

List of Figures ................................................................................................................ iii

List of Tables .................................................................................................................. v

List of Acronyms and Abbreviations .............................................................................. vii

1 Executive Summary ........................................................................................... 1

2 Introduction ........................................................................................................ 6

Background and Context .................................................................................... 6

3 Research and Sample Design ........................................................................... 7

Research Questions and Design ....................................................................... 7

Sample Design ................................................................................................... 8

4 Results by Language ......................................................................................... 9

Bahasa Sug ...................................................................................................... 10

Listening Comprehension ..................................................................... 11

Letter-Sound Identification ................................................................... 12

Non-Word Reading ............................................................................... 13

ORF Passage Reading ........................................................................ 14

Reading Comprehension ...................................................................... 16

Chavacano ....................................................................................................... 17



Non-Word Reading ............................................................................... 20



Magindanawn ................................................................................................... 24



Non-Word Reading ............................................................................... 27



Mëranaw .......................................................................................................... 32



Non-Word Reading ............................................................................... 35



2019 Philippines Regional EGRA: Bahasa Sug, Chavacano, Magindanawn, and Mëranaw iii

5 Discussion and Conclusion .............................................................................. 39

6 Next Steps ........................................................................................................ 45

Annex A: Overview of Early Grade Reading Assessment (EGRA) Subtasks .............. 49

Annex B: Student Questionnaire .................................................................................. 50

Annex C: Assessor Training Details and Data Collection Procedures ......................... 54

Annex D: Sample Methodology, Sample Weights, and Precision Estimates ............... 60

Annex E: Bahasa Sug Cumulative Distribution Graphs ............................................... 65

Annex F: Chavacano Cumulative Distribution Graphs ................................................. 68

Annex G: Magindanawn Cumulative Distribution Graphs ............................................ 71

Annex H: Mëranaw Cumulative Distribution Graphs .................................................... 74

List of Figures Figure 1. Grade 2 ORF scores, by gender and language ............................................... 4

Figure 2. Grade 3 ORF scores, by gender and language ............................................... 4

Figure 3. Bahasa Sug listening comprehension story and questions ........................... 12

Figure 4. Bahasa Sug letter-sound identification subtask student stimulus sheet ........ 13

Figure 5. Distribution of Bahasa Sug ORF, by grade (Passage 1) ............................... 14

Figure 6. Average attempted items and accuracy in Bahasa Sug, Grade 2 ................. 15

Figure 7. Average attempted items and accuracy in Bahasa Sug, Grade 3 ................. 15

Figure 8. Bahasa Sug reading comprehension scores, by grade (Passage 2) ............ 17

Figure 9. Chavacano listening comprehension story and questions ............................ 19

Figure 10. Chavacano letter-sound identification subtask student stimulus sheet ......... 20

Figure 11. Distribution of Chavacano ORF, by grade ( Passage 1) ............................... 21

Figure 12. Average attempted items and accuracy in Chavacano, Grade 2 .................. 22

Figure 13. Average attempted items and accuracy in Chavacano, Grade 3 .................. 22

Figure 14. Chavacano reading comprehension scores, by grade (Passage 2) .............. 23

Figure 15. Magindanawn listening comprehension story and questions ........................ 26

Figure 16. Magindanawn letter-sound identification subtask student stimulus sheet ..... 27

Figure 17. Distribution of Magindanawn ORF, by grade (Passage 1) ............................ 29

Figure 18. Distribution of Magindanawn ORF for Grade 2, 2014 vs. 2019 (Passage 1) ................................................................................................................... 30

Figure 19. Magindanawn reading comprehension scores, by grade (Passage 1) .......... 31

Figure 20. Magindanawn reading comprehension percent correct of attempted for Grade 2 (Passage 1) ..................................................................................... 32

Figure 21. Mëranaw listening comprehension story and questions ................................ 34

Figure 22. Mëranaw letter-sound identification subtask student stimulus sheet ............. 35

iv 2019 Philippines Regional EGRA: Bahasa Sug, Chavacano, Magindanawn, and Mëranaw

Figure 23. Non-words itemized scores in Mëranaw, Grade 2 (words 1–28) ................... 36

Figure 24. Non-words itemized scores in Mëranaw, Grade 3 (words 1–39) ................... 36

Figure 25. Distribution of Mëranaw ORF, by grade (Passage 1) .................................... 37

Figure 26. Average attempted items and accuracy in Mëranaw, Grade 2...................... 38

Figure 27. Average attempted items and accuracy in Mëranaw, Grade 3...................... 38

Figure 28. Mëranaw reading comprehension scores, by grade (Passage 2) ................. 39

Figure 29. Grade 2 percent zero scores, by subtask and language ............................... 40

Figure 30. Percentage of time spent reading during MT subject class, by language ..... 41

Figure 31. Summary of outputs from discussion groups ................................................ 46

Figure E-1. Bahasa Sug cumulative distribution of listening comprehension scores, by grade ........................................................................................................ 65

Figure E-2. Bahasa Sug cumulative distribution of ORF scores, by grade (Passage 1) ................................................................................................................... 66

Figure E-3. Bahasa Sug cumulative distribution of reading comprehension scores, by grade (Passage 1) .................................................................................... 66

Figure E-4. Percent total Bahasa Sug reading comprehension scores, by grade (Passage 1) ................................................................................................... 67

Figure E-5. Bahasa Sug cumulative distribution of reading comprehension scores, by grade (Passage 2) .................................................................................... 67

Figure F-1. Chavacano cumulative distribution of listening comprehension scores, by grade ........................................................................................................ 68

Figure F-2. Chavacano cumulative distribution of ORF scores, by grade (Passage 1) ................................................................................................................... 69

Figure F-3. Chavacano cumulative distribution of reading comprehension scores, by grade (Passage 1) .................................................................................... 69

Figure F-4. Percent total Chavacano reading comprehension scores, by grade (Passage 1) ................................................................................................... 70

Figure F-5. Chavacano cumulative distribution of reading comprehension scores, by grade (Passage 2) .................................................................................... 70

Figure G-1. Magindanawn cumulative distribution of listening comprehension scores, by grade ............................................................................................ 71

Figure G-2. Magindanawn cumulative distribution of ORF scores, by grade (Passage 1) ................................................................................................... 72

Figure G-3. Magindanawn cumulative distribution of reading comprehension scores, by grade (Passage 1) .................................................................................... 72

Figure G-4. Percent total Magindanawn reading comprehension scores, by grade (Passage 2) ................................................................................................... 73

Figure G-5. Magindanawn cumulative distribution of reading comprehension scores, by grade (Passage 2) .................................................................................... 73

Figure H-1. Mëranaw cumulative distribution of listening comprehension scores, by grade ............................................................................................................. 74

Figure H-2. Mëranaw cumulative distribution of ORF scores, by grade (Passage 1) ....... 75

Figure H-3. Mëranaw cumulative distribution of reading comprehension scores, by grade (Passage 1) ......................................................................................... 75

2019 Philippines Regional EGRA: Bahasa Sug, Chavacano, Magindanawn, and Mëranaw v

Figure H-4. Percent total Mëranaw reading comprehension scores, by grade (Passage 1) ................................................................................................... 76

Figure H-5. Mëranaw cumulative distribution of reading comprehension scores, by grade (Passage 2) ......................................................................................... 76

List of Tables Table 1. Percentage of students achieving 80 percent reading comprehension,

Grades 2 and 3 ................................................................................................ 2

Table 2. Reading comprehension scores—percent of questions correct out of total, by language and grade ........................................................................... 3

Table 3. Impact of absenteeism on mean ORF scores, by language ............................ 5

Table 4. Impact of over-age on reading ........................................................................ 5

Table 5. Percentage of lesson time students spent reading and writing in the MT, by grade .......................................................................................................... 6

Table 6. Students sample per language ........................................................................ 9

Table 7. Overview of Bahasa Sug EGRA percent zero and mean scores, by grade ............................................................................................................. 11

Table 8. Bahasa Sug ORF scores, by gender ............................................................. 16

Table 9. Overview of Chavacano EGRA percent zero and mean scores, by grade ............................................................................................................. 18

Table 10. Chavacano ORF scores, by gender .............................................................. 21

Table 11. Overview of Magindanawn EGRA percent zero and mean scores, by grade ............................................................................................................. 25

Table 12. Magindanawn ORF scores, by gender (Passage 1) ...................................... 28

Table 13. Overview of Mëranaw EGRA percent zero and mean scores, by grade ....... 33

Table 14. Mëranaw ORF scores, by gender.................................................................. 37

Table 15. Mean ORF on passage reading and comprehension scores, by language and grade ...................................................................................... 40

Table 16. Impact of absenteeism on mean ORF scores, by language .......................... 41

Table 17. Top and bottom quintile per subtask, in Bahasa Sug .................................... 42

Table 18. Top and bottom quintile per subtask, in Chavacano ..................................... 42

Table 19. Top and bottom quintile per subtask, in Magindanawn ................................. 42

Table 20. Top and bottom quintile per subtask, in Mëranaw ......................................... 43

Table 21. Teachers’ use of languages in class* ............................................................ 44

Table 22. Teachers’ use of languages in MT subject class* ......................................... 45

Table A-1. EGRA subtasks ............................................................................................. 49

Table B-1. Student questionnaire responses for Grade 2 and Grade 3 students combined, by language ................................................................................. 50

Table C-1. AAM results for Bahasa Sug: Percent averages and ranges of agreement, by EGRA subtask (n = 8) ........................................................... 54

Table C-2. AAM results for Chavacano: Percent averages and ranges of agreement, by EGRA subtask (n = 8) ........................................................... 55

vi 2019 Philippines Regional EGRA: Bahasa Sug, Chavacano, Magindanawn, and Mëranaw

Table C-3. AAM results for Magindanawn: Percent averages and ranges of agreement, by EGRA subtask (n = 8) ........................................................... 55

Table C-4. AAM results for Mëranaw: Percent averages and ranges of agreement, by EGRA subtask (n = 6) ............................................................................... 56

Table C-5. Inter-rater reliability results for Bahasa Sug: Percent averages and ranges of agreement, by EGRA subtask (n = 34) ......................................... 57

Table C-6. Inter-rater reliability results for Chavacano: Percent averages and ranges of agreement, by EGRA subtask (n=41) ........................................... 57

Table C-7. Inter-rater reliability results for Magindanawn: Percent averages and ranges of agreement, by EGRA subtask (n = 36) ......................................... 58

Table C-8. Inter-rater reliability results for Mëranaw: Percent averages and ranges of agreement, by EGRA subtask (n = 36) ..................................................... 59

Table D-1. Schools excluded from the list frame prior to sampling ................................. 60

Table D-2. Population counts of schools, by language group ......................................... 61

Table D-3. Sample methodology for the Philippines Regional Snapshot 2019 EGRA, three-stage stratified sample of schools, classrooms, and students ......................................................................................................... 62

Table D-4. Final sample counts of assessments, interviews, and observations collected for the Philippines Regional Snapshot 2019 EGRA ....................... 62

Table D-5. Weighted sample (estimated population) and sample counts, by each stage sampled (school, classroom, and student) .......................................... 63

Table D-6. Reading fluency means and precision estimates, by grade .......................... 63

Table D-7. Grade 2 reading fluency means and precision estimates, by language assessed ....................................................................................................... 64

Table D-8. Grade 3 reading fluency means and precision estimates, by language assessed ....................................................................................................... 64

2019 Philippines Regional EGRA: Bahasa Sug, Chavacano, Magindanawn, and Mëranaw vii

List of Acronyms and Abbreviations AAM assessor accuracy measure ARMM Autonomous Region in Muslim Mindanao BARMM Bangsamoro Autonomous Region in Muslim Mindanao BEIS Basic Education Information System CI confidence interval clspm correct letter sounds per minute cnonwpm correct non-words per minute cwpm correct words per minute DEF design effect DepEd Philippines Department of Education EGRA Early Grade Reading Assessment ICC intra-class correlation KG kindergarten LAC Learning Action Cell LMRSD Learning Resources Management and Development System LSB local school board MoTL medium of teaching and learning MTB-MLE mother-tongue based multilingual education MT mother tongue ORF oral reading fluency SES socioeconomic status USAID United States Agency for International Development

2019 Philippines Regional EGRA: Bahasa Sug, Chavacano, Magindanawn, and Mëranaw 1

1 Executive Summary Background In March 2019, the United States Agency for International Development/Philippines All Children Reading–Philippines activity worked with the Philippines Department of Education (DepEd) to administer an Early Grade Reading Assessment (EGRA) in representative samples of schools that teach in the mother tongues (MTs) of Bahasa Sug, Chavacano, Mëranaw, and Magindanawn. EGRA is a research-based measure of oral reading fluency (ORF; accuracy and speed) and comprehension. The assessment measured Grade 2 and Grade 3 students’ current level of reading proficiency in each of the MTs. Other subtasks were included in the study to measure underlying skills leading to fluency—listening comprehension, decoding, alphabetic principle—and are useful for more specifically understanding whether children are being taught foundational skills that support fluency and comprehension. This study measured students’ reading ability and gathered basic demographic information from children and teachers. Some standard context information was also gathered from children about their exposure to reading in the home. Thus, the data allow us to describe one outcome of the current schools sampled—children’s reading ability—but they do not provide any empirical measurement of the inputs that contribute to this outcome. To explain the current state of reading performance as measured by this study, we must rely on other contextual data from our concurrent Language Usage Study and general knowledge of mother-tongue-based multilingual education (MTB-MLE) implementation, as documented by policy and other studies carried out by other researchers. In the absence of direct measurement of “implementation fidelity” to a particular reading instruction program or materials, we must also rely on global evidence of how reading skills develop in alphabetic languages. To put it simply, children can learn to read, but only if they are taught to read. Teachers can only teach reading if they have been prepared to do so through training and are equipped with appropriate materials. Teachers and students must be present and making productive use of class time. The purpose of using EGRA as a system diagnostic is primarily to establish a baseline against which future progress can be measured and to identify priority areas for instructional improvement and teacher training.1 This study provides baseline measurements for the Bahasa Sug, Chavacano, and Mëranaw languages, but it is the third time Magindanawn has been measured using comparable methodology.2 Since student performance has not been previously measured for most of these languages, we relied on other local languages as initial reference points, as well as stated curricular objectives to make recommendations. Notably, the MTB-MLE policy expects reading instruction in second and third languages to begin as early as Grade 2. Experience from other multilingual contexts suggest that children should have sufficient first language literacy mastery before the second and third literacies are introduced. Finally, this study is not meant to be a measurement of whether the MTB-MLE policy is appropriate or working. It measures only one of many intended outcomes of using the MT as a medium of teaching and learning in primary school. Overview of Performance Across Languages The data from this study indicated that large proportions of students, especially those in Grade 2, struggled with foundational reading skills and were not yet able to read with comprehension in the MTs used in the school in which they were enrolled (see Table 1). Although students’ reading proficiency and comprehension scores improved in Grade 3

1 Gove, A., & Wetterberg, A. (Eds.) (2011). The Early Grade Reading Assessment: Applications and interventions to improve basic literacy. (RTI Press Publication No. BK-0007-1109). Research Triangle Park, NC: RTI Press. https://doi.org/10.3768/rtipress.2011.bk.0007.1109 2 DepEd previously administered EGRAs in Chavacano but has not publicly released the data or findings from those studies.

https://doi.org/10.3768/rtipress.2011.bk.0007.1109

2 2019 Philippines Regional EGRA: Bahasa Sug, Chavacano, Magindanawn, and Mëranaw

compared to Grade 2, an important consideration is whether children are ready to transition to second and third language acquisition based on sufficient linguistic and academic competency in their MT. In the languages measured, 50–65 correct words per minute (cwpm) on passage reading tends to be the rate at which Grade 3 children begin to successfully comprehend most of what they have read.3 Thus, until more data on these languages are available, we believe that a high proportion of zero scores, low percent of students reaching 80 percent comprehension, and ORF below 50–65 in Grade 3 are signs that priority must be placed on training teachers in explicit and direct instruction of reading, which includes the building blocks of oral language development, phonics, and reading for comprehension.

Table 1. Percentage of students achieving 80 percent reading comprehension, Grades 2 and 3

Language

Grade 2 Grade 3

Percentage achieving 80% comprehension (out

of total of 5 questions)

Percentage achieving 80% comprehension (out

of total of 5 questions) Bahasa Sug 19% 42% Chavacano 36% 50% Magindanawn 9% 20%

Mëranaw 34% 44% The listening comprehension subtask measures oral language comprehension and vocabulary and is an indication of whether or not the language of the assessment is also the child’s home language or otherwise a language the child understands. Oral language comprehension is an important pre-requisite to reading with comprehension. For this subtask, students in schools that teach in Chavacano and Magindanawn struggled most with oral comprehension, as evidenced by the high proportion of students unable to answer a single question (41 percent of students in Chavacano and 25 percent of students in Magindanawn). In contrast, in Mëranaw and Bahasa Sug schools, most students demonstrated oral comprehension skills in the language of instruction. The letter-sound identification subtask measured students’ ability to provide the sound of letters of the alphabet. This pre-literacy skill contributes to decoding and word identification ability. Across languages, Grade 2 students averaged less than 20 correct letter sounds per minute (clspm). A high percentages of students scored zero in Bahasa Sug, Chavacano, and Magindanawn (25–30 percent of students in each language). In Mëranaw, few students scored zero, meaning most were able to identify some letter sounds correctly, but the average was still quite low at 18 clspm in Grade 2 and 23 clspm in Grade 3. Consequently, on the non-word reading subtask, students struggled to decode and average scores were similar to those of the letter identification subtask. The percentages of students who scored zero remained high for the same languages that had high zero scores on letter-sound identification. The oral passage reading subtask measured students’ ability to read connected text. Each student was shown a grade-appropriate short story and given one minute to read aloud.

3 Each language has its own specific linguistic features that affect the expected rate of literacy acquisition and, subsequently, rate of reading that can be considered fluent. More information about each language’s fluency benchmark can be found in 2019 Benchmarks for Reading Performance for Bahasa Sug, Chavacano, Magindanawn, Mëranaw Brief (forthcoming).


Grade 2 students across all languages, on average, were reading 36 cwpm or fewer, while Grade 3 students, on average, were reading between 36 and 56 cwpm, depending on the language. The percent of students unable to read a single word on the passage reading subtask was highest for Bahasa Sug (44 percent in Grade 2 and 24 percent in Grade3). Mëranaw had the lowest percent of students scoring zero for this subtask (24 percent in Grade 2 and 8 percent in Grade 3). Comprehension is the ultimate goal of literacy instruction. To measure reading comprehension, students were given another grade-appropriate short story and an extended time of 180 seconds to read aloud. The story remained in front of the students as they responded to the comprehension questions. More than 50 percent of Grade 2 students in Bahasa Sug and Magindanawn schools scored zero on this subtask. In Chavacano and Mëranaw, the percentages were lower; however, a sizable proportion of Grade 2 students still could not answer a single question about what they read (36 percent in Chavacano and 24 percent in Mëranaw). Mean scores varied by language (see Table 2), but on average, Grade 2 students answered fewer than 50 percent of the questions correctly. Grade 3 scores were higher compared to Grade 2, but on average, Grade 3 students were only able to answer two to three questions out of five correctly.

Table 2. Reading comprehension scores—percent of questions correct out of total, by language and grade

Language Grade 2 Grade 3

Bahasa Sug 27% 49%

Chavacano 45% 55%

Magindanawn 23% 39%

Mëranaw 48% 62%

Factors Correlated with Reading Achievement Gender, Age, and Absenteeism

Consistent with patterns seen globally,4 girls performed better than boys in all languages and across most subtasks in this assessment. On average, girls read about 10 cwpm more than boys, irrespective of language or grade. Figure 1 (Grade 2) and Figure 2 (Grade 3) show mean ORF scores by gender and language. The differences are statistically significant for all languages and both grades.

4 A recent report examining the issue in across several countries, including in the Philippines, indicated that “gender stereotypes, gendered socialization, and gendered expectations both at home and in school are considered to be the main reasons for this trend to have emerged and be sustained so consistently across different contexts.” (Jha, J. & Pouezevara,S. [2016]. Boys’ underachievement in education: A review of the literature with a focus on reading in the early years. Prepared for USAID under the Education Data for Decision Making [EdData II] project, Task Order No. AID-OAA-BC-12-00003. Research Triangle Park, NC: RTI)


Figure 1. Grade 2 ORF scores, by gender and language

Figure 2. Grade 3 ORF scores, by gender and language

High rates of student absenteeism were reported across all languages, ranging from 34 percent of students in Mëranaw schools reporting absences of one or more days in the previous week to 60 percent of students in Chavacano schools. In Bahasa Sug schools, 55 percent of students indicated they were absent one or more days in the previous week, and of these students, nearly 12 percent reported being absent all five days. This study did not look into specific reasons for student absenteeism; however, it is evident that this is a widespread issue that negatively impacts student reading outcomes, as shown in Table 3.5

5 Simple linear regression analyses were run for each language and population to determine whether absenteeism was a negative factor with statistical significance.

15

29

17

31

20

39

21

41

0

10

20

30

40

50

60

70

80

90

100

Bahasa Sug Chavacano Magindanawn Mëranaw

ORF

(cor

rect

wor

ds p

er m

inut

e)

Boys Girls

29

41

29

4841

59

39

63

0

10

20

30

40

50

60

70

80

90

100


OR

F (c

orre

ct w

ords

per

min

ute)

Boys Girls


Table 3. Impact of absenteeism on mean ORF scores, by language

Language/Population

Percent of Population

Affected Difference in Mean on ORF Scores (cwpm)

Bahasa Sug schools 50% -6 cwpm

Chavacano schools 50% -13 cwpm

Magindanawn schools 62% -9 cwpm

Mëranaw schools 60% -12 cwpm Data collected on students’ age indicated that high percentages of students were over-age for their grade (22 percent in Bahasa Sug, 12 percent in Chavacano, 25 percent in Magindanawn, and 47 percent in Mëranaw). Being over-age can be caused by a variety of factors: starting kindergarten late, reintegrating into school after a period of absence, or having been required to repeat a grade. Interestingly, the impact of being over-age had a positive impact on students’ reading performance in Mëranaw schools, while it had a negative impact on performance in Chavacano schools. As shown in Table 4 below, students who were over-age in Mëranaw schools were reading 8 cwpm faster than students who were of age. In Chavacano schools, fewer students were over-age; however, the students that were, read 13 cwpm slower than students of age. This is a topic to consider for further research to learn more about the exact causes of over-age in the different areas to better understand the different effects on student outcomes.

Table 4. Impact of over-age on reading

Language/Population




Mëranaw schools 47% 8 cwpm Student and Teacher Home Language

Data collected from students and teachers indicated that their home languages did not always match the MT that was used as the medium of teaching and learning (MoTL) in schools. This creates a complex and challenging learning environment within the classroom. Mismatch of student home vs. MoTL language was most common in Chavacano schools, where 40 percent of students indicated they spoke a different language at home. An important policy question is, “Why are 40 percent of students attending a classroom where Chavacano was the MoTL when they did not speak that language at home?”


Mismatch of teachers’ home language was most common in Magindanawn schools, where 42 percent of teachers indicated they spoke a different language at home. Classroom observation data collected from the Language Usage Study showed that teachers in Magindanawn schools, the language with the highest percent of teachers speaking another home language, were using the MT the least in their teaching throughout the day. Appropriate placement of teachers is important and, as much as possible, teachers should be deployed based on whether they are native or highly comfortable speakers of the school’s MT. Access to Textbooks and Class Time Spent Reading and Writing

Students’ access to print materials (i.e., textbooks) and the time they spend actually practicing reading in class are important factors that can positively contribute to reading outcomes. Data collected through classroom observations and a classroom inventory as part of the Language Usage Study showed that textbooks were not used a majority of the time, and students were not spending enough time practicing reading in class. When observing MT classes across all four languages, textbooks were only used in 14 percent of Grade 2 classrooms and 25 percent of Grade 3 classrooms. However, when textbooks were being used, coverage tended to be high. Table 5 presents the percent of time that students were observed reading and writing in the MT during the MT class across all languages.

Table 5. Percentage of lesson time students spent reading and writing in the MT, by grade

Subject

Student Language

Mode

Percentage [Confidence

Interval (CI)] of Time in Grade

2 Lesson Observations

Percentage (CI) of Time in

Grade 3 Lesson Observations

MT Reading in MT 11% 14% Writing in MT 11% 15%

On average, students spent less than 30 percent of class time practicing reading and writing in the MT. This finding points to the fact that priority should be placed on ensuring students have books and supplementary reading materials in their respective MTs to facilitate learning and practicing reading and writing. Additionally, it is critical that teachers use class time dedicated to MT literacy instruction effectively.

2 Introduction

Background and Context In 2012, the Philippine Government began implementing MTB-MLE, a national effort to use the MT as the language of instruction in kindergarten, gradually introduce English and Filipino as second and third languages, respectively, and eventually transition to using English and Filipino as MoTL beginning in Grade 4.6 Previous to the rollout of this policy, the

6 The actual law in effect is the Republic Act 10533, approved in May 2013, which allows for use of the MT through Grade 6. In practice, DepEd guidance, materials, and teachers’ interpretation is that English and Filipino are the only languages used in Grade 4 and beyond.

http://www.unesco.org/education/edurights/media/docs/e119986abbd26ebda9c3d8c18929b4487205d4d6.pdf


Philippines used an immersive approach to language of instruction, beginning instruction in Filipino and English (depending on subject area) regardless of the child’s home language. Since 2013, DepEd, with the support of USAID and other donors, has invested considerable resources in preparing teachers to teach in 19 of the country’s more than 140 indigenous languages. This includes regular teacher professional development workshops, MT materials development and distribution, and strengthening teacher communities of practice in Learning Action Cells. With the rollout of the policy, it is becoming increasingly important to have access to data that provide insight into how well students read in MTs. The purpose of this study was to collect early grade reading data in four separate languages in the Mindanao regions, consisting of representative samples of schools that teach in Bahasa Sug, Chavacano, Mëranaw, and Magindanawn MTs to provide data on students’ current level of reading proficiencies in these languages. It is important to note that this study is not meant to be a measurement of whether the MTB-MLE policy is either appropriate or effective. It measures only one of many intended outcomes of using the MT as MoTL in primary school. However, the data can be used to understand whether children are ready to transition to second and third language acquisition based on sufficient linguistic and academic competency in their MT.

3 Research and Sample Design

Research Questions and Design The study was designed to answer the following research questions:

1. How well are Grade 2 pupils learning to read in Chavacano, Magindanawn, Bahasa Sug, and Mëranaw?

2. How well are Grade 3 pupils learning to read in Chavacano, Magindanawn, Bahasa Sug, and Mëranaw?

To answer these research questions, the EGRA was adapted for each language and used to measure students’ pre-reading, reading, and comprehension skills. The assessment included the following subtasks: listening comprehension, letter-sound knowledge, non-words, ORF, and reading comprehension. For more information on the subtasks included in EGRA and what each task measures, see Annex A. Following each EGRA administration, students were asked demographic and other background questions via a student questionnaire (Annex B). Some questions, such as whether students have electricity at home or access to certain consumer goods, were used as proxies to create a socioeconomic status (SES) index using a principal components analysis. A principal components analysis describes the association across variables and, subsequently, variables that are highly associated were used to create a SES index. The SES index was then split into three equal groups to classify students as having low, mid, or high SES. Student SES is reported as a relative measure within the population of sampled regions and schools. Experienced Filipino assessors from DepEd and a local survey research firm were trained on both instruments and subsequently collected data at schools from February 17 to March 7, 2019. Additional information about training and data collection can be found in Annex C. A separate study on language use in the classroom was conducted concurrently with this EGRA study. The Language Usage Study used classroom observations, classroom inventories, and teacher interviews to take an in-depth look at the language spoken by teachers and students inside and outside of the classroom. The study included the same


schools and classrooms as those that were sampled for the EGRA and, as such, focused on Bahasa Sug, Chavacano, Mëranaw, and Magindanawn MT usage. Some of the data from the language study are referenced in this report.7

Sample Design A total of 40 schools were randomly sampled for each of the four languages. Table 6 provides more details on the regional locations of sampled schools and the number of students assessed per grade for each language. The sample is based on a list, provided by DepEd, of schools in the regions that use these MTs. However, there may also be schools outside of the target regions that use the MT, and there may be schools within the regions that were not included on the list. Therefore, although the sample size was intended to be large enough to be representative of the language, it is not representative of the regions, and it may not be fully representative of all of the schools that teach using these MTs. It is only representative of the population of schools that was documented in the aforementioned list. It should also be noted that throughout the report we reference the Bangsamoro Autonomous Region of Muslim Mindanao (BARMM); however, the sample was drawn in January 2019 when the region was recognized as the Autonomous Region in Muslim Mindanao (ARMM). Subsequently, some divisions and cities that are now part of the BARMM were officially recognized as part of other regions. Within each sampled school, one Grade 2 and one Grade 3 classroom were randomly selected, and within each of the two selected classrooms, 10 students were selected at random. If there were fewer than 10 students in the selected classroom, all students present were automatically selected to participate in the assessment (Table 6). For more details on the sample methodology, see Annex D.

7 Harden, K., Sowa, P., & Punjabi, M. (2019). 2019 Language Usage Study in Bahasa Sug, Chavacano, Magindanawn, and Mëranaw mother tongue schools. Findings report prepared for USAID under All Children Reading-Philippines. Research Triangle Park, NC: RTI.


Table 6. Students sample per language

Language

Number of Grade 2

Students

Number of Grade 3

Students Region

Number of

Schools Sampled

Bahasa Sug

403 401 Region IX–Zamboanga Peninsula

15

BARMM 25

Chavacano

398 402 Region IX–Zamboanga Peninsula

39

BARMM 1

Mëranaw

403 401 Region X–Northern Mindanao

7

BARMM 33

Magindanawn 390 398 Region XII–

Soccsksargen 9

BARMM 31

4 Results by Language The results from this study, discussed in detail below, show that a significant proportion of children in schools using Bahasa Sug, Chavacano, and Magindanawn as the designated MT lack the foundational reading skills needed to, ultimately, be able to read fluently and with comprehension. Although fewer students in Mëranaw schools struggled with basic skills of alphabetic principal and decoding, there was still a proportion of students who were not able to read and comprehend grade-level text. Across all four languages, there was noticeable improvement from Grade 2 to 3, which indicates that more time and instruction in the MT impacted student performance. Key concerns across languages are high absenteeism, over-age students, ineffective use of class time, lack of textbooks, and lower performance among boys compared to girls. The subsequent sections of this report present detailed EGRA results of all four languages. Results for each language are intentionally presented in separate sections, and caution should be taken if comparing results across languages and contexts, particularly for timed measures that report scores as words or letters per minute. Each language has its own specific linguistic features that affect the expected rate of literacy acquisition and, subsequently, the rate of reading that can be considered fluent. For example, some languages consist of longer words, where another language might have broken the same concept into several words. Therefore, a reasonable expectation of word reading fluency by grade for one language might look very different for another language.


Bahasa Sug The sample of schools for Bahasa Sug included 15 schools from Region IX and 25 schools from BARMM. The final analysis includes student reading assessments and interviews from 403 Grade 2 and 401 Grade 3 students. The average age for Grade 2 students in Bahasa Sug schools was 8 years old and the average age for Grade 3 was 9. However, about 22 percent of the sampled students were over-age for their grade level. Fifty-five percent of students reported missing one or more days of school in the previous week, while 12 percent of students indicated they had been absent the entire previous week. Eighty-six percent of students and 77 percent of teachers reported speaking Bahasa Sug as their home language. Other home languages most frequently reported by the students included Bisaya/Cebuano and Yakan, while the teachers reported other common home languages of Filipino, Chavacano, and English. The highest proportion of students in the sampled Bahasa Sug-speaking schools are categorized as having low SES: 44 percent. Mid and high SES categories each account for just under 30 percent of students each. Table 7 below presents an overview of performance for Bahasa Sug-speaking students by subtask and grade. The table includes the percent of students who scored zero on a subtask and the mean scores, by grade. Zero scores are the percent of pupils who did not answer a single item correctly in a given subtask. Overall, there was a large proportion of students in Bahasa Sug-speaking schools who did not read with comprehension. More than 50 percent of students in Grade 2 and nearly 30 percent of students in Grade 3 scored zero on the reading comprehension subtask. Relatively low mean scores in the other subtasks (letter sounds and non-word reading) indicate that students lacked the foundational reading skills that help prepare students to read fluently, accurately, and with understanding. However, as expected, for almost all subtasks, there were significantly fewer zero scores for Grade 3 compared to Grade 2, while mean scores were higher for Grade 3 when compared to Grade 2. This means children were, on average, improving their skills from one grade to the next. Results by subtask are explained in detail in the subsequent sections.

Key Sample Descriptives Grade 2: 403 students assessed; average age 8 years old Grade 3: 401 students assessed; average age 9 years old Over-age for grade: 22% of students Absenteeism: 55% of students were absent one or more days in previous week SES: 44% high SES, 29% mid SES, 27% low SES Student home language: 86% speak Bahasa Sug Teacher home language: 77% speak Bahasa Sug


Table 7. Overview of Bahasa Sug EGRA percent zero and mean scores, by grade

Subtask

Percent Zero Scores [Margin of Error]

Mean Scores [Margin of Error]

Grade 2 (n = 403)

Grade 3 (n = 401)

Grade 2 (n = 403)

Grade 3 (n = 401)

Listening Comprehension (5 questions)

3.3% [±1.9]

3.4% [±2.2]

68.9% [±5.4]

67.4% [±3.8]

Letter-Sound Identification (clspm) (100 items)

36.6% [±8.9]

23.6% [±7.9]

12.1 [±2.9]

16.3 [±3.6]

Non-Word Reading (correct invented words per minute) (50 items)

42.0% [±7.4]

22.6% [±7.3]

10.2 [±2.0]

19.0 [±2.9]

ORF Passage Reading (cwpm) (47 items)

44.0% [±8.3]

24.2% [±8.0]

17.8 [±3.7]

35.6 [±5.7]

Reading Comprehension: Passage 2 (5 questions)

55.5% [±8.4]

31.9% [±8.0]

27.4% [±5.6]

49.2% [±6.7]

Listening Comprehension Listening comprehension is a pre-reading skill that measures oral fluency and vocabulary, which can be important to gauge whether students are able to actively listen and understand the language the teacher uses as he/she teaches. A short story is read aloud to the child, who is then asked five questions about the story; the Bahasa Sug story and comprehension questions are presented below in Figure 3. The performance across the two grades was the same—very few students scored zero on this subtask, only 3 percent in Grade 2 and in Grade 3. For both grades, on average, students correctly answered between three and four of the five questions correctly. Cumulative distribution of scores by grade are available in Annex E.


Figure 3. Bahasa Sug listening comprehension story and questions

Questions assessor asks the student 1. Hisiyu in nagtangis? 2. Mayta nagtangis in

anak-manuk? 3. Hisiyu in timabang

ha anak-manuk? 4. Unu in kita sin anak-

manuk iyaanud ha sapa?

5. Unu in hinang sin anak- manuk iban ambak pagkita nila sin batangkahuy?

Bassahan ta kaw hawpu suysuy, patanugun ku iban makaminsan ku sadja bassahun, pag-ubus awn pangasubu ku. Tumaynghug marayaw. Pag-ubus sambungan in manga pangasubu. Kiyahatihan mu na? In anak-manuk nahulug pa sapa. “Tabangi aku”, tangis sin anak-manuk. Timabang in ambak gaddung, sa, nahulug da isab siya. “Unu na in hinangun ta?” laung sin ambak. “Atud kaw! Awn batangkahuy makatabang katu bat kita di malumus, sambung sin anak- manuk. Pag-ubus, dimag na sila pa batangkahuy. Nakaulak na sila iban pag-iyan, “lappas na kita.”

On average, Grade 2 students answered 69% of listening comprehension questions correctly.


Letter-Sound Identification Knowledge of letter-sound correspondence is a fundamental skill that helps children learn to decode words and has been shown to be an early strong predictor of reading fluency, particularly for transparent orthographies (i.e., each letter is associated with a unique sound). For the letter-sounds subtask, students were presented a sheet of paper with 100 letters randomly ordered, using a mix of uppercase and lowercase forms. Figure 4 shows this grid. The subtask was discontinued for students who did not correctly identify at least one item within the first row; these students proceeded to the next subtask. Out of 100 items presented, on average, Grade 2 children were able to correctly identify 12 letter sounds in 60 seconds, while Grade 3 scores were only slightly higher at 16 clspm. Zero scores accounted for a large proportion of Grade 2 scores (about 37 percent). The percent of students scoring zero in Grade 3 was much lower, about 24 percent. The high percent of zero scores and low mean scores show that even after two and three years of schooling, children were still lacking this critical foundational skill that should be mastered by students early on.


Figure 4. Bahasa Sug letter-sound identification subtask student stimulus sheet

S n i U K B h Ny M A

i r n H s ch T g B L

G Ng u D k T Y p W r

d Y P w R Ny h K D n

U h s m B L G y p NG

J ch n d N T i h ny L

w r u H D W CH P a n

b t L G d K j N i S

T u P ng A D g m r h

ny S B D y P ch i N M

On average, students in Grade 2 attempted the first 15 items and got 12 correct in one minute.


Another way to understand performance is to remove the scores of students who were unable to read a single letter (i.e., zero scores). The average clspm score for children who could read at least one letter was only slightly different, with students in Grade 2 and Grade 3 reading 19 clspm and 22 clspm, respectively. This shows that the average reading ability cited above is not low just because of non-readers, but is, in fact, low within the whole population. The lack of progress between Grade 2 and Grade 3 may be due to the fact that the Grade 3 curriculum no longer focuses on teaching letter sounds, and students who do not learn them in Grades 1 and 2 are not likely to learn or improve on letter-sound identification once they are in Grade 3. Item-level analysis indicates that students struggled the most with the digraph sounds: /ng/, /ny/, and /ch/. It should be noted that since 2017, there has been an ongoing process to review and finalize the Bahasa Sug orthography. At the time the assessment was adapted and administered, /ch/ and /ny/ were part of the orthography, however, these letters have since been removed because they are infrequently used and predominately used in words that are borrowed from other languages.

Non-Word Reading The non-words subtask is used to measure decoding skills. It requires students to draw on their knowledge of letter-sound correspondence to read words that are unfamiliar but follow the orthographic rules and patterns of actual words. The subtask was discontinued for students who did not correctly identify at least one item within the first row; these students proceeded to the next subtask. Given the relationship of the skills needed for letter sounds and the non-words subtasks, it is not surprising that the percent zero scores and mean scores are similar between the two subtasks. On average, students in Grade 2 read 10 correct non-words per minute (cnonwpm) and Grade 3 students read 19 cnonwpm. High percentages of students in both grades were unable to read a single non-word: 42 percent in Grade 2 and 23 percent in Grade 3. If we exclude students who were unable to read a single word from the calculation, the averages are still relatively low but increase slightly to about 18 cnonwpm for Grade 2 and about 25


cnonwpm for Grade 3. Not surprisingly, the non-words that students struggled with the most contained /ng/ and /ny/. These findings are consistent with results from the letter-sounds subtask, which indicated digraph sounds were the most commonly missed by students. Girls outperformed boys in non-word reading, especially in Grade 3. On average, Grade 3 girls were reading 7 cnonwpm faster than boys (22 and 15 cnonwpm, respectively). Differences were also statistically significant in zero scores with a lower percent of girls (18 percent) scoring zero compared with boys (29 percent).

ORF Passage Reading Fluent reading depends on progressing from decoding to automatically recognizing words and also requires children to draw inferences and anticipate words based on the context of the sentence and story. The shift from decoding to automaticity is important to free up cognitive space for comprehension. The ORF passage reading subtask was presented in the form of a short, grade-level story. The subtask was administered according to the EGRA Toolkit, Second Edition8 standard procedures, such that students were given 60 seconds to read as quickly and accurately as possible. The subtask was discontinued for students who did not correctly identify at least one word within the eight words; these students proceeded to the next subtask. The distribution of scores by grade are presented in Figure 5. Cumulative distributions by grade can be found in Annex E. It is expected that fluency (a measure of automaticity and accuracy together) on the passage reading subtask is higher than in non-words subtask as passage reading presents common words in the form of sentences, which is a format that students are often more accustomed to. On average, students in Grade 2 read about 18 cwpm, while students in Grade 3 read twice as fast at 36 cwpm. However, as can be seen in Figure 5, the largest proportion of students were unable to read a single word, especially Grade 2 students (about 44 percent). The remainder of students are distributed across categories and centered around 31–40 cwpm (12 percent in Grade 2 and 17 percent in Grade 3).

Figure 5. Distribution of Bahasa Sug ORF, by grade (Passage 1)

8 For more information, see the EGRA Toolkit at http://shared.rti.org/content/early-grade-reading-assessment-egra-toolkit-second-edition

44%

8%10% 9%

12%

8%6%

3%1% 0% 0% 0%

24%

3%

7% 7%

17%

13% 11%

7%4% 4%

1% 2%

0%

10%

20%

30%

40%

50%

60%

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 101-110

Perc

ent o

f Stu

dent

s

Correct Words per Minute

Grade 2

Grade 3

http://shared.rti.org/content/early-grade-reading-assessment-egra-toolkit-second-edition

http://shared.rti.org/content/early-grade-reading-assessment-egra-toolkit-second-edition


As depicted in Figure 6, on average, Grade 2 students attempted 23 words in 60 seconds (unshaded words in the figure below) and, of those words, they correctly read only 10 words (46 percent accuracy). In Grade 3, depicted in Figure 7, on average students were reading slightly faster, attempting about 36 words within 60 seconds, (unshaded words in the figure) and with a higher level of accuracy, reading about 25 words correctly (70 percent accurately). In the figures below, the words most commonly pronounced correctly within the average attempted items are indicated in red. These words tended to be shorter words—between two and four letters long—especially for Grade 2 students.

Figure 6. Average attempted items and accuracy in Bahasa Sug, Grade 2

ORF Passage

On average, Grade 2 students attempted the first 23 words and read with 46% accuracy.

In ngan ku hi Abdul. Pitu tahun na in ummul ku. In ngan sin taymanghud ku usug hi Gulam. Awn da isab taymanghud ku babai. Naghuhula kami ha gimba. In ama ku mag-uuma. In pamilya ku magkadtu pa tabu magdagang bungangkahuy bang adlaw Sabtu. Ha hinapusan sin adlaw, magpakausaha kami sin. Makuyag tuud aku bang adlaw Sabtu na.

Figure 7. Average attempted items and accuracy in Bahasa Sug, Grade 3

ORF Passage


In ngan ku hi Abdul. Pitu tahun na in ummul ku. In ngan sin taymanghud ku usug hi Gulam. Awn da isab taymanghud ku babai. Naghuhula kami ha gimba. In ama ku mag-uuma. In pamilya ku magkadtu pa tabu magdagang bungangkahuy bang adlaw Sabtu. Ha hinapusan sin adlaw, magpakausaha kami sin. Makuyag tuud aku bang adlaw Sabtu na.

Analyzing the mean of non-zero scores helps us to better understand the performance for students who were able to read one or more words correctly. The 55 percent of Grade 2 students and 76 percent of Grade 3 students who were able to read at least one word in Bahasa Sug were doing so with a high percent of accuracy, achieving 82 percent accuracy in Grade 2 (n = 218) and about 92 percent accuracy in Grade 3 (n = 302). As shown in Table 8, girls read a passage of text more fluently than boys in both grades. For example, on average, ORF in Grade 3 for girls is 11 cwpm higher than for boys.


Table 8. Bahasa Sug ORF scores, by gender Grade 2 Grade 3

Boys (n = 98)

Girls (n = 205)

Boys (n = 196)

Girls (n = 205)

Percent Zero Scores

49% [±9.8]

39%* [±8.9]

30% [±9.6]

19%* [±8.6]

Mean Scores 15 [±3.8]

20 [±4.4]

30 [±5.4]

41** [±6.8]

* Statistical significance is p<0.05 ** Statistical significance is p<0.001

Reading Comprehension To measure reading comprehension, students orally responded to questions for two short, grade-level passages. For Passage 1 (58 words), students were given 60 seconds to read the passage, after which it was removed from in front of the student as they responded to questions (this is per the reading comprehension administration protocol, as described in the EGRA Toolkit, Second Edition). However, in Passage 2 (71 words), students were given an extended time of 180 seconds to read the passage and the story remained in front of them as they responded to the comprehension questions. Each passage is composed of five comprehension questions (four literal and one inferential question). Students are only asked questions that correspond with text they read in the story before the time ran out. The administration protocol for Passage 2 is preferable due to the fact that it is not tied to a 60-second time limit. This section discusses comprehension scores from Passage 2, while results from Passage 1 can be found in Annex E. On average, students in Grade 2 answered 27 percent of the reading comprehension questions correctly, while Grade 3 students, on average, answered 49 percent correctly. Students in Grade 2 attempted between one to two questions, while Grade 3 students attempted about three questions. Mean scores were higher for percent correct out of attempted items in both grades: 44 percent in Grade 2 and 69 percent in Grade 3. However, even with three minutes to complete the task, only 39 percent of Grade 2 and 69 percent of Grade 3 students read the entire story and attempted all five questions. Figure 8 presents the distribution of student reading comprehension scores by grade. The percent scores on the x-axis correspond to the number of questions a student answered correctly; 20 percent is equivalent to one question correct, 40 percent is equivalent to two questions correct, etc. The percent of students who scored zero is notably quite high for both grades (55 percent for Grade 2 and 32 percent in Grade 3). The high percent of zero scores for reading comprehension are a reflection of the high zero scores on the ORF subtask. Those students in Grade 3 who could read with some fluency were able to comprehend at expected levels and answer four or five out of five questions correctly (80 percent comprehension or better).


Figure 8. Bahasa Sug reading comprehension scores, by grade (Passage 2)

As they did in other subtasks, girls performed better compared to boys in reading comprehension. Only 25 percent of Grade 3 girls scored zero compared to 40 percent of boys. On average, girls in Grade 3 scored 55 percent compared with boys at only 43 percent. Grade 2 comparisons are similar, with girls scoring 31 percent and boys scoring 24 percent.

Chavacano Nearly all of the schools sampled for Chavacano were located in Region IX–Zamboanga Peninsula (29); the remaining one school was located in BARMM. The final analysis includes student reading assessments and interviews from 398 Grade 2 and 402 Grade 3 students. The average age for Grade 2 students in sampled Chavacano-speaking schools was 8 years old and the average age for Grade 3 was 9 years old. About 12 percent of students were over-age for their grade. More than 60 percent of sampled students reported being absent one or more days in the previous week, with the largest proportion of those students (24 percent) indicating they were absent two days in the previous week. About 60 percent of sampled students from both grades reported speaking Chavacano at home with their parents. Filipino/Tagalog, Bahasa Sug, and Bisaya/Cebuano were the other languages most frequently reported by students as languages they spoke at home with their parents. Eighty-four percent of sampled Grade 2 and Grade 3 teachers in Chavacano schools reported speaking Chavacano at home; the other commonly reported languages were Filipino and English.

55%

9% 8% 8% 8%11%

32%

6%9%

11%

21% 21%

0%

10%

20%

30%

40%

50%

60%

0 20 40 60 80 100

Perc

ent o

f Stu

dent

s

Percent Score

Grade 2

Grade 3

Key Sample Descriptives Grade 2: 398 students assessed; average age 8 years old Grade 3: 402 students assessed; average age 9 years old Over-age for grade: 12% of students Absenteeism: 60% of students were absent one or more days in the previous week SES: 52% high SES, 30% mid SES, and 18% low SES Student home language: 60% speak Chavacano Teacher home language: 84% speak Chavacano


More than half (52 percent) of the assessed students fell into the high SES category, about 30 percent were categorized as having mid SES, and the remaining 18 percent were in the low SES category. Table 9 below presents an overview of students’ performance by subtask and grade. The table includes the percent of students who scored zero on the subtask and the estimated mean scores, by grade. As expected, zero scores declined in Grade 3, while mean scores increased, compared to Grade 2. However, zero scores accounted for a sizable proportion of students, especially in Grade 2 (between 20 percent and 40 percent, depending on subtask). Zero scores were notably high on the listening comprehension subtask, which may be related to the percent of students who spoke other MTs at home. Results by subtask are explained in detail in the subsequent sections.

Table 9. Overview of Chavacano EGRA percent zero and mean scores, by grade

Subtask

Percent Zero Scores [margin of error]

Mean Scores [margin of error]

Grade 2 (n = 398)

Grade 3 (n = 402)

Grade 2 (n = 398)

Grade 3 (n = 402)


40.7% [±6.9]

36.5% [±7.0]

26.8% [±4.6]

30.9% [±5.2]


21.3% [±6.5]

13.8% [±5.5]

19.1 [±3.1]

17.8 [±2.0]

Non-Word Reading (cnonwpm) (50 items)

28.3% [±6.5]

23.4% [±7.1]

14.7 [±1.8]

19.9 [±2.7]


24.1% [±6.0]

13.1% [±5.3]

34 [±4.2]

50.4 [±6.8]


36.7% [±6.6]

28.1% [±7.1]

45% [±6.0]

54.8% [±6.9]

Listening Comprehension The Chavacano story and comprehension questions are presented below in Figure 9. The story and five questions were read aloud to the students; no stimulus was given to the students for this subtask. Given that listening comprehension is a pre-reading skill, it can be an important measure that reflects students’ oral fluency and vocabulary. A large portion of students in both grades scored zero in this subtask, 40 percent in Grade 2 and 36 percent in Grade 3. These students were unable to answer a single question correctly. On average, mean scores were low; students in Grade 2 scored 27 percent, and students in Grade 3 scored 30 percent. This means, on average, students in both grades were answering only about two out of the five questions correctly. The low mean scores and high percent of zero scores could be due, in part, to the percent of children reporting speaking languages other than Chavacano at home, which could hinder students’ oral language ability in Chavacano. Additionally, it is possible that the length of the story may have made it challenging for students to recall details. Cumulative distribution graphs are presented in Annex F.


Figure 9. Chavacano listening comprehension story and questions

Questions assessor asks the student 1. Cosa ta hace si

Ñor Dodong? 2. Cosa le ya sabe

cuando ta mira le televisión?

3. Porque ya espanta si Ñor Dodong?

4. Cosa ya hace si Ñor Dodong y su familia?

5. Porque ya informa si Ñor Dodong con su vecinos?

Ta mira television si Ñor Dodong. Ya espanta le kay asegun con el noticia, un mafuerza tifon hay tupa na de ila lugar. Ya anuncia tambien que el gobierno ta manda evacua con el maga residentes na de ila lugar. Con apuro, ya aregla el familia de Ñor Dodong con el de ila maga importante cosas. Pero antes sila de sale, ya informa si Ñor Dodong con el de ila maga vecinos por causa del anuncio, pero no hay estos cre con ele.



Letter-Sound Identification Knowledge of letter-sound correspondence is a fundamental skill that helps children learn to decode words. It has been shown to be a strong predictor of reading fluency, particularly for transparent orthographies (i.e., each letter is associated with a unique sound). For the letter-sounds subtask, students were presented with a sheet of paper that contained 100 letters that were randomly ordered, using a mix of uppercase and lowercase forms (see Figure 10). The subtask was discontinued for students who did not correctly identify at least one item within the first row; these students proceeded on to the next subtask. Out of items presented, on average, Grade 2 children identified 19 clspm, while Grade 3 students had a similar score, at 18 clspm. Although there was little difference in the mean scores, there was a notable difference in the percent zero scores between the two grades. About 21 percent of Grade 2 students were not able to provide the sound of a single letter. Zero scores were less common in Grade 3, accounting for only about 14 percent of students. Item level analysis indicates that students struggled most with diagraph sounds /ch/ and /ll/ followed by the sounds /j/ and /b/.


Figure 10. Chavacano letter-sound identification subtask student stimulus sheet

Z g F d T ch E b i ll

M o R y J s W ñ P v

K u N r X Ll a Q h C

L j Ñ w F z P c Y i

i e K s H d Q o CH t

N b E u V a G m X r

z i a n e i n Y a h

A D o C A L n A r p

E o i A s t E o u r

A t a e m L ch s a b

On average, students in Grade 2 and Grade 3 attempted the first 26 items and got 18 correct.

Non-Word Reading The non-words subtask is used to measure decoding skills. It requires students to draw on knowledge of letter-sound correspondence to read words that are unfamiliar but follow the orthographic rules and patterns of actual words. The subtask was discontinued for students who did not correctly identify at least one item within the first row; these students proceeded on to the next subtask. Nearly 28 percent of Grade 2 and about 23 percent of Grade 3 students were not able to read a single non-word. On average, students in Grade 2 read about 15 cnonwpm, while students in Grade 3 performed slightly better reading at 20 cnonwpm. If we exclude students who were unable to read a single word from the calculation, the averages are still relatively low but increase slightly to about 21 cnonwpm for Grade 2 and about 26 cnonwpm for Grade 3. These low scores indicate that students were not learning the most fundamental reading skills (e.g., letter sounds and decoding) that play an important role in becoming a fluent reader. Not surprisingly, some of words that students struggled with the most were words containing the letters “ch”, “j,” or “ñ.” These findings are consistent with results from the letter-sounds subtask, which indicated these sounds were the ones students had the most difficulty pronouncing correctly.

ORF Passage Reading Fluent reading depends on progressing from decoding to automatically recognizing words and also requires children to draw inferences and anticipate words based on the context of the sentence and story. The shift from decoding to automaticity is important to free up cognitive space for comprehension. The ORF passage reading subtask was presented in a short, grade-level story. The subtask was administered according to the EGRA Toolkit, Second Edition procedures, such that students were given 60 seconds to read as quickly and accurately as possible. The subtask was discontinued for students who did not correctly identify at least one word within the first eight words; these students proceeded on to the next subtask. Distribution of scores by grade are presented in Figure 11. Cumulative distributions by grade can be found in Annex F. It is expected that fluency (a measure of


automaticity and accuracy together) on the ORF passage reading subtask is higher than in the non-words subtask as passage reading presents common words in the form of sentences, which is a format that students are often more accustomed to. On average, Grade 3 students read 16 cwpm faster and more accurately than students in Grade 2 (34 cwpm and 50 cwpm, respectively). However, as can be seen in Figure 11, the largest proportion of students were unable to read a single word, especially Grade 2 students (about 24 percent). For Grade 2, the second largest proportion of students were reading 41–50 cwpm and 51–60 cwpm, with 11 percent of students in each category. Grade 3 students, however, had a higher proportion of students in the 71–80 and 81–90 categories, each with 11 percent of students.

Figure 11. Distribution of Chavacano ORF, by grade ( Passage 1)

As indicated in Table 10, girls outperformed boys with lower percent zero scores and higher mean scores. On average, Grade 2 girls read about 11 cwpm faster than boys, and Grade 3 girls read nearly 19 cwpm faster than boys.

Table 10. Chavacano ORF scores, by gender

Grade 2 Grade 3

Boys (n = 199)

Girls (n = 199)

Boys (n = 200)

Girls (n = 202)

ORF Percent Zero Scores 25.8% [±7.7]

22.5% [±7.2]

17.7% [±6.5]

8.9% ** [±6.3]

ORF Mean Scores 28.5 [±5.1]

39.2 ** [±4.9]

40.7 [±6.5]

59 ** [±7.7]

** Statistical significance is p<0.001

As depicted in Figure 12, on average, Grade 2 students attempted 34 words in 60 seconds (indicated in gray shading), of those words, they correctly read 23 words (67 percent

24.1%

7.2%9.6% 9.5%

6.7%

10.8% 11.1%7.3% 6.6%

4.0%1.2% 1.9%

13.1%9.8%

7.3%4.1% 5.3% 6.3%

8.3%10.5% 11.3% 10.8%

6.2% 7.1%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 101-110

Perc

ent o

f Stu

dent

s


Grade 2

Grade 3


accuracy). In Grade 3, depicted in Figure 13, on average, students attempted about 42 words within 60 seconds (indicated in gray shading) and read 32 of them accurately (77 percent accuracy). In the figures below, the words most commonly pronounced correctly within the average attempted items are indicated with in red.

Figure 12. Average attempted items and accuracy in Chavacano, Grade 2

ORF Passage


Ya encontra si Victor cinco peso na camino. Ya pipina le con este y ya pensa cosa puede compra con el cinco peso. En camino, ya mira le con Mila. Ta llora este. Ya habla este que ya perde de suyo cinco peso. Ya dale si Victor el cen con Mila. Ya recibe este con el cen y ya calla.

Figure 13. Average attempted items and accuracy in Chavacano, Grade 3

ORF Passage


Ya encontra si Victor cinco peso na camino. Ya pipina le con este y ya pensa cosa puede compra con el cinco peso. En camino, ya mira le con Mila. Ta llora este. Ya habla este que ya perde de suyo cinco peso. Ya dale si Victor el cen con Mila. Ya recibe este con el cen y ya calla.

When referring to means (averages) to understand student performance, it can be useful to also look at performance excluding students scoring zero, which tends to significantly decrease the mean. This is especially true in the case of Chavacano, where zero scores amounted to 24 percent of Grade 2 and 13 percent of Grade 3 students, as shown in in Figure 11. Among children who could read at least one word correctly in the first eight words, on average Grade 2 children read 44 cwpm and Grade 3 students read 57 cwpm. These students also read with a higher percentage of accuracy, achieving 88 percent accuracy in Grade 2 (n = 294) and about 89 percent accuracy in Grade 3 (n = 392).

Reading Comprehension To measure reading comprehension, students responded to questions for two short, grade-level passages. For Passage 1 (60 words), students were given one minute to read, after which the passage was removed from in front of the student as they responded to questions. However, for Passage 2 (56 words), students were given extended time—three minutes—to read the passage, and the story remained in front of them as they responded to the comprehension questions. Following each passage, five comprehension questions (four literal and one inferential question) were read aloud to the students to respond to orally. Students were only asked questions that corresponded with text they read in the story before the time ran out. The administration protocol for Passage 2 is preferable as it is not tied to a


60-second time limit. This section discusses comprehension scores from Passage 2; results from Passage 1 can be found in Annex F. Figure 14 presents the distribution of scores for the reading comprehension subtask for Passage 2. The percent scores on the x-axis correspond to the number of questions a student answered correctly: 20 percent is equivalent to one question correct, 40 percent is equivalent to two questions correct, etc. Percent zero scores were notably quite high for both grades (37 percent for Grade 2 and 28 percent in Grade 3). The high percent of zero scores for reading comprehension are a reflection of the high zero scores in the ORF subtasks. On average, Grade 2 students answered 45 percent of the questions correctly, while students in Grade 3 performed slightly better, answering 55 percent of the questions correctly. Although a large proportion of students were unable to answer a single question, another proportion of students who read with some fluency reached 80 percent comprehension or better: about 50 percent of Grade 3 students and 36 percent of Grade 2 answered 80 percent (or more) of the comprehension question correctly.

Figure 14. Chavacano reading comprehension scores, by grade (Passage 2)

Girls’ comprehension scores were higher than boys’, especially in Grade 3. Girls in Grade 3 scored 14 percentage points better than boys, and only 22% of girls scored zero, compared to 35% of boys.

37%

4%

13%10%

14%

22%

28%

5%8% 9%

24%26%

0%

10%

20%

30%

40%

50%

60%

0 20 40 60 80 100

Perc

ent o

f Stu

dent

s

Percent Score

Grade 2

Grade 3


Magindanawn A total of 40 schools that use Magindanawn as the MoTL were visited for the 2019 Regional EGRA; 31 of the schools were located in BARMM, and the remaining 9 were located in Region XII. The final analysis includes student reading assessments and interviews from 390 Grade 2 and 398 Grade 3 students. The majority of sampled Grade 2 students were 8 years old and the majority of Grade 3 students were 9 years old; however, about 25 percent were over the expected age for their respective grade (age 8 for Grade 2 and age 9 for Grade 3). About 58 percent of sampled students reported being absent one or more days in the previous week. About 87 percent of the students reported speaking Magindanawn most often at home. The second most common home language among students was Iranum, accounting for about 10 percent of the sampled students. However, only about 58 percent of sampled teachers reported that Magindanawn was their MT, with Filipino/Tagalog being the second most common home language at 18 percent. Thirty-nine percent of students were categorized as having low SES, 36 percent with mid SES, and the remaining 25 percent of students fell into the high SES category. Students were asked to complete several EGRA subtasks as part of the assessment, and Table 11 below presents an overview of performance by subtask and grade. The table includes the percent of students who scored zero on the subtask and the estimated mean scores by grade. Zero scores are the percent of pupils who did not answer a single item correctly in the given subtask. As expected, for almost all subtasks, zero scores declined in Grade 3 when compared to Grade 2, while mean scores increased. However, overall, there was a large proportion of students in Magindanawn-speaking schools who were not reading with comprehension. This is evident based on the more than 50 percent of students in Grade 2 and nearly 30 percent of students in Grade 3 who scored zero on reading comprehension subtasks. Relatively low mean scores in the other subtasks (listening comprehension, letter sounds, and non-word reading) indicate that students lacked foundational reading skills, which help prepare students to read fluently, accurately, and with understanding. Performance from the 2014 EGRA study of Grade 2 Magindanawn-speaking students is referenced in the sections below, as the same 2014 EGRA assessment was used in 2019 for Grade 2 and Grade 3 students in the Magindanawn language group.9 Results by subtask are explained in detail in the subsequent sections.

9 Pouezevara, S. DeStefano, J., Cummiskey, C., & Pressley, J. (2014). Early Grade Reading Assessment Results: A cross-language look at MTB-MLE implementation in the Philippines. Prepared for USAID under the Education Data for Decision Making (EdData II) project, Task Order No. AID-492-M-12-0000. Research Triangle Park, NC: RTI.

Key Sample Descriptives Grade 2: 390 students; average age 8 years old Grade 3: 398 students; average age 9 years old Over-age for grade: 25% of students Absenteeism: 58% of students were absent one or more days in previous week SES: 25% high SES, 36% mid SES, 39% low SES Student home language: 87% speak Magindanawn Teacher home language: 58% speak Magindanawn


Table 11. Overview of Magindanawn EGRA percent zero and mean scores, by grade

Subtask



Grade 2 (n = 390)

Grade 3 (n = 398)

Grade 2 (n = 390)

Grade 3 (n = 398)


24.5% [±5.7]

15.7% [±4.8]

36.1% [±5.2]

44.1% [±4.0]


25.8% [±12.2]

12.1% [±5.2]

13.7 [±3.3]

17 [±3.4]


30.0% [±9.9]

15.2% [±6.5]

13.6 [±3.0]

21.7 [±3.1]


28.1% [±9.6]

13.7% [±5.9]

19.2 [±4.3]

34.5 [±5.2]


52.4% [±10.3]

30.1% [±8.6]

22.7% [±5.9]

40.8% [±6.1]


52.7% [±10.1]

26.9% [±8.1]

23.1% [±5.2]

38.6% [±6.3]

Listening Comprehension In this subtask, a short story is read aloud to the child, followed by five comprehension questions, as presented below in Figure 15. This subtask measures students’ oral language ability and vocabulary knowledge. About 25 percent of Grade 2 students and 16 percent of Grade 3 students were not able to answer a single listening comprehension question correctly. For both grades, on average students answered two to three out of the five questions correctly. Based on classroom observation data from the Language Usage Study, Maguindanaoan teachers were recorded as using the MT less, compared to teachers of other languages. During the observations, Maguindanaoan teachers mixed in a fair amount of Filipino, even when teaching the MT subject. This could be due to the high proportion of teachers for whom Magindanawn is not their home language, as presented above. This lack of exposure to the MT could, in part, affect students’ abilities to listen and understand the MT.


Figure 15. Magindanawn listening comprehension story and questions

Questions assessor asks the student 1. Ngin i ngala nu ayam ni Rodel? 2. Enduken ka migkalimeng su puting/bedung? 3. Ngin i pinggula ni Rodel? 4. Ngin i pinggula ni Rodel endu si Muning guna su migkapya si Muning? 5. Ngintu ka migkagalaw si Muning?

Saki si Rodel. Aden ayam ku a puting/bedung. Ya nin ngala na Muning. Sungkulsungkul nin bu. Isa a gay, nagaipan ku ka masu malimeng sekanin. Aden besen sakit nin. Tiniakapan ku taman sa migkapya sekanin. Iganat kanu entuba na magalaw kami ni Muning pendadalement lu kanu walay.



Letter-Sound Identification Knowledge of letter-sound correspondence is a fundamental skill that helps children learn to decode words and has been shown to be a strong predictor of reading fluency, particularly for transparent orthographies (i.e., each letter is associated with a unique sound). For the letter-sounds subtask, students were presented a sheet of paper with 100 letters that were randomly ordered, using a mix of uppercase and lowercase forms. Figure 16 shows this grid. The subtask was discontinued for students who did not correctly identify at least one item within the first row; these students proceeded on to the next subtask. Out of items presented, on average, Grade 2 children identified an average of 14 clspm, while Grade 3 children identified an average of 17 clspm. Zero scores for this subtask were 26 percent for Grade 2 students and 12 percent for Grade 3 students. Excluding zero scores from the analysis, the average increased only slightly, to 18 clspm in Grade 2 and 19 clspm in Grade 3. This lack of improvement from Grade 2 to Grade 3 may be due to the fact that this beginning reading skill is no longer practiced in Grade 3.


Figure 16. Magindanawn letter-sound identification subtask student stimulus sheet

D N i y m L P A b T

K s ng G Y e W u S L

t a k g i Ng p E B d

M U n s u n w P A u

L i L n m g N S u e

N E b a d ng w n T S

D i s y u W N d i N

k t K e n U i K U E

T g p N G Y M G k L

K ng n B P M g n U i



Item-level analysis shows that students struggled most with the sounds /y/, /ng/, and /e/. In Magindanawn, the letter “e” is pronounced as /ə/ (i.e., schwa sound). It is possible that children are more familiar with and provided the Filipino sound for this letter (i.e., /e/) given the two languages use the same grapheme. The letter “y” is rarely used at the beginning of words in Magindanawn and is more frequently found at the end of the word. Therefore, it may be harder for children to produce the sound of the letter when asked to use it outside of a familiar word. In the 2014 study, many students struggled with these same three letter sounds. Students’ inability to correctly identify and pronounce these letter sounds leads to difficulties in correctly pronouncing words containing them, which is described in more detail in the sections below.

Non-Word Reading The non-word reading subtask is used to measure decoding skills. It requires students to draw on knowledge of letter-sound correspondence to read words that are unfamiliar but follow the orthographic rules and patterns of actual words. The subtask was discontinued for students who did not correctly identify at least one item within the first row; these students proceeded on to the next subtask. The percent zero scores and mean scores for the letter sounds and the non-words subtasks were quite similar for Grade 2 and Grade 3. About 30 percent of Grade 2 students and 15 percent of Grade 3 students were unable to read a single non-word. On average, students in Grade 2 read about 14 cnonwpm, while students in Grade 3 performed somewhat better, reading at 22 cnonwpm. If we exclude students who were unable to read a single word from the calculation, the averages were still relatively low but increased slightly to about 20 cnonwpm for Grade 2 and about 26 cnonwpm for Grade 3. These low mean scores indicate that students are not learning the most fundamental reading skills (e.g., letter sounds and decoding) that play an important role in becoming a fluent reader.


Not surprisingly, the invented words that students struggled with the most were words containing the letter “e.” These findings are consistent with results from the letter-sounds subtask, which indicated that the letter-sound /e/ was most frequently missed by students.

ORF Passage Reading Fluent reading depends on progressing from decoding to automatically recognizing words and also requires children to draw inferences and anticipate words based on the context of the sentence and story. The shift from decoding to automaticity is important to free up cognitive space for comprehension. To measure ORF, a short, grade-level story is placed in front of the student. The subtask was administered according to the EGRA Toolkit, Second Edition procedures, such that students were given 60 seconds to read the passage as quickly and accurately as possible. The subtask was discontinued for students who did not correctly identify at least one word within the first eight words; these students proceeded on to the next subtask. On average, Grade 2 students read 19 cwpm and Grade 3 students read 35 cwpm. Table 12 below disaggregates the proportion of zero scores and mean ORF by gender and grade. Although there was very little difference in the percentages of boys and girls who scored zero, girls scored significantly better than boys on ORF, especially in Grade 3.

Table 12. Magindanawn ORF scores, by gender (Passage 1)

Grade 2 Grade 3 Boys

(n = 181) Girls

(n = 209) Boys

(n = 199) Girls

(n = 199) ORF Percent Zero Scores 28.4%

[±10.2] 27.8% [±11.8]

15.3% [±7.5]

12.4% [±6.5]

ORF Mean Scores (cwpm) 17.4 [±4.3]

20.9 ** [±4.9]

28.6 [±5.3]

39.3 ** [±5.6]

** Statistical significance is p<0.05

The largest proportion of Grade 2 students were unable to read a single word, as seen in Figure 17 below, whereas the largest proportions of Grade 3 students read between 41 and 50 cwpm and 51 and 60 cwpm. Cumulative distributions, by grade, for both passages can be found in Annex G.


Figure 17. Distribution of Magindanawn ORF, by grade (Passage 1)

The most commonly missed words for both Grades 2 and 3 were: endu, nageda, and Pendagangen for Passage 1 and aden, sekanin, mindalemet, and kapendalemet for Passage 2. The letter “e” is present in all of these words, which aligns with the poor performance for letter sound /e/ in the letter-sounds subtask. The 2019 Magindanawn EGRA used the same passage as the 2014 Magindanawn study (Passage 1 for both years). In 2014, Grade 2 students read at an average of 20.9 cwpm, while in 2019 Grade 2 students had a similar ORF of 19.2 cwpm. Figure 18 below compares the distribution of performance in both years. Although the mean scores are similar from 2014 to 2019, an important and positive change is the reduction in zero scores from 38 percent in 2014 to 28 percent in 2019. Subsequently, there is also an increase in the proportion of students who read between 1 and 10 cwpm in 2019 (28 percent) compared to 2014 (3 percent). The ORF benchmarks set for Magindanawn in 2014 (after the 2014 Magindanawn EGRA study was completed) recommended that Grade 2 students should read at 40 cwpm. In 2014, 22 percent of Grade 2 students met this benchmark. Using Figure 17 above, we can see that only 12 percent of Grade 2 students met the benchmark in 2019, meaning there was a decrease from 2014 by 10 percent.

28%

14%11%

17% 17%

8%3% 1% 0% 0% 0%

14%

8% 10% 10%

16% 16%13%

7%5%

1% 1%0%

10%

20%

30%

40%

50%

60%

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100

Perc

ent o

f Stu

dent

s


Grade 2

Grade 3


Figure 18. Distribution of Magindanawn ORF for Grade 2, 2014 vs. 2019 (Passage 1)

Of those who demonstrated some reading ability in the 2019 study (removing the zero scores from the calculation), on average, Grade 2 students read 27 cwpm with 79 percent accuracy, and Grade 3 students read 40 cwpm with 85 percent accuracy. Both fluency and accuracy are important components of being able to read with comprehension, which is explained in detailed in the following section.

Reading Comprehension To measure reading comprehension, students responded to questions for two short, grade-level passages. For Passage 1 (47 words), students were given 60 seconds, after which the passage was taken away as students responded to questions. However, in Passage 2 (62 words), students were given 180 seconds and the story remained in front of them as they responded to the comprehension questions. For each of the passages, there were five comprehension questions (four literal and one inferential question). Students were only asked questions that corresponded with the text they read in the story before the time ran out. The administration protocol for Passage 2 is preferable because it is not tied to a 60-second time limit. Percent zero scores were similar across the two passages: 52 percent for Grade 2 students and 30 percent for Grade 3 on Passage 1 and 53 percent of Grade 2 students and 27 percent of Grade 3 students on Passage 2. Since the reading comprehension subtask is linked to performance on the oral reading passage subtask, the high percent of zero scores for reading comprehension are a reflection of the high percent of zero scores in the oral reading subtasks. The reading comprehension mean scores for Passage 1 and Passage 2 were similar, with Grade 2 students answering an average of one reading comprehension question correctly (23 percent) and Grade 3 students answering an average of two reading comprehension questions correctly (41 percent). The similarities in scores between the two passages is interesting given that students had additional time to read the second passage and

38%

3%

11% 13%15%

11%

5%3%

1%

28%

14%11%

17% 17%

8%

3%1% 0%

0%

10%

20%

30%

40%

50%

60%

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80

Perc

ent o

f Stu

dent

s


G2 2014 G2 2019


therefore, on average, were answering more questions. On average, Grade 2 students attempted about two reading comprehension questions and Grade 3 students attempted about three to four reading comprehension questions for Passage 1. For Passage 2, the average number of attempted comprehension questions increased to three questions and four questions for Grades 2 and 3, respectively. The percent correct out of attempted items shows that students had a slightly lower performance on Passage 2. Grade 2 students scored 30 percent correct out of attempted items on Passage 1 compared to 24 percent on Passage 2, and Grade 3 students scored 45 percent correct out of attempted items on Passage 1 compared to 39 percent on Passage 2. This is somewhat surprising given that students were allowed to look back at Passage 2 when they were asked the comprehension questions. This may be due to the fact that students were truly not comprehending much of what they were reading or it may reflect students’ lack of experience with this type of activity. The reading comprehension benchmarks set in 2014 (after the 2014 Magindanawn EGRA study was completed) recommended that Grade 2 students reach at least 60 percent comprehension. Figure 19 shows that when presented with the same passage and questions (Passage 1), only 21 percent of Grade 2 students met that benchmark in 2019 versus 27 percent in 2014. Although the increase in mean scores is a positive improvement, it is important to note that there was relatively no change in zero scores (50 percent in 2014 and 52 percent in 2019).

Figure 19. Magindanawn reading comprehension scores, by grade (Passage 1)

Among Grade 2 students who read at least one word (n = 188), 65 percent reached the benchmark in 2019 (Figure 20).

52%

12%15%

12%7%

2%

30%

12%17% 17% 15%

10%

0%

10%

20%

30%

40%

50%

60%

0 20 40 60 80 100

Perc

ent o

f Stu

dent

s

Percent Score

Grade 2

Grade 3


Figure 20. Magindanawn reading comprehension percent correct of attempted for Grade 2 (Passage 1)

Mëranaw The sample of Mëranaw schools included 33 schools from BARMM and 7 schools from Region X–Northern Mindanoa. The final analysis includes student reading assessments and interviews from 403 Grade 2 and 401 Grade 3 students. Although the majority of Grade 2 students were 8 years old (38 percent), a significant number of Grade 2 students were 9 years old (20 percent) and 10 years old (14 percent). This means about 42 percent of students in Grade 2 were over-age for their grade. In Grade 3, 30 percent of students were 9 years old. However, large proportions of students were 10 years old (30 percent) and 11 years old (17 percent,) resulting in a total of 52 percent of Grade 3 students who were over-age for their grade. Nearly all Mëranaw students reported that Mëranaw was the language they spoke most commonly at home with their parents. Mirroring the homogeneity in the home language spoken by students, more than 90% percent of sampled Grade 2 and 3 teachers in Mëranaw schools reported speaking Mëranaw at home. Filipino or English were most commonly reported as home languages among those teachers who did not speak Mëranaw at home. Thirty-four percent of sampled students in Mëranaw-speaking schools reported being absent one or more days in the previous week, with the highest proportion reporting two days of absenteeism (17 percent). In the sampled Mëranaw-speaking schools, there was an equal distribution of students across the three SES categories (high, medium, and low SES), with each category accounting for one-third of students.

16%20%

22% 23%20%

0

10

20

30

40

50

60

20 40 60 80 100

Perc

ent o

f Stu

dent

s (N

on-z

eros

)

Percent Correct

Key Sample Descriptives Grade 2: 403 students; average age 8 years old Grade 3: 401 students; average age 9 years old Over-age for grade: 47% of students Absenteeism: 34% of students were absent one or more days in previous week SES: 35% high SES, 34% mid SES, and 31% low SES Student home language: 99% speak Mëranaw Teacher home language: 93% speak Mëranaw


Table 13 below presents an overview of reading performance by subtask and grade. The table includes the percent of students who scored zero and the mean score for each subtask, by grade. Zero scores are the percent of pupils who did not answer a single item correctly in the given subtask. As expected, for all subtasks, zero scores were less for Grade 3 when compared to Grade 2, while mean scores were higher for Grade 3 when compared to Grade 2. This means children were, on average, improving their skills from one grade to the next. Indicated by the low percent of Grade 3 students who scored zero, the majority of students had some reading and comprehension skills. However, mean scores and the results discussed in the subsequent sections show that students were spread across different levels of reading and comprehension abilities. In Grade 2, there was still a sizable proportion of students who were not reading with comprehension (about 24 percent). Mean scores in the other subtasks (letter sounds and non-word reading) indicate that improvements are still needed in students’ foundational reading skills, which are necessary for preparing students to read fluently, accurately, and with understanding. Results by subtask are explained in detail in the subsequent sections.

Table 13. Overview of Mëranaw EGRA percent zero and mean scores, by grade

Subtask



Grade 2 (n = 403)

Grade 3 (n = 401)

Grade 2 (n = 403)

Grade 3 (n = 401)


1.4% [±1.3]

1% [±0.9]

55.2% [±4.1]

55.0% [±4.0]


5.6% [±4.2]

2.5% [±1.8]

18.2 [±2.9]

23.2 [±2.6]


14.9% [±7.6]

4.0% [±2.5]

18.2 [±2.1]

28.5 [±2.4]


12.7% [±6.7]

3.5% [±2.2]

35.8 [±4.3]

55.6 [±5.0]


23.6% [±10.4]

7.8% [±3.5]

47.6% [±8.0]

62.3% [±4.5]

Listening Comprehension For the listening comprehension subtask, a short story and questions were read aloud to the students. The Mëranaw story and comprehension questions are presented below in Figure 21. Cumulative distribution graphs are presented in Annex H. Listening comprehension is a pre-reading skill, and it can be an important measure that reflects students’ oral fluency and vocabulary. On average, students in both grades scored 55 percent. This means, on average, students answered about three out of the five questions correctly. Very few students scored zero, indicating that nearly all students had some ability to listen and comprehend in Mëranaw.


Figure 21. Mëranaw listening comprehension story and questions

Questions assessor asks the student 1. Kagiya ko

pënggoraok si Babasal , antawaa i miyakawma on?

2. Ino pagishai i kamantis si Babasal?

3. Antaa i miyaka binaning a pënggoraok?

4. Antawaa i ngaran o phakawma agu pëkamo ki Babasal?

5. Antonaa i kashuwa amay ko makaoma si rakma a khuwaan iyan si kamantis?

Adën a mala a Babasal a Phënggoraok, miyakawma si kamantis na inishaan niyan? Ino ka phënggoraok Babasal? Aya pitharo i Babasal na miyakabinaning ako. Na tigi kamantis nagodën, antonaa i marata on? Tigi Babasal, kagiya khawma ko pëman i Soraya na pëkamon ako iran. Somininga sa mala si Kamantis. Ha ha ha! Ati oway, antonaa bës i bali ngka odi ka kana? Lagidakën a miyakariga ako mambo ka phakawma si Racma na isaog ako niyan ko sabakan a tamban.

On average, Grade 2 and 3 students answered 55% of listening questions correctly.

Letter-Sound Identification Knowledge of letter-sound correspondence is a fundamental skill that helps children learn to decode words. It has been shown to be a strong predictor of reading fluency, particularly for transparent orthographies (i.e., each letter is associated with a unique sound). For the letter-sounds subtask, students are presented a sheet of paper with100 letters that are randomly ordered, using a mix of uppercase and lowercase forms. Figure 22 shows this grid. The subtask was discontinued for students who did not correctly identify at least one item within the first row; these students proceeded to the next subtask. There was little difference in performance between Grade 2 and Grade 3. On average, Grade 2 children correctly identified 18 letter sounds in 60 seconds, while Grade 3 scores were only slightly higher at 23 clspm. Zero scores accounted for a very small proportion of scores in both grades,:about 6 percent in Grade 2 and only 3 percent in Grade 3. The low percentage of students scoring zero indicates that nearly all students recognized at least some letter sounds.


Figure 22. Mëranaw letter-sound identification subtask student stimulus sheet

S N a k m i t o B ng

D p g w n K ë i h g

t n s y w R M y O d

i H u i K n a t r Y

M r k L D i p S n a

n a b O a k S a m i

A ë m W F a o L S G

y N A F a O i U ë R

a T Ng h A d N o i a

k a m L A p a S o i

On average, students in Grade 2 attempted the first 31 items, and got 18 correct in one minute.

On average, students in Grade 3 attempted the first 36 items, and got 23 correct in one minute.

Item level analysis shows that students struggled most with the letters “ë,” “ng,” “I,” “h,” and “w.” The high percent of students mispronouncing “ë” or schwa sound (/ə/) is likely due to the fact that this letter may not be widely taught by teachers or had not yet been presented in students’ learning materials at the time the assessment was administered. Starting in 2017, Mëranaw language experts have been working to revise and rewrite the Mëranaw orthography. The letter “ë” is a new grapheme that replaced “e” and was only recently added to the orthography; therefore, it is likely that it is not yet fully integrated in the teaching and learning materials.

Non-Word Reading The non-words subtask is used to measure decoding skills by requiring that students draw on knowledge of letter-sound correspondence to read words that are unfamiliar but follow the orthographic rules and patterns of actual words. The subtask was discontinued for students who did not correctly identify at least one item within the first row; these students proceeded on to the next subtask. On average, students in Grade 2 read about 18 cnonwpm, while students in Grade 3 performed slightly better, reading at 29 cnonwpm. Zero scores were relatively low, with 14 percent of Grade 2 and only 4 percent of Grade 3 students unable to read a single non-word. Item level analyses are provided in Figure 23 (Grade 2) and Figure 24 (Grade 3). The non-words are presented along the horizonal axis; however, the figures only display words that were attempted by 40 percent or more of students—that is, the first 28 words for Grade 2 and the first 39 words for Grade 3. Not surprisingly, the non-words that students struggled with the most were words containing the letter “ë.” These findings are consistent with results from the letter-sounds subtask, which indicated this sound was the ones students had the most difficulty pronouncing correctly. Excluding words with “ë,” the figures below indicate that a high percentage of children in both grades were decoding and reading words accurately. Especially in Grade 3, the majority of students were reading non-words with accuracy.


Figure 23. Non-words itemized scores in Mëranaw, Grade 2 (words 1–28)

Figure 24. Non-words itemized scores in Mëranaw, Grade 3 (words 1–39)

ORF Passage Reading ORF was measured using a short, grade-level story (Passage 1). The subtask was administered according to standard EGRA Toolkit, Second Edition procedures, such that students were given 60 seconds to read as quickly and accurately as possible. The subtask was discontinued for students who did not correctly identify at least one word within the first eight words; these students proceeded on to the next subtask. Distribution of scores by grade are presented in Figure 25. Cumulative distributions by grade can be found in Annex H. It is expected that fluency (a measure of automaticity and accuracy together) on the ORF passage reading subtask is higher than in the non-words subtask because passage reading presents common words in the form of sentences, which is a format that students are often

0

10

20

30

40

50

60

70

80

90

100

koka

sro

tanë

nba

tap

pam ing

noki

rko

ma

sapa

ngro

sëp

yatim ta

tto

lap

gato

nam

ëlna

mdë

ntka

wpa

wap law

bagn

aya

bar

toka

dtis

om

ikad lë

kna

itëm

bong

tabg

a

Perc

ent o

f Stu

dent

s

%Correct %Incorrect

0102030405060708090

100

koka

sro

tanë

nba

tap

pam ing

noki

rko

ma

sapa

ngro

sëp

yatim ta

tto

lap

gato

nam

ëlna

mdë

ntka

wpa

wap law

bagn

aya

bar

toka

dtis

om

ikad lë

kna

itëm

bong

tabg

ada

yan

daw

igo

noda

yaw

tora lani

dago toki

gadë

nbo

kal

bom

ëro

Perc

ent o

f Stu

dent

s

%Correct %Incorrect


more accustomed to. On average, Grade 3 students read 20 cwpm faster and more accurately than students in Grade 2 (36 cwpm and 56 cwpm, respectively). As can be seen in the figure below, a relatively low percent of students scored zero, especially in Grade 3 (3 percent). For Grade 2, the largest proportion of students were reading 41–50 cwpm (18 percent). In Grade 3, however, the highest proportion of students were in the 31–40 and 51–60 categories, each with 16 percent of students.

Figure 25. Distribution of Mëranaw ORF, by grade (Passage 1)

As indicated in Table 14, girls outperformed boys with a lower percent scoring zero and higher mean scores in both grades. On average, Grade 2 girls read about 11 cwpm faster than boys, and girls in Grade 3 read nearly 19 cwpm faster than boys.

Table 14. Mëranaw ORF scores, by gender

Grade 2 Grade 3

Boys (n = 200)

Girls (n = 203)

Boys (n = 203)

Girls (n = 198)

Percent Zero Scores

17.7% [±8.1]

8.0%** [±5.9]

4.6% [±3.1]

2.4% [±2.4]

Mean Scores 30.7 [±6.9]

40.5* [±4.6]

48.1 [±4.8]

62.7** [±6.3]

* Statistical significance is p<.005 ** Statistical significance is p<0.001

Fluency measures automaticity and accuracy together. As depicted in Figure 26, on average, Grade 2 students attempted 38 words in 60 seconds (unshaded text) and of those words, they correctly read 28 words (74 percent accuracy). In Grade 3, depicted in Figure 27, on average, students attempted about 50 words within 60 seconds (unshaded text) and read about 45 of them accurately (89 percent accuracy). In the figures below, the words that students most commonly pronounced correctly within the average attempted items are indicated in red. The figures indicate that students were able to read words of varying lengths and difficulty.

13%8% 9% 11%

18%13% 13%

6% 5%1% 1% 1%3% 2% 3% 5%

16%12%

16%13% 12%

7% 6% 4%

0%

10%

20%

30%

40%

50%

60%

Perc

ent o

f Stu

dent

s


Grade 2

Grade 3


Figure 26. Average attempted items and accuracy in Mëranaw, Grade 2

ORF Passage


Aya ngaran akën na si Bai. Pito ragon ako i idad. Aya ngaran o pagari akën a mama na si Abol. Mababaling kami sii sa kaporoan a palaw. Di Kami dii mamola sa mga onga Abol sapad. Phësung kami sa Padian oman Sapto. Sii ko na baling kami na adën a pirak ami. Masosonor ako igira a alongan a Sapto.

Figure 27. Average attempted items and accuracy in Mëranaw, Grade 3

ORF Passage


Aya ngaran akën na si Bai. Pito ragon ako i idad. Aya ngaran o pagari akën a mama na si Abol. Mababaling kami sii sa kaporoan a palaw. Di Kami dii mamola sa mga onga Abol sapad. Phësung kami sa Padian oman Sapto. Sii ko na baling kami na adën a pirak ami. Masosonor ako igira a alongan a Sapto.

Reading Comprehension Fluent reading depends on progressing from decoding to automatically recognizing words and also requires children to draw inferences and anticipate words based on the context of the sentence and story. The shift from decoding to automaticity is important to free up cognitive space for comprehension. To measure reading comprehension, students responded orally to questions for two short, grade-level passages. For Passage 1 (61 words), students were given 60 seconds, after which the passage was removed from in front of them as they responded to questions (in accordance with the EGRA Toolkit, Second Edition reading comprehension measure). However, in Passage 2 (61 words), students were given an extended time of 180 seconds, and the story remained in front of them as they responded to the comprehension questions. Following the passage, five comprehension questions (four literal and one inferential question) were read aloud to the students to respond to orally. Students were only asked questions that corresponded with text they read in the story before the time ran out. The administration protocol for Passage 2 is preferable because it is not tied to a 60-second time limit. As such, this section discusses comprehension scores from Passage 2; results from Passage 1 can be found in Annex H. Figure 28 presents the distributions of scores for the reading comprehension subtask for Passage 2. The percent scores on the x-axis correspond to the number of questions a student answered correctly: 20 percent is equivalent to one question correct, 40 percent is equivalent to two questions correct, etc. Percent zero scores were notably low for Grade 3, at 9 percent. The low percent of Grade 3 students scoring zero indicates that most Grade 3 students were able to comprehend some of what they read. However, the distributions show that students were spread across different levels of comprehension. On average, students in


Grade 2 answered 47 percent of comprehension questions correctly, while students in Grade 3 performed slightly better, answering 62 percent of questions correctly. Students in Grade 2 attempted about three questions, while Grade 3 students attempted four questions. Percent correct out of attempted scores was very similar to mean scores—49 percent for Grade 2 and 62 percent Grade 3. This is partially because of low zero scores but also because students understood what they read and correctly answered the questions they attempted.

Figure 28. Mëranaw reading comprehension scores, by grade (Passage 2)

5 Discussion and Conclusion The ultimate goal of this study was to measure how well students read and understand grade-level text in four MTs, to develop a baseline measure for which student progress can continue to be measured and monitored. Table 15 below summarizes mean scores for passage reading for each language, along with the proportion of children who achieved 80 percent comprehension or better. The purpose of this table is not to compare fluency across languages, but rather to recognize that, on average, current ORF scores on passage reading in these MTs, especially in Grade 2, are resulting in low percentages of students who achieve 80 percent comprehension or better. Local experts for each of these languages determined that Grade 2 students should be reading at least 40 cwpm (in Bahasa Sug, Magindanawn, and Mëranaw) to 50 cwpm (in Chavacano) to comprehend most of what they read.10

10 More information on the benchmarks set for each of these languages can be found in 2019 Benchmarks for Reading Performance for Bahasa Sug, Chavacano, Magindanawn, Mëranaw Brief (forthcoming).

24%

13% 15% 15%19%

15%

8% 8%14%

27%23%

21%

0%

10%

20%

30%

40%

50%

60%

0 20 40 60 80 100

Perc

ent o

f Stu

dent

s

Percent Score

Grade 2

Grade 3


Table 15. Mean ORF on passage reading and comprehension scores, by language and grade

Language

Grade 2 Grade 3

Mean ORF

(cwpm)

Percentage Achieving 80%

Comprehension (out of total of 5

questions)

Mean ORF

(cwpm)

Percentage Achieving 80%

Comprehension (out of total of 5

questions) Bahasa Sug 18 19% 36 42%

Chavacano 34 36% 50 50%

Magindanawn 23 9% 41 20%

Mëranaw 36 34% 56 44%

Overall, there are high percentages of students who scored zero across most of the languages and subtasks, especially in Grade 2. This indicates that students are not being taught foundational skills, such as the alphabetic principle and decoding, that are necessary for reading fluently and with comprehension (see Figure 29). Even in Mëranaw, where zero scores were lowest across subtasks, it is evident that more can be done to explicitly teach these lower order skills to see more students comprehend grade-level text. As such, a key recommendation is to prioritize training teachers in explicit instruction of reading and ensuring they directly teach oral language development, phonics, and reading for comprehension. This will help students achieve mastery in their first language, which is critical in terms of their readiness tor transition to second and third languages.

Figure 29. Grade 2 percent zero scores, by subtask and language

Access to textbooks and the amount of time students spend practicing reading can influence their reading abilities: the more practice students get with text, the better they can become at word recognition and, subsequently, reading with fluency and accuracy. Classroom observation data revealed that, on average, students spent less than 25 percent of time in MT subject classes reading text in the MT (see Figure 30). Further, MT textbooks were only

3%

37%42% 44%

56%

41%

21%

28%24%

37%

25% 26%30% 28%

53%

1%6%

15% 13%

24%

0

10

20

30

40

50

60

ListeningComprehension

Letter SoundIdentification

Non-wordReading

Oral ReadingFluencyPassage

ReadingComprehension

Perc

ent Z

ero

Scor

es



being used in 14 percent of Grade 2 classrooms and 25 percent of Grade 3 classrooms. Based on these findings, it is evident that more must be done to provide students with textbooks and supplementary reading materials that facilitate learning and practicing reading and writing—and ensure these materials are used in the classroom. The materials must be complemented by quality instruction, which is why it is critical that teachers effectively use class time dedicated to MT literacy instruction.

Figure 30. Percentage of time spent reading during MT subject class, by language

Equally important is ensuring that students start school on time and attend school regularly. Students reported high rates of absenteeism across all languages. This ranged from 34 percent of students in Mëranaw schools to 60 percent of students in Chavacano schools being absent one or more days in the previous week. In all languages, absenteeism was negatively associated with reading performance (Table 16).

Table 16. Impact of absenteeism on mean ORF scores, by language

Language/Population



Bahasa Sug schools 50% -6 cwpm

Mëranaw schools 60% -12 cwpm


Magindanawn schools 62% -9 cwpm

The percent of over-age students varied by language, but was most prevalent among students in Mëranaw schools, with nearly 47 percent of children over-age for their grade. Over-age can be caused by a number of reasons; this study did not gather specific data on the reasons for over-age students. For the case of Mëranaw schools, over-age was positively associated with better reading outcomes but for students in Chavacano schools, being over-age was negatively associated with reading performance.

22%16%

11% 11%13%16%

24%

9%

0

10

20

30

40

50

60

70

80

90

100


Tota

l Per

cent

of C

lass

Tim

e

Grade 2 Grade 3


In general, both absenteeism and age can create diverse classrooms in terms of demographics and student ability. This can create disparities in performance among the students in a classroom and make it challenging for teachers to meet the needs of all students. Table 17 (Bahasa Sug), Table 18 (Chavacano), Table 19 (Magindanawn), and Table 20 (Mëranaw), present a summary of scores for Grade 3 students performing in the top quintile and those performing in the bottom quintile, by subtask. Across all languages, there are stark differences between high and low performing students. Policy can only do so much in terms of making sure children are enrolled at the right age and that they regularly attend school. As such, it is critical that teachers are trained on and effectively use remedial teaching practices and differentiated instruction in the classroom, while having sufficient resources to support different types of learners.

Table 17. Top and bottom quintile per subtask, in Bahasa Sug

Bahasa Sug Grade 3

Listening Comprehension

(%)

Letter-Sound Identification

(clspm)

Non-Word Reading

(cnonwpm)

Passage Reading (cwpm)

Reading Comprehension

(%) Bottom 20% 26% 0 0 0 0% Top 20% 100% 41 41 73 98%

Table 18. Top and bottom quintile per subtask, in Chavacano

Chavacano Grade 3


(%)


(clspm)

Non-Word Reading

(cnonwpm)



(%) Bottom 20% 0% 1 0 1 0% Top 20% 78% 37 38 96 100%

Table 18. Top and bottom quintile per subtask, in Magindanawn

Magindanawn Grade 3


(%)


(clspm)

Non-Word Reading

(cnonwpm)



(%) Bottom 20% 5% 1 1 1 0% Top 20% 86% 39 40 64 84%


Table 20. Top and bottom quintile per subtask, in Mëranaw

Mëranaw Grade 3


(%)


(clspm)

Non-Word Reading

(cnonwpm)



(%) Bottom 20% 28% 5 10 21 16% Top 20% 83% 46 48 93 98%

Language is another possible factor affecting student outcomes. One hypothesis is that students will learn to read faster in a language they already speak and understand fluently. Therefore, reading outcomes could be negatively impacted if the language students speak at home does not match the school’s designated MT. Similarly, reading outcomes could be affected if the teacher’s home language does not match the school’s designated MT. The data show mixed results that can neither confirm nor contradict these hypotheses. On the one hand, teachers and students in Mëranaw-speaking schools, where zero scores were lowest, were more homogenous in the language they used compared to those speaking the other languages in this study. Nearly all students (99 percent) and teachers (93 percent) in Mëranaw-speaking schools reported Mëranaw as the language they most commonly used at home. Conversely, in the Bahasa Sug, Chavacano, and Magindanawn schools, the teachers and students were more heterogenous in the languages they use. For example, in the Chavacano speaking schools, 84 percent of teachers and only 60 percent of students reported Chavacano as their home language. In Magindanawn-speaking schools, 87 percent of students and only 58 percent of teachers reported Magindanawn as the language they most commonly speak at home. Magindanawn and Chavacano speaking schools had the highest percent of students scoring zero on the listening comprehension subtask. Despite this apparent lack of familiarity with the MT, the children in Chavacano schools managed, on average, to reach a level of fluency similar to that of children in Mëranaw schools. To test these hypotheses using more advanced methods, we used linear regression models. When controlling for the basic student demographics,11 the data indicate that there is no difference in ORF between students who speak the same language at home as the school’s designated MT and students who do not. Similarly, teachers’ home language seemed to have no impact on students’ ORF. Regardless, it remains important to continue to collect and appropriately use language mapping data that identify the percentage of learners who speak the MT used in a school. Appropriate implementation of the MTB-MLE policy is another plausible factor that could affect students’ reading performance. According to the MTB-MLE expectations, children should be able to read in their MT by the end of Grade 1; however, the results presented in this report show that even by Grade 2, in several of these languages, there are significant proportions of students who are unable to read. Classroom observation data collected for the Language Usage Study revealed that the majority of teachers across all four languages were implementing the policy to a high degree. For this reason, the data did not reveal any significant findings between implementation and students’ reading performance; however, there are some differences in the way teachers used language in the classroom that may be a factor contributing to students’ reading performance. It should be noted that the data collected only address language use and frequency of certain activities (e.g., reading, writing, and speaking) and do not look at the quality of teaching practices. As mentioned previously, one of the biggest issues these data raise is that teachers are not explicitly teaching students the foundational skills needed to become fluent readers.

11 Student demographics include language group, grade, gender, and SES.


Table 21 provides the average percent of time teachers for a given language used the MoTL during class. It is important to note the observations for this table included five subjects: (1) MT, (2) English, (3) Filipino, (4) math, and (5) science/social studies. Therefore, it is reasonable that teachers use Filipino and English about 20 percent of the time to account for the specific subjects that are taught in these languages (one subject of five subjects observed equals 20 percent). Magindanawn stands out as having teachers who spent significantly less time teaching in the MT and more time teaching in Filipino when compared to the other three languages.

Table 19. Teachers’ use of languages in class*

Language

Bahasa Sug

Mean % Chavacano

Mean % Magindanawn

Mean % Mëranaw Mean %

n = 398 n = 400 n = 388 n = 391

MT 52.5% [±4.8]

58.5% [±2.8]

36.3% [±8.9]

60.1% [±6.8]

Filipino 28.8% [±4.0]

21.5% [±2.6]

42.1% [±7.3]

22.8% [±4.8]

English 18.5% [±2.1]

19.7% [±1.3]

21.4% [±2.7]

15.8% [±3.2]

Other 0.2% [±0.3]

0.3% [±0.4]

0.1% [±0.3]

1.3% [±1.1]

* Average percent of time the teacher used the MT throughout the entire class time. n= Number of classrooms observed.

Table 22 provides the percent of time, on average, a teacher uses the MT when teaching the MT subject. Magindanawn stands out among the other languages. On average, teachers in Magindanawn-speaking schools spent less time using Magindanawn as the MoTL even when teaching the MT subject and tended to supplement by using more Filipino. This may be due to the fact that fewer Magindanawn-speaking teachers reported the MT as their home language when compared to teachers of the other languages (only 58 percent). About 18 percent of Magindanawn-speaking teachers reported Filipino as their home language. The percent of students who reported speaking Magindanawn as their home language was high, about 87 percent. Therefore, fidelity of implementation to the MTB-MLE policy seems to be somewhat more reliant on teachers’ home language and comfort level using the MT. Appropriate placement of teachers is important, and as much as possible, teachers should be deployed based on whether they are native or highly comfortable speakers of the school MT.


Table 20. Teachers’ use of languages in MT subject class*

Language

Bahasa Sug MT

Mean %

Chavacano MT

Mean %

Magindanawn MT

Mean %

Mëranaw MT

Mean % n = 84 n = 81 n = 95 n = 67

MT 85.1% [±6.5]

96.3% [±1.8]

67.6% [±11.5]

94.1% [±3.5]

Filipino 11.8% [±6.4]

1.9% [±1.5

29.5% [±11.6]

3.6% [±3.0]

English 2.4% [±3.0]

1.6% [±1.0]

2.9% [±2.2]

1.2% [±0.9]

Other 0.7% [±1.1]

0.2% [±.4]

0% [±0]

1.2% [±1.2]

* Average percent of time teacher used the MT throughout the entire MT subject class time. n = Number of classrooms observed.

As mentioned early on, this study was not designed or intended to measure whether the MTB-MLE policy is working—it only looks at one outcome (student reading performance) using the MT as the MoTL in the early grades of primary school. Data from the Language Usage Study and the student interviews were used to provide some insight into how implementing the policy and other factors may impact student reading performance. The reduction in zero scores and improvement in fluency from Grade 2 to Grade 3 that was seen across all the languages included in this study indicates that as children and teachers spend more time using the MT, reading skills improve. However, the policy expects reading instruction in second and third languages to begin as early as Grade 2. Experience from other multilingual contexts suggests that children should have sufficient mastery of literacy in their first language before the second and third literacies are introduced. As such, and based on the results described above, our key recommendations are as follows:

1. More can and should be done to make sure teachers are trained on and explicitly teach students to read early. This should, in particular, focus on oral language, phonics, and reading for comprehension, as well as using remedial teaching, as necessary, with students who may be falling behind.

2. Students need to be better equipped with appropriate text materials that will aid them in learning to read.

3. Emphasis should be placed on ensuring students are present at school and that both students and teachers make productive use of class time.

6 Next Steps In August 2019, government officials from the (BARMM) Ministry of Higher and Technical Education and DepEd, as well as other key stakeholders (including representatives from surrounding teacher training institutes and implementing partners), gathered for a 2-day workshop to discuss the study’s findings and work to set draft benchmarks in each language. During the sharing of findings, officials and stakeholders sat in respective language groups to review, interpret, and help contextualize the findings presented in this report. Conversely, during the final session participants were mixed across language groups and asked to discuss one of the following topics:


• (Group 1) Based on the data and findings, what are the immediate next step recommendations to strengthen implementation of the MTB-MLE policy?

• (Group 2) Based on the data and findings, what are the long-term next step recommendations to strengthen implementation of the MTB-MLE policy?

• (Group 3) To whom and how should these findings be communicated to sustain and improve implementation of the MTB-MLE policy?

Key next steps include acting on the immediate and longer term solutions, as well as further communication of the findings, as identified by the participants and summarized below in Figure 31.

Figure 31. Summary of outputs from discussion groups Group 1: Immediate next steps Conduct inventory of existing learning materials Continue work to ensure accurate language mapping of students and teachers; should

be conducted in accordance with D.O. # 21, S. 2019 Annex 4 ‒ Update the Learner Information System ‒ Conduct a full inventory of teachers who are teaching kindergarten to Grade 3 ‒ Consider language expertise/background in the assignment of new teachers in the

kindergarten through Grade 3 levels Provision of training and teacher development for MTB-MLE

‒ Refresher MTB-MLE training for teachers ‒ Development of contextualized learning resources ‒ Train kindergarten through Grade 3 teachers on teaching beginning

reading/pedagogy/ language Provision of funds (Special Education Funds (known as SEF), local funds, Maintenance

and Other Operating Expenses (known as MOOEs), stakeholder funds)

Group 2: Long-term next steps Develop, reproduce, and distribute appropriate learning resources (instructional

materials and strategic intervention materials) Conduct regular assessment of (1) teacher needs analysis and (2) student outcomes Further institutionalization and promotion of the MTB-MLE policy to gain more support

from communities and stakeholders Strengthen quality assurance technical assistance and monitoring and evaluation Strengthen learning partnership to support teachers, such as mentoring or coaching via

Learning Action Cells (LACs) Strengthen stakeholder advocacy and partnership


Group 3: Communication of findings/results Who What When/Method

Learning Resources Management and Development System (known as LMRSD) Managers

Data on language mapping and other school situation reports to request more information on existing learner resources and support future learner resources’ distributions

Other findings Proposed budget

Communication should happen before the development of learner resources

Local government units(known as LGUs), such as the Committee on Education

Heterogenous/homogeneous data in schools to better support class sectioning

Data on availability and use of teaching and learning materials to seek funding for development and production of additional learner resources

Data on teacher language background to support mapping and teacher training needs

During local school board (LSB) meetings; first quarter of the fiscal year

Regional Directors Overview of EGRA findings for general information and action

Management committee meetings

MTB-MLE Regional Coordinators

Simplified version of the findings; summary-level findings that focus on mismatch of students and teachers to the school’s MT

Division meetings District LAC meetings

School heads Teachers (kindergarten

through Grade 3) Education Program

Supervisor (known as EPS)

School Division Superintendents (known as SDS)

Reading Coordinators Parents

Data on teacher and student language usage in class (with focus on MT subject class)

Data on students’ and teachers’ background/home language

Teacher self-reported MT abilities

Data on teacher attitudes towards MTB-MLE policy and use of language

EGRA results (information on students’ language abilities/performance to read letter sounds, oral language/listening comprehension, and ORF and comprehension)

As soon as possible to support self-assessment by teachers and allow for reflection/realization to garner support for intervention at the school/district levels

LAC sessions Parent-teacher-

community association (known as PTCA) meetings

LSB meetings


Group 3: Communication of findings/results Who What When/Method

Data on teacher experience/training

Data to support class sectioning

In addition to the deciding on above listed recommendations and further communication of the information from the study, participants from the workshop stayed used the data to set draft benchmarks and targets in each language. Another next step will be the finalization and government approval the draft benchmarks and targets. Once the benchmarks are formally accepted, regional governments, schools, and implementing partners/programs can put in place monitoring and evaluation plans to measure progress in student performance for each of these languages.


Annex A: Overview of Early Grade Reading Assessment (EGRA) Subtasks The EGRA measures the basic skills that a child must possess to eventually be able to read fluently and with comprehension—the ultimate goal of reading. These skills include letter-sound knowledge, decoding, fluency, and reading comprehension. The higher order skills (e.g., fluency and comprehension) build on lower order skills (e.g., letter-sound knowledge and decoding), and the lower order skills have been shown to be predictive of later reading achievement. Therefore, even if children cannot yet read a passage with comprehension, EGRA can nonetheless measure their progress toward acquiring the lower order skills that are steps along the path to that end. Table A-1 below summarizes the subtasks of the EGRA used in this study. It should be noted that the subtasks and test items used to assess Grades 2 and 3 were identical.

Table A-1. EGRA subtasks

Subtask Scoring Methodology Assesses Listening Comprehension

Students verbally respond to different types of questions, including literal and inferential questions about a text the assessor reads to them. (Untimed subtask)

Oral language; comprehension

Letter Sounds Students are asked to read aloud a list of 100 legitimate sounds presented in random order on a page. (Timed subtask, 60 seconds)

Letter-sound knowledge

Non-Word Reading Students are asked to read aloud a list of 50 non-words presented in random order on a page. Words are constructed from legitimate letter combinations for the language being assessed but are nonsensical. (Timed subtask, 60 seconds)

Letter-sound knowledge and decoding

Oral Reading Fluency Passage Reading (Passage 1)

Students are asked to read aloud a grade-level short story printed on a page. (Timed subtask, 60 seconds)

Automatic reading and word recognition

Reading Comprehension (using Passage 1)

Students are asked to verbally respond to up to five oral questions (four literal and one inferential) based on a short story and on how far the child reads into the text after 60 seconds. (Untimed subtask)

Comprehension

Reading Comprehension (using Passage 2)

Students are asked to verbally respond to up to five oral questions (four literal and one inferential) based on a short story and on how far the child reads into the text after 180 seconds. (Untimed subtask)

Comprehension


Annex B: Student Questionnaire All Grade 2 and Grade 3 pupils that were assessed with the Early Grade Reading Assessment (EGRA) were also asked a set of questions about their home language, home environment, and parent literacy and occupation. Table B-1 below summarizes the responses to these questions by language group. About 99 percent of students from the Mëranaw language group reported speaking Mëranaw at home, while students from other language groups showed more heterogeneity in home language. Students from the Chavacano language group showed the most heterogeneity with 59 percent speaking Chavacano, 18 percent speaking Bisaya/Cebuano, 11 percent speaking Bahasa Sug, and 8 percent speaking Filipino/Tagalog at home. About 11 percent of students from the Magindanawn language group reported speaking Iranum at home. The full breakdown is shown in the table below. Questions about items in the home were asked to calculate a wealth index measure for each child. Students were able to respond with “yes,” “no,” or “don’t know/no response” to any of the questions. The student responses of “don’t know/no response” were treated as missing values for the purposes of creating this socioeconomic index. All items were then combined using a principal component analysis to obtain factor loading coefficients for each item. This principal component analysis produced a Rho of 0.293. Refer to the table below for more details on the item responses and the wealth index formula.

Table B-1. Student questionnaire responses for Grade 2 and Grade 3 students combined, by language


Counts n = 804 n = 800 n = 788 n = 804

Estimate [95% Confidence Interval Band]

Average age 8.5 [±0.1]

8.3 [±0.1]

8.6 [±0.1]

9.0 [±0.2]

Average number of days absent last week 1.8 [±0.3]

1.2 [±0.1]

1.7 [±0.2]

1.4 [±0.2]

Percent of students over-age Grade 2: age>8 Grade 3: age>9

22.1% [±4.9]

11.7% [±2.6]

26.0% [±5.2]

46.6% [±5.6]

Percent girls 52.5% [±2.6]

52.0% [±2.1]

53.2% [±2.8]

51.5% [±2.9]

Attended the same school all year 94.7% [±2.1]

89.4% [±3.3]

92.3% [±3.3]

90.4% [±3.6]

Has other reading materials at home 66.4% [±4.6]

73.7% [±6.2]

79.4% [±5.1]

76.7% [±3.2]

Mother can read and write 92.4% [±2.3]

98.1% [±1.3]

89.0% [±3.9]

92.1% [±3.0]

Father can read and write 85.3% [±3.4]

97.6% [±1.3]

89.0% [±3.0]

84.5% [±4.9]

What language do you and your parents mostly speak to each other?


Bahasa Sug 86.4% [±6.5]

11.1% [±4.5]

0% [±0.0]

0% [±0.0]

Chavacano 0.8% [±0.8]

59.4% [±8.5]

0% [±0.0]

0% [±0.0]

Magindanawn 0.6% [±1.0]

0.3% [±0.4]

86.9% [±7.1]

0.1% [±.04]



Mëranaw 0% [±0.0]

0.1% [±0.2]

0.1% [±0.3]

98.8% [±0.8]

Tagalog/Filipino 1.0% [±1.8]

8.1% [±4.0]

0.7% [±0.8]

0.3% [±0.9]

English 0% [±0.0]

0.2% [±0.4]

0% [±0.0]

0% [±0.0]

Binukid 0% [±0.0]

0.4% [±0.0]

0% [±0.0]

0% [±0.0]

Bisaya/Cebuano 3.8% [±4.3]

17.6% [±5.0]

0.9% [±0.8]

0.2% [±0.3]

Hokkien/Fukkien/Chinese 0% [±0.0]

0% [±0.0]

0% [±0.0]

0% [±0.1]

Ilonggo/Hiligaynon 0% [±0.0]

1.3% [±1.2]

0.8% [±1.4]

0.3% [±0.5]

Iranum 0% [±0.0]

0% [±0.0]

10.5% [±7.3]

0.1% [±0.4]

Sinama/Sama 4.7% [±3.7]

0.2% [±0.4]

0% [±0.0]

0% [±0.0]

Subanon 0% [±0.0]

0.3% [±0.4]

0.1% [±0.4]

0.2% [±0.4]

Yakan 2.5% [±1.9]

1.0% [±1.0]

0% [±0.0]

0% [±0.0]

What other language(s) are spoken in your home?


No other language 72.9% [±6.7]

48.4% [±6.5]

35.9% [±12.9]

69.2% [±5.8]

Bahasa Sug 3.6% [±1.9]

3.8% [±2.2]

0% [±0.0]

0.2% [±0.5]

Chavacano 1.2% [±1.0]

25.4% [±7.0]

0% [±0.0]

0% [±0.0]

Maguindanao 0% [±0.0]

0% [±0.0]

59.3% [±13.0]

0.2% [±0.4]

Mëranaw 0.1% [±0.4]

0% [±0.0]

1.1% [±1.2]

8.8% [±5.1]

Tagalog/Filipino 4.1% [±1.8]

21.0% [±5.0]

2.0% [±1.1]

13.9% [±3.8]

English 0.4% [±0.7]

2.0% [±1.2]

1.7% [±0.9]

1.8% [±2.7]

Bisaya/Cebuano 1.2% [±1.8]

13.2% [±2.7] — 1.8%

[±0.0]

Ilonggo/Hiligaynon 0% [±0.0]

0.4% [±0.6] — 1.2%

[±1.2]

Iranum 0.1% [±0.4]

0.5% [±1.1] — 0.1%

[±0.2]

Sinama/Sama 3.9% [±2.1]

0.3% [±0.5] — 0%

[±0.0]

Subanon 0.1% [±0.3]

0.2% [±0.6] — 0.4%

[±0.8]

Yakan 1.4% [±1.1]

0.7% [±1.0] — 0%

[±0.0] Does someone help you read in any language at home?


No 20.5% [±3.6]

16.0% [±3.9]

20.1% [±3.8]

29.5% [±5.1]



Yes, other 3.0% [±1.2]

7.0% [±2.4]

2.5% [±1.3]

5.4% [±4.3]

Yes, sibling 35.3% [±4.5]

29.5% [±3.4]

41.4% [±5.8]

33.6% [±6.5]

Yes, parent 41.2% [±5.4]

47.5% [±3.8]

35.9% [±6.5]

31.5% [±5.7]

What is your father’s work? Estimate [95% Confidence Interval Band]

Unemployed 3.8% [±1.6]

4.9% [±1.8]

5.7% [±1.9]

11.9% [±4.5]

Informal /manual labor/self-employed 84.5% [±4.7]

80.6% [±3.9]

85.4% [±3.4]

81.1% [±6.7]

Professional 8.8% [±4.2]

11.2% [±2.7]

7.5% [±2.7]

5.8% [±3.9]

Overseas foreign worker 3.0% [±1.3]

3.2% [±1.3]

1.4% [±0.0]

1.2% [±.9]

What is your mother's work? Estimate [95% Confidence Interval Band]

Unemployed 48.3% [±4.4]

50.2% [±4.4]

54.5% [±5.3]

45.5% [±5.7]

Informal /manual labor/self-employed 41.6% [±4.5]

38.3% [±4.1]

32.9% [±5.4]

43.2% [±5.6]

Professional 5.2% [±2.5]

6.4% [±2.1]

2.5% [±1.2]

9.2% [±2.6]

Overseas foreign worker 4.9% [±1.8]

5.1% [±1.6]

10.1% [±3.3]

2.1% [±1.5]

How many family members live in your house?


4 and below 15.6% [±3.1]

28.6% [±3.0]

22.5% [±4.2]

12.6% [±2.5]

5 or 6 38.3% [±3.9]

41.8% [±3.3]

37.6% [±4.6]

27.6% [±4.5]

7 or more 46.1% [±4.8]

29.6% [±3.4]

39.9% [±4.7]

59.8% [±4.7]

Socioeconomic Status Questions

At your house, there is: Estimate [95% Confidence Interval Band]

Refrigerator/freezer 24.4% [±6.2]

48.5% [±5.5]

27.7% [±6.6]

32.1% [±7.9]

Television set 70.6% [±6.5]

86.6% [±3.6]

70.2% [±5.5]

79.6% [±6.9]

Computer 10.3% [±3.5]

15.1% [±3.6]

5.8% [±1.9]

8.4% [±2.4]

Faucet/tap 46.7% [±8.3]

77.4% [±5.2]

24.8% [±6.6]

47.4% [±7.5]

Electricity in the house 83.9% [±7.1]

93.8% [±2.2]

80.3% [±7.0]

93.4% [±4.2]

Jeep, truck, or car 9.4% [±3.0]

14.5% [±2.9]

18.9% [±5.0]

27.7% [±9.8]

Motorcycle or tricycle 45.5% [±6.0]

60.8% [±5.0]

69.6% [±5.8]

53.9% [±4.7]

Washing machine 26.6% 38.6% 26.5% 31.8%


Bahasa Sug Chavacano Magindanawn Mëranaw [±5.6] [±7.0] [±7.5] [±8.8]

More than one bedroom 57.8% [±7.4]

73.3% [±5.6]

73.8% [±4.9]

70.6% [±5.1]

Wealth Index Formula derived using principal component analysis: wealthindex = .4069080918652335*ses_fridge + .3814358570416396*ses_tv_radio + .286825250822098*ses_computer + .3356976984940728*ses_faucet + .3327568063011612*ses_electricity + .2693243747403885*ses_car + .2693068194290627*ses_motorcycle + .3908990415472536*ses_washing_machine + .2921983706517183*ses_bedrooms Note: Variables in the equation will have a value of one if the pupil responded with “yes” and a value of zero if the student responded with “no.”


Annex C: Assessor Training Details and Data Collection Procedures

Assessor and Supervisor Training and Pilot A total of 46 participants were trained during a three-day workshop held January 28–30, 2019, in Davao City. During the workshop, participants were grouped by language; 10 participants were trained as supervisors and 36 were trained as assessors to collect data electronically with tablets using Tangerine™. Assessors participated in an assessor accuracy measure (AAM) that evaluated their ability to accurately score subtask items. Problematic items identified through the AAM exercise were addressed and improved upon during the training. The AAM evaluated each participant’s ability to accurately score the Early Grade Reading Assessment (EGRA) subtask items in relation to a Gold Standard. All participants watched a video of a child being assessed using the EGRA instrument. Each participant used a tablet to score the child as if they were conducting the assessment. The scores were then compared to a Gold Standard scoring, which had previously been completed by a master trainer scoring the same video. Assessors who performed below the conventional threshold of 90% agreement received targeted assistance to improve their performance. The results of the AAM are presented below, by language, in Tables C-1 through C-4.

Table C-1. AAM results for Bahasa Sug: Percent averages and ranges of agreement, by EGRA subtask (n = 8)

EGRA Subtask

Average Percentage Agreement of Participants

(Standard Deviation)

[Min, Max] Agreement of Participants

Total Assessors Who Achieved

90% Agreement with the Gold

Standard

Overall 95.9% (1.6%) [93.5%, 97.9%] 8


100.0% (0%) [100.0%, 100.0%] 8

Letter Sounds 97.5% (1.5%) [95.0%, 99.0%] 8

Non-Words 89.0% (4.5%) [84.0%, 96.0%] 4

Oral Reading (Passage 1)

93.8% (4.6%) [86.7%, 98.3%] 6

Reading Comprehension (Passage 1)

100.0% (0%) [100.0%, 100.0%] 8


93.8% (8.6%) [83.3%, 100.0%] 5


Table C-2. AAM results for Chavacano: Percent averages and ranges of agreement, by EGRA subtask (n = 8)

EGRA Subtask






Standard

Overall 95.9% (2.6%) [89.9%, 98.3%] 7


90.0% (15.1%) [60.0%, 100.0%] 5

Letter Sounds 95.5% (2.2%) [93.0%, 100.0%] 8

Non-Words 94.3% (3.6%) [88.0%, 98.0%] 7


99.3% (0.7%) [98.6%, 100.0%] 8


97.5% (7.1%) [80.0%, 100.0%] 7


95.0% (9.3%) [80.0%, 100.0%] 6

Table C-3. AAM results for Magindanawn: Percent averages and ranges of agreement, by EGRA subtask (n = 8)

EGRA Subtask






Standard

Overall 89.4% (7.0%) [80.3%, 96.0%] 5


57.5% (27.1%) [20.0%, 80.0%] 0

Letter Sounds 96.4% (2.5%) [93.0%, 99.0%] 8

Non-Words 94.8% (4.3%) [88.0%, 100.0%] 7


98.9% (1.6%) [95.7%, 100.0%] 8


97.5% (7.1%) [80.0%, 100.0%] 7


EGRA Subtask






Standard Reading Comprehension (Passage 2)

92.5% (21.2%) [40.0%, 100.0%] 7

Table C-4. AAM results for Mëranaw: Percent averages and ranges of agreement, by EGRA subtask (n = 6)

EGRA Subtask






Standard

Overall 96.9% (0.8%) [95.9%, 98.2%] 6


100.0% (0%) [100.0%, 100.0%] 6

Letter Sounds 97.8% (1.5%) [96.0%, 99.0%] 6

Non-Words 98.0% (3.3%) [92.0%, 100.0%] 6


97.8% (2.0%) [94.7%, 100.0%] 6


100.0% (0%) [100.0%, 100.0%] 6


86.1% (6.8%) [83.3%, 100.0%] 1

As part of the training activities, assessors returned to their divisions and visited nearby schools to practice interview skills and familiarize themselves with the tablets. After training concluded, each language’s instruments were piloted during a one-day field activity in which the trained data collectors assessed a total of 434 students across all four languages (115 in Bahasa Sug, 96 in Chavacano, 113 in Magindanawn, and 110 in Mëranaw). Using Tangerine, the instruments were administered, and data were uploaded and analyzed by RTI technical staff. Following the pilot data collection, the instruments and items within each section were reviewed. Subsequent changes were made to the instruments based on the analyses; most of these were minor.

Data Collection and Inter-Rater Reliability Results Data collection took place February 17 to March 9, 2019, following the assessor training activities. A total of 36 assessors and 9 supervisors were deployed to conduct the fieldwork, which took place in sampled schools in the Bangasamoro Autonomous Region in Muslim


Mindanao (BARMM), Region IX, Region X, and Region XII.12 Assessors were divided into teams comprising three EGRA assessors and one supervisor. General oversight and management of the data collection teams was provided by Kantar TNS. Teams uploaded data regularly, after which RTI home office statisticians provided quality checks and feedback on the uploaded data. To supplement these data quality checks, the assessors in the field also performed daily inter-rater reliability checks. For these checks, the assessors sampled one additional student. The student was given the EGRA, as normal, with one assessor, while another assessor also silently marked the students’ responses using his or her own tablet. Once the student was dismissed, the assessors compared their marked responses and discussed any discrepancies, without being able to change their original responses. Inter-rater reliability results are provided for each language in Tables C-5 through C-8.

Table C-5. Inter-rater reliability results for Bahasa Sug: Percent averages and ranges of agreement, by EGRA subtask (n = 34)

EGRA Subtask



[Min, Max] Agreement of

Participant Pairs

Overall 99.2% (1.4%) [94.9%, 100.0%]


98.2% (5.7%) [80.0%, 100.0%]

Letter Sounds 99.3% (1.1%) [95.0%, 100.0%]

Non-Words 98.5% (6.2%) [64.0%, 100.0%]


99.7% (0.8%) [96.7%, 100.0%]


99.4% (3.4%) [80.0%, 100.0%]


100.0% (0%) [100.0%, 100.0%]

Table C-6. Inter-rater reliability results for Chavacano: Percent averages and ranges of agreement, by EGRA subtask (n=41)

EGRA Subtask




Participant Pairs

Overall 99.6% (1.1%) [94.3%, 100.0%]

12 It should also be noted that the sample was drawn in January 2019 when the region was recognized as the Autonomous Region in Muslim Mindanao (ARMM). Subsequently, some divisions and cities that are now part of BARMM were previously officially recognized as part of other regions.


EGRA Subtask




Participant Pairs Listening Comprehension

99.5% (3.1%) [80.0%, 100.0%]

Letter Sounds 99.9% (0.5%) [98.0%, 100.0%]

Non-Words 99.9% (0.5%) [98.0%, 100.0%]


99.8% (0.5%) [98.4%, 100.0%]


100.0% (0%) [100.0%, 100.0%]


98.5% (6.9%) [60.0%, 100.0%]

Table C-7. Inter-rater reliability results for Magindanawn: Percent averages and ranges of agreement, by EGRA subtask (n = 36)

EGRA Subtask




Participant Pairs

Overall 97.6% (5.1%) [72.0%, 100.0%]


98.3% (10.0%) [40.0%, 100.0%]

Letter Sounds 98.4% (2.3%) [93.0%, 100.0%]

Non-Words 97.8% (5.3%) [72.0%, 100.0%]


97.9% (3.8%) [83.6%, 100.0%]


96.6% (8.9%) [60.0%, 100.0%]


96.6% (8.9%) [60.0%, 100.0%]


Table C-8. Inter-rater reliability results for Mëranaw: Percent averages and ranges of agreement, by EGRA subtask (n = 36)

EGRA Subtask




Participant Pairs

Overall 97.7% (2.6%) [91.6%, 100.0%]


98.3% (7.4%) [60.0%, 100.0%]

Letter Sounds 97.0% (4.0%) [82.0%, 100.0%]

Non-words 96.0% (8.1%) [66.0%, 100.0%]


99.2% (1.4%) [95.1%, 100.0%]


96.6% (8.9%) [60.0%, 100.0%]


98.3% (7.4%) [60.0%, 100.0%]


Annex D: Sample Methodology, Sample Weights, and Precision Estimates This annex discusses the details of the sample, the population that it is meant to represent, and how the sample is properly representative of that population. It also discusses the precision estimates for the major outcome variables from which the sample size was derived.

Population of Interest and Sample Frame The population of interest includes all primary government schools in Region IX, Region X, Region XII, and the Bangsamoro Autonomous Region in Muslim Mindanao (BARMM) that instruct in one of the four language groups of interest (Chavacano, Bahasa Sug, Magindanawn, or Mëranaw) and are not located in the Sulu division outside of Jolo City.13 The 2017–2018 Basic Education Information System (BEIS) School Census data were used as the sample frame from which the sample was drawn. It was kindly provided by the Department of Education. Table D-1 provides the total number of schools in the list frame, along with the total number of schools excluded for the given reason. It also provides the total number of schools that make up the defined population. Table D-2 provides the population of schools by language group, as well as Grade 2 and Grade 3 enrollment, by gender.

Table D-1. Schools excluded from the list frame prior to sampling

N

Schools Percent Total number of schools in the sample frame

38,913 —

Reason for Exclusion School is not located in Regions IX, X, XII, or BARMM

30,736 78.99%

School does not instruct in one of the four languages (Chavacano, Bahasa Sug, Magindanawn, or Mëranaw)

5,487 14.1%

School does not have primary Grade 1–Grade 6

1 0%

School does not have kindergarten (KG) 0 0% School was indicated as closed 22 0.06% School was located in Sulu division (except for Jolo City)

348 0.89$

School is missing language information 370 0.95% Not Excluded [Defined Population] 1,949 5.01%

13 It should also be noted that the sample was drawn in January 2019 when the region was recognized as the Autonomous Region in Muslim Mindanao (ARMM). Subsequently, some divisions and cities that are now part of BARMM were previously officially recognized as part of other regions.


Table D-2. Population counts of schools, by language group Language

Group School

Population Grade Student

Population Gender Student

Population

Chavacano 154 Second 14,547 Boys 7,428

Girls 7,119

Third 16,178 Boys 8,468 Girls 7,710

Bahasa Sug 254 Second 13,840 Boys 7,047

Girls 6,793

Third 14,297 Boys 7,266 Girls 7,031

Magindanawn 680 Second 37,265 Boys 18,871

Girls 18,394

Third 36,187 Boys 18,326 Girls 17,861

Mëranaw 861 Second 33,633 Boys 16,168

Girls 17,465

Third 29,954 Boys 14,329 Girls 15,625

Total 1,949 195,901 195,901

Sample Methodology The sample methodology follows a three-stage random sample of schools, classrooms, and students. Schools were first separated by language group and then sorted by region, division, and combined Grade 2 and Grade 3 enrollment. Schools were selected with probability proportional to the combined Grade 2 and Grade 3 enrollment. A total of 160 schools were selected (40 for each language group). For each selected school, two replace replacement schools were automatically selected. The replacement schools were selected to best represent the originally selected school with regards to location (region and division) and enrollment. They would replace the originally sampled school if the originally sampled school did not meet the requirements as defined by the population. Once the assessment team arrived at the school, they randomly selected one KG, one Grade 1, one Grade 2, and one Grade 3 classroom with equal probability. Teachers associated with the sampled classroom were asked to complete the teacher questionnaire, and various subjects were observed throughout the day. The team then randomly sampled 10 students in the selected Grade 2 classroom and 10 students in the selected Grade 3 classroom to complete the student portion of the assessment, including the Early Grade Reading Assessment (EGRA) and a student questionnaire. If there were fewer than 10 Grade 2 or Grade 3 students present in the chosen classroom, the assessor would automatically assess all students present in the selected classroom. It should be noted that no KG and no Grade 1 students were assessed in the EGRA. Table D-3 displays the sample methodology of the three-stage sample. Table D-4 provides the final counts of completed school, classroom, and student assessments. A total of 160 schools were visited, and 3,196 students were assessed.


Table D-3. Sample methodology for the Philippines Regional Snapshot 2019 EGRA, three-stage stratified sample of schools, classrooms, and students

Stage

Item Sampled (Expected Counts)

Stratification (Number of Strata)

Probability of Selection

1 Schools (160)

Language Group: Chavacano, Bahasa Sug, Magindanawn, and Mëranaw (4)

Proportional to Grade 2 + Grade 3 enrollment

2 Classrooms (640)

Grade: KG, Grade 1, Grade 2, and Grade 3 (4)

Equal

3 Students – Grade 2 + Grade 3 only (3,200)

Gender: Boys/Girls (2) Equal

Table D-4. Final sample counts of assessments, interviews, and observations collected for the Philippines Regional Snapshot 2019 EGRA

Language Group

Sampled Schools

School Info

Collected Grade

Classroom Info

Collected Teachers

Interviewed Classrooms

Observed Classes

Observed Gender Students Assessed

Chavacano 40 40

KG 40 40 40 202

Grade 1 40 40 40 201

Grade 2 40 40 40 200 Boys 199 Girls 199

Grade 3 40 40 40 200 Boys 200 Girls 202

Bahasa Sug 40 40

KG 38 38 39 177

Grade 1 40 40 40 200

Grade 2 40 40 40 201 Boys 198 Girls 205

Grade 3 40 40 40 197 Boys 196 Girls 205

Magindanawn 40 40

KG 39 39 40 136

Grade 1 40 40 40 194

Grade 2 39 39 39 191 Boys 181 Girls 209

Grade 3 40 40 40 197 Boys 199 Girls 199

Mëranaw 40 40

KG 37 38 38 164

Grade 1 39 39 39 196

Grade 2 38 38 38 190 Boys 200 Girls 203

Grade 3 39 40 40 199 Boys 203 Girls 198

Total 160 160 629 631 633 3,045 3,196

Sample Weights All sample weights were calculated as the inverse of the probability of selection at each stage of selection (school, classroom, and student). School weights were scaled to the


known populations of schools by language group. Classrooms were not scaled to the final population as that information was not available in the BEIS School Census data. The student weights were scaled to the population because the school census data contained enrollment by grade and gender. Table D-5 provides the weighted totals and the sample totals, by language group and grade.

Table D-5. Weighted sample (estimated population) and sample counts, by each stage sampled (school, classroom, and student)

Language Group

Weighted Sample (Estimated Population)

School

Classroom Students

KG Grade

1 Grade

2 Grade

3 Grade 2 Grade 3 Chavacano 154 338 416 362 371 14,547 16,178 Bahasa Sug 254 304 398 377 384 13,840 14,297 Magindanawn 680 933 1,136 927 901 37,265 36,187 Mëranaw 861 933 1,031 934 980 33,633 29,954

Total 1,949 2,508 2,981 2,600 2,636 99,285 96,616

Language Group

Sample

School

Classroom Students

KG Grade

1 Grade

2 Grade

3 Grade 2 Grade 3 Chavacano 40 40 40 40 40 398 402 Bahasa Sug 40 38 40 40 40 403 401 Magindanawn 40 39 40 39 40 390 398 Mëranaw 40 37 39 38 39 403 401

Total 160 154 159 157 159 1,594 1,602

Precision Estimates The Grade 2 and Grade 3 sample size calculations were derived using the oral reading fluency score with a desired 95 percent confidence interval (CI) band of ± 3.5 words per minute (wpm) by grade for the combined languages. The actual sample size for grades was sufficient enough to provide a 95 percent CI band of ±2.8 wpm and ±2.6 wpm for each respective grade. Table D-6 provides the mean estimates and the precision estimates for each grade.

Table D-6. Reading fluency means and precision estimates, by grade

n Range (wpm)

Mean (wpm)

95% CI Band (wpm)

Standard Deviation (SD; wpm)

Intra-Class Correlation

(ICC)

Design Effect (DEF)

Grade 2 1,594 [0,141] 26.8 ±2.8 24.3 0.255 2.3 Grade 3 1,601 [0,136] 43.9 ±2.6 29.0 0.346 1.8

It was hoped, but not planned, that the 95 percent CI bands would be within ± 5.0 wpm for each grade-language. Although the precision was met for all of the languages assessed in


Grade 2, Chavacano and Bahasa Sug in Grade 3 exceeded the precisions at ±6.6 and ±5.6 wpm, respectively. The cause for these higher than expected precision estimates has to do primarily with their high standard deviations and, to a lesser extent, the high standard deviations were combined with a moderately high intraclass correlation. Table D-7 and Table D-8 provide the mean estimates and the precision estimates for each grade by language assessed.

Table D-7. Grade 2 reading fluency means and precision estimates, by language assessed

Language Group n

Range (wpm)

Mean (wpm)

95% CI Band (wpm)

SD (wpm) ICC DEF

Chavacano 398 [0,116] 34.0 ±4.1 29.8 0.096 1.1 Bahasa Sug 403 [0,83] 17.8 ±3.6 20.7 0.218 1.3 Magindanawn 390 [0,99] 19.2 ±4.3 18.0 0.327 2.9 Mëranaw 403 [0,141] 35.8 ±4.2 25.1 0.214 2.0

Table D-8. Grade 3 reading fluency means and precision estimates, by language assessed

Language Group n

Range (wpm)

Mean (wpm)

95% CI Band (wpm)

SD (wpm) ICC DEF

Chavacano 402 [0,129] 50.4 ±6.6 36.0 0.249 1.5 Bahasa Sug 401 [0,112] 35.6 ±5.6 28.4 0.288 1.5 Magindanawn 398 [0,96] 34.5 ±5.1 23.4 0.395 2.7 Mëranaw 401 [0,136] 55.6 ±4.8 25.8 0.231 2.1

Note about Sample Replacements A total of five schools from the Chavacano group, seven schools from the Bahasa Sug group, seven schools from the Magindanawn group, and eight schools from the Mëranaw group were replaced. The reasons for replacement included the following: the school visited was using a different language of instruction than expected, there were impassable/unsafe roads or other access barriers, the school was missing grades altogether, or the school had low enrollment for Grade 2 or Grade 3.


Annex E: Bahasa Sug Cumulative Distribution Graphs The graphs below present the cumulative distributions for the listening comprehension and reading comprehension percent scores in Bahasa Sug, as well as the oral reading fluency (ORF) scores, by grade. The listening comprehension and reading comprehension percent score cumulative distributions are step-wise (increase in a step-like manner) because each comprehension subtask has only five questions, meaning the scores can only increase in increments of 20 percentage points at a time. The ORF cumulative distributions are smoother due to the fact that ORF is measured with a continuous variable (correct words per minute [cwpm]). For the distributions below, the percentile indicator is on the x-axis and the percent score, or cwpm score, is on the y-axis. These graphs can be used to identify what percentage of students are reaching certain levels of comprehension or rates of ORF. For example, we can use the y-axis and find 80 percent comprehension and then find the percentile on the x-axis that corresponds to the line graph. Looking at listening comprehension for Bahasa Sug, shown below, we see that close to 52 percent of Grade 2 and 50 percent of Grade 3 students scored at least 80 percent on the listening comprehension subtask.

Figure E-1. Bahasa Sug cumulative distribution of listening comprehension scores, by grade

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Percentile

Grade 2 Grade 3


Figure E-2. Bahasa Sug cumulative distribution of ORF scores, by grade (Passage 1)

Figure E-3. Bahasa Sug cumulative distribution of reading comprehension scores, by grade (Passage 1)

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Ora

l Rea

ding

Flu

ency

Percentile

Grade 2 Grade 3

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Percentile

Grade 2 Grade 3


Figure E-4. Percent total Bahasa Sug reading comprehension scores, by grade (Passage 1)

Figure E-5. Bahasa Sug cumulative distribution of reading comprehension scores, by grade (Passage 2)

64%

11%15%

4% 2% 2%

43%

19% 17%

9% 10%

1%0%

10%

20%

30%

40%

50%

60%

70%

0 20 40 60 80 100

% o

f Stu

dent

s

Percent Score

Grade 2

Grade 3

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Percentile

Grade 2 Grade 3


Annex F: Chavacano Cumulative Distribution Graphs The graphs below present the cumulative distributions for the listening comprehension and reading comprehension percent scores in Chavacano, as well as the oral reading fluency (ORF) scores, by grade. The listening comprehension and reading comprehension percent score cumulative distributions are step-wise (increase in a step-like manner) because each comprehension subtask has only five questions, meaning the scores can only increase in increments of 20 percentage points at a time. The ORF cumulative distributions are smoother due to the fact that ORF is measured with a continuous variable (correct words per minute [cwpm]). For the distributions below, the percentile indicator is on the x-axis and the percent score or cwpm score is on the y-axis. These graphs can be used to identify what percentage of students are reaching certain levels of comprehension or rates of ORF. For example, we can use the y-axis and find 80 percent comprehension and then find the percentile on the x-axis that corresponds to the line graph. Looking at listening comprehension for Chavacano, shown below, we see that close to 10 percent of Grade 2 and 13 percent of Grade 3 students scored at least 80 percent on the listening comprehension subtask.

Figure F-1. Chavacano cumulative distribution of listening comprehension scores, by grade

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Percentile

Grade 2 Grade 3


Figure F-2. Chavacano cumulative distribution of ORF scores, by grade (Passage 1)

Figure F-3. Chavacano cumulative distribution of reading comprehension scores, by grade (Passage 1)

0

20

40

60

80

100

120

140

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Ora

l Rea

ding

Flu

ency

Percentile

Grade 2 Grade 3

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Percentile

Grade 2 Grade 3


Figure F-4. Percent total Chavacano reading comprehension scores, by grade (Passage 1)

Figure F-5. Chavacano cumulative distribution of reading comprehension scores, by grade (Passage 2)

41%

12% 10% 9%

15%12%

30%

9% 9%11%

19%23%

0%

10%

20%

30%

40%

50%

0 20 40 60 80 100

% o

f Stu

dent

s

Percent Score

Grade 2

Grade 3

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Percentile

Grade 2 Grade 3


Annex G: Magindanawn Cumulative Distribution Graphs The graphs below present the cumulative distributions for the listening comprehension and reading comprehension percent scores in Magindanawn, as well as the oral reading fluency (ORF) scores, by grade. The listening comprehension and reading comprehension percent score cumulative distributions are step-wise (increase in a step-like manner) because each comprehension subtask has only five questions, meaning the scores can only increase in increments of 20 percentage points at a time. The ORF cumulative distributions are smoother due to the fact that ORF is measured with a continuous variable (correct words per minute [cwpm]). For the distributions below, the percentile indicator is on the x-axis and the percent score or cwpm score is on the y-axis. These graphs can be used to identify what percentage of students are reaching certain levels of comprehension or rates of ORF. For example, we can use the y-axis and find 80 percent comprehension and then find the percentile on the x-axis that corresponds to the line graph. Looking at listening comprehension for Magindanawn, shown below, we see that close to 14 percent of Grade 2 and 21 percent of Grade 3 students scored at least 80 percent on the listening comprehension subtask.

Figure G-1. Magindanawn cumulative distribution of listening comprehension scores, by grade

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Percentile

Grade 2 Grade 3


Figure G-2. Magindanawn cumulative distribution of ORF scores, by grade (Passage 1)

Figure G-3. Magindanawn cumulative distribution of reading comprehension scores, by grade (Passage 1)

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Ora

l Rea

ding

Flu

ency

Percentile

Grade 2 Grade 3

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Percentile

Grade 2 Grade 3


Figure G-4. Percent total Magindanawn reading comprehension scores, by grade (Passage 2)

Figure G-5. Magindanawn cumulative distribution of reading comprehension scores, by grade (Passage 2)

24%

13%15% 15%

19%

15%

8% 8%

14%

27%

23%

21%

0%

5%

10%

15%

20%

25%

30%

0 20 40 60 80 100

% o

f Stu

dent

s

Percent Score

Grade 2

Grade 3

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Percentile

Grade 2 Grade 3


Annex H: Mëranaw Cumulative Distribution Graphs The graphs below present the cumulative distributions for the listening comprehension and reading comprehension percent scores in Mëranaw, as well as the oral reading fluency (ORF) scores, by grade. The listening comprehension and reading comprehension percent score cumulative distributions are step-wise (increase in a step-like manner) because each comprehension subtask has only five questions, meaning the scores can only increase in increments of 20 percentage points at a time. The ORF cumulative distributions are smoother due to the fact that ORF is measured with a continuous variable (correct words per minute [cwpm]). For the distributions below, the percentile indicator is on the x-axis and the percent score or cwpm score is on the y-axis. These graphs can be used to identify what percentage of students are reaching certain levels of comprehension or rates of ORF. For example, we can use the y-axis and find 80 percent comprehension and then find the percentile on the x-axis that corresponds to the line graph. Looking at listening comprehension for Mëranaw, shown below, we see that close to 27 percent of Grade 2 and 24 percent of Grade 3 students scored at least 80 percent on the listening comprehension subtask.

Figure H-1. Mëranaw cumulative distribution of listening comprehension scores, by grade

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Grade 2 Grade 3


Figure H-2. Mëranaw cumulative distribution of ORF scores, by grade (Passage 1)

Figure H-3. Mëranaw cumulative distribution of reading comprehension scores, by grade (Passage 1)

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Ora

l Rea

ding

Flu

ency

Grade 2 Grade 3

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Grade 2 Grade 3


Figure H-4. Percent total Mëranaw reading comprehension scores, by grade (Passage 1)

Figure H-5. Mëranaw cumulative distribution of reading comprehension scores, by grade (Passage 2)

28%

16%14%

17%

22%

4%

9% 9%

26% 26%

22%

7%

0%

5%

10%

15%

20%

25%

30%

35%

40%

0 20 40 60 80 100

% o

f Stu

dent

s

Percent Score

Grade 2

Grade 3

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Perc

enta

ge o

f Que

stio

ns C

orre

ct

Grade 2 Grade 3

Date post:	04-Nov-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

ACR - 2019 Regional Early Grade Reading Assessment ...2020/04/21 · mother-tongue-based...

Documents