International Journal of Educational Development 37 (2014) 11–21

Improving reading outcomes in Kenya: First-year effects of the PRIMRInitiative

Benjamin Piper a,*, Stephanie Simmons Zuilkowski b,1, Abel Mugenda a,2

a RTI International, Nairobi Regional Office, Misha Tower, 3rd Floor, 47 Westlands Road, P.O. Box 1181 Village Market, 00621 Nairobi, Kenyab Florida State University, Center for International Studies in Educational Research and Development, Learning Systems Institute, University Center C4600,

P.O. Box 3062540, Tallahassee, FL 32306-2540, United States

A R T I C L E I N F O

Keywords:

Literacy

International education

Reading

Bilingual education

Kenya

A B S T R A C T

While educational participation is high in Kenya, literacy outcomes remain poor. The PRIMR Initiative

aims to improve literacy learning by aligning curriculum and teacher practices with current research,

providing ongoing instructional support and observation, and supplying basic instructional materials

and English and Kiswahili books for students. In a randomized control trial in more than 400 schools in

three counties in Kenya, the intervention improved oral reading fluency and in grade 1 formal and

nonformal schools and grade 2 nonformal schools for both English and Kiswahili. The findings support

the importance of in-classroom teacher support in program implementation to improve literacy

outcomes.

� 2014 Published by Elsevier Ltd.

Contents lists available at ScienceDirect

International Journal of Educational Development

jo ur n al ho m ep ag e: ww w.els evier . c om / lo cat e/ i jed u d ev

1. Introduction

Despite dramatic increases in primary-level education partici-pation over the past decade, the quality of education in Kenyaremains low. Investments by the government and families alikeoften do not result in meaningful learning (Mugo et al., 2011;National Assessment System for Monitoring Learner Achievement,2010). This problem is by no means unique to Kenya, but iscommon in the region (Uwezo, 2012). As a group, the countries ofsub-Saharan Africa have the lowest youth literacy rate in theworld, just 72% (United Nations Educational, Scientific and CulturalOrganization [UNESCO], 2012). This has dramatic consequences fornational development. While longitudinal evidence on literacy inKenya is limited, a study using the South African Cape Area PanelSurvey found that students’ literacy skills were associated withdropout three years later (Marteleto et al., 2008), aligning withsimilar results from the United States (Alexander et al., 1997;Jimerson et al., 2000). As shown in a recent analysis of 15 countries,including Malawi and Ghana, education is associated with wage-earning employment outside of the agricultural sector (Winterset al., 2009). Therefore, youth with poor literacy skills will havedifficulty completing the basic education cycle and, as adults,

* Corresponding author. Tel.: +254 733719966.

E-mail addresses: bpiper@rti.org, piperben@gmail.com (B. Piper),

szuilkowski@lsi.fsu.edu (S.S. Zuilkowski), amugenda@primr.rti.org (A. Mugenda).1 Tel.: +1 850 644 2570.2 Tel.: +254 704 300 900.

http://dx.doi.org/10.1016/j.ijedudev.2014.02.006

0738-0593/� 2014 Published by Elsevier Ltd.

finding steady employment that will support them and theirfamilies.

In order to support the Kenyan government in its efforts toimprove educational outcomes, high-quality evidence on theeffectiveness of affordable and scalable intervention options isneeded. This is the goal of the Primary Mathematics and Reading(PRIMR) Initiative. PRIMR is a program funded by the United StatesAgency for International Development (USAID) and the BritishDepartment for International Development (DFID), and operatingin partnership with Kenya’s Ministry of Education, Science &Technology (MoEST). PRIMR, focused on reading and mathematicsin grades 1 and 2, is using a nested series of randomized controlledtrials to examine the effectiveness of several interventions,including enhanced technology in the classroom and additionalteacher support. In this paper, we focus on the first-year effects of aquality improvement program aimed at grade 1 and 2 teachers informal government and low-cost nonformal private schoolsserving slum communities. In its first year, this program included:(1) teacher training to align classroom practices with currentresearch on literacy acquisition; (2) ongoing instructional supportfrom zonal-level Teachers’ Advisory Centre (TAC) tutors servingsets of 8–19 schools; (3) monthly observation and feedback; and(4) provision of English and Kiswahili books for students, as well asinstructional materials and structured lesson plans for teachers.These components are described more fully in the research designsection below.

While many programs have focused on improving the qualityof literacy instruction in sub-Saharan Africa, and some in Kenya –

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–2112

including the Aga Khan Foundation’s study of Reading to Learn, theHealth and Literacy Intervention, and Buddy Reading – thereremains a gap in the literature with respect to analyses that usemethods for assessing the impact of instructional programstargeting multilingual populations. PRIMR is designed to fill thatgap by utilizing its randomized controlled design to attempt toestimate the impact of the program on literacy outcomes in twolanguages: Kiswahili and English. Although the total researchdesign covers three years, the aim of our analyses completed forthis paper was to assess the effects of this intervention on studentliteracy after one year of implementation. Our specific researchquestions were:

(1) Has the PRIMR literacy intervention increased students’ oralreading fluency? Does this effect differ by language?

(2) Has the PRIMR literacy intervention increased students’reading comprehension? Does this effect differ by language?

In addition to responding to these research questions, we alsoidentify and discuss the differences in effects between formal(government) and nonformal schools and address the cost-effectiveness of the program.

2. Background and context

School fees were effectively abolished in Kenya in 2003, when anew administration came into power. Since that time, gross primaryenrollment rates have risen above 100% (World Bank, 2011). Thisindicates that older youth who did not previously have access toprimary school began enrolling, in addition to the great majority ofthose of primary school age. Additionally, some students repeatedgrades in order to improve outcomes, particularly as they neared thecritically important primary school leaving examination. However,this dramatic increase in enrollment over a short period putconsiderable strain on the government school system, which did notreceive funding increases commensurate with the enrollmentincreases. In 1998, the national student–teacher ratio was 28:1.In 2011, it was 47:1 (World Bank, 2011). Within the sample of publicschools in this study, the student–teacher ratio in grades 1 and 2 was44.6 in October 2012. In addition to handling large classes, Kenyanteachers often deal with space and materials shortages that impairtheir ability to teach effectively (Sifuna, 2007; UNESCO, 2005). Datafrom a study conducted by the Southern and Eastern AfricaConsortium for Monitoring Education Quality (SACMEQ) in 2005found that nationally, only 27% of students had their own readingtextbooks (Onsomu et al., 2005). These figures have improvedrecently; the government textbook policy now mandates a 3:1student-to-textbook ratio, and recent research in Kenya hasindicated that a 3:1 ratio is the average in rural and urban locations(Piper and Mugenda, 2012). While Glewwe et al. (2009) foundlimited effects of reducing the student–textbook ratio in Kenya forgrades 3 through 8, access to textbooks and other print materials iscritically important for children in the earlier grades, who arelearning to read (Neuman, 2004).

Perhaps unsurprisingly, given the high student–teacher ratios,limited teacher training, and lack of sufficient text materials,reading outcomes for students attending Kenyan primary schoolsare generally poor. The results of a series of assessments conductedover the decade since fee abolition converge on a common finding:Kenyan children are not meeting the Ministry of Education, Science& Technology’s benchmarks and on average read far below gradelevel (Mugo et al., 2011; National Assessment Centre, 2010;Onsomu et al., 2005; Piper, 2010; Piper and Mugenda, 2012;Wasanga et al., 2010). For example, the 2011 national Uwezo studyfound that just 57% of third-graders could read basic sentences,and only 30% a second-grade-level story (Mugo et al., 2011).

If children do not learn how to read in the first few years ofprimary school, they will struggle to complete the cycle and are atgreater risk of dropping out. It is therefore crucial to identify andtest interventions that have the potential of making a large impact,can be implemented quickly, and are affordable to be taken to scaleby the Kenyan government. This is the goal of the PRIMR Initiative– to test various options for improving learning outcomes andinstruction in Kenyan schools, using a randomized controlleddesign. The design helps differentiate this contribution, as manypilot programs in the sector do not test the impacts of quality-improvement methods at a medium scale and with enough rigor toidentify a causal impact. In this paper, we focus on early-gradeliteracy outcomes after one year of implementation for pupils ingrades 1 and 2.

3. Research design

3.1. Site

The schools participating in the arms of the PRIMR studyanalyzed in this paper were in peri-urban and rural zones in Nairobi,Nakuru, and Kiambu counties. Peri-urban regions are on theoutskirts of urban areas – near enough that residents can commuteto towns and cities via local transport, but still possessing many ruralcharacteristics, such as agriculture being the predominant economicactivity (Mandere et al., 2010). In Nairobi, the largest city and capitalof Kenya, more than half of the population lives in unplannedinformal settlements, sometimes referred to as slums (UnitedNations Human Settlements Programme [UN-HABITAT], 2013).Many of these settlements have no running water, access toelectricity, or basic sanitation facilities. Critically, the residents donot have ownership of the land and it is therefore illegal for them toconstruct permanent structures.

Low-cost, private nonformal schools are common alternatives topublic government schools, particularly at the primary level. Thenonformal schools participating in PRIMR generally are character-ized by low tuition rates (less than US$10 per month), substandardinfrastructure (predominantly tin roofs and unfinished floors andwalls), high student and teacher turnover, and lack of trainedprincipals and teachers. Despite these issues, many poor familiesbelieve these schools offer a better quality of education than the localgovernment schools (Ngware et al., 2012; Oketch et al., 2012; Tooleyet al., 2008). Smaller class sizes are one reason for this perception(Ngware et al., 2011; Oketch et al., 2012). In urban informal areas ofNairobi, attending a government school predicted poor achievementon the Kenya Certificate of Primary Education (KCPE) exams (Ejakaitet al., 2011). For some families, the choice may not be between agovernment primary school and a nonformal school, but betweenthe nonformal school and no school at all. Though Kenya has a FreePrimary Education (FPE) policy, many formal schools in the locationssurrounding these nonformal settlements have insufficient space forall of the pupils (Oketch et al., 2010). For example, Kibera, which ishome to both PRIMR treatment and control schools, is widely citedas being the largest nonformal settlement in sub-Saharan Africa(UN-HABITAT, 2013). Kibera has only two public primary schoolsserving a catchment area with a population estimated at between200,000 and 1,000,000, with the 2009 Kenyan census citing 250,000residents (Kenya National Bureau of Statistics, 2010).

3.2. Sample

The USAID-funded PRIMR Initiative is supporting a total of 502formal and nonformal schools during the period 2011 through2014. This study focuses on the 125 schools that beganimplementing PRIMR in January 2012 and the 101 schools thatserve as control schools and began implementing the intervention

Table 1Research design and data collected at PRIMR midterm.

Treatment group Schools Pupils assessed

Schools in PRIMR Schools assessed Baseline, January 2012 Midterm, October

2012

Public Nonformal Total Public Nonformal Total Public Nonformal Public Nonformal

PRIMR 66 59 125 40 33 73 651 528 697 558

Control 51 50 101 25 27 52 436 467 476 478

Total 117 109 226 65 60 125 1087 995 1173 1036

Table 2Demographic description of the PRIMR sample by school type and assessment (means and standard errors with p-values from T-tests comparing treatment and control).

School type Variable Baseline (January 2012) Midterm (October 2012)

Control (n = 436) Treatment (n = 651) T-test p-value Control (n = 476) Treatment (n = 697) T-test p-value

Formal Grade 1 44.2 (1.5) 44.9 (0.9) .82 45.6 (1.4) 45.1 (1.0) .81

Child age 7.2 (0.1) 7.1 (0.1) .52 7.7 (0.1) 7.6 (0.1) .38

Female 46.6 (1.0) 48.9 (0.9) .02* 47.2 (0.9) 47.3 (0.7) .09�

Wealth index (out of 9) 3.7 (0.1) 4.3 (0.2) <.01** 3.7 (0.1) 3.9 (0.1) .20

Books in home 44.2 (2.7) 55.3 (3.0) <.01** 45.8 (3.6) 38.7 (3.4) .22

Preschool, pre-unit,

kindergarten

93.7 (1.3) 95.4 (0.9) .22 96.2 (1.1) 97.2 (0.7) .57

Mother is literate 92.3 (1.8) 91.5 (1.6) .69 94.9 (1.0) 96.4 (0.7) .17

Father is literate 94.3 (1.6) 92.5 (1.2) .43 96.3 (0.9) 95.8 (1.0) .51

School type Variable Baseline (January 2012) Midterm (October 2012)

Control (n = 467) Treatment (n = 528) T-test p-value Control (n = 478) Treatment (n = 558) T-test p-value

Nonformal Grade 1 52.5 (1.3) 52.2 (1.7) .35 54.4 (1.2) 51.5 (1.3) .09�

Child age 6.8 (0.1) 6.7 (0.1) .48 7.2 (0.1) 7.3 (0.1) .23

Female 47.1 (1.3) 48.7 (1.3) .92 49.8 (1.1) 47.4 (1.1) .10�

Wealth index (out of 9) 4.3 (0.1) 4.3 (0.1) .87 4.2 (0.1) 4.4 (0.1) .08�

Books in home 61.1 (3.7) 53.4 (2.9) .16 52.1 (2.8) 52.0 (3.9) .97

Preschool, pre-unit,

kindergarten

97.7 (0.8) 93.9 (1.2) .02* 87.4 (2.3) 96.0 (1.3) <.01**

Mother is literate 93.1 (1.1) 92.1 (2.1) .88 97.7 (0.8) 98.3 (0.7) .41

Father is literate 94.6 (0.9) 89.6 (2.7) .11 96.8 (1.0) 97.6 (0.7) .71

� p < .10.* p < .05.** p < .01.

***p < .001.

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–21 13

in January 2014, after the conclusion of the PRIMR study.Government schools in Nairobi, Kiambu and Nakuru counties wereeligible for selection for PRIMR, and the unit of randomization wasthe zone, a set of approximately 15 geographically proximateschools. Each selected zone was randomly assigned to start theintervention in 2012, 2013, or 2014, with the 2014 group serving ascontrols. PRIMR supported the MoEST and several nongovernmentalorganizations (NGOs) serving the nonformal settlements to map theschools. PRIMR employed eligibility criteria to remove schools thatcharged more than US$10 per month per pupil (and were thereforenot considered low-cost primary schools), were not registered withthe government, or did not have a minimum enrollment of 30 pupilsin grades 1 and 2. The researchers then grouped the nonformalschools into clusters of 10 or 15. These clusters were then randomlyassigned to the three treatment groups, as were the governmentzones. Table 1 below presents the research design.

For the portion of the PRIMR research design focused on theimpact of the core PRIMR program at the midterm, the baselinedata set included 2082 pupils in 117 schools: 1087 from publicschools and 995 from nonformal schools. Schools were randomlyselected from the cluster and zone of schools to which theybelonged, and the population of schools in the baseline includedone half of the total number of schools in each zone. Enumeratorsselected the pupils using simple random sampling by having all ofthe students in each grade line up and then randomly selecting fiveboys and five girls each from grades 1 and 2, using a samplinginterval derived from the student population. At the one-yearmidterm data collection in October 2012, the data set utilized to

measure the impact of PRIMR included 2209 students in 117schools, with 1173 pupils in public schools and 1036 in nonformalschools. The schools and students were sampled the same way asat the baseline. Power calculations indicated that this sample sizewas sufficient to detect an impact of at least .20 standarddeviations (SD). Table 2 provides a demographic description of thesample by public/nonformal school type, treatment group, andassessment stage (January 2012 baseline and October 2012midterm). In formal schools, at baseline, pupils in treatmentschools were slightly more wealthy (p-value <.01) and 13.2% morepupils in treatment children had some reading materials at home(p-value <.01) than control schools. Those differences disappearedat the endline. For nonformal schools, we found that 3.8% fewertreatment children attended kindergarten, pre-unit, or preschoolat the baseline. At the midterm, 8.6% more treatment childrenattended kindergarten, pre-unit, or pre-school. This variabilitycould have been due to the mobility of the nonformal schoolpopulation.

3.3. Procedures

The PRIMR interventions discussed here centered on improvingteacher practices related to literacy acquisition, moving teachersaway from using whole-class oral repetition as their primarypedagogical approach and toward research-supported strategies toimprove literacy acquisition, such as explicitly teaching lettersounds, blending, and concept of word (Piper and Mugenda, 2013a;Snow et al., 1998). The goal was a balanced approach to literacy

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–2114

learning, with attention to both decoding skills and interpretation(August and Shanahan, 2006; National Institute for Child Healthand Human Development, 2000; Pressley, 1998).

The literacy arm of the program included 150 structuredlessons in both Kiswahili and English. The design of the programallowed pupils to learn letter sounds in Kiswahili before theEnglish letters, so that in English, teachers could point out whichletters sounded similar to or different from Kiswahili. Teachersreceived modest instructional aids, including pocket charts andflashcards with letters on them, and students received low-coststudent books that aligned with the scripted lessons. In PRIMRclassrooms, the student-to-textbook ratio was 1:1 rather thanthe standard 3:1 mandated by the national textbook policy andfound in PRIMR schools at the baseline (Piper and Mugenda,2012). In addition, the student books produced by PRIMR weremuch longer than the standard texts, at 150 pages. The PRIMRbooks focused on letters, phonological awareness, and decodingskills – the building blocks of reading. They also exposedstudents to controlled-text stories relevant to their local context,as well as stories for teachers to read aloud, with a heavyemphasis on comprehension strategies. PRIMR found thatexisting books on the Kenyan market placed very little emphasison letters, phonological awareness, or decoding in eitherlanguage. Together, the PRIMR lessons and materials wereintended to move children who had not attended preschooland had little exposure to the alphabet from basic letterknowledge to full fluency and comprehension within one schoolyear.

A significant amount of PRIMR’s time and technical expertisewas spent on teacher professional development. Each participatingteacher and head teacher received 10 days of training during thefirst year of implementation. The training provided brief substan-tive overviews of reading topics, then allowed ample time forteachers to practice with the scripted lesson plans and activities.TAC tutors and instructional coaches, responsible for supportingteachers in clusters of schools, received 15 days of training toensure that they would be capable of guiding teachers as theyimplemented the program. In the PRIMR pilot program, these TACtutors and instructional coaches were trained by the PRIMRtechnical staff with input and oversight by the MoEST technicalteam that manages PRIMR. This structure mirrors the QualityAssurance and Standards Officer (QASO) system that Kenya has inplace to provide technical support to the educational system. Giventheir critical role in shaping instructional improvement nationallyin Kenya, the TAC tutors and instructional coaches are seen ascritical to the program’s ongoing success and scalability. Kenya hasa relatively robust instructional support system, with each of 1052zones supported by a TAC tutor. With the relatively reasonableinstructional ratios currently in place, the TAC tutor systempresents an opportunity for Kenya to manage large-scaleinstructional reform. In fact, under PRIMR, it is the TAC tutors(for zones of formal schools) and instructional coaches (for clustersof nonformal schools) who provided the training for teachers andhead teachers as well as follow-up support. This means that PRIMRtested a model that was designed to be implemented within theexisting MoEST structures and available financial resources.

As noted, the baseline data collection was completed in January2012, at the beginning of the school year, and the midpointassessment was completed in October 2012. Oral reading fluency(ORF) and reading comprehension (RC) were assessed using aversion of the Early Grade Reading Assessment, or EGRA (see Goveand Wetterberg, 2011), adapted for use in Kenya. The EGRA usedin PRIMR included several subtasks, including letter-soundfluency, decoding fluency, oral reading fluency (timed anduntimed in the baseline study, as well as silent and aloud in themidterm), associated reading comprehension scores, listening

comprehension, and Maze comprehension scores. PRIMR usedRasch modeling to measure item-level reliability for each of thesubtasks. The measures focused on in this paper were those for thetimed, read-aloud stories and their associated reading compre-hension questions, as discussed further in the following section.

The assessments were conducted by Kenyan field staff who hadworked with PRIMR lead implementer RTI International since 2007on several studies using EGRA. These assessors received five days oftraining before assessments commenced for both the baseline andmidterm studies. Interrater reliability scores were high – 95.3% forKiswahili and 96.1% for English at baseline and 95.6% for Kiswahiliand 95.4% for English at midterm.

3.4. Measures

For both administrations, students’ oral reading fluency wasmeasured using the EGRA passage-based reading subtask andcomprehension questions, a technique derived from the DynamicIndicators of Basic Early Literacy Skills (DIBELS) instruments (Goodand Kaminski, 2002). The EGRA tool was constructed withsignificant input from an international community of practiceand has been used in over 70 countries, including many in sub-Saharan Africa (Gove and Wetterberg, 2011). In the United States,extensive research has been undertaken to examine the criterionvalidity of oral reading fluency assessments such as the DIBELS(Marston, 1989; Shinn, 1998). Researchers have found correlationsbetween .49 and .83 between oral reading fluency scores andachievement tests in a number of states, suggesting that oralreading fluency is a valid proxy for literacy skills (Barger, 2003;Shaw and Shaw, 2002; Silberglitt et al., 2006; Vander Meer et al.,2005; Wilson, 2005). However, to our knowledge, there have beenno studies comparing the EGRA oral reading fluency assessment tonorm-referenced achievement tests or long-term student out-comes in sub-Saharan Africa.

Students were given a printed passage and allotted one minuteto read it aloud to an assessor. The assessor kept time and notedany words that were read incorrectly. The score was the number ofwords read correctly per minute (WCPM). In the baseline sample,the WCPM scores ranged from 0 to 126.4 on English and 0 to 120 onKiswahili. In the midterm sample, the WCPM scores ranged from 0to 178.7 on English and 0 to 114.0 on Kiswahili. Readingcomprehension was assessed via five questions that followedthe oral reading exercise. Students were asked only the questionsthat related to material they had read – that is, if they had notreached the end of the passage within one minute, they were notasked the final question. The score was a percentage of the fiveitems answered correctly; students may therefore have had scoresof 0%, 20%, 40%, 60%, 80%, or 100%. In the baseline and midtermsamples, the scores ranged from 0% to 100% for both Kiswahili andEnglish. Afterward, PRIMR statisticians created a sample of pupilswho had been given both assessments in random order, then usedequating methods to equate the WCPM and reading comprehen-sion scores between the baseline and midterm. This impactevaluation paper uses the scores equated across the baseline andmidterm assessment. In English, the Cronbach’s alpha for the ORFsubtask was .68 at baseline, while the alpha for readingcomprehension was .74. In Kiswahili, the respective alphas were.81 and .82 (Piper and Mugenda, 2013a).

In order to estimate the causal impact of PRIMR on studentachievement, we constructed difference-in-differences (DID)models that included several covariates that the PRIMR baselinestudy showed were correlated with the outcome measures ofinterest (Piper and Mugenda, 2012). These were gender, a wealthindex derived from nine items relating to household possessions,and a dichotomous variable differentiating those pupils who hadreading materials at home from those who did not.

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–21 15

3.5. Data-analytic plan

The January 2012 baseline was designed to test whether therewere statistically significant differences in the outcome variablesbetween treatment groups. Although PRIMR randomly selectedassigned schools to treatment groups, our analyses showed thattreatment schools outperformed control schools by 2.7 WCPM inEnglish reading (p-value .04), 1.7 WCPM in Kiswahili reading (p-value .04), 2.7% in English comprehension (p-value .08), and 5.4% inKiswahili comprehension (p-value .07). These differences aresmall, but significant. There are no systematic reasons why thedifferences would occur, as the selection of zones from thepopulation of zones and the assignment of zones to treatment wasdone using random number generators. Given the small butstatistically significant differences in outcomes observed betweenthe treatment and control groups at the January 2012 baseline,which may have introduced bias, we decided to use a DID model toidentify the effects of the intervention rather than a simpler modelonly comparing results at the midterm. We fit the differences-in-differences estimator using Ordinary Least Squares (OLS) regres-sion models with covariates for the ORF and RC outcomes.

DID models compare changes in a program’s outcome variablesat two different assessment points for treatment and controlgroups by removing the secular trend (the change in outcome forthe control groups over time). This allows the analysis to separateprogram impact from changes in the population not due toprogram impact (Murnane and Willett, 2011; Murnane et al.,2006). The DID model was fit to a data set that contained fourgroups of students, differentiated by whether they attendedschools randomly assigned to treatment or control groups and byassessment round – January or October 2012. To answer theresearch questions in this paper, we fit the following statisticalmodel:

IMPACTi j ¼ b0 þ b1PRIMRi j þ b2MIDTERMi j þ b3PRIMR � MIDi j

þ gX þ ðei j þ m jÞ

Table 3Simple comparisons at midterm: program effects and effect sizes.

Subtask Language Formal or

nonformal

Grade Control

Mean

Oral reading fluency

(words correct

per minute, WCPM)

Kiswahili Formal 1 11.24

2 23.45

Nonformal 1 22.66

2 34.18

English Formal 1 10.70

2 28.42

Nonformal 1 34.85

2 57.17

Proportion of readers

(percentage of population)

Kiswahili Formal 1 0.35

2 8.19

Nonformal 1 9.17

2 30.33

English Formal 1 1.07

2 3.93

Nonformal 1 14.04

2 38.90

Reading comprehension

(percentage correct)

Kiswahili Formal 1 9.01

2 26.16

Nonformal 1 29.80

2 43.68

English Formal 1 4.29

2 11.52

Nonformal 1 19.06

2 25.91

for student i in school j, where e is an individual residual and m isthe school-level residual, and the vector of covariates X, withassociated regression parameters g, represents the impact of thecontrol predictors. Parameter b0 is the intercept, b1 represents themain effect of PRIMR, and b2 is the main effect of being in themidterm data set as opposed to the baseline. Parameter b3

represents the impact of the interaction between PRIMR andMIDTERM, and is the DID parameter that is of principal interest, asthe measure of PRIMR impact (Piper, 2009). Using the svycommands in Stata, we were able to fit a statistical model inorder to account for the nested nature of schools and students, anduse standard errors that accounted for that nesting.

The PRIMR data include a subset of 921 grade 1 pupils who werefollowed between January and October 2012. PRIMR’s initialanalyses of this longitudinal data set suggested that the impact ofPRIMR identified using the DID estimator slightly underreportedthe impact of PRIMR. This is logical given that longitudinal pupilsbenefited from the program for an entire year, but the DID estimateincluded approximately 12.7% of pupils who reported enteringPRIMR schools after the beginning of the academic year inJanuary 2012.

4. Findings

4.1. RQ1: Has the PRIMR literacy intervention increased students’ oral

reading fluency?

In order to answer the first research question, whether the PRIMRliteracy intervention increased students’ oral reading fluency rates,we estimated the average oral reading fluency rates of pupils at theOctober 2012 midterm. Given the randomized selection andassignment of the PRIMR treatment and control groups, anydifferences in the outcome would be related to the treatment.Table 3 shows the fluency rates of students in treatment and controlschools by grade and school type in both English and Kiswahililanguages. The columns labeled ‘‘program impact’’ show thedifference in the mean fluency rates between the treatment and

Treatment Program impact

Standard

error

Mean Standard

error

Standard

deviation

Program

effect

Effect

size

0.90 17.70 1.31 14.18 6.46 0.46

1.17 28.72 1.31 14.96 5.27 0.35

1.07 32.17 1.33 21.23 9.51 0.45

1.34 45.13 1.98 26.48 10.95 0.41

1.37 23.66 2.45 22.47 12.96 0.58

1.57 40.35 2.51 24.65 11.93 0.48

1.74 47.27 2.21 34.74 12.42 0.36

2.72 73.14 3.72 46.72 15.97 0.34

0.33 9.15 2.05 20.94 8.80 0.42

1.86 20.57 2.56 30.34 12.38 0.41

1.94 24.01 3.15 45.66 14.84 0.33

3.43 48.00 4.01 62.25 17.67 0.28

0.70 7.66 2.14 20.14 6.59 0.33

1.36 19.94 2.92 28.82 16.01 0.56

1.98 24.19 3.33 48.08 10.15 0.21

3.74 57.62 4.30 63.51 18.72 0.29

1.25 18.55 1.57 19.08 9.54 0.50

1.64 33.36 1.61 20.36 7.20 0.35

1.61 39.24 1.94 28.69 9.44 0.33

1.69 50.00 2.53 30.79 6.32 0.21

0.67 10.31 1.39 13.34 6.02 0.45

0.98 16.52 1.33 14.21 5.00 0.35

1.55 18.37 1.18 20.88 �0.69 �0.03

2.09 29.17 1.51 23.87 3.26 0.14

Table 4Differences-in-differences estimates of PRIMR treatment effects on outcome measures, by grade and school type.

Outcome measure Language Metric Grade 1 Grade 2

Formal Nonformal Formal Nonformal

Oral reading fluency English WCPM 8.74** 14.05*** 2.62 16.21***

Kiswahili WCPM 3.31* 10.97*** 0.77 12.88***

Proportion of readers at the benchmark English % 4.78� 10.16*** 12.09* 21.07**

Kiswahili % 7.21** 15.49*** 8.25* 17.34**

Reading comprehension (percentage correct) English % 3.07� 3.15� �4.87 9.06*

Kiswahili % 5.81* 12.00*** 1.56 9.64*

WCPM, words correct per minute.� p < .10.* p < .05.** p < .01.*** p < .001.

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–2116

control groups. PRIMR effects using this analytic method rangedfrom 5.3 WCPM in Kiswahili in formal schools in grade 2–16.0WCPM in English nonformal schools in grade 2. Effect sizes rangedfrom .34 SD (English nonformal grade 2) to .58 SD (English formalgrade 1).

During a Kenya National Examinations Council (KNEC) work-shop in August 2012, KNEC and the Ministry of Education, Science& Technology set literacy benchmarks for grade 2 at 65 WCPM forEnglish and 45 WCPM for Kiswahili Piper and Mugenda, 2013a).PRIMR’s midterm report examined whether PRIMR had an impacton the proportion of readers who could read at this benchmark inOctober 2012 (Piper and Mugenda, 2013a). Table 3 shows quite alarge difference in that proportion, with PRIMR treatment effectsranging from 6.6 percentage points in English formal grade 1 to18.7 percentage points in English nonformal grade 2. Effect sizesranged from .21 SD for English nonformal grade 1 to .56 SD forEnglish formal grade 2.

The analyses above were based on uncontrolled comparisonsbetween outcomes for treatment and control groups in October2012. This simple comparison would be valid if there were nosystematic differences in baseline scores given the randomselection and assignment. However, as discussed above, PRIMR’sanalysis showed some small but statistically significant differ-ences between treatment and control schools at the baseline(Piper and Mugenda, 2012). Therefore, we also present the resultsof our DID models in Table 4 below. This table shows statisticallysignificant impacts for PRIMR on oral reading fluency in bothEnglish and Kiswahili for formal and nonformal students in grade

Kiswahili

1.8 0.3 5.4 6.80.38.5

7.1

23.6

6.6

15.2 7.6

17.6

0

5

10

15

20

25

30

35

40

45

50

Formal Nonformal Formal Nonformal

Grade 1 Grade 2

Per

cent

age

Rea

ding

at

Ben

chm

ark

Baseline Normal Gain PRIMR Eff ect

Fig. 1. Fluent

1, and nonformal students in grade 2, but not for grade 2 formalstudents in either English or Kiswahili. The PRIMR causal effectson words read correctly per minute ranged from 3.3 WCPM inKiswahili in formal grade 1 to 16.2 WCPM in English in nonformalgrade 2.

PRIMR’s causal impact on the proportion of readers reading at thebenchmark from the DID estimator were statistically significant atthe .05 level for all combinations of grades 1 and 2, formal andnonformal, and for both English and Kiswahili – except for Englishformal grade 1, which was significant at the .10 level. PRIMRincreased the proportion of pupils who could read at the benchmarksubstantially. Fig. 1 derived from fitted DID regression models,shows this graphically for both languages. The bottom segments ofeach bar show the proportion of readers at the baseline, as defined bythe government benchmark. In all subsamples, this group was lessthan 7% of the population. The middle segments of the bars show theincrease in the proportion of readers in control schools. Gains insome control schools were substantial, with the middle segmentsshowing that control schools increased the proportion of pupils whocould read by up to 23.6 percentage points in grade 2 nonformalschools for Kiswahili and 28.8 percentage points in grade 2nonformal schools for English. The top segments of the bars showthe PRIMR causal impact. The PRIMR effect was often quite large, andnearly doubled or more than doubled the proportion of readers in allcategories (formal/nonformal, grades 1 and 2, and English andKiswahili). The size of the PRIMR effect was .23 SD in English grade 1,.34 SD in English grade 2, .39 SD in Kiswahili grade 1, and .30 SD inKiswahili grade 2.

English

2.3 06.5 6.41.1

13.7 1.4

28.8

4.2

10.3

9.5

22.2

0

10

20

30

40

50

60

For mal Nonformal For mal Nonformal

Grade 1 Grade 2

Per

cent

age

Rea

ding

at

Ben

chm

ark

Baseli ne Normal Gain PRIMR Eff ect

readers.

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–21 17

In order to answer the second part of our first research question,concerning whether the impact of PRIMR on oral reading fluencydiffered by school type, we compared the magnitude of the PRIMRcausal effect on oral reading fluency and the proportion of pupilsreading at benchmark between formal and nonformal schools. Wefound that the PRIMR causal impacts were larger in the nonformalschools than in the formal schools, in both subjects and in bothgrades. Table 4 above shows that the DID estimator impacts for oralreading fluency were nonsignificant only for grade 2 formalschools, while they were large for grade 2 nonformal schools. Incomparison, the impacts were larger for nonformal schools thanformal schools in grades 1 and 2, and for both English andKiswahili. For example, the DID impact on Kiswahili in grade 1 was3.3 WCPM for formal schools compared with 11.0 WCPM fornonformal. Similarly, for the proportion of pupils able to read at theMoEST benchmark, while there were some statistically significantimpacts for all combinations of the measures, the impacts werelarger for nonformal than for formal schools. For example, the DIDcausal effect on the proportion of readers at the benchmark ingrade 2 was 12.1 WCPM for formal schools and 21.1 WCPM fornonformal schools. Despite their apparent deficiencies in struc-tures, supplies, and staff training in comparison to the publicgovernment schools, the nonformal schools nonetheless appear tohave responded more effectively to PRIMR’s quality-improvementintervention.

4.2. RQ2: Has the PRIMR literacy intervention increased students’

reading comprehension?

Our second research question focused on the impact of PRIMRon reading comprehension (percentage correct) scores. A compar-ison of the October 2012 data, presented in Table 3 above, showedthat reading comprehension percentage-correct scores werehigher in treatment schools, both formal and nonformal, in grades1 and 2, and in English and Kiswahili. The only exception was inEnglish nonformal grade 1, where students in control schoolsslightly outperformed students in treatment schools: 19.1–18.4%.PRIMR causal effects in the simple comparison were 9.5% forKiswahili formal grade 1 and 9.4% for Kiswahili nonformal grade 1.Interestingly, the simple comparison showed larger effects forPRIMR in Kiswahili than in English across the grades and settings.

Table 4 above presented the impact of PRIMR on readingcomprehension using the DID estimator, the more conservativemethodological choice given the small but significant differencesin comprehension scores for treatment and control groups atbaseline (Piper and Mugenda, 2012). Using the DID estimates, theresults show that PRIMR had an impact on reading comprehensionfor Kiswahili grade 1 formal (5.8%), grade 1 nonformal (12.0%),grade 2 nonformal (9.6%), and English grade 2 nonformal (9.1%).PRIMR impacts were significant at the .10 level for English grade 1formal (3.1%) and English grade 1 nonformal (3.2%). While theseeffects were statistically significant, they were relatively small inmagnitude – recall that each item answered correctly was worth20 percentage points. PRIMR had no impact on reading compre-hension for grade 2 formal students in either English or Kiswahili.

5. Limitations

In Kenya, vast differences in educational participation andliteracy have been observed across regions and levels of urbanicity(Mugo et al., 2011; Wasanga et al., 2010). Therefore, we note thatthe PRIMR intervention and the impact evaluation findingspresented here reflect a sample that has a somewhat higherpercentage of peri-urban schools than is representative of Kenya.The effects of the intervention may vary by location. A DFID-fundedextension of PRIMR into two additional counties, which began in

December 2012, will address this issue and determine whetherPRIMR’s effect differs by urbanicity.

In the findings section, we compared the results of PRIMR informal and nonformal schools and showed that the impact ofPRIMR was larger in the nonformal subsector. Student assignmentto schools was not random; parents who choose to send their childto a nonformal school are likely different in unobservable waysfrom parents who do not. However, in the Kenyan context, whereno form of primary schooling is cost-free despite the FPE policy,parental selection of a fee-charging nonformal school is not merelya reflection of family socioeconomic status but a complex weighingof a number of variables, including distance from home, schoolquality, and tuition and supplementary costs. We therefore arguethat school type is important to consider, although the results werestatistically significant in both types of schools.

The random selection of zones and random assignment of zonesto treatment and control groups was done carefully using a randomnumber generator, but the results still showed small variations inperformance at the baseline, which concerned us. Utilizing the DIDmodel reduced concerns about bias, however, because DID modelsremove baseline differences from treatment effect estimates.

Although we identified substantial statistically significant,positive effects of the PRIMR intervention, two factors suggestthat the identified effects may in fact have been a lower bound.First, the Kenyan MoEST and PRIMR staff decided that all teachers –grades 1 and 2 – should use the grade 1 materials during the firstyear of the program, as the initial literacy scores were very low forboth grade levels. However, many grade 2 teachers in formalschools, as well as parents, did not support this decision, leading toreduced buy-in and commitment to the PRIMR program in thoselocations. Unsurprisingly, then, the PRIMR causal impact waslowest in grade 2 formal schools. There was some empiricalevidence that this was due to lack of buy-in rather than toapplicability to grade 2, as PRIMR found that the impact onnonformal schools (where the program had much greater buy-in ingrade 2) was actually higher in grade 2 than grade 1. The DFID-funded PRIMR Rural Expansion Programme (December 2012–February 2015) offers an opportunity to test whether using grade 1materials at both grade levels was the correct decision, as grade 2teachers in the Rural Expansion schools will have used the grade 2materials from the beginning of the program intervention in 2013.Although initial findings from the DFID Rural Expansion Pro-gramme suggested that the content of the material was toodifficult for the grade 2 students who did not undergo PRIMR ingrade 1, we hypothesize that the increased buy-in will lead tolarger student learning gains in spite of this.

The second issue that likely reduced the observed first-yearimpact of PRIMR in formal government schools was a three-weeklong-teacher strike at the beginning of the third school term(September–November) in 2012. The PRIMR midterm assessmentin October 2012 thus began the week after schools reopened,immediately following an extended term break for students inAugust 2012 and the strike. Thus, many students had been awayfrom school for nearly two months at the time the midterm datacollection team visited their schools. As has been documented inthe United States, loss of reading skills as a result of skill atrophyduring long breaks away from school immediately before theassessment might have depressed scores (Alexander et al., 2007;Borman and D’Agostino, 1996; White and Kim, 2008). Based oninternal PRIMR assessments, several formal zones had significantlyworse reading outcomes on the midterm assessment than they didin July 2012, before the term break and the strike. The presence ofstatistically significant, positive effects of PRIMR, despite theproblems with take-up and assessment timing, is promising forfuture expansion of the activities of programs like PRIMR. It alsoprovides support for the theoretical underpinning of PRIMR, which

3 TAC tutors’ non-instructional duties may interfere with their ability to provide

regular support at the teacher level. One recent study found that TAC tutors spent

60% of their time on administrative duties and just 40% working with teachers

(Kisirkoi, 2011). This difference in duties between the nonformal school coaches

and the government school TAC tutors is a factor which must be addressed in order

to plan for a successful national scale-up.

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–2118

is that simplicity and specificity are essential to ensuring buy-inand therefore improving outcomes.

Finally, we note that the observed improvements in oral readingrate – among both control and PRIMR students – could have beendue to Hawthorne effects among students and teachers. Forexample, teachers may have taught differently knowing that theywere being directly observed. While a recent review of theHawthorne effect confirmed that such an effect exists, at least inhealth-related fields (McCambridge et al., 2014), little research hasbeen conducted on its influence on teacher behavior. In Kenya,where student- and school-level test scores are publicly availableand frequently cited in the media, and where parents areremarkably sensitive to quality in their enrollment decisions fortheir children, teachers are already under a great deal of pressureto produce results. Although we cannot fully discount its potentialimpact, given that bi-monthly teacher observation via TAC tutors isan explicit component of the intervention, it appears that theinfluence of a Hawthorne effect on the performance of PRIMRteachers would be mirrored in a scale-up of the program beyondthe pilot stage.

6. Discussion

The analysis has shown that the first year of PRIMR had apositive impact on the three outcomes of interest – oral readingfluency, the percentage of pupils who read at the MoEST’sbenchmark, and reading comprehension – although not on allcombinations of language, grade, and school type. Using the DIDestimator, we found statistically significant impacts on oralreading fluency in grade 1 formal and nonformal schools andgrade 2 nonformal schools for both English and Kiswahili. Therelationship between PRIMR and oral reading fluency wasnonsignificant in grade 2 formal schools, however. The magnitudeof the impact was largest for grade 1 students in nonformal schoolsand was larger in English than in Kiswahili. In earlier research, wefound that Kiswahili words were significantly longer than Englishwords (Piper, 2010), providing one possible explanation for thedisparity between English and Kiswahili fluency impacts. Anotherpossible explanation is the considerable emphasis on English as themedium of instruction in Kenya (Piper, 2010), meaning that pupilswere exposed more to English than Kiswahili outside of theirliteracy classes. Students therefore had more practice using theirnew skills in English than in Kiswahili, leading to a more rapidimprovement in outcomes in that language.

These findings suggest that literacy improvement programs likePRIMR could contribute to Kenya’s desire to improve the quality ofliteracy outcomes, as indicated in the draft National EducationSector Strategic Plan document (Ministry of Education, Science &Technology, 2013). Our analysis showed that, holding keycovariates constant, PRIMR’s first-year intervention had a statisti-cally significant impact on the proportion of pupils who met theKNEC benchmark in both languages, both grades, and in bothschool settings (formal and nonformal) (Piper and Mugenda,2013a). Similar to the impact on oral reading fluency, themagnitude of the PRIMR impact on the proportion meeting thebenchmark was larger for nonformal schools than formal. Ouranalysis of Fig. 1 showed that the PRIMR effect was nontrivial (withassignment to treatment schools making it twice as likely that astudent could learn to read at the benchmark between January andOctober 2012), and logistic regression analyses presented byPRIMR (Piper and Mugenda, 2013a) have shown that, all thingsequal, a student in a treatment school was between 1.9 and 27.9times more likely to read at the benchmark than students incontrol schools. This evidence suggests that the PRIMR effect is notonly statistically significant, but substantively large.

Our findings also indicated that the PRIMR interventions didlead to modest improvements in children’s ability to understandwhat they read. Table 4 above showed that PRIMR had a largerimpact on Kiswahili reading comprehension than on Englishreading comprehension, in both grades and both school settings.Kiswahili and English comprehension impacts were nonsignificantin formal grade 2 and significant only at the .10 level for Englishgrade 1 (both formal and nonformal). The PRIMR design led us toexpect that the impact would be higher in Kiswahili than in Englishreading comprehension; given the focus on oral language in thePRIMR grade 1 English materials, we would not expect to observeany impacts on reading comprehension. On the other hand, theprogression of controlled-text stories and comprehension strate-gies that PRIMR has been using in Kiswahili is structured to rapidlyimprove oral reading fluency and comprehension for students,even in grade 1. Our comparisons between English and KiswahiliPRIMR impacts suggested that PRIMR had a larger impact thanexpected on English comprehension, but the impact on English wasmuch smaller than the impact on Kiswahili comprehension.

Our analysis was concerned with determining whether theimpact of PRIMR differed by formal or nonformal school type.Table 2 showed that in an absolute sense, oral reading fluency, theproportion of pupils reading at the benchmark, and readingcomprehension scores were greater in nonformal than formalschools in grades 1 and 2 and in both English and Kiswahili. This ispartially explained by the fact that PRIMR’s sample of nonformalschools was limited to the capital, Nairobi, while the formal schoolsample included schools from Nairobi, Kiambu, and Nakurucounties. PRIMR’s analyses showed no statistically significantdifferences in the outcomes between formal and nonformalschools in Nairobi at baseline (Piper and Mugenda, 2012). Ourfindings suggest that the educational performance improvementsin Nairobi were greater in high-poverty nonformal slum settle-ments than they were in peri-urban locations – even in relativelywealthy mid-sized towns, such as Nakuru town, and much higherthan in rural settings in Kiambu and Nakuru counties.

This paper presents evidence that the PRIMR first-year impactwas larger in magnitude in nonformal schools than in formal publicschools. Evidence from PRIMR’s monitoring data suggests highlevels of usage of PRIMR lesson plans in both formal and nonformalschools, with more than 95% of teachers utilizing PRIMR readinglessons for Kiswahili and English during announced and unan-nounced observations (RTI International, 2013). PRIMR officersobserved that nonformal teachers were much more open to fullyimplementing the PRIMR strategies, and that the dearth ofcompeting materials or lesson plans in nonformal settings mixedwith the lack of previous training for many PRIMR nonformalteachers made them more willing to change their instructionalmethods and perhaps acquire positive attitudes toward pupils’reading acquisition after PRIMR training. Moreover, the PRIMR-hired instructional coaches assigned to the nonformal schoolclusters made more frequent classroom observation visits (twotimes per month per teacher) than did the government’s TAC tutors(approximately one time per month per teacher). This was partlybecause of the geographic proximity of the coaches’ urban schoolscompared to the relative geographic spread of the TAC tutors’schools, and partly because TAC tutors had numerous otherresponsibilities, whereas the instructional coaches could focusexclusively on PRIMR implementation.3 Both of these factors likely

-$1.94

$0.48

-$2.25

$1.85

$0.75

-$2.50

-$2.00

-$1.50

-$1.00

-$0.50

$0.00

$0.50

$1.00

$1.50

$2.00

$2.50

Book TAC tutor training

Supervision Teac her training

Lesson Pla ns

PRIMR Spend s More

_____________

MoEST Spend s More

Cost Area

Fig. 2. Difference in per pupil cost between PRIMR and MoEST, 2012, by cost item

(US$).

10.63

6.78 7.24

4.73

0

2

4

6

8

10

12

Treatment Control

WC

PM

English WCPM Gain per U S$Kiswahili WCPM gain per U S$

Fig. 3. Per student cost effectiveness of PRIMR Initiative (treatment) and MoEST

(control) models.

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–21 19

contributed to the larger impact of PRIMR in nonformal settings,and given the interaction between the two (higher take-up andmore classroom visits), we were unable to determine which onewas more causally responsible for the differences betweennonformal and formal settings. However, the apparent willingnessof nonformal school teachers to adopt new pedagogies supportsthe contention of researchers in Kenya (Ngware et al., 2011; Oketchet al., 2010) that the urban poor may indeed be making a rationalchoice when they choose ‘‘low-quality’’ private schools over localgovernment schools with better facilities and more highly trainedstaff, as discussed above.

The meaningful first-year impacts of PRIMR on studentachievement have implications for teacher professional develop-ment in Kenya. PRIMR’s research has shown that teachers can besensitive to in-service teacher professional development (ITPD) ifthat ITPD is closely linked to the books and lesson plans used inschools, and if teachers are observed and supported frequently(Mejia and Piper, 2013). PRIMR’s decision to invest heavily inclassroom observational support (as compared to other programsfocused on improving literacy outcomes) is important to note. AKenyatta University study showed that pre-service teachereducation institutions in Kenya were unable to adequately prepareteachers for early literacy instruction, and the MoEST is consider-ing policy options to improve on the pre-service sector (Bunyiet al., 2011). The DFID-funded expansion of PRIMR is lookingparticularly at the modality of in-service professional developmentthat is able to impact pupil learning most cost-effectively. This andother ongoing studies should examine how enhanced teachersupport can change classroom behaviors.

PRIMR has several embedded randomized control trials withinthe larger research design, organized to investigate the relativeeffectiveness and cost-effectiveness of investments. The programimplementers were particularly interested in understanding howand whether PRIMR could identify the impacts of an affordableprogram model on student achievement. PRIMR’s analyses showedthat the key elements of PRIMR – TAC tutors’ training, transportreimbursements, the training of teachers and head teachers by theTAC tutors and instructional coaches, 1:1 textbook ratios in allsubjects, and ongoing instructional support for teachers – costapproximately US$2.21 per subject per pupil. That same package ofmaterials, based on the existing budgets of the MoEST and the perpupil allocation given to counties, cost the Kenyan government US$2.58 per subject per pupil in 2012 (Piper and Mugenda, 2013b).Fig. 2 below compares the PRIMR costs and the budgeted MoESTcosts per cost item. The figure shows that in 2012, PRIMR spentmore money per pupil on lesson plans (by $0.75 per pupil). This islogical given that the current system still has very little capacity todesign lesson plans or teachers’ guides internally. The 10 days ofteacher training and monthly cluster meetings in PRIMR mean thatPRIMR spent $1.85 more on teacher training than the existingsystem in 2012. The TAC tutor training invested by PRIMR (15 days

per year) equates to $0.48 more per pupil, which is a relativelysmall amount. On the other hand, the MoEST spent much more perpupil per book than PRIMR, as the average cost of a book on theKenyan market was US$4.12 in 2012, compared to US$0.76 forPRIMR’s first-year texts. Additionally, the MoEST budgeted muchmore on supervision than PRIMR spent. In total, PRIMR spentUS$0.37 less per pupil than the existing system in 2012.

Due to PRIMR’s lower per-student costs and larger studentachievement gains, the PRIMR program is much more cost effectivethan the MoEST system (see Fig. 3). The figure shows that thefluency gains (WCPM) per U.S. dollar within PRIMR were muchgreater than in the government system, in both English andKiswahili. In English, for example, PRIMR students gained 10.6WCPM per U.S. dollar, while government system students gained6.8 WCPM per dollar. These figures represent the costs at the zonallevel and below, rather than the development and managementcosts of PRIMR, which are relatively expensive. But given thatPRIMR’s design is to test a cost-effective model for eventual scale-up by the MoEST, and that future production runs of the existingPRIMR books will not have to include the design costs in their price,it appears that a decision by the MoEST to use the PRIMR methodsat scale can result in both significant cost savings and improvedstudent outcomes.

Despite these promising findings, challenges will be encoun-tered in taking this program to scale. Bold and colleagues, in theirevaluation of the scale-up of a contract teacher program in Kenya,found that the government-managed arm of the program wasineffective while the NGO-managed arm produced significantimprovements in student achievement (Bold et al., 2013). Thegovernment had difficulty in monitoring the project and the hiringprocess was flawed. Government contract teachers were often paidlate, an occurrence that was associated with student test scores. Incontrast, PRIMR was organized from the outset of the program towork within the MoEST system. The TAC tutors provide the teachertraining and the classroom support. Therefore, the effect identifiedin this paper already accounts for the somewhat more modestimplementation of educational programs by the MoEST. This doesnot negate the challenges associated with taking any programtaken to scale, but the criticism noted by Bold et al. (2013) isalready accounted for in the PRIMR design. In January 2014, theMoEST included an expansion of PRIMR in a grant application tothe Global Partnership for Education, so the ability of the PRIMRdesign to be implemented at scale will be tested.

Funding

Implementation of the Primary Math and Reading (PRIMR)Initiative in Kenya (2011–2014) is led by RTI International. The

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–2120

program is funded by the United States Agency for InternationalDevelopment (USAID/Kenya), under the Education Data forDecision Making (EdData II) Blanket Purchase Agreement, TaskOrder 13, Contract No. AID-623-M-11-00001, 2011–2014. USAID/Kenya participates in PRIMR design and implementation decisions,and also granted the authors permission to use the data set foradditional analysis and dissemination. The lead author, Dr.Benjamin Piper, serves as USAID’s Chief of Party on the PRIMRInitiative. Costs for preparing the article were covered jointly byPRIMR and by an internal Professional Development Award issuedby RTI International. The contents of the article and decisions aboutpublication were the sole responsibility of the authors. Theauthors’ views expressed in this publication do not necessarilyreflect the views of USAID, the United States Government, or RTIInternational.

Acknowledgments

We appreciate the hard work and commitment of the KenyanMinistry of Education, Science and Technology, including theProgramme Development and Implementation Team housed at theMoEST and including members from across the Ministry and otherorganizations. The USAID/Kenya education team, specifically Dr.Dwaine Lee, Dr. Christine Pagen, and Dr. Theresiah Gathenyadesigned a research study worth analysis. The PRIMR M&E team,led by Author, is a very strong team focused on collecting validdata. From a technical perspective, Jessica Mejia, Joseph DeStefanoand Melinda Taylor’s leadership and support were instrumental inthis research and in the PRIMR Initiative. We also acknowledge theexcellent editorial work of Erin Newton. Finally, this research wasmade possible due to a Professional Development Award from RTIInternational and the President of RTI, Dr. Wayne Holden.

References

Alexander, K.L., Entwisle, D.R., Horsey, C.S., 1997. From first grade forward: earlyfoundations of high school dropout. Sociology of Education 70 (2), 87–107,http://dx.doi.org/10.2307/2673158.

Alexander, K.L., Entwisle, D.R., Olson, L.S., 2007. Lasting consequences of thesummer learning gap. American Sociological Review 72, 167–180, http://dx.doi.org/10.1177/000312240707200202.

August, D., Shanahan, T. (Eds.), 2006. Developing Literacy in Second-LanguageLearners: A Report of the National Literacy Panel on Language, MinorityChildren, and Youth. Lawrence Erlbaum Associates, Mahwah, NJ, USA, http://dx.doi.org/10.2167/le129b.0.

Barger, J., 2003. Comparing the DIBELS Oral Reading Fluency Indicator and the NorthCarolina End-of-Grade Reading Assessment. North Carolina Teacher Academy,Asheville, NC, USA.

Bold, T., Kimenyi, M., Mwabu, G., Ng’ang’a, A., Sandefur, J., 2013. Scaling Up WhatWorks: Experimental Evidence on External Validity in Kenyan Education.Working Paper. Retrieved February 10, 2014, from http://www.cgdev.org/doc/kenya_rct_webdraft.pdf.

Borman, G.D., D’Agostino, J.V., 1996. Title 1 and student achievement: a meta-analysis of federal results. Educational Evaluation and Policy Analysis 18, 309–326, http://dx.doi.org/10.3102/01623737018004309.

Bunyi, G., Wangia, J., Magoma, C., Limboro, C., Akyeampong, K., 2011. Learning toTeach Reading and Mathematics and Influences on Practice: A Study of TeacherEducation in Kenya. Retrieved February 5, 2014, from https://www.sussex.a-c.uk/webteam/gateway/file.php?name=report-kenya-1july2011.pdf&site=320.

Ejakait, E., Mutisya, M., Ezeh, A., Oketch, M., Ngware, M., 2011. Factors associatedwith low achievement among students from Nairobi’s urban informal neigh-borhoods. Urban Education 46 (5), 1056–1077, http://dx.doi.org/10.1177/0042085911400323.

Glewwe, P., Kremer, M., Moulin, S., 2009. Many children left behind? Textbooks andtest scores in Kenya. American Economic Journal: Applied Economics 1 (1),112–135.

Good, R.H., Kaminski, R.A., 2002. Dynamic Indicators of Basic Early Literacy Skills, 6thed. Institute for the Development of Educational Achievement, Eugene, OR, USA.

Gove, A., Wetterberg, A. (Eds.), 2011. The Early Grade Reading Assessment:Applications and Interventions to Improve Basic Literacy. RTI International,Research Triangle Park, NC, USA. Retrieved July 26, 2013, from http://www.rti.org/pubs/bk-0007-1109-wetterberg.pdf.

Jimerson, S., Egeland, B., Sroufe, L.A., Carlson, B., 2000. A prospective longitudinalstudy of high school dropouts: examining multiple predictors across

development. Journal of School Psychology 38 (1), 525–549, http://dx.doi.org/10.1016/s0022-4405(00)00051-0.

Kenya National Bureau of Statistics, 2010. Population and Housing Census. Author,Nairobi.

Kisirkoi, F.K., 2011. Effectiveness of Teacher Advisory Centres (TACs) in teacherprofessional development in Nairobi County. Paper Presented at the DistanceEducation and Teacher Education in Africa Conference. Maputo, Mozambique.

Mandere, N.M., Ness, B., Anderberg, S., 2010. Peri-urban development, livelihoodchange and household income: a case study of peri-urban Nyahururu, Kenya.Journal of Agricultural Extension and Rural Development 2 (5), 73–83.

Marston, D.B., 1989. A curriculum-based measurement approach to assessingacademic performance: what it is and why do it. In: Shinn, M.R. (Ed.), Curricu-lum-based Measurement: Assessment of Special Children. Guilford, New York,pp. 18–78.

Marteleto, L., Lam, D., Ranchhod, V., 2008. Sexual behavior, pregnancy, and school-ing among young people in urban South Africa. Studies in Family Planning 39(4), 351–368.

McCambridge, J., Witton, J., Elbourne, D.R., 2014. Systematic review of theHawthorne effect: new concepts are needed to study research participationeffects. Journal of Clinical Epidemiology 67 (3), 267–277, http://dx.doi.org/10.1016/j.jclinepi.2013.08.015.

Mejia, J., Piper, B., 2013. Training Modalities in Kenya: Improving Outcomes Cost-effectively. RTI International, Nairobi, Kenya Unpublished manuscript.

Ministry of Education, Science & Technology (MoEST), 2013. National EducationSector Support Programme (NESSP): Priorities and Proposed Sector Pro-grammes 2013/14-2017/18 August 2013 Draft. Author, Nairobi.

Mugo, J., Kaburu, A., Limboro, C., Kimutai, A., 2011. Are Our Children Learning?Annual Learning Assessment Report. Uwezo, Nairobi. Retrieved July 26, 2013,from http://www.uwezo.net/wp-content/uploads/2012/08/KE_2011_Annua-lAssessmentReport.pdf.

Murnane, R.J., Willett, J.B., 2011. Methods Matter. Oxford University Press, NewYork.

Murnane, R.J., Willett, J.B., Cardenas, S., 2006. Has PEC contributed to improvementin Mexican public education? In: Reimers, F. (Ed.), Aprender mas y mejor.Secretarıa de Educacion, Publica, Mexico.

National Assessment Centre, 2010. Monitoring of Learning Achievement for Class 3in Literacy and Numeracy in Kenya. Kenya National Examinations Council,Nairobi.

National Institute for Child Health and Human Development [US], 2000. Report ofthe National Reading Panel. Teaching Children to Read: An Evidence-basedAssessment of the Scientific Research Literature on Reading and its Implicationsfor Reading Instruction. Reports of the Subgroups (NIH Publication No. 00-4754) NICHD, National Institutes of Health, Washington, DC. Retrieved Febru-ary 10, 2014, from http://www.nichd.nih.gov/publications/pubs/nrp/pages/report.aspx.

Neuman, S.B., 2004. The effect of print-rich classroom environments on earlyliteracy growth. Reading Teacher 58 (1), 89–91.

Ngware, M.W., Oketch, M., Ezeh, A.C., 2011. Quality of primary education inputs inurban schools: evidence from Nairobi. Education and Urban Society 43 (1), 91–116, http://dx.doi.org/10.1177/0013124510379131.

Ngware, M.W., Oketch, M., Ezeh, A.C., Mutisya, M., Ejakait, C.E., 2012. Assessingthe impact of free primary education using retrospective and prospectivedata: lessons from the Nairobi case study. International Journal of Researchand Method in Education 35 (1), 71–92, http://dx.doi.org/10.1080/1743727x.2012.666717.

Oketch, M., Mutisya, M., Ngware, M., Ezeh, A.C., 2010. Why are there proportion-ately more poor pupils enrolled in non-state schools in urban Kenya in spite ofFPE policy? International Journal of Educational Development 30 (1), 23–32,http://dx.doi.org/10.1016/j.ijedudev.2009.08.001.

Oketch, M., Mutisya, M., Sagwe, J., 2012. Do poverty dynamics explain the shift to aninformal private schooling system in the wake of free public primary educationin Nairobi slums? London Review of Education 10 (1), 3–17, http://dx.doi.org/10.1080/14748460.2012.659056.

Onsomu, E., Nzomo, J., Obiero, C., 2005. The SACMEQ II Project in Kenya: A Study ofthe Conditions of Schooling and the Quality of Education. Southern and EasternAfrica Consortium for Monitoring Education Quality (SACMEQ)/Kenya Ministryof Education, Science and Technology, Harare/Nairobi. Retrieved July 26, 2013,from http://ddp-ext.worldbank.org/EdStats/KENstu05.pdf.

Piper, B., 2009. Student-Centered Pedagogy’s Causal Mechanisms: An ExplanatoryMixed Methods Analysis of the Impact of In-service Teacher ProfessionalDevelopment on Student and Teacher Outcomes in Ethiopia. (Unpublisheddoctoral dissertation)Harvard Graduate School of Education, Cambridge, MA,USA.

Piper, B., 2010. Kenya Early Grade Reading Assessment findings report. Prepared forthe William & Flora Hewlett Foundation under the Monitoring of LearningOutcomes in Sub-Saharan Africa Project, Contract No. 2008-3367. RTI Interna-tional and East African Development Consultants, Research Triangle Park, NC,USA. Retrieved July 26, 2013, from https://www.eddataglobal.org/countries/index.cfm?fuseaction=pubDetail&ID=275.

Piper, B., Mugenda, A., 2012. The Primary Math and Reading (PRIMR) Initiative:baseline report. Prepared under the USAID Education Data for Decision Making(EdData II) Project, Task Order Contract No. AID-623-M-11-00001. RTI Interna-tional, Research Triangle Park, NC, USA. Retrieved July 26, 2013, from https://www.eddataglobal.org/countries/index.cfm?fuseaction=pubDetail&ID=480.

Piper, B., Mugenda, A., 2013a. The Primary Math and Reading (PRIMR) Initiative:midterm impact evaluation. Prepared under the USAID Education Data for

B. Piper et al. / International Journal of Educational Development 37 (2014) 11–21 21

Decision Making (EdData II) project, Task Order Contract No. AID-623-M-11-00001. RTI International, Research Triangle Park, NC, USA. Retrieved July 26,2013, from https://www.eddataglobal.org/countries/index.cfm?fuseaction=pub-Detail&ID=486.

Piper, B., Mugenda, A., 2013b August. Teaching and teacher change. In: Presented atthe USAID 2013 Education Summit, Washington, DC.

Pressley, M., 1998. Reading Instruction That Works: The Case for Balanced Teaching.Guilford, New York.

RTI International, 2013. The Primary Math and Reading (PRIMR) Initiative AnnualReport. Prepared under the USAID Education Data for Decision Making (EdDataII) Project, Task Order Contract No. AID-623-M-11-00001. Author, ResearchTriangle Park, NC, USA.

Shaw, R., Shaw, D., 2002. DIBELS Oral Reading Fluency-Based Indicators of Third-Grade Reading Skills for Colorado State Assessment Program (CSAP). Universityof Oregon, Eugene, OR, USA. Retrieved February 10, 2014, from http://dibel-s.uoregon.edu/docs/techreports/shaw_csap_technical_report.pdf.

Shinn, M.R., 1998. Advanced Applications of Curriculum-based Measurement.Guilford, New York.

Sifuna, D.N., 2007. The challenge of increasing access and improving quality: ananalysis of universal primary education interventions in Kenya and Tanzaniasince the 1970s. International Review of Education 53 (5–6), 687–699, http://dx.doi.org/10.1007/s11159-007-9062-z.

Silberglitt, B., Burns, M.K., Madyun, N.H., Lail, K.E., 2006. Relationship of readingfluency assessment data with state accountability test scores: a longitudinalcomparison of grade levels. Psychology in the Schools 43 (5), 527–535, http://dx.doi.org/10.1002/pits.20175.

Snow, C.E., Burns, M.S., Griffin, P., 1998. Preventing Reading Difficulties in YoungChildren. National Academy Press, Washington, DC.

Tooley, J., Dixon, P., Stanfield, J., 2008. Impact of free primary education in Kenya.Educational Management Administration and Leadership 36 (4), 449–469,http://dx.doi.org/10.1177/1741143208095788.

United Nations Educational, Scientific and Cultural Organization (UNESCO), 2005.Challenges of Implementing Free Primary Education in Kenya. UNESCO,Nairobi.

United Nations Human Settlements Programme (UN-HABITAT), 2013. State of theWorld’s Cities 2012/2013: Prosperity of Cities. Routledge, New York. Retrieved July25, 2013, from http://www.unhabitat.org/pmss/listItemDetails.aspx?publicatio-nID=3387.

UNESCO, 2012. Adult and Youth Literacy. UNESCO Institute for Statistics, Montreal.Uwezo, 2012. Are our Children Learning? Literacy and Numeracy Across East Africa.

Uwezo, Nairobi. Retrieved July 26, 2013, from http://www.uwezo.net/wp-content/uploads/2012/09/RO_2012_UwezoEastAfricaReport.pdf.

Vander Meer, C.D., Lentz, F.E., Stollar, S., 2005. The Relationship between OralReading Fluency and Ohio Proficiency Testing in Reading. University of Oregon,Eugene, OR, USA.

Wasanga, P.M., Ogle, A.M., Wambua, R.M., 2010. Report on Monitoring of LearnerAchievement for Class 3 in Literacy and Numeracy. Kenya National Examina-tions Council, Nairobi.

White, T.G., Kim, J.S., 2008. Teacher and parent scaffolding of voluntary summerreading. Reading Teacher 62 (2), 116–125, http://dx.doi.org/10.1598/rt.62.2.3.

Wilson, J., 2005. The Relationship of Dynamic Indicators of Basic Early Literacy Skills(DIBELS) Oral Reading Fluency to Performance on Arizona Instrument toMeasure Standards (AIMS). Tempe School District No. 3, Tempe, AZ, USA.Retrieved February 14, 2014, from https://dibels.uoregon.edu/docs/techre-ports/arizona.pdf.

Winters, P., Davis, B., Carletto, G., Covarrubias, K., Quinones, E.J., Zezza, A., Azzarri, C.,Stamoulis, K., 2009. Assets, activities and rural income generation: evidencefrom a multicountry analysis. World Development 37 (9), 1435–1452, http://dx.doi.org/10.1016/j.worlddev.2009.01.010.

World Bank, 2011. EdStats – Education Statistics. World Bank, Washington, DC.Retrieved July 26, 2013, from http://go.worldbank.org/ITABCOGIV1.