Evaluating Conditional Schooling and Health Programs1

Susan W. Parker Centro de Investigación y Docencia Económicas

Luis Rubalcava

Centro de Investigación y Docencia Económicas

Graciela Teruel Universidad Iberoamericana

Revised draft October, 2006

1 Draft prepared for chapter in forthcoming volume of Handbook of Development Economics, Volume 4, North Holland,. We gratefully acknowledge very helpful comments by T. Paul Schultz and Edward Miguel as well as by conference participants at the Rockefeller Conference Center in Bellagio in May, 2005.

1

Index: 1. Introduction. 2. Theoretical considerations of conditional programs. 2.a. Income and substitution effects. 2.b. Bargaining effects. 2.c. Targeting effects. 3. Progresa Program description. 3.a Program description. 3.b. Discussion. 4. The Progresa evaluation: Design, data and empirical issues. 4.a. Design and data collection. 4.b. Estimators of the evaluation. 4.c. Attrition. 5. Progresa rural evaluation results.

5.a. Short term results: rural areas 1998 to 2000. 5.b. Other studies related to the evaluation.

5.c. Medium term impacts: rural areas 1998-2003. 6. Evaluating conditional schooling-health programs around the world.

6.a. Urban Progresa 6.b. Other programs in Latin America 6.c. Other countries.

7. Analysis: Some lessons learned.

2

1. Introduction The design of social policies which encourage human capital accumulation among the poor, thus breaking the transmission of poverty from one generation to the next is a basic concern for development economists. Roughly speaking, these policies can be classified as either supply side interventions attempting to improve the infrastructure or quality of education or “demand-side” interventions attempting to provide incentives for poor parents to keep their children longer in school. We analyze in this chapter the development and evaluation of a new genre of social programs termed conditional cash transfer programs which have become widespread across Latin America. Conditional transfer programs typically link monetary transfers to human capital investment, generally education, health or a combination of both. They now represent a major instrument for combating poverty in Latin America, an area noted for high inequality and high rates of extreme poverty. Furthermore, their popularity is spreading outside the region. A number of other countries including Turkey and Cambodia now also have conditional programs, and others such as Indonesia are in the planning/implementation stages. The programs are also under consideration in Africa. These transfer programs are considered innovative because of their conditionality, e.g. they condition the receipt of monetary benefits on such behaviors as regular school attendance and preventive clinic visits. Effectively they are a subsidy to schooling and health, reducing the shadow price of human capital acquisition. The programs typically thus have a price or substitution effect due to the conditionality and an income effect from the increase in household income. While most research focuses on the conditionality aspect, the programs also are likely to have an important effect on alleviating current poverty. Most also give the cash directly to the mother and thus may have a bargaining effect as well. Note that from an economist’s point of view, conditioning might not necessarily seem welfare enhancing, although there are economic arguments which would justify conditionality, which we discuss in more detail in the chapter. The popularity of conditional transfer programs may also reflect that governments find the programs more attractive politically than unconditional transfers, e.g. they may be less likely to be viewed as a “handout.” The programs have generally had little relation to supply or quality improvements in schools and health clinics, although some programs have provided money for supply side interventions in order to reduce potential reductions in quality due to crowding. While improving quality of public education and health institutions has been viewed to be outside the scope of conditional cash programs, the issues of how quality may limit or constrain the impacts of the programs is an area beginning to receive more attention. We focus here primarily on a case study of Progresa (the Education, Health and Nutrition Program), a Mexican anti-poverty program which has served as a model for the implementation of conditional programs in other countries including Colombia, Nicaragua, Honduras, and Jamaica, and on which most evidence exists on impacts. Progresa conditions cash transfers on children’s enrollment and regular school attendance as well as clinic attendance. Besides its design the program has become noteworthy because it was subject to a large scale rigorous evaluation effort in rural areas which included an experimental design. While experimental designs to evaluate social programs have become more common within the United States, they are still relatively rare within the developing world. The Progresa evaluation has also become a

3

model for the evaluation of other conditional programs in the region, some of which also have experimental designs. In the evaluation design of Progresa, a subset of eligible communities was randomly assigned to a treatment (320 communities) or control (186 communities) group to receive benefits about 18 months later. A number of impact studies were produced taking advantage of the experimental design which we review. We also review results from the newer conditional cash transfer programs. The chapter thus analyzes what we know about the success or potential success of conditional cash transfer programs as a mechanism for reducing poverty. We comment on the potential defects and identify what research which is still needed in order for broader conclusions to be drawn. Note that because of their objective aim of reducing the poverty of the next generation, the evaluation of conditional cash programs would ideally be long-term, e.g. from early childhood through adulthood. Progresa, the oldest of the conditional cash transfer programs has impacts results based only on 6 years, clearly insufficient for measuring long term impacts. Thus, many questions as to the long term impacts of conditional cash transfer programs will necessarily need to be studied in the future. The chapter is organized as follows. In Section 2, we present theory of the potential effects of conditional cash transfer programs. Sections 3, 4 and 5 present the design, evaluation and results of the Progresa program. Section 6 reviews the initial findings of conditional cash programs in other countries. Section 7 concludes with some lessons learned from the evaluation of conditional cash transfer programs and areas for future research. 2. Theoretical considerations of conditional programs The main innovation of conditional cash transfer programs is the linking of benefits to human capital investment, particularly of children. The aim is thus to alleviate current poverty through monetary transfers as well as future poverty, by increasing human capital of children.2 In this section, we discuss the main effects of conditional cash transfer programs from the point of view of economic theory. We focus the analysis on school subsidies, which constitute the largest fraction of the monetary benefits provided under these programs. 2.a. Income and substitution effects In order to discuss the income and substitution effects of conditional cash transfers, we present a two-period model extending Behrman, Parker and Todd (2005b). In the first period of the model, individuals are children and can allocate their time between leisure, market work, home production, and school. In the second period, they are adults and can choose between leisure, market work, and home production. Market work in the second period has a wage that depends on the time spent in school in the first period, and work in home production in the second period has a productivity that depends on

2 An area of some debate, which we retake later on in the text, is the extent to which the program can fulfill both objectives e.g. of reducing both current and future poverty simultaneously.

4

time spent in school in the first period. Let 1C and 2C denote consumption of the individual as a child and as an adult,3 1L and denote leisure, and denote work in home production, amount of schooling,

2L 1h 2hS T the time endowment, the child wage

rate, the adult wage rate, the child labor contribution to house production, the adult labor contribution to house production, and

1W)(2 SW )( 11 hM

),( 22 ShM B transfers to the individual from parents and other family members. We assume that and are increasing in labor at a decreasing rate. Also, let denote amount of time spent working in period 1 and the amount of time spent working in period 2.

1M

2M 1t

2t β denotes the discount rate and the subsidy paid for amount of schooling attended. We assume diminishing marginal utility of consumption and of leisure in each period and diminishing marginal productivity of schooling on second period marginal product of labor in house production and on the second period wage. We also assume that consumption and leisure are normal goods in both periods. There is no direct utility from schooling, which solely provides a technology for increasing the second period income.

sp

Individuals maximize the objective function

),,(),( 222111 LCULCU β+

subject to the constraints

,111 TLhtS ≤+++

,222 TLht ≤++

).1/(),()1/()()()1/( 2222111121 rShMrSWthMWtSpBrCC s +++++++=++

An optimality condition that holds in any interior solution of the problem is

)),,()('( 2222211 ShMSWtMUpMUMU SCsCL +=− β

or equivalently,

)).,()('()( 2222211 ShMSWtMUpWMU SCsC +=− β

The left-hand side is the cost of time spent in school, in term of monetary direct costs and foregone leisure (or equivalently, foregone earnings from work, since another optimality condition equates the marginal utility from leisure with the marginal utility in added consumption derived from work). The right-hand side is the marginal benefit of spending additional time in school, i.e. higher earnings as an adult.

The optimality condition discussed above shows that the subsidy affects the marginal costs of schooling in the same way as would a decrease in the child wage rate: the subsidy reduces the shadow wage (or relative value) of children’s time in activities other than school.

3 Let P1=P2=1.

5

The benefit to schooling depends on how much time the individual as an adult spends working either at the market or at home. Another optimality condition that holds in any solution to the problem above is

).(),( 222 SWShM h =

If labor productivity in home production does not rise as rapidly as in the market with schooling, at least at the secondary school level and beyond, an implication of a school subsidy will be a reduction in the time spent in home production. If women dedicate more time to housework in the absence of subsidies, and if the marginal effect of schooling on labor productivity in housework is decreasing in the time devoted to housework, a result of school subsidies may be an increased participation of women relative to men in the labor market.4

A decrease in the direct costs of schooling, resulting from the subsidy, has both substitution and income effects. The substitution effect decreases the amount of time spent in leisure and working at home or in the market as a child and increases the amount of time spent in school. The income effect (an increase in life-time earnings) increases consumption of all normal goods, namely of leisure and consumption in the first and second periods. Thus, the net effect on childhood time spent in leisure in the first period is ambiguous. The net effect on time spent working and time spent in school is also, in principle, ambiguous. However, if the substitution effect on leisure dominates the income effect, then leisure in the first period will go down and time spent in nonleisure activities will go up. Because the subsidy increases the relative benefit of school relative to work, we expect that time spent in school will go up and time spent working will go down. The optimality condition can be rewritten as a tangency condition

)/()),()('(/ 1222221 sSCC pWShMSWtMUMU −+=β

where the left-hand side gives us the slope of the indifference curves between present and future consumption for the child and the right-hand side gives us the slope of the frontier of the feasible consumption set. The model so far ignores the indivisibilities built in the program with respect to school subsidies. Consider a version of the model in which there is no proportional school subsidy (i.e. 0=sp ) but rather a subsidy is paid to every child that obtains a level of education larger or equal than and a subsidy of zero to every child that does not. Here may represent the minimum mandatory attendance to receive the benefits of the program. The resource constraint of the individual is now

p

minS

minS

),1/()()1/()()()()1/( 2222111121 rhMrSWthMWtSpIBrCC +++++++=++

where

⎩⎨⎧

=<

=min

min

S S if1S S if0

)(SI .

There is a discontinuity now in the frontier of the feasible consumption set. This subsidy scheme will affect children differently according to the decisions regarding

4 We thank T. Paul Schultz for bringing this point to our attention.

6

schooling before school subsidies were introduced. For children who were already receiving more than units of education before the introduction of school subsidies, the program has only an income effect.

minS5 For children who were not receiving that level

of education and who do not receive it either after the introduction of the subsidies, the program has neither an income nor a substitution effect. And for children who did not receive minS units of education before the program but who do so after, there are both income and substitution effects. Given the heterogeneity of preferences and constraints, the extent to which the program has a significant impact on the human capital and work of children can only be determined through empirical analysis. 2.b. Bargaining effects Note that it is not obvious why a government interested in increasing the welfare of the child would prefer school subsidies over unconditional transfers in the context of the simple model discussed above. Many might argue that conditionality requirements are paternalistic and therefore not necessarily welfare improving relative to unconditional transfers. One potential motivation for conditioning benefits is that there are some social returns to investing in education which are not reaped by the individual. A review of the literature, however, was unable to find empirical studies which demonstrated that social returns were significantly higher than private returns (see the discussion in Martinelli and Parker 2003). An alternative motivation to condition transfers is the role of conditionality in implementing outcomes that are favorable to the child in the context of intrahousehold bargaining. For instance, unconditional transfers to the child may be counterweighed by a reduction in the resources transferred by family members to the child. In fact, conditional cash transfer programs almost always specify the monetary benefits to be received by the mother of the household. This design feature was motivated by a growing social science literature which argues that resources under the control of women tend to have a greater impact on the well-being of children than resources under the control of men (see e.g. Thomas 1990). In order to discuss the bargaining effects of conditional cash transfers, we present in here a simple two-period model based on Martinelli and Parker (2003). We consider a family composed by a man, a woman and a child. There is a single consumption good. The man cares only about his own consumption and the consumption of the child; similarly, the woman cares only about her own consumption and that of the child. That is, from the point of view of the two adults, child’s consumption is a public good. We have chosen this simple formulation as the paper focuses on the possibility of the adults disagreeing with respect to child consumption. Preferences of the man and the woman are given by the utility functions and , respectively, where

and are the (nonnegative) consumption levels of the man, the woman and the child.

),( kmm CCU ),( kff CCU

fm CC , kC

In the first period, only the two adults consume. The man and the woman supply inelastically each T units of time in a labor market. The adults also decide how to allocate the child's time, T, between child labor and schooling, S. Finally, the adults decide how much to leave as a (nonnegative) bequest to the child, B. Wages for the

5 We are ignoring here for simplicity the fact that the frontier of the feasible consumption set may be nonlinear.

7

three types of labor are equal to , and the price of consumption goods is normalized to one. The first period budget constraint for the family is given by:

1W

.3 1TWBSCC fm ≤+++

In the second period, only the child consumes. Besides any bequests from the parents, the child obtains a labor income given by . The second period budget constraint is then:

)(2 STW

).(2 STWBCk +≤

Decisions about and B are made by the two adults in the first period according to the Nash bargaining solution, that is maximizing

SCC fm ,,

),),()(),(( d

mkmmdfkff UCCUUCCU −−

where and represent the utilities the two adults would experience if they were unable to reach an agreement about household decisions. It may reflect what would happen if the marriage is dissolved or if the adults stay married but behave with respect to the household in a non-cooperative way (see e.g. Lundberg and Pollak 1993).

dfU d

mU

Define as the solution to *S 1)('2 =STW . This is the “efficient” level of schooling in terms of maximizing the family's lifetime income. From the first order conditions of the household problem, we obtain that any interior solution belongs to one of two cases. In the first case, and We refer to a family in this situation as bequest-constrained. In intuitive terms, a family in this situation would like to leave “negative bequests” to the child; since negative bequests are ruled out, the family instead sets schooling below the efficient level, or equivalently, sets child labor above the efficient level. In the second case, and . We refer to a family in this situation as bequest-unconstrained. In intuitive terms, adults leaving positive bequests are better off setting human capital investment at its efficient level as they can trade between bequests and investment (see e.g. the discussion in Becker and Murphy 1988). We can expect the poorest families to be bequest-constrained, as a family is bequest-constrained if and only if It is immediate that, holding preferences constant, inefficiently low levels of human capital will be associated with poverty.

*SS < ).(2 STWCk =

*SS = 0≥B

).( *2 STWCk ≤

We now introduce in the model a government agency with a positive budget G. The agency has two available policies: it can either give G as an unconditional transfer to the adults, or it can provide a schooling subsidy . In this last case, we assume that is set so as to induce the family to choose schooling level of . For a bequest-constrained family, the level of schooling corresponding to a conditional transfer is necessarily higher than that corresponding to an unconditional transfer since the former includes a substitution effect. Since for a bequest-constrained family , we obtain that the child is better off with a conditional transfer. Indeed, under plausible conditions discussed by Martinelli and Parker (2003), the mother is also better off with

sp sp

spG /

)(2 STWCk =

8

conditional transfers. Intuitively, if the mother gives more weight than the father to the children’s future consumption, conditioning transfers moves the solution to the family’s bargaining problem in the direction of the mother’s preferences. If the family is bequest-unconstrained, however, conditioning transfers makes every member of the family worse-off. Intuitively, in this case conditional transfers induce an inefficiently high level of schooling, which depresses the family’s lifetime income. 2.c. Targeting effects Gahvari and de Mattos (2005) argue that conditional cash transfer programs can be useful to implement redistribution without distortionary losses, using the combination of cash and in-kind transfers as a screening device. They consider a situation in which an indivisible good such as formal education can be provided in different qualities and people must decide which of the qualities to consume. In a similar situation, Besley and Coate (1991) have shown that some redistribution can be obtained if the state provides a low quality for free and charges a head tax, at the cost of some deadweight loss. Gahvari and de Mattos (2005) show that such deadweight loss can be avoided if the state can provide a tax rebate or conditional transfer to those consuming the lower quality good. Intuitively, without conditional transfers, the only instrument that the government has to achieve self-targeting is the quality of the publicly provided good, so the government is (generically) forced to provide a quality that is different from what would be optimal for the poor. Introducing conditional transfers provides the government one more instrument, so that in many cases there are combinations of good quality/conditional transfer that achieve self screening (in the sense of being undesirable for the non poor) and are efficient (in the sense that the poor would not be willing to accept an offer of increasing or decreasing the quality of the publicly provided good financed by a change in their net tax). 3. Progresa Program Description 3.a. Program Description Progresa began operating in 1997 and has grown quickly to become the principal anti-poverty strategy of the Mexican government, occupying about half the annual poverty budget. By the end of 2004, 5 million families were receiving benefits, corresponding to almost one fourth of the Mexican population (Appendix Table 1). Progresa conditions cash transfers on children’s enrollment and regular school attendance as well as clinic attendance. The program also includes in-kind health benefits and nutritional supplements for children up to age five, and pregnant and lactating women. The program thus provides benefits in education, health, and nutrition. Combining three different components e.g. education, health and nutrition in one program was aimed at creating synergies. For instance, children who suffer from malnutrition might be more likely to drop out of school or repeat years of school, which would imply that attempts to insure children go to school will be more effective if combined with adequate nutrition and health programs. There is, however, little direct evidence which has tested the hypothesis of synergies, nor was the evaluation or program design of Progresa designed to do so. However, based on a review of the literature on the determinants of education, health and nutrition investments from around the world, Behrman (2000)

9

suggests that synergies in Progresa might be substantial, particularly with regard to the impact of preschool nutrition on schooling outcomes. Education Under the education component, Progresa provides monthly educational grants; and monetary support or in-kind school supplies. Education grants are given for children under 22 years of age and enrolled in school between the third grade of primary and the third grade of senior high school (e.g. up until twelfth grade).6 Originally, the program provided grants only for children between the third and ninth grade. In 2001, the grants were extended to high school. Table 1 shows the monthly grant levels available for children between the third grade and the twelfth grade in the first semester of 2006 (Appendix Table 2 provides the historical trends of benefits since the program initiated, grants are nominally adjusted for inflation every semester). Grants increase as children progress to higher grades and beginning at the junior high level, are slightly higher (by 10 to 15%) for girls than for boys. Higher grants for girls were originally motivated, according to Progresa, by the observation that in rural areas, girls tended to have a higher dropout rate than boys after finishing primary school. Thus, the higher grant levels were aimed at compensating for this lower achievement. 7 In the first semester of 2006, the specific grant amounts range from $US11.00 (120 pesos) in the third grade of primary to about $US60 (665 pesos) for boys and $US69 (760 pesos) for girls in the third year of senior high school. For comparison, note that the minimum wage in Mexico was 48 pesos per day in 2006 (with some minor variations by region), corresponding to about 1060 pesos monthly for full time work (22 days). By senior high school then, the grant amounts represent about two thirds of a minimum wage. The grants are given every two months during the school calendar. To receive the grant parents must enroll their children in school and ensure regular attendance (i.e., students must have a minimum attendance rate of 85%, both monthly and annually). Failure to comply will lead to the loss of the benefit, at first temporarily, but eventually permanently. Program rules allow students to fail each grade once, but students are not allowed to repeat a grade twice, at that point education benefits are discontinued permanently for the youth. Note this allows a student theoretically to receive two years of grants for the same grade for each grade in which the student enrolls. Enrolment and attendance are verified before grants are paid. 8 All monetary grants are given to the mother of the family with the exception of scholarships for upper-secondary school, which can be received by the youth themselves.

6 We refer to grades 1 through 6 as primary school (primaria), 7 through 9 as junior high school (secundaria) and 10 to 12 as senior high school (media superior) throughout the text. 7 Nevertheless, as shown in Behrman, Sengupta and Todd, 2000 and Parker and Pederzini, 2001, actual attainment of girls in terms of years of completed schooling in rural areas is quite similar to that of boys. The seeming paradox between lower enrollment and similar attainment in years of schooling arises because boys tend to have higher repetition rates than girls. 8 Operationally, there are two basic forms which verify enrolment and regular attendance. The E1 form is given to parents, who take it to the specific school where each child is to be registered to be signed by a school teacher/director to certify enrolment at the beginning of each school year. The E2 form, for monthly attendance records, is sent directly to the schools with names of registered children taken from the E1 forms and then forwarded from the schools to the Progresa offices, which process the information before sending out the corresponding transfers.

10

Health and Nutrition The health care component provides basic health care for all members of the family, with some emphasis on preventive health care (Table 2). These services are provided by public health institutions in Mexico including the Secretary of Health and the Mexican Social Security Institute. The nutritional component includes a fixed monetary transfer equal to about $US16.50 (180 pesos) monthly (specified to be for “improved food consumption” although Progresa does not monitor the expenditures of beneficiaries), as well as nutritional supplements, which are principally targeted to children between the ages of four months and two years, and pregnant and lactating women. They are also given to children aged 2 to 4 if any signs of malnutrition are detected. Mothers visit the clinic at least once a month (more if they are pregnant or have small children) and pick up nutritional supplements monthly. To receive the fixed health and nutrition transfer, all members of beneficiary families must adhere to a regular schedule of health clinic visits. The calendar of visits varies by the age and gender of each individual (Table 3). 9Beneficiaries (generally mothers) are also required to attend monthly health and nutrition talks at the clinic on topics such as nutrition, hygiene, infectious diseases, immunization, family planning, and chronic diseases detection and prevention. Under the recent extension of education grants to the high school level, high school students are also required to attend (separate) talks on topics aimed towards adolescents. Size of Monetary Transfers Progresa has a maximum limit of monthly benefits for each family equivalent in 2006 to about $US100 for families with children in primary and junior high school and $US175 for those with (at least one) children in senior high school. The maximum amount of benefits is intended to reduce any incentive the program might provide to have additional children. Benefits are provided directly to the female beneficiary by wire transfer in offices and modules which are installed nearby the communities. In some urban areas, benefits are transferred directly to beneficiary bank accounts. The average monthly transfers during the twelve-month period of 2003 (the last year of the rural evaluation survey) was 309 pesos monthly per beneficiary family or about $US27.50. Payments are higher during non-summer months when education grants are received in addition to the fixed nutrition transfer. Targeting and continued program eligibility10

9 Like the E1 form in the case of education, the schedule of clinic visits is entered on an S1 form, which is brought to the clinic by the beneficiary, ensuring that a record of attendance by household members is kept at the clinic. Each clinic receives an S2 form every two months that contains the names of all individuals in beneficiary families. The S2 form registers compliance or non-compliance by the household and is filled out by a nurse or doctor at the health unit every two months, certifying whether family members visited the health units as required. This form is then sent back to Progresa and processed before receipt of the bi-monthly health and nutrition transfer. 10 A number of add-ons to the program are occurring. Jóvenes con Oportunidades was added in 2003, which extends program benefits past the end of upper high school. The program consists in opening an account for each youth in the last year of lower secondary school and depositing a certain amount of points (equal to pesos) for each year completed onward from ninth grade until

11

The program is means tested with an elaborate targeting mechanism, which varies somewhat between rural and urban areas. In both rural and urban areas, the first stage of targeting is geographic, using aggregate local indicators to select poor rural communities and urban blocks. To identify household level beneficiaries, in rural areas, Progresa carries out a survey of socio-economic conditions for all households denominated the ENCASEH, in the selected communities. With this data, discriminant analysis is used to identify eligible households from non-eligible households. In essence, the program makes an initial classification of poverty depending on a household’s per-capita income. Using this initial classification, a discriminant analysis is carried out relating this initial classification to a number of other household characteristics including dwelling characteristics in the household, dependency ratios, ownership of durable goods, animals and land, and the presence of disabled individuals. According to the predicted scores, a final classification of households as poor (eligible) or non-poor is made. 11 Individuals sign their acceptance as program beneficiaries and receive registration forms for schools and the family clinic. Nearly all selected families enrolled in the program in rural areas, so that self-selection in program participation is not a significant evaluation issue in the first years of the evaluation. However, Alvarez Devoto and Winter, 2006 analyze the dropout program in the first five years of Progresa and show that the relatively poorer beneficiaries are less likely to dropout, which they attribute to program conditionality potentially acting as a screening device. In urban areas, the targeting includes an element of self-selection where individuals are required to apply for the program at modules set up in poor urban areas throughout the country. As in rural areas, basic socio-economic levels are assessed, for those that pass this initial qualifying test at the module, a home visit is programmed to verify socio-economic information and based upon this information, a similar discriminant analysis as in rural areas is used to decide whether the household is eligible for Progresa. The issue of self-selection is important from the beginning in the urban program evaluation, as many eligible households do not apply. (See Coady and Parker, 2005 for more description). Once families become beneficiaries, they remain in the program for three years without further verification of their economic status. However, after three years, a re-interview takes place, at which point, either their beneficiary status is renewed or they are transitioned to a scheme of partial benefits (called EDA Esquema Diferenciado de

finishing high school. When the youth finishes high school, he/she can choose between waiting two years and being able to have the accumulated account balances with interests to use as he/she wishes or having immediate access to the funds if they are used to participate in one of the following initiatives: 1) attend college; 2) purchase health insurance; 3) get a loan to start a business; or 4) apply for public housing. In 2006, a pension for the elderly was added to the program, providing a monthly payment to each adult age 70 or over who is part of an Oportunidades family, equal in 2006 to 250 pesos monthly (about US$22). The impact of these new components have not yet been analyzed in the Progresa evaluations. 11 Once beneficiary households have been identified, an assembly is arranged in the community where the list of selected families is made public. An interesting third stage in the targeting is that community leaders have the opportunity to express reservations about any of the families selected and potentially remove families from the list who are not deemed to be poor. While hard evidence is unavailable about the importance of this last stage, program officials claim that this last step rarely resulted in significant changes to the list of beneficiary families.

12

Apoyos which includes only secondary and high school educational grants but excludes primary school scholarships and cash transfers for food.) 3.b. Discussion Section 2 illustrated the general impact of conditional transfer programs is to increase time spent in school and reduce time spent in work, broadly defined. It is worth elaborating however on other potential impacts. While the conditionality of the program should unambiguously affect enrollment, the potential impact on passing grades is more ambiguous. While children who fail the same grade twice during the cycle of grants are permanently excluded from the Program, Progresa does allows children to fail each grade once (e.g. for each grade they can repeat the year and receive the grant). This taken at face value might imply there are incentives to fail some (or each) grades, so as to increase the years receiving grants. However, this incentive must be balanced against the point that passing a grade leads to a higher grant received during the next grade, as well as that the opportunity cost of not working presumably increases with higher age. Clearer incentives might derive from not permitting grants to be received in two years for the same grade, however, the high failure rates in Mexico during primary school (as high as 30 percent) suggest that this might be too harsh, and possibly in the longer term lead to lower schooling impacts. The maximum benefit of the program also presumably complicates the modeling of the program. The maximum benefit was set as a disincentive for families to have additional children, although note that children do not receive a grant until grade 3 (around age 8) and so the discount rate on these expected benefits would presumably have to be low to motivate families to have additional children. Under the initial program with grants offered only up until the 9th grade, about 10 percent of families would run up against the maximum benefit, e.g. in these families if all children were enrolled in school, at least 1 child would not receive the stipulated grant level. Relative to a program with no maximum benefit, the maximum benefit might lead families to send fewer children to school, since the marginal benefit for the last children sent would be less than the grant amount. 12

This was further complicated by the introduction of grants at the senior high school level (10 to 12th grades) in 2001 where families with children enrolled at this level are subject to a higher maximum benefit than families with children only in primary or lower secondary school. Note that this endogenizes to some extent the maximum monetary benefit families are eligible to receive. With respect to work, while, as indicated by the model, the program is likely to reduce time spent in work of children, the potential impacts on adult work are more ambiguous.

12 When a family runs up against the maximum benefit in a particular bimester, the Program “reduces” the amount of each child’s grant so that the sum is equal to the maximum benefit, (e.g. rather than indicating that one child is not receiving a grant). Some studies have used this program feature to argue that this provides variation in the amount of grants for students in the same age (de Janvry and Sadoulet, 2006) which might be used to estimate how program impacts vary with changes in grant amounts. However, a family might strategize by for instance by sending the “last” child to work and not to school, so that this sort of variation seems a bit suspect.

13

An unconditional income transfer, through the income effect, would presumably as in other contexts reduce the incentives to work. Nevertheless, with children likely to spend less time in work, the conditionality effect could increase the time adults spend in work. The structure of program benefits, at least for the first three years of benefits, unlike traditional welfare benefits, does not penalize work. However, after three years, a re-interview takes place, at which point, either their beneficiary status is renewed or they are transitioned to a scheme of partial benefits which lasts for another three years before benefits are discontinued. Thus, particularly in the longer term, it remains possible that the program may reduce work incentives in an effort to maintain beneficiary status. 4. The Progresa evaluation: Design, data and empirical issues. In this section we present the initial evaluation design in rural areas, based on the randomized design carried out where a subset of communities eligible to receive Progresa was randomly assigned to a treatment (320 communities) or control (186 communities) group. The control group under this design, however began to receive treatment about 18 months after the experiment began. A new comparison group of communities never receiving benefits was brought into the evaluation design in 2003. Throughout this chapter, we will refer to original treatment households as T1998 (e.g. in treatment community eligible for benefits in 1998), original control households as T2000 (treatment community eligible for benefits in 2000), and the new comparison group as C2003 (eligible communities never receiving benefits by 2003). 4.a. Design and data collection. The randomized trial evaluation of Progresa was carried out in the initial stages of its operation, which resulted in a couple of clear advantages. First, the small scale of the program in its initial phases implied the potential ethical issue of maintaining eligible households out of the program was less binding. E.g. given a limited budget, many eligible households (indeed, in the early program stages, millions of households were in this category) could not receive program benefits, so that randomization was arguably an equitable method of assigning benefits in the context of limited resources (although this argument was not made publicly in Mexico at the time). A second advantage is that the early evaluation allowed the quick production of a number of studies on the program, which in term, made possible policy changes in the program while the program was still continuing to grow and evolve. There is unfortunately little written evidence on how precisely the randomization was done.13 The available government documentation suggests that a universe of potential treatment communities and control communities was randomly selected from the overall universe of communities in high poverty eligible for Progresa. From these two “sample” universes, a treatment sample of 320 communities was selected and a sample of 186 control communities were randomly selected. The communities are from seven states which were among the first states to receive Progresa benefits. Larger communities were apparently given less weight in the sampling scheme. 13 The lack of documentation by government officials may reflect their perception of the controversial nature of carrying out an evaluation with an experimental design. In fact when the results of the initial evaluation studies were made public in 2000, a number of Mexico City newspapers ran articles criticizing the “unethical” nature of the evaluation.

14

The best evidence on the quality of the randomization derives from analyses based on the baseline data, for instance whether there are any significant differences in the distribution of characteristics between treatment and control groups, or regressions attempting to predict treatment or control status. Behrman and Todd (1999) compare characteristics in the treatment and control group for a wide variety of indicators, prior to program implementation. In general, they conclude that at the community level, the level at which the randomization was done, treatment and control groups appear to be random. Nevertheless, at the individual level, where most of the analysis was done due to the larger sample size, they find some generally small but significant differences in pre-program characteristics between the treatment and control group for a number of different characteristics. 14

Treatment beneficiary household began to receive benefits in May of 1998, whereas control households began to receive benefits in December of 1999. However, prior to beginning to receive benefits, households were informed they had been deemed eligible by the Program, it is unfortunately not clear either for the T1998 or T2000 group the exact date they were informed. The duration of the “experiment” lasted approximately 18 months, as measured from the time the first families in T1998 began to receive benefits to when the first households in the T2000 group began to receive benefits. During this period, the program grew quite quickly, from less than 400,000 households in 1997, its first year of operation, to more than 2 million households in over 50,000 communities by the year 2000. This rapid growth created a scenario where many of the original control communities began literally to become “surrounded” by communities (presumably similar to themselves) receiving program benefits. The evaluation entered a second stage in 2003, with the introduction to the evaluation sample of a new comparison group, composed of communities who had never received benefits. Households in 152 communities selected on the basis of matching to the original treatment communities were added to the sample for a new follow-up round carried out in 2003. The 152 matched localities were chosen from a pool of 14,000 potential matches that had not yet begun to receive benefits. Matching was based on locality-level information on the average characteristics of households in each locality from the 2000 Mexican Census data, including demographics of household residents, characteristics of the dwelling and community characteristics as well as distance to schools (see Todd, 2004 for more details). Given the communities are drawn from different geographic areas from the treatment group, they may experience different local area effects (labor market conditions, quality of schooling, quality of health clinics, prices) that may affect the evaluation outcomes of interest. In an effort to have “pre-program” data on households in the new comparison group, the fieldwork included a retrospective questionnaire for those in the new comparison group on basic characteristics in 1997. This retrospective questionnaire was unfortunately not applied to the original evaluation group, so that comparisons of pre-program 1997 14 In fact, at the individual level, a much larger number of significant differences exist than would be expected by chance alone (32 percent of 187 characteristics studied). Behrman and Todd argue that this may in part reflect the large sample size at the individual level (e.g. 24,000 households and more than 100,000 individuals) and thus the “tendency to reject even minor differences”.

15

information between the original evaluation group and the new comparison group use actual 1997 information for the original evaluation (T1998 and T2000) and retrospective information for the new C2003 comparison group. Recall bias may thus unfortunately affect the estimated propensity scores as well as the impact estimations. An additional point is that differences in outcomes may suffer from sample selection if households in the comparison group who in 2003 answered the questionnaire retrospectively differ from those living there in 1997. The new comparison communities were chosen based on a matching of community characteristics between the original evaluation communities and the set of communities not yet incorporated into the program. Not surprisingly then, comparisons based on community level characteristics show few significant differences (Attanasio and DiMaro, 2004). At the individual level, however, the story is quite different. Table 4 presents general characteristics based on data on 1997 characteristics between the groups T1998-T2000 and C2003 groups. (Comparisons between the groups based on the after program 2003 data are contaminated by program impacts.) Nearly all of the individual characteristics presented in the table show significant differences between the original evaluation group and the new comparison group. In particular, along a number of dimensions, the new comparison group appears to be less poor than the original treatment group. The table makes clear the inappropriateness of simple comparisons between the original sample and this new 2003 comparison group for obtaining impacts and the need for some non-experimental approach to account for these differences. We turn now to the data collection. The data collection carried out in the rural evaluation has been relatively extensive and now includes seven rounds of the evaluation survey. In each of the survey rounds, all households in each community were interviewed. The first data available for the original evaluation sample (T1998 and T2000) is the Survey of Socio-Economic Characteristics (ENCASEH), carried out in 1997, applied to households in eligible communities for the purpose of selecting households eligible for the Program. This data contains information on household demographics and composition, child schooling, labor force behavior, income, durable goods and assets including agricultural assets. This data has proved useful as a baseline survey, although this was not its original intention. Note that it is a credible base line survey in part because it was carried out before any households might have been informed they would be beneficiaries. The intended baseline evaluation survey (ENCEL) was carried out in March of 1998, just prior to the initial of payment of families. While no family received payments prior to March of 1998, many or most families in the T1998 group were likely informed they would be beneficiaries prior to the ENCEL of March, 1998. The follow up Evaluation Surveys ENCEL surveys were carried out approximately every 6 months between the fall of 1998 and the end of 2000, for a total of 5 after program rounds during the first phase of the evaluation. These household surveys focused mainly on socio-economic information, e.g. including demographic information, a schooling module, a labor module, a health module with self-reported health indicators and clinic/hospital attendance history, expenditures, income, and assets information. For most after program rounds, identical or similar modules were applied

16

through the rounds, facilitating the comparison of impacts over time. For a small sub-sample, mainly children, some anthropometrics, e.g. weight and height were collected.15

A final ENCEL follow up round, which included the addition of the new comparison group to the sample, was carried out in the fall of 2003. This last round contains the most extensive information yet of the survey, with a number of biological and educational tests not previously applied, including cognitive and behavioral tests applied to young children and their mothers, as well as achievement tests applied to adolescents. Additionally, detailed surveys of prices, school characteristics, and medical clinics were carried out. Appendix Table 3 provides a summary of the information available for the different rounds of the evaluation survey. Note that administrative information on who is a beneficiary and the amounts of benefits received are available from the beginning of the Program for those in the evaluation sample. Table 5 provides a timeline on the evaluation design and data collection. 16

The collection of data on all households in all of the selected communities has the advantage of providing information on both eligible households and non-eligible households. This, in principal allows the use of non-eligible households as a potential control group under impact estimators such as regression discontinuity. However, the definition of who is eligible for the program has evolved over time. The initial eligibility criteria was established based on the ENCASEH information from 1997. Under this criteria, approximately 50 percent of all households in T1998 were declared eligible for the program in the original evaluation sample, informed and began to receive benefits by early 1998. 17 Nevertheless, shortly thereafter, the Program perceived that the selection mechanism was excluding elderly households with few children in school age, and an adjustment, called a “densification” was carried out. This densification increased the percentage of eligible households in the selected communities to a total of 78 percent by the fall of 1998. (Because of operational errors, many of these “densified” households did not actually begin to receive benefits until much later.) Additionally, after three years families are re-interviewed at which time non-poor households can solicit incorporation, providing an opportunity for additional households to be incorporated at that point. Table 6 shows the number and distribution of eligible households, as well as the percentage of those receiving benefits over time. Given the changes in eligibility and numbers of households receiving benefits, many of 15Fieldwork for the collection of these anthropometric measures was done at different times and by different teams than the ENCEL survey, which appears to have complicated somewhat the use of the information (see Behrman and Hoddinott, 2005 for some discussion). 16 The first generation of Progresa evaluation studies was coordinated by the International Food Policy Research Institute, lasting between 1998 and 2000. Since 2000, however, the Progresa evaluation has been coordinated by a Mexican institute, the Institute for Public Health (Instituto Nacional de Salud Publica-INSP) in Cuernavaca as well as an evaluation committee composed of national (Mexican) as well as international researchers. All Progresa databases, questionnaire and other documentation are publicly available at http:\evaloportunidades.insp.mx. 17 Once households are declared eligible, there is an “incorporation ceremony” at which households are informed they have been selected as eligible and where they sign a form agreeing to be beneficiaries. The receipt of monetary benefits generally starts several months later, however, although arguably from the moment of incorporation, households become beneficiaries. Note that for some variables, such as school enrollment or attendance, given program rules, one would expect behavior to change once households are aware they are beneficiaries. For others, such as food expenditures, one might not expect behavior changes until actual transfers begin.

17

the evaluation studies concentrate on the comparison of households initially chosen to be eligible to receive benefits in 1997, which is a relatively clean definition of program participation. 4b. Estimators of the evaluation. In this sub-section we present the main estimators used in the first and second generation studies of the impacts of Progresa. Beginning with some standard notation: Let denote the outcome with treatment, and denote the outcome for persons without treatment. The gain to the individual of moving from to , we can denote as . Let if persons receive treatment,

1Y 0Y

0Y 1Y∆ 1=D 0=D if not. Let X denote other

characteristics used as conditioning variables and let )|1Pr()( XDXP == . The typical parameter estimated in the literature is the impact of treatment on the treated, e.g. )1,|()1,|()1,|()1,|( )01)01 =−===−=∆= DXYEDXYEDXYYEDXETT Normally, one has data on for those who participate in the program and data on for those who do not participate but do not have for participants. A randomized design solves the evaluation problem under the assumption that is a good approximation of , that is the control group provides a good estimation of what would have happened to the treatment group in the absence of treatment.

1Y 0Y

0Y)0|( 0 =DYE

)1|( 0 =DYE

To the extent possible, the initial Progresa evaluation papers use double difference methods, in order to control for any pre-program differences in the impact variables of interest (Behrman and Todd, 1999). When relevant impact variables were not available in the baseline, cross-sectional estimators are generally used to estimate program impacts that is, comparing differences between the treatment and control group after program implementation. The cross-sectional estimator assumes:

)0),(|()1),(|()1.( 00 === DXPYEDXPYECS tt

at some post program time period t and for some subset of characteristics X. The difference-in-difference estimator requires longitudinal (or repeated cross-section data) on program participants and nonparticipants. Let t and t′ be two time periods, one before the program start date and one after. Y0t is the outcome observed at time t. The main condition needed to justify the application of the estimator is:

)0),(|()1),(|()1.( '00'00 =−==− DXPYYEDXPYYEDID tttt where t is a post-program time period and t’ a pre-program time period. The standard equation used to estimate double difference impact estimates in almost all of the initial reports is of the following type:

18

itc

J

jjitcjticictitc XRTTRY εβαααα ∑

=

+++++=1

*3210

where reflects the impact variable of interest, RitcY t refers to the round of the ENCEL; Tic refers to whether the individual/household (i) lives in a treatment or control community (c); refers to the vector of j control observed characteristics for individual/household i in period t in community c; ε

jitcX

itc is an error term assumed to be equally distributed across individuals/households; The equation assumes only 2 rounds of the evaluation survey (before and after), although in practice additional rounds are often used. 18

Note that this framework provides double difference estimators of the impact of Progresa. The coefficient 2α is expected to be statistically insignificant from 0 and provides an indication of whether pre-program differences exist between the treatment group and the control group. The coefficient provides an estimate of the differences between the treatment and control group in the relevant round after program implementation relative to

*3α

2α . This framework extends to allowing additional interaction terms between R and T to analyze how the estimated impacts might differ over time. When only data after the program are available, as was the case for a number of indicators such as consumption, the regression equation simplifies with 2α capturing the direct impact of the program in the following equation under the assumption that pre-program differences are not significantly different from zero (although this may be questionable given evidence by Behrman and Todd described above):

it

J

jjitjiit XTY εβαα ∑

=

+++=1

20

As the above discussion implies, the most frequent variable used to measure program impact is the simple treatment/control dummy, with most analyses restricting attention to the sample eligible for the program. This estimator provides an estimate of the “intent to treat” estimator. When using the initial eligibility definition, most households (96 percent) did in the beginning of the program participate, so that, for this eligibility definition there are few differences between an intent to treat estimator and a treatment on the treated estimator.19

18 Note that in specifications which use the amount of benefits households receive or other measures of program benefits which vary within households over time, an individual/household fixed effect can be added to the specification. See Rubalcava, Teruel and Thomas, 2004 and Behrman and Hoddinott, 2005. 19 Available data includes dates and amounts of program benefits for beneficiaries, thus facilitating the comparison of estimators based on eligibility versus actual program status. See Hoddinott and Skoufias, 2004 for a description.

19

With the original control group receiving benefits as of 2000 and the addition of the non-experimental comparison group, estimation of longer run program impacts rely on non-experimental estimators, which generally compare the original treatment group with the new comparison group, which provides an estimate of the impact of receiving benefits for 5.5 years versus never receiving benefits. The studies using this comparison we term “second generation” Progresa studies, because of the non-experimental nature of the selection of the new comparison group. Mainly matching estimators have been used (see Heckman, Ichimura and Todd, 1997). The second generation studies concentrate on estimating the mean impact of treatment on the treated by using only those persons/households who actually participated in the program, although as in the first phase evaluation, one could estimate an intent to treat parameter by using all individuals potentially eligible for the program, as is done by Angelucci, Attanasio and Shaw, 2004 in their study of the effects of urban Progresa on consumption, which we discuss further in Section 6. Analogous to the first stage studies, there are two types of matching estimators that have been used in the evaluations, the cross-sectional matching estimator and the difference in difference matching estimator. 20

This cross-sectional matching estimator assumes:

)0),(|()1),(|()1.( 00 === DXPYEDXPYECS tt

)1)|1Pr(0)2.( <=< XDCS

at some post program time period t and for some subset of characteristics X. Under these conditions, an estimator for TT is

)0),(|())(()/1( 0111 =−=∆ ∑= DXPYEXPYn iiii iD

20 Given that many of the second generation studies use the retrospective questionnaire on 1997 characteristics applied to the C2003 group to construct matching estimators (with 1997 data collected in 1997 for the T1998 sample), reporting bias may affect the estimated impacts. For instance, if the C2003 tended to underreport their economic conditions in 1997 relative to their actual values, the matching based on 1997 characteristics would systematically match the T1998 group to households with upwardly biased propensity scores. Ideally, the module on retrospective information in 1997 would also have been applied to the T1998 and T2000 groups, thus providing a direct way to estimate reporting bias by comparing actual characteristics in 1997 with information in 2003 on characteristics in 1997 for the original sample. Lacking such information, there are (at least) two alternatives exist to analyze possible reporting bias. One is to use for the propensity score matching only 2003 characteristics which are unlikely to have been altered by the program, so as to use questions asked in an identical manner during the same time period. Another alternative would be to explore alternative propensity score matching excluding and including variables which might be more or less theoretically subject to reporting bias.

20

where the sum is over , the number of treated individuals with 1n X values that satisfy CS.2.21 )0),(|( 0 =DXPYE ii represents the matched outcome for each treated individual, which can be estimated nonparametrically by nearest neighbor, kernel or local linear regression. The difference-in-difference matching estimator requires longitudinal (or repeated cross-section data) on program participants and nonparticipants. Let t and t′ be two time periods, one before the program start date and one after. Y0t is the outcome observed at time t. Conditions needed to justify the application of the estimator are:

)0),(|()1),(|()1.( '00'00 =−==− DXPYYEDXPYYEDID tttt

)1)|1Pr(0)2.( <=< XDDID where t is a pre-program time period and t’ a post-program time period. The matching estimator, based on longitudinal data, is

DID

)]0,(|()0),(|()())(([)/1( '00'0111 =−=−−=∆ ∑= DXPYEDXPYEXPYXPYn iitiitiitii itD

Where and are the number of treated observations in the two time periods.tn1 '1tn 22

The propensity score matching estimators are estimated in two stages. In the first stage,

is estimated using a probit or logit model and a set X consisting of pre-program (1997) household and locality level characteristics.

)(XP23 In the second stage, the matched

outcomes are constructed, i.e. )0,(|( 0 =DXPYE t for the cross-sectional estimator and, additionally, for the difference-in-difference estimator. )0),(|( '0 =DXPYE t

With regard to how matching performs relative to alternative estimators, this has been a matter of some recent debate, available research is mainly based on programs in the United States. Within the current context, difference and difference matching e.g. comparing say the T1998 group with the C2003 group can only be carried out for very limited indicators, at least until an additional follow up round of the ENCEL is carried out. The early nature of the second generation studies implies there has been thus far limited explorations of how different matching estimators perform under varying dimensions or how matching compares to other non-experimental estimators. How the results vary with different estimators as well as the appropriateness of the estimators should be an

21 The CS.2 condition that insures that matches can be found for the treated individuals. See Heckman, Ichimura and Todd, 1997, for a discussion of the relevance of common support restrictions for matching estimators. 22 Note that at the baseline time period we observe Y0it’ (no treatment outcomes) for the D=1 and D=0 groups. 23 The distribution of X should be unaffected by the receipt of treatment. Using preprogram characteristics makes likely that this requirement is satisfied, because, in 1997, none of the respondents had any knowledge of the program.

21

important part of analyzing the robustness of program impacts in the second generation of studies, we elaborate further below. 24 25

4.c. Attrition in Progresa Attrition has turned out to be important in the ENCEL surveys. Fieldwork protocols throughout the evaluation of Progresa have been to revisit only the original dwellings. Households who move have not been followed. Individuals who leave households that remain are not followed although some limited demographic and schooling information was captured in the ENCEL2003 for individuals who left. Note, however, that households are never dropped entirely from the interview roster. Thus, a household which was not interviewed in a given round because of temporary absence, say, might reappear in the next ENCEL round. Table 7 illustrates this. Of the original 24,077 households of the evaluation sample, only 14,495 report information in every round of analysis. However, 20,067 households have information in both 2003 and 1997. Similarly for the case of individuals, of the original 125,669 individuals only 58,958 have information in all of the evaluation rounds whereas 98,471 have information in (at least) 1997 and 2003. Most of this attrition is caused by apparent changes of residence or migration (more than 80%) and the rest is related to non-response and deaths (Teruel and Rubalcava 2005). 26 If attrition differs between the T1998 and T2000 groups (e.g. treatment is a good predictor of attrition), this selective attrition may bias the estimated impacts of the program. Also, in the context of a program like Progresa which improves education in highly poor areas with few non-agricultural employment options, presumably larger or at least different impacts are likely to be obtained by those who migrate out of their communities after participating in the program. It is obviously of interest to know the impacts of the program not only on the population remaining in their home communities, but also the population that leaves.

24 Substitution bias is frequently mentioned as a potential limiting factor of evaluation design e.g. non-beneficiaries may look elsewhere for a substitute program (Heckman and Smith, 1998). In the rural areas where the program began, this seems fairly unlikely as there are few alternative programs available (or received) pre-program. It is however the case that Progresa specifically prohibits the receipt of programs considered to be “similar” to Progresa, these included school breakfast, subsidized milk, tortilla, and other education grant programs. If one compares the percentage of households receiving these benefits before and after Progresa, there are some declines in the percentage of treatment households reporting these benefits, compared with the controls. However, the percentage of households receiving benefits (in 1997) from other programs was quite low so that at least in the early stages of the Program, participation in other programs seem unlikely to substantially alter the estimated impacts. See Skoufias, 2005 for a summary of participation in other social programs before and after the implementation of Oportunidades in the original treatment and control groups. The topic is important, however, to reconsider in the future. 25 Smith and Todd (2005) use evidence from the NSW (National Supported Work Demonstration) to analyze the performance of different non-experimental estimators and conclude that difference in difference matching is likely the best, in terms of obtaining impacts closest to those derived from an experimental evaluation, due to “eliminating potential sources of temporally-invariant bias, such as geographic mismatch”. 26 There have also been some reported problems with matching identifiers at the individual level across the different rounds. See Teruel and Rubalcava, 2005 for some discussion.

22

In the first stage evaluation studies, none of the analysis considered the possible biasing effects of attrition/migration on estimated program impacts. Some more recent studies begin to look at the potential bias caused by non-random attrition. Teruel and Rubalcava (2005) analyze attrition during the first phase of the evaluation (1997-2000) and argue that households in the original treatment group (T1998) are more likely to leave the household than those in the original control group in 2000. This estimation in effect compares households with two years of receiving benefits with those who have received benefits for only about 6 months. Their results show that at the household level, poor treatment households are 5.3 percent more likely to have left the original 1997 sample by 2000, relative to control households. Bobonis, 2004 in his study of Progresa’s impacts on marriage dissolution and spending patterns analyzes the probability of leaving the sample, and concludes for his sample of women in reproductive age, that while attrition rates are balanced across treatment groups, the likelihood of attrition is correlated with some observable characteristics. This is suggestive that program impacts, even when concentrated only on those who remain in the community, may be biased by differential attrition rates, although the potential magnitude of this bias has not yet been studied. In the case of the second stage evaluations, Behrman, Parker and Todd, 2005a show that for the original treatment and control group, out-migration rates were very high by the 2003 round for those aged 15 to 21 with approximately 40 percent no longer living in the household in 2003 compared with 1997. For a few variables of interest, though, including years of schooling and occupation, actual attrition is less than 20 percent, because information on outcomes is provided by the parents or other informants. Nevertheless, for other variables, the relevant level of attrition for youth is 40 percent. Note, however, there are few significant differences between the original treatment and control group, estimates which effectively compare those with 5.5 years of benefits versus 4.0 years of benefits. (This does not necessarily mean that Progresa has no impact on attrition/migration for these age groups, rather that differential exposure in the program does not seem to have an impact.) As in Bobonis, 2004 a number of individual and household characteristics can predict attrition, implying that the estimates found only reflect the impacts for those who have remained in the community. 27

With respect to the second phase of evaluation, under potential selective attrition, the matching estimators provide unbiased impact estimates for those who remain in the community only under the assumption that attrition out of the program is on observables, e.g. that it can be taken into account by conditioning on observed child and family characteristics. If unobservables that are related to program impacts are determinants of who remains in the sample, this may potentially be addressed through difference in difference matching that allows for time-invariant unobservable differences in the outcomes between participants and non-participants. This assumption is however, not easily tested or verified. A final point is that even if the attrition from the survey does not reflect differential sample selection between participants and non-participants, the matching estimators only estimate the impact of the program on those

27 Behrman, Parker and Todd, 2005a use a difference-in-difference approach combined with a density reweighting method to take into account attrition occurring between the baseline and follow-up surveys.

23

who remain in the survey which may be the sample less likely to benefit from the Program in the longer run. 28

In very partial recognition of the importance of capturing information on migrants, the ENCEL2003 survey includes a number of basic questions applied to the main informant of the household when an individual is no longer living in the household, including schooling levels, occupation, and marital status. To the extent that measurement error in these variables is not severe, this information allows the migrants to be re-included in the sample for some limited variables. The alternative to the collection of this information on migrants is actually following up and locating movers, note that in the Mexican case a fair proportion of movers migrate to the United States. 29

5. Progresa rural evaluation results In this section we present the main results of the rural Progresa evaluation as of mid 2006. The evaluation remains relatively new and is ongoing, thus many of the papers cited are preliminary and/or have not yet been published, particularly those from what we term the “second” phase of the evaluation. Tables 8 and 9 provide a list of the main studies, we focus on those measuring principal indicators of program impacts. 5.a Short term results: rural areas 1998 to 2000. 30

Targeting and current poverty Before turning to the program impact studies, a first obvious question to answer is the extent to which the program is well-targeted. Skoufias, Davis and de la Vega, 2002 provide some fairly convincing evidence that the program in rural areas has been reasonably targeted. They analyze Progresa’s accuracy in targeting at both the community level, and the household level by comparing Progresa’s selection to an alternative selection of households based on per-capita consumption. They also evaluate Progresa’s targeting performance by comparing its potential impact on poverty alleviation relative to other targeting and transfer schemes with the same total budget. 28 Note that an alternative for assessing potential biases in impact estimates due to attrition is to place bounds on the treatment effects when there is differential attrition between treatment and control group. In general, the idea is to estimate bounds of treatment effects through assuming that missing observations are either 1)highest or 2)lowest. See Lee, 2005 and Bobonis, Miguel and Sharma, 2005 for applications. These methods have thus far not been attempted within the Progresa evaluation. 29 Parker and Gandini, 2005 in a pilot study following up and interviewing migrant individuals which have left the Progresa sample both within Mexico and the United States show that information reported by the mother is fairly accurate for variables such as marital status and school enrollment, but with somewhat greater errors for employment and occupation. 30 A number of the studies we describe in this section were originally commissioned and coordinated by the International Food Policy Research Institute (IFPRI). IFPRI was hired by the Mexican Government in 1998 to carry out the initial evaluation reports on a wide variety of impact indicators, including education, health, nutrition, women’s status, expenditure, and community outcomes, cost-benefit analysis as well as operations. The hiring of IFPRI represented a strategy on the part of the program to provide a credible evidence of the program’s impacts, potentially affecting the probability the program would continue to exist under future governments.

24

The overall undercoverage and leakage rates are relatively low (about16 percent respectively). More convincing, however, are the Foster Greer Thorbecke (FGT) adjusted leakage and undercoverage rates which show that the targeting errors in the Program tend to be made around the cutoff point, e.g. the program is very good at including the extremely poor and very good at excluding the non-poor. Similar conclusions derive from the simulations of poverty after Progresa transfers (assuming unchanged pre-program income). After Progresa’s cash transfers, the headcount ratio, which simply measures the percentage of the population with income levels below the poverty level in a community, is reduced by about 10%. However, higher order α (=1, =2) show much greater reductions (30 percent and 45 percent respectively), almost as high as that achieved under the “perfect targeting” alternative based on household consumption. While the targeting is generally shown to be effective, a potential question arises about the merits of targeting in very small communities, e.g. less than 50 families, where more than 90 percent are typically declared eligible. In these particular communities, it is plausible that the costs of targeting and excluding such a small fraction of households are larger than the benefits of targeting. We now turn to the program’s impacts on current poverty. While an obvious hypothesis is that Progresa will reduce the income poverty of its beneficiaries, there are a number of possible incentive effects which might affect or crowd transfer income. First, if by subsidizing schooling, Progresa reduces child labor, then children’s income is likely to fall. Secondly, given the program has an income effect, adults might choose to work less and consume more leisure, reducing family labor income. Finally, the program might affect other income receipt, for instance, transfers from outside the household, in particular remittances to beneficiary households from the United States. Skoufias, 2005 uses difference in difference estimates to compare poverty and income trends over time between the original treatment (T1998) and control (T2000) groups. Using household income and a basic food basket as a poverty line, the results show that by the fall of 1999, Progresa had reduced the headcount poverty rate in 11.7 percentage points, a decline of 17 percent in poverty relative to poverty in the absence of the program. Reductions of higher order poverty terms show even larger proportional reductions, with the squared poverty gap showing reductions as high as 46 percent in poverty. Note these impact estimates are similar to those based on simulations assuming no changes in pre-program income described above, suggestive that crowding out effects do not appear to be overly large (although this could mask for instance reductions in income from child work and increases from other sources). 31

31 Attanasio and Rio-Rull (2001) and Albarran and Attanasio (2003) provide some evidence on the potential impacts of the program on crowding out, analyzing whether individuals in control villages, relative to treatment villages are more likely to receive transfers, or to receive higher amounts of transfers. Their results, based on cross-sectional probit and tobit models for transfers (excluding the program and remittances) finds some evidence consistent with a reduction in the size of monetary transfers to the household as a result of the program. They do not specifically focus on the impact of the program on remittances from abroad.

25

Hoddinott and Skoufias, 2004 examine the impacts of the program on food consumption, finding significant positive impacts on calorie consumption (increasing average consumption by 6.4 percent), with larger impacts on vegetable and animal products, suggesting that families are not just consuming more, but consuming a more diverse and presumably healthier diet. In an attempt to analyze whether the positive impacts on consumption reflect only the increased income of the families, Hoddinott and Skoufias control for total expenditures, finding that the increased income explains about 50 percent of the total consumption impact. Hoddinott and Skoufias interpret the remaining impacts as possibly reflective of the health talks attended by Progresa mothers whereas Rubalcava, Teruel and Thomas (2004) find that the consumption of a healthier diet can be to some extent explained by the fact that women receive the cash benefit. Note that information on consumption was only carried out after the program, so that the estimation is limited by the inability to control for pre-program differences. However, because of operational issues, few and irregular payments were received by households in the first months of operation, which Hoddinott and Skoufias exploit to estimate difference in difference regressions with household fixed effects in effect taking the first after program round as a pre-program baseline. In summary, the program at least in the early years has, apparently sizably, reduced the poverty level and increased consumption and expenditures of the rural population. Large crowding out effects do not appear, in the initial evidence. An important caveat is that these results are based on impacts from the first 18 months of the program and many of the issues discussed here of potential negative incentive effects warrant a re-examination of data in the medium and longer term. Recall that beneficiaries are re-evaluated after three years of benefits, and this additional means testing may increase the importance of potential negative disincentive effects of the program. Education impacts. One of the first studies of the impacts of Progresa comes from Schultz (2004), who focuses on school enrollment. Using difference in difference estimations which compare enrollment before and after the program for the treatment and control group, he finds that positive enrollment impacts are concentrated at the junior high level, and in particular at the transition between primary (elementary) and junior high school, where many children prior to the program tended to leave school. Limited impacts are observed on enrollment in primary, reflecting already very high enrollment rates (generally over 95 percent) in primary school prior to the program. The preferred point estimates based on double difference regression analysis translate to about a one percentage point increase in enrollment at the primary level. At the junior high level, the estimates are higher implying between a 7 and 9 percentage point increase for girls and between a 5 and 6 percentage point increase for boys overall. In another study, Schultz, 2000 shows that the program has little impact on attendance rates, e.g. days attended per month, likely reflecting the very high (97 percent) number of days children enrolled in school report attending. 32

32 Given objectives of the program, parents may have an incentive to over-report their children’s enrollment and attendance, alternative evidence on attendance and enrollment would be highly useful, say through unannounced visits to schools (Duflo and Hanna, 2005).

26

Assuming these impacts do not change over time and assuming no program impacts on other related variables (e.g. grade failure and repetition), Schultz estimates the long run increase on schooling attainment to be 0.72 years of schooling for girls and 0.64 years for boys, an increase of about 10 percent of completed years of schooling pre-program. Using actual returns to education on wages in urban areas of Mexico, Schultz compare the gain in earnings associated with additional schooling to the cost of the grants, finding an internal rate of return of about 8 percent. Behrman, Sengupta and Todd (2005) use a different approach to estimate the education impacts of Progresa. Using a Markov schooling transition model, they compare transition matrixes between the treatment and control group, in this way looking at, for each age, program impacts on enrollment, repetition, dropout and re-entry. Unlike the case of enrollment, younger children (6 to 10) experience large reductions in grade repetition and better grade progression. At the junior high school level, the program reduces the dropout rate and also encourages re-entry among those who have dropped out. Using the transition matrixes from the first year of program operation (1997-1998), they then simulate the long run impacts of the program, assuming the transition matrixes are stationary over time. They estimate that the average child, by age 14 will have accumulated 0.68 additional years of schooling. These estimates are similar to those of Schultz described above, although one might expect Behrman et al.’s estimates to be larger, given their analysis takes into account the positive impacts of the program on reducing failure and other variables besides enrollment. Health and nutrition Behrman and Hoddinott (2005) provide the first study of the potential impacts of Progresa on child height. Unfortunately, the initial data collected on child outcomes in 1998-1999, by the National Institute of Public Health (INSP), have some important limitations. Only a sub-sample of the original ENCEL sample were applied child nutrition questionnaires and there are difficulties linking children back to the larger ENCEL data. Two rounds of nutritional data were carried out, in August-September 1998 and October-December of 1999. While overall sample sizes of the original sub-sample were large, very few children (663) originally interviewed in 1998 were re-interviewed in 1999. Furthermore, by the fall of 1998, many families had begun to receive program benefits, so that the “baseline” was technically carried out post-program. Behrman and Hoddinott focus on impact results for those receiving nutritional supplements. They demonstrate, however, that only about two thirds of children actually report receiving and taking the nutritional supplements and furthermore, that those children taking the supplements have a greater degree of malnutrition than children not taking the supplements. Their impact results thus compare children taking supplements with children not taking supplements in the treatment communities over time, using child fixed effects to control for unobserved heterogeneity (e.g. factors correlated with the treatment variable and the outcome measure, in this case, height). Their analysis focuses on impact results for children aged 12 to 36 months in the first survey round in 1998, leaving out infants under 12 months who might be less likely to benefit from the nutritional supplements because of breastfeeding. They show that controlling for child fixed effects, rather than community fixed effects, results in the estimated impact coefficient changing from a negative sign to positive and significant,

27

with the resulting magnitude implying an increase in height of 1cm due to a year of benefits. This estimator is an impact of treatment on the treated although the issue of selection in who takes the supplements still remains. Estimates of the intent to treat parameter (e.g. using the entire sample of program eligible children rather than only those taking the supplement) show an overall positive but insignificant impact of the program. Rivera et al. (2004) study program impacts on child height and on the prevalence of anemia. Their evidence is based on the same samples, with an extra round of analysis in 2000, although they make no attempt to link the data back to the ENCEL surveys, instead relying on more limited socio-economic surveys carried out at the time of the 1999 and 2000 to construct control variables. This is likely problematic, however, as these socio-economic surveys were applied post-program so that the reported households’ characteristics had potentially been altered by the program and thus including post-program indicators as independent variables is likely to bias potential treatment impacts. The observed impacts on height compare children receiving 2 years of benefits to those receiving only one year of benefits, imply an increase in height of about 1 cm, for infants 12 months or younger in 1998. A third study by Gertler (2004) reports results based on the original data from 1998 and 1999 for children aged 12 to 36 months in 1998 and reports similar results, finding a program impact of 1 cm. He also reports a program impact of a reduction in 25 percent in the probability of a newborn becoming ill (during 4 weeks previous to survey) and a corresponding reduction of 22 percent for those age 0 to 3 at baseline. Work Using double difference estimators, Skoufias and Parker (2001) study the schooling, work and time use decisions of beneficiary children in the early years of the program. The results strongly support the hypothesis that the program reduces work. Looking first at employment (e.g. excluding domestic work) boys aged 12 to 13 and aged 14 to 15 show significant reductions in working. For boys aged 12 to 13 pre-program, the reductions range from 2.8 to 4.1 percentage points and 5.4 to 6.0 percentage points for boys aged 14 to 15, corresponding to a reduction relative to pre-program levels of about 15 to 20 percent. Girls, age 14 to 15 who have very low pre-program labor force participation rates than girls, show reductions of between 2.6 and 3.9 percentage points, corresponding to an overall reduction of between 15 to 25 percent in the probability of working. Note that for boys, the impact estimates are a majority (between 65 to 82 percent) of the size of the Program increase in school enrollment, implying that school and work operate as substitutes in Progresa communities. Using time use data carried out after the program (which permit only difference estimations to be carried out), Skoufias and Parker also analyze the impact of the program using a broader definition of work (e.g. to include market or paid work, home agricultural work and domestic work). This analysis confirms significant reductions in work of both boys and girls with Progresa. Interestingly, both boys and girls show reductions in participation in domestic work, with impacts ranging from a reduction between 5 and 10 percent of pre-program levels. Parker and Skoufias, 2000 analyze program impacts on adult labor supply using double difference estimators comparing participation and hours worked before and after the

28

program. The results in general show no significant impacts of the program on participation or on hours worked. Furthermore, using after program information on time use, there also is no significant impact of Progresa on time spent in leisure. The evidence in the early years of the program then, is that adult beneficiaries do not use the benefits to work less and increase their leisure. These results may in part reflect the design of Progresa, where benefits are provided to families for three years, irrespective of family income, so that there is no (immediate) disincentive effect on work, as opposed to transfer programs in other countries which often reduce benefits with work income. The work of Gertler, Martinez and Rubio-Codina, 2006 also is consistent with evidence that the program has not reduced incentives to work. They analyze the impact of Progresa on participation in micro-enterprises and agriculture and find significant impacts on the amount of land in use, the probability of having a micro-enterprise and the ownership of animals. In particular, program participation increase the probability of having a micro-enterprise in about 3 percentage points, a significant increase given the overall level in the control group of 5.8 percent. Migration.

How might Progresa affect the incentives to migrate? The higher income Progresa provides might, in a context of credit constraints, make migration more feasible. Nevertheless, the conditionality of the income might reduce migration, e.g. because those receiving the education grant must attend school and thus are presumably less likely to migrate. The woman head of the household furthermore is responsible for picking up benefits, which in rural areas would likely require her to at least be in the area. Thus, the potential incentives seem likely to vary by the household member. The topic of the potential impact of the program on international migration is particularly interesting, given the high fraction of Mexicans who migrate. Two recent studies look at the effect of Progresa on migration in the early years of the program, focusing in particular on international migration. Unlike studies of most other topics of Progresa in the early years under the experimental design which tend to coincide in the estimated impacts, the two available studies find completely different impacts of Progresa on migration.

Angelucci, 2004 argues that the net theoretical effect of the program on migration is ambiguous. Unconditional (to schooling) income can be expected to increase migration by reducing the financial constraints to migrating. Nevertheless, conditioning benefits (to schooling) is likely to reduce migration, at least for individuals eligible to enroll in school, by providing incentives to remain in the home village. Angelucci finds that overall international migration (to the U.S.) is substantially (by about 60 percent) increased by the Program although the impacts only appear in 1998 and disappear by 1999, which is puzzling given that in 1998 much less money had been transferred to the Progresa households than in 1999 and presumably the transfers would affect migration by providing additional resources to finance migration. Domestic migration does not appear to be significantly affected on average by the Program. Angelucci attempts to separately identify the impact of an unconditional income transfer from that of the conditional income transfer. This is done by controlling for the conditionality effect through controlling for the proportion of the income grant thought to be “conditional”

29

and attributing the rest of the impact to the unconditional income transfer. The results are suggestive that unconditional income increases migration whereas conditional income reduces it. Nevertheless, given other program aspects (for instance, a bargaining effect by giving women the transfers) and the high correlation of conditional income with the demographic structure of the household, it appears difficult to isolate the impact of “conditional” income. It is plausible that migration might increase after completing schooling, if youth move out of the rural areas with likely limited non-agricultural employment to areas where they can expect the largest return to their schooling. To get some insight into this issue, Angelucci analyzes the impact on migration of those having some junior high/secondary school schooling at the end of the program and finds no significant impact on the probability of migrating for this group. 33 Stecklov et al., 2005 also analyze the short-term program impact on migration, concluding that the Program reduces international migration, in contrast to Angelucci’s work above, and in particular, reduces international migration by almost 40 percent. Their indicator of migration is based at the household, e.g. whether anyone from the household migrates. Stecklov et al use both cross sectional and double difference (comparing before and after program migration) which one might normally expect to give similar results, given the randomization. Results based on the after program cross sectional results show an insignificant impact of the program on migration. Rather the main results emphasized by the authors are those based on the difference in difference which use pre-program trends in migration based on reported migration in the household during the five year period previous to the program. Given the use of the same data sources, it is difficult to understand the differences for such highly opposing results in the two studies. One potential reason might be use of a household measure as opposed to an individual measure of migration. Stecklov et al. do not address the presumably differing incentives to migrate that the program might induce between household members as well as the different program effects (e.g. conditional versus unconditional income). Nevertheless, Angelucci also analyzes a household level indicator of migration to the United States and continues to find a positive and significant effect in 1998, so that the level of analysis does not in principal explain the differences. Other potential explanations might be different definitions of who migrates. Also Angelucci claims that pre-program migration trends show no significant differences while Stecklov et al. claim to find some pre-program differences. In summary, the theoretical work of Angelucci seems more complete in terms of addressing the different incentives to migrate according to different program components. However, neither set of empirical results is entirely convincing. This is clearly an issue where further work is needed, and is important both for understanding impacts of the program on well-being of beneficiaries as well as overall public policy initiatives as to whether social programs might affect Mexican migration to the United States. 33 Nevertheless, it is not clear that comparisons of the treatment and control group for this purpose are valid. Individuals with some completed junior high school in 1999 in the treatment group have likely already been affected by the program, and thus may have different characteristics than their counterparts in the control group who achieved their schooling in the absence of program impacts.

30

Fertility

Although benefits are capped, a large fraction of transfers are related to the number of children through the educational grants. Should beneficiary households perceive the program to be permanent that would presumably induce an increment in the desired number of children. Schultz, 2004 analyzes the program effects on fertility in the first 18 months of the program, and finds no statistically significant impacts on fertility. Similarly, Steklov et al, 2006 analyze the impact of Progresa on fertility in the initial years of program operation. Their overall analysis shows no effect of the program on fertility, marriage or use of contraceptives in the early years of the program for the group of women aged 15 to 49. Todd and Wolpin (2006) simulate the impact of Progresa on permanent fertility, and conclude fertility effects are likely to be insignificant.

Impacts related to intra-household allocation One of the distinctive design features of Progresa is that the monetary transfers were given directly to the woman, typically the mothers, who pick up the payment at the local post office. This design feature reflects research in the social sciences that indicates that men and women do not share the same preferences. In carefully controlled experimental settings, women have been shown to be more altruistic and more risk averse than men. (See Eckel and Grossman (2004a, 2004b) for reviews.) Non-experimental evidence, based on population surveys, suggests that in some contexts women allocate resources under their control towards goods they or their children consume (such as clothing, see Lundberg, Pollak and Wales, (1997) and also to investments that improve child health and well-being (Thomas, 1990; Duflo 2000). Legitimate concerns, however have been raised regarding the extent to which this evidence against the unitary model of household behavior is contaminated by unobserved heterogeneity that is correlated with the distribution of resources within households. Therefore, a central stumbling block in the empirical literature has been identifying sources of "power" that vary exogenously to better understand household behavior. In the case of Progresa, the randomization at the community level provides an instrument of the share of income under the control of woman. Nevertheless, the program in addition to giving money to the woman increases the total income to the family. Furthermore, this income is conditional on human capital investment and thus likely to alter these investments as well as others complementary or substitutes with human capital investment which may also affect the outcomes to be studied. In this sense, the randomization does not provide an ideal instrument for analyzing the relevance of the unitary model. (An ideal randomization design would randomly assign some households where the woman receives benefits and others where the man receives.) Attanasio and Lechene (2003) use the Progresa data to test the common preferences household model using expenditure shares as outcome indicators. In their analysis, the Progresa treatment dummy instruments for the proportion of female income in the household. As they note, however, their approach requires valid instruments that induce variation not only in the income share, but also in total expenditures and schooling

31

because of their endogeneity. Total expenditures are first instrumented by total income, but given total income is also likely endogenous in the current case, total expenditures are instrumented by the community agricultural wage. Community agricultural wages information in the Progresa data is however somewhat limited, it is unclear from the paper how adequate community wages function as instruments in this context. Controlling for the endogeneity of schooling provides another challenge, Attanasio and Lechene include in their regression controls for schooling enrollment prior to the program in an effort to separately control for the conditionality program impact. Attanasio and Lechene estimate that the coefficient on the share of woman’s income is positively and significantly related to spending on both boys’ and girls’ clothing. There are, however, no impacts on spending on other goods, even on those such as male and female adult clothing where one might expect to observe impacts of the female share of income. If the conditionality impacts are not fully controlled, however, it is plausible that the share of female income instrumented by Progresa would also be picking up conditionality impacts of the program. Rubalcava, Teruel and Thomas (2004), use Progresa administrative records on actual payments that beneficiary household receive to analyze the effect of the Program on allocation patterns over time. The analysis is restricted to beneficiary households living in treatment communities with a married or cohabitating couple. To isolate the Program’s “power effect” from income or other Program’s effects, the authors examine the marginal effect of Progresa income on allocations, controlling for total household resources (including Progresa income). Their findings suggest that Progresa income increases the power of women towards investments in the future. Specifically, more money is spent on children, higher quality nutrient intake and there is investment in small livestock which, in the communities of study, are traditionally cared for by (and under the control of) women. The results are robust to household fixed effects, to variation in the timing of Progresa payments within treatment households and also to controlling for expected future benefits.34 They find that in households headed by single females or single males, Progresa income is treated no differently from any other income, concluding that Progresa benefits increase the power of women to allocate resources; that preferences of women differ from those of men; and that women are more inclined to invest. 35

Bobonis 2004 also finds evidence against the income pooling hypothesis, particularly for the indigenous population. He uses information from the Encaseh 1997 and the first four rounds of the evaluation survey and exploits the variation from the randomization in the program and variation attributable to localized rainfall shocks to instrument for the overall level of family spending. By showing that rainfall shocks are uncorrelated with observed time-variant and time-invariant characteristics of households and that the distribution of total spending is not significantly affected by the combination of

34 Results are shown to be robust to the inclusion of income variance. It is also possible that the expected payments (from the program’s rules), as opposed to actual payments (from administrative records), are the underlying decision making variable. Models that include both expected and actual benefits show no significant change in the effects of actual benefits. 35 The evidence presented in these studies is in line with the qualitative evidence from interviews conducted with Progresa households that indicate that Progresa income was perceived as being under the control of women (Adato, et. al. 2000).

32

program treatment and the shocks, his model identifies changes in the effective share of income in the household earned by women, while total household income is unchanged. He finds a 40 percent increase in the share of children’s clothing among households where women received cash transfers and suffered a rainfall shock; this percentage increases to 60 percent among the indigenous. For this analysis, Bobonis uses a subsample of the evaluation survey since his identification strategy relies on the assumption that conditionality constraints are not likely to be binding for households with primary school children at baseline and that confounding factors with the program conditionality would be minimal in a sample of eligible households with children ages 9 years and younger at baseline and households with mothers between the ages of 16 and 55. Spillover effects A majority, but not all, of families within a community are eligible and receive Progresa benefits. In the evaluation surveys, all households within a community are interviewed so that four types of households can be identified, eligible households in treatment villages, ineligible households in treatment villages, eligible households in control villages and ineligible households in control villages. Thus, the evaluation data provides an opportunity to study potential spillover effects of Progresa on non-eligible households. Bobonis and Finan, 2005 analyze Progresa schooling impacts on non-eligible children. They argue that significant spillover effects of enrollment exist for non-beneficiary children, which are primarily concentrated on ineligible children closer to the poverty cutoff. They estimate an “endogenous peers” parameter by using treatment in the program to instrument the effect of eligible school participation on ineligible children. The IV exclusion restriction is that an increase in school participation among ineligible children in treatment villages is the result of the exogenous increase in school participation among the eligible secondary-school children within the village and not the result of changes in contextual variables affected by the program. The reported estimates are quite high, the peer effects with the IV estimates imply a 0.72 percentage point increase in a child’s probability of enrollment as a result of a 1 percentage point increase in the reference group’s enrollment rate. Angelucci and DeGiorgio analyze the effects of Progresa on consumption, finding Progresa increases food consumption for non-beneficiary households in between 5 and 6 percent after 12 to 18 months of program benefits. They argue that the increased consumption is explained by higher loans/transfers to the non-beneficiary families and a reduction in savings of crops and animals, which they interpret as evidence that the program, seen as a positive income shock to beneficiary families, benefits non-beneficiary families through improving consumption smoothing. 36

5.b. Other studies related to the evaluation. Can non-experimental estimators replicate experimental estimators?

36 However, when exploring possible mechanisms to explain program effects on non-beneficiary households, Bobonis and Finan find no significant effect of Progresa on the consumption of non-beneficiary households, inconsistent with the work of Angelucci and DeGiorgio. It is not clear the reasons for this.

33

Given the relative rarity of experimental evaluations as well as their generally low duration, an interesting issue is the extent to which non-experimental estimators can replicate the impact estimates based on the experimental evaluation. A large and recent literature on this topic exists in the United States, beginning with LaLonde (1986) and more recently as demonstrated by Smith and Todd (2005). In the case of Progresa, two papers exist which use alternative estimators to estimate impacts and compare these with those based on the experimental design. Skoufias and Buddelmeyer (2004) use regression discontinuity (RD) analysis to estimate program impacts on work and schooling and compare these impacts to those estimated using the experimental design of the program. Under RD design, comparing individuals within a very small range around the threshold score is equivalent to conducting a randomized experiment at the threshold score (see Hahn, Todd and Van der Klaauw, 2001 for a formal elaboration). Because of the original eligibility criteria where those above or below a critical value were either selected or excluded from program benefits, Buddelmeyer and Skoufias are able to construct a “sharp” RD design. The results in Skoufias and Buddelmeyer show that the RD estimator (using a variety of bandwidth and kernel functions), performs well in approximating the pre-program differences between treatments and control as well as the impacts in 1999. Nevertheless, in 1998, the RD design finds no significant impacts on child school enrolment whereas substantial impacts are reported based on the experimental design estimates although the RD estimates for 1999 are similar to those obtained from the experimental design. Buddelmeyer and Skoufias argue that these mixed findings can be explained by potential problems with the control group (on which the experimental estimates are based), for instance if the control anticipates receiving benefits in the future and alters behavior in the present. Diaz and Handa (2005) attempt to replicate the estimates from Progresa’s experimental design on expenditure shares, school enrollment and child work using non-experimental estimators, in their case matching. Diaz and Handa use the Survey of Income and Expenditures in Mexico (ENIGH), a nationally representative repeated cross-section survey to construct a comparison group to be matched to beneficiary households in the ENCEL treatment group. Their work provides some supportive evidence to Heckman, Ichimura and Todd (1997) and Smith and Todd (2003) who argue that obtaining credible results for matching is greatly facilitated when survey instruments are similar for both the treatment group as well as the comparison group from which the matches are drawn. For the indicators where similar survey questions and structure are available (school enrollment and child work), Diaz and Handa are able to closely replicate the experimental design estimates using cross-sectional matching estimators, with insignificant differences between the experimental design estimates and those based on their matching analysis. For those based on expenditures, where the survey instruments vary substantially, the estimates are significantly different. Structural estimation of program impacts A number of studies have considered the estimation of program impacts and simulation of potential design changes through structural estimation (Todd and Wolpin, 2006, Attanasio, Meghir, and Santiago, 2004). An advantage of the use of the Progresa data is

34

that the experimental design provides the potential for validating the model’s predictions by seeing how well the model predicts the experimental impact of the program, under the assumption that the behavior model for the control group should be the same as the model for the treatment group. Todd and Wolpin (2006) estimate a behavioral model of parental decisions of fertility and child schooling using the Progresa data, using information on child wages to identify the impact of changes in subsidies. They estimate an increase in average schooling for boys and girls of about 0.55 years, which is similar to those based on extrapolations of the experimental results carried out in Schultz (2004) and Behrman, Sengupta and Todd, (2005). An important point is how good is the model at predictions, and in general the model is adequate at predicting the level variables, e.g. the percentage of children enrolled in school. They remain good at approximating the experimental impact results for girls, but to a much lesser extent for boys. The reasons for this are not clear, but may relate to the selection of the sample which is restricted to landless households, which might increase the probability that children, particularly boys would be in the labor force. On the whole, the study is an interesting example of combining the use of detailed evaluation data from a randomized evaluation with structural estimation, allowing the comparison of experimental impacts with those generated from a behavioral model. An Additionally, Todd and Wolpin simulate a number of potential policy changes in Progresa, including eliminating the conditionality, eliminating primary level grants, and doubling the size of secondary grants. According to their estimates, program impacts would be significantly lower if the conditionality is removed with schooling increasing by only 0.1 years compared with the current program estimate of 0.55 years. Reallocating primary grants to increase secondary school grants would increase impacts over the actual program in 0.15 additional years of schooling, with, by design, no increase in program costs. Attanasio, Meghir and Santiago (2004) develop and structurally estimate impacts of Progresa on schooling, allowing income generated by working children to have a different effect on schooling decisions than income generated by the school subsidy and using post-program treatment data to estimate the model. They also allow for anticipatory effects, e.g. that the control group may have anticipated plans to bring them into the program at a future date. While the model is substantially different, the empirical estimates of Progresa are qualitatively similar to Todd and Wolpin, and no anticipatory effects are found. 5c. Medium term results: rural areas 1998-2003. We now discuss the first studies of the “second generation” which use the new follow up round of 2003 as well as the new comparison group (C2003) to generate medium term impacts of Progresa, e.g. after 5.5 years of program benefits. Note that this second generation studies have just begun, all of the papers thus far produced represent unpublished drafts. (Table 9 provides a list). Infant Development A critical question relating to longer term impacts is the impact of the program on infant development. Gertler and Fernald (2004) analyze the impacts of the program on a number of different dimensions of infant development for children aged 3 to 6 in 2003. Their sample includes children born to mothers who were receiving/taking the

35

Program’s nutritional supplement as well as those who were already infants at the time their household became beneficiaries of the program, allowing some analysis of how impacts might vary if “participation” in the program began during the pre-natal period. The indicators analyzed include: 1) cognitive development, as measured by Woodcock Johnson tests of short term, long term memory and visual integration, language development measured by the Peabody Picture vocabulary test for 3 to 6 year olds and the McArthur Communicative Development Inventories for 2 year olds; 2) physical development, measured by gross motor skills (McCarthy scale), fitness measured by the resting heart rate, growth measured by height for age, and stunting, and 3) socio-emotional development, measured by the Achenbach Child Behavior Checklist. Note that all of these tests were carried out for the first time in 2003, thus all estimations in Gertler and Fernald (2004) are based on cross-sectional matching. 37

The results based on the cross-sectional matching show some important impacts on the physical development of children of both boys and girls as well as some improvement in socio-emotional development for girls. For the 8 different gross motor skills tests (for instance, walking backwards, jumping etc.), improvements of 15 percent for boys and 10 percent for girls on average in the proportion who can carry out each of the skills are observed. While these results would seem to be quite positive, they contrast sharply with those observed in the area of cognitive development, no significant effects were observed for any of the 6 different indicators used, which covered children from age 2 to 5. These results are disappointing, particularly given the overall low rates of cognitive development in the communities where Progresa operates. The authors speculate that a potential explanation might be the lack of stimulation within the household, a context where parents have very low levels of educational attainment, and few toys, books or other stimulation or educational tools. Education and work The early nutrition interventions motivating the impacts observed in Gertler and Fernald described above were also hoped to increase the educational performance of children once they began to enter school. Behrman, Parker and Todd, 2006 analyze program impacts in the medium term on the early education of those aged 0 to 8 prior to the program, or 6 to 14 in 2003. A particular group of interest are children aged 0 to 2 in 1997 who, prior to 2003, were exposed directly only to the infant nutritional supplement and check-up components of the program (though they may have been affected indirectly by other aspects of the program, such as income transfers to other household members) The matching estimates based on comparisons of those receiving 5.5 years of benefits versus never having received benefits show some positive impacts of the program on these children, in particular showing a reduction in the age of entry to primary school for girls. Overall, both boys and girls aged 0 to 8 pre-program show increases in years of schooling by 0.45 years with higher impacts as expected for those age 6 to 8 (12 to 14 in 2003). Grade progression also show significant increases with the 37 For these age groups, longitudinal (difference in difference) matching would clearly not be possible as only those aged 6 in 2003 were born before 1997. In an ideal world, it would have been useful to have information on a cohort of children aged 0 to 6 in 1997 which would have allowed difference in difference matching using repeated cross-sections and provided some perspective on possible pre-program difference between the T1998 and C2003 in these different child indicators.

36

program, with impacts for children age 9 to 14 post-program increasing the probability of progressing on time (e.g. without failing any grade) by 13.5 percentage points for girls and 16.0 percentage points for boys. This group, however is only recently entering school age, it thus is early for final conclusions on the eventual impacts of the early nutritional intervention to be drawn. Prior evaluations showed the largest impacts on education of the program on enrollment in junior high school (Schultz, 2004), implying that the largest impacts thus far of the program may be seen by children at or near the transition to junior high school pre-program. Behrman, Parker Todd, 2005b, focus on those plausibly close to or undergoing this transition, children aged 9 to 15 prior to the program (15 to 21 in 2003), examining a variety of indicators in education and work, including years of schooling, achievement test scores in reading, math, and writing, employment and wages. They carry out two types of estimators, differential exposure estimates, which are based on comparing the original treatment group T1998 (receiving 5.5 years of benefits by 2003) with the original control group T2000 (receiving 4.0 years of benefits) and matching estimates, based on comparing the new comparison group with the original treatment, thus estimating the impact of 5.5 years of benefits versus never having received benefits. Overall, the difference in difference matching estimates show impacts of about a year of schooling for youth in households receiving benefits. In particular, youth aged 9 to 12 prior to the program (and thus close to the important transition from primary and junior high school) the impacts show an increase in schooling between 0.8 and 1.0 years. The differential exposure estimates show an increase of about 0.2 years of schooling for boys and girls aged 9 to 12 prior to the program. These estimates based on the original experimental design reflecting an additional 1.5 years of program benefits for the original treatment versus the original control group are apparently consistent with the matching estimates based on 5.5 years of benefits. The paper also considers the impact on achievement tests, which were for the first time applied in 2003 in the areas of reading, writing and mathematics (Woodcock Johnson) to youth aged 15 to 21 in 2003. The achievement tests were applied in the household to all youth in this age group, thus avoiding the problem of selection on those enrolled in school which typically arises in program evaluation of impacts on tests applied at school. Nevertheless, for this analysis, only difference matching estimates could be constructed, which is problematic. Behrman, Parker and Todd (2005b) find some important pre-program differences in schooling between the treatment and new comparison groups favoring the new comparison group; it is probable that impacts based only on difference matching may underestimate potential impacts on test scores. The matching estimates show some very limited positive impacts on achievement tests, principally concentrated on boys. 38

38 Low observed impacts on achievement scores compared with the important increases in years of schooling attained may be due to the lack of a baseline of test scores. Alternative explanations might include low school quality, if enrollments induced by Oportunidades lower school quality both through congestion and through adding marginal students who may have diverted resources and attention from the students who would have been in school in the absence of the program. Note, however, that even if Progresa has low measured impacts on achievement tests, schooling, rather than test scores, may be the more important variable on

37

With regard to program impacts on labor, note first that the overall impact on employment and wages in the medium run is ambiguous. The schooling grants will presumably lead to delayed entry to the labor market for many of those in the sample whereas those completing their schooling (with higher grades of schooling due to the program) would presumably be more likely to be employed and at higher wage jobs. Behrman et al. 2005b find for those aged 15-16 in 2003, there is a negative and significant impact on the probability of employment, consistent with the point that at this age, many boys are still attending school and thus less likely to be working. For the older youth, e.g. 19-21 in 2003, for whom most have likely finished their schooling, there is a positive and significant impact of working, approximately equal to 6 percentage points, or an increase in 6 percent of the probability of working. Similar patterns results for girls. For girls aged 19-21 in 2003, a significant positive impact of the program is observed, equivalent to 5-6 percentage points or a percentage increase of between 14 and 16 percent. These impacts overall confirm the model and hypotheses presented earlier, with school being a deterrent to work for younger youth and for older youth, greater years of completed schooling increasing work. Particularly for girls who tend to have lower labor force participation in the traditional rural communities studied here, the percentage increases in work are quite important. Consumption Attanasio and DiMaro, 2004 analyze the long run effects of Progresa on household consumption patterns. Their analysis is however constrained by the lack of pre-program information on consumption patterns for the new comparison group which preclude the use of double difference matching estimators. Their first set of estimates, based on cross-sectional matching between the original treatment households and the new control group households, finds no significant differences between the two groups, and in fact finds negative “impact” coefficients for almost all expenditure categories. Given the results from Hoddinott and Skoufias (2004) on consumption derived from the first stage evaluation under an experimental design, these results seem implausible. Similar to the analysis of achievement tests described above in Behrman, Parker and Todd, 2005b, Attanasio and DiMaro’s results may reflect pre-existing differences in consumption patterns prior to the program which could not be separately identified from the impact estimators in cross-sectional matching. Living arrangements Rubalcava and Teruel (2005) study the impact of Progresa on living arrangements and migration decisions in the medium term. They compare original households 1997 (T1998) to the new comparison group added in 2003 (C2003), and using double difference propensity score matching they find that households who benefited from the Program reveal a higher rate of rotation of their members. The evidence suggests that the program promotes young adults, sons and daughters of the household head with their children to exit the household, suggesting a partition effect in which the Program may provide greater independence to individuals that wish to form their own families.

longer run labor market outcomes, a hypothesis which could be tested in future evaluations. These are all clearly important topics for future research.

38

There also seems to be an inflow of new members, (e.g. not previously in the household prior to the Program’s implementation), which supports the hypothesis of providing support to members of the extended family, such as parents and grandparents. The implication of this is that members of the extended family—for which the program was not directly intended—also benefit from the Program. Rubalcava and Teruel (2005) also look at the longer term impact of Progresa on migration decisions. Using propensity score matching difference estimates with household fixed effects they find that Progresa increases the probability of females of moving to a different community due to marriage and that teenagers of beneficiary households are more likely to have left to another state or even to the United States relative to teenagers in the comparison group. The impact of migrating to the United States--given Progresa--is greater for males than females.39 In summary, some clearly positive impacts on longer term measures of completed schooling and infant and child health are emerging as well as some interesting impacts on migration and living arrangements. Nevertheless, few positive impacts in the areas such as cognitive development and cognitive achievement have been found thus far. There is some evidence that, given the apparent differences between the new comparison group and the original treatment group, that double difference matching appears to provide higher estimates in the current context than cross-sectional matching, which might be expected given that the comparison communities appear to be less poor in some ways than the original treatment areas. Unfortunately, for many of the indicators of interest, pre-program levels were not available for the new comparison group. For the cross-sectional estimators, the extent to which impacts may be contaminated by pre-program differences is not yet clear. Much further research will be necessary to analyze how robust the estimators used are, how they might vary with different types of matching and with the use of other estimators. Obviously, there are also a number of important indicators which have not yet been studied, including other health outcomes, such as children’s height and weight as well as adult health indicators. 6. Other conditional schooling-health programs around the world. A progressively growing number of other countries in Latin America and the developing world have implemented conditional cash transfer programs with some similarities to Progresa, and an important sub-set of these have also implemented rigorous evaluations. It is useful to review the evidence of these other experiences, in order to gauge how representative the impacts in Progresa might be and also to the extent possible evaluate how different structure of benefits might affect impacts. The evidence, however, is more preliminary than that of Progresa, all of these evaluations are still undergoing what we have termed “first-stage” evaluations, with most studies still in report form, and none published in academic journals. We also review the experience of the urban chapter of Progresa, a non-experimental evaluation which began in 2002 and is also in the first stage of evaluation studies. 6.a. Urban Progresa: 39 Analyzing migration pre and post-program requires knowing about household membership pre-program, information which is based on retrospective information for the comparison group whereas the treatment group is based on actual reporting in 1997. Recall bias on who was in the household in 1997 might thus affect the estimated impacts.

39

In 2001, Progresa was extended to urban areas of Mexico and re-named Oportunidades. The urban program retains the identical benefits of the rural program, although the targeting mechanism was changed to include an element of self-selection where individuals are required to apply for the program at modules set up in poor urban areas throughout the country. At the module, their basic socio-economic levels are assessed, for those that pass this initial qualifying test, a home visit is programmed to verify socio-economic information and based upon this information, a similar discriminant analysis as in rural areas is used to decide whether the household is eligible for Progresa (See Coady and Parker, 2004 for a description). 40

The urban evaluation design is not experimental, but rather uses the method of matching to choose comparison groups. From urban localities eligible for the program, a sample of 149 poor blocks was selected. 41 All 20,859 households in these 149 treatment blocks were initially interviewed to gather information on the socio-economic characteristics used to calculate the proxy-means score. Using this information, a discriminant score was calculated using the same formula as Progresa for each household and households were classified into three groups: Poor, Quasi-Poor (i.e. those just above the cut-off), and Non-Poor. A stratified random sample, based on these classifications in addition to the self-reported beneficiary status was used to select the treatment urban households. The sampling procedure used for selecting a comparison group (households living in areas planned to be incorporated to the Program until 2004) involved matching treatment blocks with non-participating localities using a logistic regression approach and data from the Census of 2000. In all, 388 control blocks (matched to the 149 treatment blocks) were selected for further sampling of households. A similar procedure to that followed in the treatment localities was used to sample control households, including a census tamizaje in selected blocks and probability-weighted sampling. Both in treatment and control areas, a socio-economic survey was applied the Urban Household Socio-economic Characteristics Questionnaire, henceforth “ENCELURB”), beginning in 2002 (the baseline) with after program follow-ups in 2003 and 2004. The survey includes both socio-economic information as well as some anthropometric and biological measures and cognitive development and achievement tests (more details available at evaloportunidades.insp.mx). Most of the studies that have been carried out thus far use matching. There are several possible comparison groups including eligible households in control areas as well as

40 Martinelli and Parker, 2006 take advantage of data on reported conditions at the module and actual household characteristics found in the verification visit to analyze the extent of mis-reporting. As might be expected, under-reporting is substantial. Surprisingly, over-reporting in some goods also occurs, but only in goods where most applicants have the good (e.g. toilet, concrete floor, running water), and thus where “embarrassment” of reporting, say, not having a toilet might be higher. This is important because while under-reporting can theoretically be corrected by the verification (assuming no hiding), households who over report, may over-report themselves out of the program. 41 All blocks with poor populations greater than 50 households were selected and an additional 50 blocks were selected weighted by the inverse of their poor population. The blocks chosen represent a very heavily weighted sample towards urban blocks with the highest density of poor households, thus the impacts derived in the urban evaluation are only relevant for this population.

40

eligible households in treatment areas who did not apply to the program. (See Parker, Todd and Wolpin, 2005 for some discussion in the context of education impacts on how results vary with the comparison group used.) Take up in the urban program was initially low, with less than half of eligible households taking up the program. This low take up has, however, provoked some debate within the evaluation, as to the appropriateness of using matching in a context with low program take up. Angelucci and Attanasio, 2005 argue that the conditional independence assumption is inappropriate in such a context and thus that estimating the average treatment on the treated effect through matching is not appropriate. They suggest an estimator approximating this parameter, using the intent to treat parameter and adjusting by the program participation share. For the case of impacts on household consumption, more intuitive program impacts are obtained with their method relative to matching, however, note this is does not appear to be the case for education, as discussed below. Nevertheless, most available studies report impacts based only one estimation method, in such cases the reported results should be interpreted with some caution. There are two studies which analyze education impacts, both provide estimations of the program impacts after two years of operation. Beginning with the educational outcomes of the beneficiary population, Behrman et al., 2006 provide estimates of the program impact on both school enrollment and years of completed schooling, based on difference in difference matching between eligible participants in treatment areas and eligibles in comparison areas. After two years, the findings indicate that the program has a significant impact on both variables for boys and girls, with the largest impacts generally for the groups aged 12 to 14 and 15 to 18 pre program (2002). For boys, the estimated increase in years of schooling for the group age 12 to 14 is about 0.25 years of schooling and about 0.28 for those aged 15 to 18. For girls, the estimated impacts are slightly smaller, ranging from 0.15 to 0.17 for those aged 12 to 14 and 0.15 to 0.19 for those aged 15 to 18. For school enrollment, the program shows significant results for boys and girls aged 6-7, 8-11 and boys aged 15 to 18. School enrollment for those aged 6 to 7 (reflecting earlier enrollment as well as additional enrollment) is increased by almost 7 percentage points for girls and about 4.5 percentage points for boys. From age 8 to 11, the impacts show about a 2 percentage points increase for both sexes. For boys age 15 to 18, the program increase enrollment after one year by 8 percentage points although the effect becomes insignificant after two years. (Behrman et. al 2006). Parker, Todd and Wolpin (2005) use a dynamic panel data model allowing for unobserved heterogeneity and sibling based estimation procedures to identify the program impact (intent to treat) for younger siblings on school enrollment and years of completed schooling in urban areas. The analysis takes advantage of data which collect a complete education history for all siblings. The preferred estimates, based on a sibling difference IV estimator show significant enrollment estimates, largely concentrated on youth age 12 to 17. For this age group, the impacts range between 9 and 12 percentage point increases for boys and 12.6 to 14.4 percentage point increases for girls. However, the estimated impacts on grades completed show slightly higher impacts for boys, ranging from 0.1 to 0.15 for boys and 0.08 to 0.1 for girls. The authors also carry out simulations of long term exposure, e.g. if the program were able from age 6 to 17, estimating an increase of almost 0.6 years in overall schooling years. The estimates from this study are slightly lower than those of the Behrman et al., 2006 study described

41

above which may reflect that in the first study, average treatment effects are estimated, whereas intent to treat estimates are provided in Parker, Todd and Wolpin. 42

Gutiérrez et. al 2004 analyze program impacts after a year of program benefits on health status, morbidity and utilization of medical services of the beneficiary population (Gutiérrez et. al 2004). Methods based on difference in difference propensity score matching are presented, however, only one potential comparison group is used (based on households in control areas) so that it is difficult to assess how variable the estimates might be to different estimators. Limited impacts on health are however observed. The number of hospitalization in the past year is reduced in about 0.1 for both youth and adults. The reported estimates are a reduction of 0.27 sick-days in the last 30 days for children 6-15 years-old, and 0.23 for adults aged 16 to 49. There is some positive evidence on the capacity to perform activities of daily living for adults and the elderly. With regard to nutrition and indicators, Neufeld et al. analyze the impacts of the program on hemoglobin, anemia, height and weight and language development of small children. Only a small sample of the urban evaluation surveys (children age 6 to 23 months) were selected to receive the biomarker tests and the sample was not random, but rather chosen to minimize the number of geographic areas visited to reduce field costs, so it is not clear how representative the sample is of other areas. Control children were eligible children in treatment areas not receiving benefits. Estimates using this sub-sample indicate that the program reduced the prevalence of anemia in 46%. There is no effect on hemoglobin levels, weight-for-height and height-for-age. (Neufeld et. al 2004). One factor which may constrain nutritional impacts, as in the rural case, is the extent to which nutritional supplement provided by Progresa to children and women are actually consumed. Nuefeld et al., 2004 report that only one fourth of lactating mothers consumed the supplement, and only about 60% prepared the supplement according to the recommendations given. Only about half the children of 6-23 months-old take the nutritional supplement at least once a week. Among those taking the supplement, 66.4% consume the supplement regularly. The median of the supplement consumption, among those who reported consuming it, was about 20g, significantly less than the 44g recommended by the program. Finally, Angelucci, Attanasio, and Shaw, 2004 analyze the impact of Progresa on the level and composition of consumption in urban areas. Difference-in-difference estimates, measuring the intent-to-treat effect of the program based on baseline (2002) and first-year (2003) data from households in treatment and control areas show that the program increases total consumption by 4%. Furthermore, this increase is mainly captured by food consumption, which goes up by 9%. Additionally, as in rural areas, the areas where consumption increases the most are proteins, fruits and vegetables 42 Note that the estimated education impacts in the short run in urban areas seem similar to those in the early years of the program in rural areas. Perhaps one would have expected significantly larger impacts priori, given that the urban program included high school grants from the beginning and thus would be expected to have larger impacts on enrollment than in rural areas. Further research is needed to study the differential impacts between rural and urban areas and whether the differing impacts might reflect higher opportunity costs in urban areas (recall the grants offered are identical in both rural and urban areas).

42

(Angelucci, Attanasio, and Shaw, 2004). Matching program estimates, which estimate the average effect of the treatment on the treated show estimates of similar magnitude although they would presumably be expected to be larger, given the low participation rates in the urban Progresa program. These last findings, as discussed above, lead the authors to question whether propensity score matching in this context are adequate to control for the factors that are driving participation. Overall, the initial short-run impacts in urban areas appear to be positive and relatively consistent with those estimated in the initial evaluation of Progresa in rural areas. Further follow up is obviously necessary. Additionally, given the non-experimental nature of the evaluation design and relatively low take up, alternative estimators and comparison groups should be used to explore the robustness of the impacts. Perhaps a lesson, as compared with the rural evaluation, is that the lack of the experimental design has resulted in some controversy over the appropriate estimators to be used, points which did not particularly arise in the rural evaluation. The issue of attrition has also not yet been addressed in the urban areas, but like rural areas is likely to be important. 6.b. Other programs in Latin America Nicaragua Nicaragua was among the first countries following Mexico to implement a cash transfer program. The Red de Proteccion Social (RPS) focuses on reducing school dropout in the first four years of primary school, improving the health and nutritional status of children under 5 years old, and improving consumption. The program began with a pilot phase subject to an experimental evaluation, and was then extended to other rural communities. The RPS education component stipulates that households with children age 7-13 receive a cash transfer under the requirement that they enroll their children in school and that children comply with 85% attendance. Part of the education grant is fixed (US$8 per household per month), and part varies with the number of children (US$20 per child per year for school supplies). 4344 The health and nutrition component includes a cash transfer under the condition that mothers attend community workshops, and that they take their children under age 5 to periodical medical controls and immunization programs. As in Progresa, the nutrition transfer is fixed per household, equivalent to about US$17 per household/month. The money is given to the mother. Relative to total annual household expenditure, the nutrition transfer represents around 13% and the average education grant 18% of total household expenditure per year. This is closely comparable to Progresa, although unlike Progresa, the RPS transfer is not inflation adjusted45. The pilot phase was implemented in six municipalities of two of the 17 country departments. Within these municipalities, 42 rural census communities (comarcas) were targeted based on a marginality index computed from a 1995 census. Half the comarcas were randomized into the program, while the other half were incorporated after two 43 There is also a small transfer to the teachers per participant children, with the purpose of compensating teachers for the potential increase in class size that the program would cause. 44 According to the 2000 RPS census, 63% of the households had at least one child age 6-12. 45 The real value of the transfer fell by 8% after two years.

43

years. The program’s enrollment rate in the intervention areas turned out to be relatively high, with almost 90% of the eligible population participating, representing around 6,000 households. A household panel data survey was collected with interviews in treatment and control areas in 2000 (baseline), 2001, and 2002, with about 1,600 households. The randomization appeared to be adequate with no few significant differences in pre-program levels of program impact variables (Maluccio and Flores, 2004). Maluccio and Flores (2004) use difference-in-difference estimates between the treatment and control areas to get measures of the intent-to-treat effect of RPS. After two years of the program, results show that RPS increased per-capita household expenditure by 13%, and that most of that increase was allocated to food. In the context of education, enrollment in grades 1-4 rose by 17.7 percentage points, while attendance increased by 11 percentage points and the retention rate improved by 6.5%. Furthermore, the program had a significant effect on some health indicators. The use of primary care services increased significantly, although child vaccination rates are not differentially higher for the treatment group. Finally, RPS also had positive effects on the nutritional status of children, decreasing the prevalence of stunting prevalence in children under age 5 by 3 percentage points, and the prevalence of underweight by 6 percentage points. Honduras In 2000 Honduras began its conditional cash transfer program Programa de Asignacion Familiar (PRAF) with a similar focus on education, health and nutrition. The educational component was targeted to primary school enrollment, while the health component aimed at poor households with children age 0-3 and/or with pregnant women. PRAF had an innovative evaluation design, with communities randomized into one of four groups: a group receiving demand incentives, one receiving supply incentives, another one with both supply and demand incentives, and one receiving no incentives (control group). Eligible households were given an educational voucher of US$58 per child per year conditional on fulfilling requisites of school enrollment and not having more than 7 absences every 3-month period. They were given a health voucher equivalent to US$46 per year upon compliance with a certain frequency of health care visits. In the supply-side subsidy, health facilities and schools received a subsidy (US$6,020 per health facility/year, US$4,000 per school/year, on average) under the commitment that they guaranteed adequate supply and improved the quality of their services. The program was implemented in the seventy poorest municipalities of the country –ten in the supply group and twenty in each of the rest. For the evaluation of PRAF, a household survey was conducted at baseline and then at year-one and year-two of implementation, covering around 6,000 households. Impact measures after the first two years of the program (IFPRI, 2003) show that the distribution of demand side incentives generally reached the target population, with low leakage rates. On the other hand, the execution of supply subsidies was substantially below the planned target. Transfers to health facilities amounted to only 17% of the stipulated. In the field of education, supply incentives were partially implemented, teacher training schemes were executed at a 74% of planned budget but only 7 percent of school subsidies were spent.

44

Transfers focused on boosting demand were generally successful. Use of children’s health services increased by 15 to 21% relative to the control group (Morris, Flores, Olinto and Medina 2004). A significant effect on increasing vaccination rates by 4-7% was also observed. Regarding education, the program caused an increase in attendance of about one day per month, but there does not seem to be a significant impact on enrollment of children age 6-12. There was no significant impact of the program on consumption patterns, interpreted by the authors as reflecting the low PRAF transfers, representing only 3.6% of total household expenditures (IFPRI, 2003). With the possible exception of health clinic visits, the overall impacts for Honduras are more limited, plausibly relating to the relatively low level of grants offered by the program. Supply-oriented grants show no significant impact on the health and educational status of children but given their very partial implementation described above, it is difficult to draw any conclusions on the efficacy of the supply-side interventions. Colombia Another example of a conditional transfer program with a rigorous, although non-experimental, evaluation is Familias en Accion (FA), implemented in Colombia in 2001. Also inspired by Progresa, FA has an education component targeted to poor households with children 6-17, and a health component for poor households with children in ages 0-5. The program delivers a subsidy of about US$6 per month for every child in primary school, and US$12 per month for each child in junior high school. The transfer is conditional on an 80% attendance requirement. The nutritional subsidy amounts to US$20 per family/month, while requiring regular visits to health care centers for children’s growth and development checks, and attendance of mothers to workshops for basic health education. Targeting of the program was done first at the geographic and then at the household level. The selection was based on municipalities that, having less than 100,000 inhabitants, had adequate school and health facilities, a bank, and up to date household poverty information. Based on means testing in eligible municipalities, about 400,000 households became eligible, of which almost 90% enrolled in the program (Attanasio et al. 2004). In the case of the FA evaluation, the program was not randomly assigned between treatments and controls, but rather matching was done between treatment and control municipalities. Control communities were matched on the basis of population size, urbanization and quality of life indicators, and generally would have qualified for FA but were lacking bank facilities. Attanasio et al. (2005) examine the impacts of the program on education, nutrition and consumption after one year of program benefits, using of difference-in-difference regressions. The authors find no impact on enrollment of children 8 to 11 (reflecting already high enrollment rates pre-program), but positive effects on enrollment for the 12-17 age group. Effects are sizable both for urban and rural areas (about 5 and 10 percentage points respectively). Results also show that FA had a significant impact on household consumption, with increases in expenditure of 19% in rural areas and 9% in urban areas. Furthermore, in rural (urban) communities, 80% (90%) of the increase was allocated to food. Within food expenditures, the highest portion of the increase is concentrated in proteins and

45

cereals. Most of the remaining share of total consumption is allocated to children’s clothing and shoes while expenditures on alcohol and tobacco are not significantly affected (Attanasio et al. 2005). Regarding child health and nutrition, the impact of FA appears to be encouraging as well. The percentage of children with preventive care increased 23% for children less than 24 months old, and 33% for those between 24 and 48 months. The incidence of diarrhea was reduced by 10 percentage points in rural areas. Finally, FA increased the height of children under 24 months old by 0.44 centimeters. (Attanasio et al. 2005). Brazil Brazil has a long history of CCT program dating back to the mid 1990’s. The conditional cash transfer Programs Programa de Eradicaçao de Trabalho Infantil (PETI), Bolsa Alimentação, and Bolsa Escola, were unified into Bolsa Familia in 2003, becoming the largest CCT program in the developing world. The Bolsa Alimentação (BA) program consists of a cash transfer to low income families with pregnant and lactating women and/or children under 7 years of age. Mothers must comply with prenatal care attendance, child growth checkups, and vaccination schedules. Household benefits depend on the quantity of eligible members, and range from US$6.25 to US$18.70, given monthly for a semester, and renewable conditional on remaining eligible. 46 The Bolsa Escola (BE) program was targeted to children in the ages 6 to 15, and set its conditionality on children’s attendance to school. A particular characteristic that differentiates the Brazilian conditional cash transfer programs from other programs such as Progresa, is that—they were administered in a decentralized fashion by municipalities, both for the selection of beneficiaries and receipt of the cash transfer itself. A study by De Janvry et al (2005) of Bolsa Escola finds that there seemed to be considerable heterogeneity in the quality of implementation across local governments. De Janvry, Finan and Sadoulet (2006) assess the impact of BE on dropout and failure rates. They use administrative data, choosing 2 schools randomly with a weight proportion to the number of Bolsa Escola beneficiaries, for each of 261 municipalities in four states in Northesast Brazil. School records including school enrolment and repetition were collected for all children in the school, on average 500 per school. The records included school enrollment and repetition for five years, two years prior and three years post-program. Their analysis is based on a double difference method, using as a control group children that were not selected by the municipality and including individual fixed effects. The main findings of the study are that BE led to a decrease of 7.8 percentage points in the dropout rate of children 6-15 years of age, but increased failure rate by 0.8 percentage

46 Bourguignon, Ferreira and Leite (2003) carry out a simulation of Bolsa Escola’s impacts using a round of the annual PNAD (Pesquisa Nacional por Amostra de Domicilios, a nationally representative household survey. Their simulation results suggest that as many as 6 out of 10 children out of school in absence of the program would enroll in school with Bolsa Escola.

46

points. Nevertheless, methodological problems include the lack of information on children who transfer out of the school and the fact that treatment children appear significantly different pre-program in education indicators than control children so that it is not clear that fixed effects are adequate to resolving the selection problem. Other countries Conditional transfers have become popular not only in Latin America, but also in other developing countries around the globe. We elaborate on two programs, in Bangladesh and Cambodia, with impact evaluations. Both aim at fostering female education, given the sharp gender disparities typical of the region. In the Bangladesh case, the program was introduced in the mid 1990’s and was targeted to rural households with girls in secondary school age. Apart from a 75% attendance requirement, the transfer was conditional on satisfactory academic performance, and on the girl remaining unmarried. The transfer covers 100% of tuition costs and 50% of other direct expenses. In total, the subsidy is equivalent to around 6% of per capita income, and it benefits over 2 million girls. Even though the program included all rural areas of the country, it was launched at different times in different regions. Exploiting this variation in timing, Khandker, Pitt and Fuwa (2003) study the impact of the program using the 1991 and 1998 rounds of a cross-sectional household and school survey. Given that all the villages in the survey had the program in 1998, and none in 1991, the authors can estimate the marginal effect of the program, not the average effect. Results show that, after controlling for village-level unobserved heterogeneity, the program had a significant effect on the schooling of girls, and no impact on that of boys. An additional year of program duration increases the probability of enrollment by 12 percentage points for 11-18 year-old girls, large compared with the pre-program enrollment rate of 44%. In the case of Cambodia, the program—Japan Fund for Poverty Reduction (JFPR)—targeted girls making the transition between primary and secondary school, given the context of extremely low secondary school enrollment of girls. Using data from program applications and surprise school visits and comparing girls receiving the scholarship with those who do not, Filmer and Schady (2006) conclude that JFPR had a large and positive effect on high school enrollment and attendance of girls. Results based on propensity score matching indicate that JFPR beneficiaries are 43% more likely to be enrolled and attending class at program schools, and 33% more likely to be enrolled and attending at any school. In summary, the initial results of other conditional cash transfer programs, present a picture of fairly similar results in terms of increasing school enrollment, health clinic attendance, and consumption levels, particularly of food. The PRAF program in Honduras had the lowest impacts, but also is a program with the smallest size of benefits.

47

7. Analysis: Some lessons learned. In this section we analyze the evaluation experience of conditional cash transfer programs. We focus first on what we have learned from the evaluations of conditional cash transfer programs and what future research would be useful. We also comment on some lessons learned on carrying out evaluations of large scale programs with experimental designs in developing countries. Unlike most cash transfer programs, which generally focus on alleviating the conditions of current poverty, conditional cash transfer programs aim to both alleviate current poverty and future poverty, by increasing human capital investment. The “first generation” of studies of the impact of Progresa focused on indicators likely to be highly correlated with this human capital investment. The diverse studies showed almost uniformly positive results on the short-term indicators measured. Among other impacts, better health, greater school attendance, less child work, and improved household consumption were reported. The evidence, although more preliminary, from other conditional cash transfer programs is supportive of important impacts of conditional cash transfer programs in other contexts on schooling and health outcomes. Conditional cash transfer programs can be considered both anti-poverty programs and human capital investment programs. From a cost-benefit perspective then, the programs can presumably be evaluated in terms of how well they achieve these goals relative to other potential programs. For instance, one might judge conditional cash transfer programs merely on their merits as cash transfer programs, whereas indicators such as the reductions in poverty of its beneficiaries would be the main indicators of interest to be contrasted with potential costs of the program. If one takes the longer term view, that the program is primarily a human capital investment program, then the program can be evaluated as such, where the impacts on schooling/health are estimated as well as the potential value of these benefits and contrasted with the costs of the program to obtain a cost-benefit analysis of the program which can be compared to other education programs. However, given the program’s significant impacts on current poverty, it is not clear that it is correct to judge the program simply as a human capital program, to be compared with other potential programs such as improving school quality which focus only on human capital outcomes. Assuming the estimated short run effects of Progresa are stable over the longer term, Schultz, 2004 estimates an internal rate of return of program grant expenditures on private labor market earnings for Progresa. The exercise is to compare program expenditures on education grants with estimated increases in labor market earnings based on the schooling increment generated by the Program. The returns to schooling are based on urban schooling returns, implying that beneficiaries are assumed to migrate after finishing their schooling, although Schultz assumes a 20 percent less than the estimated urban return (lower school quality in rural areas might imply a lower return to schooling for migrants than for urban dwellers). Under these assumptions, Schultz finds an internal rate of return on program expenditures of 8 percent. These findings are suggestive that even judged only as a human capital program, the program is obtaining a

48

reasonable return on its expenditures, assuming the human capital returns are not far off those used here.47 Note that there are other potential effects of conditional programs which are perhaps relevant to weight in a cost benefit type analysis. For instance, deJanvry, Finan, Sadoulet and Vakis show that for families suffering income shocks such as natural disasters or illness, the potential negative effect on schooling is mitigated if the household is a beneficiary (although the same does not appear to be true for child labor). Of course it is possible that other potential human capital programs might have higher returns. In fact, in other contexts, there are a number of recent randomized interventions showing significant education impacts, including impacts on test scores, obtained at a low cost (in fact substantially less than most conditional cash transfer programs). 4849 Nevertheless, most of these interventions have been carried out in much poorer contexts than the primarily middle income countries of Latin America so it is hard to extrapolate potential impacts. There is nevertheless, a clear need to extend impact evaluations in Latin America so as to compare cost-benefit ratios between conditional cash transfer programs and alternative human capital investment programs. In Latin America, there are relatively few education programs which have been evaluated and can thus be compared to conditional transfer programs. An exception is Coady and Parker (2004) who compare the cost effectiveness of Progresa to a program of constructing additional secondary schools in rural areas, and find, that under all plausible scenarios of discount rates, Progresa grants are a far more cost effective intervention for increasing enrollment than building schools, even in rural areas where most communities do not have a secondary school in their community. Unanswered questions There are still many unanswered questions related to the longer term impacts of conditional cash transfer programs. The short term evaluations are an input into the longer term question of whether the central goal between the linking of the program to investment in human capital will be fulfilled. That is, are children who achieve more education today and a better health and nutrition, less likely to be poor in the future as a direct result of program benefits? The second generation of studies in the case of Progresa begins to examine these important issues, showing significant accumulation of schooling for youth and some important positive effects on the gross motor development of toddlers. Further studies and follow-up data will be necessary in order to provide more conclusive evidence of the extent to which conditional transfer programs can promote cognitive development and learning. One of the most important outstanding questions on conditional cash transfer programs is the extent to which the additional schooling they achieve with the program will impact their life long earnings, as would be expected assuming significant returns to schooling. Substantial direct evidence on this is still lacking largely, in great part 47 It is difficult to estimate a rural return to schooling using the Progresa surveys given most workers are self-employed agricultural workers. This reiterates the difficulty of simulating the potential returns to increases in schooling through Progresa, so that the best way to estimate these returns is likely to be through direct assessment of wages in the future. 48 See Duflo, 2006 and Duflo and Kremer, 2003 for reviews. 49 See Coady, 2000 for a description of Progresa program costs.

49

because of the relative recent development of conditional cash transfer programs and the point that the medium term impacts may take 15 or 20 years to observe if one is interested in evaluating the impacts into early adulthood . The cognitive development and academic achievements of the Progresa beneficiaries appear to be limited and there is as yet no other evidence on this topic from other conditional cash transfer programs. (Gertler and Fernald, 2005). While conditional cash transfer programs were not explicitly designed to improve the cognitive and academic development of young children, some positive effects were presumably expected through better nutrition and school attendance and the limited effects thus far observed are worrying. In other contexts, some randomized evaluations show significant impacts of other education interventions on learning, including remedial education programs, incentives to reduce teacher absenteeism. and other incentives for learning such as linking benefits explicitly to performance (Duflo, 2006.) Most conditional cash transfer valuations have not even included the collection of cognitive development and achievement tests. Such indicators of child development and learning should form part of conditional cash transfer evaluations, evidence of the impact of conditional transfer programs on these indicators is urgently needed. In a sense, the design of conditional cash transfer programs as a demand program is neutral on the topic of available school quality.50 The issue of school quality in the areas where conditional cash transfer programs are received is, however, a topic with an urgent need for analysis. Higher enrollment might induce crowding and lower school quality, unless education authorities react by compensating such crowding with additional school resources to schools with higher enrollment increases. And such effects might constraint the actual impacts of the program, particularly on learning indicators. There is little concrete evidence on the issue, however. Parker, 2003 shows that student teacher ratios in Progresa remained approximately constant between 1997 and 2002, implying that the education ministry reacted by increasing the number of teachers at schools with large enrollment increases. Additionally, a recent study (Behrman, Parker, Todd and Gandini, 2006) suggests that schooling impacts of Progresa are higher when available school quality is higher. If school quality affects schooling returns, then this is potentially suggestive that higher labor market impacts of the program may be on those children fortunate to have access to higher quality schools. The impacts of school quality are thus clearly related with the long term impacts of the program on youth and the next generation. Migration is clearly related to the long term impacts of the program by interacting with program impacts in at least two ways. First, conditional cash transfer programs may affect migration, and in fact, particularly in rural areas would be expected to, given the general lack of employment opportunities other than agricultural work. So, the possible impact of the program on migration is of interest. There are however, few studies of the topic and, at least for Progresa, conflicting findings on the existing studies. This is an area, thus, where the impact of conditional cash transfers programs is really not known. Furthermore, migration is plausibly a variable whose impacts might be less immediate than others such as school enrollment, so that additional follow up studies are clearly needed.

50 Although from a welfare point of view, in a context with extremely low school quality, unconditional cash transfers might increase welfare more than conditional cash transfers.

50

A second important reason for studying migration is the point that the impacts of conditional cash transfer programs thus far studied are based on individuals who remain in their communities of interest, and it is quite possible that different impacts will be seen on those who leave their communities. A pilot study following Progresa migrants (Parker and Gandini, 2006) points to the possibility of locating and interviewing migrants and shows that migrants effectively do have different program impacts than non-migrants. For the longer term, in order to provide a picture of the long-term impacts of the program, interviewing migrants as well as non-migrants will be crucial. Attrition is likely to worsen as the collection of data progresses over time. Careful preparation of protocols and fieldwork plans should be developed at the beginning of the evaluation (Thomas, Frankenburg and Smith, 2001). A final issue for the medium term we mention here is that of program dependency. Research on welfare programs in developed countries have typically argued that cash transfer programs create negative incentive effects of work of transfer programs. Beneficiaries of conditional cash transfer programs have generally been promised benefits for a certain period of time (e.g. 2 or 3 years) without further means testing of the prospect of losing benefits, presumably work disincentives might be lower under such a design. In the context of Progresa, Parker and Skoufias, 2000 find no impact of the program on adult labor force participation rates or on hours worked after a year of program benefits. Nevertheless, since eligibility is presumably not for life in conditional programs, at some point households will likely be reassessed so that program receipt in the longer term might affect work incentives differently than in the short run. Note that while conditional cash transfer programs, have at least initially, be judged as successful, there is little evidence on what particular aspects are most relevant. E.g. is the most important aspect the conditionality (price effect), the income effect or the potential intra household allocation effect deriving from women receiving the transfer. A couple of studies have tried to isolate particular impacts of different components (e.g. Angelucci, 2004, Rubalcava, Teruel and Thomas, 2004, Hoddinott and Skoufias, 2004). Since there is no experimental variation within a particular conditional cash transfer program, this is of course inherently difficult. Wolpin and Todd, 2006 simulate different “programs” and suggest that conditionality is responsible for most of the program impacts e.g. unconditional income transfers would have low impacts on schooling. Bourguignon, Ferreira and Leite, 2002 suggest the same in the context of Brazil. Lessons from experimental evaluations of conditional programs We close with a few comments on the benefits/limitations of experimental evaluations of conditional cash transfer programs. As the first of the experimental evaluations of conditional cash transfer programs, the Progresa evaluation has played an important role in raising the visibility of conditional cash transfer programs and of experimental designs in their evaluation. The Program has clearly had quite a significant impact at the international level, as indicated by a number of Latin American countries adopting similar programs and evaluations after Progresa. It has also, however had an impact on public policy in Mexico. The evaluation likely played an important role in ensuring that

51

the program was not eliminated with the change of government in 2000, as had been common in previous administrations, but rather was expanded. Generally speaking, the experimental evaluations of conditional cash transfers have taken place in the initial phases of the program, thus ensuring the feasibility of having a control group, e.g. when there are a lot of eligible households who were not yet incorporated due to budget and operative limitations. With rapid program growth, it would have been much more difficult to incorporate an experimental design at a later date. For instance, in the case of Progresa, by the time the control group was treated, one in every three households in rural areas was a beneficiary. In a context of such rapid growth, it was difficult to continue to maintain the control group, indeed, the control group communities began to pressure for their inclusion, given they were literally becoming surrounded by communities being incorporated into the program. Early evaluations also ensured that results were available at an early juncture in the program, when perhaps program changes are easier to carry out and when programs may be more susceptible to budget cuts. In Mexico, the experimental design played a critical role in increasing the impact and visibility of the evaluation, and by providing credible, easily understood results at a critical time when the program was under potential scrutiny. 51

Most randomized experiments, particularly in the context of developing countries are however unlikely to survive a long period of time. Certainly, expecting a randomized experiment to survive more than 5 years is not very reasonable, particularly in the context of programs where coverage is expanding quickly, as was the case of Progresa. For programs where the immediate program impacts is the main interest, this is not particularly problematic. However, given the design of conditional cash transfer programs, the most important impacts are those in the medium to longer run. The same holds true of the other experimental evaluations described here, e.g. Honduras, Nicaragua, which also did not last over 2 years. Thus, one obvious lesson for evaluations of conditional cash transfer programs or at least for new evaluations endeavors, is to anticipate that any experiment is likely to last only a short time, and thus contemplate alternative strategies for evaluating long term impacts from the initial stages of program evaluation. For instance, in the Progresa context, the new comparison communities, which were initially not eligible at the beginning of the Program, could have been incorporated into the evaluation sample from the beginning. This would at a minimum, have resulted in identical survey instruments from the beginning and allowed the non-experimental impact estimators to be constructed and directly compared with impacts derived from the original experimental design. One might ask whether the experimental design is useful in the longer term, e.g. after the original control groups become treated. Presumably, the experimental design is generally still valid, but the interpretation of comparison changes from treated versus untreated to differential treatment exposure. We would argue that the experimental 51 A recent source of evaluations with experimental design have been the initiatives by MIT/Poverty Action Lab as well as Innovations for Poverty Action in New Haven. Note, however, these programs have a different focus, e.g. small scale interventions generally carried out by academics/NGO’s rather than large scale government transfer programs, where the operational issues associated with evaluation are likely to be different.

52

designs are still useful by comparing the effects of the program on families/individuals who received benefits for X years versus the control group of for instance, X-2 years. These comparison are also useful as a way to judge the extent to which the non-experimental impacts are reasonable. With relatively short lengths of experimental design, the fundamental nature of the evaluation of longer term impacts of conditional cash transfer programs is likely to shift to non-experimental methods. 52 Given that most conditional programs use proxy means tests (which generally are define eligibility on a basis of a continuous variable based on a strict cut-off), the method of regression discontinuity continues to be another potential option, assuming that the definition of program eligibility does not change over time (as has happened in the case of Progresa). Another potential comparison group for the future includes older siblings who did not participate in the program alternatives (Parker, Todd and Wolpin, 2005).

52 Note that, in the context of Progresa, the “long-run” impacts simulated by the short term evaluations in the area of education (Schultz, 2004; Behrman, Sengupta and Todd, 2005) look quite plausible in comparison with those beginning to emerge in the second generation studies based on non-experimental evaluations.

53

Table 1

Cash benefits of Progresa (pesos per month, 2006)

Boys Girls

Primary School

Grade 3 120 120 Grade 4 140 140 Grade 5 180 180 Grade 6 240 240 Middle School Grade 7 350 370 Grade 8 370 410 Grade 9 390 450 High School Grade 10 585 675 Grade 11 630 715 Grade 12 665 760 Fixed monthly nutrition grant per household 180 pesos Support for adults aged 70 or more 250 pesos Maximum household monthly transfer with no children in HS

1095 pesos

Maximum household monthly transfer with children in HS

1855 pesos

Exchange rate: 11 pesos=$1US Note: Progresa also provides in-kind benefits including school supplies, medical consultations and nutritional supplements.

54

Table 2 Interventions in the basic health services package: Progresa

ω Basic hygiene

ω Family planning

ω Prenatal, childbirth and post-natal care

ω Supervision of nutrition and children's growth

ω Vaccinations

ω Prevention and treatment of outbreaks of diarrhea

ω Anti-parasite treatment

ω Prevention and treatment of respiratory infections

ω Prevention and control of tuberculosis

ω Prevention and control of high blood pressure and diabetes mellitus

ω Accident prevention and first-aid for injuries

ω Community training for health care self-help

Source: Oportunidades, 2004 (Program Operating Rules) oportunidades.gob.mx

55

Table 3

Annual Frequency of Health Care Visits Required by Progresa

Age Group Frequency of Check-Ups Children Less than 4 months 4 months to 24 months 2 to 4 years old 5 to 16 years old

3 check-ups: 7 and 28 days, and at 2 months 8 check-ups: 4, 6, 9, 12, 15, 18, 21 and 24 months with 1 additional monthly weight and height check-up 3 check-ups a year: 1 every 4 months 2 check-ups a year: 1 every 6 months

Women Pregnancy

Post-pregnancy

5 check-ups: prenatal period 2 check-ups: 1 immediately following birth and 1 during lactation

Adults and youths 17 to 60 years old Over 60 years old

One check-up per year One check-up per year

Source: Oportunidades, 2004 (Program Operating Rules) oportunidades.gob.mx

56

Table 4

Differences between original treatment group and new comparison group in 1997 T1998, T2000 C2003 Mean Mean Difference P-value HH, spouse characteristics Age of Household head 46.90 43.28 3.63 00 Age of Spouse 40.30 37.86 2.44 00 Gender of Household head 0.89 0.86 0.03 00 Hh head indigenous 0.34 0.27 0.08 00 Spouse indigenous 0.28 0.21 0.07 00 Years schooling HH head 2.70 4.50 -1.80 00 Years schooling spouse 2.68 4.47 -1.79 00 Employed HH head 0.88 0.91 -0.03 00 Employed spouse 0.12 0.18 -0.07 00 Demographic Children 0 to 5 0.85 1.30 -0.44 00 Children 6 to 21 1.05 0.95 0.10 00 Women 20 to 39 0.65 0.63 0.02 05 Women 40 to 59 0.37 0.34 0.03 00 Women 60+ 0.20 0.27 -0.07 00 Men 20 to 39 1.28 1.22 0.06 00 Men 40 to 59 0.37 0.35 0.02 05 Men 60+ 0.21 0.27 -0.06 00 Dwelling charact. # Rooms 1.84 1.66 0.18 00 Electricity in HH 0.73 0.70 0.03 00 Water in HH 0.37 0.45 -0.08 00 Dirt floor 0.59 0.65 -0.06 00 Room material (inferior) 0.72 0.70 0.02 0.011 Wall material (inferior) 0.16 0.15 0.01 0.048 Own animals 0.38 0.26 0.12 00 Own land 0.63 0.45 0.17 00 Score 2.56 2.51 0.05 04 Total HH income 1146.86 2051.93 -905.06 00 Durable goods 0 Blender 0.34 0.27 0.06 00 Refrigerator 0.15 0.14 0.02 01 Gas stove 0.31 0.31 0 0.877 Radio 0.63 0.52 0.12 00 Television 0.46 0.37 0.10 00 Washer 0.05 0.03 0.02 00 Car 0.02 0.03 -0.01 08 Truck 0.08 0.05 0.03 00 Source: ENCASEH for T1998 and T2000 and ENCEL 2003 retrospective questionnaire for C2003

57

Table 5: Timeline for Progresa rural evaluation and data sources

Fall, 97

March, 98

May, 98

Nov, 98

May, 99

Nov, 99

1) ENCASEH survey to determine program eligibility X 2) Experimental design (randomization) X 506 communities, 320 T1998, 186 T2000 4) Treatment (T1998) begins to receive benefits X 5) Follow-up ENCEL X X X X

Jan, 00 May, 00

Nov, 00

Oct, 03

5) Follow-up ENCEL X X X 6) Control group (T2000) begins to receive benefits X 7) New comparison group added (C2003) to sample X 152 new rural communities Definitions: T1998=original treatment communities under experimental design, began receiving benefits in May 1998 T2000=original control communities under experimental design, began receiving benefits in January 2000 C2003=new matched comparison communities never receiving benefits before 2003 Source: Oportunidades, 2004. Nota metodologica de la muestra rural.

58

Table 6

Number of Total Households in Evaluation Sample, Eligible Households and Beneficiary Households in the Progresa Evaluation

Communities Households Eligible households (under 1997 eligibility criteria)

Eligible households under new densified criteria adopted in 1998

Households incorporated in 1998

Households incorporated by 2000

Households incorporated by 2003

Treatment group (T1998)

320 14,856 7837 11,623 8,009 8,478 11,387

Control group (T2000)

186 9,221 4682 7,173 0 6134 7262

New control group (C2003)

152 6,768 6218 0 0 0

Source: Author’s calculations using 1997 ENCASEH and ENCEL surveys from 1998, 2000 and 2003

59

Table 7 Presence of ENCASEH Original Households and Individuals across waves

Round t Households in 1997 and in Wave t

Households in wave t and in all previous waves

1997 24,077 24,077 1998 (Oct) 22,551 22,551 1999 (May) 20,857 20,192 1999 (Nov) 20,908 18,632 2000 (May) 20,496 17,033 2000 (Nov) 20,223 15,777 2003 20,067 14,495 Round t Individuals in 1997

and in Wave t Individuals in wave t and in all previous waves

1997 125,669 125,669 1998 (Oct) 111,886 111,886 1999 (May) 102,123 98,550 1999 (Nov) 105,648 90,355 2000 (May) 94,979 75,978 2000 (Nov) 97,426 67,984 2003 98,471 58,958 Source: ENCASEH 1997, ENCEL 1998o, ENCEL 1999m, ENCEL 2000m, ENCEL 2000n y ENCEL 2003.

60

Table 8 Impact Indicators of the Progresa evaluation: first generation (1997-2000)

Impact indicator

Type of estimator

Data used

Authors

EDUCATION Enrollment in School Experimental design, DD

estimator ENCASEH-Nov97 ENCEL-Mar98 ENCEL-Nov98 ENCEL-Jun99 ENCEL-Nov99

Schultz (2004)

Proportion of School Days Attended Experimental design, DD estimator

ENCASEH-Nov97 ENCEL-Mar98 ENCEL-Nov98 ENCEL-Jun99 ENCEL-Nov99

Schultz (2000)

Failure, repetition, dropout and progression

Experimental design, Markov transition model

ENCASEH-Nov97 ENCEL-Mar98 ENCEL-Nov98

Behrman, Sengupta, Todd (2005)

Child school achievement test scores Experimental design, DD estimator

ENCASEH-Nov97 ENCEL-Nov98 ENCEL-Nov99 Ministry of Public Education (SEP) test scores 1997,98,99

Behrman, Sengupta, Todd (2000)

School enrollment Regression discontinuity compared to experimental

DD estimator

ENCASEH-Nov97 ENCEL-Nov98 ENCEL-Nov99

Buddelmeyer and Skoufias, (2003)

School enrollment Matching CS compared with experimental DD

estimator

ENCASEH-Nov97 ENCEL-Nov98 ENCEL-Nov99, ENIGH1998

Diaz and Handa, (2005)

School enrollment, years of schooling, completed fertility.

Structural model ENCASEH-Nov97 ENCEL-Mar98 ENCEL-Nov98 ENCEL-Jun99 ENCEL-Nov99

Todd and Wolpin, (2003)

HEALTH Daily Consultations per Clinic

DD estimator IMMS Solidaridad Administrative records 1996,97,98

Gertler (2000)

Visits by Provider type (public vs. private)

Experimental design, CS estimator

ENCEL-Jun99 ENCEL-Nov99

Gertler (2000)

Nutrition monitoring visits (0-5 yr old children)

Experimental design, DD estimator

ENCASEH-Nov97 ENCEL-Mar98 ENCEL-Nov98 ENCEL-Jun99 ENCEL-Nov99

Gertler (2000)

Child illness (0-5 yr old children) Experimental design, DD estimator

ENCASEH-Nov97 ENCEL-Mar98 ENCEL-Nov98 ENCEL-Jun99 ENCEL-Nov99

Gertler (2000)

Adolescent and Adult Health Status Experimental design,

CS estimator

ENCEL-Jun99 ENCEL-Nov99

Gertler (2000)

NUTRITION Child height (12-36 month old children)

Experimental design, DD estimator

INSP Evaluation data Behrman and Hoddinott (2004), Rivera et al. (2004); Gertler (2004)

CONSUMPTION Total Consumption (Food and Non-Food) and Total Caloric Availability

Experimental design, CS and DD estimator

ENCEL-Nov98 ENCEL-Jun99 ENCEL-Nov99

Hoddinott and Skoufias, (2004)

Total Consumption (Food and Non Food)

Matching CS compared with experimental DD

ENCASEH-Nov97 ENCEL-Nov98

Diaz and Handa, (2005)

61

estimator ENCEL-Nov99, ENIGH1998

INTRAFAMILY ALLOCATION OF TIME Employment (measure excludes domestic activities)

Experimental design, DD estimator

ENCASEH-Nov97 ENCEL-Nov98 ENCEL-Jun99 ENCEL-Nov99

Parker and Skoufias (2000), Skoufias and Parker, (2001)

Employment Regression discontinuity and experimental DD

estimator

ENCASEH-Nov97 ENCEL-Nov98 ENCEL-Nov99

Buddelmeyer and Skoufias, (2003)

Time Spent in a Wide Range of Activities (including Domestic Activities) During Previous Day

Experimental design, CS estimator

ENCEL-Jun99 Parker and Skoufias (2000)

WOMEN’S STATUS AND INTRAHOUSEHOLD RELATIONS Decisions Regarding Children, Household Expenditure Decisions (such as food, durables, and house repairs), Decisions on how to spend Women’s Extra Income

Experimental design, CS estimator

ENCEL-Nov98 ENCEL-Jun99

Adato et al. (2000), Attanasio and Lechene (2002)

TRANSFERS Probability of HH receiving transfers, amount of transfers

Computable General Equilibrium model

ENIGH 1996 Albarran and Attanasio (2003), Attanasio and Rios-Rull (2001)

OTHER Social Welfare Computable General

Equilibrium model ENIGH 1996 Coady and Harris (2004)

National and International Migration Experimental design, CS and DD estimator

Angelucci (2004)

62

Table 9

Impact Indicators of the Progresa evaluation: second generation (1998-2003)

Impact indicator Estimator Data used Authors EDUCATION Years of schooling Experimental design, DD estimator

Matching DD estimator, T1998 vs. C2003

ENCASEH-Nov97 ENCEL-Oct03

Behrman, Parker and Todd (2004, 2005a, 2005b)

Woodcock Johnson achievement tests for adolescents

Experimental design, CS estimator

Matching CS and DD estimator, T1998 vs. C2003

ENCEL-Oct03 Parker, Behrman and Todd (2005b)

Progression Experimental design, DD estimator

Matching DD estimator, T1998 vs. C2003

ENCASEH-Nov97 ENCEL-Oct03

Behrman, Parker and Todd (2005b)

LABOR Employment (measure excludes domestic activities)

Experimental design, DD estimator

Matching DD estimator, T1998 vs. C2003

ENCASEH-Nov97 ENCEL-Oct03

Behrman, Parker and Todd (2005b),

Agricultural work Experimental design, DD estimator

Matching DD estimator, T1998 vs. C2003

ENCASEH-Nov97 ENCEL-Oct03

Behrman, Parker, and Todd (2005b),

CHILD DEVELOPMENT Woodcock Johnson cognitive achievement

Matching CS estimator, T1998 vs. C2003

ENCELOct.-03 Gertler and Fernald (2004)

Gross motor development (McCarthy Scale)

Matching CS estimator, T1998 vs. C2003

ENCELOct.-03 Gertler and Fernald (2004)

Peabody Picture vocabulary test

Matching CS estimator, T1998 vs. C2003

ENCELOct.-03 Gertler and Fernald (2004)

Achenbach Child Behavior Checklist

Matching CS estimator, T1998 vs. C2003

ENCELOct.-03 Gertler and Fernald (2004)

CONSUMPTION Total Value of Consumption (Food and Non-Food)

Experimental design, CS estimator

Matching CS estimator, T1998 vs. C2003

ENCELOct-03 Attanasio and DiMaro (2004)

DEMOGRAPHICS Migration, HH partition, attrition.

Matching CS estimator, T1998 vs. C2003

ENCASEH-Nov97 ENCEL-Oct03

Rubalcava and Teruel, (2005); Teruel and Rubalcava, (2005)

63

Table 10

CCT around the world – Selected Impacts

Program/Source Education Health Other RPS Nicaragua Maluccio &Flores 2004 Experimental Design, DD estimator

18% increase in enrollment grades 1-4, 23% increase in attendance, 6.5% improvement in retention rate

5% decrease in prevalence of stunting children under 5, 6% decrease in prevalence of underweight children under 5

4% increase in food share in household budget

PRAF Honduras IFPRI 2003 Experimental Design, DD estimator

no significant impact on enrollment of children age 6-12, but increased attendance

15-21% increase in children’s health services, 4-7% increase in vaccination

no impact on consumption

FA Colombia Attanasio et al 2005 Non-experimental Design, matching estimator

5-10% increase in enrollment children age 12-17

.44cm increase height infants under 24 mths, 23-33% rise in preventive care use

9-19% increase in expenditure

BA/PETI Brazil Non-experimental, matching estimator

no impact on height-for-age children ages 0-7 31g decrease in weight children under 3 (Morris et al 2004)

4.5-18% decrease in child labor ages 7-14 (Yap et al 2002)

BE Brazil De Janvry et al 2006 Non-experimental, DD estimator

7.8% decrease dropout children ages 6-15 6% decrease grade retention

BDH Ecuador Schady & Araujo 2006 Experimental, DD & IV estimator

10% increase enrollment children ages 6-17

17% decrease child labor ages 6-17

Bangladesh Khandker et al 2003 Non-experimental, fixed effects estimator

Marginal effect: 12% increase probability of enrollment girls ages 11-18

JFPR Cambodia Filmer & Schady 2006 Non-experimental, matching estimator

33-43% increase 8th grade enrollment girls

Oportunidades Urban Mexico Non-experimental, matching estimator

.12-.20 additional grades boys, .08-.15 for girls 6-10% increase matriculation age 6 (Todd et al 2004)

reduction of 6.1 sick-days for children 6-15 years-old 17% increase preventive care use (Gutierrez et al 2004)

4% increase in total consumption, 9% increase in food (Angelucci et al 2004)

64

Appendix Table 1:

Growth of Progresa over time by area. By date and geographic area

Year Rural Urban Total

1997 205,318 14,626 219,944

1998 1,474,972 143,564 1,618,536

1999 1,895,385 271,077 2,166,462

2000 1,915,747 273,973 2,189,720

2001 2,409,432 609,249 3,018,681

2002 2,922,911 1,114,185 4,037,096

2003 3,059,721 1,180,279 4,240,000

2004 3,453,872 1,546,128 5,000,000 Source: Progresa administrative records Note: Urban area incorporation officially began in 2001, the vast majority of urban households prior to this date correspond to semi-urban areas (2,500-50,000 inhabitants) or to areas previously defined as rural but whose classification changed in the 2000 census.

65

Appendix Table 2 Historical monthly amounts of Progresa benefits

Jul - Dec

2004 Jan - Jun

2004 Jul - Dec

2003 Jan - Jun

2003 Jul - Dec

2002 Jan - Jun

2002 Jul - Dec

2001 Fixed grant from nutrition component (conditioned on attending scheduled visits to health centers) 165 160 155 155 150 145 145 Education grant per child (conditioned on child school enrollment and regular attendance) Primary: 3rd grade 110 110 105 105 100 95 95 4th grade 130 125 120 120 115 115 110 5th grade 165 160 155 155 150 145 145 6th grade 220 215 210 205 200 195 190 Junior high : 1st year Male 320 315 305 300 290 285 280 Female 340 330 320 315 310 300 295 2nd year Male 340 330 320 315 310 300 295 Female 375 370 355 350 340 330 325 3rd year Male 360 350 335 335 325 315 310 Female 415 405 390 385 375 365 360 Upper high school 1st year Male 540 530 510 505 490 475 470 Female 620 610 585 580 565 545 540 2nd year Male 580 570 545 545 525 510 505 Female 660 645 625 620 600 585 575 3rd year Male 615 600 580 575 555 540 535 Female 700 685 660 655 635 620 610 Max. monthly benefit (with upper HS grants) 1,010 985 950 945 915 890 880 Max. monthly benefit (w/o upper HS grants) 1710 1685 1635 1605 1570 1525 1500 School supplies support Primary: 220 210 200 195 First semester 145 140 135 130 Second semester 75 70 65 65 Junior high school 275 260 250 240 Upper high school 275 260 250 240 Note: Grants are adjusted every semester according to consumer price index (INCP).

66

Appendix Table 2(continuation) Monthly amounts of Progresa benefits (historical)

Jan - Jun

2001 Jul - Dec

2000 Jan - Jun

2000 Jul - Dec

1999 Jan - Jun

1999 Jul - Dec

1998 Jan - Jun

1998 Fixed grant from nutrition component (conditioned on attending scheduled visits to health centers) 140 135 130 125 115.0 105.0 95.0 Education grant per child (conditioned on child school enrollment and regular attendance) Primary: 3rd grade 95 90 85 80 75.0 70.0 65.0 4th grade 110 105 100 95 90.0 80.0 75.0 5th grade 140 135 130 125 115.0 105.0 95.0 6th grade 185 180 170 165 150.0 135.0 130.0 Junior high: 1st year Male 275 260 250 240 220.0 200.0 185.0 Female 290 275 265 250 235.0 210.0 195.0 2nd year Male 290 275 265 250 235.0 210.0 195.0 Female 320 305 295 280 260.0 235.0 220.0 3rd year Male 305 290 280 265 245.0 225.0 205.0 Female 350 335 320 305 285.0 255.0 240.0 Upper high school 1st year Male Female 2nd year Male Female 3rd year Male Female Max. monthly benefit (with senior HS grants) Max. monthly benefit (w/o senior HS grants) 855 820 790 750 695.0 630.0 585.0 School supplies support Primary: 180 165 135.0 First semester 120 110 90.0 Second semester 60 55 45.0 Junior high school 225 205 170.0 Note: Grants are adjusted every semester according to consumer price index (INCP). 11 pesos = $US 1 in January, 2005. Source: www.oportunidades.gob.mx

67

Appendix Table 3 Content of Baseline and Follow-up Surveys in rural Progresa

Baseline Baseline Follow

-up 1 Follow-up 2

Follow-up 3

Follow-up 4

Follow-up 5

Date of Survey 10-97 3-98 10-98 5-99 11-99 5-00 5-03 Demographic Characteristics Age, sex, relation to head, schooling, language, marital status, etc. X X X X X X X Verification of ID, demographics, ∆ in HH membership & reasons X X X X X Receipt of PROGRESA Benefits Nutrition Cash transfer X X X X X Educational Cash transfer X X X X X Nutrition Supplement X X X X X Education of those 5 years & older Literacy, school enrollment, years completed schooling X X X X X X X Parental opinions of school quality & student achievement. X X X Age started school, grade repetition, drop out X X X X School choice X X X X X Consumption (113 items) Weekly consumption of food booth purchased & home produced X X X X X Weekly & monthly expenditures on non-food items X X X X Health & Health Care Child > 5: Immunizations, ill in last 4 weeks, child was breastfed X X X X X X Preventive/Curative Care by provider type, contraceptive use X X X X X Work/school days lost & days in bed due to illness X X X X Self-reported physical function measures X X X X Maternal & child anthropometrics & blood tests X X X DHS-like fertility & child mortality histories X X Cognitive Development & Achievement X Woodcock Johnson of cognitive development (age 2 to 6) X MacArthur test of language development (age 2 to 3 years) X Peabody test (age 3 to 6) X McCarthy gross motor (age 2 to 5) X Woodcock Johnson of cognitive achievement (age 15 to 21) X Peabody vocabulary test (adults) Work and Earning Labor force participation, days & hours worked X X X X X X X Labor earnings from wage & farm/enterprise X X X X X X X Non-labor earnings, public & private transfers X X X X X X X Women’s Status & Household Decision Making X X X Migration Characteristics of child & adults not at home, 5 yr. History X X X X Assets (level and change) House ownership & characteristics, household appliances X X X X X X X Housing improvements X X X X X Land ownership, land use, & crops produced X X X X X X X Animals X X X X X Community Questionnaire Transportation & public infrastructure and services X X X X X X X Health Facilities and School characteristics X X X X X X Local labor market & wages X X X X X X Private infrastructure & services X X X X X X Biologicals Height, weight, pulse, blood pressure X Blood sample, glucose tests and cholesterol (adults) X Blood sample -anemia and herpes, (age 15 to 21) X Urine sample clamydia and pregnancy, (age 15 to 21) X Woodcock Johnson of cognitive achievement (age 15 to 21) X Other Fertility history X 1997 Characteristics for new comparison group (like baseline 1997 info) X Source: ENCASEH and ENCEL questionnaires 1997-2003

68

Appendix Table 4

Incorporation of new rural comparison group by 2004 (C2003) State Beneficiary Not beneficiary as of 2004 Communities Households Communities Households Guerrero 8 181 2 313 Hidalgo 18 236 9 461 Michoacan 15 334 4 562 Puebla 14 349 1 354 Queretaro 4 216 1 409 San Luis Potosi 20 397 2 570 Veracruz 46 1186 7 1151 Total 125 2899 26 3820 Source: Progresa administrative data

69

References: Adato, M. 2000. “Final Report: The impact of PROGRESA on community social relationships. September. Report submitted to PROGRESA.” International Food Policy Research Institute, Washington, D.C. Adato, M., de la Brière, B., Mindek, D. and Quisumbing, A. 2000. “Final report: The impact of PROGRESA on women’s status and intrahousehold relations.” International Food Policy Research Institute. Albarran P. and Attanasio O. 2003. “Limited commitment and crowding out of private transfers: evidence from a randomised experiment” The Economic Journal. Vol. 113: 486. C77-C85(1). Álvarez, C., Devoto F, and Winters, P. 2006. "Why do the poor leave the safety net in Mexico? A study of the effects of conditionality on dropouts," Working Papers 2006-10, American University, Department of Economics.

Angelucci, M. 2004. “Aid and Migration: An Analysis of the Impact of Progresa on the Timing and Size of Labour Migration.” IZA Discussion Papers 1187.

Angelucci, M. and O. Attanasio. 2005. “Estimating ATT effects with non-experimental data and low compliance.” Mimeo.

Angelucci, M and De Giorgi G. “Indirect effects of an aid program: the case of Progresa and consumption.” Jan. 2006. Mimeo.

Angelucci, M, Attanasio O and Shaw J. 2004. “The Effect of Oportunidades on the Level and Composition of Consumption in Urban Areas.” Technical Document #8 on the Evaluation of Oportunidades. Instituto Nacional de Salud Pública, Mexico. Attanasio, O and Di Maro V. 2004. “Medium Run Effects of Oportunidades on Consumption in Rural Areas.” Mimeo. Institute for Fiscal Studies. Attanasio, O, Meghir C and Szekely M. 2004. “Using randomized experiments and structural models for 'scaling up': evidence from the PROGRESA evaluation”. Accelerating Development (eds. Francois Bourguignon and Boris Pleskovic). Oxford University Press. Attanasio, O, Meghir C and Santiago A. 2004. “Education choices in Mexico: using a structural model and a randomized experiment to evaluate PROGRESA.” IFS Working Papers, EWP04/04. Attanasio OP, and Rios Rull, JV. 2001. "Consumption smoothing in island economies: Can public insurance reduce welfare?," European Economic Review, Elsevier, vol. 44(7), 1225-1258. Attanasio, O and V. Lechene. 2002. “Tests of Income Pooling in Household Decisions”. Review of Economic Dynamics 5:4 720-748.

70

Attanasio, O., E. Battistin, E. Fitzsimons, A. Mesnard, and M. Vera-Hernandez, 2005. “How Effective Are Conditional Cash Transfers? Evidence From Colombia.” The Institute for Fiscal Studies. Attanasio, O, Fitzsimons E, and Gomez A. 2005. “The Impact of a Conditional Education Subsidy on School Enrolment in Colombia.” Institute for Fiscal Studies, London. Attanasio, O, Gomez LC, Heredia P, and Vera-Hernandez, M. 2005. “The Short-Term Impact of a Conditional Cash Subsidy on Child Health and Nutrition in Colombia.” Institute for Fiscal Studies, London. Bautista-Arredondo S. Bertozzi SM, Gertler PJ, Martinez S. 2006. “How Health Care Supply Constraints Limit the Effectiveness of Conditional Cash Transfer Programs: Evidence from Mexico’s OPORTUNIDADES Program.” Mimeo. Becker, G. S. 1999. “Bribe” Third World Parents to Keep their Kids in School”, Business Week, November 22nd. Behrman, J. 2000. “Literature Review on Interactions Between Health, Education and Nutrition and the Potential Benefits of Intervening Simultaneously in All Three.” Mimeo. IFPRI. Behrman, JR and Hoddinott, J. 2005. "Program Evaluation with Unobserved Heterogeneity and Selective Implementation: The Mexican Progresa Impact on Child Nutrition" Oxford Bulletin of Economics & Statistics 67: 547-569 Behrman, JR, Gallardo-García J, Parker SW, Todd PE and Vèlez-Grajales V. 2006. “How conditional cash transfers impact schooling and working behaviors of children and youth in urban Mexico. Mimeo.” University of Pennsylvania. Behrman, JR., Parker SW and Todd PE. 2006. “Medium-Term Effects of the Oportunidades Program Package on Young Children.” (Revised version). Mimeo. Behrman, JR., Parker SW, Todd PE and Gandini L. 2006. “Impacts of Oportunidades and available school supply in rural communities.” Mimeo. Behrman, JR, Parker SW and Todd PE, 2005a. "Long-Term Impacts of the Oportunidades Conditional Cash Transfer Program on Rural Youth in Mexico," Ibero America Institute for Econ. Research (IAI) Discussion Papers 122, Ibero-America Institute for Economic Research. Forthcoming Klasen, S. and Nowak-Lehmann, F. (eds.) Poverty, Inequality, and Policy in Latin America. Cambridge, MA: MIT Press. Behrman, JR, Parker, SW and Todd, PE. 2005b. “The Longer-Term Impacts of Mexico’s Oportunidades School Subsidy Program on Educational Attainment, Cognitive Achievement and Work.” Mimeo. Behrman, JR and Skoufias, E. “Mitigating Myths about Policy Effectiveness: Evaluation of Mexico’s Antipoverty Programme.” The ANNALS of the American Academy of Political and Social Science.2006; 606: 244-275.

71

Behrman, JR, Sengupta P, and Todd, PE. 2005. “Progressing through PROGRESA: An Impact Assessment of a School Subsidy Experiment”. Economic Development and Cultural Change. Vol. 54:1. 237-276. Behrman, JR, Sengupta P, and Todd PE. 2000. “Final report: The impact of PROGRESA on achievement test scores in the first year.” September. Washington, D.C.: International Food Policy Research Institute, Processed. Behrman, JR. and Todd, PE. 1999. “Randomness in the experimental samples of PROGRESA (education, health, and nutrition program).” February. Report submitted to PROGRESA. International Food Policy Research Institute, Washington, D.C. Bobonis, GJ., Miguel, E and Sharma DP. 2005. “Anemia and School Participation. Mimeo.” University of California at Berkeley. Forthcoming, Journal of Human Resources. Bobonis, G. 2004. “Income Transfers, Marital Dissolution, and Intra-Household Resource Allocation: Evidence from Rural Mexico.” Mimeo. Bobonis, G and Finan F. 2005. “Endogenous Peer Effects in School Participation.” University of Toronto, Ontario, Canada and UC-Berkeley, CA. Bourguignon, F., Ferreira FH, and Leite PG. 2003. “Conditional Cash Transfers, Schooling, and Child Labor: Micro-Simulating Brazil's Bolsa Escola Program”. World Bank Econ. Rev., 17: 229 - 254. Cardoso, E and Portela Souza A. 2004. “The Impact of Cash Transfers on Child Labor and School Attendance in Brazil.” Paper No. 04-W07, Department of Economics, Vanderbilt University, Nashville, TN. Coady, DP and Harris R. 2004. “Evaluating transfer programmes within a general equilibrium framework”. Economic Journal Vol 114: 498, 778-799. Coady, DP 2000. “Final report: The application of social cost-benefit analysis to the evaluation of PROGRESA.” November. Report submitted to PROGRESA. International Food Policy Research Institute, Washington, D.C. Coady, D and Parker, SW, 2005. "Program participation under means-testing and self-selection targeting methods," FCND discussion papers 191, International Food Policy Research Institute (IFPRI). Coady, DP and Parker SW. 2004. “A Cost-Effectiveness Analysis of Demand and Supply Side Education Interventions: The Case of Progresa in Mexico”. Review of Development Economics. Vol. 8:3 440-451. Das J, Do K, and Özler B. 2005. “Reassessing Conditional Cash Transfer Programs.” World Bank Research Observer 20(1): 57-80.

72

delaBriere, B. and Rawlings L. 2006. “Examining Conditional Cash Transfer Programs: A Role for Increased Social Inclusion.” Social Safety Net Primer Series. The World Bank. deJanvry, A, Finan F, Sadoulet E and Vakis, R. 2006. “Can Conditional Cash Transfers Serve as Safety Nets to Keep Children out of School and out of the Labor Market”. Journal of Development Economics, 79(2): 349-373. deJanvry, A. and Sadoulet E. 2006. "Making Conditional Cash Transfers More Efficient: Designing for Maximum Effect of the Conditionality." World Bank Economic Review, 20:1-29. de Janvry, A., F. Finan, E. Sadoulet, D. Nelson, K. Lindert, B. de la Biere, and P. Lajouw, 2005. “Brazil’s Bolsa Escola Program: The Role of Local Governance in Decentralized Implementation.” Social Safety Nets Primer Series, The World Bank, Washington D.C. de Janvry, A., F. Finan, and E. Sadoulet, 2006. “Evaluating Brazil’s Bolsa Escola Program: Impact on Schooling and Municipal Roles.” Mimeo, University of California at Berkeley. de Janvry, A , Finan F, Sadoulet E, Nelson D, Lindert K, de la Briere B, and Lanjouw P. “Evaluating the Impact of Decentralized Conditional Cash Transfer Programs: A Study of Brazil's Bolsa Escola Program.” A report prepared for the World Bank. July 2005.

Diaz JJ, and Handa S. 2006. “An Assessment of Propensity Score Matching as a Nonexperimental Impact Estimator: Evidence from Mexico’s PROGRESA Program.” Journal of Human Resources 41(2): 319–345.

Djebbari, H. and Smith J. 2005. “Heterogeneous Program Impacts in PROGRESA.” Mimeo. University of Maryland. Dubois, P., A. deJanvry, and E. Sadoulet. 2004. “Effects on School Enrollment and Performance of a Conditional Transfer Program in Mexico.” Mimeo. Duflo, E. 2006. “Field Experiments in Development Economics.” Mimeo. Duflo, E. and Hanna R. 2005. “Monitoring Works: Getting Teachers to Come to School.“ Mimeo. MIT. Duflo, E. and Kremer M. 2003. “Use of Randomization in the Evaluation of Development Effectiveness.” Paper prepared for the World Bank Operations Effectiveness Department Conference on Evaluation and Development Effectiveness.

Duflo, E. 2000. “Child Health and Household Resources in South Africa: Evidence from the Old Age Pension Program.” American Economic Review, 90 (2), 393-98.

Eckel, C.C. and Grossman, P. 2005a. “Men, Women and Risk Aversion: Experimental Evidence.” Forthcoming in Handbook of Experimental Results, edited by C. Plott and V. Smith. New York, Elsevier.

73

Eckel, CC., and Grossman, P. 2005b. “Differences in Economic Decisions of Men and Women: Experimental Evidence.” Forthcoming in Handbook of Experimental Results, edited by C. Plott and V. Smith. New York, Elsevier.

Edmonds, E., Mammen, K. and Miller, DL. 2005. “Rearranging the Family? Income Support and Elderly Living Arrangements in a Low Income Country.” Journal of Human Resources, forthcoming.

Fernald L. Gertler PJ, Neufeld, L. “How Important is the Amount of Cash in Conditional Cash Transfer Programs for Child Development?” Mimeo.

Filmer D., and Schady N, 2006. “Getting Girls into School: Evidence from a Scholarship in Cambodia.” World Bank Policy Research Working Paper 3910, Washington D.C.

Gahvari F and de Mattos, E. 2005. Conditional Cash Transfers, Public Provision of Private Goods, and Income Redistribution. Forthcoming American Economic Review. Gertler PJ 2000. “Final report: The impact of PROGRESA on health.” November. Report submitted to PROGRESA. International Food Policy Research Institute, Washington, D.C. Gertler PJ. 2004. “Do Conditional Cash Transfers Improve Child Health? Evidence from PROGRESA’s Control Randomized Experiment”. American Economic Review 94(2): 336-341. Gertler PJ and Fernald L. 2005. “The Medium Term Impact of Oportunidades on Child Development in Rural Areas.” Mimeo. Gertler, P. and Boyce S. 2001. “An Experiment in Incentive Based Welfare: The Impact of Progresa on Health in Mexico.” Mimeo. University of California at Berkeley. Gertler P, Martinez,S and Rubio-Codina M. 2006. "Investing cash transfers to raise long term living standards. " Policy Research Working Paper Series 3994, The World Bank. Gitter, Seth. 2005. “Conditional Cash Transfers, Credit Remittances, Shocks, and Education: An Impact Evaluation of Nicaragua’s RPS.” University of Wisconsin- Madison, WI. Gutiérrez, JP, Bautista S, Gertler P, Hernández M, Bertozzi S, 2004. “Impacto de Oportunidades en el Estado de Salud, Morbilidad y Utilización de Servicios de Salud de la Población Beneficiaria: Resultados de Corto Plazo en Zonas Urbanas y de Mediano Plazo en Zonas Rurales.” Documento Técnico #3 en la Evaluación de Oportunidades 2004, Evaluación Externa de Impacto del Programa de Desarrollo Humano Oportunidades, Instituto Nacional de Salud Pública, Mexico. Gutiérrez, JP, Gertler P, Hernández M, Bertozzi S, 2004. “Impacto de Oportunidades en Comportamientos de Riesgo de los Adolescentes y en sus Consecuencias Inmediatas: Resultados de Corto Plazo en Zonas Urbanas y de Mediano Plazo en Zonas Rurales.”

74

Documento Técnico #12 en la Evaluación de Oportunidades 2004, Evaluación Externa de Impacto del Programa de Desarrollo Humano Oportunidades, Instituto Nacional de Salud Pública, Mexico.

Hahn, J., Todd PE, and van der Klaaus W. 2001. “Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design”. Econometrica, Vol 69:1 201-209.

Heckman, JJ. 1992. “Randomization and social policy evaluation.” In Evaluating welfare and training programs, ed. C. Manski and I. Garfinkel. Cambridge, MA: Harvard University Press. Heckman, JJ, Ichimura, H and Todd PE. 1997. “Matching as an Econometric Evaluation Estimator,” Review of Economic Studies 65:261-290. Heckman, JJ, Ichimura H, Smith J and Todd PE. 1997. “Characterizing Selection Bias using Experimental Data”. Econometrica 66: 1017-1089. Heckman, JJ and Smith, J. 1995. “Assessing the Case for Social Experiments.” Journal of Economic Perspectives. 9(2): 85-110. Heinrich C. 2005. “Demand and Supply-Side Determinants of Conditional Cash Transfer Program Effectiveness.” Mimeo. La Follette School of Public Affairs and IRP Affiliate, University of Wisconsin–Madison. Hotchkiss J. 2004. “Do husbands and wives pool their income: Further evidence.” Mimeo. Georgia State University.

Hoddinott, J, Skoufias, E. 2004. “The impact of PROGRESA on consumption.” Economic Development and Cultural Change 53:1 37-63. IFPRI (International Food Policy Research Institute), 2003. Proyecto PRAF/BID Fase II: Impacto Intermedio, Sexto Informe, Washington D.C. Kakwani, N. and Veras Soares F and Son HH. 2005. “Conditional Cash Transfers in African Countries.” Mimeo. International Poverty Center. United Nations Development Programme. Khandker, S, Pitt M and Fuwa N. 2003. “Subsidy to Promote Girls’ Secondary Education: The Female Stipend Program in Bangladesh.” Mimeo. Krueger, AB 2002. Economic Scene: A Model for Evaluating the Use of Development Dollars, South of the Border. New York Times. May 2nd. La Londe R. 1986. “Evaluating the Econometric Evaluations of Training Programs with Experimental Data”. American Economic Review 76: 604-620. Lee, D. 2005. “Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects.” Mimeo. University of California at Berkeley.

75

Lundberg, S. and Pollak, RA. 1993. “Separate Spheres and the Marriage Market.” Journal of Political Economy 101 988–1010.

Lundberg, Shelly J., Robert A. Pollak, and Terence J. Wales. 1997. "Do Husbands and Wives Pool Their Resources? Evidence from the United Kingdom Child Benefit, by Journal of Human Resources 32(3):463-480.

Maluccio, J., and Flores R, 2004. “Impact Evaluation of a Conditional Cash Transfer Program: The Nicaraguan Red de Proteccion Social.” FCND Discussion Paper No. 184, IFPRI, Washington D.C. Martinelli, C. and Parker SW. 2003. “Should Transfers to Poor Families be Conditional on School Attendance: A Household Bargaining Approach”. International Economic Review. Vol. 44(2): 523-544. Martinelli, C. and Parker SW. 2004. “Do School Subsidies Promote Human Capital Investment Among the Poor?” Mimeo. Martinelli, C. and Parker SW. 2006. “Deception and Misreporting in a Social Program.” Mimeo.

Morris S, Flores R, Olinto P, Medina JM. 2004. “Monetary incentives in primary health care and effects on use and coverage of preventive health care interventions in rural Honduras: cluster randomised trial.” Lancet 364: 9450 2030-37.

Neufeld, L., D. Sotres-Álvarez, L. Flores-López, L. Tolentino-Mayo, J. Jiménez-Ruiz, and J. Rivera-Dommarco, 2004. “Consumo del Suplemento Alimenticio Nutrisano y Nutrivida de Niños y Mujeres Beneficiarios de Oportunidades en Zonas Urbanas.” Documento Técnico #6 en la Evaluación de Oportunidades 2004, Evaluación Externa de Impacto del Programa de Desarrollo Humano Oportunidades, Instituto Nacional de Salud Pública, Mexico.

Neufeld, L, Sotres-Álvarez, R. García-Peregrino, A. García-Guerra, L. Tolentino-Mayo, L. Fernald, J. Rivera-Dommarco, 2004. “Evaluación del Estado Nutricional y Adquisición de Lenguaje en Niños de Localidades Urbanas Con y Sin el Programa Oportunidades.” Documento Técnico #7 en la Evaluación de Oportunidades 2004, Evaluación Externa de Impacto del Programa de Desarrollo Humano Oportunidades, Instituto Nacional de Salud Pública, Mexico. Parker, SW. 2003. “Evaluación de impacto de Oportunidades sobre la inscripción escolar: primaria, secundaria y media superior.” México: Secretaría de Desarrollo Social. Parker, SW and Gandini L. 2006. “Poverty Alleviation and Migration in Mexico.” Mimeo. Parker, SW. and Pederzini C. 2001. Gender Differences by Education in Mexico. In The Economics of Gender in Mexico: Work, Family, State, and Market. (Eds. Elizabeth Katz and Maria Correia), The World Bank, Washington D.C. 2001.

76

Parker, SW and Skoufias E. 2000. “The impact of PROGRESA on work, leisure and time allocation.” October. Report submitted to PROGRESA. International Food Policy Research Institute, Washington, D.C. Parker SW and Teruel, GM. 2005. “Randomization and Social Program Evaluation: The Case of Progresa.” The ANNALS of the American Academy of Political and Social Science, 599(1): 199-219. Parker, SW, Todd PE and Wolpin KI. 2005. "Within Family Treatment Effect Estimators: The Impact of Oportunidades on Schooling in Mexico". Mimeo. University of Pennsylvania. Poder Ejecutivo Federal. 1997. Progresa: Programa de Educacion, Salud y Alimentacion. Rawlings, L. 2004. “A New Approach to Social Assistance: Latin America’s Experience with Conditional Cash Transfer Programs.” Mimeo. World Bank. Rawlings, L. and G. Rubio. 2003. “Evaluating the Impact of Conditional Cash Transfer Programs: Lessons from Latin America.” The World Bank. Policy Research Working Paper 3119. Washington D.C. Rivera, J. A.; D. Sotres-Alvarez, J.P. Habicht, T. Shamah, S. Villalpando. 2004. “Impact of the Mexican Program for Education, Health, and Nutrition (Progresa) on Rates of Growth and Anemia in Infants and Young Children.” Journal of the American Medical Association 291:2563-2570. Rubalcava, L., Teruel G, and Thomas D. 2004. "Spending, Saving and Public Transfers Paid to Women". California Center for Population Research. On-Line Working Paper Series. Paper CCPR-024-04. Rubalcava, L. and Teruel G 2005 “Conditional transfers, living arrangements and migration decisions: PROGRESA, six years of evidence.” CIDE working paper. Schady, N., and Araujo M, 2006. “Cash Transfers, Conditions, School Enrollment, and Child Work: Evidence from a Randomized Experiment in Ecuador.” World Bank Policy Research Working Paper 3930, Washington D.C. Schultz, TP. 2004. “School subsidies for the poor: Evaluating a Mexican strategy for reducing poverty”. Journal of Development Economics 74:1 199-250.

Schultz, TP. 2000. “Impact of Progresa on School Attendance Rates in the Sampled Population.” International Food Policy Research Institute, Washington, D.C.

Skoufias, E. and Buddelmeyer, H. 2004. "An evaluation of the performance of regression discontinuity design on PROGRESA," Policy Research Working Paper Series 3386, The World Bank.

77

Skoufias, E. 2005. “PROGRESA and its Impacts on the Human Capital and Welfare of Households in Rural Mexico.” IFPRI Research Report No. 139. International Food Policy Research Institute, Washington, D.C. Skoufias, E, Davis B, and de la Vega S. 2001. “Targeting the poor in Mexico: evaluation of the selection of beneficiary households into PROGRESA”. World Development 29: 10 1969-1984. Skoufias, E. and Parker SW, 2001 “Conditional cash Transfers and their Impact on Child Work and Schooling: Evidence from the PROGRESA program in Mexico”. Economia 2:1 45-96. Smith, J., Todd, P. (2005) “Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators?” Journal of Econometrics, 125(1-2), 305-353.

Stecklov, G, Winters, P, Stampini, M, Davis, B. 2005. “Do Conditional Cash Transfers Influence Migration? A Study Using Experimental Data From the Mexican PROGRESA Program.” Demography 42(4)769-790 Stecklov, G, P Winters, J Todd and F Regalia, 2006. "Demographic Externalities from Poverty Programs in Developing Countries: Experimental Evidence from Latin America," Working Papers 2006-01, American University, Department of Economics. Strauss, JA, and Thomas, D. 1995. “Human resources: Empirical modeling of household and family decisions”. In Handbook of Development Economics, ed. T. N. Srinivasan and J. Behrman. Amsterdam: North Holland. Teruel, G. and Rubalcava L. 2005. “Attrition in PROGRESA.” Mimeo. Thomas, D. 1990. “Intrahousehold resource allocation: An inferential approach”. Journal of Human Resources 25:4 635-664. Thomas, D., Frankenberg E, and Smith JP. 2001. “Lost but Not Forgotten: Attrition in the Indonesian Family Life Survey”. Journal of Human Resources 36:3 556-592. Todd, PE “Technical note on using matching estimators to evaluate the Oportunidades program for six year follow-up evaluation of Oportunidades in rural areas.” Mimeo. Philadelphia: University of Pennsylvania; 2004.

Todd, PE and Wolpin KI. 2006. “Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility: Assessing the Impact of a School Subsidy Program in Mexico.” Forthcoming American Economic Review. Yap, Y., Sedlacek G, and Orazem P. 2002. “Limiting Child Labor Through Behavior-Based Income Transfers: An Experimental Evaluation of the PETI Program in Rural Brazil.” The World Bank. Washington D.C.

78