Stories of Evaluations that Made a Difference tprint a Footprint Stori… · Leaving a Footprint:...

editorialUNSJ

Pablo Rodríguez Bilella Esteban Tapella

When we hear the word evaluation we tend to frown and to associate it with inspec-tions, requirements and stress.

On top of that, if we link evaluation with the field of development interventions (projects, programmes or policies), it’s like entering an unknown and scary land. This is because many evaluations do not contri-bute to decision-making and their use tends to be very limited if non-existent. However, there can be a different reality.

This book compiles seven evaluation stories in development contexts, which in a diversified and innovative manner, produ-ced positive effects in the place in which they took place. Scattered all over Latin

America and the Caribbean, these narrati-ves cover evaluation of interventions that worked with children, rural young people, indigenous women, health programmes and university self-evaluation.

Through surprising and entertaining narratives, these stories identify the factors that allowed evaluation to enhance local development.

This book will be of great use for social programmes’ managers and technicians, as well as academic, evaluators and for the public in general who are interested in processes where social change is enhanced by evaluation.

Pabl

o R

odrí

guez

Bile

llaEs

teba

n T

apel

la

editorialUNSJ

Stor

ies

of E

valu

atio

ns

that

Mad

e a

Diff

eren

ceLe

avin

g a

tpri

nt

Stories of Evaluationsthat Made a Difference

FOREWORDby Bob Wiliams

LEAVING A FOOTPRINT: EVALUATION STORIES THAT MADE A DIFFERENCEPablo Rodríguez Bilella / Esteban Tapella

Graphic design: Maria Clara GraffignaBook trailer and illustrations: Ana Clara BusteloTranslation: Ana del Valle Navas English editing and proofreading: Bob Williams

This book is the result of the research project "Evaluation of Public Policies in Argentina and Latin America: Analysis of the Facilitating Factors of its Demand, Use and Adoption of Results", carried out by the PETAS (Programa de Estudios del Trabajo, el Ambiente y la Sociedad), Instituto de In-vestigaciones Socioeconómicas, Facultad de Ciencias Sociales, Universidad Nacional de San Juan, Argentina.

This publication was supported and financed by:

Rodriguez Bilella, Pablo

Leaving a Footprint: Stories of Evaluation that Made a Difference / Pablo Rodríguez

Bilella; Esteban Tapella. - 1a ed. - San Juan : Editorial UNSJ, 2018.

186 p. ; 21 x 23 cm.

ISBN 978-987-3984-77-8

1. Evaluation. 2. Política. I. Tapella, Esteban II. Título

CDD 320

Copies of this book have been filed as required by Argentine law 11,723. Reproduction prohibited without express written authorization. © Pablo Rodríguez Bilella / Esteban Tapella© Editorial de la Universidad Nacional de San Juan

Mitre 396 (este), primer piso, 5400, Capital, San Juan, Argentina. Tel: +54 (264) 4295096 / 5097 / 5104 Email: [email protected] editorial

UNSJ

Pablo Rodríguez Bilella Esteban Tapella

When we hear the word evaluation we tend to frown and to associate it with inspec-tions, requirements and stress.

On top of that, if we link evaluation with the field of development interventions (projects, programmes or policies), it’s like entering an unknown and scary land. This is because many evaluations do not contri-bute to decision-making and their use tends to be very limited if non-existent. However, there can be a different reality.

This book compiles seven evaluation stories in development contexts, which in a diversified and innovative manner, produ-ced positive effects in the place in which they took place. Scattered all over Latin

America and the Caribbean, these narrati-ves cover evaluation of interventions that worked with children, rural young people, indigenous women, health programmes and university self-evaluation.

Through surprising and entertaining narratives, these stories identify the factors that allowed evaluation to enhance local development.

This book will be of great use for social programmes’ managers and technicians, as well as academic, evaluators and for the public in general who are interested in processes where social change is enhanced by evaluation.

Pabl

o R

odrí

guez

Bile

llaEs

teba

n T

apel

la

editorialUNSJ

Stor

ies

of E

valu

atio

ns

that

Mad

e a

Diff

eren

ceLe

avin

g a

tpri

nt

Stories of Evaluationsthat Made a Difference

FOREWORDby Bob Wiliams

This publication is the result of work from many people who contributed their resources and willingness in different ways. The National University of San Juan (UNSJ) funded two research projects that became the foundation of this publication. The National Scientific and Technical Research Council (Consejo Nacional de Investigaciones Científicas y Técnicas, CONICET) provided academic support to both projects. The German Institute for Development Evaluation (DEval) contributed through editing and printing of this book. As part of our research programme, our colleagues, Vanesa Castro, Cecilia Luna and María Alejandra Lucero, played a key role in conducting and analysing the interviews that led to the development of

a large portion of the evaluation stories presented here. María Clara Graffigna and Ana Clara Bustelo contributed with their professionalism and passion in designing the layout and illustrating the book. We are particularly grateful to all those interviewed for allowing us to recount these stories in which they are portrayed.

We wish to express our gratitude to Mike Hendrix, Martha McGuire, Jim Rugh, Bonnie Koenig, Jane Davidson, Cristina Casado, Karen Russon and Julia Patricia Carobonado for revisiting the English version of the book. We aslo wish to extend special thanks to Bob William for doing the English proofreding and the foreword of this publication.

Acknowledgements

Local Knowledge and Institutional Actors

An Evaluation With a Caribbean Accent

From Indifference to Appropriation

Evaluation of the Safe and Family Centred Maternity Hospitals Initiative -MSCF-(Argentina)

(Saint Lucia, Caribbean)

The Self-evaluation Process in the National University of Lanús -UNLa-(Argentina)

The View of Those Who See

Participatory Evaluation of the Cancer Care and Prevention Programme in Valle de la Estrella(Costa Rica)

Page 91 Page 113 Page 133 Page 157

Introduction

If you Don’t Ask, you Won’t See it!

Giving Account and Becoming Fully Aware of the Account

Indigenous Women, Territoriality and Evaluation

Qualitative Evaluation of the Oportunidades Programme(Mexico)

Process and Impact of Youth Participation in Territorial Development in Santa Catalina Valley

(Peru)

The Story of a (de)Constructed Road (Colombia)

Page 35 Page 51 Page 71

Page 11

Roving Caregivers Program

ForewordPage 7

7Foreword

For many in the English speaking world, Central and South America is that place you travelled around in your early 20s; cheap, fun, ancient and slightly mysterious. Somewhere you always meant to go back to but never quite made it. The reality is that it is also bracingly modern, strikingly diverse and nurtures unique intellectual traditions little known and barely understood outside the Spanish and Portuguese speaking worlds.

Within the evaluation field, translation tends to go one way – from

English to Spanish or Portuguese. So while in Central and South America much is known about North American and European evaluation theories and practices, not much has travelled the opposite direction. And that is our loss. We are missing the experience and insights of evaluators working with 600 million people; taking Latin American concepts and blending them with those from other parts of the world. This book gives voice to that experience.

But this is much more than a collection of historias latinas. There

Foreword

9Foreword8

is an orientation here that informs issues that go to the heart of current evaluation debates; how do we handle complexity, ensure usefulness and gain legitimacy. In other words, Latin American’s can teach us a few things that we need to know.

How? To make some massive generalisations, the English speaking world – or perhaps more accurately Anglo American culture – tends to simplify the complex; whereas in Latin America life is more accepting and skilled at navigating complexity as complexity. Secondly individual responsibility to the collective social good bear heavier in Latin America, the notion of ‘independence’ personally or professionally is not such an obsession. And finally legitimacy is awarded by working towards broader issues of social justice - the core of liberation theology. That an

intervention works is nice, and the fact that evaluation is used is fine but legitimacy is awarded if both actually improve people’s lives? Usefulness trumps use.

What do I mean by usefulness trumps use? For years we Anglo-American evaluators have worried about how to get our evaluations used; the evaluation literature abounds with stories of use. But today we are frequently asked make our evaluations useful. There has been a shift in emphasis from the technical matter of getting our evaluations used to the social process of ensuring that our evaluations are useful. And that poses the ethical challenge of deciding to whom it should be useful. As befits this ethically based social justice orientation, this book indeed pushes beyond the immediate impacts of evaluation use and asks us

to consider the more long term ideas of evaluation usefulness to social goals. Finally someone is addressing the question “what ought to be the consequences of an evaluation?” In the authors’ own words,

"[...] the ultimate reason for evaluation is to contribute to this social betterment or impact. This includes, but at the same time goes beyond the mere use of evaluation results that change policies or programmes. In this way, the use of evaluation per se stops being evaluations’ final objective, since it aims at changes that promote improvements in people’s lives. The stories illustrate how evaluation itself has this potential to produce a positive impact in people’s lives."

So this book is for you both if you are curious to read about places you visited in the past and even more so

if you are wanting to engage with the forces that will determine the future of our evaluation craft.

Bob Williams

Introduction

13Introduction

Evaluation is the systematic process of assessment and critical analysis of projects, programmes, policies, or other types of social interventions. For that purpose, evaluations (a) apply methodologies aimed at assessing whether the design, management, and results produced are consistent with what had originally been anticipated; (b) assess whether the actions carried out were suitable in order to produce the desired changes, identifying contextual factors that

had bearing on the results; and (c) obtain evidence that backs up the evaluative judgement.

Evaluation practice does not follow a unique method, but includes a range of methodological strategies, scopes and audiences (authorities, extension workers, intermediate technicians, mass media, NGOs and citizens in general). Its results provide material for decision making, contribute to the learning of teams and organisations

Recognising Evaluation Traces

15Introduction14

actively is a way of ensuring that evaluations will not only have to meet the users’ needs but evaluation will also meet quality standards¹ based on credible evidence.

However, the literature about evaluation quality is scarce. In the pursuit of bridging the gap between theory and practice, or between resources invested in evaluation and its use, this publication aims to gather and analyse a collection of stories about evaluations in Latin America and the Caribbean that have left a trace, and made a difference.² Analysis of the evaluation stories led to identification of factors that facilitate

valuable evaluation and contribute to the body of knowledge of evaluations aimed at social betterment; meaning evaluations that have a positive impact on people’s lives.

From this perspective, the ultimate reason for evaluation is to contribute to this social betterment³ or impact. This includes, but at the same time goes beyond the mere use of evaluation results that change policies or programmes. In this way, the use of evaluation per se stops being evaluations’ final objective, since it aims at changes that promote improvements in people’s lives. The stories illustrate how evaluation itself has this potential to produce a positive impact in people’s lives.

The request for evaluation stories was issued through multiple forums and social networking sites. Consent from the evaluation or evaluation team, together with the political officer in

There is not sufficient literature in regard to the fundamental matter of evaluation quality

and increases knowledge of the problems that are the focus of public action.

The interest in evaluation of public policies and development interventions in general has grown significantly in the last fifteen years. This phenomenon is reflected in the range of theories and methodologies, the increase of national evaluation policies in countries from all continents, the increasing institutionalisation of evaluation and the emergence of

different initiatives oriented towards professionalising this practice. The relevance of evaluation is a global reality in which the interests and the actions of the main international development agencies, networks and regional evaluation associations, foundations and non-governmental organisations, different State agencies, and the academics converge.

Together with increased interest in evaluation, the launch of the Sustainable Development Goals presents a turning point in evaluation theory and practice. In this context, there is commitment to building a new set of priorities for evaluation on a global scale from 2016 to 2030. This aims at reducing the gap between the evaluation community (supply) and the policy making community –executive and legislative powers, together with organisations and social actors – (demand). Giving civil society an opportunity to participate

Evaluation is the systematic process of assessment and critical analysis of projects, programmes, policies, or other types of social interventions

16

charge of the policy or intervention was required. Seven stories were selected from the region.

1. Qualitative Evaluation of the Oportunidades Programme (Mexico).

2. Process and Impact of Youth Participation in Territorial Development in Santa Catalina Valley (Peru).

3. The Story of a (de)Constructed Road (Colombia).

4. Evaluation of the Safe and Family Centred Maternity Hospitals Initiative –MSCF- (Argentina).

5. Roving Caregivers Program (Saint Lucía, Caribe).

6. The Selfevaluation Process in the National University of Lanús -UNLa- (Argentina).

7. Participatory Evaluation of the Cancer Care and Prevention Programme in Valle de la Estrella (Costa Rica).

This introduction seeks to describe the elements that facilitate and characterise an evaluation that makes an impact. These elements cannot be seen as written in stone, they should be taken as a starting point in order to understand how and why evaluations can become a turning point.

Evaluation can have a transformative impact on the lives of the users of the programmes and policies by giving them a space for their voice and their expression, contributing to their inclusion in decision-makers’ mental models. This situation is enhanced in the many cases where decision-makers do not have a close connection to the reality of program beneficiaries, not knowing their needs or contexts. The story of qualitative evaluation of the programme Progresa / Oportunidades (México) illustrates how evaluation identified language barriers that prevented very poor natives from benefiting from a money transfer programme. Changes to the program

19Introduction18

of the evaluation team was highly significant. Without that technical rigour there would have been no guarantee that the evaluation would benefit anyone. For example, in the evaluation of the Mexican programme, the key factor was the evaluators’ anthropological approach, whereas in the evaluation of the UNICEF programme in Argentina, the interdisciplinary nature of the evaluation team was very important. Beyond the technical rigour, communication of the evaluation results to the various relevant actors involved is becoming increasingly important. Communication draws attention to the type of report used, adapting language to different audiences and generating learned lessons that fall within the ability of the organisations’ abilities to respond.

Linked to this, there appears a recurrent tension in evaluation programmes. Evaluators often try

to keep a certain distance from the evaluated programme in order to protect their independence. However this increases the possibility that the evaluation becomes distant and irrelevant for those who need to act on the results. The closeness of the evaluators to the evaluated programme and its actors gives opportunities to make a difference through the use of the evaluation processes. This recognises that the benefits and impacts of evaluation emerge as much —or even more— from how an evaluation is carried out (usefulness of the process) as much as in relation to its findings (usefulness of its results).⁴ This situation is illustrated in the self-evaluation process of the National University of Lanús (UNLa), with the involvement of the university community, which allowed democratic access to information, innovative interaction, discussion and debate, as well

allowed communication in local languages, which greatly improved the ability of people to understand the programme’s requirements (such as children’s regular school attendance) and therefore, be able to benefit from the money transfer the programme offered.

Recently programmes and development policies designed to improve people’s lives are expected to be based on credible evidence. However, there is not one single way of generating believable and convincing information, given that what is ‘believable’ depends on the situation and on the specific actors. In some cases, the key is to assess the programme results using an approach that helps the process to be perceived as methodologically rigorous. In other situations, a believable evaluation entails understanding the perspectives of the most relevant actors in the intervention (as in the

case of the Safe and Family-Centred Maternity Hospitals Initiative, in Argentina), whereas in others evaluations credibility was achieved through the active participation of the users in the evaluation process (e.g. the participatory evaluation experience in the cancer prevention and care programme in Valle de la Estrella in Costa Rica).

In the cases described here, the technical ability and competence

The use as an end in itself stopped being evaluation’s final objective but was aimed at making the changes it promoted to produce improvements in people’s lives

21Introduction

as strengthening abilities of the participants.

The evaluation stories revealed in the project illustrate the value of a positive approach.5 They emphasize the evidence of what works and/or might work and is worthy of being continued, enlarged or modified. This can be contrasted with evaluations that are focused mainly on the technical problems and deficiencies present in most interventions. This type of approach has been shown to be highly relevant in getting beyond defensive and suspicious attitudes, and instead promoting a constructive focus on possible solutions. For example, the evaluation of the initiative Strengthening the Abilities of Indigenous Women to Set up and Have a Bearing on the Implementation of Public Policies (Colombia) validated many of the approaches adopted, and indicated the potential that could be developed by having an additional

training to the objectives outlined originally in the project. In addition to that, in evaluations with a positive approach, evaluators usually develop a close relationship with the actors of the intervention, understanding them and supporting them. The evaluators' task in these cases is not limited to indicating what to be modified. Many times local actors regard this as evaluators demonstrating ‘commitment’ to the project and its future.

Participation of the actors in the collection and use of the evaluation data is a powerful way of including users and beneficiaries. This allows participants to get involved and understand the data better. An evaluation characterised by a collaborative approach leads participants to take responsibility for the evaluation and then for the change and transformation that follows. In this way, active participation in the

23Introduction22

explicitly participatory (as in the self-evaluation of the UNLa in Argentina and the evaluation cases in Costa Rica and Perú), the actors-beneficiaries had a say in the formulation of the central questions and decisions in the evaluation design.

The willingness and motivation of the organisation and the actors of the programme to carry out an evaluation are also key factors that help the evaluation make a difference and

have an impact. This is usually called 'political willingness' since it refers to the higher management levels of the organisation where the initial foundation for the development of the evaluation occurs. This factor is highly important and was part of all the analysed evaluation cases that were assessed as making an impact.

Participation in evaluation processes requires the willingness of the evaluators to adopt approaches that allow people’s active involvement and participation. The thought that most evaluations are carried out in order to fulfil accountability requirements is very common. In such instances the final report is too often merely stored on a shelf (or hard drive) and the programme continues unaltered. However, in evaluations that make a difference it is always possible to find an actor or group of actors truly interested in making the most of

evaluation process helps to develop better understanding of evaluation and contributes to commitment and use. This is illustrated by the participatory evaluation in Costa Rica, in which regional technical teams were involved and deeply interested in getting to know how the evaluated program worked in their area. In contrast, the higher authorities limited their participation to approving the evaluation. In this way, recommendations at regional and local levels were applied soon after the evaluation finished, whereas general recommendations —dependent upon the higher authorities— have not yet been applied.

Clearly, the more participatory the evaluation is, the more necessary it is to ensure the willingness and motivation of the most relevant actors in the intervention (beneficiaries, local technicians, officers) in order to

promote the impact of the evaluation and for it to make a difference. This motivation is less notable when the actors play merely an advisory role. Their relevance clearly decreases if they ignore the central questions of the evaluation effort. For example, in some cases of experimental or quasi-experimental evaluation designs in which the opinion or the perspective of the actors is secondary to other components (their behaviour, the impact of vaccines or diets on their body, etc.), the role of the participants is limited to answering questions and allowing the evaluator to carry out some measurements. However, this does not always prevent the evaluation from making an impact on the people's quality of life. Referring to the evaluation of the UNICEF programme in Argentina, direct users were not involved in the evaluation, yet this did not stop the evaluation from having an impact. In the cases

Evaluations with this positive approach are usually described with characteristics that show there was support, empathy and closeness attitudes towards the actors of the intervention

24

the learning that evaluations can provide. These evaluations are valued as positive since they allow users to make appropriate decisions to achieve meaningful changes in the intervention. In the story of the evaluation of the Caregiver programme in Saint Lucia island, the commitment to the use of the evaluation helped the process make a difference in people’s lives. In that experience, the ‘personal factor’ —understood as the presence of an individual or group of identifiable individuals who are personally concerned with the evaluation and its ability to improve people’s lives— worked as the main driver for the use and impact of the evaluation.

The description of an internal evaluation or self-evaluation as participatory depends not only in its final aim but also of the involvement of the actors-participants. When an internal evaluation is participatory,

it faces an important challenge when configuring evaluation teams, because in becoming evaluators team members must go beyond their normal organisational roles. This challenge entails developing evaluation skills that allow reasonable levels of competency in the team responsible for the participatory evaluation.

The evaluation story in Peru indicates how the evaluation team was developed over an extended period of time, and supported by a range of stakeholders. The evaluation story in Costa Rica shows that, without being a strictly speaking self-evaluation, it had the key components of a participatory evaluation. Having a limited time frame for the training of the team members resulted in certain limitations during the evaluation process. The key factors to be taken into account when developing the skills of evaluation team members are time, the content to be taught

27Introduction26

they were being involved in an evaluation process; this was made clear soon after the women arrived.

The impact of an evaluation can be increased by having champions able to influence those who make key decisions and necessary changes. It is usual that the people who have real authority to make decisions are external to the programme and have not participated in the evaluation process. Thus, even though the programme staff as well as the directors who took part in the evaluation are committed to improving the programme, other interested parties need to be convinced that the changes are necessary. The champions in the evaluation stories were people who cared deeply for the affected families and communities, and at the same time had an influence on people who were able to make decisions, playing a fundamental role

so that the changes could take place. In the example of the evaluation of the Progresa / Oportunidades programme (Mexico), that role was played by an actor who believed in the potential of the evaluation effort, and facilitated the implementation of some of the suggested recommendations.

It was difficult to find evaluation cases relevant and at the same time approachable for this study making the selection of evaluation stories particularly challenging. It was ex- tremely difficult to have access to evaluations whose main actors (officers, administrators, evaluators) considered that the evaluation had made a (positive) difference. This could be because most evaluators tend to distance themselves from the impact of their evaluation, once the

(relevant to the specific evaluation task they have to carry out), support and supervision by one or more experienced evaluators, as well as applying skills people already have.

At the early stages of the evaluation, most of the beneficiaries of the

intervention as well as the actors who implement the programmes (field technicians, officers in charge of the implementation, etc.) tend to consider evaluation from a point of view of control and accountability. Generally, the start of an evaluation process does not create excitement or expectations connected to the learning dimension. In the evaluations that made a difference, the situation changes when the evaluator or evaluation team are able to show through their words and actions that evaluation has the potential to improve programs, overcoming narrow views connected with monitoring and control, accountability, rewards and sanctions. The evaluation story in Colombia illustrates the way in which indigenous women were invited to participate in the evaluation. They were invited for a talk or chat with the intention of ‘lowering anxiety’, without explicitly mentioning that

The evaluations that leave a footprint require evaluation ‘champions’. That is to say, people who care deeply for the affected families and communities, who are capable of having influence on those who have authority to make decisions and encourage necessary changes

Leave Nothing But Your Footprints

29Introduction

final report has been submitted. This is a paradoxical situation, because often evaluators are not willing to do the self-reflection and assessment as to whether an evaluation made a difference. What they want their clients to pursue is too often not what they tend to do in their evaluation practice, stating that they have no control over the use of evaluation findings. Only a small fraction of the evaluation stories initially selected for the project were able to show a clear connection between the evaluation and the benefits for the people derived from it. Generally, evaluators do not go into detail on how their work can have a positive impact on people’s lives. They assume that their responsibility and tasks do not extend beyond selecting the appropriate methodology or method capable of influencing decision-making.

In the global neoliberal context, evaluation runs the risk of becoming

another service which gives answers wanted by those who pay for it. Evaluations tend to concentrate excessively on efficiency, effectiveness and measurable results on a short-term basis, rather than contributing to democratic, transformative and participative purposes that the evaluation community holds as central. ‘Speaking the truth to power’ may be naive and insufficient if the inherent political nature of evaluation is not recognised. This entails extending the focus of action of the evaluation to contribute to public good, broadening its interest towards medium and long-term results, to unexpected consequences of development interventions and investigating the causes of some social problems that programmes and policies aim to deal with. Giving evidence to the subjects of the political intervention entails ‘speaking truth to the powerless’,6 considering them as legitimate stakeholders in the

30

evaluation results and aiming at their empowerment to speak for themselves and act for their own benefit. Developing strategies for that is a rich field for evaluators, as the evaluations that made a difference have shown.

1. See in this regard the "Evaluation Standards for Latin America and the Caribbean", developed participatively by the ReLAC (the Latin American Evaluation, Monitoring and Systematization Network), FOCEVAL and DEval.

2. This publication deepens the work produced for the Latin American and the Caribbean regions in the context of the project "Evaluations that Make a Difference: Stories from Around the World", carried out by an international team led by Burt Perrin and financed by the EvalPartners initiative. The production of this project can be consulted in the following link:https://evaluationstories.wordpress.com/

3. Mark, Melvin M., Gary T. Henry, and George Julnes. 2000. "Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs". San Francisco, CA: Jossey-Bass.

4. Cousins, J. Bradley, Elizabeth Whitmore, and Lyn M Shulha. 2013. "Arguments for a Common Set of Principles for Collaborative Inquiry in Evaluation". American Journal of Evaluation 34 (1): 7–22.

5. Perrin, Burt. 2014. "Think Positively! And Make a Difference Through Evaluation". Canadian Journal of Program Evaluation / La Revue Canadienne D’évaluation de

Programme 29 (2): 48–66.Stame, Nicoletta. 2014. "Positive Thinking Approaches

to Evaluation and Program Perspectives". Canadian Journal of Program Evaluation 29 (2): 67–86.

6. Mathison, Sandra. 2018. "Does evaluation contribute to the public good?" Evaluation, 24(1), 113–119.

Stories of Evaluation

If you Don’t Ask, you Won’t See it! Qualitative Evaluation of the Oportunidades Programme(Mexico)

37If you Don’t Ask, you Won’t See it!. Qualitative Evaluation of the Oportunidades Programme (Mexico)

Contemplating the indigenous women of the Sierra Tarahurama, a mountain area in the state of Chihuahua in northern Mexico, evokes bucolic scenes from the time of the Spanish conquest. These communities have maintained themselves for centuries far from cities, clustered in small villages and scattered family groups, cultivating a few seasonal crops and raising chickens, goats, and cattle. Almost all live in poverty, and in many cases they are semi-nomadic.

Access to the Sierra Tarahurama is difficult. Sometimes it takes several days to reach the settlements.

It is therefore quite difficult to make contact with them to interview and select candidate families for the Programa de Desarrollo Humano Oportunidades (Human Development Opportunities Programme, hereafter referred to as Oportunidades).1 But

The programme aims at nothing less than breaking the seemingly endless cycle of poverty that typifies many rural communities

38

the effort is well worth it because the programme is quite remarkable.

This programme, implemented in Mexico since 1997, aims at nothing less than breaking the seemingly endless cycle of poverty that typifies many rural communities. It does so by using a conditional cash transfer (CCT) approach, where families are provided with payments that are conditional upon undertaking certain activities, such as ensuring regular attendance of their children in school or obtaining certain health services. These incentives assist in the achievement of higher standards of education, health, and nutrition, and also provide necessary support to the people of the Sierra Tarahumara to undertake economic activities that enable them to increase their family income and quality of life.

The indigenous communities were among the intended beneficiaries of

the programme since its inception, and they also participated in their evaluations from 1999 to 2006. But although these evaluations confirmed that indigenous people were effectively participating in the programme, it was unclear if they were achieving the stated objectives in terms of health, education, and ultimately, economic performance. According to Mercedes Gonzalez de la Rocha, anthropologist and head of the qualitative impact evaluation of Oportunidades, the absence of a clear focus on indigenous communities had left a blind spot in the programme’s knowledge base. Or as she puts it, If you don’t ask, you won’t see it!.

This statement constituted a starting point for raising the influence of an ethnicity variable in the new qualitative evaluation to be conducted in 2008. By then, the programme had a decade of experience in implementation in Mexico, which allowed for a thorough

41If you Don’t Ask, you Won’t See it!. Qualitative Evaluation of the Oportunidades Programme (Mexico)40

evaluation. Evaluators designed a strategy for field work comprising eleven indigenous intercultural regions in the states of Chiapas, Chihuahua, Oaxaca, and Sonora. In each location, the programme’s coverage and operations were analysed. The evaluation identified the main obstacles to programme implementation with particular attention to the relationship between the extensionists (promotores) and the women representing the indigenous communities, the vocales.

What they found was quite unexpected: there were serious communication problems with language. Virtually none of the promotores and only a few of the vocales were bilingual. Although the programme had been operating since 1997 with more impact in indigenous areas than in non-indigenous areas (as had been documented by previous evaluations), the important

problem of communication had not been sufficiently addressed. For instance, the evaluation found that young indigenous women – most of whom were bilingual, although their Spanish was often limited – did not accurately understand the Oportunidades employees and the technical information they provided. This situation was exacerbated in the case of elderly women, who spoke no Spanish at all. The promotores assigned to the region did their best to overcome the language barrier, but the results were unsatisfactory.

In some areas, the majority of indigenous women did not understand what the programme was for. They couldn’t understand what good it did to spend hours listening to medical specialists who spoke about issues they did not understand in a language they could barely comprehend. Sometimes the information that was being communicated even conflicted

with their traditional customs. For example, when indigenous women participated in training on the importance of a physical examination for possible breast cancer, it was clear that the idea of a stranger touching them in their private parts made them very uncomfortable. These women do not even undress in front of their husbands! Thus, a practice intended to save their lives was totally unacceptable for cultural reasons.

The evaluation found that although the programme’s coverage had improved in some indigenous areas, in the Sierra Tarahurama, a whopping 30% of the population remained outside the programme. It was clear that the lack of access to health services and education was brought about by the problem of monolingual families who could not benefit from the written and oral information conveyed by Oportunidades staff members to the women.

It was urgent to take measures to solve this serious problem, and Oportunidades did just that. The qualitative evaluation suggested that bilingual promotores be recruited from indigenous youth alumni so as to contribute to better communications and smoother operations in indigenous communities. This arrangement also led to a further

The evaluation identified the main obstacles to programme implementation with particular attention to the relationship between the extensionists and the women representing the indigenous communities


positive impact by providing jobs and wages to the young in their own living space, allowing them to share the learning they had acquired right there in their own communities.

The then general coordinator of Oportunidades, Dr Salvador Escobedo, recalls:

We managed to make the change in rules of operation to include bilingual extensionists, and that was the first step. Then in parallel we ran a training programme with the National Institute of Indigenous Languages (INALI) in order to generate a mechanism to evaluate the indigenous youths we wanted to hire as extensionists who speak an indigenous language. We have sought to work with the 13 most spoken languages such as Tzotzil, Maya, and Totonac.

To do this, INALI began training a group of 15 young men and women with diplomas as trainers of social

programmes in indigenous languages. They were the first to reach the remotest areas of the country to carry the voice of the programme to where it was most needed.

Says Escobedo, The project trained 350 promotores in order to achieve almost total coverage of the monolingual indigenous populations in Yucatan,

The evaluation suggested that bilingual promotores be recruited from indigenous youth alumni so as to contribute to better communications and smoother operations in indigenous


Oaxaca, Chiapas, Jalisco, and the Sierra Tarahumara. By the time I left the programme, we had trained a total of 250 extensionists, and awaited evaluation on the success of this implementation.

The process of bringing bilingual promotores into the programme unfolded steadily over the next 2 years. As their understanding increased about the importance of sending their children to school, feeding them properly, and learning how to use resources, indigenous women became

increasingly willing to participate and interact with the promotores and with each other. In some areas they even formed groups for early child education. The role played by the qualitative evaluation with its remit to address the marginalisation caused by monolingualism was key to this transformation.

Another important aspect that was pointed out by the qualitative evaluation of 2008 concerned the inefficient (and even absurd) survey of each household to assess whether or not it was eligible for the programme. In the case of the communities of the Sierra Tarahumara, this was redundant because every household was unquestionably poor – not to mention the difficulty and cost of getting there just to conduct the survey.

Escobedo, who understood the realities of indigenous communities

The evaluation pointed out the inefficient (and even absurd) survey of each household to assess whether or not it was eligible for the programme

in Mexico, supported the proposals arising from the qualitative evaluation that were eventually endorsed by the president.

The first step was to convince Oportunidades collaborators such as government officials and international organisations like the Inter-American Development Bank and the World Bank of the need to change the rules for coverage in isolated areas. That involved a major challenge for the programme managers because, beyond being convinced of the real value of this recommendation, they needed to effect those changes without appearing to refute the spirit of Oportunidades, which was historically focused on the poor while requiring certain conditionalities of the beneficiaries such as health checks and school attendance.

In late 2011 and early 2012, a severe drought hit the Sierra Tarahumara.

Rumours were rife about indigenous people committing suicide for lack of food, although these were subsequently found to be false. Dr Iliana Yaschine, former director of evaluation of the programme from 2002 to 2006, who coordinated a study documenting the work of Oportunidades in the Sierra Tarahumara during the drought crisis, recalls:

That report was published in a Chihuahua newspaper and generated an immediate response from the federal government, which decided to intervene in various regions with the Secretary of Social Development. It was then that they communicated the decision that Oportunidades should intervene to solve the problem precisely in the area of the Sierra Tarahumara.

The important mobilisation and attention to the area made it possible to detect 8,000 families (40,000


indigenous people) who had been dropped from the programme because they had not fulfilled such co-responsibilities as sending children to school or attending health talks. The figure was worrisome, and returning these people to the programme would be difficult, if not impossible, given the rules of operation. Reaffirming this contention, Escobedo asserted:

The World Bank and Inter-American Development Bank opposed eliminating the requirement of the co-responsibilities in these areas, in part because they would undermine the programme, thus generating a new programme. That ́s the reason why it could not be done overnight. It is sad and painful, but very real.

However, given the magnitude of the problem in the context of the severe drought affecting indigenous people in the Sierra Tarahumara, the situation was resolved after the

intervention of the President Felipe Calderón himself, who ordered the immediate reinstatement of 8,000 families. This was accomplished by integrating the reinstatement to the Food Support Program, a transfer programme that did not require the fulfillment of co-responsibilities that was also operated by Oportunidades. The indigenous promotores brought the good news to their communities, while a renewed training of bilingual extensionists was encouraged to address immediately the indigenous families.

The rules of the Oportunidades programme were changed to implement the strategy of full coverage as it had been recommended by the qualitative evaluation


In addition, the rules of the Oportunidades programme were changed to implement the strategy of full coverage (as it had been recommended by the qualitative evaluation). This allowed in-corporating families into the programme from small and distant locations without undertaking a home survey. This change helped broaden the programme’s coverage and, in addition to the reincorporation of the families mentioned above, it made serving the entire indigenous population of the Sierra Tarahumara possible.

According to Yaschine:Without the qualitative evaluation, it would have been impossible to make that change, even with the drought crisis. That is my perception. I observed the implementation of this decision in the field at a time of crisis. Special efforts were made to deploy support for the affected families, not at the household level but in the care centres where groceries were delivered. I remember what the qualitative evaluation said on this subject, and how that had prompted the changes during the crisis.

The findings and recommendations of the qualitative evaluation in the indigenous communities provided crucial information for decision-making. Managers made changes that impacted the participants of the programme directly. In the case of the bilingual promotores, the evaluation noted a dimension of cultural adaptation that had been neglected in

The history of the qualitative evaluation shows the relevance of cultural sensitivity to the evaluation of social projects

the original design of the programme, and once it was incorporated the benefits were obvious.

The history of the qualitative evaluation shows the relevance of cultural sensitivity to the evaluation of social projects. In the words of Gonzalez de la Rocha: If you don’t ask, you won’t see it!

1. The name of this programme, originally called Progresa, was later changed to Oportunidades, and still later to Prospera, the name by which it is known today. In this report we use the second name since that was in use during the evaluation.

Interviewed: Mercedes González de la Rocha and Agustín Escobar Latapí were co-authors of this story, and directed the external qualitative evaluation of the Oportunidades Programme, making the recommendations that lead to great improvements in the lives of the beneficiaries.Iliana Yaschine was the Evaluation Director in the Oportunidades Programme from 2002 to 2006.Salvador Escobedo (Director of the programme at that time) provided valuable ideas for this story.

Writers: Pablo Rodríguez Bilella and Omar Zevallos.

Giving Account and Becoming Fully Aware of the AccountProcess and Impact of Youth Participation in Territorial Development in Santa Catalina Valley (Peru)

53Giving Account and Becoming Fully Aware of the Account. Process and Impact of Youth Participation in Territorial Development in Santa Catalina Valley (Peru)

Santa Catalina Valley is an extensive area in La Libertad region, in North Peru, whose capital is the renowned city of Trujillo. This region has been an important agricultural area for fruit and vegetable farming since pre-Columbian times, nowadays having a significant growth in the tourist sector. Thousands of visitors are usually drawn by its restaurants, mini zoos and country houses, making the most of the modern paved access roads in order to visit Poroto, Laredo and Simbal districts. These rural areas have historically undergone

an increasing emigration of its young population, in addition to a context of poverty which explains the emergence of different development NGOs in the region.

These rural areas have historically undergone an increasing emigration of its young population, in addition to a context of poverty

55Giving Account and Becoming Fully Aware of the Account. Process and Impact of Youth Participation in Territorial Development in Santa Catalina Valley (Peru)54

The project was developed in two consecutive stages named respectively ‘Building Local Development Together’ and ‘Strengthening Local Development Processes in Santa Catalina Valley’. As a whole, and as the interviewed actors remember, the initiative aimed at promoting business organisations or networks in order to encourage the emergence of workers’ cooperatives. Their purpose was to go beyond the scattering of each producer in their own plot of land and the typical problems that arise: being prisoners of informal intermediaries or financial services,

as well as the absence of access to technology transfer services. In order to evaluate progress in these projects, the ‘Evaluation Team’ was created, who received training technical guidance from the development NGO called DESCO together with monitors and coaches from the Kellogg Foundation.

The Kellogg Foundation team had created a general Theory of Change for the Integrated Group Project initiative, which was taken as reference and adapted in each region through a group of consultants and coaches who worked with the organisations in project implementation. In Santa Catalina Valley, this theory was regularly updated, showing the evolution which reflected the support of local linking strategies between different organisations (social capital), the different opportunities for income generation (economic capital), as well as the development of educational

As a whole, the initiative aimed at promoting business organisations or networks in order to encourage workers’ cooperatives

Towards the middle of the first decade of this century, these NGOs had the possibility to bring together their efforts, integrating even governmental and academic actors as part of the Integrated Group Project initiative, a proposal made by the Kellogg Foundation Latin America. This proposal was developed towards the end of the 90s, trying to go beyond the thematic approaches focused on health and nutrition that the foundation had developed before, now moving towards others of a highly territorial nature. The initiative’s main purpose was to break the intergenerational circle of poverty by working with young people, developing their individual and group abilities, and at the same time having an influence in their local context so that it could enhance their development. In this way, all proposals backed by financing and technical guidance would be submitted in an alliance of organisations: local council,

ministries, grassroot organisations, private actors, NGOs, etc. The initiative was developed in the most critical areas of the region: Central America, North-East of Brazil, and the Andean Region (Bolivia, Ecuador, Peru). In the Santa Catalina Valley region, four NGOs agreed to submit a proposal and to start working on a project (CEPEDAS Norte, MINKA, CIPS, Sara Lafosse, CEPROCUT1), together with the National University of Trujillo and some local governments. This involved the first challenge, bringing together different organisations and people from different backgrounds, and different intervention objectives in order to work towards a common goal. The project that resulted included the Poroto, Simbal and Laredo districts, in which organisations, institutions, town halls and Irrigation Board of Users participated. This work was developed in the period between 2005 and 2009, the key of this work being the evaluation component.

56

and leadership opportunities for young people (human capital). In this respect, Federico Tenorio —member of the CEDEPAS NGO— gave his opinion:

The Theory of Change was improved and refined as a product of four years of work. There was a preliminary idea of the Theory of Change at the beginning, we were thrilled about this and along the way it was adjusted; what we wrote at the end was much more accurate and precise than what we had at the beginning. We always highlight that the Theory of Change worked very well for this field and it depended on the level of participation of the local actors (young people).

With the support of the regional NGOs involved in the project, young people of Santa Catalina Valley were gathered in business groups or networks. Many of these networks were oriented towards small cattle and vegetable

farming, and the groups consisted of 10 to 15 producers, including young people and adults, who got together to exchange information and produce as a group, generate shared learning and gain an economy of scale for the purchase and sale of consumables products. Other networks worked by linking themselves to a bigger business sector, for example: linking small carpentry business to supply for constructors or ports, or in the growing tourist sector of the region. The work with youth leaders entailed creating 40 youth organisations, promoting their leadership roles in topics of political incidence, social awareness activities and actions of cultural promotion. Susana Shoaie, from the Kellogg Foundation, was linked to the first stages of the project in the Andean region, giving her support in the development of the Theory of Change, and she said the following:


main motivation for implementing this process was to develop those abilities to enable local actors to carry out their own evaluations directly in the future. In each of the territories where the Integrated Group Project was developed, evaluation groups or teams were created with whom an evaluation training programme was carried out. This involved attending a year-long evaluation course in Lima and their follow-up in the field for two more years.

In Santa Catalina Valley, members from different participating support

organisations were selected to be part of the evaluation group, creating the team coordinated by Dr Esther Ramírez, from the Trujillo University, and the participation of Luciana Alfaro (CEDEPAS), Cecilia Montenegro (CEPROCUT), Sonia Mendoza (MINKA), Marco Luján from the Laredo Local Council and Nelly Aliaga (Ministry of Agriculture), as well as other members from the NGOs who were included throughout the years. Together with the team and other members of the local teams —who made up an audience of 90 people—, Desco began the evaluation training course. The course is remembered by the evaluation team members from Santa Catalina Valley as a ‘training and practical course’, given that what they learned in the training was immediately put into practice. Esther Ramirez shared her memories of this moment:

The main motivation aimed at developing evaluation abilities, so that the local actor could carry out their own evaluations directly in the future

This was a large and complex project consisting of two stages. Each of these stages lasted two years and were both funded by Kellogg. It involved a cultural change in young people’s view, from rebellious and problematic to being the agents of change, believing that in this way young people can have a positive impact in their own lives, their families’ and their communities. Likewise, it involved the local organisation’s commitment by supporting these initiatives.

The general evaluation of the Integrated Group Project initiative started to develop from the creation of the project itself, starting from the logic model production and the construction of the evaluation matrix and their corresponding instruments. The complex initiative entailed a complex evaluation, with relevant efforts as regards the baseline configuration, design and data processing. It was agreed that the evaluation nature of the initiative, apart from contributing to the subregion and Latin America, should adopt a profile highly focused on its usefulness locally.

To give support to the evaluation process in the Andean region, the NGO called Desco was brought in and it chose to implement a capacity development process for the evaluation, virtually at the beginning of the project. The

Evaluation in Process

The work with youth leaders involved creating 40 youth organisations promoting their leadership roles in topics of political incidence, social awareness activities and actions of cultural promotion

61Giving Account and Becoming Fully Aware of the Account. Process and Impact of Youth Participation in Territorial Development in Santa Catalina Valley (Peru)

It was an interesting work group, the work involved carrying out an evaluation with a new approach, which gave us some results while we were implementing it. It was an evaluation where we had to give an account, but starting with becoming fully aware of what to account for, give an account of what was going on. Becoming fully aware of learning was the most important part of the evaluation.Trying to achieve learning and growth in every member of the different teams we were working with, while at the same time making decisions about the project. The idea was not simply to evaluate the impact but also the process itself, based on the results and according to the different project components.

Marco Luján, civil servant of the Laredo Local Council, contributes with some details about this process:

In the first place, there was screening to create the team. We did a test, and our CVs were evaluated. It was interesting

because, if I remember correctly, there were 22 institutions who had been asked to be included in the evaluation, from which only six were selected. So, it was a very demanding process considering the selection process we went through. Then we went through a training process in Lima; it was a year-long and consisted in theoretical lessons during a whole week and later applying what we had learnt in the field.

After the baseline of the project was elaborated, the team scheduled their actions in such a way that every three

The idea was not simply to evaluate the impact but the process itself, based on the results and according to the different components


In CEDEPAS, we have a lot of experience with different projects and we have been through many evaluations where there was a strong control and vigilance, that is to say, a police type of evaluation. In this case, the evaluation proposed was more like a learning process. This was achieved, through a participatory process, drawing important lessons for everybody, for the Local Council, the producers and their leaders.

Given that the members of the evaluation team were part of the organisations directly involved in the project implementation, they understood each other and took over the task as an internal evaluation team.2 In turn, the internal nature of the evaluation team, comprising members of the organisations implementing the project, resulted initially in some tension as well as learning. Its members remember that in this way:

Esther Ramírez: At the beginning, each organisation instructed their representatives: ‘You have to stand up for our work’, ‘You have to make sure we do well’ But the idea was not to do well in the exam, but realise that what we need to work well and that those who are working in the project can grow.

Marco Luján: So, when you had to evaluate your own organisation, many

The members of the evaluation team understood each other and took over the task as an internal evaluation team, whereas the external evaluation would be carried out by hired consultants

months they would meet and receive the evaluation reports from each of the institutions involved in the initiative. Based on those reports and guided by the idea that the evaluation should help strengthening the human capital, the evaluation team carried out field visits and worked with the technical teams, as well as the people involved in the project,. Esther Ramírez remembers the way the work took place:

In the field visits, based on the reports, technical teams were asked to show us a typical activity, which we knew they had already done. We worked with farmers, cattle breeders, artisans, carpenters, school teachers, and we also worked with young people and observed in the field if the work reported to the management team was being accomplished. We chose, for example, an activity X, and observed how they carried out this activity from start to finish, keeping their accounts,

following the guidelines stipulated by the management team.

The evaluation team little by little generated an evaluation culture in which, based on the technical reports received and their discussions in meetings with different actors, they would carry out field visits and meetings with the head officers of the implementation teams and with the targeted population. They were clearly interested in allowing some space to hold interviews in depth so that they would help to overcome the control or accountability viewpoint (‘we didn’t go to the field to check if everything was working nicely, and that it was arranged to look in a certain way because the evaluation team was coming’), The evaluation was trying to contribute to continuous improvement to the work being carried out. Federico Tenorio commented about it:


In reference to the objective of developing evaluation abilities, the experience of the evaluation team in Santa Catalina Valley has also been fruitful. To that effect, Molvina Zeballos says:

In Santa Catalina Valley it is worth it to highlight how the evaluation team included members of the local government in this evaluation process. As a result, in different territories of the Andean region, abilities were developed, Many of the team members are dedicated to evaluation now, some of them are consultants, and others replicated some evaluations.

Marcos Luján, linked to a local government of the territory, also gives a review of some of these learning experiences:

An important aspect is that a part of this methodology was included by local governments in the oversight committees in relation to participatory budgets. These committees oversee that everything that is budgeted for is executed in an effective manner. In that case there was an interesting methodology: a brochure was used, and I understand that this format is still used to evaluate public investment projects. From the point of view of the evaluation and the actors, they have taken on board the methodology and have become leaders. The development of abilities that was included in the evaluation process contributed with an image systematisation methodology which was so useful, practical and didactic that I still use it and I have taken it on board.

times you wanted to show the best aspect, but all of your team mates told you: ‘We need to be objective, we need to show what we find. And where you haven’t made progress, show why there hasn’t been any progress and look to how that can be improved.’ That was a learned lesson, that the evaluation team was so empowered that it risked becoming the ugly duckling of the project. Sometimes the rest of the participating organisations did not think well of us because when we met every three months to submit the project evaluation, and observed that the progress was not enough in accordance with the schedule we had presented ... Then there was some friction, that was overcome little by little; but the meetings were heated. It is about learning a lesson because the fact that you are evaluating yourself often means you are more demanding.

Susana Sohaie gives an additional perspective with respect to this topic from the initiative’s management:

For us, the work that these teams carried out was foreign to the image of external evaluation. There were some organisations that were eager for their own staff to be trained in evaluation and made the most out of it. For others it meant a bigger workload and looked at it as something imposed on them, Besides they regarded it as an external evaluation but with local resources! (laughs). Not all organisations experienced it in the same way, but many of them did deem it as an opportunity in many respects and they stamped their own seal to their involvement.

Not all organisations experienced it in the same way, but many of them did deem it as an opportunity in many respects and they stamped their own seal to their involvement

The importance of the use of evaluation was emphasised throughout the training implementation process

66

For me, someone who has been working in public organisations, it was very important to get to know the NGOs and their work, this had an impact on me.

The importance of evaluation use was emphasised throughout the training and implementation process. It was understood that once the information had been processed and the results discussed, the fundamental question that evaluators and people who implement the project had to ask was how they were going to use it. In order to modify (or reinforce) certain processes evaluators had to find specific methods and approaches through which the direct participants, as well as the local authorities, could use the findings and assessments. This recurring emphasis on use is highlighted by different actors of the evaluation who strongly integrated the notion of outcome as going beyond the outputs developed.

At the level of the general initiative, it turned out to be an additional challenge to communicate the evaluation results to the Kellogg Foundation management. Whereas the local actors considered that the participatory instruments and strategies helped to indicate the changes that appeared —in their consolidation or in their emergence— there appeared larger difficulties to show to Kellogg the cultural and process changes. Together with that, changes in the institutional context generated additional difficulties for using the evaluation, just as Susana Sohaie mentions:

Something happened: the foundation had a change in management during this period. The new management had a different approach for the intervention, focusing on childhood —the origin of the foundation—, that is why some changes had to be made as regards the theme


nature with the actors, combining the internal evaluation with the external consultancy. The participants in this project consider the independence that each evaluation team had as valuable and positive, so that it allowed them to deepen and include their own elements over the minimal required in the general evaluation of the region. Members of the evaluation team have tended to recall the emphasis that the evaluation had in the growth and strengthening of the evaluation capacities of each of the organisations, expressed in the idea that ‘the project has finished, but the organisations and the people continue’.

In the formulation of development projects, evaluation always tends to have its own space, even has its own budget funds. But local organisations and NGOs do not always have the capacity needed to implement those evaluation processes. In that way, one of the greatest benefits of evaluation in the Integrated Group Project initiative in the Andean region was improving the evaluation capacities in the territories where

it was implemented, from a learning perspective, through the processes, the empowering of all the actors involved and the use of the results for continuous improvement.

1. CEDEPAS: Centro Ecuménico de Promoción y Acción Social Norte; MINKA: Centro de investigación. Estudio y Promoción del Desarrollo, Centro Lafosse: Centro de investigación and Promoción Social Sara Lafosse.

2. In contrast the external evaluation would be carried out by consultants hired by the Kellogg Foundation for that purpose.

Interviewees: Molvina Zeballos, Federico Tenorio, Susana Shoaie, Marco Luján, Esther Ramírez, Cecilia Montenegro Salgado.

Writers: Vanesa Castro and Pablo Rodríguez Bilella.

and geography. It happened in different organisations, not only in Kellogg, who started to reduce their international commitments. Towards the years 2007-2008, the Kellogg Foundation decided to concentrate in Mexico and other areas of Central America: areas with a greater link to the USA.

As soon as the project and the final instances of the evaluation finished, Kellogg Foundation moved away from the Andean area. Some of the experiences seemed to need longer to make progress in their consolidation, especially since they were focused in the work with young people, who because of their age, were defining their own lives, looking for new opportunities, and prioritising their education many times outside the territory, reducing their level of participation. However, there were

young people who found in the proposal a very valid alternative for their social and economic development; they started with a small amount of capital, reached a notable diversification, and still participate in the cooperative efforts until the present. As regards the training of leaders by developing new perspectives in local political processes, the results were less effective than expected.

Different from other regions where the Integrated Group Project was implemented, the internal evaluation in the Andean region adopted a participatory and collaborative

The Evaluation Capacity as a Result

The internal evaluation in the Andean region adopted a participative and collaborative nature with the actors

Indigenous Women, Territoriality and EvaluationThe Story of a (de)Constructed Road (Colombia)

73Indigenous Women, Territoriality and Evaluation.The Story of a (de)Constructed Road (Colombia)

Over the last century, Latin American women have unitedly spoken out as a means to resolve their ancestral struggles. One example are the in-digenous women organisations in Colombia, who know what they want and have a defined road towards achieving their goal.

First and foremost, their struggle focuses on land entitlement or, as they refer to it, land restitution. Their displacement resulted from an armed conflict in which almost 70% of the indigenous population was removed from their ancestral lands

—a process which has, in recent years, started to be reversed but still needs to consolidate. Second, their struggle aims at restoring the rights of the victims of armed conflict, with special attention given to those who have disappeared– especially women. Third, this struggle reasserts indigenous women’s rights, in light of a history of systematic rapes, forced disappearances, and murders. This is how their struggle focuses on land, restoration of their land rights, the people who disappeared (husbands, brothers, and fathers). This last struggles takes into consideration

75Indigenous Women, Territoriality and Evaluation.The Story of a (de)Constructed Road (Colombia)74

(councils). At the regional level, the Indigenous Organisation of Antioquia (Organización Indígena de Antioquia - OIA) stands out; whereas at a local level, the Association of Indigenous Councils of Northern Cauca (Asociación de Cabildos del Norte del Cauca - ACIN)is worth mentioning.

These organisations have a relevant trajectory of fighting for indigenous women’s rights. ACIN has a long history working for women’s rights, starting in 2007 with a Family and Women Programme, aimed at making women aware of their general situation, promoting their organisation and training in order to consolidate their ethnic identity and their political participation as agents of transformation. In 2003, OIA started working by creating awareness on gender inequality in Antioquian communities, developing the Gender, Generation and Family

Programme (which was responsible for managing the evaluated project). This programme is being overseen by 165 local councils and its objective is to achieve equality, helping women run their own government through statute regulation of women political participation organisation in the councils, boards, and offices. Finally, ONIC has a Women’s Office, which is where the evaluated intervention is carried out.

Three indigenous organisations developed a project with the objective that women be recognised as development agents in the context of the Colombian armed conflict

gender-based violence against women in femicide cases, not only external but also accounting for strong sexist patterns found within the indigenous movement. They are indigenous, they are poor, and they are women – an ethnic, socioeconomic and gender triad of vulnerabilities.

In this way, the process of the struggle of indigenous women in Colombia must pay attention to three

different fronts, for which they have been generating different alliances and communications. One of the organisations that has accompanied their processes is a development NGO called MUNDUBAT, originally centred on the Colombian indigenous movement, and more recently focused on indigenous women’s organisations. Its presence facilitated a response to a European Union request for proposals (RFP) focused on human rights. A key element aimed to include different-level organisations who embrace diversity of ethnicities and indigenous cultures, among them aboriginal cultures such as the Emberá-Chamí, Emberá-Katío, Dóbida, Tule, Zenú and Nasa. The National Indigenous Organisation of Colombia (Organización Nacional Indígena de Colombia – ONIC), is a national group comprised of 43 regional and local branches, which include local authorities and associated cabildos

The process of the struggle of indigenous women in Colombia must pay attention to three different fronts, for which they have been generating different alliances and communications

76

The three indigenous organisations mentioned above participated actively in the intervention together with the NGO called MUNDUBAT. Between 2013 and 2014, they developed a project funded by the European Union, with the objective of recognizing indigenous women as development agents within the context of the Colombian armed conflict. In this way, their actions were oriented towards strengthening their abilities to set up and influence the implementation of local, regional, and national public policies. The project had three components: (a) Education, aimed at training indigenous women on their rights, so that they cannot only influence in their local communities but also influence the regional and national levels; (b) Impact, which aimed at lobbying the women’s rights arena; (c) Awareness, with the purpose of making indigenous women’s work known outside their organisations.

A year after the project began – and in line with the sponsor’s (European Union) guidelines, a mid-term evaluation was required to make adjustments to the project’s implementation and boost its efficiency. The design of the evaluation process was agreed with the organisations involved. As part of this process, MUNDUBAT, playing the role of evaluation manager, coordinated meetings with the three indigenous organisations linked to the project to review and give feedback on the methodology of the evaluation and the results as they were being generated. The evaluation team was led by Jenny Luz Mayta Navarro, a young Spanish evaluator with experience working in the Latin American region. This is how she remembers this experience:

It was our first time working as a team which consisted of two people, a local

An Evaluation: An Ally


As a women’s organisation, we hadn’t had an evaluation, it was our first time. At first, the leaders, and ourselves, were scared because we didn’t know what it was going to be like. We didn’t say to the grassroots women that it was an evaluation because when they hear those words, they tend to get scared. We told them that they were invited to a meeting, that’s all. We held the first meeting in a context different than the territory, but we held the rest here in the territory. We told them that we were going to meet and that a lady was just going to ask some questions because if we had told them that they were coming to do an evaluation, they surely wouldn’t have participated.

Some of the participants remembered a monitoring experience carried out a couple of years ago by an English expert who visited the area for four days, held meetings with them and made a battery of very specific

questions oriented towards the financed project under consideration. Once the field visit was finished, he kindly said goodbye to the interviewed candidates and they never heard from him again or saw his report. This first experience was considered invasive, which provided additional motivation for the evaluation team to include a fun and engaging participatory aspect to break away from the feeling that they were extracting information.

Moreover, a very important point that the participants brought to the evaluators’ attention was that in the previous evaluation, as well as other preceding monitoring and evaluation experiences, the external actors —whether they were donors or evaluators— who had approached them did not understood the following concepts, as expressed in the participants owns words: ‘We are

person who worked on human rights in Colombia and knew of the indigenous women’s movement and me. We had had three meetings through Skype with the leaders —organisation coordinators— in preparation for the fieldwork stage. There was previous knowledge of the methodology and some guidelines had been given regarding the field work and all that was going to carried out. So, the field work consisted first of having an assembly, to which the representative leaders were summoned, for which we prepared a PowerPoint presentation they could see in a friendly and fun way —given that many of them were illiterate— so that they could get to know us through a motivating and dynamic engagement process. We applied a qualitative methodology with participatory workshops in each section where they were intervening. We organised focus groups, some in depth-interviews and also some semi-structured interviews with the indigenous leader authorities.

For all the organisations involved in the initiative, the mid-term evaluation was generally a new experience for them. In past, some of the organisations had been through auditing and control on handling received donations, but this time the evaluation entailed field visits and meetings with different indigenous grassroots women. Mónica Yalanda Chilo, member of ACIN, shared her impressions of the evaluation:

We applied a qualitative methodology with participatory workshops in each section. We organised focus groups, some in depth-interviews and also some semi-structured interviews with the indigenous leader authorities

81Indigenous Women, Territoriality and Evaluation.The Story of a (de)Constructed Road (Colombia)

not a project; we are a process’. They suggested that this idea has always been very clear for the indigenous women organisations from Colombia: they are going through a process and one or more projects contribute to this process, but the heart belongs to the struggle process that defines them.

Amelicia Santacruz, member of OIA, Indigenous Organisation of Antioquia, remembers and strongly appreciates the evaluators’ intentions by visiting women directly in their territory:

Jenny, as an evaluator, wanted to go to the communities. She visited the communities so that the project was more transparent. Many things can be said from the office or the desk, but she wanted to get in, observe and listen directly to the voices of indigenous women.

The previous working experience of the evaluators with the women’s organisation of the region favoured these in-depth dialogues with the project participants. As one of the participants said:

It was mostly a dialogue, a conversation between women. It was an enjoyable moment that allowed us to chat just as we indigenous women like to do, telling each other things. It wasn’t like an evaluation of someone who asks and then writes; this was a conversation. She followed her methodology, her questions, she had prepared it very well, but she made us feel that we could trust

"We are not a project; we are a process"The indigenous women organisations from Colombia are in a struggle process to which they contribute with one or more projects


ritual and they would guide it. They have their own protocols and these formalities are very common for them. We all decided to include these elements because it was a way of giving the evaluation a context. We also set up commissions so that they could participate, one of them dedicated to environment, another to materials, another group wrote an account of what happened every day and the leader in charge coordinated. This wasn’t planned, it came up from the meetings and I think it was an element of success. We reduced our action considerably to basically becoming facilitators.

The different accounts repeatedly highlighted the notion of trust that existed between the group of women and the evaluator. For example, Amelicia shared the following with us:

If there hadn’t been any trust, which allowed the women to speak and

express what they felt, the evaluation wouldn’t have had the effects that it did. Grassroots women placed a lot of trust in Jenny, they see her as an ally, as one of the members of the group, she has kept directly in touch with us. And I think that this is good because one tries to be able to give their best and what this type of evaluations allow is to be better, because it helps to know what is good and be able to make it excellent and turn the bad things into good. And I believe that they were very clear, very straightforward and said everything they had to say.

Recommendations for improvement were made to the intervention and a reflection about them was carried out with the organisations

her to be able to talk, which is one the biggest difficulties indigenous women have, talking. But as I was saying, it was a space that generated trust and allowed us to confide in her about what is happening to us and how the project has helped us.

The fact that indigenous grassroots women took to the floor is an explicit empowering purpose of the different organisational stages involved in the project. Overcoming centuries of silence is a hard and long task and it should be placed in the context of an educational and empowering process

just like the women’s organisations involved in the initiative encouraged them to do. The evaluators were able to articulate this evaluation to this dynamic, getting closer, in this way, to the reality of the organisations involved in this intervention.

The different sessions were carried out with lively dynamics that promoted dialogue and reflection over the actions accomplished in the context of the project. In turn, towards the end of each session, a short evaluation process was carried out, highlighting satisfactory aspects of the day as well as others that could be adjusted for the next meeting. Jenny Luz mentions some characteristics of the evaluation approach:

We included their rituals because they perform a ritual in every meeting. Then we decided that in each of the workshops on the different days we would perform a

Overcoming centuries of silence is a hard and long task and it should be placed in the context of an educational and empowering process


Once the field work was finished, a meeting for reporting results was held and a timeline was made marking the main project landmarks and identifying its strengths and weaknesses. This is how recommendations for improvement were made in regards to the intervention and a reflection about them was carried out with the organisations, defining improvement plans to boost the intervention’s efficiency during the rest of the implementation of the project.

The mid-term evaluation described here was carried out after the first year of the initiative development and shortly before the elaboration and submission of a second project that would give continuity to the one evaluated here. In this sense, the contributions presented by the evaluation for the different actors

involved in the proposal indicated that the project objectives were being accomplished; moreover, at the same time it was possible to identify some weaknesses in that process, which the next project proposed to overcome.

The evaluation made specific contributions to the MUNDUBAT training model. These contributions were very welcomed, and entailed the adoption of a new educational strategy, planning with an educational

logic process, with materials designed for indigenous women. This made an impact not only in MUNDUBAT, but also in the indigenous women organisations, who in the same evaluation process deepened and understood concepts within the context of their own projects, such as what is the purpose of indicators, how to build a logic model framework, etc. In this way, the evaluation itself became an instance of capacity training in project managing. The evaluator made some comments about those aspects:

We found that there weren’t any training plans. The training courses were disorganised; they were conducted but they weren’t generating an effect on indigenous women. We made a clear recommendation about making a training plan adapted to each ethnic group, taking into consideration their specific cultural elements. They included this in the second stage which has

already started, and they are

Date post:	19-Oct-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Stories of Evaluations that Made a Difference tprint a Footprint Stori… · Leaving a Footprint:...

Documents