1
What we think is what matters: the role of impact assessment (IA)
in the CGIAR change management
Javier M. Ekboir
2
Overview of the presentation
•Review of evaluation and IA approaches
• How can philosophy of science contribute to evaluation and IA?
•The nature of organizational change
•Evaluation and IA in the history of the CGIAR
• Evaluation and IA in the on-going change process
3
Evaluation and IA can be categorized from two perspectives: goals and methods
Goals: • summative or judgment-oriented to determine
the value of a program• formative or improvement-oriented to identify
steps to improve a program• developmental to facilitate learning during
implementation
4
Evaluation and IA can be categorized from two perspectives (2)
Methods: • Qualitative• Quantitative – rates of return and cost-benefit analysis – other approaches, especially related to how
to measure changes in indicators in non-experimental settings
• Combination of qualitative and quantitative
5
Which approaches and methods are appropriate depends on what important
stakeholders think is appropriate!
6
The role of data in learningArgentina’s amazing discovery
Every data set (experimental or not) is open to an infinite number of hypotheses (underdetermination thesis)
Let me show you the picture of a bird
Or is it a rabbit?
Data are collected and have a meaning within a “theoretical” framework (“theory-ladenness” of data)
Data are important because they provide information to build and “confirm” theories
7
Hypotheses cannot be tested (T, H, A1, A2 … An)
•Then, what do researchers do?
•They “build explanations”
•But there is no consensus on what a valid explanation is
• Scientists build narratives that reflect how they see the processes they study
8
The social nature of knowledge
The acceptance or rejection of a narrative depends on the consensus among the majority of scientists and other stakeholders about what is a good explanation
This does not mean that all knowledge is relative
Just that knowledge is determined by the interaction of data and social conventions
Consensus about climate change
9
Organizational change depends, among other factors, on:
The nature of leadership
The organization’s culture
The organization’s learning routines
Governance
In NGOs, also external consensus about what needs to be done
10
The nature of change in organizations
Stakeholders influence organizational change according to their beliefs, resources and connections
Similar minded stakeholders form informal coalitions
The more effective a coalition, the greater its influence on the change process
11
The nature of change in organizations (2)
Influence can be gained in two ways:
•commanding resources
• convincing other stakeholders by presenting a compelling argument about what change is needed and how to measure it
Evaluations do not bring organizational change by themselves, but influence the perception of what changes are needed
12
Why was impact assessment implemented in the CGIAR?
In the 1990s, CGIAR centers had to justify their existence, i.e., accountability was the most important motive for IA
For decades, economists had calculated rates of return to research, which was considered by some professionals a rigorous method to measure impacts
Some of these professionals occupied important positions among donors, in the science council and in the centers
Over time, other approaches for IA were developed by CGIAR and external scientists, but they remained at the fringes of the system
13
Why was impact assessment implemented in the CGIAR? (2)
In recent years, some donors and scientists realized that an alternative to the rates of return was needed
New approaches for evaluation, IA and organizational learning are being adopted by donors, CGIAR scientists, other organizations working on development, and important segments of the evaluation community
The issue today is what type of IA meets scientific standards and is useful for the donors and the CGIAR
14
IA and change management in the CGIAR
Evaluations influence the change management process by helping to create a consensus about the new role of the CGIAR and the changes needed
IA, as designed by the SC, did not induce change in the system
How the new approaches for evaluation can influence change in the CGIAR depends on the system’s ability to use the new information
This depends, among other factors, on the coalitions that influence the process
15
How can new evaluation approaches help change in the CGIAR?
Generating new frameworks and information to help the Consortium, similarly-minded donors and DGs to provide effective leadership for change
Synthesizing the new approaches for evaluation and IA developed by CGIAR scientists and providing a unifying framework
16
How can new evaluation approaches help change in the CGIAR? (2)
Developing new approaches for IA based on novel frameworks for the analysis of social processes
Building a new, compelling argument for reassessing the role of science (including the CGIAR) and IA in development by bringing in new ideas and challenging orthodoxies
17
On its own, IA will have little impact on the change process
To have an impact, it is necessary to work simultaneously along three lines of action:
• Give more visibility to the new methods for evaluation and IA developed by CGIAR and/or external scientists
• Develop a conceptual framework that unifies the new methods for evaluation and IA and provides a compelling argument for using evaluation as a tool for organizational learning
• Build a coalition of similarly-minded stakeholders
18
Thank you!