Real-time evaluationIn theory and practice

Jessica Letch, University of MelbourneThesis: Master of Assessment and Evaluation

Overview

Real-time evaluation

Questions and rationale

Methodology

Logic and fidelity

RTE in theoretical context

Suggestions

Origins of RTE

Pivotal conflictsPersian Gulf Crisis, Rwanda, Kosovo

Early agenciesUNHCR, World Bank, Danida

Humanitarian reformOCHA, Reliefweb, IASC, ALNAP

Rationale

Humanitarian evaluation “tends to mirror humanitarian practice – it is often rushed, heavily dependent on the skills of its key protagonists, ignores local capacity, is top-down, and keeps an eye on the media and public relations implications of findings”

- Feinstein & Beck (2006)

Rationale“atheoretical and method-driven”– a less thoughtful and rigorous cousin of mainstream evaluation.

- Feinstein & Beck (2006)

The ‘wild west’ of evaluation

- AES conference Canberra, 2009

The centrality of theory

Without a strong theoretical base, “we are no different from the legions of others who also market themselves as evaluators today”

- Shadish (1998)

Aid evaluation emergentAt the time of the Rwanda evaluation, “there were no manuals, guidelines or good practice notes to follow on evaluating humanitarian action”

- Beck (2006)

The value of research on evaluationRigorous and systematic study “can provide essential information in the development of an evidence base for a theoretically rooted evaluation practice, as well as provide the evidentiary base for the development of practice-based theory”

- Miller (2010)

Research questions

1. What is the conceptual logic behind real-time evaluation?

2. How is real-time evaluation applied in practice?

3. How can the theory and practice of real-time evaluation be strengthened?

Methodology

Drawn from:Miller and Campbell (2006)

Multistage sampling approach; examination of fidelity between theory and practice

Hansen et al (in press)Logic modeling from coding framework

Methodology

Identify core texts

Code theory data

Summarise theory data

Develop theory logic

model

Establish fidelity

measures

Identify RTE

reports

Code report data

Measure fidelity

Summarise report data

Develop report logic

models

Compare data and

logic models

Compare to theory

Espoused theory

Six items of literatureBroughton, WFP (2001)Jamal and Crisp, UNHCR (2002)Sandison, UNICEF (2003)Cosgrave, Ramalingham & Beck (2009)Waldon, Scott & Lakeman (2010) Brusset, Cosgrave, MacDonald, ALNAP

(2010)

Logic of theory

Theory in use56 case examples

Logic of practice

• Theory: Concerns about programme performance

• Practice: Silent on these concerns

Impetus

• Theory: Agency knowledge• Practice: External evaluators

with sectoral expertise and diverse backgrounds

Evaluator

• Theory: Field based planning• Practice: Reference groupsPlanning

Contrasts in logic models

• Theory: Field and management response.

• Practice: More optimistic picture of beneficiary consultation.

Stakeholders

• Theory: Effective planning.• Practice: Relationships, transparency,

meta-evaluation.Credibility

• Theory: Establishing M&E systems.Organizational capacity

Contrasts in logic models

• Theory: Learning• Practice: Communication and

coordination.Process use

• Theory: Understanding at headquarters.

• Practice: Field team ownership and action plans.

Utilization

• Practice: Political environment.Constraints

Contrasts in logic models

• Theory: Modest expectations of organizational change.

• Practice: The importance of lessons learned.

Assumptions

• Practice has a stronger emphasis on bottom-up influence and approaches.

Overall

Contrasts in logic models

Fidelity

Change in scores pre- and post- ALNAP guide

Element Pre-March 2009 Post-March 2009 % change

Median no of beneficiaries 40 beneficiaries 136 beneficiaries 240%

Matrix of recommendations 9% 29% 222%

5 to 10 recommendations

16%average 24

33%average 15

106%

Inception report included 9% 17% 89%

List of informants 41% 63% 54%

Group interviews 50% 67% 34%

Workshop in field 59% 75% 27%

Change in scores pre- and post- ALNAP guide

Element Pre-March 2009 Post-March 2009 % change

Workshop in field 59% 75% 27%

Average fidelity score

45%6.25 out of 14

53%7.38 our of 14

18%

Beneficiary consultation

78% 88% 13%

1 to 4 evaluators 81% 88% 9%

7 to 21 days in field50%

median 13 days54%

median 16 days8%

Average fidelity score

45%6.25 out of 14

53%7.38 our of 14

18%

Beneficiary consultation

78% 88% 13%

Change in scores pre- and post- ALNAP guide

Element Pre-March 2009 Post-March 2009 % change

Describes methods 97% 92% -5%

Describes triangulation and validity

38% 33% -13%

Timeline 38% 33% -13%

Report 15 to 40 pages

59% average 30 pages

46%average 38 pages

-22%

Matrix of evidence 0% 21% N/A

   Median change (all scores) 18%

Highest fidelity scores

Found in Humanitarian Accountability Project, IASC and among external evaluators.

Many of these evaluators also contributed to the literature on RTE.

Lowest scores

Found in mixed and internal teams, multi-country evaluations.

Some reports seem to be labeled RTE simply for its cachet.

In theory

Is RTE new?Though described as innovative, it has

many antecedents outside the humanitarian field.

Is RTE evaluation at all?‘Purists’ would argue that it’s

pseudoevaluation. RTE is part of an increasingly vague distinction between evaluators and organizational development consultants.

Classifying RTE

2nd stage evaluationShadish, Cook and Leviton (1991)

“Use” branchAlkin & Christie (2004)

Utilization-focused evaluation

Type of use Potential for real-time evaluation

InstrumentalInfluence actions and decisions.Develop action plans.Change policies and programs.

ConceptualLessons learned for country teams, headquarters and donors.

SymbolicInformation for donors.Demonstrate transparency, accountability.

Process Communication, coordination and morale.

Developmental evaluationProgram in a continuous state of change; operations will never become fixed or stable.

Patton (2008)

Not to prove

…but to improve

Krueger & Sagmeister (2012), Stufflebeam (2004)

Connoisseurship

“There are no algorithms, rules, recipes or the like to use”

Eisner (2004)

Expert-led, lightweight and agile design.

Credibility (and supply) of experts a key limitation.

Stufflebeam & Shinkfield (2007), Miller (2010)

Summary

There is a strengthening relationship between theory and practice

A strong logic is emerging from RTE

RTE has roots in mainstream evaluation, especially developmental and utilization-focussed approaches. Must be wary of the risks of connoisseurship.

Suggestions

Humanitarian evaluators: stronger engagement with theory and better training in guidance.

Mainstream theorists: attention to the specificities of emergencies, to adapt traditional models.

Further research on evaluation in humanitarian programs.

Thank you

Jess LetchMasters candidateUniversity of Melbourne, Australiajessicaletch@gmail.com

Special thanks to supervisor Brad Astbury

Special acknowledgement to Ros Hurworth

References Alkin, M. C., & Christie, C. A. (2004). An evaluation theory tree. In M. C. Alkin (Ed.), Evaluation roots:

tracing theorists' views and influences (pp. 12-65). Thousand Oaks, Calif.: Sage Publications.

Beck, T. (2006). Evaluating humanitarian action using the OECD-DAC criteria: An ALNAP guide for humanitarian agencies: ALNAP.

Broughton, B. (2001). Proposal Outlining a Conceptual Framework and Terms of Reference for a Pilot Real-Time Evaluation (O. O. o. Evaluation), Trans.). Canberra: World Food Program.

Brusset, E., Cosgrave, J., & MacDonald, W. (2010). Real-time evaluation in humanitarian emergencies. [Article]. New Directions for Evaluation(126), 9-20.

Cosgrave, J., Ramalingham, B., & Beck, T. (2009). Real-Time Evaluations of Humanitarian Action: An ALNAP Guide (Pilot Version): ALNAP.

Eisner, E. (2004). The roots of connoisseurship and criticism: A personal journey. In M. C. Alkin (Ed.), Evaluation Roots: Tracing Theorists' Views and Influences (pp. 8p). Thousand Oaks, Calif.: Sage Publications.

Feinstein, O., & Beck, T. (2006). Evaluation of Development Interventions and Humanitarian Action. In I. F. Shaw, J. C. Greene & M. M. Mark (Eds.), The Sage Handbook of Evaluation. London: Sage.

Hansen, M., Alkin, M. C., & LeBaron Wallace, T. (in press). Depicting the logic of three evaluation theories. Evaluation and Program Planning. doi: 10.1016/j.evalprogplan.2012.03.012

Jamal, A., & Crisp, J. (2002). Real-Time Humanitarian Evaluations: Some Frequently Asked Questions (E. a. P. A. Unit, Trans.): UNHCR.

Krueger, S., & Sagmeister, E. (2012). Real-Time Evaluation of Humanitarian Assistance Revisited: Lessons Learned and the Way Forward. Paper presented at the European Evaluation Society, Helsinki.

Miller, R. L. (2010). Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31(390). doi: 10.1177/1098214010371819

References Miller, R. L., & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review.

American Journal of Evaluation, 27(3), 296-319.

Owen, J. M., & Rogers, P. J. (1999). Program Evaluation: Forms and approaches Retrieved from SAGE Research Methods database Retrieved from http://srmo.sagepub.com/view/program-evaluation/SAGE.xml doi:10.4135/9781849209601

Patton, M. Q. (2008). Utilization-Focused Evaluation. Thousand Oaks, California: Sage Publications.

Sandison, P. (2003). Desk Review of Real-Time Evaluation Experience. New York: UNICEF.

Shadish, W. R. (1998). Evaluation theory is who we are. American Journal of Evaluation, 19(1), 18.

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: theories of practice. Newbury Park, CA: Sage Publications.

Stake, R. E. (2004). Stake and Responsive Evaluation. In M. C. Alkin (Ed.), Evaluation Roots: Tracing Theorists' Views and Influences (pp. 204-216). Thousand Oaks, Calif.: Sage Publications. Retrieved from http://www.loc.gov/catdir/toc/ecip048/2003019866.html.

Stufflebeam, D. L. (2004). The 21st Century Cipp Model. In M. Alkin (Ed.), Evaluation Roots: Tracing Theorists' Views and Influences (pp. 245-266). Thousand Oaks, Calif.: Sage Publications.

Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications / Daniel L. Stufflebeam, Anthony J. Shinkfield: San Francisco : Jossey-Bass, c2007.

Walden, V. M., Scott, I., & Lakeman, J. (2010). Snapshots in time: using real-time evaluations in humanitarian emergencies. Disaster Prevention and Management, 19(3), 8.