+ All Categories
Home > Documents > Note to evaluator…

Note to evaluator…

Date post: 13-Mar-2016
Category:
Upload: megan-douglas
View: 37 times
Download: 0 times
Share this document with a friend
Description:
Note to evaluator…. The overall purpose of this presentation is to guide evaluators through the completion of steps 4 to 6 of the UFE checklist. The main goal is to suggest a process that can help UFE evaluators facilitate the formulation of key evaluation questions. - PowerPoint PPT Presentation
Popular Tags:
32
Note to evaluator… The overall purpose of this presentation is to guide evaluators through the completion of steps 4 to 6 of the UFE checklist. The main goal is to suggest a process that can help UFE evaluators facilitate the formulation of key evaluation questions. Please adapt this presentation to the context of the project that you are evaluating and to your
Transcript
Page 1: Note to evaluator…

Note to evaluator…

The overall purpose of this presentation is to guide evaluators through the completion of steps 4 to 6 of the UFE checklist. The main goal is to suggest a process that can help UFE evaluators facilitate the formulation of key evaluation questions.

Please adapt this presentation to the context of the project that you are evaluating and to your facilitation style.

Page 2: Note to evaluator…

Facilitating UFE step-by-step:a process guide for evaluators

Joaquín Navas & Ricardo RamírezDecember, 2009

Module 2: Steps 4-6 of UFE checklist

Page 3: Note to evaluator…

Meeting’s objectives

1. To review the most relevant topics of previous meeting(s) – comments on distributed report(s).

2. To formulate the first draft of the key evaluation questions.

Page 4: Note to evaluator…

Agenda

1. Summary and brief discussion on the previous meeting and/or report.

2. Group reflection on some of the discussed topics.

3. Break.

4. Formulation of key evaluation questions – first draft.

Page 5: Note to evaluator…

What have we achieved so far?

1. Common understanding among participants on basic UFE principles.

2. Stakeholder identification within the overall project.

3. Role definition within the evaluation process.

4. Identification of possible challenges of the evaluation process.

Page 6: Note to evaluator…

Review – Basic UFE principles

1. UFE is a PROCESS for helping primary intended users select the most appropriate content, model, methods, theory, and uses for their particular situation.

2. UFE is a COLLABORATIVE APPROACH that seeks to generate learning.

3. Evaluation should be JUDGED by its utility and USE in the real world.

4. Evaluation plan needs to be part of the INITIAL DESIGN of project.

5. The evaluator’s role is to COLLABORATE with those engaged in the design of the evaluation process.

Page 7: Note to evaluator…

In the previous meeting you identified x, y, z, as the most limiting resources or factors of the evaluation process. Have you identified any other limiting resources/factors since that meeting?

¿Do you still think that the available resources are enough (or not enough) to carry out the evaluation process?

Group reflection – Available resources…

Page 8: Note to evaluator…

From the stakeholder groups that you identified in the previous meeting(s), what group(s) do you think you represent as primary intended users of the evaluation?

Group reflection – Stakeholders (1/4)…

Page 9: Note to evaluator…

Identified stakeholders (from previous meetings)

Group Stakeholder Assigned role

Stakeholder Group 1 Stakeholder 1

Stakeholder 2

Stakeholder 3

Stakeholder Group 2 Stakeholder 4

Stakeholder 5

Stakeholder 6

Stakeholder Group 3 Stakeholder 7 Primary intended user

Stakeholder 8

Page 10: Note to evaluator…

What key stakeholder groups are not represented by the primary intended user?

Would there be any implications on the use of the evaluation as a result of not having these stakeholder groups represented by the primary intended users?

What would those implications be?

Group reflection – stakeholders (3/4)…

Page 11: Note to evaluator…

Can you think of any political factors within the project that could affect the use of the evaluation?

Group reflection – Stakeholders (4/4)…

Page 12: Note to evaluator…

In previous meeting(s) you made the following comments regarding your previous evaluation experiences: X, Y, Z. What did you learn from those experiences?

Group reflection – previous experience (1/3)

Page 13: Note to evaluator…

Suppose that at the beginning of this project you had the required resources and total freedom for implementing or not implementing a formal evaluation plan. What factors would have discouraged you about implementing the evaluation plan? Why?

Group reflection – previous experience (2/3)

Page 14: Note to evaluator…

What would have motivated you to implement the evaluation plan? Why?

Group reflection – previous experience (3/3)

Page 15: Note to evaluator…

Do you think that this evaluation process can contribute to program improvement? How?

Group reflection – evaluation utility (1/3)

Page 16: Note to evaluator…

Do you think that this evaluation process can contibute to making major decisions? How?

Reflexión – evaluation utility (2/3)…

Page 17: Note to evaluator…

Do you think that this evaluation process can contribute to the generation of knowledge? How?

Group reflection – evaluation utility (3/3)

Page 18: Note to evaluator…

What are your expectations regarding the completion date of this evaluation?

Can you think of any milestone dates that could be critical or useful to have in mind for major decision-making throughout the project subject of this evaluation?

Group reflection – critical dates…

Page 19: Note to evaluator…

BREAK

Page 20: Note to evaluator…

FORMULATION OF KEY EVALUATION QUESTIONS -

FIRST APPROACH

Page 21: Note to evaluator…

What is the purpose of the evaluation?

Page 22: Note to evaluator…

The trajectory of change…

INPUT / RESOURCES ▼

ACTIVITIES ▼

OUPUTS ▼

OUTCOMES ▼

IMPACT / RESULTS

CONTROL

&

PREDICTION?

Page 23: Note to evaluator…

According to Patton (2008) evaluation can be oriented towards different purposes based on the findings’ primary intended USES.

The evaluation’s purpose

Page 24: Note to evaluator…

MENU OF INTENDED USES (1/2)

Typical primary users

Knowledge generating

Program designers, planners, modelers, theorists, scholars, policy-makers.

Formative improvement & learning

Program administrators, staff, those involved in the day-to-day management.

Overall summative judgment

Those charged with making major decisions: funders, directors, other adopters of model, etc.

“To look across findings from different programs to identify patterns of effectiveness” (p. 131).

Primary intended uses

“To provide data for judging the overall value of a program and deciding whether it is worth continuing with it or not” (p. 114).

“To provide data for program improvement” (p. 116).

Evaluation Purposes

Page 25: Note to evaluator…

Monitoring Program managers responsible for internal accountability and information system management (Adaptaded from Patton p. 139 – Ch. .4).

MENU OF ITENDED USES (2/2)

Typical primary users

Accountability Those with administrative and funding authority, responsible for resource use.

Social innovators, those involved in bringing about major systems change in dynamic environments.

Organization / Program Development

“To provide data for adapting interventions to emergent conditions” (p. 137).

Intended primary usesPurposes

“To provide data for describing and explaining achievements” (p. 121).

“To provide information about key areas that require managerial attention” (Pg 126).

Page 26: Note to evaluator…

Formulation of key evaluation questions

Page 27: Note to evaluator…

• Specific enough to be useful in guiding you through the evaluation

• Broad enough to be broken down - are not the same as a question in a survey

• Data (qualitative/quantitative) can be brought to bear on the KEQ

• KEQs are open questions (can’t answer yes or no!)

• Have meaning for those developing the plan

• Lead to useful, credible, evaluation

• There aren’t too many of them (2-4 is enough).

What makes good KEQs? (adapted from Dart, 2007)

Page 28: Note to evaluator…

Categories of key evaluation questions

INPUT / RESOURCES

IMPACT

OUTCOMES

APPROACH / MODEL

PROCESS

QUALITY

COST- EFFECTIVENESS

Page 29: Note to evaluator…

Linking KEQ to the project objectives

List the specific project objectives here…

Page 30: Note to evaluator…

Conclusion and next steps

Page 31: Note to evaluator…

Next steps (for the evaluator only)

Page 32: Note to evaluator…

References

Dart, J. 2007. “Key evaluation questions”. Presentation at the Evaluation in Practice Workshop. Kualal Lumpur, December. http://evaluationinpractice.files.wordpress.com/2008/01/keyquestionschoices.pdf

Patton, M.Q. (2008) Utilization focused evaluation, 4th Edition. Sage.


Recommended