The 2010 User-Friendly Handbook for

Project Evaluation by Joy Frechtling Westat

Chapters 1 and 2Summary by Farran Coker, Tracy Crosby,

Savannah Baker, and Manda Cawthon

Follow the link to view The 2010 User-Friendly

Handbook for Project Evaluation

https://www.purdue.edu/research/docs/pdf/2010NSFuser-friendlyhandbookforprojectevaluation.pdf

Table of Contents

•Chapter 1: Reasons for Conducting Evaluations

•Chapter 2: Evaluation Prototypes

Chapter 1: Reasons for Conducting Evaluations

What is an evaluation?

• A comprehensive definition, as presented by the Joint Committee onStandards for Educational Evaluation (1994), holds that evaluation is “systematic investigation of the worth or merit of an object.”

• This idea of evaluations revolves around the idea of using evaluations for apurpose meaning that once an evaluation is conducted, action must be taken based on the results.

Why should evaluations be conducted?

• Evaluations allow for improvement. It is essential to understand what

objectives are being met and what objectives are not in order to make decisions to facilitate improvement.

• Evaluations can always provide information on new insights that were

unanticipated. These insights can help drive future improvements.

What is the best way to conduct an evaluation?

• Evaluations should not be conducted in an adversarial mode, but instead

conducted in a manner that will provide insight for improvements.

Chapter 2: Evaluation Prototypes

Evaluation: Includes different models and methods of data collection within the lifespan of a project

Two basic types of evaluation:

1) Program evaluation2) Project evaluation

What is the difference between a program and a project?

Programs and Projects

• Program: a coordinated approach to exploring a specific area related to the NSF’s mission of strengthening science, mathematics, and technology.

• Project: a particular investigation or developmental activity funded by that program. (NSF funds discrete projects to test the effectiveness of a program based upon student need and performance.)

Program Evaluation

• The evaluation of a program determines the total value of a series of projects. • The process of evaluation includes the examination of the

viability of activities and strategies employed. • The collection of data may be summative or done throughout

the program while it is underway.• In any case, data is collected annually.

Project Evaluation• Project evaluation focuses on an individual project funded under

the umbrella of a specific program. The collected information is used to enhance the project as it develops. • The data is used to guide the project while it is underway. This

promotes the obtainment of the defined program goals and objectives of the project. • Examination of specific components could also be implemented.

To what extent have the goals been met? What is needed to make the project more successful?

The Different Kinds of Evaluation: FORMATIVE AND SUMMATIVE

EVALUATIONSFormative Assessment:

Purpose: provide information for project improvement. Begins during project development and continues

throughout the duration of the project. Two components: 1) Implementation Evaluation and 2)

Progress EvaluationSummative Assessment:

Purpose: to assess the quality and impact of a fully implemented project.

Formative Evaluation

Formative Evaluation: Implementation Evaluation

Purpose: to assess whether the project is being conducted as planned. • This assessing of the project is conducted several

times throughout the span of the grant or contract provided by the NSF. • The project must be operating to the proposed plan.

If not, then modifications may be necessary. • Also provides for documentation of the activities

carried out. Research based plans done in alignment with the program goals are paramount.

Formative Evaluation: Progress Evaluation

Purpose: to assess progress in meeting the project’s ultimate goals. • Internal and External (monitoring) is done to check

for proper procedures during the project. • Progress Evaluation involves the collection of data

to determine if the established benchmarks have been met and to identify unexpected deviations. The project is adjusted accordingly based on this information. • It is important early problems are corrected to

promote individual effectiveness.

Summative Evaluation  

Purpose: to assess a mature project's success in reaching its stated goals

• takes place after the project has been established and the time frame forchange has occurred• collects information about outcomes and related processes, strategies, andactivities that have led to them• needed for the decision making about the future of the intervention

• disseminate the intervention to other sites or agencies• continue funding• increase funding• continue on probationary status• modify and try again• discontinue

Unanticipated Outcomes

• findings that emerge during data collection or data analyses that were never anticipated when the study was first designed

Evaluation compliments but is different from other kinds of data collection activities

FastLane allows for basic data to be collected across all programs in a consistent and systematic fashion.

Types of Data Collection Activities

Project Descriptors and Statistics

Formative Evaluation & Summative Evaluation

• gather information to answer• descriptive information• in-depth data collection activities• support decision making• range in cost (depending on questions asked and project complexity)

Performance Factors/Indicators• performance indicator system: collection of statistics that can be used to

monitor the ongoing status of a program against a set of targets and metrics• play critical role in the GPRA and PART activities• provide information that can be measured against a set of goals and objectives• examine accomplishments of the projects in a program• unlikely to provide an explanation of why a project has succeeded or failed• focus on tangible results used to focus policymakers, educators, and the public

on1. key aspects of how an educational program is operating2. whether progress is being made3. where there are problems

Basic Research• descriptive information

• provide targeted in-depth exploration of issues

• broaden understanding

• explore conceptual models and alternative explanations for

• observed relationships

Using Evaluation Information

Why assess worth or merit?

• to make changes or improvements in the status quo• "evaluations should be judged by their utility and actual use"

Patton (2008)• create a context in which evaluation findings are actually

used for decision making and improvement• provide an ongoing source of information that can aid

decision making at various steps along the way

Discussion Question

What types of evaluations have you used or plan to use in your classroom? How have you used the results or how can the results of these evaluations be used? Provide an example of each type of evaluation, formative and summative.

References

Frechtling, Joy A., and Melvin M. Mark. The 2010 User-friendly Handbook of Project Evaluation. Arlington, VA: National Science Foundation, Directorate for Education and Human Resources, Division of Research and Learning in Formal and Informal Settings, 2010. Web.