Focused Program Evaluation

Randolph R. “Randy”Weigel

&

Susan James

EPIC – 2004Lusk, WY

September 21

“Your tax dollars support this learning experience either directly or indirectly. Do you want your tax dollar to continue supporting this type of program?”1. Yes2. No

Robert J. Fetsch, Extension Spec.Colorado State University

Definition Of Evaluation

“To evaluate is to make an explicit judgment aboutthe worth of all or part of a program by collectingevidence to determine if acceptable standardshave been met.”’

Andrews & Warner (1988)

Types of Evaluation

•Process (Formative)

•Impact (Summative)

When should we conduct an evaluation?

Before?

During?

After?

When should we conduct an evaluation?

1. Project design2. Start up3. In-progress4. End of program5. Follow-up

When should we NOT conduct an evaluation?

• No one cares• Not going to change the program• One time program• No time, $, resources or energy• No goals or objectives• To satisfy your “performance”

Evaluability Assessment

What are you going to evaluate?

What is the purpose of the evaluation?

Who will use the evaluation

What resources do you have and need?

What questions will the evaluation hope to answer?

Knowledge Item

How important is nitrogen fertilizer management formaintaining our water quality?

*************

Farmers’ proper management of nitrogen fertilizerapplications is essential to maintain good waterquality.

TRUE FALSE DON’T KNOW

Attitude Item

How important is nitrogen fertilizer management formaintaining our water quality?

*************

Farmer’s proper management of nitrogen fertilizerapplications is essential to maintain good waterquality.

Strongly Stronglyagree Agree Disagree disagree

Experience Item

How important is nitrogen fertilizer management formaintaining our water quality?*************

Do you follow recommendations for applyingnitrogen fertilizer when the recommendations arebased on deep soil sampling?1. No2. Yes—How much do you apply

1. About 20% less than the amount recommended2. About 10% less than the amount recommended3. The exact amount recommended4. About 10% more than the amount recommended5. About 20% more than the amount recommended

Feeling Item

How important is nitrogen fertilizer management formaintaining our water quality?

*************

How do you feel when others imply that farmersmust better manage nitrogen fertilizer applicationsto maintain good water quality?

Extremely Extremely angry ___ ___ ___ ___ ___ calm

Collecting the Data:What sources will you use?

•Existing Information

•People

•Observations

•Pictorial Records

What data collection method(s) to use?

• Which method will give you the information you need?• Which method is appropriate for participants?• Which method is least disruptive?• Which method can you afford and manage?

Common Types of Data Collection

Surveys Interviews

Observation Case Study

Group Techniques Tests

Photos

Common Types of Data Collection(cont.)

Document Review Testimonials

Expert Panel Simulations

Journals Unobtrusive measure

“Triangulation”

Using the Information

How will the data be analyzed?

How will the information be interpreted?

Who will receive the evaluation?

How will the information be disseminated?

Avoid These Mistakes in

Your Program Evaluation

http://uwacadweb.uwyo.edu/Research/

Did today’s in-service provide you with informationto review and if necessary improve your programevaluation skills?

1. Yes2. No

Do you believe LISA research should or shouldnot be funded by CSREES/USDA?

1. Should2. Should not3. Undecided or unsure

Please interpret each of these words below as apercentage. Example: Average = 50%

Frequently

Occasionally

Seldom

Sometimes

How much profit did your business earn in2003?

_____________ Dollars

http://www.extension.psu.edu/evaluation/

http://www.uwex.edu/ces/pdande/evaluation/index.html

http://cyfernet.ces.ncsu.edu/cyfres/browse_2.php?search=Evaluation

http://cru23.cahe.wsu.edu/lifeskills/

http://www.four-h.purdue.edu/fourfold/main.htm

http://www.sagepub.com/

http://www.uwyo.edu/CES/LIFE/Program_Planning_Main.html