Focusing Evaluation

Getting ready

Focusing: Making Ready

• Stages of Evaluation• Evaluability

– Stakeholders– Standards

• Target Indicators– Outcome indicators– Process indicators

• Evaluation Management

Focusing: Making Ready for Process

• Stages of Evaluation: What’s ahead?– Planning process– Focusing: Identifying context, questions, targets– Designing study methods– Selecting study tools– Collecting data– Interpreting data– Using results

Focusing our own process

• Stages of Evaluation: Where are you?

How often do you thinking about what to evaluate after the program is over?

a) Pretty typical

b) Somewhat typical

c) Not typical

d) Never wait until after

e) What evaluation?

Focusing: Making Ready for People

• Stakeholders: Interests and Readiness– Sponsors (STEM careers; mentors from field)– Teachers (STEM achievement; collaboration)– Parents (STEM achievement; volunteerism)– Youth (STEM hands-on; participation)

Not whether it’s evaluation-ready,

but what evaluation it’s ready for.

Focusing: Making Ready for Program

• Standards: Investments and Readiness– Utility (credibility, relevance, collaboration)– Feasibility (management, viability)– Propriety (principled, facilitative)– Accuracy (sound and scientific)

Not whether it’s evaluation-ready,

but what evaluation it’s ready for.

Focusing: Ready Questions

• Evaluation Question:

What do we want to know about a program?

– Outcome questions: What does it do?

– Process questions: How does it work?

Focusing: Making Ready Indicators

• Target Indicators

How do we know a program is worth its salt?

– Outcomes: KASA (Short-term); BA (Long-term)

– Processes: Program Quality, Fidelity

Focusing: Making Ready Quality

• Target Indicators:

Quality First

– Program (Delivery) Processes• In Preparation: Credibility, Consistency• In Process: Engagement, Guided Inquiry

Focusing: Making Ready Outcomes

• Target Indicators: Now & Later

– Program Outcomes for Individuals• In Preparation: Credibility, Consistency• In Process: Content (subject matter concepts); and

Process (inquiry, creativity, problem solving)• In Prospect: Current activities that contribute to

future outcomes

Focusing our own goals

• Evaluation Targets: Where are you?

How often do you check program quality or fidelity before collecting data on outcomes?

a) Pretty typical

b) Somewhat typical

c) Not typical

d) No need to check

e) Fidelity to what?

Focusing: Making Ready Resources

• Evaluation Management

– Timeline– Budget– Human Capital/Deployment– Facilities/Optimization