Post on 08-Jan-2018
description
transcript
YOUTH & FAMILIES AGRICULTURE HEALTH ECONOMY ENVIRONMENT ENERGY COMMUNITIES
Evaluation Based Program Development
Based on work by Lydia B. Blalock, Ph.D., Youth Development Specialist
New Brunswick, New Jersey
YOUTH & FAMILIES AGRICULTURE HEALTH ECONOMY ENVIRONMENT ENERGY COMMUNITIES
From Assessment to Impact: Day 1
Karen Barale, Pierce CountyRenee Overath, Kitsap County
Christy Price, Grant County
Introduction
• Why are you here?• When I hear Program Evaluation
I think . . .• Education Philosophy
– Experiential: I can’t teach anything, but I can provide opportunities to learn!
– There’s no such thing as a stupid question!
How We Got . . . Here
• Various backgrounds• Program Evaluation Philosophy
– We can’t develop or perform an evaluation if the program wasn’t built upon a strong foundation!
• “Train the trainer” Evaluation-Based Program Development
Process Objectives
If I:• Develop and present seminar on EBPD• Develop and distribute appropriate handouts• Engage participants in group activities• Provide participants an opportunity to
practice • Encourage questions
THEN
Outcome Objectives
• Program Development–Understand how to clearly define issue program will address.
–Use program theory to plan activities.
–Write clear outcome and process objectives.
–Use EBPD Model to plan programs.
Participants will:
Outcome Objectives
•Program Evaluation– Develop skills to evaluate programs.– Select appropriate evaluation
methods and designs.– Select criteria to evaluate objectives.– Use EBPD Model to plan evaluation.
Participants will:
Outcome Objectives
• Scholarship of Engagement– Understand scholarship opportunities in
EBPD process. – Identify several avenues to share work. – Use EBPD to plan scholarly activities.
• Cultural Aspects of EBPD– Understand influence of culture on EBPD
(personal & audience).– Understand importance of learning culture of
target audience/s.– Select evaluation methods appropriate for
diverse audiences.
Participants will:
Commit to putting new knowledge and skills into practice!
BUT FIRST: Is it a Program or a program
program/s
event/s
activity/ ies
PROGRAM
PROGRAM
program/s event/sactivity/ ies
•Stakeholder leadership•Faculty leadership•Multi-disciplinary•Sustainable•Academic department •Student component•Research, teaching, outreach
PROGRAM
program/s event/sactivity/ ies
Group Assignments
• Form groups of 4• Sit with new group• Select bag• Develop program based upon
contents of bag• Develop 10-15 minute (tops)
program presentation
Program Development Project: Requirements
• Define problem– What method/s did you use– Identify stakeholders/collaborators
• Develop outcome objectives– Target audience– Resources needs– Short, medium, long term objectives
Program Development Project: Requirements
• Plan program activities (process objectives)
• Develop evaluation plan– Success indicators– Research design and data collection
methods• Identify scholarship opportunities
5. Implementation6. Dissemination & Scholarship 4
Evaluation-Based Program Development: By the Numbers
1. Define the Problem4. Plan Program Evaluation
1. Define the Problem
4. Plan Program Evaluation
5. Implementation
6. Dissemination & Scholarship 4
3. Plan ProgramÑ
2. Develop Outcome Objectives Ï
3. Plan ProgramÑ2. Develop Outcome Objectives Ï
Step 1. Define the Problem
• Describe Current Situation – How does the world look today?
• Why a Problem– Clearly articulate why a problem – Ask why until get to root of
problem– Symptom vs. Cause
Step 1. Define the Problem
• Identify Stakeholders– Population: Who is affected? How
many?– Stakeholder Leadership: interest in
solving problem?– Potential Collaborators: Already
working on problem?
Step 1. Define the Problem
• How: Evidence? Learn more? References & Facts– Learn all you can about the
issue/problem – • “prove” the problem exists• Identify potential solutions
• “Needs” Assessment – Need: difference between actual and
target “state”– A.K.A.: Asset, Gap, etc.
Small Group Activity:Needs Assessment• Purpose of “needs” assessment is to
clearly define or clarify issue.• What kinds of information might you need?• Who or what can provide the information?• What methods could you use to gather
information?• Where can you gather or find the
information?
Needs Assessments
Needs Assessments
• How will we gather the information?– Surveys – Interviews– Focus groups– Card sort– Archival/secondary data
• Criteria for selecting methods– Consider characteristics of target and
respondent groups– Choose combo of methods to gain
different types of info• When
– Develop target dates and timelines
Using Results
• Set Priorities– Factors that influence priorities
• Difference between actual and target states
• Degree of difficulty in addressing needs• Cost of implementing solutions• Political and other factors
– Derive priorities from several data source
– Priorities without resources
Using Results
• Select Solution Strategies to Meet Needs– Consider alternative solutions for
each need– Evaluate alternatives – Select solutions to implement– Scholarship Opportunities?
Table 1. Define the ProblemIdentify who, what, where, when, why.
1. Describe Current Situation How does the world look today?
2. Why a Problem? 3. Identify Stakeholders
Clearly articulate why current situation is a problem. Ask why until you get to root of problem.
Population. Who is affected by problem? How many people are affected?
Stakeholder Leadership. Who’s interested in solving problem?
Potential Collaborators. Who’s interested in solving problem and/or already working on problem?
What is the evidence? How can I learn more? Provide references and facts.
15 Minute Break
Step 2. Develop Outcome Objectives
• Who do you want to change? • What do you want to accomplish? • Do you have resources to
accomplish changes?
Step 2. Develop Outcome Objectives
• Describe Ideal Situation– How would world look w/o
problem (perfect world)?• Define Target Audience/s
– Who/what can we change (youth, families, policies, etc.)?
– What do we know about culture/s of audience?
Step 2. Develop Outcome Objectives
• Identify Outcome Objectives– What changes can I influence
in my target audience/s?• Identify Resources
– What do we need?– What do we have?
Step 2. Develop Outcome Objectives
2. Develop Outcome ObjectivesÏWhat can we change in audience?
Define Target Audience
Who can we change to solve problem?
Identify ResourcesWhat resources are required, available,
needed?
Short Term LEARNING& Awareness, Skills, Attitudes, Opinions,
Knowledge Aspirations, Motivations
Medium Term ACTIONS·Behavior, Practice, Decision-making,
Policies, Social Action
Long Term CONDITIONS þSocial, Economic,
Civil, EnvironmentalIMPACT?
Writing Objectives
• Specific– Will increase knowledge vs. Will know
3 sources of Vitamin C• Time delineated
– At the end of the program, in 6 weeks, etc.
• Measurable– 75%, all, etc.
Writing Objectives
• Example– Participants will increase knowledge
of nutrition.– 75% of program participants will
know 3 sources of Vitamin C 6 weeks after the program.
Table 2. I dentify Outcome Objectives
1. Describe I deal Situation How would the world look if the problem did not exist (the perfect world)?
3. I dentify Outcome Objectives What changes can I influence in my target audience/s?
2. Define Target Audience/ s Who or what can we realistically change (youth, families, farmers, policies, etc.)? What do we know about the culture/s of the audience? Do we need to learn more?
Learning Objectives
Short term changes in awareness, skills,
knowledge, attitudes,
aspirations, motivation. List for
each target.
Action Objectives
Medium term changes in
behavior, policies, practice, decision-
making, social actions. List for
each target.
Condition Objectives
Long term changes in social, civic, economic, environmental
conditions. Impact?
Target 1:
Target 2:
4. Identify Resources Resources Required Revenue Enhancement Development
Opportunities
Step 3. Plan Program
Plan ProgramÑ(Activities)
Develop Process ObjectivesÑWhat will you do to facilitate changes in
target audience/s?
Use “Program Theory” to develop process objectives!
Program Theory: The Key to Unlocking Change
IF we
do this . . .
THEN they
will change . . .IF:THEN Statements
IF we ABC: THEN XYZ
where ABC = the events, programs, activities you will conduct, deliver, develop, train, provide, access, facilitate, etc.
and whereXYZ = participant outcome objectives previously developed
ABC will become your process objectives.
Process Objectives
The actions you will take to produce the desired
outcomes (write curriculum, conduct
training)
Outcome Objectives
Changes in target population as result of
your actions(teachers adopt
curriculum, kids are respectful)
PROCESS:OUTCOME Statements
Delivery Modes
Projects and ActivitiesResultsPRODUCE
Small Group Activity:Process Outcome Challenge
1. Outcome Objectives Insert Outcome Objectives from Table 2.
2. Process Objectives Develop Program Theory (IF:THEN) If we = Conduct, deliver,
develop, train, provide, etc. Then = Outcome
objectives from Table 2
Learning Objectives
Short term changes in awareness, skills,
knowledge, attitudes, aspirations, motivation.
List for each target.
Action Objectives Medium term changes in behavior, policies, practice, decision-
making, social actions. List for each target.
Condition Objectives
Long term changes in social, civic, economic,
environmental conditions.
Impact? If we (insert activities ABC on lines below)
Then (insert Learning Objectives)
Then (insert Action Objectives)
Then (insert Condition Objectives)
3. Plan Program Outputs Deliverables
Program, Curriculum, Emergency Kits Scholarship Go to Table 6!
Template: Process Objectives
WORKING LUNCH1.Define the problem2.Outcome objectives3.Process objectives (program
activities)
Step 4. Plan Program Evaluation
4. Plan Program Evaluation
Specify Process Indicators ÑHow will you know if you
successfully implemented program?
Specify Outcome Indicators ÏHow will you know if the audience achieved the specified outcomes?
Why Evaluate?
• I dunno• I think I knew why–once• They make me do it
– Department chair– Funding organization– My mom
• I don’t
Why Evaluate?
• Demonstrate success• Understand failure• Provide feedback for program
improvement– Process– Outcome
• Nip problems in the bud
Why We Don’t Evaluate
• Too hard• Not qualified• Costs too much• Takes too much time• Don’t think about it “in time”• No one ever reads evaluation
reports• Everyone “knows” it’s a good
program
Small Group Activity: Setting Criteria
• Establish criteria– What are criteria (benchmarks,
indicators) for judging your item?• Construct standards
– How well should your item perform on benchmarks (standards)?
• Develop criteria rating system
Small Group Activity:
• AFTER developing criteria: measure performance and compare with standards.
• How well does each type of item measure up against criteria?
Template: Develop Success Indicators
1. Develop Success Indicators NOTE: May have multiple indicators for each objective.We will know (insert Learning Objectives) If (insert criteria for success)
The chocolate chip cookie meets our standard of excellence for aroma
It receives a 3.5 average score for “aroma” from our test panel.
Process Indicators
• How will you know you’ve successfully accomplished your process objectives?
Step 5. Implement Program and Conduct Evaluation
• You planned your work.– Now is the time to work the
plan.• If all goes well – great!• If problems arise – check plan,
revise as needed
15 Minute Break
Cultural Aspects of EBPD
• What is culture?– “Behaviors and beliefs characteristic
of a specific social, ethnic, or age group.”
– Product of our culture– Culture influences (unconsciously)
what we think of people from other cultures.
– Recognize cultural biases towards participants and strive to eliminate them.
Cultural Aspects of EBPD
• Neutralize cultural biases and prevent embarrassing (or worse) EBPD design flaws is:– Know your audience (Step 2 – Develop
Outcome Objects)• Be aware of how differences between
cultures may impact EBPD– Develop cross-cultural efficacy!
Cross Cultural Efficacy
Cultural Competence vs. Cross-cultural Efficacy
Cultural competence I can deal with them.
My culture is superior to their culture. (It’s all about me!)
Cross-cultural efficacy
I can effectively interact with people from other cultures.
I can benefit from our interactions. Neither culture is superior to the other. (It’s all about us!)
Small Group Activity:Cultural Influences
Step 6. Dissemination of Results -Extension Scholarship
1. Define the Problem
2. Develop Outcome Objectives Ï
3. Plan ProgramÑ
4. Plan Program Evaluation
5. Implementation
6. Dissemination & Scholarship 4
Step 6. Dissemination of Results - Extension Scholarship
• Tenure track faculty engage in appropriate scholarly activities
• Scholarship – Development and implementation of
creative concepts . . . – . . . broadly communicated and
validated by appropriate peers
Step 6. Dissemination of Results - Extension Scholarship
• Forms– Publications
• peer-reviewed journals, books, proceedings, abstracts
• approved extension publications– Peer-validated presentations – Creation of other forms of durable
scholarly products • includes digital and electronic media• that have been scrutinized by peers and
widely communicated. • Scholarship not simply act of doing
creative work . . .• . . . must be peer validated and
communicated
Examples of Outreach Scholarship
• PowerPoint Presentation• Original Handouts and Templates• Professional presentation
(abstract, talks, papers) • Participant Workbook• Curriculum• Train the Trainer Curriculum
Impact Reporting: A Core Competency
• Evaluate & clearly document programmatic outcomes & impacts
• Short, intermediate, long-term• http://ext.wsu.edu/pd/
documents/GuidelinesforDocumentingOutcomesFinal.pdf
Table 6. Dissemination and Scholarship How will you share your work?
Evaluation-Based Program Development
1. Identify Scholarship Activities
2. Identify Products & Deliverables
Table 1. Define Problem
Table 2. Identify Outcome Objectives and Resources
Table 3. Plan Program
Table 4. Plan Program Evaluation
Table 5. Implement Program, Conduct Evaluation and Document Impact
Table 6. Dissemination and Scholarship
Report Results
Demonstrate Impact on Profession
Attain Recognition
Small Group Activity: Extension Scholarship
Day 1: It’s Over!!!
Day 2 Overview• 8:30 Sign-in, Continental Breakfast • 9:00 Review, Questions• 9:15 Part I Program Evaluation• 10:15 Break• 10:30 Part II Program Evaluation• 11:15 Program Development Template• 12 Working Lunch• 1:00 Questions• 1:30 Group Projects • 3:35 Day 2 Post-Assessment, Day 3 Overview
Thank You! Have a Good Evening
YOUTH & FAMILIES AGRICULTURE HEALTH ECONOMY ENVIRONMENT ENERGY COMMUNITIES
Evaluation-Based Program Development
Day 2
Welcome
• Evaluation Models, Approaches and Design
• Program Template• Group Projects
Step 4. Plan Program Evaluation
4. Plan Program Evaluation
Specify Process Indicators ÑHow will you know if you
successfully implemented program?
Specify Outcome Indicators ÏHow will you know if the audience achieved the specified outcomes?
Select Evaluation Design Select Data Collection Method
Evaluation Design 101
• One-Shot– Data collected following program,
• Retrospective Pretest– Data collected following program; participants
recall behavior prior to and following program. • One Group Pretest-Posttest
– Data collected prior to and following program
Evaluation Design 101
• Time Series– Data collected prior to, during, after program
• Pretest-Posttest Control-Group– Data collected from 2 different groups prior to
and following program.– Treatment group: program– Control group: no program
• Post-Only Control-Group– Data from 2 different groups collected following
program only
Evaluation Design 101
Evaluation Design Pretest Posttest # Times Assessed Follow-up Control
Group
One-Shot Design X 1
Retrospective Pretest X X 1
One Group Pretest-Posttest X X 2
Time Series X X 3 (min) X
Pretest-Posttest Control-Group X X 2x per group X
Post-Only Control-Group X 1x per group X
Data Collection Methods
• Face to Face Surveys– Respondent completes survey in
presence of evaluator– Evaluator may record responses and is
able to answer questions• Self Administered Questionnaires
– Respondent given survey to complete on his/her own.
– Respondent’s responsibility to return survey.
Data Collection Methods
• Personal Interviews– Respondent asked pre-determined
questions. – Evaluator records responses, can ask
respondent to clarify answers.• Focus Groups
– Several respondents meet with evaluator.
– Evaluator facilitates discussion around pre-determined questions.
– Discussion recorded.
Small Group Activity: Evaluation Designs and Methods
Data Collection Template
Learning Objectives Short term changes in awareness, knowledge, skills, attitudes, aspirations, motivation
1. Develop Success Indicators NOTE: May have multiple indicators for each objective.
2. Select Evaluation Design*
3. Select Data Collection Method*
We will know (insert Learning Objectives)
I f (insert criteria for success) Type Pro/Cons Method Pros/Cons
5. Implementation6. Dissemination & Scholarship 4
Evaluation-Based Program Development: By the Numbers
1. Define the Problem4. Plan Program Evaluation
1. Define the Problem
4. Plan Program Evaluation
5. Implementation
6. Dissemination & Scholarship 4
3. Plan ProgramÑ
2. Develop Outcome Objectives Ï
3. Plan ProgramÑ2. Develop Outcome Objectives Ï
WORKING LUNCHPlease return by 1:30
5. Implementation(Program & Evaluation)
1. Define the ProblemWhat is issue or problem? Why* is this a problem?
Who is affected by or interested in solving it?
4. Plan Program Evaluation
Specify Process Indicators ÑHow will you know if you
successfully implemented program?
Specify Outcome Indicators ÏHow will you know if the audience achieved the specified outcomes?Develop Process ObjectivesÑ
What will you do to facilitate changes in target audience/s?
3. Plan ProgramÑ(Activities)
1. Define the ProblemWhat is issue or problem? Why* is this a problem?
Who is affected by or interested in solving it?
IF:THENIF we (conduct, provide, train, etc.): THEN we can expect these outcomes
5. Implementation(Program & Evaluation)
6. Dissemination & Scholarship 4
Develop Process ObjectivesÑWhat will you do to facilitate changes in
target audience/s?
3. Plan ProgramÑ(Activities)
Long Term CONDITIONS þSocial, Economic,
Civil, Environmental
Medium Term ACTIONS·Behavior, Practice, Decision-making,
Policies, Social Action
Identify ResourcesWhat resources are required, available,
needed?
Define Target Audience
Who can we change to solve problem?
Short Term LEARNING& Awareness, Skills, Attitudes, Opinions,
Knowledge Aspirations, Motivations
2. Develop Outcome ObjectivesÏWhat can we change in audience?
Specify Outcome Indicators ÏHow will you know if the audience achieved the specified outcomes?
4. Plan Program Evaluation
Specify Process Indicators ÑHow will you know if you
successfully implemented program?
6. Dissemination & Scholarship 4
EBPD: Full Model
Long Term CONDITIONS þSocial, Economic,
Civil, Environmental
Medium Term ACTIONS·Behavior, Practice, Decision-making,
Policies, Social Action
Identify ResourcesWhat resources are required, available,
needed?
Define Target Audience
Who can we change to solve problem?
Short Term LEARNING& Awareness, Skills, Attitudes, Opinions,
Knowledge Aspirations, Motivations
2. Develop Outcome ObjectivesÏWhat can we change in audience?
The Template
Day 3 Overview
• 8:00 – 9:30 Sign In, Working Breakfast (projects)
• 9:30 – 11:00 Presentations• 11:00 – 11:15 Debriefing• 11:15 – 12 :00 Lunch (non-
working!)• 1:00 – 3:00 Planning Session:
Discuss Training Strategies• 3:00 Travel Home Safely!
Small Group Activity:Develop Your Program
YOUTH & FAMILIES AGRICULTURE HEALTH ECONOMY ENVIRONMENT ENERGY COMMUNITIES
Evaluation-Based Program Development
Day 3
Group Presentations
Resources
• Extension Websites– www.cyfernet– www.sammie.osu.edu– http://www.uwex.edu/ces/pdande/evaluat
ion/index.html
• Journals– Journal of Extension– Youth Development Journal– American Evaluation Association
Resources
• Databases– Discipline Journals– PsycLit– OVID
• Evaluation Specialists • Faculty with assessment & evaluation
experience
Evaluation-Based Program Development: Process Objectives
If I, as facilitator:• Develop and present seminar on EBPD• Develop and distribute appropriate
handouts• Engage participants in group activities• Provide participants an opportunity to
practice • Encourage questions THEN
Outcome Objectives
• Program Development–Understand how to clearly define issue program will address.
–Use program theory to plan activities.
–Write clear outcome and process objectives.
–Use EBPD Model to plan programs.
Participants will:
Outcome Objectives
•Program Evaluation– Develop skills to evaluate programs.– Select appropriate evaluation
methods and designs.– Select criteria to evaluate objectives.– Use EBPD Model to plan evaluation.
Participants will:
Outcome Objectives
• Scholarship of Engagement– Understand scholarship opportunities in
EBPD process. – Identify several avenues to share work. – Use EBPD to plan scholarly activities.
• Cultural Aspects of EBPD– Understand influence of culture on EBPD
(personal & audience).– Understand importance of learning culture of
target audience/s.– Select evaluation methods appropriate for
diverse audiences.
Participants will:
Commit to putting new knowledge and skills into practice!
1. Define the ProblemWhat is issue or problem? Why* is this a problem?
Who is affected by or interested in solving it?
IF:THENIF we (conduct, provide, train, etc.): THEN we can expect these outcomes
5. Implementation(Program & Evaluation)
6. Dissemination & Scholarship 4
Develop Process ObjectivesÑWhat will you do to facilitate changes in
target audience/s?
3. Plan ProgramÑ(Activities)
Long Term CONDITIONS þSocial, Economic,
Civil, Environmental
Medium Term ACTIONS·Behavior, Practice, Decision-making,
Policies, Social Action
Identify ResourcesWhat resources are required, available,
needed?
Define Target Audience
Who can we change to solve problem?
Short Term LEARNING& Awareness, Skills, Attitudes, Opinions,
Knowledge Aspirations, Motivations
2. Develop Outcome ObjectivesÏWhat can we change in audience?
Specify Outcome Indicators ÏHow will you know if the audience achieved the specified outcomes?
4. Plan Program Evaluation
Specify Process Indicators ÑHow will you know if you
successfully implemented program?
1. Define the Problem
4. Plan Program Evaluation
5. Implementation
6. Dissemination & Scholarship 4
3. Plan ProgramÑ
2. Develop Outcome Objectives Ï
The Last Word: Begin with the End in Mind
If you don’t know where you’re going, how are you gonna’ know when you get there? Yogi Berra
Alice asked: Would you tell me please which way I ought to go from here? That depends a good deal on where you want to get to, said the Cat. I don’t much care where . . . said Alice. Then it doesn’t matter which way you go, said the Cat. From Alice’s Adventures in Wonderland
THE END
• Thank You!• Final Questions• Post-Assessment
YOUTH & FAMILIES AGRICULTURE HEALTH ECONOMY ENVIRONMENT ENERGY COMMUNITIES
The “Logic” Behind Evaluation Based Program Development
A Rose by Any Other Name . . .Can Still Have Thorns!
3. Plan ProgramÑ 2. Develop Outcome Objectives Ï2b.
ResourcesÏ2a. Target
Audience Ï
4. Plan Program Evaluation 5. Implementation 6. Dissemination & Scholarship 4
1. Define the Problem