Date post: | 04-Jan-2016 |
Category: |
Documents |
Upload: | peter-cummings |
View: | 220 times |
Download: | 0 times |
Program EvaluationProgram Evaluation
Planning Programs for Adult LearnersChapter 11: Formulating Evaluation PlansCafarella (2002)
Definitions in Program EvaluationDefinitions in Program Evaluation
Program Evaluation: a process used to determine whether the design and delivery of a program were effective and whether the proposed outcomes were met.
Formative Evaluation: Evaluation done to improve or change a program while it is in progress.
Summative Evaluation: Evaluation focuses on the results or outcomes of a program.
Formative vs. SummativeFormative vs. Summative
Formative Summative
Were changes occurring in knowledge and skill levels as they go through the program?
What context issues came up that might have changed the nature of the program?
Were changes evident from pre / post data?
How applicable would this program data be for other settings / contexts?
Goals of Program EvaluationGoals of Program Evaluation
Judging the value or worth of an education/ training program (cost/benefit analysis).
Demonstrate that program outcomes are ties to what happened in the program.
Develop clear criteria to make your judgments that can be changed as the program evolves.
Be aware of the audience for which the program data will be shared and the population from which it will collected. Each will have their own agendas.
Program met the original objective set forth by administration and/or participants.
Program did not have any unintended consequencesLearners are able to transfer the information obtained
to another setting.Usually specific to the program and not as interested
in generalizability.
Elements of a Systematic EvaluationElements of a Systematic Evaluation
Secure support for the evaluation from the stakeholders.
Identify who will plan and oversee the evaluation.
Define the purpose of the evaluation and the use of the results.
Specify what is judged and formulate evaluation questions.
Determine who supplies needed evidence and/or currently available data.
Delineate the evaluation approach.
Choose data collection and use plan.
Indicate the analysis procedure.
Stipulate how judgments are made.
Determine timeline, budge, needed resources.
Monitor and complete the evaluation and make judgments.
When to Collect Evaluation DataWhen to Collect Evaluation Data
Collecting DataCollecting Data
How to Collect DataHow to Collect Data
ObservationsInterviewsWritten QuestionnairesTestsSurveysProduct ReviewsPerformance ReviewsReview of RecordsPortfoliosCost-benefit AnalysisFocus GroupsSelf-Assessment
Levels of Evaluation ApproachLevels of Evaluation Approach
Evaluate◦Participant reactions◦Participant learning◦Behavior change or use new knowledge/skills◦Results or outcomes
Focus◦Primarily on participant reactions and changes
Accountability Planner ApproachAccountability Planner Approach
Evaluate◦Skills/ knowledge/ attitudes◦Achievement of broad objectives◦Learning, tasks, and materials.◦Anticipated changes◦Evidence of change
Focus◦Data collected at each level of program.◦What should you know or believe at the end?◦What were the techniques to reinforce the
objectives?
Situated Evaluation FrameworkSituated Evaluation Framework
Evaluate◦Review of program◦Reflection on criteria (who determines and how)◦Product Review ◦Performance Review
Focus◦Programming (what and how)◦Valuing (who decides what is valuable and how)◦Knowledge Construction (what counts as evidence)◦Utilization of Evaluation Findings (for what ends and
by whom)
Systems EvaluationsSystems Evaluations
Evaluate◦Written Questionnaires◦Cost-benefit Analysis
Focus◦Educational unit function◦Efficiency of use of resources
Case Study MethodCase Study Method
Evaluate◦Observations◦Interviews◦Review of Records◦Qualitative Data
Focus◦What does the program look like from the different
individuals involved Participants Staff Sponsors Stakeholders
◦How was it implemented and received
Quasi-Legal EvaluationQuasi-Legal Evaluation
Evaluation◦Interviews◦Product Reviews◦Tests◦Cost-Benefit Analyses◦Review of Records
Focus◦Quality determined through adversarial
hearings◦Panel of judges determines effectiveness based
on presented evidence
Professional or Expert ReviewProfessional or Expert Review
Evaluate◦Interviews◦Review of Records
Focus◦Panel of experts evaluate based on set of
standards◦Often done for large education and training
programs
EthnographyEthnography
Evaluate◦Qualitative Focus◦Interviews, Observations, Story-telling◦Building of life histories and memoirs
Focus◦Enter and close, prolonged interaction with
people in their everyday lives◦Purpose: placing actions and changes within a
larger context