30
Video Lectures for MBA By: Video.edhole.com
Video Lectures for MBA
By:
Video.edhole.com
PPA 502 – Program Evaluation
Lecture 2c – Process Evaluation
Video.edhole.com
Introduction You must figure out exactly what the program
theory is—that is, you must articulate the program theory; you must make it explicit, so that it can be examined and evaluated.
Typically, a program theory will be implicit, at best. That’s why the evaluator must understand how the program is intended to operate.
Program theory (like needs assessment) is very important, because it is foundational for later types of evaluation in the RFL model (i.e., implementation assessment, impact assessment, and efficiency assessment).
Video.edhole.com
Introduction
Overview of Program Theory
Target population
Service
Arena
Program
Service Utilization Plan
Program's Organizational Plan
Impact Theory
Target interaction with delivery system
Program - target service interactions
Program facilities, personnel, activities
Proximal
Outcomes
Distal
Outcomes
Video.edhole.com
Program Theory
A social program centers on the transactions that take place between a program’s operations and the population it serves.– Program impact theory.– Service utilization plan.– Program organizational plan.
Video.edhole.com
Program Theory
Program impact theory.– Assumptions about the change process started
by the program and the improved conditions expected to result.
Video.edhole.com
Program Theory
Service utilization plan.– The program’s assumptions and expectations
about how to reach the target population, provide and sequence service contacts, and conclude the relationship when services are no longer needed or appropriate.
Video.edhole.com
Program Theory
Program’s organizational plan.– Program resources, personnel, administration,
and general organization.– Resources, organization, and activities will
produce a viable organization that will successfully implement the service utilization plan.
Process theory.– The service utilization plan and the
organizational plan combined.
Video.edhole.com
Process Evaluation
Process evaluation assesses the success or failure of the program theory.
Video.edhole.com
Three Major Questions of Process Evaluation What is the program intended to be?
– Methods to develop and specify program components
What is delivered in reality?– Methods for measuring program implementation
Why are there gaps between program plans and program delivery?– Assessing influences on the variability of
implementation.
Video.edhole.com
Chain of Events Logic for Process and Impact Evaluation
Phase I Phase II Phase III Phase IV
Program
Events ContinuedConcerning: Activities
Recipients
DesignInterventionFormative evaluation.
Evaluability assessment.
Use theory and prior studies.
Develop delivery plan
Specify components.
Assign resources.
Design work flow.Build networks.
Use theory and prior studies.
Deliver interventionImplement
components.
Manage resources.
Manage work flow.
Communicate.
Collect data.
Assess impactsOrganizations
change?
Collect data.
IdentifyPopulation
Needs assessment.
Population studies.
Information flow
Recruit/select participants
Collect data.
Participants receive intervention
Collect data.
Participants change?
Collect data.
Select nonparticipants
Collect data.
Nonparticipants do not receive intervention
Collect data.
Nonparticipants do not change?
Collect data.
Information flow
Information flow
Information flow
Video.edhole.com
Process evaluation to develop and specify program components A major function of process evaluation is
using data to help design interventions to clarify and obtain agreement on what is the intended program.
Video.edhole.com
Types of program components
Intended recipients.– Background characteristics appropriate for program.
– Eligibility requirements.
– Recruitment mechanisms.
– Selection process.
Intended context.– Types of agencies that will deliver the program, and
their characteristics.
– Community context.
Video.edhole.com
Types of program components
Intended delivery.– Activities: Who does what? With whom?– Staffing: What types, backgrounds, and skills
will be used?– Materials needed for delivery.– Information and information systems.
Intended scope of program.
Video.edhole.com
Criteria for measurable program components Specify activities as behaviors that can be observed
rather than as goals or objectives. Ensure that each component is separate and
distinguishable from other components so that each one can be separately measured.
Explicitly link each component to its underlying theoretical rationale.
Include all activities and materials intended for use in the intervention.
Identify the aspects of the intervention that are intended to be adapted to the setting, and those that are intended to be delivered as designed.
Video.edhole.com
Techniques for Program Specification Formative Evaluation
– Data collected from pilot situations and recipients while developing an intervention to obtain feedback about the feasibility of proposed activities and their fit with intended settings and recipients.
Evaluability Assessment– A set of systematic processes for developing the underlying
program theory and clarifying intended uses of data before initiating full-scale evaluation.
Use of Theory to Aid Program Specification– Applying normative and causal theories relevant to the
content area and using data to elucidate the underlying processes.
Video.edhole.com
Data collection methods for formative evaluation Focus groups Observation Open-ended interviews Ethnographic analysis Message or form analysis Expert judgment Equipment trial
Video.edhole.com
Process evaluation methods for measuring program implementation The program components are the basis for
selecting or developing instruments to measure two key aspects of program delivery: the extent of implementation (number and quality of components delivered) and the scope of implementation (number of recipients reached and their characteristics).
Video.edhole.com
Uses of implementation data
Monitor current activities to identify problems in implementation, then improve service delivery.
Measure variability in program delivery for later statistical analyses of program impacts.
Use as dependent variables in assessing why delivery is or is not carried out as intended.
Video.edhole.com
What to measure
Number of participants reached and their characteristics.
The number of program activity components and their characteristics.
Video.edhole.com
How to measure implementation
Use of technical equipment. Indirect unobtrusive measures. Direct observation. Activity or participation log. Organizational records. Written questionnaires. Telephone or in-person interviews. Case studies.
Video.edhole.com
Problems in implementation measurement Single measures. Reliability and validity. Representativeness. Implementation should be explicitly linked
to outcomes for impact assessment.
Video.edhole.com
Assessing influences on the variability of implementation The implemented program is likely to vary
from original intentions.– Why does it occur?– How does it effect program effectiveness?– What might be done to create full
implementation?
Video.edhole.com
Macro-implementation versus micro-implementation Macro-implementation: large scale
programs. Focus on intergovernmental management.– Decision points, competing priorities, interests
and resources of the various actors, and diverse strategies for change.
Micro-implementation: the extent of compatibility between the pre-existing organization and the new program requirements.
Video.edhole.com
Influences on micro-implementation Organization as a whole
– Types of decision-making processes used – centralized or participatory.
– Procedures and priorities for allocating resources.
– The supportiveness of an organization’s overall culture toward programs of the type implemented.
– The types of pressures emanating from the organization’s environment of beneficiaries, supporters, competitors, and regulators.
Video.edhole.com
Influences on micro-implementation Structures and processes of an organization’s work
units.– Expectations and performance feedback from the
supervisors of each unit.– Technologies in use and their fit to the new program.– Standard operating procedures used to simplify and
regulate work flow.– Use of time and availability of time to learn activities for
the new program.– Social norms governing ways of working that have
developed within a unit.– Communication processes both within and between units.
Video.edhole.com
Influences on micro-implementation The actions of individual deliverers (“street-level
bureaucrats”).– Their own capabilities or skills in using the innovation.
– Extent and types of training they are given for delivering the new program.
– Their concerns about how it affects them personally, such as its possible effects on their future employment and advancement.
– External incentives and internal motivations to learn and to deliver the new program.
Video.edhole.com
Methods for Data Collection and Analysis Case Studies Systematic surveys of organizational
members.
Video.edhole.com
Suggested roles for process evaluation The usefulness of evaluation for improving
program management and delivery will be substantially increased by more emphasis on data about program processes.
Evaluations of the impact of program interventions should always include measures of the extent of program delivery.
Video.edhole.com
Suggested roles for process evaluation (contd.) Understanding of program impacts, whether or not
the desired change occurred, will be greatly strengthened by process evaluation data.
The more that variation in program delivery is expected among multiple sites, the greater the need for process evaluation.
The larger the scale of an evaluation study in number of sites and participants, the greater the need to measure the extent and processes of intervention delivery.
Video.edhole.com
