126
v.2.2.1
v.2.2.1
The Guide to
The SYSTEMS EVALUATION PROTOCOL
ii
iii
TABLE OF CONTENTS
7 7
é
PREFACE
ACKNOWLEDGEMENTS
Figure 1.
Phases of
Program
Evaluation
Throughout the Protocol there are sidebars that will be of interest to systems
These are added for their supplementary value and will enhance understanding of the
reader should be able to use the Protocol even
Figure 2.
Protocol
Terminology
for System
Hierarchy
Throughout the Protocol
want to use as a group
Universe Multiverse
Atom
Solar SystemPlanet
Galaxy
Figure 3.
Phases of Evaluation Planning
7
of the human thrower and use mechanical devices like a catapult we can predict even
This discussion format
workgroup members should be broken up into small groups (2-5 people in each
is presented and at each
given 2-3 minutes to think about and take personal
5-7 minutes to share and
process can be repeated
prompts generally allow
share the notes created at
mind when you think about
Figure 4.
Hypothetical map
of stakeholders
especially colleagues
of stakeholders can be
program member list the stakeholders individually onto separate pieces of
it notes should also work
place the stakeholders on
stakeholders near their
taking turns in a formal
own criteria for similarity
When the diagram is complete (when everyone
discuss the process and
have developed - this may
stakeholders on their own
stakeholders they had not
VII.
for stakeholders because they lead to discussions about the meaning of what they are
Figure 5.
Phases in
Program
Lifecycle
iNITITATION dEVELOPMENT sTABILITY Dissemination
PHASE I pHASE II PHASE III PHASE IV
State of the Program
cHARACTERIZING A PROGRAM'S EVOLUTION
discuss how and why the program has changed
The following prompt is designed to help the
the group to best communicate the story of the
are several working groups engaged at the
groups take turns sharing and describing their
Figure 6.
Phases in
evaluation
Lifecycle
Process &
Response
PHASE I pHASE II PHASE III PHASE IV
State of the Evaluation
cHARACTERIZING An Evaluation'S EVOLUTION
changeComparison
& ControlGeneralizability
Figure 7.
Lifecycle
Alignment
is concerned with the origin and the
Generalizability
Phase I Phase II Phase III Phase IVPhase I Phase II Phase III Phase IV
Prog
ram Lif
ecycle
Evaluatio
n Lif
ecycle
Phase IA
Phase IB
Phase Iv
A
Phase Iv
B
Phase iIIB
Phase iIIa
Phase IIa
Phase IIB
Prog
ram and Evaluatio
n Lif
ecycle Def in
itio
ns and alig
nment
the theory of natural
all of the life sciences
Figure 9
Sample Logic
Model
Figure 10
Sample Path
from activity to
outcome
Figure 11
Pathway from
activity to
outcome
Figure 12.
pROGRAM pathway
model
feedback can be especially valuable to the logic and pathway model development process when programs reviewing each
(
useful insights about the answers to
in a manner that is
makes strategic
contributes to our
Figure 13.
Stakeholders
located on
pROGRAM pathway
model
Review
Figure 14.
Evaluation Scope
within it as shown in
Figure 15.
lITERATURE
MAPPING (Golden
Spike)
to delve deeper into understanding stakeholder
key stakeholders about
stakeholder whom they would be interested in
agree on a plan for how to capture stakeholder
workgroup member should
that the stakeholder input
interview the stakeholder
and if agreeable establish
group should then create
interview based on the list provided in
and pathway model during the interview can provide
the program before sharing
to a peer
brainstorm as many strategies as possible and then narrow them down based on your
stakeholders outside of the working group can provide a
program implementers and/or stakeholders who are
most column with each of their
peer or stakeholder is then
may also include columns
working group can also use the brainstorm guide to develop
for an imagined or well-known
neutral program will help
of their choices and the challenges facing them as they
and
do not
TimeRules out
other possible
Shows Program lifecycle phase it may be appropriate for
No No No
post pre No No No
No No
and
Analysis the process of deriving order and meaning from data
that may be outside the program
must screen the data for accuracy - allowing you to go back and clarify
the programan abstract or general idea inferred or derived from empirical evidence
not receive the treatment
on one scale are associated with low values on another
WikipediaWikipedia
building
person in the leadership role who will facilitate the protocol and guide
the larger systems within which the program is embedded
statement
in the future
Lifecycle analysis
Long-term outcomes
outcomes to long-term outcomes
Pathway model
Post-only
Pre-post
Program
Program boundary
a step-by-step guidethe process of deriving order and meaning from data using non-numerical methodsthe process of deriving order and meaning from data using numerical
they already possess
Wikipedia
Random assignmentin the study
Reliability
Sample
Wikipedia
SEP
for that programShort-term outcomes
Stakeholder analysis
Stakeholders
Subscale a smaller set of items on a measure that have shared validity and reliability independent of the larger measure
SurveySystem
take place
which the program is embedded
between each
Timeline
completedTreatment group
program change
that they were designed to measure
[
and [
[
[ORGANIZATION
ORGANIZATION
[
The [
of the [
and the [
[
PERSONNEL
TRAINING
HARDWARE
SOFTWARE
THANK YOU!
DESCRIPTION
FUNDING
INPUTS
PROGRAM STAGE
PARTICIPANTS
LOGIC MODEL
DESCRIPTION
EVALUATION TRAINING
PROGRAM
IN OUT
INITIATION DEVELOPMENT STABILITY DISSEMINATION
PHASE I PHASE II PHASE III PHASE IV
STATE OF THE PROGRAM
CHARACTERIZING A PROGRAM'S EVOLUTION
INITIATION DEVELOPMENT STABILITY DISSEMINATION
STATE OF THE EVALUATION
PROCESS &RESPONSE
PHASE I PHASE II PHASE III PHASE IV
CHANGE COMPARISON &CONTROL GENERALIZABILITY
CHARACTERIZING AN EVALUATION’S EVOLUTION
and the
dist
ribut
edth
is
Phase I Phase II Phase III Phase IVPhase I Phase II Phase III Phase IV
Phase IA
Phase IB
Phase Iv
A
Phase Iv
B
Phase iIIB
Phase iIIa
Phase IIa
Phase IIB
ev
alu
ae
le
e e
B
A
C
ev
alu
ae
le
e e
checklistchecklist
checklistchecklist
checklistchecklist
outputs
with a
on the
Use:
or an
weaker
