BUILDING PROGRAM THEORY FOR EVALUATION: THE PROCESS THROUGH A POLITICAL LENS by MARY KATHRYN ROBERTSON (Under the Direction of Ronald M. Cervero) ABSTRACT HRD professionals have an ethical and pragmatic obligation to evaluate training programs (Russ-Eft & Preskill, 2001). However, outcome focused evaluation models have dominated the organizational literature. Theory-driven evaluation has been suggested as an alternative. Theory-driven evaluation uses a program theory to make “explicit the underlying assumptions about how programs are expected to work…and then using this theory to guide the evaluation” (Rodgers, Petrosino, Huebner, & Hacsi, 2000). However, the program theory building process has not been explored. The purpose of this research was to explore the ways that multiple stakeholders’ interests are represented in the program theory building process when conducting a theory-driven evaluation. Three research questions guided this study: 1) what are the program theories of stakeholders?, 2) what similarities exist among the stakeholders’ program theories?, and 3) what aspects of the program theories are unique to each stakeholder? A case study methodology was used, focusing on a new employee orientation program at a 196 bed hospital. Findings included a description of the orientation program and a program
Transcript
BUILDING PROGRAM THEORY FOR EVALUATION: THE PROCESS THROUGH A
POLITICAL LENS
by
MARY KATHRYN ROBERTSON
(Under the Direction of Ronald M. Cervero)
ABSTRACT
HRD professionals have an ethical and pragmatic obligation to evaluate training
programs (Russ-Eft & Preskill, 2001). However, outcome focused evaluation models have
dominated the organizational literature. Theory-driven evaluation has been suggested as an
alternative.
Theory-driven evaluation uses a program theory to make “explicit the underlying
assumptions about how programs are expected to work…and then using this theory to guide the
evaluation” (Rodgers, Petrosino, Huebner, & Hacsi, 2000). However, the program theory
building process has not been explored. The purpose of this research was to explore the ways
that multiple stakeholders’ interests are represented in the program theory building process when
conducting a theory-driven evaluation. Three research questions guided this study: 1) what are
the program theories of stakeholders?, 2) what similarities exist among the stakeholders’
program theories?, and 3) what aspects of the program theories are unique to each stakeholder?
A case study methodology was used, focusing on a new employee orientation program at
a 196 bed hospital. Findings included a description of the orientation program and a program
theory for each of the nine stakeholders interviewed. The program theories were compared for
similarities and differences.
This study had three major conclusions: 1) similarities exist between stakeholders
program theories, 2) individual stakeholders also have differing, and conflicting, components of
their program theories, with many of the differences accounted for by the role of the stakeholder
within the organization, and 3) in developing a program theory for evaluation, a broad range of
outcomes and processes were specified by program stakeholders.
The major assumption in the theory-driven evaluation literature is that there is a single
program theory underlying a program. This research confirmed that there is some agreement
among stakeholders, but also showed that stakeholders have unique understandings about how
and why programs function. Finally, the research showed many of the outcomes and processes
identified as important by stakeholders would not be included in an outcome focused evaluation.
Recommendations for practice included the need for evaluators to negotiate a single
program theory from stakeholders’ multiple theories, the role of the evaluator, and when to use
theory-driven evaluation.
INDEX WORDS: evaluation, theory-driven evaluation, stakeholder evaluation, Human
Resource Development evaluation, negotiating power and interests
BUILDING PROGRAM THEORY FOR EVALUATION: THE PROCESS THROUGH A
POLITICAL LENS
by
MARY KATHRYN ROBERTSON
A.B., The University of Georgia, 1998
M.S., The University of Tennessee at Chattanooga, 2000
A Dissertation Submitted to the Graduate Faculty of The University of Georgia in Partial
participatory/collaborative, empowerment, organizational learning, and theory-driven. Yet note
that “most of the approaches for evaluating training and performance follow in the tradition of
the behavioral objectives approaches, with some limited use of theory-driven evaluation” (Russ-
Eft & Preskill, 2001, p. 88). None of evaluation approaches utilized for training evaluation
included using a participatory, collaborative, responsive, or use-focused approach.
The black box evaluation. For forty years, the field of training evaluation has limited the
scope of evaluation by asking only a few outcome oriented questions, and using only a few tools
and methods (Hilbert et al., 1997). This has resulted in “a constricted view of what training
evaluation can offer organizations” (Hilbert et al., 1997, p. 145). Bickman (1994) supports this
view, stating, “most evaluations have been conducted in the public sector. The private sector is a
decade behind the public sector in the use of evaluation” (p. 255). Hilbert, Preskill, and Russ-Eft
(1997) suggest that organizational evaluation should draw upon the literature of large-scale
social programs, and incorporate logic models into the organizational literature.
Narrowly defining training evaluation has resulted in creating the black box evaluation,
characterized by a focus on outcomes with little attention given to how those outcomes were
achieved or not achieved. There is a “focus on the overall relationship between the inputs and
outputs of a program without concern for the transformation process in the middle” (Chen,
1990b, p. 18). The information resulting from such evaluations is difficult for policy makers or
practitioners to apply, and is often misleading.
In summary, training plays an important and costly role in organizational functioning.
Evaluation is a necessary part of the training process, given the amount of resources dedicated to
the endeavor. However, Kirkpatrick’s hierarchy of training outcomes, a behavioral objectives
approach, has dominated the training evaluation literature. This focus on program outcomes has
15
essentially created a black box around the training process. In light of this restricted view, the
field should look to the evaluation literature of large-scale social, educational, and health
programs. These bodies of literature offer more advanced theories and tools for conducting
evaluation. Specifically, this literature has utilized the concept of theory-driven evaluation and
related concepts. The use of these types of evaluation models holds promise for advancing
training evaluation.
Theory-driven Evaluation
Theory-driven evaluation has been a movement of critical importance in the last two
decades, in that it is a broader perspective of evaluation than has been conceptualized in the past
(Chen, 1994). Theory-driven evaluation is a process for evaluating programs in which a program
theory, or logic model is created that represents program functioning. This program theory is
then tested to see if the program is operating as conceptualized, and to understand if the program
is creating the anticipated impact. This opens up the black box created by simple outcome
focused evaluations, and explores the way the outcomes are created. By answering the “why”
question evaluators will be able to contribute to our understanding of how to plan effective HRD
programs (Cervero, Rottet, & Dimmock, 1986). In this section the concept of theory-driven
evaluation will be explored, with a focus on the definition, history, and an overview of the use of
theory-driven evaluation in organizational settings.
Definition of Theory-driven Evaluation
Theory-driven evaluation is an approach in which the evaluator “makes explicit the
underlying assumptions about how program are expected to work- the program theory- and then
using this theory to guide the evaluation” (Rogers et al., 2000, p. 5). Central to theory-driven
evaluation is the use of a program theory to examine the expectations of the program (Reynolds,
16
1998). Program theory is essentially a blueprint of how the HRD program is supposed to work.
In other words, it is a “plausible and sensible model” of the program (Bickman, 1987, p. 5), in
which the relationship between the treatment and outcomes is specified. The complete model
should be a broad view of the channels through which the program will achieve the desired
results. The theory-driven evaluation process includes the development of the program theory
and the testing of the program theory.
Theory-driven evaluation has received widespread attention in the evaluation literature in
recent years, yet there has not been consensus on the terms and definition of the concept. Rogers
et al. (2000) note that “program theory is known by many different names, created in many
different ways, and used for any number of purposes” (p.7). Examples of the various
terminologies associated with the concept include: outcome hierarchies (Bennett, 1975), theory-
of-action (Schon, 1997), program theory (Bickman, 1987), theory-based evaluation (Weiss,
1995, 1997), program logic (Funnell, 1997), theory-driven evaluation (Chen & Rossi, 1983), and
program philosophy (Conrad, 1987); (Rogers et al., 2000). Definitions range from very broad
definitions that include any specification of program components, to highly structured definitions
that depend on specific program theory development techniques.
I have chosen to use the terminology theory-driven evaluation, as defined as an explicit
theory or model of how the program causes the intended or observed outcomes, that is used in
some way as a guide during the evaluation (Rogers et al., 2000). The theory should be explicit
about what happens in the black box when program inputs are transformed into program
outcomes (Bickman, 1987).
17
Nature of Program Theory
The program theory specifies more than simple inputs and outputs, and it is not simply a
“to do” list of program activities (Rogers et al., 2000). Instead it is model showing a series
intermediate outcomes, and moderating variables that lead to the desired outcomes. Including
such moderators results in a more complete understanding of the organizational conditions,
individuals, and characteristics of programs that are the most likely to produce the desired
outcomes (Cervero, 1985). The channels through which the program achieves (or fails to
achieve) the desired results can then be mapped.
Theory is a term that can be used in a variety of contexts. Chen (1990b) defines theory
“as a set of interrelated assumptions, principles, and/or propositions to explain or guide social
actions” (p. 40). More specifically, Weiss (1997b) defines program theory as “the set of beliefs
and assumptions that undergird program activities” (p. 503). Program theory can also be defined
as “a specification of what must be done to achieve the desired goals, what other important
impacts may also be anticipated, and how these goals and impacts would be generated” (Chen,
1990b, p. 43). These three definitions can be used as guides for describing the nature of program
theory.
Program theory is similar to the notions of personal theory, practical knowledge, and
theory in use. Jarvis (1999) states that practitioners develop what he calls personal theory or
practical knowledge, which is closely related to what Argyris and Schon refer to as theory in use
(Argyris, 1993; Schon, 1987). Knowledge learned in practice, or personal theory, is flexible, and
responsive to the demands of practice. The knowledge is a combination of learning from doing
and thinking about practice, and learning from information sources. This combination creates a
pragmatic knowledge base that can guide practice. Usher et al (1997) support Jarvis’s view of
18
practitioner knowledge by stating that it supports the decision making process when practitioners
are faced with particular, complex, and problematic situations in practice. More specifically,
practical knowledge is “knowing how to act appropriately in relation to the circumstances of a
particular situation or context” (Usher et al., 1997, p. 128).
The concept of program theory can be conceptualized as having two dimensions,
prescriptive and descriptive. The descriptive dimension focuses on an explanation of the
program. That is, the focus is on what actually takes place during program functioning. This
notion is represented when program theory is defined as “program resources, program activities,
and intended program outcomes, and specifies a chain of causal assumptions linking program
resources, activities, intermediate outcomes, and ultimate goals” (Wholey, 1987, p. 78). The
prescriptive dimension of the program theory is value-laden, and is focused on issues such as
how to design and implement the program (Chen, 1990b). The prescriptive dimension focuses
on what should be done in ideal circumstances. This portion of the theory gives guidance by
telling practitioners how to conduct the program in the best possible way. Definitions of
program theory have focused on the descriptive theory. However, Chen (1990b) advocates for
the inclusion of prescriptive theory as well, and to advance its use, names three characteristics of
a prescriptive theory. First, the prescriptive theory has an action orientation. This means that the
theory contains “specific strategies for achieving a goal or solving a social problem” (Chen,
1990b, p. 41). The second characteristic of a prescriptive theory is the inclusion of treatment
format and implementation strategy. This will cover issues of how the treatment is constructed
or formatted with regard to program components and program strength. Program strength can be
measured using dimensions of the program such as purity of the program, specificity, dosage,
19
frequency, and duration (Scott & Sechrest, 1989). Finally, the prescriptive theory should deal
with choosing the outcome criteria for the theory from a wide range of options.
Testing the Program Theory
Once established, the program theory can be confirmed or denied by investigating the
linkages between the elements of the theory. Theory-driven evaluation is unique in that it does
not rely on one type of research methodology (Chen & Rossi, 1989). Instead, a variety of
research methods can be used to develop and test the program theory. In fact, because of the
comprehensive nature of theory-driven evaluation, multiple methods are often needed to
complete the entire process. Quantitative methodologies have dominated the discussion of
appropriate methodology and will be discussed here. However, it should be noted that
qualitative methods hold great promise in the future of theory-driven evaluation.
There are several techniques that have proven useful for program theory verification. Causal
modeling is one technique used to verify the program theory. Causal models can take several
forms including, linear structural relations (LISREL), confirmatory factor analysis, latent
variable structural models, and path analysis (Lipsey & Pollard, 1989). The use of causal
modeling techniques requires that each of the components in the model be a variable. In other
words, the components must vary in such a way that different levels have different values for
each component (Lipsey & Pollard, 1989). This leads directly to the second requirement, which
is that variables must be quantifiable, so that the differing levels or values for each component
can be identified (Chen & Rossi, 1989). The advantage of using causal modeling is that it forces
evaluators to get specific on the program theory components and the relationship between those
components. Furthermore, advanced computer programs are available to assist the evaluator
when testing the model. Lipsey and Pollard (1989) also present a simpler form of causal
20
modeling called the “basic two-step” (p.320). The basic two-step narrows the focus of the
evaluation to include the first intervening variable in the program theory, that is, the variable that
is expected to change as a direct result of the treatment. The second “step” is the more distant
variable that is expected to be impacted as a direct result of the changes in the first intervening
variable. This technique broadens the range of a simple “black box” evaluation considerably by
considering major mediating variables, but is not as likely to explain complex program contexts
(Lipsey & Pollard, 1998).
A second methodological approach used in conducting theory-driven evaluation is known
as confirmatory program evaluation (Reynolds, 1998). This approach is also used to establish
causal relationships between program, intervening variables, and identified outcomes proposed
in the program theory, specifically in the post-program stage and periods of follow-up. While
the exact process for conducting a confirmatory program evaluation is beyond the scope of this
discussion, the process is similar to the causal modeling techniques described previously. The
process may be applied to experimental, quasi-experimental, or non-experimental data. One
benefit of this approach is the ability to strengthen causal inferences “in most quasi-experimental
and non experimental designs” (Reynolds, 1998, p. 207).
Third, Mohr (1995) presents methodology designed to generate measures of effectiveness
or adequacy. These two measures are based on a comparison between actual results and the
counterfactual, the state of the world without the program. For this comparison to be complete
Mohr advocates the use of a control group. While the identification of this control group is often
a difficult task, and by far the strongest limitation of this evaluation strategy, Mohr does offer
many strategies for working with quasi-experimental designs. In addition, regression-
discontinuity designs, and time series designs have been offered as strategies for evaluation when
21
no control group is available. The benefit of this rigorous process is the ability to attribute
causality to findings, if and only if designs are meticulously executed.
History of Theory-driven Evaluation
Early evaluation efforts emphasized scientific research methods in an attempt to establish
the field as a scientific field of study. This focus on methodological issues created an
environment where major evaluation perspectives have been oriented towards methodology.
This focus was further propagated by the 1965 passage of the Elementary and Secondary
Education Act (ESEA), which included a large amount of federal spending. The passage of this
act prompted Senator Robert Kennedy to insist that each federal grant recipient conduct an
evaluation for federally sponsored programs (Worthen et al., 1997). It was with this single act
that “program evaluation had been born overnight” (Worthen et al., 1997, p. 32). As a result,
educators around the county were required to perform evaluations. As there were few trained
evaluators at that time, individuals trained in social science often were called on to perform these
tasks. Commonly used techniques included experimental research designs, and standardized
testing, which focused almost exclusively on program outcomes. As a result of the prominence
of social scientists filling evaluation roles, social science methodology has had a huge impact on
the field.
Social science methodology was advanced when Campbell and Stanley (1963/1966) were
first introduced into the field of evaluation through the publication of Evaluation Research:
Principles and Practices in Public Service and Social Action Programs, written by Edward
Suchman in 1967 (Campbell, 1994). The original purpose of Campbell and Stanley’s work was
not aimed at evaluation, but designed to guide social psychologist and educators in the use of
causal inference in the “uncontrolled real world.” The recommendation that Campbell and
22
Stanley be used as evaluation methodology has strongly influenced the field of evaluation,
especially in the area of impact evaluation. Throughout the 1970’s Campbell’s ideas of
experimental design and internal validity ruled the field (Baker & Niemi, 1997).
So while progress was made as to the appropriateness of differing methodological
approaches, the field still lacked the conceptual bases for integrating contextual factors and
research methods. It was with the development of theory-driven evaluation that the causal
mechanisms of programs began to be explored. A theory-driven evaluation advocates the
development of a program theory, or a map of how a program is constructed to achieve the
desired outcomes. In this way the “black box” is opened allowing evaluators to examine how
and why program inputs get translated into program outcomes.
Worthen (1996) suggests that “casual reading of current literature on theory-based
evaluation could leave the impression that this view suddenly burst on the evaluation scene in the
1980’s” (p. 169). Yet the ideas surrounding theory-driven evaluation developed slowly, with
many of the early writings going virtually unnoticed. Origins of the idea of basing program
evaluation on a causal model of the program can be seen as early as the 1960’s (Rogers et al.,
2000). Suchman (1967) introduced the idea of a “chain of objectives” (p. 55). This idea stressed
the importance of understanding the program, the objectives, and the intervening processes. This
work was followed by Weiss’s (1972) writings on the use of a model in evaluation. This paper
stresses the importance of the identification of the steps by which a program is intended to work.
In 1975 Fitz-Gibbon and Morris published an article titled, Theory-based evaluation, in which
the essential characteristics of this approach are outlined. In 1980 Chen and Rossi published The
multi-goal, theory-driven approach to evaluation: A model linking basic and applied social
science, which was reprinted later in 1981. These authors further elaborated the ideas of theory-
23
driven evaluation in a 1983 publication. Few authors writing on the topic of theory-driven
evaluation reference work previous to Chen and Rossi (1980/81; 1983). Yet, this work relies
heavily on those earlier writing mentioned above. It was during the late 1980’s and 1990’s that
the idea of theory-driven evaluation became a “powerful movement in program evaluation”
(Worthen, 1996, p. 169). In the late 1980’s writing on the topic of theory-driven evaluation
became increasingly prevalent. For example, Bickman (1987) published the first book on the
topic and in 1989 a special issue of Evaluation and Program Planning was devoted to the topic.
Program theory offers guidelines for overcoming the barriers to the traditional method-
driven approach. For example, the issues most important in the evaluation can be identified;
from there the methods most relevant to addressing these issues can be selected and applied
(Chen, 1990b). It can be seen that methods play quite a different role in theory-driven
evaluation. Instead of viewing evaluation as a data collection technique, an emphasis is placed
on developing a body of literature that is unique, systematic, and theoretically based (Chen,
1990b). That is, it is not appropriate to view evaluation as a group of methods or data collection
techniques. Instead, a comprehensive framework is needed to link to various components of the
program under consideration. Because a wide array of both quantitative and qualitative
techniques exist, it is appropriate to turn our attention away from the question of whether the
program made a difference and focus on the question of how and why to program made a
difference.
The use of theory-driven evaluation has grown as more agencies have adopted the idea.
For example, the National Institutes of Health now require an explanation of the program theory
in applications for evaluation support (Rogers et al., 2000). Chapters on theory-driven
evaluation have also been added to popular evaluation text books such as Program Evaluation: A
24
Systematic Approach by Rossi, Freeman, and Lipsey (1999), and Evaluation Models:
Evaluation of Educational and Social Program, by Madus, Stufflebeam, and Scriven (1983).
Theory-Driven Evaluation in Organizations
Within theory-driven evaluation several derivatives have been seen in the organizational
literature. Russ-Eft and Preskill (2001) present four evaluation models that have characteristics
of theory-driven evaluation: logic models, the Input-Process-Output model, Holton’s HRD
Evaluation Research and Measurement Model, and Brinkerhoff’s Six Stage Model. The
similarities and differences between each of these four models and theory-driven evaluation will
be explored.
Logic models. A logic model or map is essentially a tool to help evaluators “determine
the extent to which the program has clearly defined and measurable objectives, a logic or
rationale for reaching the training goals, and a sequence of activities that represents that logic or
rationale” (Hilbert et al., 1997, p. 143). The core elements of a logic model include resources,
activities, outputs, customers reached, short-term, intermediate, and long-term outcomes, and
other external or contextual influences (McLaughlin & Jordan, 1999). The information used to
construct the logic model can be derived from several sources, including program
documentation, interviews with stakeholders and related literature. The external literature can be
searched for examples of what others have done to solve similar problems and literature on key
factors in the model. In addition to external literature, the logic model is developed in an
interactive and collaborative process with program stakeholders to create the model (Russ-Eft &
Preskill, 2001). One technique that can be used to develop the logic model is to use a series of
“if, then” statements. Other techniques include asking a series of structured question to
understand the program logic.
25
Logic models and theory-driven evaluation are very similar concepts. In fact, many
authors consider the terms interchangeable. For example, Russ-Eft and Preskill (2001) present
logic models as “one way to develop and illustrate a program’s theory” (p. 59). Similarly,
McLaughlin and Jordan (1999) quote Bickman (1987) in there definition of a logic model. That
is, McLaughlin and Jordan (1999) define logic models “as a plausible and sensible model of how
the program will work under certain conditions to solve identified problems” (p. 66). That is the
same definition that Bickman (1987) uses to define program theory.
While conceptually the two are almost identical, the difference can be seen in the focus of
the literature. The literature associated with logic models tends to focus more on the theory
development phase of the process, while the theory-driven evaluation literature focuses more on
testing the theory. It is not to say that either excludes discussions of theory development or
theory testing, but that more emphasis is given to the topics. For example, McLaughlin and
Jordan (1999) give a five-stage process to be used when building a logic model. Similarly, Russ-
Eft and Preskill (2001) provide two strategies for the development of a logic model, yet no
attention is given to the testing of the model. On the other hand the literature on theory-driven
evaluation has given more attention to theory testing. For example, Reynolds (1998) provides
confirmatory program evaluation as a method for conduction theory-driven evaluations. This
method is designed to assess the outcomes associated with social and education programs. Other
examples of strategies for testing program theories that have appeared in the theory-driven
evaluation literature include: Scott and Sechrest’s (1989) suggestions for quantitatively linking
program strength with outcomes, Hennessy and Greenberg’s (1999) presentation of structural
equation modeling as a means of assessing program effectiveness, and Trochim’s (1989)
description of pattern matching as a way of confirming the program theory and outcomes. It can
26
be seen that while theory-driven evaluation and logic models share similar definitions and
conceptual foundations, the focus of the literature concerning these two topics has been slightly
different.
Another concept that is related to logic models is evaluability assessment. Evaluability
assessment involves interviewing key stakeholders to develop an understanding of the structure
and operations of the program and to understand the perspectives of key stakeholders and
participants on the impacts of the program and how these might be measured (Youtie, Bozeman,
& Shapira, 1999). The key to this evaluation approach is the determination of the degree to
which the particular elements of the program can be measured to provide evaluation data. These
elements include, “clearly defined and measurable objectives, a discernible logic for reaching
them, a set of sequenced activities that are plausible representations of the logic, and clear
provisions for policymakers’ and managers’ use of information about the program” (Russ-Eft,
1986, p. 40). The questions posed by this approach are “to what extent is evaluating the
program’s effect actually possible?”, and “what should be measured” (Youtie et al., 1999, p. 59).
Evaluability assessment differs from logic models and theory-driven evaluation because it asks
fundamentally different question. Yet, they share a focus on using stakeholder inputs to
understand program functioning.
Input-process-output (IPO) model. Another model of evaluation that has been
categorized as fitting within the theory-driven family is the Input-Process-Output (I-P-O) model.
This model of training program evaluation was developed by IBM and used as their corporate
educational strategy for the year 2000 (Russ-Eft & Preskill, 2001). An I-P-O model of
evaluation covers a broad range of elements in the training process. First, inputs include trainee
qualifications, instructors experience, the availability of already tested instructional materials, the
27
types of equipment and training facilities available, and the economic resources put into the
program. Second the process stage of evaluation considers the instructional objectives, the
development of design criteria, and assembly of training materials. Finally, the outputs are
evaluated. At this stage a distinction is drawn between outputs and outcomes. Outputs are short-
term effects such as, student reactions, knowledge and skills learned, and performance
improvement on the job. Outcomes, on the other hand, refer to longer-term benefits such as
profitability, competitiveness, and survival (Bushnell, 1990).
The I-P-O evaluation process has four steps. First, the evaluation goals are identified.
During this step of evaluation, decisions are made about the purpose of the evaluation. The
second step of the evaluation process is developing an evaluation design or strategy. The focus
is deciding on measures and data collection strategies. The third step in the I-P-O evaluation
process is the identification or construction of measurement tools. The data requirements
established in step two, and the reliability and validity of the measurement tools, should be used
make decisions about appropriate measure. Finally, the data should be tied to the evaluation
goals established in step one of the evaluation process.
The difference between theory-driven evaluation/logic models and the IPO model is the
element of causality. For instance, theory-driven evaluation attempts to link program
components to program outcomes. While the I-P-O model does attempt to open the “black box”
by specifying the inputs, process, and outputs, there is no clear way in which these elements are
linked. So at best the results of an I-P-O evaluation can reveal that a specific combination of
inputs and processes lead to the outcomes identified. However, there is no way to determine
which specific components affected the results. Furthermore, if no impact is found, there is no
way to identify at what point the program failed. The I-P-O evaluation model provides
28
considerably more information to evaluators and stakeholders that the traditional behavioral
objectives approach, yet is does not go as far a theory-driven evaluation in the explanation of
program functioning.
Holton’s HRD evaluation research and measurement model. Russ-Eft and Preskill
(2001) also categorize Holton’s HRD Evaluation Research and Measurement Model as related to
the theory-driven approach to evaluation. Holton’s (1996) model uses the outcomes identified
by Kirkpatrick’s four-level model as a foundation. Yet, this model attempts to move from
Kirkpatrick’s simple taxonomy to a more fully specified, researchable evaluation model (Holton,
1996). While learning, individual performance, and organizational results are primary outcomes
of the model, trainee reactions are not. Instead, it is considered a moderating variable between
motivation to learn and learning. The elements of Holton’s model that move it beyond the
behavioral objectives approach are the inclusion of both primary and secondary influences on the
proposed outcomes. The primary influences on outcomes are identified as the participants’
behavior in training, influenced by ability, motivation, and environment. Secondary influences
are the factors that influence ability, motivation, and environment, with more attention given to
influences on motivation. The model specifically outlines the influences on learning in a single
learning experience. Furthermore, the model created “should be used for research purposes”
(Holton, 1996, p. 17). That is, research surrounding this model should focus on operationalizing
each of the influences on outcomes so that the hypothesized relationships can be further
understood.
The difference between Holton’s model and theory-driven evaluation is in purpose.
Holton hopes to develop a model of training that can explain the learning process by considering
both primary and secondary influences. In other words, this is general model of a single learning
29
event. This concept differs from theory-driven evaluation in two ways. First, this integrated
model does not address characteristics of the program. Cervero (1985) suggests that processes in
the design and implementation of the program should be related to the outcomes. In other words,
this model does not account for the type or length of the program. Second, there is little room in
the model for intervening variables other than those specified. The model does allow for
unnamed external events to be added, otherwise, the model is a fixed set of relationships that do
not allow situationally specific variables to be added.
Holton’s model will however, aid in the use of theory-driven evaluation in organizational
evaluation, by establishing a well researched model of organizational learning and transfer. This
will provide the evaluator with information that can be used to create the program theory. A well
researched model can be combined with practitioner input to create a complete model of program
functioning.
Brinkerhoff’s six-stage integrated evaluation model. Brinkerhoff (1987; 1988) presents a
six-stage integrated evaluation model for HRD. This model integrates needs analysis, design,
and delivery. By focusing on program development, as well as program outcomes, this model is
intended to help the program succeed. It is designed to be a part of the entire HRD process
(program planning, development, and operation), carried out in six steps. In the first stage
evaluation takes the form of needs assessment. Data is used to determine which problems should
be addressed and if the problem can be addressed through a learning program. In stage two,
evaluation “can help a trainer decide whether a training design is sound enough or which of
several competing alternatives will work best” (Brinkerhoff, 1988, p. 68). Stage three focuses on
program implementation. That is, the evaluation is used to determine if the program is operating
as planned and whether the desired results are being produced. Stages four, five, and six of
30
Brinkerhoff’s six-stage model closely resemble Kirkpatrick’s levels two, three, and four. For
example, stage four is used to determine how much trainees have learned. Stage five is used to
determine the amount of knowledge that is retained and used or transferred to the job. Stage six
considers the benefits to the organization.
Brinkerhoff’s model, because of its similarities to Kirkpatrick’s model, could be
categorized as a behavioral objectives approach to evaluation. However, by adding evaluation
steps at the early stages of program development, this evaluation approach attempts to
understand and explain program functioning. Brinkerhoff (1988) states that using this approach
“requires articulating the assumptions about why and how each HRD activity is supposed to
work” (p. 68). Russ-Eft and Preskill (2001) suggest that a logic model could be used to guide
stages one and two of this evaluation approach. So while Brinkerhoff’s six-stage model
approaches program evaluation in a different way than theory-driven evaluation, it is possible for
elements of program theory to be incorporated. It is unlikely that an evaluator following
Brinkerhoff’s evaluation approach would create and test the program theory as is done in theory-
driven evaluation. Yet, the program theory could be developed and used as a guide during the
execution of Brinkerhoff’s six stages.
Value of Theory-Driven Evaluation for HRD Practitioners
Introducing theory-driven evaluation as an alternative to existing HRD evaluation models
holds great promise. The unique contribution of theory-driven evaluation is the ability to
consider a broad range of influences on program functioning, in a situationally specific way.
However, asking practitioners to change the way they evaluate programs will not be easy. In
order for practitioners to give up the traditional behavioral objective approaches to evaluation
that now dominate the field; a strong case has to be made for the use of theory-driven evaluation.
31
Because similar approaches, for example, logic models and Holton’s 1996 model, can be found
in the organizational literature, there is a possibility for theory-driven evaluation to be accepted
as well. Three benefits of using theory-driven evaluation in practice will be presented in an
attempt to influence practitioner understanding and appreciation for this evaluation approach.
The first strength of theory-driven evaluation is that the problem being addressed by the
training program must be specified. That is, a detailed account of the problem and the target
population must be identified. The information can help both the evaluator and the program
planner understand why the program is needed, and can be used to justify the program to
stakeholders. Similarly, the specification of the causal mechanism of the program must be
identified. This information can be used to clearly define how a program will address the
problem. So by simply developing a program theory, evaluators can create useful information
about the program to share with program planners, and other stakeholders. Furthermore, the
program theory development process can be used to gain consensus among the various program
stakeholders about the problem, the purpose of the program, and the means through which the
outcomes will be produced. There does not have to be one “right” theory developed, but by
involving multiple stakeholders, there is the possibility for consensus. In the situation where
little or not consensus exists among stakeholders, the program theory development process can
be used to address this conflict. By discussing the foundational components of the program,
stakeholders may be able to gain more insight into why the program is needed and how it is
designed to work. The development of the program theory is also a time when unrealistic
expectations stakeholders may have for a program can be addressed. By specifying the
outcomes of the program and the way the outcomes will be achieved, stakeholders can see that
no one program can be a “fix-all” solution to organizational problems. So while the program
32
theory identifies what outcomes can be expected, it also puts limits or boundaries on what can be
expected from the program.
The second strength of theory-driven evaluation for practitioners is that it does not rely
on one type of research methodology. Instead, the practitioner can choose the methods most
relevant for collecting data and testing the program theory, given the organizational context. In
fact, a variety of research methods can be used in any one evaluation. Most often qualitative
methods are used during the theory development phase. For example, the evaluator may use
interviews with stakeholders and observation of the program to understand the causal links
between the program components. Quantitative methods have dominated the discussion of
testing the program theory. For example, causal modeling techniques such as linear structural
relations (LISREL), confirmatory factor analysis, and path analysis, have been proposed for
theory testing. A simpler quantitative measure of the program theory is the “basic two-step”
approach (Lipsey & Pollard, 1989). However, as the literature on theory-driven evaluation
expands, and more examples of applications of the approach become available, the possibilities
for methods to be utilized during the process should expand as well.
While the flexibility in the research methods is considered a benefit to skilled evaluators,
it could put some evaluators at a disadvantage. Evaluators attempting to implement a theory-
driven evaluation should be technically competent in a variety of data collection and analysis
methods. However, the evaluator is also free to use the research methods with which she/he is
most familiar or feels the most competent with during the evaluation process.
The ultimate strength of theory-driven evaluation for practitioners is the quality of
knowledge that is produced. This approach provides practitioners with a tool for creating
knowledge about the types of training programs that are successful in the organization. Instead
33
of providing information about the outcomes of one specific program, theory-driven evaluation
allows for the understanding of why the program worked. This knowledge can then be used in
the next program planning process. This moves evaluation away from a tool to provide
information about outcomes, which are used to justify HRD practices, to a tool that also provides
knowledge for those working within HRD. By creating new knowledge for HRD practitioners,
theory-driven evaluation has the potential to improve HRD functioning within an organization
and the field as a whole.
In summary, theory-driven evaluation is an evaluation approach in which the underlying
assumptions about program functioning are captured in a program theory. This program theory
is then used as a guide to the evaluation, and can be tested to better understand how and why the
program works. This concept has received growing attention in many different fields, with some
limited use in the organizational evaluation literature. This evaluation approach holds promise
for the field of HRD, with the potential for creating new knowledge.
Program Theory Development
The formulation of the program theory is an important first step of conducting a theory-
driven evaluation, creating the foundation of the evaluation. However, “evaluators have often
disagreed on how to construct program theory” (Chen, 1990a, p. 10). There are three alternate
views of program theory development offered in the literature. The first view advocates the
development of the program theory through the discussions with program stakeholders. The
second approach suggests that program theory be developed by consulting existing bodies of
social science knowledge. The third approach is an integrated approach, in which social science
theory is combined with stakeholder views of program functioning.
The Stakeholder Approach to Program Theory Development
34
The stakeholder approach to program theory development focuses on the perspectives,
ideas, and expectations of the program stakeholder. According to this view the program theory
is “relevant only insofar as it reflects the stakeholders’ values” (Chen, 1990a, p. 10). The use of
practitioner knowledge, or personal theory can provide valuable insight into program
functioning. Most often practitioners rely on a combination of experience, practice knowledge,
and intuition instead of explicit theories. Yet, some theory is present in every program, despite
the fact that it may be “implicit, fragmented, and not well conceptualized” (Bickman, 1987,
p.14). Weiss (1997) points out, “theories do not have to be right, and they do not have to be
uniformly accepted” (p. 503).
According to this approach, the program theory is developed by capturing stakeholder
perceptions of the program through review of relevant documents and interviews. Discussions
with key stakeholders can be used to resolve differences in perspective on the program theory.
Chen (1990a) states, “after the differences are resolved, a formal program theory can be
completed” (p. 11). The evaluator’s role in the development of the stakeholder approach to
theory building is passive. That is, evaluators should “avoid relying on their own knowledge or
expertise in constructing program theory” (Chen, 1990a, p. 11). Instead the program theory
should emerge from the data collected. A theory used by a practitioner may be nothing more
than a small number of simple assumptions concerning how the program will work, it is the role
of the evaluator to capture the assumptions as a more explicit causal chain of events concerning
program functioning (Bickman, 1987).
One of the key features of the program theory development process is the identification
of program outcomes. During the process, all possible outcomes should be identified. This list
should include both positive and negative potential outcomes (Chen & Rossi, 1989). Because
35
the various stakeholders will have different interests regarding the program, there will be
variance in what are considered important outcome criteria to be investigated. However, it will
not be possible to submit all outcomes identified to actual (Mohr, 1995). Instead the evaluator
and stakeholders must collaborate to choose from among all outcomes identified, selecting only
those most relevant. This is a decision that should be made with input from stakeholders so that
the most relevant outcomes are selected. This decision will have large ramifications for the
results of the evaluation. Careful selection of outcomes to be tested has the potential to increase
utilization of evaluation results (Mohr, 1995).
An example of a program theory developed through the stakeholder approach is given by
Torvatn (1999). In this example the program theory is developed for an educational program
aimed at teaching participants about product development methods in the food and food
processing industry. The program theory was developed through the use of a chain of reasoning,
which is “a combination of text and graphic imaging which presents the activities of a program
or project, the goals to be attained, additional assumptions and the link between them” (Torvatn,
1999, p. 74). The chain of reasoning was developed by the program evaluator with input from
the program management.
The Social Science Approach to Program Theory Development
According the social science approach to program theory development, existing research
and theories can inform program theory during the development and interpretation phases
(Riggin, 1990). During the development of the program theory, social science theory can help
identify variables and interrelationships that are not apparent in the practitioner theories.
Advocates of using social science theory to guide program theory development warn against
accepting stakeholders’ viewpoints uncritically because “vested interests may cause stakeholders
36
to stress desirability over the plausibility of the program” (Chen, 1990b, p. 11). When using
social science to guide the devolvement of the program theory the evaluator is urged use
specialized knowledge and expertise.
Umble, Cervero, Yang, and Atkinson (2000) provide an example of using established
social science theories to guide the development of a program theory. The program under study
was a continuing education program aimed at increasing professionals’ compliance with
recommendations for practice. Four behavioral theories were used to construct the program
theory including, the health belief model, social cognitive theory, the transtheortical model, and
the theory of reasoned actions. The authors “used the behavioral science theories…to construct a
model with these outcomes to open up the ‘black box’ between the course and practice-that is, to
yield causal explanation and to show a causal connection between the course and practice” (p.
1219).
The Integrated Approach to Program Theory Development
A third approach to building program theory is an integrated approach in which social
science theories are combined with program stakeholders’ viewpoints. While this approach
combines the strengths of both the social science approach and the stakeholder approach, it is
problematic because there are not yet guidelines or recommendations about how these two types
of theories can be combined into one program theory.
This approach to developing program theory is problematic, and therefore not prevalent
in the literature. One example was found in which a program theory was developed using both
stakeholder input and social science theory. A youth drop-in center was being evaluated, not
simply one educational program (Mercier, Piat, Peladeau, & Dagenais, 2000). The evaluators
37
constructed the program theory based on three data sources, concept mapping with staff, a
literature review, and the youth’s perceptions of the center. The merging of these three data
sources into one program theory was possible because of the many of the components were
“identified unanimously as being essential” (Mercier et al., 2000, p. 86). However, it was noted
that the three data source were not consistent in regards to the amount of importance attributed
different components in the model.
The Use of Multiple Program Theories
Another solution to the problem of developing one guiding program theory is to utilize
multiple theories in the evaluation process. The inclusion of several theories can be helpful for
several reasons. First, by following the chain of assumptions of several theories, the evaluation
can determine which theory is best supported by the data that is collected. It is also useful to use
several theories when there is little consensus and the assumptions guiding the program are
unclear or unspecific (Weiss, 2000). This will reveal a more clear understanding of how the
program actually works and how the outcomes were achieved.
Because it is unrealistic to attempt to evaluate more than three or four theories in one
evaluation study, criteria are needed to select appropriate theories to be included in the
evaluation (Weiss, 2000). The first criterion is the beliefs of program designers/developers,
administrators, practitioners, and members of the funding source. The beliefs these
individuals/groups have about how the program functions should be considered in the evaluation.
A second criterion is the plausibility of the program theory. In other words, can the
program actually achieve the outcomes described in the program theory? This criterion can be
investigated by observing the program in action to determine if the elements are in place to
produce the specified changes. The third criterion is the amount of uncertainty about current
38
assumptions. When there is a lack of knowledge or an overly simplistic view about how the
program works, the assumptions should be included in the evaluation. Finally, a theory that is
central to program functioning should be included in the evaluation. Because some theories are
essential to program functioning, the success of the program will depend on the viability of the
theory (Weiss, 2000).
In summary, the program theory that is the central characteristic of theory-driven
evaluation can be developed through input from stakeholders, through a review of relevant
literature or relevant social science theories, or through a combined approach in which social
science theory and stakeholder input are merged. An alternative to creating on overarching
program theory is to create several program theories. Given the confusion surrounding the
program theory development process, the political nature of educational programs and evaluation
need to be considered.
Politics of Training and Evaluation
Cervero and Wilson (1994) establish program planning as a political process where the
planners have the opportunity to promote substantive democratic planning with those involved in
and affected by the planning process. The political context that affects the planning process
continues to affect the program after the planning stage is over. Evidence is presented here that
suggests that stakeholders continue to have divergent interests in the program that affect the
evaluation process. Several types of evaluation strategies acknowledge multiple stakeholders in
the evaluation process will be reviewed, including responsive evaluation,
participatory/collaborative evaluation, and empowerment evaluation. The development of
program theory is discussed through a political lens.
Planning is Political
39
Cervero and Wilson (1994) establish that program planning is “inherently a social,
political, and ethical activity” (p. 24). The authors draw from the critical perspective of program
planning which states that the purpose of education is to allow citizens to find out about and
communicate meaningfully about issues affecting their lives. This viewpoint takes a clear value
orientation by suggesting that program planners should work in such a way that leads to a more
just and democratic society, thereby placing responsibility of promotion of the democratic
process on the planner. The result of the attention organizing duties of the program planner
should be a well-informed, democratic planning process.
The context in which the planning process takes place must also be closely examined
according to the critical viewpoint of program planning (Forester, 1989). In the planning
process, there are four political boundaries to the decision-making process. First, the program
planner or decision maker is fallible. That is, there are cognitive limits on this individual,
creating a situation where the decision making process becomes more ambiguous, less well
defined. A second boundary on the decision making process is the socially differentiated
environment in which decisions get made. That is, a program planner works in a context in
which there are many different actors, each having a different perspective, a different amount of
time, and a different area of expertise. Third, these various actors in the planning process will
lend the program planner opposition and support throughout the planning process. As each actor
brings a different interest in the planning process, it is unrealistic to think that they will be
cooperative 100 percent of the time. Finally, the power to act is unevenly distributed in our
society and this inequality shapes the context in which program planned. Program planners
therefore need active interventions that promote social justice and seek to repair structural
imbalances of those involved in and affected by the planning process.
40
Evaluation is Political
The Cervero and Wilson (1994) program planning model has established that the
planning of a program is an inherently political and ethical activity. Evaluation is closely related
to program planning, with the planning of the evaluation often taking place during the process.
Many evaluators have argued that evaluation too is a political and ethical activity (House &
Howe, 2000; Palumbo, 1987; Ryan & DeStefano, 2000; Weiss, 1987). Weiss (1987) states that
the political context of a program effects an evaluation in three major ways. First, the program
being evaluated was created through a series of political decisions. This point is supported by
the Cervero and Wilson planning model. However, the political context does not disappear after
the planning stage is over; the programs “remain subject to the pressures-both supportive and
hostile- that arise out of the play of politics” (p. 48). Second, the results of an evaluation are
often used for decision making, yet other information will also be available to decision makers.
“Evaluative evidence of program outcomes has to compete for attention with other factors that
carry weight in the political process” (p. 48). These other factors include issues such as the
allocation of power, the development and protection of coalitions, and trade-offs with interests
groups, unions, or the affected public. These issues can often over power a report on the
effectiveness of a program. Finally, evaluators make assumptions about the evaluation process
that are political in nature. If an evaluation uses only stated program goals to guide the
evaluation, it accepts these goals as being justified and achievable. Outcome focused evaluations
are especially susceptible to this political problem, because they do not take the program context
into account or question the feasibility of the outcomes given the program (Weiss, 1987).
One key to navigating the political environment surrounding a program is to recognize
that there are multiple stakeholders with “divergent priorities and interests” (Bernstein, Whitsett,
41
& Mohan, 2002). Amba (2000) presents a case study which challenges the assumption that
consensus exists concerning evaluation criteria. Instead a pluralist perspective is taken, and it is
assumed that “it is more likely that stakeholders will have different, and sometimes conflicting
views on an evaluated program” (p. 199). It is argued that most evaluations allow certain criteria
and values to dominate at the exclusion of other criteria, values, voices and people. The focus of
the case study is a program in which conflict arose between differing stakeholders. Two options
for addressing stakeholder conflict are given, compromise and “new synthesis.” Compromise is
a process of bargaining and exchanging of views to reach a consensus. A new synthesis is
characterized by the formation of a solution that had not been conceptualized before the
evaluation negotiation began. The outcome of this negotiation process was consensus and a
heightened personal and mutual understanding.
Michalski and Cousins (2001) explore differences in stakeholder perceptions concerning
the purposes, processes, and consequences of evaluation within an organization. The views of
different stakeholder groups were divergent, with most of the difference related to the job role of
a stakeholder in the organization. The authors conclude that for “training evaluation to have
fuller importance, assumptions about stakeholder homogeneity in training evaluation practice
need to be questioned more thoroughly” (p. 50).
Several evaluation approaches have emerged that recognize the importance of
considering multiple stakeholders viewpoints during the evaluation process. These approaches
strive to create a more democratic evaluation process and to represent as many interests as
possible. These approaches include responsive evaluation, participatory/collaborative
evaluation, and empowerment evaluation.
42
Responsive evaluation focuses on program stakeholders, including program participants,
staff and other audiences (Russ-Eft & Preskill, 2001). The key to this evaluation approach is to
identify individual stakeholders concerns about the program and address the concerns through
the evaluation process. The issues raised by the program stakeholders become the guiding
framework for the evaluation, instead of formal program objectives. Participatory or
collaborative evaluation uses a democratic decision making process as a way allow stakeholders
to reach an understanding about program effectiveness. Empowerment evaluation works with
program participants to help them improve their own programs through self-evaluation and
reflection. The goal of this process is to empower the stakeholders to improve program
functioning.
In summary, it has been established that educational programs are created through a
political decision making process which has an influence on the evaluation of the program.
When conducting an evaluation it is important to recognize that different stakeholder groups
exist with varying interests in the program. While this divergence of view points has been
established, only a few evaluation models exist that address multiple stakeholder groups. Yet,
little attention has been given to resolving differences in stakeholder perspectives when creating
the program theory. The stakeholder approach to building program theory rests on the notion
that any differences among stakeholders can be easily resolved through a discussion process.
Chen (1990b) states,
this process should facilitate agreements on the intended program theory in terms
of program resources, activities, intended outcomes, important side effects, and
assumed causal links among resources, activities, and outcomes. After the
differences are resolved, a formal program theory can be completed. (p. 11)
43
This assumption is problematic, but necessary for theory-driven evaluation to proceed.
Even if multiple program theories are developed and tested, all possible stakeholder viewpoints
will not be represented in the evaluation process. It will be the role of the evaluator to make
decisions about whose views are represented and whose views are not.
Summary
This literature review has addressed four major topics: evaluation in HRD, theory-driven
evaluation, program theory development and the politics of training and evaluation. The review
of the literature of evaluation in HRD revealed that training plays an important and costly role in
organizational functioning, and given the importance, evaluation is a necessary part of the
training process. However, a behavioral objectives/outcomes focused approach has dominated
the training evaluation literature. This focus on program outcomes has essentially created a
black box around the training process. In light of this restricted view, the field should look to the
evaluation literature of large-scale social, educational, and health programs. Specifically, this
literature has utilized the concept of theory-driven evaluation and related concepts. The use of
these types of evaluation models holds promise for advancing training evaluation.
Review of the theory-driven evaluation literature was used to define theory-driven
evaluation as an evaluation approach in which the underlying assumptions about program
functioning are captured in a program theory. This program theory is then used as a guide to the
evaluation, and can be tested to better understand how and why the program works. Closely
related is the literature review of the program theory development process. The program theory
is the central characteristic of theory-driven evaluation and can be developed through input from
stakeholders, through a review of relevant literature or relevant social sciences theories, or
through a combined approach in which social science theory and stakeholder input are merged.
44
An alternative to creating an overarching program theory is to create several program theories.
Given the confusion surrounding the program theory development process, the political nature of
educational programs and evaluation were considered. The final section of this literature review
focused on the political nature of the program planning and evaluation process. It was
established that educational programs are created through a political decision making process
which has an influence on the evaluation of the program. When conducting an evaluation it is
important to recognize that different stakeholder groups exist with varying interests in the
program. While this divergence of view points has been established, only a few evaluation
models exist that address multiple stakeholder groups.
45
CHAPTER 3
METHODOLOGY
Before theory-driven evaluation can be incorporated into the repertory of evaluation
approaches utilized by HRD practitioners, issues surrounding the development of program theory
must be addressed. The theory-driven evaluation literature is explicit about using both formal
social science theory and stakeholders’ informal theory in the development of a sound program
theory. However, little attention is given to incorporating multiple stakeholders’ viewpoints into
one theory. The purpose of this research was to explore the ways that multiple stakeholders’
interests are represented in the program theory building process which can be used when
conducting a theory-driven evaluation.
The objective of this chapter is to outline the methods used to accomplish my purpose.
Included is an outline of the design of the study, case and sample selection, data collection
methods, data analysis, validity and reliability, and researcher bias. The results of a pilot study
are also be given.
Design of the Study
A qualitative case study was used in this research. Yin (1994) defines case study as “an
empirical inquiry that investigates a contemporary phenomenon within it’s real-life context,
especially when the boundaries between phenomenon and context are not clearly evident” (p.
13). Bogdan and Biklen (1998) also offer a definition of case study that states it “is a detailed
examination of one setting, or a single subject, a single depository of documents, or one
particular event” (p. 54).
46
Attempts to define the case study method usually involve naming what is unique about
the research design (Merriam, 1998). That is, understanding the key characteristics of case study
may help in the definitional process. Merriam (1998) states that the “single most defining
characteristic of case study research lies in delimiting the object of study, the case” (p. 27). The
notion of bounded system is useful in defining what is meant by a “the case.” That is, the case
should be “a thing, a single entity, a unit around which there are boundaries” (p. 27). The
boundedness of a system can be understood by considering how finite that data collection would
be for one context. For example, the researcher may consider if there is a limit to the number of
people involved who could be interviewed.
Merriam (1998) names three “special features” of the case study method. First, the case
study is particularistic. This refers to the study of a particular system. By specifically targeting
one event or phenomenon, the case study method is able to address practical problems,
questions, situations, or occurrences (Merriam, 1998). Furthermore, by studying one particular
system, many aspects of the case can be uncovered including “the nature of the case, its
historical background, the physical setting, other contexts, including economic, political, legal,
and aesthetic” (Stake, 1994, p. 238).
Second, the descriptive feature of case study research refers to the rich description which
results from the case study. This means the results include a “complete, literal description”
(Merriam, 1998, p. 30) of the case under study. This descriptive output takes the form of
detailed text, supported by documents, quotes, and artifacts. Third, the heuristic feature of case
study method means that “case studies illuminate the reader’s understanding of the phenomenon
under study” (Merriam, 1998, p. 30). For instance, the reader of a case study may find new
meaning in a phenomenon, or confirm what is known.
47
Within the qualitative tradition, case studies are different from other types of qualitative
research because of the focus on a single, bounded system (Merriam, 1998). This may take the
form of a program, an event, an individual, a group, an intervention, or a community.
Additionally, knowledge produced from case study research is unique in four ways. Because
experience is the source of knowledge, the knowledge produced is more concrete. Second,
because case study knowledge comes from experiences, the knowledge is specific to the context
in which the events occurred. Third, readers of case study are able to interpret the study through
their own experiences, thus producing new generalizations. Finally, because each reader can
have a different reference population in mind when interpreting a case study, the generalizations
are spread to many populations (Merriam, 1998).
These unique aspects of case study research were the primary reason that this study
design was chosen. The focus of this research was one educational program operating within an
organizational setting, and qualifies as a bounded system. That is, there were a finite number of
people that could be interviewed within the boundaries of the program. The context specific
nature of the knowledge produced also fits well with the guiding theoretical framework, theory-
driven evaluation.
Case Selection
“Perhaps the most unique aspect of case study in the social sciences and human services
is the selection of cases to study” (Stake, 1994, p. 243). Setting criteria for selecting the case can
make this process easier and insure that an appropriate case is selected for study. Two criteria
were established to reflect the purpose of the study, and were used as a guide to insure that the
case selected was pertinent to the research purpose. The case was chosen based on the following
criteria:
48
1. the case is located within an organizational setting, in the private sector;
2. the case is an employee training program;
The first criterion, the case is located within an organizational setting, is pertinent to the
purpose of the study. Theory-driven evaluation has been used in the public sector, but the
purpose here is to investigate its utility in the private sector. The second criterion is that the case
be an employee training program. Despite the fact that HRD covers various elements of
organizational development, the focus of this research was a learning intervention. By
specifically naming a training program, various organizational development programs were not
considered as possible sites for this research. Based on these two criteria, a new employee
orientation program at a 196 bed hospital was chosen. A detailed description of the case is
included in chapter four.
Selection within the Case
Within the case, purposeful sampling was used to guide participant selection. Unlike
probability or random sampling, purposeful sampling is used when “the investigator wants to
discover, understand, and gain insight and therefore must select a sample from which the most
can be learned” (Merriam & Associates, 2002, p. 61). More specifically, the type of purposeful
sampling used was network sampling. The initial contact was made with the Director of the
Training Department. Network sampling method allowed other potential participants to be
identified through the discussions with participants.
To further guide the selection of stakeholders to be interviewed, the framework presented
by Cervero and Wilson (1994) was used. This framework names five categories of stakeholders
that may have an interest in any educational program, including: learner, teacher, planners,
49
institutional leadership, and the affected public. Multiple participants from each of these
categories of stakeholders were selected through network sampling.
Because the nurses are the only group of employees to attend all five days of training,
they were given more attention in this research. Choosing to focus on the nurses had several
implications for the selection of participants for this study. First, one of the participants selected
in the category of learner was a nurse. Second, the participants selected in the categories of
teachers, planners, institutional leadership, and affected public had an interest in all three
sections of the orientation program. For the purpose of this study the affected public was
represented by interviewing a patient representative. An actual patient would have no
information about the program and this compromise was made to access this category of
stakeholder. The following table contains the interview participants:
Table 3.1. Interview Participants Category Interview Participants Learners New Nurse
New Non-Nurse Employee
Teachers Two Orientation Trainers
Planners Director of Education and Training
Resource Development Coordinator
Staff Development Educator
Institutional Leadership Manager of Employee Relations
Affected Public Patient Representative at Hospital
50
Data Collection
“The important aspect of case study data collection is the use of multiple sources of
evidence-converging on the same set of issues” (Yin, 1993, p. 32). Unlike many other types of
research, case study does not rely on one specific data collection or analysis method. With a
case study, many different types of data gathering techniques can be used. Case study method is
not only a data collection technique or a design feature, but instead it is a comprehensive
research strategy (Yin, 1994). When using a case study approach it is necessary to rely on
multiple sources of evidence. The multiple sources of data provide many perspectives to clarify
meaning, or verify the repeatability of an observation (Stake, 1994). This case study research
employed interviews with stakeholders, observations of the program, and collection of relevant
documents.
Interviews
Kvale (1996) defines an interview as “a conversation that has a structure and a purpose”
(p. 6). Interviews can be conceptualized as unstructured, semi-structured, or structured (Davies,
1999). This research employed semi-structured interviews, defined as an
interview with some sort of interview schedule: it may be as structured as a set of
written questions or it may be a very informal list, perhaps memorized, of topics.
However, in contrast to structured interviews, researchers may alter the wording
and order of these questions, perhaps omitting some that seem inappropriate; they
may introduce new topics and supplementary questions not included on the list.
(Davies, 1999, p. 95)
Semi-structured person-to-person interviews were conducted with stakeholders in five
categories: learners, teachers, planners, institutional leadership, and the affected public. (See
51
Appendix A for Interview Guide). The conditions in the learner interviews were slightly
different, as they were interviewed about one event. All other stakeholders were interviewed
about the ongoing educational program. Additionally, learners were provided with an orientation
agenda, creating a prompted recall situation. Interviews provided the primary data source and
were used to answer the research questions. The interviews were audio tapped and transcribed.
Observations
Observations were made of the five-day orientation program, consisting of the general
orientation, clinical orientation, and nursing orientation. Observations served as a secondary
data source. While the observational data was not used to directly address the research
questions, it was used to gain an understanding of the case.
The role of the researcher when observing can range from complete participant to
complete observer. At one end of the continuum, the researcher is a member of the group being
studied, at the other end the researcher is hidden from or unknown to the group. As an observer
in a lecture-focused classroom, I was able to assume a role of observer. The participants knew
my role as an observer. In this role, the primary objective was to gather information;
participation in the group is secondary (Merriam & Associates, 2002).
Observations were recorded through the use of field notes and expanded field notes
(Davies, 1999). Field notes were taken during the observational period. Expanded field notes
were also written after each observational period to capture information about the experience not
reflected in the field notes.
The observation focused on six elements of the classroom, the physical setting,
participants, activities and interactions, conversations, subtle factors, and my behavior (Merriam
& Associates, 2002). Observations of the physical setting focused on the physical characteristics
52
of the classroom, including the objects, resources, technologies, and the kinds of behavior the
setting is designed for (Merriam & Associates, 2002). Observations of the participants focused
on who is in the classroom and their roles. Activities and interactions were observed for
sequence, norms, and duration of activities. Observation of conversations focused on content,
the speakers, and the listeners. Subtle factors such as unplanned activities and nonverbal
communication were also noted. Finally, my role in the classroom and the impact of my
presence on events was documented.
Documents
Similar to the observations, documents provided a secondary source of data. The
documents collected were used to gain a fuller understanding of the case. Documents were
collected through two primary means. First, documents were collected during the observation
period. Any handouts or other materials given to the employees during the educational activities
were collected. Second, documents were collected during interviews. Interviewees were asked
for any materials, documents or “official records” concerning program. Each document
collected was inventoried. The inventory information for each document included: item name,
description, use, title, creator, date, site found, and significance of item.
Data Analysis
The first step in the data analysis process was to create a detailed description of the case
based on observations and documents. The case description provides an overview of the
program being studied and the context in which the program exists. The second step in the data
analysis process was to address each of the research questions.
In order to address research question one, the constant comparative method was used to
analyze the interview data. While the constant comparative method originated with the grounded
53
theory approach to qualitative data collection and analysis, the method is used widely in all types
of qualitative research (Merriam & Associates, 2002). When using the constant comparative
method, “units of data deemed meaningful by the researcher are compared with each other in
order to generate tentative categories and properties” (Merriam & Associates, 2002, p. 143).
Data analysis began with open coding within each interview, in which data was dissected and
given a name or code (Strauss & Corbin, 1990). The categories represented elements of the new
employee orientation program which the stakeholder thought significant to program functioning.
The related codes were grouped to form categories. The second step was to reduce the number
of categories through the process of axial coding. During axial coding the categories identified
through open coding were connected, resulting in fewer main categories (Strauss & Corbin,
1990). Specifically, like categories were combined and temporal linkages between the categories
were identified. The result of this data analysis process was a program theory for each of the
nine stakeholders. For an example of a developed program theory, see the pilot study findings
(Appendix C). In addition to the program theory, the organizational role of each of the
stakeholders was identified.
To address research question two, the program theories of the nine stakeholders were
compared so that similarities could be identified. “Like elements” or “similarities” were defined
as similar ideas expressed by at least two stakeholders. These similarities or like elements were
combined to form one program theory.
Once all of the similar elements, or shared ideas had been extracted form the nine
program theories, the unique elements were easily identifiable. Unique elements were the
concepts or ideas which were only expressed by one stakeholder. At the end of the data analysis
process the following data had been specified: nine program theories, organizational role of each
54
stakeholder, one program theory representing the like ideas expressed by stakeholders, and a list
of elements which were unique.
Pilot Study
A pilot study was conducted in which the three types of qualitative data collection
methods named above were pilot tested, as well as the data analysis techniques. Initial entry into
the site was made through contact with the Director of Education and Training at the hospital,
who identified an educational program to study and provided contact information for the
Resource Development Coordinator, who oversees the orientation program.
Observation
The first data collected were two one-hour observations. The observations were
conducted on two consecutive days of the general orientation program. The dates and times of
the program were identified through contact with the Resource Development Coordinator. As an
observer in a lecture-focused classroom, I was able to assume the role of observer, and take
extensive field notes (Davies, 1999). Expanded field notes were also written after each
observational period to capture information about the experience not reflected in the field notes.
Interview
Three interviews were conducted with administrators of the orientation program. A
semi-structured interview guide, composed of five questions was used (see Appendix B). The
interview guide was designed to include a variety of types of interview questions, including
introducing questions, direct questions, and indirect questions. Probing and interpreting
questions were also used throughout the interview as needed (Kvale, 1996). Each interview was
between forty-five minutes to one hour in length. The selection of interview participants was
done through network sampling. The first interviewee, the Resource Development Coordinator,
55
was identified during the site entry process with the Director of Education and Training at the
hospital. The other two interviewees were identified during the first interview. The interviews
were audio taped and transcribed.
Documents
Three sources were used to collect artifacts. First, publicly available information about
the hospital was collected from the Internet. I searched not only the hospital’s website, but also
did a broad search to identify any other information about the hospital from other sources.
Through this process I found several documents directly from the hospital’s website, and found
comments about the hospital from a sponsor organization. Archival data was also collected
during both of the one-hour observation periods. This data was in the form of materials that
were handed out to the program participants. Finally, two of the three interviewees shared
program material with me during the interview process. These three data sources provided
useful information about the program that was not available from other data sources. Each of the
artifacts collected were inventoried. The inventory information collected for each artifact
included: item name, description, use, title, creator, date, site found, and significance of item.
Methods and Findings
The constant comparative method was used to analyze one of the interviews. The coding
process resulted in a visual representation of the program theory as described by this interviewee
(see Appendix C). The program theory has two main trajectory points, joint commission and
human resources standards, and corporate mission and values. Intermediate variables link these
two trajectory points to three main outcomes of the program, compliance with corporate policies
and regulating bodies, employee prepared for the job, and customer service/patient satisfaction.
The elements of the program supporting these outcomes were also included.
56
As a result of this pilot study the interview guide was changed. While the interview
guide used in the pilot study did elicit some information about the program, a modified interview
guide was developed to better target the research questions. The pilot study also confirmed that
network sampling will be an effective means to identify interview participants.
Validity and Reliability
In order to improve the quality of this qualitative study, attention has been given to the
internal validity, external validity, and reliability of the findings. Internal validity is the extent to
which the findings of the study represent reality and reliability is the extent to which the findings
of the study can be replicated (Merriam & Associates, 2002). Several elements have been built
into the research to improve both the internal validity and reliability of the findings. First,
triangulation was used. Triangulation is a process where multiple data sources are used to
support the findings of the research. In addition to using multiple data sources, triangulation can
be achieved through the use of multiple investigators, multiple theories, or multiple methods to
confirm findings (Merriam & Associates, 2002). To incorporate triangulation into this research
study, data from multiple sources were collected, including conducting interviews and
observations, and collecting archival data. The information revealed in the interview sessions
was checked against observations and documents collected. The use of peer review also
strengthened the internal validity of the research findings. The committee supporting this
dissertation research reviewed and commented on the findings (Merriam & Associates, 2002).
Reliability was also addressed through the use of an audit trail and member checks. An audit
trail is a detailed description of the exact process that was used to derive the findings of the
research (Merriam & Associates, 2002). Finally, member checks were done with two interview
participants to strengthen the internal validity of the findings. Interview participants were
57
presented with preliminary findings of the research so feedback could be received. Member
checks affirmed the findings of this research.
Generalizing the results of one study to other sites is way of questioning the external
validity of the study. In qualitative research, the goal is to understand the particular in depth, not
what is generally true of many. Furthermore, small non-random samples cannot be statistically
generalized (Merriam & Associates, 2002). However, if it is done, it is the reader or the user of
research that generalizes the results to other settings. To aid the user of this research in
generalizing to other settings, thick rich description was incorporated to give the users of this
research the information needed to decide if the findings can be applied to their contexts.
Researcher Bias and Assumptions
As a graduate student, I had the opportunity to participate with a team conducting an
evaluation of a county mandated parenting and divorce education program, which was part of a
larger divorce initiative. We used a modified version of Kirkpatrick’s four-level evaluation
approach. After months of data collection and analysis, we presented our results to the county,
with hopes of providing useful data about continuing the program. However, the data on
participant reactions and learning provided little help to the decision makers. Clearly a more
detailed model of program functioning was needed, especially in how the program supported
other parts of the divorce initiative.
As a result of these negative experiences with other evaluation approaches, I went in
search of an evaluation approach that I believed could provide sound evaluation knowledge.
Consequently, I have chosen a framework, theory-driven evaluation, to guide my research.
Theory-driven evaluation appeals to me because the context in which the educational program
occurs is emphasized. As I read and learned more about my topic, I realize that it is not perfect.
58
I can see flaws in the logic of this approach to evaluation, yet it continues to be a major factor in
my research. I have to investigate this evaluation approach with a critical eye to use it
effectively in my research. That is, I need to be open to the possibility that this evaluation
approach does not work in the real world as it seems to on paper. Because I chose this
evaluation approach, I obviously believe that it has potential to yield valuable evaluation
information. However, my optimism should not blind me to the possibility that flaws in this
method exist.
59
CHAPTER 4
CASE DESCRIPTION
Based on the case selection criteria named in chapter three a case was chosen for
research. The case is an educational program for new employees at a hospital. This chapter
explores the case, including a description of the organizational context and a detailed description
of the educational program central to the case study. The data provided in this chapter are based
on observations and document review.
Organizational Context
The site of the research is a 196 bed hospital in the Southeastern United States. The
hospital is located in a three-county metropolitan area of 126,000 people. The hospital is a
private, non-profit, self-supporting acute care network, which is part of a larger Catholic
organization which includes 32 hospitals, 41 free-standing and hospital-based skilled nursing
facilities, 22 residential facilities, 3 free-standing behavioral health facilities, and numerous
ambulatory and community based health services with more than 14,000 beds. Within the
hospital, one educational program will be considered, the new employee orientation program.
The program is operated by the division of Human Resources and Support Services (See
Figure 4.1). Under this division, direct program operation is overseen by the department of
Education and Training. The director of the department, the resource development coordinator,
and the staff development educator each has a role in the design and operation of the program.
60
Description of the Program
The educational program at the center of the case study is a new employee orientation
program at the hospital. The orientation program is an ongoing program which takes place each
month. The orientation program is composed of three parts: general orientation (two days),
clinical orientation (one day), and nursing orientation (two days). While the total orientation
program is five days in length, not all employees have to participate in the full program. All
incoming employees are required to attend the first two days of the general orientation program.
Day three, clinical orientation, is only for employees with direct patient contact. The final two
days of orientation are nursing orientation and is designed only for new nurses. The general and
clinical orientations are held twice monthly, but the nursing orientation is only offered once a
month (see Appendix D for Observation Schedule).
This case study is based on the new employee orientation program that was held in May
and June, 2003. Observation of general and clinical orientation at the hospital was consecutive,
taking place on May 19, 20, and 21, 2003. Observation of nursing orientation took place on June
BOARD OF DIRECTORS
COO/CFO
Vice President, Human Resources
Vice President, Professional Services
Executive Director, Community Care Services
Vice President, Medical Affairs
Vice President, Mission
Vice President, Nursing
Vice President and General Council
Figure 4.1. Organizational Chart
61
5 and 6, 2003. The following is a day-by-day account of the events that took place during the
orientation program with the program agenda for each day included.
Notes on Agenda and My Role
Note that the physical agenda given to learners (the one included in this description) was
only a loose guide to the actual activities which took place during the course of the orientation
program. The first two days of general orientation closely followed the actual agenda. In
clinical orientation, day three, all the topics on the agenda were covered, but not in the order
presented. Finally, the nursing orientation agenda was radically different from the actual events.
As an observer unfamiliar with many of the topics, I was often unsure about the topic of the
presentation and how the topic related to the agenda.
As an observer in the orientation program, I fluctuated between total participation in the
program with the new hospital employees and observing activities in which they participated. I
did not participate in many of the small group activities, but I did listen in on the discussions
learners at my table were having. I did participate in the group anytime we did introductions,
and informed the group and the instructor of my role as observer. However, in several sessions
no introductions were done, so the presenter had no way of knowing that I was not a new
employee.
I feel like I established very good rapport with the learners in the general and clinical
orientation program. I sat at the table with the same learners for three consecutive days. During
this time there was opportunity to talk and get to know one another. I had lunch with different
learners each day, which also allowed time to talk. I do not feel that I was able to establish as
much rapport with the learners during the nursing orientation for several reasons. First, many of
the learners had been together all week during general and clinical orientation, and I joined the
62
group on day four, after many connections had already been established. Second, nursing
orientation moved at a much quicker pace and there was less time for social interaction between
sessions. Overall, I was treated as if I were a learner in the program with the exception of
participating in some of the small group activities.
Day One
The first day of general orientation began on Monday morning at 8:00 a.m., in an
education classroom at the hospital (see table 4.1). There were seventeen learners attending the
program and all were female. The learners were seated around five small circular tables. As the
learners arrived, Sally, the Resource Development Coordinator, who also acts as the coordinator
of the general orientation program, greeted people.
Sally opened the orientation program with a brief overview of basic information such as
the need for learners to sign in, how to obtain an employee name badge, and the location of
bathrooms and exists. She then proceeded to review the orientation program schedule and other
materials that were part of a packet that learners received as they arrived. Many of the learners
had to leave to go to the Human Resources office to pick up their name badge and to clock in.
The initial welcome and review of materials took approximately thirty minutes.
The first session of the orientation program covered the mission and values of the
hospital. The lecture, given by Sally, focused on the history of the hospital and the impact being
sponsored by a religious organization had on hospital functioning. A video was shown, which
was created by the hospital’s sponsor organization, outlining the values of the sponsor
organization which were expected to be carried out by hospital employees. After the video,
Sally reviewed the values stated in the video and then broke the learners into small groups by
table. In small groups, we discussed ourselves, our jobs at the hospital, and named the values
63
that we most related to. After the group exercise, Sally read the mission statement for the
hospital and closed the session with an orientation prayer.
The topic of the second session was the vision and strategic plan, and the speaker was
listed as the President/CEO of the hospital. However, the CEO was not available, so Sally
continued to lead this session. A video was shown which outlined the renovations that are taking
place at the hospital and explained the organizational strategy and growth. The video was
followed by a description of all of the resources that the hospital offers the community, such as
long term care and wellness facilities. During the discussion, three hospital Vice Presidents
stopped by to introduce themselves to the new employees, the Vice President of Nursing, the
Vice President of Professional Services, and the Vice President of Medical Affairs.
Table 4.1. General Orientation- Day One Monday May 19, 2003
8:00 - 9:00 Mission & Values
9:00 - 9:30 Vision & Strategic Plan
9:30 - 9:45 “This is [the hospital]” Video
9:45 - 11:30 Customer Relations
11:30 - 12:15 Lunch
12:15 - 1:00 Ethics/Advance Directives
1:00 - 2:15 Culture of Safety
2:15 - 2:30 Break
2:30 - 3:00 Abuse & Neglect
3:00 - 4:30 Benefits Introduction
64
The third session of orientation was listed on the agenda as “This is [the hospital]” video,
and Sally continued as presenter. In this section, a brief video was shown, which introduced the
hospital’s specialty units including the women’s center, pediatrics, orthopedics, the
gastrontology unit, and the neurology unit.
After a break the orientation session resumed with a new presenter, the Director of
Patient/Guest Relations, to talk about customer relations at the hospital. She introduced herself
by telling stories about herself and used story telling throughout her presentation. The first topic
covered was patients’ rights, and the mandate that all patients be informed of their rights. She
then moved on to the topic of customer service. The presentation used the theme “take PRIDE in
your work,” with pride being a descriptor used to describe the actions expected of employees.
She then asked learners to share stories about experiences they have had with either good or poor
customer service in the past. She also shared a story about an experience with excellent
customer service. She then gave an overview of the individuals and groups considered
customers of the hospital, and the ways in which customer satisfaction data is collected. “The
Customer” program was also introduced to the learners. This program was described as a seven-
week customer service program which all employees must complete. The learners were
introduced to the basic concepts during orientation, including the six customer service steps,
greet, value, ask how to help, listen, help, and invite back. The presenter explained that twenty-
five percent of the performance appraisal for employees at the hospital is based on customer
service. After more lecture on “The Customer” program, a behavior styles inventory was
completed by learners. The presenter reviewed each of the four possible behavior styles used in
the inventory and talked about the importance of understanding the behavior styles of coworkers.
65
The final segment of the customer relations session focused on phone etiquette. A short video
was shown about how to handle an irate caller. This was the last session before lunch.
When the orientation program resumed, the Director of Patient/Guest Relations again was
the presenter, but now was representing the ethics committee. The topic of the session was
ethics and advance directives. She began with a lecture on ethical/religious directives for
healthcare and outlined the role of the ethics committee at the hospital. After telling two stories
about rulings by the ethics committee, she presented each table with a case study and asked them
to act as an ethics committee and make a recommendation. The learners seemed very engaged in
this activity and then reported their group decisions to the large group. The presenter then
reviewed the decision the ethics committee actually made in this situation.
The next session on the orientation schedule was titled Culture of Safety, and covered a
wide variety of topics. The presenter was the Officer of Performance Improvement, Risk
Management, and Corporate Compliance and he began by talking about performance
improvement. His presentation was lecture based and used a “dashboard” metaphor to describe
the performance improvement processes at the hospital. Overheads were used including tables,
flow diagrams, fishbone diagrams, and bar charts to outline successful performance
improvement activities. The next topic was risk management, which focused on filing
occurrence reports. Information about what types of problems to document, how to complete the
report, and the importance of talking with the risk management staff if a problem does occur was
given. The presenter then began to discuss corporate compliance. A booklet titled “Corporate
Compliance Plan” was used as a guide during this discussion. The booklet covered topics such
as patient referrals, billing and cost reporting, conflicts of interests, fund-raising, record keeping,
and political activities. Each of the topics listed in the booklet were addressed and then the
66
learners were asked to sign a statement indicating that the corporate compliance plan summary
had been received.
The final section of the Culture of Safety session of orientation covered the privacy
policy of the hospital and was presented by the Privacy Officer. This section had recently been
added to the orientation agenda to cover new national standards on protected health information,
also referred to as HIPPA. The presenter described the types of information protected by
HIPPA, such as the name or other identifying information of the patient, the treatment, and the
payment. The presenter discussed practical ways for employees to comply with the policy such
as shredding key documents, protecting computer passwords, and using a computer screen filter.
This presentation generated more questions from learners than any other on this day. The
session ended after the scheduled time so that all questions could be addressed.
After a short break the orientation program resumed with a session on abuse and neglect,
presented by a case manager at the hospital. She used a lecture presentation style to identify “red
flags” that could indicate domestic violence, elder abuse, and child abuse, including neglect,
sexual abuse, and emotional abuse. A PowerPoint presentation was used to guide the session,
yet the presenter used stories of events at the hospital to demonstrate the importance of reporting
suspicious events.
The final two presentations of the day covered employee retirement and benefit plans.
Only full-time employees were asked to stay for this session, resulting in 12 learners remaining.
The first presenter talked about the importance of investing in a retirement plan. She cited the
unreliability of Social Security and the growing credit card debt of older Americans as reasons to
invest in a retirement plan. She reviewed each of the retirement plans offered by the hospital,
which are based on the percentage of income invested, and gave a packet of information
67
outlining the retirement options. The presenter identified herself as an outside consultant to the
hospital, and stated that she is available to meet with employees one-on-one to set up a
retirement plan. The final presenter of the day was from Human Resources and talked about
employee benefits. She outlined the differences between a PPO and an HMO and discussed the
strengths and weaknesses of each. This presentation generated several questions from the
learners. Next, she discussed basic life insurance and supplemental life insurance. The presenter
encouraged the learners to think about their options during the evening and to arrive at 8:00 a.m.
the next morning to sign-up for a benefits package.
Day Two
The second day of general orientation began informally at 8:00 a.m. (see table 4.2). Only
full-time employees wishing to sign-up for a benefits package were in attendance, all other
employees were told to arrive at 9:00 a.m. There were eight employees at the opening session,
all were female. The presenter from Human Resources leading yesterday’s closing session was
again in the room. She was assisting individual learners with paperwork regarding the benefits
plan, and was aided by another representative from Human Resources. By 8:30 a.m. all of the
learners present had completed their paper work, yet the next session did not begin until 9:00
a.m. Some of the learners left the room, but the learners at my table stayed and talked.
The next session was titled infection control and began at 9:00 a.m. At this time there
were eighteen learners in the room. The presenter arrived and mentioned that she was a
substitute presenter taking the place of the regularly scheduled presenter. She handed out a
packet of information and asked the learners to sign a statement titled “OSHA’s Bloodborne
Pathogen and Tuberculosis Standard.” As the learners completed this, she left the room to get a
68
Table 4.2. General Orientation-Day Two Tuesday May 20, 2003
8:00 – 9:15 Benefits
9:15 – 10:30 Infection Control
10:30 – 10:50 Electrical Safety
10:50 – 11:00 Break
11:00 – 11:45 Ergonomics
11:45 – 12:30 Lunch
12:30 – 1:00 Home Care Services
1:00 – 4:00 Environmental Control
4:00 – 4:30 Evaluation
video. The video was titled “Blood Bugs,” and primarily discussed “PPE” but never defined the
acronym. After the video, the presenter demonstrated the types of infection control materials
used at the hospital including, disinfectant, cloths, hand wash, masks, goggles, red bags and
gowns. The presenter then briefly reviewed material in a packet she handed out. During her
discussion she used jargon such as “N95” and “BSI”. Many of the learners were exchanging
confused glances. She finished the session by passing around an example of an infection control
sign used in the hospital. The session ended at 10:15 a.m., which was fifteen minutes earlier
than scheduled.
The next session did not begin until 10:40 a.m. and most of the learners used the time to
talk with the other learners at their tables. The topic of the session was electrical safety, and the
presenter was from biomedical services. After a brief introduction, the presenter showed a video
pertaining to electrical plugs, equipment, cords and how to check for problems. After the video,
69
the presenter described how to put in requests for service, and how to get in touch with other
people at the hospital responsible for electrical problems. This session ended at 10:55 a.m.
At this time Sally, the Resource Development Coordinator, arrived to check on the
progress of the program. She started the next video at 11:00 a.m. before the next presenter
arrived. The topic of the session was ergonomics, and the video was about protecting your back
on the job. The presenter arrived during the video, yet she did not introduce herself once the
video ended. She handed out a sheet showing back exercises, and demonstrated each exercise
listed on the sheet. At the close of this session the learners went to lunch.
The orientation program resumed at 12:30 p.m. and focused on the topic of Home Health
Care, with a presenter from this department. The presenter used PowerPoint and lecture to give
the history of home health, the scope of home health care, and hospice. She only used 15
minutes of the 30 minutes allotted on the schedule. Once again, the learners have time to talk
among themselves.
At 1:00 p.m. a representative from the Public Safety division arrived to speak on the topic
listed as environmental control. He began the presentation with a description of the role that
Joint Commission plays in the policy and plans that the hospital has for environmental control.
He then reviewed hospital emergency codes, and discussed how disasters have changed the way
in which the hospital handles emergencies. He gave employees specific instructions on what to
do if a code is announced, the codes included fire, community disaster, baby abduction,
bio/chemical radiological event, bomb threat, violent situation, medical emergency, and tornado.
He gave examples from events that have taken place at the hospital and at other hospitals.
The presenter then began to talk about OSHA requirements for the hospital, which
focused on the hazardous material communication standard, or employee right to know. He
70
showed an example of a reference book, which is located on each floor of the hospital. The
reference book included chemical spill response, hazardous material labeling, and instructions
for exposure to chemicals. The learners are then asked to sign a verification sheet, indicating
that all OSHA materials had been covered. The environmental control session continues with a
review of security procedures at the hospital, including the occurrence of alcohol/drugs,
weapons, suspicious persons, or theft in the hospital. The presenter also reviewed parking for
hospital employees, radiation safety, and workplace violence.
At this point in the presentation, a video about the events of September 11, 2001 was
shown. The video highlighted scenes of events, people, and victims at all three sights of disaster,
and ended with Bible verses displayed on the screen. The presenter stated that the purpose of
showing the video was to remind employees that disaster can still strike and that it is the
hospital’s responsibility to be prepared. After the video, Sally began a new session on
emergency management. The first part of the presentation was a lecture on the types of disasters
that the hospital is prepared to handle. This included a review of hazardous materials such as
materials that are dangerous due to exposure, materials that are contagious, and materials that are
contaminants. She also talked briefly about what to do if the hospital resources are exhausted.
The specific codes for each type of hazardous material were presented, and an overview of how
the hospital will respond in each of these instances was covered. The main emergency response
team, the HAZMAT team, was introduced and their role was briefly outlined. The
decontamination showers were reviewed as a way to handle a hazardous material incident.
At 3:00 p.m. the lecture ended and the room was opened up to include the adjoining
room. The adjoining room contained the equipment used during the decontamination process.
Sally and two members of the HAZMAT team were present to assist in the demonstration. The
71
session began with a demonstration of how the decontamination suits are put on by the
HAZMAT team, with one of the team members being fully dressed in the decontamination gear.
A demonstration of how the decontamination showers are assembled was also included. Once
the shower was constructed, a lecture on how a person would proceed through the
decontamination process was given. At the end of the demonstration, the learners were given the
opportunity to assemble the shower. The learners were divided into two groups, and each group
assembled and disassembled the shower.
At 3:30 p.m. the final session, live evaluation, began. During this time, Sally asked
learners for feedback on what worked and what did not work during the two day orientation
process. She made a list on the white board of the comments that the participants made.
Comments were generally positive, but suggestions for improvement were also given. At the
end of the session, Sally offered to take learners on a tour of the hospital. This was an optional
part of orientation, and none of the learners stayed for the tour.
Day Three
Day three of orientation was Clinical Orientation, and only employees with direct contact
with patients were required to attend. There were 13 learners in attendance, all were female.
While an agenda was given for the day, the presentations did not closely follow this agenda (see
table 4.3). The agenda indicated that orientation began at 8:00 a.m., but learners were instructed
to be there at 9:00 a.m. The first presenter was from the Performance Improvement Division,
and indicated that she worked closely with the Joint Commission. She introduced herself giving
a brief work history and current job overview. She then asked learners to give their names and
work locations. The first content presented covered safety levels of falls. Three levels of falls
were described to the learners. The second topic covered was the use of restraints on patients,
72
Table 4.3. Clinical Orientation-Day Three Wednesday May 21, 2003
8:00 – 8:30 Pain Management
8:30 – 9:00 Restraints
9:00 – 9:30 Age-Appropriate Care
9:30 – 9:45 Break
9:45 – 11:45 Infection Control
11:15 – 12:00 Lunch
12:00 – 4:00 Basic Life Support (CPR)
indicating that the goal of the hospital is to maintain the rights and dignity of patients. The Joint
Commission standards for the use of restraints were reviewed, including documentation, and
time frames. The next topic covered was pain management. The presenter from the
Performance Improvement Division continued to lead the session. Once again, she referenced
the Joint Commission, this time discussing standards for pain management. These standards
include education of hospital staff concerning the pain management process, and specifically the
education of new employees. She began with patient’s rights about pain, and then discussed the
biological and cognitive issues of pain. Biologically, she reviewed the nerve fibers and spinal
cord as the pathways for pain. She also described the cognitive and behavioral interventions that
could be used to reduce pain.
At 10:00 a.m. a presenter from Infection Control came to discuss drug resistant
organisms, which she referred to as MRSA, VRE, and Antimicrobials. She gave definitions of
these types of organisms, the cost associated with infection, and precautions that the hospital
takes to prevent resistance from occurring.
73
At 10:45 a.m. a presenter from the Training and Education Department began a session
on the computer system used at the hospital known as Micromedex. She began by demonstrating
how to search for different disease types. The computer system gave a variety of types of
information about each disease. After a brief demonstrations, the presenter moved to a new
topic. The new topic was not readily apparent from the information being presented, but I
figured out that the topic was age appropriate care after the presentation was underway. The
learners were divided into small groups. The presenter went from group to group asking true
false questions about the care of toddlers, adolescences, and those in early, middle, and late
adulthood. After the exercise, she read a case description and asked questions about the case.
The learners were somewhat non-responsive to the questions during both learning exercises. At
11:00 a.m. the session ended and the learners went to lunch.
For the afternoon session, there were two learning tracks, a CPR certification track and a
learning tack for those needing to be recertified in CPR. There were only two learners in the
certification track and that session was four hours long. I was only able to attend the first part of
the CPR certification track, because certification was designed for hospital employees only. I
did attend the first part of the session which consisted of a four-hour video on the CPR process. I
was told that a hands-on CPR training activity followed the video. I was unable to attend the
recertification track, but the session was two hours.
Day Four
Days four and five of orientation were targeted at new nurse employees. The nursing
orientation is only offered once a month, while the general and clinical orientation is offered
twice a month. Therefore, I observed the nursing orientation on June 5 and 6, two weeks after
74
observing general and clinical orientation. Note that the order and time of topics presented did
not match the agenda presented (see table 4.4).
Nursing orientation opened with an overview of the agenda, which was presented by,
Marie, the Clinical Staff Educator in the Training and Education Department and coordinator of
nursing orientation. There were ten participants in attendance. She then presented some basic
information which new nurses needed to know about passing a medication test. The first session
of the nursing orientation was only about 10 minutes in length. The session was not listed on the
schedule, but focused on teaching patients and families how to use medical supplies. A brief
review of how to order medical supplies was also given.
The next session began at 8:20 a.m. and was on the topic of hemoccult testing and wound
care. The first part of the session was a hands on activity in which the learners use the products
involved in hemoccult testing to test mock sample materials. After finishing the activity, the
learners had a chance to wash their hands. The wound care presentation began with a
demonstration of the different types of wound care products, including saline wash, lotions,
cleansers, and dressings. Specialty beds were also discussed and a demonstration was given of a
decision tree that is available on the intranet to help nurses make decisions about the type of
specialty bed needed. The presenter briefly reviewed the hospital policy concerning wound care,
showed the policy to the group, but did not disseminate a copy to the learners.
At 9:30 a.m. a new session began which focused on Diabetes and Diabetes education.
The presenter was a Diabetes educator at the hospital. She began the session by having each of
the learners introduce themselves. At the start of the session the learners were presented with
several handouts which were used to guide much of the content discussion. The content began
with a review of insulin medications, and continued with a review of the symptoms of low blood
75
Table 4.4. Nursing Orientation-Day Four Thursday June 5, 2003
8:00 – 8:15 Introduction/Greeting
8:15 – 9:15 Functional Health Patterns
9:15 – 9:30 Break
9:30 – 11:00 Documentation
11:00 – 12:00 Pharmacology
12:00 – 12:45 Lunch
12:45 – 1:15 Case Management- Discharge Planning
1:15 – 2:00 Advance Directives
2:00 – 2:15 Break
2:15 – 3:00 Pyxis Usage
sugar and high blood sugar. The review also included a discussion of hospital protocol for
treating hypoglacemia and hyperglacemia. After the handouts were reviewed, the presenter gave
learners a test and a check sheet which pertained to the second portion of the session. The
presenter indicated that learners would be tested on all information covered during the second
portion of the presentation on the accu-check meter; she also briefly talked about the annual
recertification that nurses must complete in this area. The session continued with a
demonstration of the accu-check meter, which the hospital uses for testing patients’ blood sugar
levels. The learners all gathered around a table to see the demonstration of how the accu-check
machine works. Each learner then had the opportunity to work the machine. After the
demonstration and practice session, the learners completed the test.
76
At 11:00 a.m. the next session began on the topic of advanced directives. This topic was
briefly covered during general orientation, but covered in more depth during this nursing
orientation. The presenter was the Director of Patients/Guest Relations and the same person who
presented the advance directives content during the general orientation. The session began with
an introduction to the advance directives booklet which the hospital provides to patients. The
presenter covered the legal requirements concerning giving information about advance directives
to patients. The presenter then went through the booklet section by section and covered topics
such as living wills, receiving pain medications, and organ donation. The presenter told a story
about an incident at the hospital and a learner also shared a relevant story. The session ended
with an opportunity for question and answer, and several questions were asked. The focus of
most of the questions was how to comply with the legal requirements. At the close of this
session the learners went to lunch.
The next session after lunch was on the topic of case management. The presenter, a case
manager, arrived slightly late and rushed into presenting content with little introduction. She
reviewed the number of case managers and social workers on each unit. She also talked about
the role of the case worker in the hospital, which included work with patients, insurance, home
care, abuse, and homelessness. The presenter asked for questions, and when there were none she
ended the session. The total session was only about fifteen minutes in length.
The next session of the day was on the topic of pharmacology and was presented by a
pharmacist. The presenter arrived and gave each learner a packet of papers and a calculator. He
began the session by reviewing his role at the hospital. The first content presented was a hospital
protocol for the dispensing of a commonly used drug. The presenter then directed the learners to
a case study which focused on medication calculations and gave them twenty-five minutes to
77
complete the problems. During the activity, the presenter encouraged the participants to talk
among themselves and provide help to each other as needed. At the end of twenty-five minutes,
the presenter reviewed the case and the math problems. A final lecture focused on the use of
micro-medics, how to properly abbreviate pharmacy terms, non-punitive reporting of errors, and
the importance of patient safety. The presenter ended the session by telling the learners the
number to call to report medication errors and adverse reactions, and the presenter collected the
math tests.
The final session of the day was a video on the use of pyxis, which is a
medical/pharmaceutical computer system, used by the hospital. The presenter indicated that if
learners were familiar with the computer system, it was not necessary for them to stay through
the video. Only three of the original ten learners remained to watch the video.
Day Five
Day five was the final day of nursing orientation, and was only a half a day session (see
table 4.5). Learners arrived at the hospital at 8:00 a.m. for an orientation to the laboratory, but
most of the learning activities were to be done one-on-one, so I did not attend this portion of the
orientation program. The learners reconvened at 9:00 a.m. at in the education room. There were
again ten participants attending the orientation program. The presenter was Marie, from the
Education and Training Department and the topic was documentation. She began the session by
talking about the history of the documentation system at the hospital and described how it came
to be in its current form. She then gave the learners a packet of the sixteen different forms used
by the hospital staff. She demonstrated how to get the forms off of the hospital’s Intranet. She
then reviewed each of the sixteen forms, including the purpose and the important features of each
form. Several of the forms were flow sheets and the basic design and purpose of the sheets were
78
Table 4.5. Nursing Orientation-Day Five Friday June 6, 2003
8:00 – 9:00 Laboratory
9:00 – 10:15 Skin/Wound Care Hemoccult Testing
10:15 – 10:30 Break
10:30 – 12:00 IV Therapy/Blood Transfusions
12:00 – 12:45 Lunch
12:45 – 2:15 Diabetes Education
Blood Glucose Monitoring
explained. At the end of the session she took several interested learners to her office to further
review programs available to nurses on the Intranet.
At 10:30 a.m. the final session of the nursing orientation began. The topic was
intravenous (IV) therapy and was presented by the IV therapy nurse. She began the presentation
with a review of the veins appropriate for IV therapy; most of the veins discussed were in the
arm. She then reviewed the duties of the nurse concerning IV’s, including site prep, and
checking for allergies. After this discussion, a brief video was show which introduced the type
of needle used by the hospital. The video demonstrated the use of the needle, and after it was
over the learners had a chance to practice using the needle on a practice pad. The practice pad
was a mock “skin and vein” with the consistency of skin and tubes or “veins” running through.
Those familiar with using the needle product helped other learners use the needle on the practice
pad. The presenter then demonstrated how to secure the IV to the patient’s arm. The learners
were then divided into pairs and each was given the opportunity to practice using the bandaging
materials to secure the IV to their partner’s arm. During this activity there were many
79
conversations between the learners while the presenter walked around offering assistance as
needed. This session concluded the five day orientation program at the hospital.
Summary
The orientation program was five days in length, with the first two days devoted to
general orientation, the third day devoted to clinical orientation, and days three and four devoted
to nursing orientation. The format of all five days was similar. Presenters, considered experts in
their topics, spoke from thirty minutes to several hours on their assigned topic. The presentation
styles ranged from lecture-based to hands-on, and a variety of supporting materials were used,
including videos, small group activities, tests, and handouts. While an agenda was given to
learners, it was only a loose guide to the topics and the time frames; and often the program
deviated from the agenda, especially during nursing orientation. My role in the program ranged
from observing activities to participating fully in the learning experience. Observing the
program provided me with an understanding of the types of information conveyed during
orientation and gave me a better understanding of the learners’ experiences with the program.
80
CHAPTER 5
DESCRIPTION OF STAKEHOLDERS AND THEIR PROGRAM THEORIES
This chapter presents an answer to this study’s first research question: “What are the
program theories of the stakeholders?” Specifically, this chapter provides a description of the
stakeholders and the program theories which they have for the new employee orientation
program which is the focus of this case study. Five categories of stakeholders were represented
in this study: program planners, hospital administration, teachers, learners, and the affected
public. It should be noted that while the five categories of stakeholders were used as a guide
when selecting stakeholders to be included in the study, most of the stakeholders served multiple
roles. The table below gives a brief overview of the stakeholders represented in the study (see
table 5.1).
Table 5.1. Overview of Stakeholders Name (Stakeholder Group)
Title/Department Role
Sally (Program Planner)
Resource Development Coordinator/Training and Education
Program Coordinator for the General New Employee Orientation Program and Presenter
Ann (Program Planner)
Director of Education and Training Department/Training and Education
Supervisor of New Employee Orientation Program and Presenter
Marie (Program Planner)
Clinical Staff Educator/Training and Education
Program Coordinator for the New Employee Nursing Orientation and Presenter
Donna (Hospital Administration)
Manager of Employment and Employee Relations/Human Resources
Coordinator for New Hire Scheduling into the New Employee Orientation Program
81
Table 5.1. Overview of Stakeholders Continued Name (Stakeholder Group)
Title/Department Role
Janice (Teacher)
Diabetic Educator Presenter in New Employee Nursing Orientation Program
Ellen (Learner)
Director of Women’s Services Recent Program Participant and Coordinator of Unit Specific Orientation
Betsy (Learner)
Unit Secretary/ Women’s Services Recent Program Participant
Linda (Affected Public)
Director of Patient and Guest Relations
Patient’s Rights Advocate and Presenter
This chapter first provides information about the role of the stakeholder in the hospital,
the role of the stakeholder in the new employee orientation program, and the program theory of
the stakeholder. The program theories are displayed in a model format, with a discussion
accompanying each model. The discussion first addresses the outcomes identified by the
stakeholder and then addresses the parts of the program which support each particular outcome.
Finally, the discussion turns to how each stakeholder’s view of the program is influenced by her
or his position within the organization.
Stakeholder One: Sally
Sally works in the Education and Training Department at the hospital and her title is
Resource Development Coordinator. Her role in the new employee orientation program is that
of program coordinator. In this role, Sally is responsible for overseeing the program agenda,
selecting presenters from the hospital staff, overseeing program functioning, and making
improvements to the program as needed. In her general oversight of program functioning, Sally
describes her role as, “ensuring that all the new employees are getting the information that they
need.” She also acts as a presenter in the program on the first day of general orientation.
82
Sally’s role also involves some interaction with Human Resources. She states, “I also
have to work with Human Resources because they send us the people and we orient them.” In
addition to her role in the Education and Training Department, Sally serves on the environmental
control committee within the Public Safety Division.
Sally’s view of the orientation program focuses around three major outcomes, employee
fit with the organization, employees having tools to perform their jobs, and hospital compliance
with regulatory bodies (see figure 5.1). The first outcome, employee fit with the organization, is
important given the hospital’s religious foundation. Sally states,
So what we hope to accomplish is one, that the new employee will be able to judge by the
end of that first day really, whether or not they fit with the organization. Because being a
Catholic health organization we have a set of ethics that we have to follow that may be.
different than other hospitals. And we are religious based, and some people may not be
comfortable with that. So they need to know that up front
In order to achieve this outcome, the orientation program introduces new employees to
the mission and values of the hospital, as well as the strategic plan or the vision. The mission
and values of the hospital are set by the hospital’s parent organization. The parent organization
provides an educational video which introduces the mission and values. Another piece of
orientation that pertains to employee fit with the organization is an introduction to the ethics of
the hospital, specifically the role of the ethics committee. Finally, in order to fit within the
organization, employees need to have an understanding of the health care system, or a broader
view of the hospital’s functioning. Sally sees the orientation program as crucial to the view
83
Figure 5.1. Sally
Employee Fit withthe Organization
Tools to Perform the Job
Compliance with Regulatory B di
Learning about Ethics
Learning Mission, Values, and Strategic Plan
Meeting Presenters Hospital Tour Overview of all
Desire for Employees to have Understanding of the total Health Care System
Parent Organization responsible for Mission and Values
Information is matched with actual work
Learning customer satisfaction, ethics, culture of safety, performance improvement, risk management, corporate compliance, and HIPPA
Measured by signing statements
Desire for employee competence in regulationsHIPPA
Health Information Management Safety Officer
In-house Corporate Compliance, Risk Management, and Infection Control
Environmental Control Committee of the Public Safety Division
Human Resources
OSHA
Joint Commission
84
which employees have of the system. She states, “we are also making a first impression as a
system.” Part of understanding the hospital system is meeting employees from a cross-section of
the hospital system. Sally states, “we get all orientees, no matter where they work, but we
understand that we are in the hospital setting.” Another part of the overview of the hospital
system is accomplished by introducing employees to long-term care, the nursing home facility,
home healthcare, the wellness center, and the women’s center. The orientation program ends
with a hospital tour which also facilitates an understanding of the physical layout of the hospital.
The second outcome identified by Sally is that new employees need tools to perform their
job. Sally sees orientation as a place to give new employees “their tools so that they can go
immediately on the job and feel like they fit in and they have an idea of what they need to do.”
The parts of the program that Sally views as giving employees tools that they need to do their job
include, customer satisfaction, the culture of safety, performance improvement, risk
management, corporate compliance, and HIPPA. There is an important link between the
orientation program and the actual use of the tools on the job, in that the orientation program
needs to be representative of what actually happens in the workplace. Sally states,
What they are doing in orientation and hopefully what they are seeing on their floors or
wherever they work is going to match. We don’t want them to go back to their unit and
say “but they said in orientation we are doing this.”
The third major outcome identified by Sally is compliance with regulatory bodies. Sally
describes the regulatory aspect of the hospital as originating from two sources, “the required
things they have to have by the regulated bodies and what in-house we think that we need.” The
regulations which come from within the organization or from in-house include corporate
compliance, risk management, and infection control. There were three outside regulating bodies
85
named. The first regulatory body influencing the orientation program content is HIPPA, which
is represented at the hospital by the Safety Officer in the Health Information Management
Division. The second regulatory body is OSHA which is represented at the hospital by the
Public Safety Division. Within the Public Safety Division is the environmental control
committee, which oversees much of the orientation material. Finally, the Joint Commission
requirements for the orientation program come through the Human Resources Department. In
order to prove compliance with the regulatory bodies, the orientees “have to sign several pieces
of paper.” Sally states,
I know this does not measure competency, but they are saying that yes, I understand
this…So it is more just an acknowledgement by them that they understand and you are
not really going to know until it gets put into action.
Sally’s program theory is limited to the outcomes associated with the general orientation.
She did not discuss the nursing orientation program or the ongoing orientation which employees
receive at the departmental level. As the Coordinator of the general new employee orientation
program, Sally is concerned with the importance of each of the components or topics covered in
the orientation program. She acknowledges that there are large amounts information that need to
presented in a short period of time and her program theory represents the intended outcome for
each of the topics presented. She also shows concern for compliance with the many regulatory
bodies represented in the orientation program.
Stakeholder Two: Ann
Ann’s role at the hospital is the Director of the Education and Training Department. As
director, she is responsible for both employee development and nursing education. She is a
nurse by training and began her career in education as a nurse educator. However, in her current
86
role she is in charge of training and education for staff of all levels from “administration and
management training to those that work in housekeeping and the kitchen.” Her role in the new
employee orientation program is to oversee program functioning and support Sally in her role as
coordinator of the orientation program. However, in the past she has acted as a presenter and
continues to act as a substitute presenter if needed.
Ann has identified five major outcomes for the new employee orientation program at the
hospital, including employees having a high level of comfort performing their job, decreased
hiring costs, employees being prepared to do their jobs, employees fitting with the organization,
and providing mandatory and essential education (see figure 5.2).
First, Ann hopes that after completing orientation, new employees will have a high level
of comfort performing their jobs. She acknowledges that “it is scary to start a new job” and that
often just having some basic information can relieve some of the anxiety associated with a new
workplace. She also acknowledges that not all the information presented during orientation will
be remembered by the learners, but that it is more important for them to have knowledge of
where to go to find information and who to contact for help. Ann states, “if you can at least give
them a phone number, link them to a face” they can use that information to answer questions that
arise on the job. Ann goes on to say that if learners know at the end of orientation, “that is the
person I need to go to for that, that is where I really felt we met our goal.” Ann also thinks that
the way in which the information is delivered during orientation impacts the learners’ knowledge
of reference points. She states, “The more information delivered in different ways that we can
give people, the better off we are.” Ann goes on to discuss the importance of providing an
orientation booklet and handouts for learners to refer to once on the job.
87
Figure 5.2. Ann
High Level of Comfort Performing the Job
Knowing reference points and contact people
Using a variety of delivery formats
Using handouts and other materials in classroom
Providing contact information
Decreased Hiring Costs
Employee RetentionEmployees feel welcome and valued
Upper management present at general orientation
Employees Prepared to Perform Job
Demonstration of Job Specific Skills and Knowledge on Checklist
Department Specific Orientation Overview of Job at
General Orientation Orientation Process Influenced by Joint Commission Standards
Employee Fit with Organization
Understanding the Mission and Values of the Hospital
Religious Nature of the Hospital
Providing Mandatory and Essential Education
Teaching Fire Safety, Infection Control
Compliance with Joint Commission, OSHA, and HIPPA
88
Another outcome Ann expects from the program is a decreased hiring cost, which she
views as a natural outcome of making employees feel welcome and valued. When asked about
her expectations, Ann states “my goal over and above everything else is that they feel welcome,
and that they feel like we care about them as an institution.” Ann does admit that there is an
“ulterior motive” to employees feeling welcome and valued, which is that employees will want
to remain a part of the organization. She states, “We hope then that we would get some retention
from them because it is so expensive to hire new people.” So feeling valued by the organization
leads to employee retention, which ultimately leads to reduced hiring costs. Much of this desire
for employee retention is driven by the deficit in nurses in the job market. Ann’s frustration with
this situation is captured in her statement, “there aren’t any nurses out there any more.” As part
of making employees feel welcome, all the directors from the various departments are invited to
the first fifteen minutes of orientation. She acknowledges that growing time constraints have
caused there to be a decrease in the attendance of managers at orientation. Ann states, “I think
that was really a good idea, because that had lots of welcoming to it, especially when someone at
that high level would come in.”
A third outcome which Ann views as stemming from the new employee orientation
program is that employees are prepared to perform their jobs. Because of the general nature of
the information presented at the orientation program, Ann does not expect that orientation alone
will fully prepare people to perform their jobs. When speaking about orientation program she
states, “There is some preparedness, but it is pretty general.” Instead, Ann thinks that employees
will only become fully prepared to perform their jobs after completing the department specific
orientation and completion of the skills and knowledge checklist which is in place for every job
at the hospital. For example, Ann states,
89
A lot of what we do in general orientation, I don’t know that that really get them
necessarily technically prepared for their job. Because that is why we leave that up to the
departments, to really do that fine tuning.
Once new employees have participated in a department specific orientation, they are required to
complete a skills and knowledge checklist. The checklist represents competencies required for
their specific job function. The competencies “have to be written in behavioral terms so that we
can actually see something happen…the actual demonstration, interaction with the patient, or the
interaction with the piece of equipment.” The structure of the orientation process, general
orientation followed by departmental specific orientation, is mandated by the Joint Commission.
Ann explains, “They expect kind of this general overview, but then they demand this department
specific training.”
A fourth outcome Ann expects from the orientation program is that employees will know
if they fit with the organization. The hospital has a religious focus, that is, it is sponsored by a
religious organization and uses religious doctrine to guide the mission and values. Ann states,
“they do need to know about our mission and values…if your own values are not in line with
what our values are, it maybe a hard fit.” It is important for new employees to understand the
religious nature of the hospital early on so that they can decide if they fit within this culture.
Ann explains:
This is the time, during these first couple of weeks for you to really examine that and
make sure that this a good fit for you, and so it is important that we have put those
mission and values right up at the front, so people can know if it is the right place for
them to work.
90
Finally, Ann expects that the orientation program will fulfill mandatory education
requirements imposed by outside regulatory bodies and provide employees with education which
is essential to hospital functioning. Ann explains, “There are some things that we have to go
over.” As mentioned earlier, Joint Commission mandates that some general orientation be given
to new hospital employees. But in addition to the Joint Commission requirements, the hospital
must also comply with OSHA and HIPPA and provide basic education in these areas. Internally,
the hospital has a need to convey the fire safety plan and the infection control procedures. Ann
admits that this information maybe redundant to new employees that have experience in the
healthcare field. She states, “if they have been in healthcare before, they are going to know,” but
stresses that it is important to cover all of this material.
As the director of the Training and Education Department, Ann is concerned with a wide
variety of outcomes for the orientation program. While she is concerned with outcomes
associated with the individual learners, such as job preparation and employee fit with the
organization, she is also concerned with broader outcomes such as decreasing hiring costs and
providing mandatory education.
Stakeholder Three: Marie
Marie’s role at the hospital is in the Education and Training Department as a clinical staff
educator. In this role she coordinates educational programs for the clinical staff, or the
employees directly involved in patient care, with nurses being the largest group. As the clinical
staff educator, she is responsible for the clinical orientation and the nursing orientation. She
notes that “the orientation program has pretty much been in effect for a good number of years,
with a few minor changes depending on new things.” Therefore, she sees her role as more of a
coordinator rather then a planner. As she states, “I don’t provide all of that education, but I do
91
coordinate or make sure the right people are in the right plaice at the right time.” There are
certain topics in both the clinical and the nursing orientation programs on which she is a
presenter, and these include CPR and documentation education.
Marie’s central goal for the new employee orientation program is that nurses feel
comfortable performing their jobs (see figure 5.3). She thinks this is important because if “you
don’t feel that you have been given enough preparation, it is very uncomfortable, it’s a patient
safety issue, professional license issue.” Marie explains that in order for nurses to feel
comfortable, their education starts at nursing orientation which is “a foundation or an
introduction.” She has three expectations for nurses completing the nursing orientation program:
that nurses understand their job functions, that nurses know how to obtain further education, and
that nurse feel that education and support are available. First, she expects “that you would have
an understanding of your job and an understanding would lead to being confident that you have
the ability to do your job.” Second, she expects that a nurse would feel:
comfortable in that she knows what she may need to address further and who she would
need to see to get that, so that she feels when she is finished with orientation that she can
go to work and be comfortable in the setting.
Finally, upon the completion of nursing orientation she expects nurses will feel that education
and support are available to them. She states, “you have got to have the expectation that we are
willing to work with you, provide you with what you need and the people who would have the
ability to help you as well.”
Marie views nursing orientation as a process, with the orientation program providing the
“basic preparation,” but she notes, “It does not end with our two day program.” She explains,
“We provide the basic or foundation at the orientation, but then we also have the ongoing
92
Figure 5.3. Marie
Patient Safety
Maintaining Professional License
Nurses Feel Comfort-able Performing Job
Nurses complete competency checklist
Nurses work with preceptor to learn unit specific information
Understanding Job Functions
Knowledge of how to obtain further education
Feeling that education and support are available
Introduction to Job at Basic Nursing Orientation
Ongoing Education
93
orientation and part of that is an initial competency checklist.” So the orientation process
involves the orientation program, and the ongoing orientation, including the competency
checklist. Each nurse has competency checklist which is based on their job description and it
“covers all of the skills that they would be expected to perform.” As nurses demonstrate
competency in the skills on the checklist they are checked off. Because they are not expected to
perform the skills until they have been checked off, the expectation is “within six months, the
majority of those skills would be checked off, and they would feel comfortable once that is done
in performing those skills.”
A second part of the ongoing education in which nurses participate is work with a
preceptor. A preceptor is a trained nurse who can “take a new nurse on and spend up to six to
eight weeks with her, orienting her to her unit.” There are trained preceptors in all areas of the
hospital, prepared to spend up to eight weeks with a new nurse. The length of time a new nurse
spends with a preceptor depends on their level of ability. The preceptor also has a role in the
competency checklist described above. Marie explains that nurses “have an opportunity to make
sure that they understand the procedure, the process, the skill before they sign themselves off as
being checked off and then the preceptor does as well.”
So in summary, Marie’s view is that
the basic orientation then leads to the more specific unit orientation, in which their skills,
their expectation, their job performance is evaluated and checked off before they actually
perform that…So then they should feel comfortable and confident in their ability.
Marie’s program theory focuses exclusively on nurses. Her program theory is very broad
in scope, ranging from nursing orientation, to ongoing orientation, and patient safety and
maintaining licensure. In her work with the nursing orientation Marie acknowledges that the
94
nursing orientation program is only once piece, and that further orientation and training is
necessary. As the Clinical Staff Educator, it is understandable that Marie would focus on
continuing education and development.
Stakeholder Four: Donna
Donna’s title at the hospital is the Manager of Employment and Employee Relations in
the Human Resources Department. Her role is “overseeing, coordinating the whole process of
how people get employed here, also coordinating the whole pre-employment process and
orientation.” Her role in the Human Resources Department is to make sure the new employee
satisfies all of the hiring requirements, including lab work, drug screens, and participating in new
employee orientation.
Donna’s role in the orientation is program is “to make sure that people get scheduled into
orientation.” Her department produces the orientation list, or the list of all new hires that need to
attend orientation. Then they “send that out to the appropriate people, so that everybody knows
who is supposed to come to orientation and when they are supposed to be there.” She also
follows up with any employee not attending orientation on the scheduled date. Donna also notes
that she is no longer a speaker at orientation, but has been in the recent past.
Donna names three outcomes she expects from the orientation program: employee
retention, compliance with regulatory bodies, and providing new employees with information
(see figure 5.4). First, Donna thinks the orientation program can have an effect on employee
retention. She states, “If you don’t have a good orientation program and a good overview of [the
hospital] healthcare system, I think it directly impacts retention; it impacts the perception the
new employee has of the hospital.” Donna thinks that the orientation program is “vital to giving
that applicant or that employee a really good overview, a good first impression of the hospital.”
95
Figure 5.4. Donna
Employee Retention
Employee Perception of the hospital
Overview of Hospital System
Meeting Hospital Management and Other Employees Employee
Participation in Orientation Program
Smooth Hiring Process
Compliance with Regulatory Bodies
HR Keeping Personnel Files
General Orientation
Department Specific Orientation
Requirements from Joint Commission and other State Agencies
Providing New Employees with Information
Polices and Public Safety
Culture, Mission, and Values
Benefits Information
96
But the orientation program is also important in that new employees have the chance to
meet hospital management. For example, new employees “meet lots of people they may not see
otherwise, people responsible for the policies that they are going to follow.” In addition to
meeting hospital management, new employees also begin to develop a social network during
orientation. Donna states, “You have got that connectedness and interaction in orientation, and
you start to form friendships.” Donna goes on to tell a story about a friend she met in orientation
ten years ago, whom she still talks with occasionally. Overall, Donna sees participation in the
orientation program as one step in a smooth hiring process: “The process needs to run very
smoothly from the time the applicant walks in the door until they get to the department.” And
ideally, the new employee will “attend orientation before they even set foot in the department.”
The second outcome Donna expects from the orientation program is hospital compliance
with regulatory bodies, mainly Joint Commission. She states, “Joint Commission or different
state agencies… are going to make sure that we properly oriented an employee to [the hospital]
and of course real importantly to their job once they get to the department.” Human Resources
plays a role in the Joint Commission review process by keeping evidence of employee
participation in orientation procedures. Donna states that, “our role in HR especially when it
comes to Joint Commission … we keep the personnel files here in this office, the main personnel
file. So in that file, there has to evidence that they have been oriented.” Donna acknowledges
that, “we have got to do orientation, but I think we do more than is absolutely necessary, we try
to go above and beyond the bare requirements.”
The third outcome Donna expects from the orientation program is providing employees
with basic information. She states, “The main reason you have orientation is to inform.” Donna
discusses the information presented during orientation as falling within three categories. First
97
the employees need to know about the hospital culture. Orientation should “set the stage for just
the culture, what to expect here.” And this comes through an overview of the culture, mission,
and values. Second, employees need to know the “nuts and bolts” of the hospital, which are
things like pubic safety, back safety, and other hospital policies. Donna states, “Giving them
information, that is the big piece of it. They need to know what the policies are, and we do cover
a lot of that.” The third piece of information presented at orientation is knowledge about the
benefits options at the hospital:
We cover benefits of course in orientation, so for benefits eligible folks this is the time
they answer questions and fill out their enrollment forms, so that is just a big piece,
giving them the knowledge they need, and a place to go as a resource if they have
questions later on.
Donna’s program theory focuses primarily on the role of the orientation program in terms
of the hiring process. In her role as Manager of Employment and Employee Relations, her
concern is insuring that the hiring process is smooth and that that employees have a good first
impression of the hospital. She hopes that this process will reduce new employee attrition.
Donna also acknowledges that the orientation program serves the purpose of providing new
employees with information and complying with regulatory bodies.
Stakeholder Five: Brad
Brad’s role at the hospital involves four main components performance improvement,
risk management, corporate compliance, and Joint Commission compliance. The performance
improvement aspect of the job involves measurement of organizational activities with the intent
of improving the measured activity. Risk management is similar to the performance
improvement process but concerns the activities that are involved with legal regulations. In his
98
role as a risk manager, Brad also purchases and administers insurance for the hospital. The
corporate compliance role involves insuring that the hospital is in compliance with federal, state,
and local laws. Corporate compliance also involves insuring that the hospital is billing
appropriately for the services provided. Finally, his role involves insuring compliance with Joint
Commission standards. Compliance with Joint Commission is necessary for the hospital to
receive funding from Medicare and Medicaid. In both his role as Corporate Compliance Officer
and Joint Commission Officer, Brad must make sure that any new regulations are implemented
at the hospital.
Brad’s role in the orientation program is as a presenter. He presents during general
orientation on the topics of performance improvement, risk management, and corporate
compliance. His presentation is a general overview of each of these functions within the hospital
and how it will affect the employees.
Brad’s view of the orientation program involves two major outcomes, employee
involvement in the performance improvement, risk management, and corporate compliance
processes at the hospital, and continued success of the hospital’s business (see figure 5.5).
Starting with the performance improvement process, Brad states “my goal is to give the new
orientees just a basic understanding of how we develop our process, how it relates to the vision
of the organization and then the statistical tools that are utilized.” When discussing his
expectations for the risk management portion of his presentation, Brad wants “them to
understand their involvement in that, in identifying things that go wrong and reporting them on
an occurrence report form.” When discussing corporate compliance Brad states:
I want to leave them with the understanding that if any of those things that we cover
under corporate compliance happen or they think, even think it is happening that they
99
Figure 5.5. Brad
Involvement in Process and Use of Tools
Understanding of Issues Specific to Department
Understanding of Performance Improvement Process
Understanding of Employee Role in Risk Management Process
Understanding of Corporate Compliance Issues
Relationship between Performance Improvement and Vision
Identifying Problematic Occurrences
Knowledge of how to complete an occurrence report
Joint Commission Compliance
Understanding reporting of Problems
Continued Success of Hospital Business
Return Customers
Good Reputation in the Community
Understanding the importance of good customer relations
All Knowledge Reinforced through Clinical Orientation
100
need to notify me so that I can get involved in and investigate and find out what is really
going on.
Brad notes that “a lot of the stuff that I cover is really reinforced through the clinical side.” For
example,
They will be really demonstrating, here is an occurrence report, if you give a wrong
medication how you fill out the occurrence report. So they take it another step further in
the clinical aspect, and they really have more hands on demonstration.
Finally, Brad summarizes his expectations of the program by stating that employees
should learn how their specific departments handle each of these topics, and become involved in
the process:
I expect them to be able to go back to the departments and find out specifically what the
issues are in their department… I give them the understanding that we do it, and this is
how we derived it, and this is how we look at it and some of the tools that we use. And
then they go back to the department and they see it happening and get involved in it and
utilize the tools.
Brad also stresses the importance of the guest relations session which is presented during
the orientation program. He says that employees
need to understand that, you know, they are our customers and you got to treat them right
or they are not going to come back… and also if we don’t treat them right then our
reputation goes down the drain in the community.
Brad goes on to say that the satisfaction of the customer and the reputation of the hospital within
the community are what keep it a viable business.
101
Brad’s program theory is directly related to his job function at the hospital and the subject
matter about which he presents. He discussed the program in terms of his three job functions,
performance improvement, risk management, and corporate compliance. He was also concerned
for the continued success of the business end of the hospital.
Stakeholder Six: Janice
Janice’s role at the hospital is that of a diabetic educator. While she is a nurse, she works
exclusively as an educator at the hospital. She teaches both inpatients and outpatients. The
outpatient group is reached through a class, which is offered several times each month. The
inpatient group receives educational support when they enter the hospital for diabetes related
illnesses. She also teaches several community programs in locations such as assisted living
homes. Her role in the orientation program is that of presenter. She presents a session during the
nursing orientation on diabetes.
Janice has three expectations of the new employee orientation program, compliance with
the hospital’s requirement that all nurses be certified to use the accu-check meter, nurses giving
good patient care, and nurses being comfortable on the job (see figure 5.6).
Janice states, “My goal would be for them to be able to use the accu-check meter.” The
accu-check meter is a blood glucose monitor which is used by the hospital, and all nurses must
be certified to use the meter. Janice states, “It is mandatory, they have to be recertified every
year.” The nurses learn to use the meter during orientation through a process of watching a
demonstration and then demonstrating it themselves. Janice feels that the learning technique she
uses is very important, because when she sees “a return demonstration, I feel like they have
learned, because then they have learned by listening to me, and then they have learned by doing
102
Figure 5.6. Janice
Compliance with Hospital Requirement for Certification using Accu-check meter
Test on using Accu-check Meter
Learners Demonstrate use of Accu-check meter
Presenter Demonstration of Accu-check meter
Nurses Giving Good Patient Care
Understanding of High and Low Blood Sugar
Comfort Level on the Job
Knowledge of Hospital System
Knowledge of Hospital Protocols
Knowledge of Reference Points
Interaction with Learners during Orientation Presentation
103
it.” In addition to the demonstrations, the learners are also required to complete a test concerning
the use of the accu-check meter, which becomes part of their employee file.
Janice also names good patient care as an outcome she expects from the orientation
program. She talks about this goal as it relates to nurses understanding of diabetes:
My goal for a good outcome is to have good patient care, because I know that if they
have an understanding of high blood sugars and low blood sugars and if they have an
understanding of what causes it, how it can be prevented, and then how to test the blood
sugar, I know that they are giving better patient care.
Finally, she hopes to make nurses “feel more comfortable” on the job by providing them
with key pieces of knowledge. The first piece of knowledge which she discusses is hospital
protocol, or “knowing what the hospital expects out of them.” Second, nurses need to know
reference points or they need to be “acquainted with where to find things.” And finally, they
need to “have some idea of what the system is about before they go onto the floor.” In summary
Janice states, “the more you can learn before you go up and actually have patient interaction the
better you are.”
Each of the outcomes named above is contingent on the learning environment. More
specifically, Janice wants interaction from the learners when presenting this material. She states,
“I want the nurses to ask questions, I want interaction from them. I want them to recall maybe
an episode of when they have had a patient that had low blood sugar or high blood sugar.” She
thinks that interaction between the learners and the presenter is an indication of good teaching
and learning. She states, “I think that if you can teach and are doing a good job, you are going to
get questions and you are going to get interaction and you are going to get a good
demonstration.”
104
Janice’s role of educator is reflected in her program theory. Interaction with learners
during the presentation is a key piece of her program theory, and is a necessary to produce all the
outcomes which she names. She also specifies the channels through which she expects learners
to acquire knowledge about the topic. Like Marie, Janice’s program theory is limited to the
nursing orientation portion of the process. She has no involvement with general orientation.
Stakeholder Seven: Ellen
Ellen is the Director of Women’s Services at the hospital. This department includes the
labor and delivery unit, the post-partum unit, and the well baby nursery. As director of this
department, Ellen oversees about sixty-five employees. She has only been on the job three
months. Her role in the orientation program is as a learner only. However, she is involved in the
ongoing orientation that takes place at the departmental level. At the department level, Ellen
develops the orientation checklist used to guide the orientation process. The checklists are
created for all the different job classifications within the department. She is also involved in
linking new nurses in her department with a trained preceptor to further the orientation process.
Ellen sees the orientation program as leading to two outcomes, preparing new employees
to participate in the ongoing orientation at the departmental level and preparing the new nurses to
work with a trained preceptor (see figure 5.7). As a new employee entering a leadership position
she also thinks orientation gave her “a focus.” She states the orientation program helped her
understand,
the hospital’s focus on where they were planning to go. So, I think that was very helpful
to me as far as having to follow through with that plan. So I would expect that it would
be the same way for all new employees, but I had to take it even more to heart because I
have to preach it to people that work for me.
105
Figure 5.7. Ellen
Ongoing Orientation through checklist
Employee Work with a Trained Preceptor
Ability to lead employees and follow plan
Overview of Hospital System
Knowledge of Policies
Knowledge of Mission, Values, Vision, and Strategic Plan
Participation in General Orientation
106
In order to prepare the new employees for orientation at the departmental level, Ellen has
three expectations for the general orientation program. First, she expects it to provide an
overview of the hospital system. She states, “I basically expected it to be a general overall view
of the hospital and how the hospital works.” And she further states that as a learner she thought
that the program “was very successful in providing an overview of the hospital.”
Second, Ellen expects that the general orientation program will provide learners with knowledge
of hospital wide policies, such as “employment policies and scheduling policies.” When
discussing hospital wide policies Ellen states, “If they know that coming up here, before they
ever got up here, it would be much easier on the directors.”
Finally, Ellen expects the general orientation program to provide new employees with
knowledge of the mission, values, vision and strategic plan of the hospital. Again, Ellen notes
that because these issues are hospital wide, it is important for them to be covered before the
employee comes to the ongoing, unit specific orientation which is more job specific.
Ellen’s program theory reflects her role as a learner and as a new unit director. She
expected the orientation program to give her the knowledge that she needed to lead employees
using the hospitals mission and values. In her role as unit director, she also acknowledged that
the ongoing unit specific orientation was a necessary part of new employees understanding the
job.
Stakeholder Eight: Betsy
Betsy’s role at the hospital is that of unit secretary for Women’s Services at the hospital.
In addition to serving as unit secretary, she will also be serving as a nurse’s aid when needed.
She has only been on the job for two weeks, but will begin working the night shift as soon as her
orientation period is over. Presently, she is working with the unit secretary on the day shift to
107
learn “this is what we do, the papers that go in the charts, that kind of stuff.” Her only role in
orientation is that of learner.
Betsy sees the orientation program as producing one main outcome, good patient care
(see figure 5.8). She states, “Here it’s all about patient care.” She thinks that this is an important
part of the hospital’s philosophy and is evident not just in the orientation program, but in her unit
as well. Betsy states that during her unit specific orientation she felt supported by other
employees. She feels that people are willing to spend a lot more time with you, getting you used
to how it works here and what people expect out of you and what you expect out of them, and
you have just a lot more time, instead of them just throwing you, and saying alright do your job.
However, Betsy also thinks there are several outcomes that are specific to the general
orientation program. First, she expected to “learn more about the hospital.” She states, “the
mission and values, you know the ‘this is [the hospital]’ video, all told about the hospital.”
She also expected “to learn more about my job… basically just get an overview of what I
would be doing, and what was expected.” She states that several topics helped her to understand
her job expectations. For example, “abuse and neglect, you know the benefits of course, safety,
the ethics course, all that you know pertains to my job, so it all helped.”
Finally, Betsy expected to learn who “could help me in case I had questions” or general
reference points in the hospital. And she stated that she felt that orientation “helped to like, if I
had an abuse or neglect situation, who to call.” Along these lines, part of the orientation that
Betsy found exceptionally helpful was the use of expert presenters from various parts of the
hospital. She stated that the presenters were from upper management, or “if it was not the
directors of you know like electrical safety or infection control, it was somebody, you know like
the step down.
108
Figure 5.8. Betsy
Good Patient Care
Help from other Employees
Unit Specific Orientation
Knowledge of Job
Knowledge of Reference Points and Contact People
Knowledge of the Hospital Learning
Mission and Values
Learning about Abuse and Neglect, Benefits, Safety, and Ethics
Meeting Presenters in Various areas of Hospital
Presenters are upper management
109
As a new employee, Betsy had a very broad view of the orientation program. She
thought of each of the topics presented as helping her to know more about the hospital and her
job. She also articulated that the orientation program was only the first step in the learning
process. Because she is still in the job specific portion of her orientation phase, she emphasized
the importance of help from other employees in the learning process. Finally, she viewed the
purpose of the whole orientation process as improving her ability to give the best patient care
possible.
Stakeholder Nine: Linda
Linda is the Director of Patient/Guest Relations at the hospital. When describing her role
she states,
I represent or advocate for all the patients that come into [the hospital]. I also listen to
and resolve any grievances that any of our customers voice concerning their stay or
expected services that they felt that they did not receive or etcetera. I a.m. involved in the
advance directive process, end of life process; I am also involved in the ethics committee.
A major part of her job is to facilitate the complaint management process. In addition, she
teaches a seven week course called “The Customer,” which addresses personal relations and
customer satisfaction.
Her role in the orientation program is “to present to new orientees our philosophy as it
relates to patient rights.” She does this through
teaching employees the expected behavioral standards of [the hospital]. Teaching them
good customer relations, teaching them how to handle patient complaints, the importance
of good customer satisfaction, and then teaching them somewhat, some of the major
110
things that [the hospital] requires of every employee in order to assist them through their
performance evaluation process and that is in relationship to the customer.
Linda identifies four major outcomes of the orientation program (see figure 5.9). The
first outcome is excellence in patient satisfaction. For this outcome, general orientation is only
the first step in a series that is designed to create this outcome. Linda states, “I think the main
importance that it would be for me, based upon what my responsibility is at [the hospital], that
we would be able to continue to achieve excellence in patient satisfaction.” Linda goes on to say
that is achieved through having employees that “understand basically how to handle problems
and be able to handle problems at the grass root rather than having to go through a long process.”
In order to understand how to handle problems, the employee learns
basic behavioral expectations at [the hospital]…we talk about psychological reciprocity,
doing unto other as you would have them do unto you, we talk about problem solving, we
talk about the six step patient satisfaction program, we talk a lot about attitudes”.
This material is introduced during the general orientation session, and then is presented in full
during a “seven-week training period later that is certainly going to support what I have already
told them in orientation.” The seven-week course is designed so that employees attend a four-
hour course on week one, and then attend six weeks of one hour per week follow-up sessions.
Linda views the follow-up sessions as a key to the success of the program:
Ninety five percent will eventually be lost unless you have set up some type of a follow
up, in which you can continue to go over and over and over and cause them to repeat
these behaviors over and over, until it becomes a part of them.
111
Figure 5.9. Linda
Excellence in Patient Satisfaction
Reduced Risk Management Problems
Good Patient Care
Fulfillment of Mandatory Requirements
Employees Prepared to handle problems at the "Grassroot"
Follow up Sessio
Understanding of Behavioral Expectations, Psychological Reciprocity, Problem Solving, Six-Step Patient Satisfaction, and Attitudes
Attendance at 7-week "The Customer" Program
Introduction to "The Customer" at the Orientation Program
Knowledge of Department Specific Policies and Procedures
Ongoing Orientation with Manager
Teaching Reference Points for Policies and Procedures at General Orientation
Teaching General Information on Performance Improvement, Safety, Quality Improvement and Corporate Compliance
HIPPA, Medicare/Medicaid, and Joint Commission
Learning Specific Information on the Job
112
The second and third outcomes identified by Linda, good patient care and reduced risk
management problems, are produced through the same channels. Linda states that knowledge of
general policies and procedures would be presented at general orientation, but then her
expectation is that that manager then would take over, and would help the employee
through the process of understanding policies and procedures, not necessarily knowing all
policies and procedures, but know where they can go to find out those policies and
procedures.
According to Linda, this understanding of policies and procedures is linked with giving
good patient care and reduced risk management problems. Linda states, “the more knowledge
that our employees have, that this certainly gives us a better opportunity to prepare to take care
of our patients” and goes on to say, “if you do not have a good orientation program and give this
to your employees there is a big risk management situation that could evolve from this.”
Finally, Linda views fulfilling mandatory requirements an outcome of the orientation
program. She states, “I think that a lot of our orientation program is set up around mandates.”
She describes the mandates as stemming from a variety of sources, including HIPPA,
Medicare/Medicaid, and Joint Commission. Some of the orientation components which relate
directly to these mandates include, performance improvement, safety, quality improvement, and
corporate compliance. Linda’s expectation is that orientation is designed “to give everybody a
general overview, and then the expectation is that these people go back then to their various
departments, to get the specifics that is related to their department.”
Linda’s program theory is multi-faceted. Her role as the Director of Patient/Guest
Relations is evident in her desire for excellence in patient satisfaction. She clearly articulates the
channels through which she hopes to obtain this outcome, which includes educational
113
interventions beyond the orientation program. She also acknowledges that the orientation
process is ongoing and that knowledge of the job at the department level is key to fewer risk
management issues and good patient care.
Summary
This chapter has presented an overview of the nine stakeholders of the orientation
program interviewed as part of the case study. The overview included a description of each of
the stakeholders’ role at the hospital, as well as, their roles in the orientation program. Based on
interview data, a program theory was created for each stakeholder. These program theories were
presented and discussed. A variety of program theories were expressed. Some were narrow in
scope, focusing only on general orientation, while others were broad and encompassed ongoing
orientation and education. The program theories expressed by stakeholders were reflective of
their roles at the hospital and in the orientation program.
114
CHAPTER 6
DEVELOPING A PROGRAM THEORY TO GUIDE AN EVALUATION
In the previous chapters a description of the case or educational program has been
established, and nine program stakeholders have been introduced along with their program
theories. This chapter addresses research questions two and three:
2. What similarities exist among the stakeholders’ program theories?
3. What aspects of the program theories are unique to each stakeholder?
In order to address research question two, a program theory was created which represents
the similarities that exist among stakeholders’ program theories. That is, a review was done of
the nine program theories articulated by the program stakeholders and similar components were
identified. The components were then compiled into one program theory, which is
representative of these like ideas. This program theory will be presented and a discussion will
follow. Secondly, a review will be done of the components which are not represented in the
combined program theory. That is, the unique elements of each program theory will be
presented.
Similarities
A program theory was created to represent the similarities among stakeholders (see figure
6.1). The discussion of this program theory will follow the format of the individual program
theories presented in chapter five, beginning with the outcomes and moving to a discussion of
the parts of the program which support the outcomes.
115
Figure 6.1. Similarities among Stakeholders
Retention
Good Patient Care • Patient Safety• Patient
Satisfaction
Employee Fit with Organizational Philosophy
Understanding Religious Nature of Hospital
Employees Able to Perform Job Well
Employees Feel Comfortable Performing
Participation Job Specific Orientation
Knowledge of Ethics, Mission, Values, Strategic Plan, Overall Healthcare System, Safety, and Benefits
Participation in Orientation Program
Knowledge of Reference Points for More Information on all Topics Presented
Compliance with Regulatory Bodies
116
The program theory has three major outcomes, employee retention, good patient care,
and compliance. Retention was named as an outcome of the orientation program by Donna, the
Manager of Employment and Employee Relations, and Ann, the Director of the Education and
Training Department. Donna states, “if you don’t have a good orientation program and a good
overview of the [the hospital] healthcare system, I think it directly impacts retention.” Likewise,
Ann states, “we hope then that we would get some retention from them because it is so expensive
to hire new people, that whole process and then to train.” Ann goes on to say that retention
among nurses is especially important because of the shortage of nurses in the workforce.
One way in which the orientation program affects employee retention is through helping
employees know if they fit with the philosophy of the hospital. Because the hospital is religious
in nature, it is important that employees understand this early in the employment process. Sally
states that the orientation program should give new employees “the tools required to be a team
member, to fit into the organizational goals, mission, vision, and values.” Sally goes on to say,
Being a Catholic health organization we have a set of ethics that we have to follow that
may be different than other hospitals. And we are religious based, and some people may
not be comfortable with that. So they need to know that up front.
Ann also discusses the importance of new employees understanding the religious orientation of
the hospital, so that they can determine if it is a good fit. When discussing the organizational
mission and values she states, “This is the time; during these first couple of weeks for you to
really examine that and make sure that this is a good fit for you.” She goes on to say, “Since we
are Catholic too, you know that has a barrier on it, because there are not many places that pray.”
Good patient care is the second outcome in the program theory. Three stakeholders
specifically named this outcome as important, Linda, the Director of Patient Guest Relations,
117
Betsy, the new Unit Secretary, and Janice, the Diabetic Educator. After going through the
orientation program and beginning her unit-specific orientation Betsy states, “Here it is all about
patient care… you can tell”. And Janice states, “My goal for a good outcome is to have good
patient care.”
This outcome is achieved through patient satisfaction and through patient safety. As the
Director of Patient/Guest Relations, Linda is was the main advocate for patient satisfaction
stating, “I think the main importance that it would be for me, based upon what my responsibility
is at [the hospital], that we would be able to continue to achieve excellence in patient
satisfaction.” However, Brad, the Officer of Performance Improvement, Risk Management, and
Corporate Compliance, also had concerns about patient satisfaction. He stresses that good
customer relations are key to the continued success of the hospital. He states that employees
“need to understand that, you know, they are our customers and you got to treat them right or
they are not going to come back.”
The related factor affecting patient care is patient safety. Marie, the Clinical Staff
Educator, thinks that orientation is important, because if “you don’t feel that you have been
given enough preparation, it is very uncomfortable, it’s a patient safety issue.” Sally also thinks
that the knowledge gained during orientation will affect patient safety. She says that the
information learned “protects them and the safety of their patients, plus their own family that
they go home to. You know, so those are just the important things that keep everybody safe.”
In order to assure patient safety and satisfaction, it is expected that the orientation
program will prepare new employees to perform their jobs. Sally states that one of the main
outcomes of orientation is “to give them their tools so that they can go immediately on the job.”
Similarly, Brad states when talking about the information which he presents during the
118
orientation program, that he wants employees to “get involved in it and utilize the tools.” Ann
acknowledges that the orientation program cannot fully prepare employees to perform their jobs.
However, she does expect the program to prepare new employees for some aspects of the job
stating, “I hope that we would prepare people, given that little short time that we have with them,
with some aspects of their job.”
In addition to being simply able to perform their jobs, employees should also feel
comfortable and confident performing their jobs after orientation. When speaking about new
nurses Marie states, “we want her to feel comfortable when she leaves orientation and goes to the
unit.” She goes on to say that at the end of the orientation process nurses “should feel
comfortable and confident in their ability.” Similarly, Ann states, “we just try to give them some
kind of comfort level.” Janice expresses the same idea when talking about teaching new nurses
the protocol for using the accu-check meter. She states that nurses “feel more comfortable if
they know the protocol.”
It has been established that participation in the orientation program prepares employees to
perform their jobs and feel comfortable and confident in their abilities. This in turn should lead
to patient safety, satisfaction, and overall good patient care. However, as Ann expresses,
a lot of what we do in general orientation, I don’t know that that really gets them
necessarily technically prepared for their job, because that is why we leave that up to the
departments, to really do that fine tuning orientation, in terms of their department specific
orientation.
Of the nine stakeholders interviewed, six named ongoing or departmental specific
orientation as a key piece of their program theory. Betsy, the new unit secretary was still in the
process of job specific orientation at the time of interview, and she stated that the department
119
specific orientation helped her know “what people expect out of you and what you expect of
them and you have just a lot more time, instead of them just throwing you, and saying alright do
your job.” Donna, from HR, states that departmental specific orientation is important because
regulatory agencies “are going to make sure that we properly oriented an employee to [the
hospital] and of course real importantly to their job once they get to the department.” Ann, the
Director of the Training and Education Department also discussed the role of the regulatory
agencies in the structure of the orientation program. That is, she states it is required that the
hospital provide both a general hospital orientation and a job specific orientation.
Other stakeholders discussed ongoing orientation or departmental orientation in terms of
a job specific checklist. Ann describes the checklists in this way:
We have some orientation checklists, that would be really specific, for instance to an RN
or LPN, and then another one that would be specific to a clinical tech and then another
one for the secretaries. And that would be driven by their job description and their
performance evaluation.
As with many other aspects of the orientation, the need for the checklist is driven by regulatory
agencies. Ann states, “the Joint Commission really only requires for clinical people, but when
we are talking in terms of organizational improvement, we have required every job description to
have a checklist.” Marie discusses the importance of the checklist for new employees learning
the skills they need to perform their job and facilitating a comfortable work environment. She
states, “they demonstrate the skills and are checked off on their competency on those initial skills
and they are not expected to perform those until they do feel comfortable.”
Another aspect of the ongoing or department specific orientation which was discussed
was the preceptorship for nurses. The preceptor program begins with current nurses being
120
trained as preceptors. Then the trained nurse is paired with a new nurse to help with the
orientation process. Ellen describes this pairing as a “mentor type” relationship. She goes on to
describe the preceptor program as “picking our preceptors, have them go through training so that
they can assist in developing the nurses the way they need to be developed to function
independently on the unit.” The precepting relationship as described by Marie, is one that
“could last from four weeks to maybe eight weeks.”
The ongoing or department specific orientation was recognized by six of the nine
stakeholders as being an important part of the orientation process. More specifically, the
stakeholders the discussed the importance of the orientation checklist and work with a preceptor
as being important parts of the ongoing orientation process. Ann goes on to say that the
department specific orientation eventually turns into continuing education. She states, “that unit
specific kind of orientation … it goes on forever, it turns into continuing education, then after
that it is all joined together.”
However, before ongoing orientation can take place, employees must first be exposed to
the general orientation program. Employee participation in the general orientation program was
described by all nine stakeholders as a way for information to be distributed to new employees.
Stakeholders named topics such as hospital ethics, mission, values, strategic plan, the overall
healthcare system, safety, and employee benefits as topics which new employees should know
upon leaving the general orientation program. Betsy, the new unit secretary had just participated
in the orientation program as a learner, she recalled that
the mission and values, you know the “This is [the hospital]” video all that told about the
hospital, as far as all the other, like abuse and neglect, you the benefits of course, safety,
the ethics course, all that you know pertains to my job, so it all helped.
121
Ann talks about general orientation as fulfilling new employees’ basic need to know about
hospital culture and safety. She states, “they do need to know if there is a fire what they need to
do and they do need to know about our mission and values.” Likewise, Donna states, “the main
reason you have orientation is to inform, but also to set the stage just for the culture.”
While all nine stakeholders named providing information as an important part of the
orientation process, four of the stakeholders acknowledged that so much information is presented
during the orientation program, that it is important for new employees to at least understand how
to get more information once on the job. Janice states that as new employees “we have lots of
materials to learn and we can’t be expected to remember everything, but I want to make sure
they know where to find it.” Ann also hopes that orientation will prepare a new employee to find
answers to questions once the employee is on the job:
So if you can at least give them a phone number, link them to a face, you know that is
why I think that is so important for them to get information because after they are out of
that classroom, they will think ‘you know they told me that.’ Now if they could look
back, you know have something concrete. That is why I really like what [Sally] has done
with the orientation booklets, because there is a map in there and some essential kinds of
things that you might need, but you could take notes on it too.
So in addition to providing the basic knowledge needed to function in the hospital, the
orientation program is also expected to help new employees understand how to get questions
answered once on the job.
Finally, compliance with regulatory bodies is an important outcome of the orientation
program. Six of the nine stakeholders named compliance as a part of the orientation process.
As mentioned previously, the Joint Commission plays a role in the way in which the orientation
122
program is structured. Joint Commission requires that orientation be conducted at a general
hospital wide level and at the level of the job. Therefore, to comply with these standards the
structure of orientation includes both a general orientation program and a job specific orientation.
However, Joint Commission is not the only regulatory body involved in the orientation process.
Sally outlines all of the regulatory bodies as follows:
The HIPPA comes from the person; I think she calls herself the safety officer. And then
we have, well we have a safety officer that is over the electronic format. So that resides
really with health information management. Corporate compliance and the Risk Manager
are the same person. So those things come from that area. And then the OSHA is going
to come from public safety. And then the Joint Commission is really the piece that
comes from HR.
While parts of the orientation program were mandatory, Donna felt that the hospital tried to more
than simply what is required. She states, “we have got to do orientation, but I think we do more
than is absolutely necessary, we try to go above and beyond the bare requirements.” Therefore,
compliance is an important influence on the orientation program, but not the driving factor.
In summary, creating a program theory that is representative of the like ideas among
stakeholders is one way to create a program theory which then can be used to guide an
evaluation. In this case, the initial step in the program theory is participation in the orientation
program, resulting in general knowledge of the hospital. This knowledge helps employees
understand the religious nature of the hospital, and their fit with this philosophy. From there
employees move to ongoing or department-specific orientations, that prepare employees to
confidently perform their job duties. The end result is good patient care achieved through patient
safety and satisfaction.
123
Differences
The program theory created to represent the like ideas among stakeholders presents one
view of how the new employee orientation program operates. However, this program theory is
not representative of all the ideas expressed by stakeholders. Each of the nine stakeholders had
unique elements in their program theory, usually specific to their job or role in the hospital.
These unique elements were essentially lost when the program theories were combined. The
following table outlines the components of each stakeholder’s program theory not included in the
combined program theory. (See Table 6.1)
Table 6.1. Overview of Unique Elements of Stakeholders’ Program Theories Name (Stakeholder Group)
Title/Department Unique Elements of Stakeholders’ Program Theories
Sally (Program Planner)
Resource Development Coordinator/Training and Education
• Use of signed statements to indicate a topic has been covered
Ann (Program Planner)
Director of Education and Training Department/Training and Education
• Transfer of information through a variety of delivery formats, handouts, and providing contact information
• Upper management’s role in helping new employees feel welcome
Marie (Program Planner)
Clinical Staff Educator/Training and Education
• Nurses feel that education and support are available
• Nurses maintain professional licensure Donna (Hospital Administration)
Manager of Employment and Employee Relations/Human Resources
• Orientation is part of a smooth hiring process
• Keep accurate personnel files about participation in orientation process
Brad (Teacher)
Officer of Performance Improvement, Risk Management, and Corporate Compliance
• Employees gain an understanding of the performance improvement process, role in the risk management process, and an understanding of corporate compliance issues
• Employees gain an understanding of department specific issues related to these topics
124
Table 6.1. Overview of Unique Elements of Stakeholders’ Program Theories Continued Name (Stakeholder Group)
Title/Department Unique Elements of Stakeholders’ Program Theories
Ellen (Learner)
Director of Women’s Services
• Provide information to lead employees in her unit
Betsy (Learner)
Unit Secretary/ Women’s Services
• Having experts as presenters • Help from other employees during unit
specific orientation Linda (Affected Public)
Director of Patient and Guest Relations
• Introduce new employees to “The Customer” program
As the table indicates, each stakeholder expressed unique element of her or his program
theory. In most cases the elements unique to each stakeholder were reflective of the job or role
held at the hospital. For example, Marie, the Clinical Staff Educator, had two unique aspects of
her program theory. First, she wanted nurses to leave the orientation program feeling that further
professional development and support were available to them once they began working. As the
Clinical Staff Educator, Marie is responsible for providing continuing education to nurses, so it is
understandable that she would emphasize this aspect of the program. Marie also expected that
participating in the orientation program would help nurses maintain their professional license.
This outcome was expected to be achieved by insuring that nurses felt comfortable performing
the entire job.
Janice, a Diabetic Educator at the hospital, also had unique expectations of the orientation
program that reflect her role as an educator. She considers interaction with learners a key part of
her program theory. As a matter of fact, all of the outcomes she hoped to achieve through the
orientation program were contingent on having interaction with learners during her presentation.
She also valued the use of demonstration and return-demonstration as teaching technique. She
uses this teaching technique in the orientation program with nurses, and also on the job when
teaching patients to use their glucose monitoring equipment.
125
Summary
This chapter has outlined the similarities and differences which exist between
stakeholders’ program theories. In order to address the similar aspects among program theories,
a program theory was created which represented like ideas among stakeholders. However, this
program theory was not representative of all ideas expressed by stakeholders. In order to address
these differences, the unique aspects of each stakeholder’s program theory were outlined. These
unique aspects were not represented when the program theories were combined. Many of the
unique aspects identified were representative of the job or role the employees held at the
hospital.
126
CHAPTER 7
SUMMARY, DISCUSSION, AND RECOMMENDATIONS
This chapter includes a summary of this study’s problem, purpose, research questions,
method, and findings, followed by a discussion of the findings. The chapter concludes with a
presentation of recommendations for theory and practice and future research.
Summary of the Study
HRD professionals are challenged with the task of providing education which will change
behavior and improve organizational functioning. However, the dominant outcome-focused
evaluation strategies do little to help HRD professionals explain how these challenging, yet
important outcomes will be achieved (Russ-Eft & Preskill, 2001). Theory-driven evaluation has
been proposed as an alternative evaluation strategy, which addresses some of the weakness of
outcomes-focused evaluation (Hilbert et al., 1997). Theory-driven evaluation is an evaluation
strategy that uses a program theory to guide the evaluation. A program theory is a map of the
channels through which outcomes are achieved, opening up the black box in which training has
long functioned. During the evaluation process the program theory can be developed through a
variety of ways, including using social science theories, practitioner theories, or some
combination of the two. Practitioner theories, the focus of this research, consist of the
stakeholders’ understanding of how the program works to achieve the desired outcomes. When
practitioner theories about program functioning are used to create a program theory, many
political issues must be addressed. Cervero and Wilson (1994) have establish that program
planning is a political process. Likewise, many evaluators have argued that evaluation is a
127
political and ethical activity (House & Howe, 2000; Palumbo, 1987; Ryan & DeStefano, 2000;
Weiss, 1987).
However, it is unclear how these practical theories, held by program stakeholders, which
exist in a complex sociopolitical context are combined to form a program theory that can be used
to guide an evaluation. Before theory-driven evaluation can be incorporated into the repertoire
of evaluation approaches utilized by HRD practitioners, issues surrounding the development of
the program theory must be addressed. The purpose of this research was to explore the ways that
multiple stakeholders’ interests are represented in the program theory building process which can
be used for a theory-driven evaluation. More specifically, this study investigated the
development of a program theory in an organizational setting. The following three questions
were used to guide the research:
1. What are the program theories of stakeholders?
2. What similarities exist among the stakeholders’ program theories?
3. What aspects of the program theories are unique to each stakeholder?
In order to address the purpose and research questions used to guide this study, a
qualitative case study methodology was chosen. Bogdan and Biklen (1998) offer a definition of
case study that states it “is a detailed examination of one setting, or a single subject, a single
depository of documents, or one particular event” (p. 54). Merriam (1998) states that the “single
most defining characteristic of case study research lies in delimiting the object of study, the case”
(p. 27). To assist in the selection of a case to study, two criteria were established to reflect the
purpose of the study. The case was chosen based on the following criteria:
1. the case is located within an organizational setting, in the private sector;
2. the case is an employee training program.
128
Based on these criteria, the case chosen for study was an educational program for
employees at a 196 bed hospital located in a three-county metropolitan area of 126,000 people.
The hospital is a private, non-profit, self-supporting acute care network. Within this hospital one
educational program was identified, an orientation program for new hospital employees. The
program was five days in length and included general orientation, clinical orientation, and
nursing orientation.
Three data collection methods were used to address the research questions, one primary
and two secondary. First, interviews with program stakeholders were used as the primary data
collection method. Program stakeholders were selected in the following categories: program
planner, teacher, learner, hospital administration, and affected public. Within these categories
nine program stakeholders were interviewed. Secondary data sources consisted of observations
and document reviews. The secondary data sources were used to gain a more in-depth
understanding of the case.
Based on the observations and document collection, a detailed description of the five day
new employee orientation was generated. Interview data were used to address the three research
questions. For research question one: What are the program theories of stakeholders?, data was
analyzed and a program theory was developed for each of the nine stakeholders. These nine
program theories were then compared to answer research questions two and three: What
similarities exist among the stakeholders’ program theories? and What aspects of the program
theories are unique to each stakeholder? To represent the similarities among stakeholders, a
program theory was created which included all the like ideas expressed in the stakeholders’
program theories. The unique aspects of each stakeholder’s program theory were then presented.
129
Based on the data analysis three conclusions were drawn and recommendations for practice were
suggested.
Conclusions and Discussion
This study had three major conclusions: 1) similarities exist between stakeholders’
program theories, 2) individual stakeholders also have differing and conflicting, components of
their program theories, with many of the differences accounted for by the role of the stakeholder
within the organization, and 3) through the program theory building process, a broad range of
outcomes and processes were specified by program stakeholders.
Conclusion One
The first conclusion of this research is that similarities exist between stakeholders’
program theories. As shown in the analysis of research question two, a model containing like
ideas among stakeholders was developed. This program theory is representative of all nine
stakeholders, and each element included was expressed by at least two stakeholders. That is,
some consensus existed in every stakeholder’s program theory with at least one other
stakeholder. It is important that stakeholders had similar ideas about the foundations of the
program, such as compliance with regulatory bodies and good patient care.
This finding supports the major assumption in the theory-driven evaluation literature that
a program theory is one fixed concept of how a program functions. For example, the following
definitions were found: “the set of beliefs and assumptions that undergird program activities”
(Weiss, 1997b, p. 503), and “a specification of what must be done to achieve the desired goals,
what other important impacts may also be anticipated, and how these goals and impacts would be
generated” (Chen, 1990b, p. 43). In both of these definitions program theory is presented as if it
is one set of beliefs or one specification of how the program works.
130
Furthermore, the assumption of one definitive program theory existing is necessary for
the testing of the program theory. That is, one program theory must be established so that the
linkages can be tested and confirmed or denied. To test the program theory using techniques
such as linear structural relations, confirmatory factor analysis, latent variable structural models,
and path analysis can be used (Lipsey & Pollard, 1989). However, these techniques focus on the
pathways of one model, in this case one program theory. This research confirms that there is
validity to the assumption that among stakeholders there are shared concepts of how the program
operates to achieve the desired outcomes.
For analysis purposes, one program theory was created which represented the similarities
among stakeholders. This program theory was developed by the researcher, based on previous
analysis, and leaves out significant differences between the stakeholders. It therefore should not
be the basis for an evaluation. Despite these weaknesses, the exercise of identifying similarities
was useful for several reasons. First, the exercise confirmed that similarities do exist among
stakeholders’ concepts of program functioning. Second, this program theory could be a useful
starting place in developing greater consensus among program stakeholders. Presenting this
program theory to a group of stakeholders, which contains some of the elements they agree are
important, would allow for discussion to begin. The resulting program theory could be more
inclusive of all stakeholders’ ideas and have greater levels of ownership by stakeholders.
Conclusion Two
The second conclusion of this research is that in developing a single program theory,
evaluators will encounter individual stakeholders, who have differing, and possibly conflicting,
program theories. This research has shown that there are as many program theories as there are
stakeholders. The differences found in stakeholders’ program theories ranged from Janice giving
131
priority to learner interaction in the classroom, to Donna feeling that new employee orientation
was one essential part of a smooth hiring process. While the elements unique to each
stakeholder were not in direct conflict, only so many can be included in the evaluation process.
In the process of reaching consensus about what is important to be included in the evaluation, it
is possible that conflicts could arise.
Yet, the theory-driven evaluation literature has not addressed the complexity of involving
multiple stakeholders in the development of the program theory. As established in conclusion
one, program theory is often discussed as one fixed concept of how the program is supposed to
work. This assumption is necessary if the program theory is to be tested, so that the linkages can
be confirmed or denied.
However, outside of the theory-driven evaluation literature, much work has been done on
the integration of stakeholders into the evaluation process. Specifically, responsive evaluation,
participatory/collaborative evaluation, and empowerment evaluation are evaluation techniques
which directly address program stakeholders and their interests in the evaluation process.
Responsive evaluation focuses on program stakeholders, including program participants, staff
and other audiences (Russ-Eft & Preskill, 2001). The key to this evaluation approach is to
identify individual stakeholders concerns about the program and address the concerns through
the evaluation process. Participatory or collaborative evaluation uses a democratic decision
making process as a way to allow stakeholders to reach an understanding about program
effectiveness. Empowerment evaluation works with program participants to help them improve
their own programs through self-evaluation and reflection.
In addition to these three evaluation techniques, research has been done which supports
the idea that multiple stakeholders have varying interests in any given educational program.
132
Amba (2000) presents a case study which challenges the assumption that consensus exists
concerning evaluation criteria. Instead a pluralist perspective is taken, and it is assumed that “it
is more likely that stakeholders will have different, and sometimes conflicting views on an
evaluated program” (p. 199). Michalski and Cousins (2001) explore differences in stakeholder
perceptions concerning the purposes, processes, and consequences of evaluation within an
organization. The authors conclude that for “training evaluation to have fuller importance,
assumptions about stakeholder homogeneity in training evaluation practice need to be questioned
more thoroughly” (p. 50).
The findings of this research study challenge the premise of program theory as it is
discussed in the theory-driven evaluation literature. By outlining the program theories of various
stakeholders, it can be seen that multiple and sometimes conflicting understandings of program
functioning exist. These findings are supported by Amba (2000) and Michalaski and Cousins
(2001). While the program theory development literature has not addressed these issues,
stakeholder focused evaluation techniques have addressed the multiple stakeholders, with
varying interest, in evaluation process. This research bridges the gap between what is known in
stakeholder focused evaluation and what is known in program theory development.
This research has demonstrated that not only do program theories vary from stakeholder
to stakeholder, but that some of the differences can be attributed to the position of the
stakeholder within the organization. That is, each stakeholder expressed ideas about how the
orientation program functioned which reflected their job or role in the organization. Again
Michalaski and Cousins’s (2001) findings support this conclusion. While exploring
stakeholders’ perceptions about the purposes, processes, and consequences of evaluation within
an organization, the researchers found that the views of different stakeholder groups were
133
divergent, with most of the difference related to the job role of a stakeholder in the organization.
For example, the program planners were interested in evaluation results which highlight training
merit and worth, in order to sustain and expand the training budget. Managers on the other hand
were interested in evaluation results which could improve training for their employees.
Michalaski and Cousins’s (2001) findings support the conclusion that stakeholders’ interest in
program evaluation is influenced by their role in the organization.
Because the theory-driven evaluation literature has given little attention to the process of
building a program theory, differences among stakeholders have not been addressed. It is
important for evaluators to know that stakeholders’ program theories do not vary at random.
Instead, the program theories reflect the interests of the stakeholder, interests which are related to
the position the stakeholder holds within the organization. This finding is especially important
for HRD practitioners who use theory-driven evaluation within an organizational setting. The
evaluator must be aware that stakeholders have unique interests in any given educational
program based on the role or job held within the organization. These unique interests will affect
the stakeholders’ understanding of how the program functions to achieve the desired outcomes.
Conclusion Three
The third conclusion of this research is the program theory building process allows a
broad range of outcomes and processes to be specified by program stakeholders. In the process
of eliciting program theories from stakeholders, important outcomes such as learners’ feelings on
the job and completing necessary steps in the compliance process were expressed. Theory-
driven evaluation has been offered as an alternative to outcome focused evaluation techniques
which have dominated the HRD evaluation literature, namely, Kirkpatrick’s four-level model.
Hilbert, Russ-Eft, and Preskill (1997) reviewed evaluation models in the HRD and psychology
134
literature. Of the 57 journal articles describing evaluation models, 44 (or 77%) included
Kirkpatrick’s model. The dominance of outcome focused evaluation techniques has left training
in a black box, giving HRD professionals no understanding of how outcomes are achieved.
Therefore, Russ-Eft and Preskill’s (2001) recommendation that theory-driven evaluation should
be a future direction for HRD evaluation practice is much needed. This suggestion provides
HRD evaluators with an alternative which will allow for a more in-depth understanding of
program functioning. This research confirms that a broad range of outcomes and necessary
processes can be identified when Kirkpatrick’s four-levels of outcomes are not used to define the
evaluation.
Outcomes elicited through the theory building process could be perceived as existing
outside of Kirkpatrick’s four-levels of outcomes: reaction, learning, behavior, and results. For
example, Marie expressed that an important outcome of the new employee orientation program
was that it helped nurses maintain their professional licensure. This outcome does not fit neatly
into any of Kirkpatrick’s four levels. Similarly, several stakeholders express the importance of
employees feeling comfortable performing their jobs as a result of attending the orientation
program. Once again, feelings on the job are not captured by Kirkpatrick’s evaluation model.
In addition to a broad range of outcomes being identified, building program theory
allowed for important process component of the new employee orientation program to be
identified. For example, Sally expressed that it was necessary for learners to sign statements,
indicating that topics had been presented. This step was necessary to insure that the program
was in compliance with the regulatory bodies. Likewise, Betsy found that having experts as
presenters was a key to her learning experience. Both of these are example of important
processes or elements that were necessary to facilitate the outcomes of these programs.
135
Kirkpatrick’s four-level outcome model does not capture these important process variables that
facilitate the outcomes. By building program theory through discussions with stakeholders,
necessary processes can be identified which facilitate the outcomes of the program. This allows
evaluators to understand not only if the program worked (outcomes), but how the program
worked (processes). This research confirms that the program theory building process can open
up the black box of training, to specify the channels through which the outcomes are achieve.
Recommendations
Several recommendations have been made for practice and research, based on the
findings and conclusions of this research. Recommendations for practice focus on the uses of
theory-driven evaluation by HRD practitioners, specifically need for evaluators to negotiate a
single program theory from stakeholders’ multiple theories, the role of the evaluator in this
process, and when to use theory-driven evaluation. Recommendations for research suggest
further research that is needed in the area of building program theory.
Negotiating a Single Program Theory
If HRD professionals adopt theory-driven evaluation as an evaluation strategy, the
negotiated nature, as well as the stakeholders’ whose ideas are represented must be considered.
That is, evaluators need to negotiate a single program theory from stakeholders’ multiple theories
within socially and organizationally defined contexts of power. Not all ideas expressed by all
stakeholders can be incorporated into a single program theory, and the situation is further
complicated when conflicting ideas about program functioning are presented. Because the
evaluator will be faced with multiple, and often conflicting, program theories, it will be
necessary to negotiate a single theory. This negotiation process will take place in an
organization with socially and organizationally defined contexts of power. Organizational power
136
differences are those tied to organizational design and structure. Social power differences are
those that are interpersonal or caused by temporary organizational conditions.
Little direction has been offered to evaluators in navigating the complex political process
involved in creating one program theory. According to the stakeholder approach to program
theory development, the program theory is developed by capturing stakeholder perceptions of the
program through review of relevant documents and interviews. While most definitions of
program theory given in the theory-driven evaluation literature did assume one program theory
exists, some authors did acknowledge the complexity involved in the theory building process.
Weiss (1997a) states, “theories do not have to be right, and they do not have to be uniformly
accepted” (p. 503). This statement acknowledges that several program theories may exist and
that they may vary from stakeholder to stakeholder, supporting the second conclusion of this
study. Likewise, Rogers et al. (2000) defines program theory as “the underlying assumptions
about how programs are expected to work” (p. 5). Once again this definition acknowledges that
multiple assumptions underlie any given educational program.
However, little guidance is given on how to handle the multiple assumptions which exist.
In the theory-driven evaluation literature it is suggested that discussions with key stakeholders
can be used to resolve differences in perspective on the program theory. Chen (1990a) states,
“after the differences are resolved, a formal program theory can be completed” (p. 11). Bickman
(1987) suggests that the practitioner may be using nothing more than a small number of simple
assumptions concerning how the program will work, and it is the role of the evaluator to capture
the assumptions as a more explicit causal chain of events concerning program functioning. The
literature goes on to suggest that the evaluator’s role in building program theory building is
passive. That is, evaluators should “avoid relying on their own knowledge or expertise in
137
constructing program theory” (Chen, 1990a, p. 11). Instead the program theory should emerge
from the data collected. While these suggestions give evaluators some guidance for dealing with
stakeholders’ practical understandings of program functioning, there are no clear cut mechanisms
for resolving differences.
Cervero and Wilson (1994) outline a process for negotiation during the program planning
process and the same framework could be extended to the development of the program theory.
More specifically, Cervero and Wilson (1994) have identified strategies that can be used to
negotiate the interests of various stakeholders in different planning environments where there are
organizational and social power differences. For example, in situations where there are
systematic power differences (those tied to organizational structure) and there is consensus about
the appropriate actions, networking can be used as a tool to negotiate the power relationships.
Networking involves forming relationships with stakeholders that share similar interests so that a
program can be designed to satisfy the interests (Cervero & Wilson, 1994). There will also be
situations in which there are systematic power differences, but conflictual relations among
stakeholders. Networking will not be sufficient in these situations, because of the possibility for
only the interests of those with more power to be served. In these situations the program planner
must keep asking whose interests are being served and whose interests should be served. In the
face of opposing interests in systematically differentiated power relationships it will be necessary
to counteract these interests. Counteracting can take the form of mobilizing groups of people to
counter the effects of established interest (Cervero & Wilson, 1994). This will give all the
stakeholders a voice in the planning of educational programs that matter to them.
These are strategies for negotiating power relationships that are based on organizational
design. Cervero and Wilson (1994) also offer strategies for negotiating power relationships that
138
are interpersonal or temporary due to the organizational conditions. When there is general
consensus among stakeholders in socially differentiated power relationships, evaluators should
use the most satisfactory alternatives available in the given context (Cervero & Wilson, 1994).
When socially differentiated power relationships exist, and there are conflicts of interest, a
bargaining strategy can be used. Bargaining is a way of negotiating that says “I’ll give you X, if
you give me Y.” Through this process, compromise can be reached.
This process of negotiating (using appropriate strategies for the given situation) can help
create a democratic planning process and also a democratic evaluation process. It will not
always be easy to determine the political context in which you are working. Deciding which
negotiation strategy will be appropriate may be difficult and a combination of strategies may be
needed in any given planning context.
These negotiation strategies offered by Cervero and Wilson (1994) for the program
planning process can be transferred to the development of the program theory. Like program
planners, evaluators using theory-driven evaluation will be presented with multiple stakeholders
with varying and sometimes conflicting interests. It is important that evaluators have the
negotiation skills needed, so that stakeholders have a voice in the evaluation of the educational
programs that matter to them. Furthermore, the evaluator will be expected to conduct those
negotiations in contexts in which there are variations in power on both an organizational and
social level.
Evaluator’s Role
This research should caution HRD professionals about the use of theory-driven
evaluation. More specifically, HRD professionals choosing to use theory-driven evaluation
within their organizations should consider the negotiated nature of the program theory
139
development process. According to existing literature, the evaluator’s role in the program theory
development process is a passive one. The assumption also exists that any discrepancies among
stakeholders can be resolved through a discussion process. However, this is a simplistic view of
a very complex situation which involves multiple stakeholders, each with unique understanding
of how and why the program works to achieve the desired outcomes.
If HRD professionals chose to use theory-driven evaluation, then they must be prepared
to take on the role of a negotiator during the program theory building phase of the evaluation.
That is, the interests of multiple program stakeholders and the power relationships of these
stakeholders must be considered. The political context of the educational program must be
negotiated, so that the various stakeholders are represented in educational evaluations which
matter to them. While determining the political context of any given educational program will
not be easy, Cervero and Wilson (1994) provide negotiation strategies for various power
differences and levels of consensus. By using the negotiation framework spelled out by Cervero
and Wilson (1994) during the program theory building phase of theory-driven evaluation, HRD
professionals have the opportunity to create an evaluation which both represents multiple
stakeholders’ interest and also helps explain program functioning.
Use of Theory-Driven Evaluation in HRD
Introducing theory-driven evaluation as an alternative to available HRD evaluation
models holds great promise. Like the Holton (1995) integrated model, theory-driven evaluation
goes beyond a simple output model to consider important contextual factors affecting the
educational program. The unique contribution of theory-driven evaluation is the ability to
consider a broader range of influences on educational program success in a situationally specific
way.
140
Of course theory-driven evaluation is not appropriate in every evaluation context. The
barriers to the use of this type of evaluation are quite serious. Bickman (1989) identified
multiple barriers to use. First, increased costs can be incurred from activities such as, a lengthy
planning session, the development of measurement tools and the collection of data on the various
components of the model. Second, the evaluator must be technically competent in the strategies
used to implement a theory-driven evaluation approach. In addition, the evaluator is expected to
be knowledgeable in the problem and treatment literature. This creates the need for evaluators
who are technically competent in evaluation research techniques as well as HRD.
Despite these limitations, theory-driven evaluation remains an important addition to the
HRD literature. The ultimate strength of theory-driven evaluation for practitioners is the quality
of knowledge that is produced. This approach provides practitioners with a tool for creating
knowledge about the context specific characteristics of training programs that are successful in
the organization. Instead of providing information about the outcomes of one specific program,
theory-driven evaluation allows for the understanding of why the program worked. This
knowledge can then be used in the next program planning process. This moves evaluation away
from a tool to provide information about outcomes, which are used to justify HRD practices, to a
tool that also provides knowledge for those working within HRD. By creating new knowledge
for HRD practitioners, theory-driven evaluation has the potential to improve HRD functioning
within an organization.
In addition to creating new knowledge about educational programs, theory-driven
evaluation can contribute to new knowledge in the field of HRD. First, it may be possible to
generalize the results of the evaluation to other situations (Chen &Rossi, 1989). In other words,
if desired outcomes were produced through specified intervening variables, it may be possible to
141
replicate the findings in similar educational programs. Second, knowledge can be generated
which contributes to the broader body of evaluation knowledge (Chen, 1994). This generation of
new knowledge is crucial to the development of program evaluation as a field, and in turn to the
field of HRD.
However, as Finney and Moos (1989) point out, “the greatest benefit of a theoretical
foundation for evaluation research is a very practical one- the enhanced usefulness of findings”
(p. 314). More, specifically, findings can help make instrumental changes to improve and
develop the educational program and also influence the way stakeholders conceptualize the
problem and the educational program.
Recommendations for Research
While this research explored the program theory development process and illuminated
the complex political context in which this process happens, further research is still needed. The
most obvious next step in the incorporation of theory-driven evaluation into the field of HRD is
the use of the theory in an applied setting. Chen (1994) reiterates this need, calling for more
research examples in various evaluation situations. This research offers a partial example, but
does not carry the evaluation through the negotiation and testing of the program theory.
Future research on the program theory building part of theory-driven evaluation should
focus on negotiation among stakeholders to derive one program theory. This research should
highlight both the social and organizational contexts of power in which the program stakeholders
operate, and the negotiations strategies needed to address stakeholders’ interests. More
specifically, research is needed to demonstrate the utility of Cervero and Wilson’s (1994)
program planning negotiation strategies in the program theory building process. While this
research study identified the need for a mechanism to create program theory, when multiple
142
stakeholders are involved, it did not actually implement these negotiation strategies. Therefore,
further research is needed.
Final Note
This case study explored the program theory development process associated with
theory-driven evaluation in detail. Based on this in-depth look at program stakeholders, their
roles in the organization, their roles in the program, and their program theories,
recommendations as to the role of the evaluator have been offered. That is, when developing
program theory, evaluators must negotiate the interests of the various stakeholders of the
program to insure that the program theory used to guide the evaluation represents those
stakeholders to whom the outcomes matter. It has been suggested that Cervero and Wilson’s
(1994) negotiation strategies for program planners be used as a guide when constructing program
theory.
This research also recommends that theory-driven evaluation will not be appropriate in
every evaluation situation. There are increased costs and time associated with this evaluation
strategy. The use of theory driven-evaluation also demands evaluators to be highly trained
evaluation practitioners with the skills to carry out the rigorous evaluation designs. This
involves qualitative skills needed for the development of the program theory, as well as,
quantitative skills to test the program theory. While the theory-driven evaluation process maybe
more demanding on practitioners, the result is a comprehensive understanding of the problem,
the educational program and the outcomes produced.
143
REFERENCES
Abma, T. A. (2000). Stakeholder conflict: A case study. Evaluation & Program Planning, 23,
199-210.
Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick's levels of training criteria: Thirty years later.
Personnel Psychology, 42, 331-342.
Argyris, C. (1993). Knowledge for action: A guide to overcoming barriers to organizational
change. San Francisco: Jossey-Bass Publishers.
Baker, E. L., & Niemi, D. (1997). School and program evaluation. In D.C. Berliner & R. C.
Calfee (Ed.), Handbook of educational psychology (pp. 926-944). New York: Macmillan.
Baldwin, T. B., & Ford, J. K. (1988). Transfer of training: A review and directions for future
research. Personnel Psychology, 41, 63-105.
Bateson, G. (1972). Steps to an ecology of mind. Chicago: The University of Chicago Press.
Bernstein, D. J., Whitsett, M. D., & Mohan, R. (2002). Addressing sponsor and stakeholder
needs in the evaluation authorizing environment: Trends and Implications. In M. D.
Whitsett (Ed.), Responding to sponsors and stakeholders in complex evaluation
environments (pp. 89-99). San Francisco: Jossey-Bass.
Bickman, L. (1987). The functions of program theory. In L. Bickman (Ed.), Using program
theory in evaluation (pp. 5-18). San Francisco: Jossey-Bass Inc., Publishers.
Bickman, L. (1994). An optimistic view of evaluation. Evaluation Practice, 15, 255-259.
Bogdan, R. C., & Biklen, S. K. (1998). Qualitative research in education. An introduction to
theory and methods . Massachusetts: Viacom Company.
144
Brinkerhoff, R. O. (1987). Achieving results from training: How to evaluate human resource
development to strengthen programs and increase impact. San Francisco: Jossey-Bass
Publishers.
Brinkerhoff, R. O. (1988). An integrated evaluation model for HRD. Training and Development
Journal, 42, 66-68.
Bushnell, D. S. (1990). Input, process, output: A model for evaluating training. Training and
Development Journal, 44, 41-43.
Campbell, D. T. (1994). Retrospective and prospective on program impact assessment.
Evaluation Practice, 15, 291-298.
Cervero, R. M. (1985). Continuing professional education and behavioral change: A model for
research and evaluation. The Journal of Continuing Education in Nursing, 16, 85-88.
Cervero, R. M., Rottet, S., & Dimmock, K. H. (1986). Analyzing the effectiveness of continuing
professional education at the workplace. Adult Education Quarterly, 36(2), 78-85.
Cervero, R. M., & Wilson, A. L. (1994). Planning responsibly for adult education: A guide to
negotiating power and interests. San Francisco: Jossey-Bass Publishers.
Chen, H.-t. (1990a). Issues in constructing program theory. In L. Bickman (Ed.), Advances in
program theory (pp. 229-306). San Francisco: Jossey-Bass INC., Publishers.
Chen, H.-t. (1994). Current trends and future directions in program evaluation. Evaluation
Practice, 15(3), 229-238.
Chen, H.-t., & Rossi, P. H. (1980). The multi-goal, theory-driven approach to evaluation: A
model linking basic and applied social science. Social Forces, 59, 106-122.
145
Chen, H.-t., & Rossi, P. H. (1983). Evaluating with sense: The theory-driven approach.
Evaluation Review, 7(3), 283-302.
Chen, H.-t., & Rossi, P. H. (1989). Issues in the theory-driven perspective. Evaluation &
Program Planning, 12(4), 299-306.
Davies, C. A. (1999). Reflexive ethnography: A guide to researching selves and others. New
York: Routledge.
Dixon, N. M. (1987). Meeting training's goals without forms. Personnel Journal, 66, 108-115.
Forester, J. (1989). Planning in the face of power. Berkeley: University of California Press.
Foxon, M. J. (1994). A process approach to transfer of training. Part 2: Using action planning to
facilitate the transfer of training. Australian Journal of Educational Technology, 10(1), 1-
18.
Garavaglia, P. L. (1996). Applying a transfer model to training. Performance & Instruction,
35(4), 4-8.
Goldstein, I. L. (1993). Training in organizations (3rd ed.). Pacific Grove, California: Brooks
Publishing Company.
Hennessy, M., & Greenberg, J. (1999). Bringing it all together: Modeling intervention processes
using structural equation modeling. American Journal of Evaluation, 20(3), 471-480.
Hilbert, J., Russ-Eft, D., & Preskill, H. (1997). Evaluating training. In D. Russ-Eft (Ed.), What
works: Assessment, development, and measurement (pp. 109-150). Alexandria, VA:
ASTD.
Holton, E. F. I. E. (1996). The flawed four-level evaluation model. Human Resource
Development Quarterly, 7(1), 5-21.
146
House, E. R., & Howe, K. R. (2000). Deliberative democratic evaluation. In L. DeStefano (Ed.),
Evaluation as a democratic process: Promoting inclusion, dialogue, and deliberation
(pp. 3-12). San Francisco: Jossey-Bass Publishers.
Industry Report. (2000). Training Magazine, 37(10), 45-94.
Jarvis, P. (1999). The Practitioner-researcher: Developing theory from practice. California:
Jossey-Bass Inc., Publishers.
Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development
handbook (pp. 18-1 – 18-12). New York: McGraw-Hill Book Company.
Kvale, S. (1996). InterViews: An introduction to qualitative research interviewing. Thousand
Oaks: Sage Publications.
Lipsey, M. W., & Pollard, J. A. (1989). Driving toward theory in program evaluation: More
models to choose from. Evaluation & Program Planning, 12(4), 317-328.
McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: A tool for telling your program's
performance story. Evaluation and Program Planning, 22(1), 65-72.
Mercier, C., Piat, M., Peladeau, N., & Dagenais, C. (2000). An application of theory-driven
evaluation to a drop-in youth center. Evaluation Review, 24(1), 73-91.
Merriam, S. B. (1998). Qualitative research and case study applications in education. Revised
and Expanded from Case Study Research in Education. California: Jossey-Bass Inc.,
Publishers.
Merriam, S. B., & Associates. (2002). Qualitative research in practice. San Francisco: Jossey-
Bass.
147
Michalski, G. V., & Cousins, J. B. (2001). Multiple perspectives on training evaluation: Probing
stakeholder perceptions in a global network development firm. American Journal of
Evaluation, 22(1), 37-53.
Mohr, L. B. (1995). The qualitative method of impact analysis. American Journal of Evaluation,
20(1), 69-84.
Palumbo, D. J. (1987). The politics of program evaluation. Newbury Park: Sage Publications.
Reynolds, A. J. (1998). Confirmatory program evaluation: A method for strengthening causal
inference. American Journal of Evaluation, 19(2), 203-221.
Riggin, L. J. C. (1990). Linking program theory and social science theory. In L. Bickman (Ed.),
Advances in program theory (Vol. 47) (pp. 109-121). San Francisco: Jossey-Bass.
Rogers, P. J., Petrosino, A., Huebner, T. A., & Hacsi, T. A. (2000). Program theory evaluation:
practice, promise, and problems. In T. A. Hacsi (Ed.), Program theory in evaluation:
Challenges and opportunities (pp. 5-13). San Francisco: Jossey-Bass.
Rothwell, W. J. (1996). A survey about training evaluation practices in selected organizations:
Summary of results. Proceedings of the Academy of Human Resource Development,
USA, 2, 3-2.
Russ-Eft, D. (1986). Evaluability assessment of the adult education program (AEP): The results
and their use. Evaluation and Program Planning, 9(1), 39-47.
Russ-Eft, D., & Preskill, H. (2001). Evaluation in organizations. Cambridge, Massachusetts:
Perseus Publishing.
Ryan, K. E., & DeStefano, L. (2000). Evaluation as a democratic process: Promoting inclusion,
dialogue, and deliberation. San Francisco: Jossey-Bass Publishers.
148
Schon, D. A. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass
Publishers.
Scott, A. G., & Sechrest, L. (1989). Strength of theory and theory of strength. Evaluation &
Program Planning, 12(4), 329-336.
Stake, R. E. (1994). Case Studies. In N. K. Denzin (Ed.), Handbook of qualitative research (pp.
236-247). Thousand Oaks: Sage Publication.
Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures
and techniques (2nd ed.). Newbury Park, California: Sage.
Suchman, E. A. (1967). Evaluative research: Principles and practice in public service and
social action programs. New York: Russell Sage Foundation.
Torvatn, H. (1999). Using program theory models in evaluations of industrial modernization
programs: Three case studies. Evaluation & Program Planning, 22, 73-82.
Trochim, W. M. (1989). Outcome pattern matching and program theory. Evaluation & Program
Planning, 12, 355-366.
Tyler, R. W. (1949). Basic principles of curriculum and instruction. Chicago: The University of
Chicago Press.
Umble, K. E., Cervero, R. M., Yang, B., & Atkinson, W. L. (2000). Effects of traditional
classroom and distance continuing education: A theory-driven evaluation of a vaccine-
preventable diseases course. American Journal of Public Health, 90(8), 1218-1224.
Usher, R., Bryant, I., & Johnston, R. (1997). Reconceptualising theory and practice. In R. Usher,
I. Bryant, & R. Johnston (Eds.), Adult Education and the postmodern challenge (pp. 123-
141). New York: Routledge.
149
Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness.
Reprinted in Evaluation Practice, 17, 50-53.
Weiss, C. H. (1987). Where politics and evaluation research meet. In D. J. Palumbo (Ed.), The
politics of program evaluation (pp. 47-72). Newbury Park: Sage Publications.
Weiss, C. H. (1997a). How can theory-based evaluation make greater headway? Evaluation
Review, 21(4), 501-524.
Weiss, C. H. (1997b). Theory-Based evaluation: Past, present, and future. New Directions for
Evaluation, 76, 41- 55.
Weiss, C. H. (2000). Which links in which theories shall we evaluate? In T. A. Hacsi (Ed.),
Program theory in evaluation: Challenges and opportunities (pp. 35-45). San Francisco:
Jossey-Bass.
Wholey, J. S. (1987). Evaluability assessment: Developing program theory. In L. Bickman
(Ed.), Using program theory in evaluation (pp. 77-92). San Francisco: Jossey-Bass.
Worthen, B. R. (1996). Editor's note: The origins of theory-based evaluation. Evaluation
Practice, 17, 169-171.
Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1997). Program evaluation: Alternative
approaches and practical guidelines. New York: Addison Wesley Longman.
Yin, R. K. (1994). Case study research: Design and methods. Thousand Oaks: Sage
Publications.
Youtie, J., Bozeman, B., & Shapira, P. (1999). Using an evaluability assessment to select
methods for evaluating state technology development programs: The case of the Georgia
research alliance. Evaluation & Program Planning, 22, 55-64.
150
APPENDIX A
INTERVIEW GUIDE
1. What is your role in the orientation program?
2. What are your expectations of the orientation program? What outcomes do you expect
from the orientation program?
3. You mentioned_________ as a possible outcome of the program. What parts of the
program support or facilitate this outcome?
4. Can you explain why (this outcome) is particularly important to you?
Note: Questions three and four will be repeated for each of the outcomes identified in question
two.
151
APPENDIX B
PILOT DATA COLLECTION INTERVIEW GUIDE
I want to start by thanking you for agreeing to talk with me. Our purpose is to explore your
perspective of the orientation program at St. Mary’s. I am going to ask several questions about
the orientation program. My purpose is to gain your perspective about how the program works
and the outcomes that result.
1. What is your role in the orientation program?
• How long have you been in this role?
• Can you give me a history of the orientation program?
2. Why does the St. Mary’s need an orientation program?
• How was this need identified?
• What types of content are presented?
• What specific outcomes are desired?
3. What people or groups of people does the orientation program affect?
• Are there other people or groups of people affected by the program?
4. What has changed as a result of the orientation program?
• What benefits have you seen resulting from the orientation program?
5. I would like to ask you to think about specific parts of the orientation program:
• What specific parts of the program are effective?
• Which specific parts of the program are not effective?
152
APPENDIX C
HOW DOES THE PROGRAM WORK TO ACHIEVE THE DESIRED OUTCOMES?
Joint Commissions and HR Standards
Customer Service/Patient Satisfaction
Follow-up 7-week program
Compliance with Corporate Policies and Regulating Bodies
Employee Satisfaction
Safety procedures
25% of performance evaluation
Corporate Mission and Values
Employee –Organization Fit
Employee prepared for job
System Overview- Senior Leadership and Departments
Environmental and Infection Controls
Ethical Issues
Legal Issues
Knowledge of Reference Materials
153
APPENDIX D
OBSERVATION SCHEDULE
The following chart represents the dates that the new employee orientation program was offered. The dates in bolded represent observation days. Month General
