Augmenting Behavior-Modeling Training: Testing the Effects of Pre- and Post-Training Interventions Jon M. Werner, Anne M. O’Leary-Kelly, Timothy T. Baldwin, Kenneth N. Wexky A number of recent training authors have suggested that pre- and post-train- ing interventions may enhance training outcomes. In the present study, pre- and post-training interventions were added to an established behavior-mod- eling program on assertiveness, creatingfour conditions: (1) no intervention, (2) pretruining interventions, (3) post-training interventions, and (4) both. One hundred fifty trainees completed the module, and measures of trainee reaction, learning retention, and behavioral change were obtained. Results indicated that the post-training intervention strongly affected learning reten- tion, as well as reactions immediately following training, with moderate ef- fects on behaviol: No sign$cant effectswere observed between the pretrain- ing intervention and any ofthe trainee outcome measures. lmplications ofthe findingsfor training research and practice are discussed. An enormous amount of money is spent annually on organizational training- and-development programs. Unfortunately, much of this has little (or at least an unknown) impact on measures of training effectiveness. One estimate is that as little as 10 percent of the money spent on training leads to changes in trainee behavior back on the job (Georgenson, 1982). More recently, New- strom has written that only 15 percent of the skills learned in training remain with trainees one year after training (cited in Garavaglia, 1993).An increasing body of literature is looking for ways to combat this pervasive “transfer prob- lem” (Baldwin and Ford, 1988). One approach to this problem is to augment existing training with interventions designed to motivate trainees to learn and transfer training skills. Hw RESOURCE DEVELOFMENT QUARTERLY, vol. 5, no. 2. Summer 1994 8 Jossey-Bass Publishers 169
Transcript
Augmenting Behavior-Modeling Training: Testing the Effects of Pre- and Post-Training Interventions
Jon M. Werner, Anne M. O’Leary-Kelly, Timothy T. Baldwin, Kenneth N. Wexky
A number of recent training authors have suggested that pre- and post-train- ing interventions may enhance training outcomes. In the present study, pre- and post-training interventions were added to an established behavior-mod- eling program on assertiveness, creatingfour conditions: (1) no intervention, (2) pretruining interventions, (3) post-training interventions, and (4) both. One hundred fifty trainees completed the module, and measures of trainee reaction, learning retention, and behavioral change were obtained. Results indicated that the post-training intervention strongly affected learning reten- tion, as well as reactions immediately following training, with moderate ef- fects on behaviol: No sign$cant effects were observed between the pretrain- ing intervention and any ofthe trainee outcome measures. lmplications ofthe findingsfor training research and practice are discussed.
An enormous amount of money is spent annually on organizational training- and-development programs. Unfortunately, much of this has little (or at least an unknown) impact on measures of training effectiveness. One estimate is that as little as 10 percent of the money spent on training leads to changes in trainee behavior back on the job (Georgenson, 1982). More recently, New- strom has written that only 15 percent of the skills learned in training remain with trainees one year after training (cited in Garavaglia, 1993). An increasing body of literature is looking for ways to combat this pervasive “transfer prob- lem” (Baldwin and Ford, 1988). One approach to this problem is to augment existing training with interventions designed to motivate trainees to learn and transfer training skills.
H w RESOURCE DEVELOFMENT QUARTERLY, vol. 5 , no. 2. Summer 1994 8 Jossey-Bass Publishers 169
1 70 Wernel; O’Leary-Kelly, Baldwin, Wexley
Behavior-modeling training is one of the most popular approaches for training in interpersonal skills (Decker and Nathan, 1985). Meta-analyses have shown that behavior modeling typically has some of the strongest effects ob- served for any training technique currently in use (Burke and Day, 1986). Also, modeling has been found to have strong effects not only on outcomes such as learning and behavior but also on measures of organizational results (Werner and Crampton, 1992). However, despite the fact that behavior modeling ap- pears to be one of the best techniques available for skills training, empirical re- search on this technique has waned, with little work reported recently on ways to refine and improve this method (Baldwin, 1992).
In this study, we used an established training program (Baldwin, 1992; Smith, 1975) and investigated whether the addition of pre- and/or post-train- ing interventions would enhance training effectiveness over and above the training program alone. The idea here was to take an effective technique and make it even more effective. Leifer and Newstrom (1980) suggest that re- searchers should not only investigate the period of skill acquisition (that is, during training) but also the periods before and after training. The purpose of our research was to test this notion. These authors recommend that, prior to training, trainees must understand their need for training, as well as how this training can help them in the future (the instrumentality of training). Noe (1986) proposes that trainees’ expectancies regarding training will influence their motivation to learn, as well as their subsequent performance. It was pre- dicted that subjects receiving a motivational intervention prior to training would exhibit enhanced performance on various training outcome measures.
Once training is completed, the question arises as to whether some moti- vational post-training intervention would further enhance the effects of train- ing. A large body of literature now attests to the strong effects generally found for goal setting in a wide variety of situations (Locke and Latham, 1990). Locke and Latham (1990) describe how goal setting is thought to influence the choice of action, as well as the intensity and duration of that action. There is growing evidence that goal setting also serves to increase positive transfer. For example, supervisory-level trainees who used checklists to monitor their own behavior after training were better at applying training content than a control group, because the checklists allowed trainees to monitor their performance on assigned goals (Wexley and Nemeroff, 1975). Wexley and Baldwin (1986) found the greatest behavioral change when subjects were assigned goals or goals were set participatively, compared to subjects receiving no goal setting or a self-management approach patterned after Man (1982) being used. Learn- ing, however, was greatest for trainees in the assigned goal-setting condition.
Finally, Gist, Bavetta, and Stevens (1990) provided trainees with an in- tensive negotiation-skills training program, followed by either goal setting or self-management (see Frayne and Latham, 1987). Voluntary, self-set goals were used in both conditions. Subjects in the goal-setting condition were less able to generalize what they had learned to a novel situation than those in the self-
Augmenting Behavior-Modeling Training 171
management condition. However, the skills these trainees used were demon- strated more repeatedly than those of the selfimanagement trainees. In both conditions, the level of self-set goals was significantly related to the outcome variables.
Following Wexley and Baldwin (19861, in the present study we focused on the use of assigned goals, with behavioral checklists as our primary post- training intervention. This made sense, in that we presented the training as a “package” of ten related learning points, all useful for trainees to successfully demonstrate assertive behavior. Based on past research (Baldwin, 1992), both learning and demonstrating such shlls should pose a sufficiently difficult goal to master.
However, this situation is not directly comparable to those described re- cently by Kanfer and Ackerman (1989). They utilized an information-pro- cessing framework to study motivation and ability variables thought to be im- portant in the skill-acquisition process. These authors provided subjects with a complex air traffic control simulation. In successive experiments, subjects were provided with specific, difficult goals either earfier or later in a ten-trial experiment. Kanfer and Ackerman (1989) found that assigning goals in the early stages of learning had a significant negative influence on task perfor- mance, compared to the results when goals were assigned later. They argue that goal setting hinders the learning of declarative knowledge that is required for later procedural knowledge to take place (see Anderson, 1985). In the pres- ent study, trainees in the goal-setting conditions would have already received the full training program, including two attempts to practice the assertiveness behaviors, before goals were assigned. Thus, it was not expected that these goals would interfere with skill acquisition by preceding an understanding of what the task was about (Kanfer and Ackerman, 1989).
Practically, the use of assigned goals in training interventions does not ap- pear to be common, apparently because of a belief that trainees will not like assigned goals, and that this negative reaction will affect subsequent perfor- mance. This is an empirical question, which the present study can address. There is evidence that training outcome measures often do not “hang together” to the extent that is commonly assumed (Alliger and Janak, 1989).
Overall, then, we hypothesized that both pre- and post-training interven- tions would increase training effectiveness over and above training by itself. Specifically, we expected two main effects for each of our outcome measures.
Method
In the following paragraphs, we describe the subjects, design, and measures used in the study
Subjects and Design. In the early weeks of the university term, 150 stu- dents were recruited from an undergraduate management course at a large Mid- western university T b number represented approximately 45 percent of the
1 72 Werner, O’kmy-KeIly, Baldwin, Wexley
students in the class. Subjects volunteered to participate in our training pro- gram, whch taught assertiveness skills, in exchange for course credit. Upon ar- riving at the training session, subjects were told the general nature of the re- search and provided their informed consent to participate. The mean age of the subjects was 2 1.5 years, with seventy-three males and seventy-seven females.
All subjects received the same one-hour behavior-modeling training pro- gram conducted by one of the first two authors. The fifteen minutes before and after training were vaned by condition, with half the subjects receiving the pretraining intervention and half receiving the post-training intervention. We utilized a 2 x 2 factorial design, where pretraining intervention (provided or not provided) and post-training intervention (provided or not provided) were the two variables that were crossed. Condition 1 involved no pre- or post- training intervention; condition 2 involved a pretraining intervention, but no post-training intervention; condition 3 involved a post-training intervention, but no pre-training intervention; and condition 4 involved both a pre- and post-training intervention. In order to keep the total time of training constant across the conditions, those subjects who were not receiving a particular intervention were given a placebo, consisting of a lecture by one of the train- ers on other training techniques. Subjects in condition 1 received two such lectures (pre- and post-training). Subjects in condition 2 received a lecture after the assertiveness workshop, and subjects in condition 3 received a lec- ture before the workshop.
All training was conducted within a two-week period. Subjects were trained in groups of sixteen to twenty-two individuals. Subjects were ran- domly assigned to a condition, and each trainer trained approximately half the subjects in each condition. The order in which the conditions were presented was randomly varied for each trainer. The trainers followed a scripted trainer’s manual for each condition.
Assertiveness Training Intervention. The workshop, patterned after Smith (19751, utilized ten learning points such as “Is persistent in a request or an- swer by using calm repetition” and “Avoids apologizing for one’s feelings” (see Exhibit 1). Learning points were presented visually and then modeled in two videotaped situations. For each situation, both positive and negative role mod- els were presented (Baldwin, 1992). During the workshop, trainees had two opportunities to practice using the learning points in dyadic role plays. Feed- back was provided after each role play by the trainee’s role-playing partner, with general feedback provided by the trainer.
Pretruining Intervention. During the pretraining session, the trainers sought to influence the trainees’ perceived need for assertiveness training. Discussion was generated concerning specific situations where trainees could have acted more assertive1y The instrumentality or value of the training was also dis- cussed in terms of how such training could benefit the trainees’ future careers.
Post-Training Intervention. In the post-training session, trainees were as- signed behavioral goals based on the learning points presented in the training
Augmenting Behavio r-Modeling Training 173
Exhibit 1. Learning Points Used in This Research
1. Looks directly at the focal person with head erect. 2. Speaks clearly and to the point. 3. Speaks with no dramatic changes in voice tone. 4. Exhibits honesty about one’s feelings and needs, and accepts responsibil-
5. Is persistent in a request or answer by using calm repetition. 6. Avoids using softening statements and qualifiers. 7. Deals with manipulative or sidetracking statements by calmly acknowl-
edging the probability that there may be some truth in the statement, but that does not change one’s feelings.
8, Avoids sarcasm, angry denials, aggression, or critical attacks toward the focal person.
9. Avoids apologizing for one’s feelings and needs.
ity for them.
10. Seeks to close the conversation with a two-way understanding of the out- come or compromise.
workshop. The subjects received activities checklists, which they were to use twice a week for four weeks. The trainees were asked to consider the most chal- lengng situation they had faced since their last self-evaluation, and to check how many of the goals they had achieved. After four weeks, trainees submit- ted their completed checklists to the trainers.
Measures. Following Kirkpatrick (1976) and Baldwin and Ford (1988), we collected five measures of training effectiveness. Trainee reactions were measured immediately after the initial 1%-hour training block and again four weeks later. Learning retention and behavioral reproduction were measured four weeks after training. This period of delay in collecting outcome measures was primarily driven by the constraints of collecting data within a ten-week term. This is, however, the same length of time used in previous research (Wexley and Baldwin, 19861, and it provides a reasonable period of time to test whether skills have been acquired. Approximately six weeks after training, a measure of behavioral generalization (transfer) was obtained. These mea- sures are described below.
Reaction. A six-item hkert scale was developed, aslung subjects to state their degree of agreement with the following statements:
“I would recommend this workshop to others.” “This workshop got me more excited about becoming more assertive in the
“This workshop will help me act more assertively in the future.” “The workshop’s learning points clearly showed me how to act more
future.”
assertively
174 Wernel; O’Leary-Kelly, Baldwin, Wexley
“I plan to use what I’ve learned in the weeks ahead.” . “This workshop will help me in my future career.”
Values could range from 1 to 5 , with 5 as most favorable. This was the only outcome measured immediately after training. These items were also mea- sured four weeks later, after the subjects had completed the behavioral-repro- duction role play.
Learning Retention. Learning retention was assessed four weeks after train- ing by asking subjects to recall and write down as many of the ten learning points as they could remember. This is similar to learning measures used in previous research (see Mann and Decker, 19841, and can be viewed as a strin- gent test of learning. These written responses were later coded by two of the authors, who were blind to condition, by comparing the subjects’ answers to the written learning points. Learning points were coded as present or not pres- ent in the subjects’ written responses, with no partial credit assigned. Subject scores on this measure could range from 0 to 10. An initial sample of subject responses was coded by both raters, with 90 percent agreement attained. This was deemed acceptable, and further coding was done by one rater only
Behavioral Reproduction. Four weeks after training, but before collecting re- action and learning measures, the trainees met with one of the authors to role- play a situation that called for assertive behaviors. This situation was similar to the situations the subjects had. role-played during training. The subjects were to assume that the role player wanted to borrow class notes the night be- fore an exam. Role plays were audiotaped and later scored as to whether or not the subject exhibited nine of the learning points (the first, “Looks directly at the focal person,“ was scored immediately after the role play). Scores could range from 0 to 10. The subjects never completed the role play with the same individual who had served as their trainer. Role players were blind to the con- dition in which the subject had participated, and audiotapes were similarly scored by a researcher who was bIind to condition. The subjects arrived for these follow-up sessions in no particular order concerning which condition they had participated in. Thus, with no knowledge of condition by role play- ers or coders, it is most unlikely that any systematic bias occurred in favor of our hypotheses.
Behavioral Generalization. To tap the subjects’ ability to apply asSertlVeneS skills in a real-life situation, an unobtrusive measure was adapted from Bald- win (1992). About six weeks after training, approximately fifteen randomly se- lected trainees from each experimental condition were contacted by telephone by one of two male graduate students, who asked the subjects to purchase an expensive package of business publications. The telephone solicitors had been coached by the researchers to be persistent in their sales attempts. As antici- pated, all the subjects declined the offer, despite repeated attempts made by the caller (such a strategy was required to test learning point 5, “Is persistent in a request or answer by using calm repetition”). The subjects were rated con-
Augmenting Behavior-Modeling Training 175
cerning the extent to which they demonstrated the eight learning points that could be exhibited over the phone (two required face-to-face contact). At the end of the call, the callers identified themselves as being with the research pro- ject and thoroughly debriefed the subjects. Generalization measures were col- lected from a sample of subjects in each cell, because of the difficulty of reach- ing subjects on the phone.
Results
All subjects in the goal-setting conditions (conditions 3 and 4) arrived at their follow-up session with their checklists completed, creating the strong impres- sion that the checklists had in fact been used as prescribed in the preceding four weeks. As a control variable in this study, we asked subjects to report their grade-point averages (GPAs) (M = 2.89, SD = 0.39). GPA was significantly cor- related with our measure of learning (r = .28, p < .Ol) , but not with the other four dependent variables. None of the results reported below were changed when they were rerun including GPA as a covariate. Further, there were no sta- tistically significant differences in GPA across the four conditions (ranges were from 2.84 to 2.99). This provides evidence of the random distribution of sub- jects across conditions.
The intercomelation matrix for all variables is presented in Table I. As can be seen, reactions immediately after training were strongly related to reactions four weeks after training (r = S2). Moderate but statistically significant relation- ships were also observed between reactions four weeks after training and leam- ing retention (r = .29) and between learning retention and behavioral reproduc- tion (r = .22, both p e .01). Surprisingly, there was no statistically significant relationshp between behavioral reproduction and behavioral generalization.
The means and standard deviations, broken down by condition, are shown in Table 2. At both time periods, the subjects’ reactions to the training program were generally favorable. The overall measures of learning retention, behavioral reproduction, and behavioral generalization were all moderately high.
Note: N = 150 for both reaction measures, learning retention, and behavioral reproduction; n = 65 for behavioral generalization. Reliabilities (coefficient alpha) are on the diagonal.
* p < .01.
Tab
le 2
. M
eans
and
Sta
ndar
d D
evia
tions
for A
ll V
aria
bles
Cond
ition
1,
Cond
ition
2,
Cond
ition
3,
Cond
ition
4,
Wor
ksho
p Pr
ein te
nent
ion,
W
orks
hop,
Then
W
orks
hop,
plus
O
vera
ll O
nly
Then
Wor
ksho
p Po
stint
erve
ntio
n Pr
e- an
d Po
stint
erve
ntio
n
Variables
M SD
M SD
M SD
M SD
M SD
Rea
ctio
n, im
med
iate
' 3.
81
.75
3.93
.8
1 4.
04
.51
3.70
.8
4 3.
57
.75
Rea
ctio
n, d
elay
eda
3.90
.5
4 3.
84
.52
4.03
.5
3 3.
89
.55
3.85
.5
5 Le
arni
ng r
eten
tion'
5.
87
2.13
5.
12
2.03
5.
03
1.85
7.
05
1.96
6.
22
2.09
Be
havio
ral r
epro
duct
ion'
5.
64
2.20
5.
18
2.19
5.
79
2.08
5.
62
2.34
5.
90
2.20
Be
havio
ral g
ener
aliz
atio
nb
5.49
1.
47
5.28
1.
49
5.47
1.
19
5.27
1.
91
5.94
1.
25
~~
' N =
150
; con
ditio
n 1,
n =
34; c
ondi
tion
2, n
= 38
; con
ditio
n 3,
n =
37;
cond
ition
4, n
= 4
1.
N =
65;
con
ditio
n 1,
n =
18;
cond
ition
2. n
= 1
5; co
nditi
on 3
, n =
15;
cond
ition
4, n
= 1
7.
Augmenting Behavior-Modeling Training 177
Multivariate analysis of variance (MANOVA) was conducted to test for the overall effects of pre- and post-training intervention strateges on these five training outcome measures. MANOVA results were significant for the main ef- fects of the post-training intervention (Hotellings F[1, 601 = 7.31, p < .OOl>. There was no overall effect for the pretraining intervention. However, a signif- icant pre- by post-training interaction was observed (Hotellings F[1,60] = 2.48, p < .05). Univariate analyses of variance (ANOVAs) revealed that subjects in conditions 2 and 3 reacted more favorably to the training four weeks after train- ing than did subjects in conditions 1 and 4.
Since these measures are conceptually distinct, and Table 1 demonstrated that they are only modestly intercorrelated, further analyses were conducted by measure. Specifically, ANOVA was used for the two reaction measures, whereas multiple analysis of variance, followed by univariate ANOVA, was used for learning retention, behavioral reproduction, and behavioral general- ization. Individual item responses were entered separately for each of the lat- ter three training outcomes. This procedure was used to detect significant effects by learning point, while still seeking to minimize the type 1 error that could result from conducting twenty-eight separate analyses of variance. Fur- ther, this approach allowed all 150 subjects to be utilized in all analyses except those concerning behavioral generalization (where 65 subjects were available).
The reaction measures taken immediately after training displayed a sigmf- icant effect for the post-training goal-setting intervention (F(1, 144) = 10.33, p < . O l ) such that those who were assigned goals liked the training significantly less than those receiving no goal setting. Interestingly, no statistically significant differences were observed between the conditions four weeks after training. As can be seen in Table 2, the mean reactions in condition 1 (no pre- or post- training intervention) had fallen, while those in the goal-setting conditions (conditions 3 and 4) had risen over time. A repeated-measures MANOVA using reaction as the repeated measure revealed a sipficant effect on reactions over time for subjects in the assigned goal conditions (F(1, 145) = 6.80, p < .O l ) .
For the learning retention measure, strong effects were again found for the post-training intervention only. Subjects in the assigned goal-setting conditions could recall over 1.5 more leaming points than those in the no-goal-setting conditions (F(1, 146) = 4.81, p < ,001, partial eta’ = .26). Looking at the ef- fects by leaming point revealed that four of the learning points had statistically significant differences in favor of the goal-setting conditions (p c .05). These were “Looks directly at the focal person,” “Exhibits honesty about one’s feel- ings, and accepts responsibility for them,” “Avoids sarcasm or critical attacks toward the other person,” and “Seeks to close the conversation with a two-way understanding of the outcome.” Results for three other learning points ap- proached sigdicance (p < .lo>, all in favor of the goal-setting conditions.
For the behavioral-reproduction measure, MANOVA results were signifi- cant for the post-training intervention (F[l, 1441 = 2.64, p < .01, partial eta’ = .16), as well as for the pre- by post-training interaction (F[1,144] = 1.99,
178 Wernec O’Leary-Kelly, Baldwin, Wexley
p c .05, partial eta’ = .13). Even though subjects receiving the pretraining in- tervention were generally better at reproducing the learning points in the role play than subjects with no pretraining intervention, MANOVA results for this effect were not significant (F[1, 1441 = .48, ns.). Univariate ANOV’ at the item level revealed that the effects of the post-training interventions were pri- marily observed for three learning points. Subjects in the goal-setting condi- tions were more likely to look directly at the focal person and to avoid apolo- p i n g for their feelings (p c .O l ) . There was also an interaction for the first learning point, “Looks directly at the focal person,” such that subjects in con- dition 3 (post-training intervention only) did best of all (M = 0.92), and sub- jects in condition 1 (no pre- or post-training intervention) did poorest (M = 0.58, p c .05). The learning point concerning closing the conversation with a two-way understanding of the outcome approached significance (p = .07). Contrary to predictions, subjects in the goal-setting conditions were also less likely to exhibit honesty about their feelings or to accept responsibil- ity for them (p c .Ol).
The pre- and post-training interventions had no significant effects on the behavioral-generalization measure. The mean interitem correlation was ex- tremely low (r = .04). Viewing these items as distinct aspects of assertiveness and proceeding directly to analysis at the item level (see Mann and Decker, 1984) still produced only one statistically significant finding, that is, subjects in the goal-setting conditions were judged as better able to speak clearly and to the point (p c .05).
Discussion
One of the more interesting findings in this study concern the differential effects for trainee reactions and learning retention. Subjects who were told to monitor their assertiveness behaviors for four weeks reacted less favorably when they were questioned immediately after the training program. Goal set- ting appears to have been viewed as a bitter pill to swallow. However, four weeks later, subjects in the goal-setting conditions recalled more of the learn- ing points than subjects in the other conditions. Further, after using the checklists for four weeks, subjects in the goal-setting conditions viewed the training more favorably while subjects who received no concrete means of maintaining their newly learned behaviors became less favorably inclined to- ward the training.
While we did not measure self-efficacy one explanation for these findings is that assigning goals initially threatened the subjects’ belief in their ability to carry out the assertive behaviors taught in training (Gist, Stevens, and Bavetta, 1991). However, use of the activities checklists may have enhanced the self- efficacy of subjects in the goal-setting conditions; therefore, they reacted more favorably four weeks later and, in fact, could recall considerably more of the learning points.
Augmenting Behavior-Modeling Training 179
An alternative (and simpler) explanation for these findings was proposed by a reviewer of this article, namely that participants who received the check- lists realized that their “training” wasn’t really finished-until they had com- pleted eight checklists. Thus, their initial reactions were more negative than subjects in the other conditions. Given this, the change in their reactions over time may be explained using cognitive dissonance theory Although they did not initially like the idea of completing the checklists, they became more posi- tive over time because they were required to complete them, that is, their re- actions changed in order to deal with the dissonance of doing something they disliked over a four-week period. Since both explanations are speculative, we see this as an area worthy of future research.
For behavioral reproduction, there is evidence that the post-training in- tervention worked as predicted. From a practical standpoint, it is valuable to find that behavioral monitoring following training had a significant impact on both learning retention and behavioral reproduction. Unfortunately, these ef- fects were not observed for the behavioral generalization (transfer) measure.
Clearly, these findings would be more compelling had we found stronger evidence for both behavioral measures. It is possible that the training work- shop by itself was strong enough to produce moderate behavioral changes in all conditions, regardless of pre- or post-intervention strategies. Conversely, it is also possible that our choice of training content, assertiveness skills, was suf- ficiently difficult to master so that even those who learned more (the subjects in conditions 3 and 4) were not able to demonstrate all that they had learned in our behavioral role play. In either case, our results bolster Kirkpatricks con- tention (1976) that demonstrating competence on a written test of learning is not synonymous with demonstrating behavioral change after training.
Our measure of behavioral generalization was patterned after Baldwin (1992). In that study, a confederate made a similar sales pitch to trainees im- mediately following their completion of the assertiveness-training workshop. Unlike the current study, Baldwin’s study (1992) found meaningful differences in generalization across the four conditions. One problem for our study was that the practical difficulties of contacting subjects on the phone caused a large loss in the statistical power to detect differences across conditions if they existed. Further, the change in context (from face-to-face to a telephone encounter) also seemed to have an effect on our measure. Overall, 56 percent of the learning points were demonstrated in our behavioral-reproduction role play, whereas 70 percent of the learning points were demonstrated in the generalization scenario. This difference was statistically significant (t = 5.49, p < .OOl). Com- paring this 70 percent “success rate” (that is, success in demonstrating the learning points) with the percentage of learning points demonstrated in Baldwin’s generalization measure also produced significant differences (t = 5.76, p < .OOl). It would appear that, in general, it is easier to say no over the phone than in person. Thus, despite finding no differences between the conditions on behavioral generalization, we urge future researchers to collect such data. It is
180 Wernet; O’Leary-Kelly, Baldwin, Waley
vitally important to demonstrate the positive transfer of training content to real- life situations. Despite the widespread acceptance of this idea, the problems of transfer remain relatively underresearched.
The lack of significant effects for our pretraining motivational intervention was disappointing. The most likely explanation for this was that our pretrain- ing intervention was not strong enough to produce noticeable changes. A ma- nipulation check revealed that subjects receiving the pretraining intervention reported that their trainer generated more interest and enthusiasm about as- sertiveness training than did subjects who received no pretraining intervention (p < .05). However, there were no significant dfferences in their perceptions concerning whether the trainer increased their understanding of the instru- mentality of training-in other words, how this training would help them in the future. It is hoped that future research will improve such pretraining inter- ventions and tie them to a needs assessment, in order to determine specifically who needs training, and of what type (Wexley and htharn, 1991).
One suggestion for strengthening pretraining motivation is to extend the pretraining intervention overtime, just as the post-training intervention was ex- tended over time. That is, prior to training, participants could be asked to com- plete checklists of times when they could have been more assertive. These sit- uations would clearly show panicipants how important assertiveness might be to them in their everyday lives, thereby strengthening the valance of training. It may in fact turn out that pretraining interventions are relatively less effective than post-training interventions at influencing training transfer, However, it is premature to make such a claim based on the results of this study alone.
Some limitations of this research should be noted. First, as described above, our pretraining intervention was very brief, and likely not very engagmg. Sec- ond, our lack of results for our behavioral generalization measure may be largely the result of the fact that our sample for this measure was so much smaller than the sample available for our other measures. Third, we analyzed our data using MANOVA and ANOVA procedures, despite the fact that the learning points were scored binomially Thus, the results at the learning point level should be treated with some caution. Fourth, while we obtained sizable effects on our measure of learning retention, we did not collect pretest measures, or measure learning immediately after training. Nevertheless, our efforts at randomization, plus the lack of differences in participant grade-point average by condition, lead us to argue that the observed differences in learning retention were the result of the training interventions we carried out. Ultimately organizations should be far more interested in the retention of learning over time than in how much par- ticipants know at the end of a training session. Finally, our results were obtained using one particular training method and content. It remains to be seen whether pre- and post-training interventions have similar effects in other training set- tings, where different methodology or content is utilized.
Two further points should be made concerning the relationship of the cur- rent study to recent research. First, when discussing training criteria, it is easy
Augmenting Behavior-Modeling Training I81
to think of the outcomes as levels in a hierarchy, with reaction measures as the “worst” and results as the “best” (in terms of either informational or dollar value to the organization). Recently, Alliger and Janak (1989) challenged the notion that Krkpatrick’s four criteria exhibit a direct causal relationship, or that they are necessarily all positively intercorrelated. These authors searched the literature and located twelve studies reporting twenty-six correlations be- tween Kirkpatricks four criteria. Some research has been published since Alliger and Janak (1989) appeared (see Baldwin, 1992; Mathieu, Tannenbaum, and Salas, 1992; Tziner, Haccoun, and Kadish, 1991). The current study re- ports correlations between reaction, learning retention, and behavior. It sup- ports Alliger and Janaks argument (1989) that there are times when reaction and learning measures will not be highly correlated. In this study, trainees ini- tially liked training plus goal setting less than training alone, yet they learned considerably more with the addition of assigned goals. Though we had as- sumed no causal ordering between the criteria, we had implicitly assumed a “positive manifold” between these measures (Alliger and Janak, 1989). We hope this study serves to caution researchers that training-effectiveness mea- sures do not align themselves in any simple hierarchy of worth or causal order.
Second, recent research has sought to expand our understanding of the influences on training-effectiveness measures (Tziner, Haccoun, and Kadish, 1991). For example, Mathieu, Tannenbaum, and Salas (1992) tested the ef- fects of several individual and situational variables on measures of learning and behavior. Their results did not strongly support their hypothesized model. However, they did find evidence that trainee reactions served as a moderator between trainee motivation and learning. They observed the best results on their measures of learning and behavior when trainee motivation and reactions were both high. Mathieu, Tannenbaum, and Salas (1992) urged training re- searchers to consider more sophisticated models concerning the relationships between effectiveness measures. They also utilized an expectancy-theory framework to measure trainee motivation (Noe, 1986). This is much preferred to the common practice of using trainee-reaction measures alone to tap trainee motivation.
In summary, adding a relatively brief goal-setting intervention to an exist- ing behavior-modeling training program led to a sizable increase in learning retention by trainees, as well as moderate behavioral change. Trainee reactions, either immediately following training or four weeks later, were not a good proxy for the learning of training content. Kirkpatrick (1978) presented data indicating that the majority of training programs were evaluated using reac- tion measures. Approximately half used learning measures, and fewer than 20 percent were evaluated using either behavioral or results measures. A more re- cent survey showed little change in this general pattern (Saari, Johnson, and McLaughlin, 1988). We agree with Alliger and Janak (1989) that there may be times when it is inappropriate to measure behaviors or results, but certainly such occasions are not so common as to justify the weak efforts typically made
182 Wernet; O'Leary-Kelly, Baldwin, Wexley
at training evaluation. If there is no necessary hierarchy or positive manifold among firkpatricks criteria, then in most situations it is unacceptable to mea- sure only reactions and learning. Measuring behaviors and results is both dif- ficult and frustrating. However, in this era of global competition and corporate restructuring, it is imperative for training professionals to demonstrate the effectiveness of their efforts. This must include measures that tap the transfer of behaviors from training to real-life settings (Baldwin and Ford, 1988; Garavaglia, 1993).
References Alliger, G. M., 6r Janak, E. A. (1989). Kirkpatricks levels of training criteria: Thirty years later.
Anderson, J. R. (1985). Cognitive psychology and its implications (2nd ed.). New York Freeman. Baldwin, T. T. (1992). Effects of alternative modeling strategies on outcomes of interpersonal-
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future re-
Burke, M. J., & Day, R. R (1986). A cumulative study of the effectiveness of managerial training.
Decker, P J.. & Nathan, B. R. (1985). Behavior modeling training. New York Praeger. Frayne, C., & Ltham, G. I? (1987). Application of social learning theory to employee self-man-
agement of attendance. Journal of Applied Psychology, 72,387-392. Garavaglia, P L. (1993). How to ensure transfer of training. Training and DevelopmentJournal, 47
(10),63-68. Georgenson, D. L. (1982). The problem of transfer calls for partnership. Training and Develop-
mentjouml, 36 (10). 75-78. Gist, M. E., Bavetta. A. G., & Stevens, C. K. (1990). Transfer training method: Its influence on
skill generalization, slall repetition, and performance level. Personnel Psychology, 43,501-523. Gist, M. E., Stevens, C. K.. & Bavetta, A. G. (1991). Effects of self-efficacy and post-training in-
tervention on the acquisition and maintenance of complex interpersonal skills. Personnel Psy- chology, 44,837-861.
Kanfer, R., & Ackerman, F! L. (1989). Motivation and cognitive abilities: An integrativdaptitude- treatment interaction approach to slall acquisition [Monograph]. Journal ofApplied Psychology, 74,657-690.
Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook (pp. 18.1-18.27). New York McGraw-Hill.
Kirkpatrick, D. L. (1978). Evaluating in-house training programs. Training and DevelopmentJour- MI, 33 (61, 6-9.
Leifer, M. S., & Newstrom, J. W. (1980). Solving the transfer of training problem. Training and DevelopmentJouml, 34 (2). 42-46.
Locke, E., & Latham, G. €! (1990). A theory ofgoal setting and task performance. Englewood Cliffs, NJ: Prentice Hall.
Mann, R. B., &Decker, F! J. (1984): The effect of key behavior distinctiveness on generalization and recall in behavior modeling training. Academy of ManagementJouml, 27, 900-910.
Marx, R. D. (1982). Relapse prevention for managerial training: A model for maintenance of be- havior change. Academy of Management Rrview, 5,433-441.
Mathieu, J. E., Tannenbaum, S. 1.. & Salas, E. (1992). Influences of individual and situational characteristics on measures of training effectiveness. Academy of Management Journd, 35, 828-847.
Noe, R. A. (1986). Trainees’ attributes and attitudes: Neglected influences on training effective-
Saari, L. M., Johnson, T. R., & McLaughlin, 5. D. (1988). A survey of management training and
Smith, M. J. (1975). When I say no, Ifeel guilty. New York: Bantam Books. Tziner, A., Haccoun, R. R., & Kadish, A. (1991). Personal and situational characteristics influ-
encing the effectiveness of transfer of training improvement strategies. Journal of Occupational Psychology, 64, 167-177.
Werner, J. M., & Crampton, 5. C. (1992). The impact of behavior modeling training on leam- ing, behavior, and results criteria: A meta-analytic review. In M. Schnake (Ed.), Proceedings from the annual meeting of the Southern Management Association, New Orleans
Wexley, K. N., & Baldwin, T. T. (1986). Posttraining strategies for facilitating positive transfer:
Wexley, K. N., & Latham, G. I? (1991). Developing and training human resources in organizations
Wexley, K. N., & Nemeroff, W (1975). Effectiveness of positive reinforcement and goal setting
ness. Academy of Management Review, 11,736-749.
education practices in U.S. companies. Personnel Psychology, 41, 731-743.
(pp. 279-284).
An empirical exploration. Academy ofManagement]ournaf, 29, 503-520.
(2nd ed.). New York: HarperCollins.
as methods of management development. ]ournal of Applied Psychology, 60,446450.
Jon M. Werner is assistant professor, Department of Management, University of South Carolina, Columbia.
Anne M. O’Leary-Kelly is assistant professor, Department of Management, Texas A&M University, College Station.
Timothy T. Baldwin is associate professor, Department of Management, lndiana University, Bloomington.
I Kenneth N. Wexley is managingpartnel; Human Resource Decisions, lnc., Okemos, Michigan.
