Post on 07-Sep-2018
transcript
Evaluating Training Impact Level 1 -6 | Paul Leone, Ph.D. | 2013
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 1
he only way training is ever going to be perceived as a profitable business
imperative, instead of a costly distraction, is if training professionals and
their organizations measure their results. While most training
organizations and vendors recognize the need and urgency to report the impact of
their contributions, the overwhelming majority just don’t end up doing it (or just
don’t do it right). Further, when it comes to measuring the powerful effects of
leadership training, the numbers get even more dismal and the impact studies just
about disappear. This study not only measures the results of a leadership program
from Level 1 (reaction) to Level 5 (Return on Investment), but also introduces a
Level 6 (Transfer Climate) which identifies
the environmental factors that help or
hinder a learner from applying their new
knowledge and skills back on the job.
In this study, the training measured was a
popular leadership program that was
deployed organization-wide at a major
credit card company. Over a period of 12
months, it was launched globally to over 10,000 managers and director-level people
leaders. The sample size for this particular study was 1974 participants across many
diverse regions, lines of business, and staff groups.
The Evaluation Strategy
The evaluation and overall impact of the leadership development efforts were
measured in part using the traditional Kirkpatrick/Phillips 5-level approach and then
T
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 2
a sixth level was added to identify the critical factors that can influence the impact of
the training back on the job (see table below).
Measurement Method of Evaluation
Level 1: Reaction Learner responds to survey upon completion of learning event (satisfaction with experience)
Level 2: Knowledge Learner responds to survey upon completion of learning event (new knowledge and skills acquired)
Level 3: Behavior Assessment completed by self and direct reports 3 months after learning event
(observed improvement in leadership skills)
Level 4: Impact Assessment completed by self, and direct reports 3 months after learning event (improved
productivity of direct reports)
Level 5: ROI Cost vs. Benefit analysis based on increase in sales productivity of direct reports over 3 months
Level 6: Transfer
Climate
Assesses factors in participant’s work environment (climate) that will help or hinder the transfer of
learning
Gathering the Data
Participants attending the one-day instructor-led program were given a 5-minute
survey immediately after training. This survey captured satisfaction with the
experience (Level 1) and the level of new knowledge and skills acquired (Level 2).
Three months after the training was completed, participants were then e-mailed a
link to take another survey online and were instructed to invite their direct reports
to complete a similar survey. For participants, this 15-minute survey captured
behavior change (Level 3) and performance
improvements. For direct reports, the survey
included questions about their leaders’ behavior
change and also about how those changes
affected their own performance and
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 3
productivity metrics (Level 4). Using these performance increases, a return on
investment (ROI) analysis (Level 5) was then conducted for all respondents
reporting “sales revenue” as their primary performance measure. Also, included in
participants’ self-assessment was a transfer climate index which asked participants
about the work environment they returned to post-training (Level 6).
Results
Level 1 and Level 2
As far as how well the program was received (Level 1) and the amount of learning
that took place (Level 2), participants took no more than 5 minutes to answer a few
key questions about their experience. The results of the quantitative questions are
shown below. One important thing to remember is that most organizations never
get past these Level 1 or 2 measures and often use these rather limited metrics to
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 4
extrapolate the ultimate value and potential impact of the program. As you will see
when we get into higher levels of impact, these very first levels become far less
relevant and in no way predict the ultimate outcome and ROI of a training program.
They are useful however to get some preliminary data on how a program is being
received and some important feedback on how to improve the delivery of content.
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 5
Level 3
When it came to behavior change (Level 3), we looked at the percentage of
participants who were demonstrating observed levels of improvement. Using a 6-
point improvement scale, we asked both participants and their direct reports to
rate the level of improvement they observed in 6 critical, training-related
behaviors (this number can vary and is primarily defined by the business). One
important thing to look for in these results is similarity in the response
distributions. That is, did direct reports corroborate the self-ratings? Also note
that “high improvement” was defined as the self and direct reports rating a
participant’s improvement either “significant” or “exceptional” on our scale. As
you’ll see shortly, we use this group to calculate higher levels of impact. Below
are our overall response distributions:
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 6
Level 4 Of course the next question is “so what…?” What do high improvements in
certain key leader behaviors actually mean to the business? More specifically,
how does improving your leadership behaviors actually affect your direct reports’
levels of productivity? This brings us to our Level 4 measure - Business Impact.
Here we separated our entire sample of direct reports into two distinct groups:
Group A: direct reports of “high improvement” leaders vs. Group B: direct
reports of “no improvement” leaders. When we looked at the extent to which
each of these two groups of direct reports improved their primary measures of
performance, some significant and quite astounding differences emerged. The
direct reports of “high improvement” leaders were increasing productivity by an
average of 48%, while the direct reports of “no improvement” leaders showed
increases of only 11%. See graph below:
While this makes an incredible case for how the behavior of a leader can impact
productivity, the next step was to isolate just how much of this difference in
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 7
productivity can be directly attributed to the training. After all, there may be a
multitude of factors that can either make an employee perform better or make a
leader improve his/her behavior over a three-month period.
To isolate the unique influence of the leadership program, we factored in how
much each direct report was influenced by their leaders’ specific changes
(expressed as a percentage)
and then how much each
leader’s changes were
influenced by the program
(expressed as another
percentage).
That is, on our direct report
survey, we asked how much
of their productivity
increases over the past 3
months could be attributable
to improved leadership (e.g. more direction and support from their leaders) and
on our participant survey we asked how much of the improved leadership could
be attributable to the training experience. We then multiply these two
percentages for our Level 4 calculation and that gives us “total productivity
improvement directly attributable to the training”. See next diagram for Level 4
calculation:
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 8
From these results we saw that direct reports increased their productivity by an
overall average of 7% as a direct outcome of their leaders attending the
leadership training.
Level 5 – ROI
So what does a 7% increase in productivity mean in dollar
value? To calculate the ROI, we looked at all the employees
reporting Sales revenue as their primary productivity metric.
According to the business, a sales specialist contributed
approximately $50,000 in revenue over a three month period. If an employee’s
contribution was to increase by 7% due to better direction and support from
his/her leader, the added benefit to the company would be approximately $3500.
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 9
Now consider that each leader in the study impacted not one employee, but had
an average of 6 employees that were being directly influenced every day. That
brings the total benefit for the company to $21,000 (3500 X 6). If the benefit is
$21,000 in additional revenue, and the cost of the training was approximately
$2,000, the return on investment is 950%. See formula and calculation below:
Level 6 - Transfer Climate
Which factors in the participant’s immediate work environment were predictive
of how much training would be applied and sustained on the job? To measure
this we used the results of our “transfer climate index” attached to the 3-month
post-program survey. Here, we essentially correlated the scores on the transfer
climate questions to the scores on the questions about behavior change and
performance improvements. When we looked at the results of this analysis, we
found 3 environmental or climate factors that were consistently predicting
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 10
whether a learner would have “no improvement” or “high improvement” back
on the job. These factors were:
Their manager follows up with them and has one-on-one meetings after the event to discuss
what was learned and how to apply
Their manager endorses the training and supports their development
Their manager specifically recognizes and rewards the improved leadership behaviors
The illustration below depicts just how much these factors could discriminate
between levels of impact. For example, if you look at the first set of data on the
left, 74% of learners in the “high improvement” group were having one-on-one
conversations about how to apply their training, while only 13% of the “no
improvement’ group were having these same conversations. In fact, all 3 factors
were present and pervasive for the “high improvement” group, while they were
all significantly lower for the “no improvement” group.
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 11
Conclusion
When it comes to ROI, the training that can (and should) pack the biggest punch is
your leadership training. When leaders increase their effort and improve their
leadership behaviors on the job, they not only become more productive, but they
have a tremendous, positive ripple effect on all the employees they lead and
influence every day. If captured and measured correctly, the “benefit” of these
leader improvements can result in some stunning “returns”. I call this the
“Leadership ROI”.
For this particular study, it was shown that a significant
number of leaders improved their leadership behaviors
and were doing a few crucial things different after the
training. These changes by the leaders clearly made their
immediate direct reports more productive on the job. In fact, these direct reports,
on average, showed a 7% increase in productivity in the 3 months after their leader
attended the training. Each leader also had an average of 6 immediate direct reports
who were being influenced every day by his or her leadership. That meant for every
ONE leader that attended the training and did a few things different afterwards,
there were SIX employees that were enhancing their performance and improving
their primary business metrics. This led to a final ROI for each participant of 950%.
This is the power of the Leadership ROI.
What our evaluation also told us was that participants were clearly taking their
learning back to very different work environments, and these often disparate
“climates” can have a profound effect on how they apply the training. That is, those
who had tactical support from their immediate leader and were deliberately
HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING
Page | 12
rewarded for their training-related changes were demonstrating far greater
improvement and impact than those who had less supportive environments.
Further, the results of this Level 6 analysis told us which specific climate factors
were having the biggest quantifiable influence on our training outcomes and ROI.
Imagine the power of knowing what your top factors are and very precisely targeting
them for future training roll-outs and initiatives. Consider how much you could
improve your training’s impact and ROI. This is the power of evaluation. This is
the power of measuring to Level 6.
From these results, it should be clear that measuring and reporting the impact of
your leadership training is not only achievable, but also quite simple if you create
the right strategy and design. In total, the
time I asked from participant’s for this
entire evaluation was no more than 20-25
minutes. While I spent some more time
behind the scenes getting to understand the
behaviors and business metrics we needed
to include, it all still added up to a small
drop in the bucket compared to the time and resources an organization spends
launching any one given training program. By investing just a little in your
measurement strategy, and asking a few simple questions, to the right employees, at
the right times, you can end up a few months later telling an incredible story of
bottom-line impact. How’s that for an ROI?