1www.exploreHR.org

Measuring ROI of Training

2www.exploreHR.org

You can download these excellent slides at :

www.HR-Management-Slides.com

3www.exploreHR.org

Agenda

1. Measuring the Effectiveness of Training Program

2. Measuring ‘Return on Investment’ of Training

3. Enhancing the Effectiveness and ROI of Training

4www.exploreHR.org

Measuring the Effectiveness of Training Program

5www.exploreHR.org

Training Need Analysis

Training Objectives

Training Delivery

Training Evaluation

Training Process

What are the training needs for this person and/or job?

Objective should be measurable and observable

Techniques include on-the-job-training, action learning, etc.

Measure reaction, learning, behavior, and results

6www.exploreHR.org

The Four Levels of Evaluation

Level 1 - Reaction

Level 2 - Learning

Level 3 – Behavior Application

Level 4 – Business Impact

Four Levels of Training

Effectiveness

7www.exploreHR.org

Test the trainees to determine if they learned the principles,

skills, and facts they were to learn.

Evaluate trainees’ reactions to the program. Did they like the

program? Did they think it worthwhile?

Level 1 - Reaction

Level 2 - Learning

The Four Levels of Evaluation

8www.exploreHR.org

What final results were achieved in terms of the training objectives previously set? Did the number of customer complaints about employee drop? Did

the reject rate improve? Was turnover reduced, and so forth.

Ask whether the trainees’ behavior on the job changed because of the training program. For

example, are employees in the store’s complaint department more courteous toward disgruntled

customers than previously?

Level 3 – Behavior

Application

Level 4 – Business

Impact

The Four Levels of Evaluation

9www.exploreHR.org

Value of Information

Frequency of Use

Difficulty of Assessment

Level

I. Reaction

II. Learning

III. Behavior

IV. Results

The Four Levels of Evaluation

Least valuable

Most valuable

Frequent

Infrequent

Easy

Difficult

10www.exploreHR.org

Level 1 - Reaction

Evaluate trainees’ reactions to the program: Did they like the program? Did they like the facilitators?Did they like the training accommodation and facilities?

Level 1 - Reaction

11www.exploreHR.org

Guidelines for Evaluating Reaction

1. Determine what you want to find out

2. Design a form that will quantify reactions

3. Encourage written comments and suggestions

4. Get 100 percent immediate response

5. Get honest response

6. Develop acceptable standards

7. Measure reactions against standards, and take appropriate action

8. Communicate reaction as appropriate

12www.exploreHR.org

Please give us your frank reactions and comments. They will help us to evaluate this program and improve future programs.

Program :

Facilitator :

1. How do you rate the subject (interest, benefit, etc)

a. Excellent

b. Very Good

c. Good

d. Fair

e. Poor

2. How do you rate the facilitator? (knowledge, ability to deliver and communicate?)

a. Excellent

b. Very Good

c. Good

d. Fair

e. Poor

3. How do you rate the facilities? (comfort, convinience, etc)

a. Excellent

b. Very Good

c. Good

d. Fair

e. Poor

4. What would have improved the program?

………………………………………………………………………………… ………………………………………………………………………………… …………………………………………………………………………………

Sample of Reaction Form

13www.exploreHR.org

Level 2 - Learning

Measuring learning means determining one or more of the following :• What knowledge was

learned?• What skills were

developed or improved?• What attitudes were

changed?

Level 2 - Learning

14www.exploreHR.org

Guidelines for Evaluating Learning

1. Use a control group if practical

2. Evaluate knowledge, skills and/or attitudes both before and after the program

3. Use a paper-and-pencil test to measure knowledge

4. Use a performance test to measure skills

5. Get 100 percent immediate response

6. Use the results of evaluation to take appropriate action

15www.exploreHR.org

Guidelines for Evaluating Learning

Experimental Group Control GroupPre Test Score 45 46

Post Test Score 55 48

Gain 10 2

Pretest and Posttest Scores on Change Management TrainingExample :

16www.exploreHR.org

Level 3 – Behavior Application

• The frequency of application of new skills/knowledge/ attitudes (on the job)

• The effectiveness of the skills/knowledge/ attitudes (as applied on the job)

Level 3 – Behavior Application

17www.exploreHR.org

Guidelines for Evaluating Learning

1. Use a control group if practical

2. Allow time for behavior change and application to take place

3. Evaluate both before and after the program if practical

4. Survey and/or interview one or more of the following : trainees, their immediate supervisor, their subordinates, and others who often observe their behavior

5. Get 100 percent response or a sampling

6. Repeat the evaluation at appropriate times

7. Consider cost versus benefits

18www.exploreHR.org

Example of Survey to Measure Behavior Application

Understanding and Motivating1. Trying to understand my subordinates 5 4 3 2 1

2. Listening to my subordinates 5 4 3 2 1

3. Praising good work 5 4 3 2 1

4. Talking with subordinates about

their family and personal interests

5. Asking my subordinates for their ideas 5 4 3 2 1

6. Applying "Management by Walking Around" 5 4 3 2 1

Time and energy spent after the prgram compared to time and energy spent before

the program

5 4 3 2 1

Instruction: The objective of this questionnaire is to determine the extent to which those who attended the recent program on Leadership have applied the principles and techniques that they learned there to the job.

Circle the answer that you consider appropriate for each question.5 = Much more 4 = More 3 = Same 2 = Less 1 = Much less

19www.exploreHR.org

Level 4 – Business Results

• Indicate the extent to which you think this program has influenced each of these measures in your work unit, department, or business unit:• Productivity• Quality• Customer Response Time• Cost Control• Employee Satisfaction• Customer Satisfaction• Other

Level 4 – Business Results

20www.exploreHR.org

Guidelines for Evaluating Learning

1. Use a control group if practical

2. Allow time for results to be achieved

3. Measure both before and after the program if practical

4. Repeat the evaluation at appropriate times

5. Consider cost versus benefits

6. Be satisfied with evidence if proof is not possible

21www.exploreHR.org

Performance Indicators

HARD DATA INDICATORS

• Downtime duration• Number of defect products• Sales volume• Production unit• Customer satisfaction index• Response time to orders• Number of accidents at work• Others

22www.exploreHR.org

Performance Indicators

SOFT DATA INDICATORS (intangible impacts) • Job satisfaction• Conducive working relationship• Effective communication• Stress rate• Quality in decision-making

23www.exploreHR.org

Example : Measuring Training Results

0.00

50.00

100.00

150.00

200.00Program : TQM Training

Results after 3 months of training, number of defects dropped to 80 units/day

120 units

80 units

Before training After training

24www.exploreHR.org

0.00

10.00

20.00

30.00

40.00

50.00

Example : Measuring Training Results

Program :Sales Training

Results after 3 months training, number of sales per salesman increase to 30 units/month.

20 units

30 units

Before training After training

25www.exploreHR.org

Measuring Return on Investment of

Training

26www.exploreHR.org

Level 5 : Return on Investment of Training

Level 1 - Reaction

Level 2 - Learning

Level 3 – Behavior Application

Level 4 – Business Impact

Level 5 – Return on Investment of

Training

27www.exploreHR.org

Criteria for Selecting Programs for Levels 4 and 5 Evaluation

• Importance of the program in meeting the organization’s goals

• Cost of the program

• Visibility of the program

• Size of the target audience

• Extent of management interest

28www.exploreHR.org

Benefits of ROI of Training

• Measure contribution

• Set priorities

• Focus on results

• Alter management perceptions of training

29www.exploreHR.org

ROI of Training Model

Collect Data

Isolate the Effects of Training

Convert Data to Monetary

Values

Calculate ROI of Training

Tabulate Program

Costs

Identify Intangible Benefits

30www.exploreHR.org

Net Program Benefits

Program CostsX 100

Return on Investment Formula

ROI =

Example :

• Costs per program (25 participants) $ 88,500

• Benefits per program (1st year) $230,625

$ 230,625 – 88,500

$ 88,500ROI = X 100

ROI = 161 %

31www.exploreHR.org

Collecting Data

• Identify appropriate performance indicators

• Develop a collection plan

Collect Data

32www.exploreHR.org

Example of Performance Indicators

• Units produced• Items sold• Work backlog• New accounts opened• Productivity• Inventory turnover• Etc.

Output

• Equipment downtime• Overtime• Time to project

completion• Processing time• Repair time• Lost time days• Etc

Time

33www.exploreHR.org

Example of Hard Indicators

• Unit costs• Variable costs• Overhead costs• Operating costs• Number of cost

reduction• Etc.

Cost

• Scrap• Waste• Rejects• Error rates• Rework• Product defects• Product failure

Quality

34www.exploreHR.org

Example of Performance Results

• Some performance results after training program:• Scrap was reduced from 11 % to 7.4 %

• Absenteeism was reduce from 7 % to 3.25 %

• The annual turnover rate was reduced from 30 % to 16 %

• Lost time accidents were reduced 95 %

35www.exploreHR.org

Isolating the Effects of Training

Methods to Isolate the Effects of Training

Using Control Group

Trend Lines

Participants Estimate

Supervisors of Participants

Estimate

36www.exploreHR.org

Using Control Group

• A control group arrangement can be used to isolate training impact.

• With this strategy, one group receives training, while another, similar group does not receive training.

• The difference in the performance of the two groups is attributed to the training program.

Isolating the Effects of Training

37www.exploreHR.org

Trend Lines

• Trend lines are used to project the values of specific output variables if training had not been undertaken.

• The projection is compared to the actual data after training, and the difference represents the estimate impact of training.

Isolating the Effects of Training

38www.exploreHR.org

Trend Lines Analysis

Jan Feb Mar Apr Jul Jun May Aug

At the beginning of May, a Sales training Program session was

held

Volu

me

of S

ales

The difference

represents the estimate

impact of training.

Trend Projection

Actual sales performance

39www.exploreHR.org

Participants and Supervisors of

Participants Estimate of

Training’s Impact

• This method rests on the assumption that participants (and their supervisors) are capable of estimating how much a performance improvement is related to the training program.

• Because their actions have produced the improvement, participants (and their supervisors) may have very accurate input on the issue.

• They should know how much of the change was caused by applying what they have learned in the program.

Isolating the Effects of Training

40www.exploreHR.org

Participants and Supervisors of

Participants Estimate of

Training’s Impact

• Typical Questions to Estimate :• What percent this improvement

can be attributed to the application of skills/techniques/knowledge gained in the training program?

• What confidence do you have in this estimate, expresses as a percent?

• What other factors contributed to this improvement in performance?

Isolating the Effects of Training

41www.exploreHR.org

1 Training Program 50% 70%2 Change in Procedures 10% 80%3 Adjustment in Standards 10% 50%4 Revision to Incentive Plan 20% 90%5 Increased Management Attention 10% 50%6 Other - -

Percent Improvement

Caused by

Confidence Expressed as

a PercentFactors Which Influenced

ImprovementNo.

Isolating the Effects of Training

Example of a Participant’s Estimation

The confidence percentage is multiplied by the estimate (50 % x 70 %) to produce a usable training factor value of 35 %

42www.exploreHR.org

1 Training Program 50% 70%2 Change in Procedures 10% 80%3 Adjustment in Standards 10% 50%4 Revision to Incentive Plan 20% 90%5 Increased Management Attention 10% 50%6 Other - -

Percent Improvement

Caused by

Confidence Expressed as

a PercentFactors Which Influenced

ImprovementNo.

Isolating the Effects of Training

Example of a Participant’s Estimation

• The confidence percentage is multiplied by the estimate (50 % x 70 %) to produce a usable training factor value of 35 %

• This adjusted percentage is then multiplied by the actual amount of improvement (post-program minus pre-program value) to isolate the portion attributed to training

• The adjusted improvement is now ready for conversion to monetary values, and used in the return on investment

43www.exploreHR.org

Converting Data to Monetary Values

Converting Data to

Monetary Values

• Steps to Convert Data to Monetary Values

1. Focus on a unit of improvement

2. Determine a value of each unit

3. Calculate the change in performance data

4. Determine an annual amount of change

5. Calculate the annual value of improvement

44www.exploreHR.org

Steps to Convert Data to Monetary Values

An example to illustrate the steps to convert data to monetary values

1. Focus on unit improvement

• One grievance reaching step two in the four-step grievance resolution process

Steps Illustration

2. Determine a value of each

unit

• Using internal experts, the cost of an average grievance was estimated to be $ 6,500 when considering time and direct costs (V = $ 6,500)

45www.exploreHR.org

Steps to Convert Data to Monetary Values

3. Calculate the change in

performance data

• Six months after the program was completed, total grievances per month reaching step two declined by ten.

• Seven of the then grievance reductions were related to the program as determined by supervisors (isolating the effects of training)

Steps Illustration

4. Determine an annual amount for the change

• Using the six month value, seven per month (grievance reductions), yields an annual improvement of 84 (7 x 12 months)

46www.exploreHR.org

Steps to Convert Data to Monetary Values

5. Calculate the annual value of improvement

• Annual value = 84 x $ 6,500 = $ 546,000

Steps Illustration

47www.exploreHR.org

Tabulating cost of the program

• Tabulating the costs involves monitoring or developing all of the related costs of the program targeted for the ROI calculation.

Tabulating Cost of the Program

48www.exploreHR.org

Tabulating cost of the program

• Cost components that should be included are :

• The cost to design and develop the program

• The cost of all program materials provided to each participant

• The cost for facilitator

• The cost of the facilities of the training program

Tabulating Cost of the Program

49www.exploreHR.org

Tabulating cost of the program

• Cost components that should be included are :

• Travel, lodging, and meal costs for the participants

• Salaries, plus employee benefits of the participants who attend the training

• Administrative and overhead costs of the training function, allocated in some convenient way

Tabulating Cost of the Program

50www.exploreHR.org

An Example to Illustrate ROI analysis

Weeks after training Trained Group Control Group

1 US$ 9,723 9,698

2 9,978 9,720

3 10,424 9,812

13 13,690 11,572

14 11,491 9,683

15 11,044 10,092

Average for weeks 12,075 10,449 13, 14, 15

Post Training Data

Average Weekly Sales

51www.exploreHR.org

An Example to Illustrate ROI analysis

Data US$

Average weekly salesTrained Group 12,075

Control Group 10,449

Increase 1,626

Profit Contribution from Training 2 % 32.50

(Training Impacts)

Total weekly improvement 1,495

(32.5 x 46 participants)

Total annual benefit 71,760

($ 1,495 x 48 weeks)

Annualized Program Benefits

Note : 46 participants were still in job after 3 months

52www.exploreHR.org

An Example to Illustrate ROI analysis

Cost US$

Facilitation fees 11,250

Program materials : $ 35 x 46 1,610

Meals : 3 days x $ 28 x 46 1,288

Participant salaries plus benefit (35 %) 12,442

Coordination and Evaluation 2,500

Total Cost 29,090

Cost Summary : An Illustration

ROI (%):

$ 71,760 - $ 29,090

$ 29,090

146 %

x 100=

=

53www.exploreHR.org

Enhancing Training Effectiveness

54www.exploreHR.org

How Effective is Your Training Program?

Broad and Newstrom (1992) report studies have shown less than 30% of what is actually taught transfers to the job in a way that enhances performance.Source : Broad, M., & Newstrom, J. W. (1992). Transfer of training: Action packedstrategies to ensure high payoff from training investments. Reading, MA: Addison-Wesley.

55www.exploreHR.org

Source of Barriers to Training Transfer

• Lack of reinforcement on the job

• Interference from immediate (work) environment

• Nonsupportive organizational culture

• Trainees’ perception of impractical training programs

• Separation from the inspiration or support of the trainer

56www.exploreHR.org

The Transfer Partnership

Trainee Trainee recognizes need for new skills

Trainee Trainer Trainer designs and/or delivers learning experiences

Trainee Trainer Manager supports learning and application on the job

Manager

57www.exploreHR.org

The Transfer Matrix

Before During After

Manager

Trainer

Trainee

Time PeriodsRo

le P

laye

rs

58www.exploreHR.org

Manager

Before Training

• Build transfer of training into supervisory performance standards

• Collect baseline performance data• Involve supervisors and trainees in needs

analysis process• Involve trainees in program planning• Brief trainees on the importance of the training

(course objective, content, process, and application on the job)

• Review instructional content and materials• Plan to participate in training sessions• Encourage trainees attendance at all sessions

59www.exploreHR.org

Trainer

Before Training

• Align the training plan with the organization's strategic plan

• Systematically design instruction• Provide proactive opportunities• Design a peer coaching component for the

program and its follow-up activities

60www.exploreHR.org

Trainee

Before Training

• Provide input into program planning• Actively explore training options• Participate in advance activities

61www.exploreHR.org

Manager

During Training

• Prevent interruptions• Transfer work assignment to others• Monitor attendance and attention to training• Recognize trainee participation• Participate in transfer action planning• Review information on employee in training• Plan assessment of transfer of new skills to the

job

62www.exploreHR.org

Trainer

During Training

• Develop application-oriented objectives• Answer the “WIIFM” question• Manage the unlearning process• Provide realistic work-related tasks• Give individualized feedback• Provide job performance aid

63www.exploreHR.org

Trainee

During Training

• Maintain an ideas and application notebook• Participate actively• Form support groups• Plan for applications• Create behavioral contracts

64www.exploreHR.org

Manager

After Training

• Plan trainees’ reentry• Provide opportunities to practice new skills• Have trainees participate in transfer-related

decisions• Reduce job pressures initially• Give positive reinforcement• Schedule trainee briefings for co-workers• Set mutual expectations for improvement• Arrange proactive (refresher) sessions• Provide and support the use of job aids

65www.exploreHR.org

Trainer

After Training

• Provide follow-up support• Conduct evaluation surveys and provide

feedback• Develop and administer recognition system• Provide refresher/problem-solving sessions

66www.exploreHR.org

Trainee

After Training

• Practice self-management• Review training content and learned skills• Develop a mentoring relationship• Maintain contact with training ‘buddies’

67www.exploreHR.org

References/Recommended Further Readings

1. Donald Kirkpatrick, Evaluating Training Programs : The Four Levels, Berrett-Koehler Publishers. You can obtain this excellent book at this link : http://www.amazon.com/Evaluating-Training-Programs-Four-Levels/dp/1576753484/ref=sr_1_1?ie=UTF8&s=books&qid=1219800986&sr=1-1

2. Jack J. Phillips and Patricia Phillips, In Action : Measuring Return On Investment, American Society for Training & Development. You can obtain this excellent book at this link : http://www.amazon.com/Action-Measuring-Return-Investment/dp/1562860089/ref=sr_1_3?ie=UTF8&s=books&qid=1219801000&sr=1-3

3. Mary Broad and John Newstrom, Transfer of training: Action Packed Strategies to Ensure High Payoff from Training Investments, Addison-Wesley. You can obtain this excellent book at this link : http://www.amazon.com/Transfer-Training-Action-Packed-Strategies-Investments/dp/0738205672/ref=sr_1_1?ie=UTF8&s=books&qid=1219801069&sr=1-1

68www.exploreHR.org

End of Material

69www.exploreHR.org

You can download these excellent slides at :

www.HR-Management-Slides.com