Chapter 6 Training Evaluation

Copyright © 2010 by the McGraw-Hill Companies, Inc. All rights reserved.McGraw-Hill/Irwin

6-2

-Benefits for trainees: learning new skills or behaviors.

-Benefits for the company: increased sales , more satisfied customers.

The training evaluation measures specific outcomes or criteria to determine the benefits of the program.

Introduction

*Training effectiveness : *Training effectiveness : the benefits that the company and the trainees receive from training. “shows that the time, money and effort devoted to training actually made difference.

Chapter 6: Training Evaluation

“ Why we evaluate training programs and how?”

6-3

*Training outcomes or criteria : *Training outcomes or criteria : measures that the trainer and the company use to evaluate training programs. - to determine the effectiveness of the program we have to conduct a training evaluation.

Introduction

Chapter 6 : Training Evaluation

6-4

Training evaluationTraining evaluation - the process of collecting the outcomes needed to determine if training is effective.

Evaluation designEvaluation design - collection of information, including whom, what, when, and how, for determining the effectiveness of the training program.

Introduction

Chapter 6 : Training Evaluation

6-5

Introduction

- The information from the needs assessment, the characteristics of the learning environment and the steps taken to ensure transfer of training should all be used to develop an evaluation plan.

- In order to identify appropriate training outcomes, a company needs to look at its business strategy, its organizational analysis ( why are we conducting training? How is it related to the business?) , its person analysis ( who needs training?), its task analysis ( what is the training content?), the learning objectives of the training, and its plan for training transfer.

6-6

1- Companies make large investments in training programs to help gain a competitive advantage.

-Companies invest in training because learning creates knowledge that distinguishes successful companies and employees from those who are not.

Reasons for Evaluating Training

Chapter 6 : Training Evaluation

6-7

2- Companies that conduct training are more likely to have more positive human resource outcomes (attitudes and motivation, behaviors, human capital) and grater organizational performance outcomes (performance and productivity) or financial outcomes (profits and financial indicators)

Reasons for Evaluating Training

6-8

3- Training evaluation provides a way to understand the investment that training produces and provides information needed to improve training.

- if the company receives an inadequate return on investment in training, the company will likely reduce its investment in training or look for training providers outside the company who can provide training experiences that improve performance, productivity, customer satisfaction.

Reasons for Evaluating Training

6-9

A) Formative evaluation :the evaluation of training that takes place during program design and development.

1. It helps ensure that the training program is well organized and runs smoothly, and trainees learn and are satisfied with the program.

2. It provides information about how to make the program better; it involves collecting qualitative data about the program. “ opinions, beliefs and feelings about the program”

Reasons for Evaluating Training

-Training evaluation involves formative and summative evaluation.

6-10

3. formative evaluation ask customers, employees, managers and subject-matter experts their opinions on the description of the training content and objectives and the program design.

4. these people are also asked the clarity and ease of use of a part of the training program that is provided to them in the way that it will be delivered “ face to face, online, video”

5. the formative evaluation is conducted either individually or in groups before the program is made available to the rest of the company.

6. Trainers may also be involved to measure the time requirements of the program.

Formative evaluation (cont)

6-11

Formative evaluation (cont)

7- as a result of the formative evaluation, training content may be changed to be more accurate, easier to understand, or more appealing. The training method can be adjusted to improve learning (e.g. provide trainees with more opportunities to practice or give feedback)

9- introducing the training program as early as possible to managers and customers helps in getting them to buy into the program, which is critical for their role in helping employees learn and transfer skills. It also allows their concerns to be addressed before the program is implemented.

6-12

Pilot testing: the process of previewing the training program with potential trainees and managers or with other customers (persons who are paying for the development of the program).

- The information gained from this preview would be used by program developers to improve the program before it is made available to all employees.

Formative evaluation (cont)

6-13

Summative evaluation: an evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program.

“ have trainees acquired knowledge, skills, attitudes, behavior or other outcomes identified in the training objectives ?”

- It may include measuring the monetary benefits ”also known as return on investment” that the company receives from the program.

- It involves collecting quantitative data through `tests, ratings of behavior, or objective measures of performance such as volume of sales, accidents or patents.

Reasons for Evaluating Training

6-14

1. To identify the program’s strengths and weaknesses.2. To assess whether content, organization, and administration of

the program including the schedule, accommodations, trainers and materials contribute to learning and the use of training content on the job.

3. To identify which trainees benefited most or least from the program.

4. To gather data to assist in marketing training programs.5. To determine the financial benefits and costs of the program.6. To compare the costs and benefits of training versus non-

training investments such as work redesign or a better employee selection system.

7. To compare the costs and benefits of training versus different training programs to choose the best program.

Reasons for Evaluating Training

Why a training program should be evaluated:

6-15

The Evaluation Process

To identify what knowledge, skills, behavior or other learned capabilities are needed.

to assess the extent to which learning and transfer have occurred.

Consider : expertise ,how quickly the information is needed, change potential and the organizational culture

Previewing the program (formative evaluation),collecting training outcomes according to the design.

Consider the expectations of stakeholders.Determine how training content can be used on the job

6-16

Table 6.1 - Kirkpatrick’s Four-Level Framework of Evaluation Criteria

Chapter 6 : Training Evaluation

6-17

Outcomes Used in the Evaluation of Training Programs

The hierarchical nature of Kirkpatrick’s framework suggests that higher level outcomes should not be measured unless positive changes occur in lower level outcomes.

The framework implies that changes at a higher level are more beneficial than changes at a lower level.

Chapter 6 : Training Evaluation

6-18

Outcomes Used in the Evaluation of Training Programs (cont.)

Kirkpatrick’s framework criticisms:Research has not found that each level is

caused by the level that precedes it in the framework, nor does evidence suggest that the levels differ in importance.

The approach does not take into account the purpose of the evaluation.

Outcomes can and should be collected in an orderly manner, that is, measures of reaction followed by measures of learning, behavior, and results.

Chapter 6 : Training Evaluation

6-19

6-20

Outcomes Used in the Evaluation of Training Programs

1. Reaction outcomes: trainees perception of the program, including the facilities, trainers and content.

They are often called class or instructor evaluations. It is collected at the program’s conclusion.Reaction outcomes are typically collected via a

questionnaire completed by trainees.A reaction measure should include questions related

to the trainee’s satisfaction with the: instructor, training materials and training administration (ease of registration, accuracy of course description) as well as the clarity of course objectives and usefulness of the training content.

6-21

Outcomes Used in the Evaluation of Training Programs

Reaction outcomes

Include items related to the trainer’s preparation, delivery, ability to lead a discussion, organization of the training materials, ability to answer questions and stimulate trainees’ interest in the course.

Research results suggests that reactions have the largest relationship with transfer of learning.

6-22

2. learning or Cognitive outcomes: Used to determine the degree to which trainees are

familiar with principles, facts, techniques, procedures or processes in the training program.

Measures what knowledge trainees learned in the program.

Pencil-and-pen are used to assess cognitive outcomes. Measures whether a trainee knows how to make a

decision. They do not help to determine if the trainee will actually use decision-making skills on the job.

Outcomes Used in the Evaluation of Training Programs

6-23

3- Behavior and Skill-based outcomes: Used to assess the level of technical or motor skills and

behaviors. Skill-based outcomes include acquisition or learning of skills (skill learning) and use of skills on the job (skill transfer).

The extent to which trainees have learned skills can be evaluated by observing their performance in work samples such as simulators.

Trainees may be used to provide ratings of their own behavior or skills (self ratings). Peers, managers and subordinates may also be asked to rate trainees’ behavior or skill based on their observations.

Outcomes Used in the Evaluation of Training Programs

6-24

4- Affective outcomes: “attitudes and motivation” :

- Include tolerance for diversity, motivation to learn, safety attitudes and customer service orientation.

Outcomes Used in the Evaluation of Training Programs

6-25

5- Results: used to determine the training program’s payoff for the company.

e.g: Increased production and reduced costs related to employee turnover, accidents and equipment downtime as well as improvements in product quality or customer service.

Outcomes Used in the Evaluation of Training Programs

6-26

Outcomes Used in the Evaluation of Training Programs

6-Return on investment: comparing the training’s monetary benefits with the cost of training.Direct costs :salaries and benefits for all

employees involved in training; program material and supplies; equipment or classroom rentals or purchases; and travel costs.

Indirect costs : not related directly to the design, development, or delivery of the training program.

Benefits :value that the company gains from the training program.

6-27

6-Return on investment Training quality index (TQI): a computer application that

collects data about training dapartment performance, productivity, budget and courses and allows for detailed analysis of data.

TQI tracks all department training data into five categories: effectivenss, quantity, perceptions, financial impact and operational impact.

Questions such as: how can the cost of training be justified, what is the operational impact of training, and what amount of training have technicians received can now be answered through TQI.

Outcomes Used in the Evaluation of Training Programs

6-28

Determining Whether Outcomes are Appropriate

CriteriaRelevance

The extent to which training outcomes are related to the learned capabilities emphasized in the training program.Criterion contamination: the extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions.Criterion deficiency : the failure to measure training outcomes that were emphasized in the training objectives.

Reliability The degree to which outcomes can be measured consistently over time-allows the trainer to have confidence that any improvements in posttraining test scores from pretraining levels are the result of learning that occurred in the training program not test characteristics.

Discrimination The degree to which trainees’ performance on the outcome actually reflects true differences in performance.

Practicality The ease with which the outcome measures can be collected.

6-29

Criterion Deficiency, Relevance, and Contamination

6-30

Which training outcomes should be collected?

Surveys of companies’ evaluation practices indicate that reactions and cognitive outcomes are the most frequently used outcomes in training evaluation.

It is not necessary to collect all five levels of outcomes to evaluate the program, the training program objectives determine which ones should be linked to the broader business strategy.

To ensure adequate training evaluation, companies should collect outcome measures related to both learning ( levels 1 and 2) and transfer of training (levels 3,4 and 5)

6-31

Training Program Objectives and Their Implications for Evaluation

6-32

Evaluation Designs

Threats to validity - factors that will lead an evaluator to question either the:Internal validity - the believability of the study

results.External validity - the extent to which the

evaluation results are generalizable to other groups of trainees and situations.

6-33

Table 6.7 - Threats to Validity

6-34

Methods to Control for Threats to ValidityPretests and Posttests

A comparison of the posttraining and pretraining measures can indicate the degree to which trainees have changed as a result of training.

Random assignment - assigning employees to the training or comparison group on the basis of chance.

Helps to reduce the effects of employees dropping out of the study, and differences between the training group and comparison group in ability, knowledge, skill, or other personal characteristics.

Evaluation Designs (cont.)

6-35

Methods to Control for Threats to ValidityUsing a comparison group - employees who

participate in the evaluation study but do not attend the training program.

Helps to rule out the possibility that changes found in the outcome measures are due to factors other than training.

Evaluation Designs (cont.)

6-36

Table 6.8 - Comparison of Evaluation Designs

6-37

Time series - training outcomes are collected at periodic intervals both before and after training.

It allows an analysis of the stability of training outcomes over time.

Reversal - time period in which participants no longer receive the training intervention.

Types of Evaluation Designs

6-38

Table 6.12 - Factors That Influence the Type of Evaluation Design

6-39

Determining Return on Investment (ROI)

Cost-benefit analysis - process of determining the economic benefits of a training program using accounting methods that look at training costs and benefits.

ROI should be limited only to certain training programs, because it can be costly.

6-40

Determining Return on Investment (ROI) (cont.)

Determining costsMethods for comparing costs of alternative

training programs include the resource requirements model and accounting.

Determining benefits – methods include:technical, academic, and practitioner

literature.pilot training programs and observance of

successful job performers.Estimates by trainees and their managers.

6-41

Determining Return on Investment (ROI) (cont.)

To calculate ROI, divide benefits by costs. The ROI gives an estimate of the dollar return expected from each dollar invested in training.

6-42

Table 6.13 - Determining Costs for a Cost Benefit Analysis

6-43

Determining Return on Investment (ROI) (cont.)

Utility analysis - a cost-benefit analysis method that involves assessing the dollar value of training based on:estimates of the difference in job performance

between trained and untrained employees.the number of individuals trained.the length of time a training program is

expected to influence performance.the variability in job performance in the

untrained group of employees.

6-44

Table 6.16 - Training Metrics