Training Effectiveness, design and delivery-

an ISO-9000 perspective.By Saroj Ku. Behera

1

Agenda

• Overview:• ISO Model• Requirements (9K, 14K, 18K).

• Kirpatrick Model for Training Evaluation.

• ISO-10015: Guidelines for Training.

2

Background

3

Customer

Requirements

Customer

Satisfaction

CONTINUAL IMPROVEMENT OF THEUNIFIED MANAGEMENT SYSTEM

Managementresponsibility

Resourcemanagement

Measurement,analysis,

improvement

Productrealization

ProductOutputInput

Quality Management System Model

4

6. Resource Management6. Resource Management

6.2 Human Resources

6.2.1 General6.2.1 General

Personnel whose work affect product quality to be COMPETENT on the basis of appropriate education, training, skills and experience

5

6. Resource Management 6. Resource Management (Contd.)(Contd.)

6.2 Human Resources6.2.2 Competence, Awareness and Training6.2.2 Competence, Awareness and TrainingThe organization to

• Determine necessary competence for personnel performing work affecting product quality,

• Provide training or take any other actions to satisfy these needs***,• Evaluate effectiveness of the actions taken.***

Ensure that its personnel are aware of• relevance and importance of their activities• how they contribute to the achievement of the Quality Objectives

Maintain records of education, experience, training and skills*** Changed in 2008 version

6

6. Resource Management 6. Resource Management (Contd.)(Contd.)

6.2 Human Resources (Contd)

6.2.2 Competence, Awareness and Training 6.2.2 Competence, Awareness and Training ((Guidelines)Guidelines)

Make employees at each relevant function level aware of• importance of conformance with the Quality Policy & QMS

requirements• significant impacts of their work activities on quality, actual or

potential • benefits of improved personnel performance• roles and responsibilities in achieving conformance to quality policy• the potential consequences of departure from specified procedures

7

Paragraph 6.2

Words, "affecting product quality" changed to "affecting conformity to product requirements”

Paragraph 6.2.2

Sub-clause (b) -"provide training or take other actions to satisfy these needs" changed to "where applicable training needs to be provided to achieve the necessary competence".

Sub-clause (c) – obligation to ensure that the staff attained the expected competence.

Changes in ISO-9001-2008

8

ISO-14000 & OHSAS-180014.4.2 Training Awareness &

competence

• Identify training needs• Establish training procedures

• Importance of conformance• Consequences of departure from

procedures

• Personnel competent by education, training, and/or experience

9

4.4.2 Training Awareness & competence

• Training department completes Training Needs Analysis (TNA) and training schedule, reviews at least annually and revises as necessary

• Permanent on-site contractors included in training program

• Training records maintained

10

Effectiveness

Extent to which planned activities are realized and planned results achieved.

ISO 9000:2000 3.2.14

ISO 9001 specifies requirements for a quality management system that can be used for internal application by organizations, or for certification, or for contractual purposes. It focuses on the effectiveness of the quality management system in meeting customer requirements.

ISO 9001: 2000 0.3

11

Training Design: ISO-9001

Part of Product Realization Process, Clause 7.3

Training Design Planning Design Inputs Design Output Design Review Design Verification Design Validation Control of changes

1212

7.3 Design & Development (D/D)7.3.1 Design and Development Planning7.3.1 Design and Development Planning• Plan and control Design and Development of the product• During planning, determine

• Stages of design and development• Review, verification and validation appropriate to each Design

and Development stage • Responsibilities and authorities for Design and Development

• Manage interface between different involved groups to ensure effective communication and clear responsibilities

• Update planning output, as appropriate, as Design and Development progresses

7. Product Realization 7. Product Realization (Contd.)(Contd.)

13

7.3 Design & Development (D/D) (Contd)

7.3.2 Design & Development Inputs7.3.2 Design & Development InputsDetermine the inputs related to product requirements and

maintain recordsDesign & Development inputs to include

Functional and performance requirements Applicable statutory and regulatory requirements Where applicable, Information derived from previous similar

designs Other requirements essential for Design and Development

Review inputs for adequacy. Requirements to be complete, unambiguous and non-conflicting

7. Product Realization 7. Product Realization (Contd.)(Contd.)

14

7.3 Design & Development (D/D) (Contd)

7.3.3 Design & Development Outputs7.3.3 Design & Development OutputsOutputs of D/D provided in a form that enables verification

against D/D inputs. This shall be approved prior to release D/D Outputs shall :

meet D/D input requirements provide appropriate information for purchasing, production and

for service provisions contain or reference product acceptance criteria specify product characteristics essential for safe and proper use

7. Product Realization 7. Product Realization (Contd.)(Contd.)

15

7.3 Design & Development (D/D) (Contd)

7.3.4 Design & Development Review7.3.4 Design & Development Review At suitable stages, conduct systematic D/D reviews

to evaluate the ability of results of D/D to fulfill requirements to identify any problems and proposed necessary actions

Involve and include representatives of functions concerned with design stage(s) being reviewed

Records of results of reviews and necessary actions to be maintained

7.3.5 Design & Development Verification7.3.5 Design & Development Verification Perform verification to ensure that D/D outputs have satisfied D/D

input requirements Records of results of verification & necessary actions to be

maintained

7. Product Realization 7. Product Realization (Contd.)(Contd.)

16

7.3 Design & Development (D/D) (Contd)

7.3.6 Design & Development Validation7.3.6 Design & Development Validation Perform D/D validation in accordance with planned arrangements

(see 7.3.1) to ensure Resulting product is capable of fulfilling the requirements for the

specified or known intended use or application Wherever practicable, validation to be completed prior to delivery

or implementation of the product Records of results of validation and necessary actions to be

maintained

7. Product Realization 7. Product Realization (Contd.)(Contd.)

17

7.3 Design & Development (D/D) (Contd)

7.3.7 Control of Changes7.3.7 Control of ChangesChanges in D/D to be identified and records maintained The changes to

be reviewed, verified and validated, as appropriate be approved before implementation include evaluation of the effect of changes on constituent parts

and delivered products Records of the results of validation and necessary actions

to be maintained

7. Product Realization 7. Product Realization (Contd.)(Contd.)

18

Training EvaluationTraining Evaluation

19

How the training is evaluated?

Factors affecting evaluation?

20

Evaluation- Myths

I Can’t Measure the Result of My Training Efforts.I Don’t Know What Information to Collect.If I Cannot Calculate the ROI, the Evaluation Is

Useless.My CEO Does not Require Evaluation, So, Why

Should I Do It ?There Are Too Many Variables Affecting the

Behavior Change for Me to Evaluate the Impact of Training.

Evaluation Will Lead to Criticism.I Don’t Need to Justify My Existence; I Have a

Proven Track Record.

21

FactorFactor How Factor Influences Type of Evaluation How Factor Influences Type of Evaluation DesignDesign

Change potential Can program be modified?

Importance Does ineffective training affect customer service, product development, or relationships between employees?

Scale How many trainees are involved?

Purpose of training

Is training conducted for learning, results, or both?

Organization culture

Is demonstrating results part of company norms and expectations?

Time frame When do we need the information? 22

Questions we must answer …How do participants feel about our

training program?

Are participants learning?

Is their learning transferring to the job?

Does the organization benefit from our training efforts?

23

Donald Kirpatrick’sModel

for Training Evaluation

24

Finding answers …Kirkpatrick’s 4 Levels of Evaluation

Level 1: ReactionHow do participants feel about our training

program?

Level 2: LearningAre participants learning?

Level 3: BehaviorIs their learning transferring to the job?

Level 4: ResultsDoes the organization benefit?

Source: Kirkpatrick,1998 25

 DONALD KIRKPATRICK'S 4 LEVELS OF EVALUATING TRAINING

Levels Description Comments

Level 1 Reaction Trainee reaction to the course. Does the trainee like the course? Usually in the form of evaluation forms, sometimes called "smile sheets".

Most primitive and widely-used method of evaluation. It is easy, quick, and inexpensive to administer. Negative indicators could mean difficultly learning in the course.

Level 2 Learning Did trainees learn what was based on the course objectives?

Learning can be measured by pre- and post tests, either through written test or through performance tests.

Level 3 Behavior Trainee behavior changes on the job - are the learners applying what they learned?

Difficult to do. Follow-up questionnaire or observations after training class has occurred. Telephone interviews can also be conducted.

Level 4 Results Ties training to the company's bottom line.

Generally applies to training that seeks to overcome a business problem caused by lack of knowledge or skill. Examples include reductions in costs, turnover, absenteeism and grievances. May be difficult to tie directly to training.

26

Appeal of Kirkpatrick’s Model:

Assesses important areas

Widely known

Simple framework

Easy to explain and understand

27

However . . . Widely Know ≠ Widely Used

Level 1: Often (over 90%)Level 2: Sometimes (less than 35%)Level 3 & 4: Rarely (less than 15%)

Why is this a problem?Level 3 and 4 often perceived as:

Difficult to measure

Time consuming

Beyond the realm of most trainers

Level 1 result does not always mean similar Learning / Transfer / ROI results

Source: Pershing & Gilmore, 200428

Other problems …Undermines Management Partnership

Training ≠ “Silver Bullet”Training is only one strategy within entire

Performance SystemLevel 3 & 4 should include evaluations of entire

Performance System - not just training

Lacks Performance System FocusWhat about rest of Performance Environment?What factors impede / enable usage of training?

Feedback Goes to Wrong PeopleFeedback to training function only is incompleteMust include Performance Environment owners

Source: Brinkerhoff & Dressler, 200229

In Summary:An evaluation tool must be integrated

into the training programs.

Kirkpatrick’s Four Levels of Evaluation is well known, but has limitations:Unlikely completion of all 4 LevelsLacks performance system focus

30

ISO-10015Guidelines for Training

9 31

Background of ISO-10015

22 country representatives developed the draft text over several years, culminating in the publication of ISO 10015 issued December 1999.

The advantages:

• based on the process oriented concepts of the

new 9000:2000 ISO family of standards.• Offers specific guidance for training design,

review and implementation.• Provides context to organization's needs.

32

1. Linking training investment with company

performance • Key client is the organization, not only the persons being trained.

(ROI Approach)

• A company has to recognize first what is the performance challenge it faces and the causes for it.

• ISO 10015 offers a clear road map to connect training to performance goals and use it for individual and group performance improvement.

2. Organizing training on the basis of pedagogical principles and processes

• Training as a tool to cover an identified performance gap.

• Establishing an appropriate training design and effective learning processes.

• Ensures that training uses resources (finances, time and energy) optimally.

Key features of ISO 10015

33

Training: for organizational needs

34

4.1Training is 4-step processFor selecting and implementing training to

close the gaps between required and existing competence, management should monitor the following stages:

a) defining training needs;b) designing and planning training;c) providing for the training;d) evaluating the outcome of training.

35

The Training Process

36

4.2 Defining Training Needs4.2.1 General4.2.2 Defining the needs of the

organisation4.2.3 Defining the competence needs4.2.4 Reviewing Competence4.2.5 Defining Competence gaps4.2.6 Identifying solutions close to

competence gaps4.2.7 Defining the specifications for

training needs

37

Training Processes-1

38

Training Processes-2

39

Training Processes-3

40

Training Processes-4

41

4.3 Designing and planning training

4.3.1 General4.3.2 Defining the constraints4.3.3 Training methods and criteria

for their selection4.3.4 Training Plan specifications4.3.5 Selecting a training provider

42

43

4.4 Providing the training

4.4.1 General4.4.2 Providing Support

1. pre-training support2. training support3. end-of-training support

44

45

4.5Evaluating training outcomes4.5.1 General4.5.2 Preparing evaluation report

Contents of evaluation report:– specification of training needs– Evaluation criteria and description of

sources, methods and schedule for evaluation.

– Analysis of data and interpretation– Review of training costs– Conclusions and recommendations for

improvement.

46

47

5 Monitoring and improving the training process

5.1 General5.2 validation of training process

Monitoring is the key tool:– Monitoring is to be done by competent

personnel only.– Monitor records at all stages. Detect

nonconformity at all stages. Take preventive and corrective actions.

– if procedures followed and requirements met, then update the personal competence records.

– if procedures not followed yet requirements are met, revise procedures and update training records.

– If procedures followed and requirement not met, either correct the training process or seek a non-training solution.

• Identify further opportunities for improvement. 48

49

Thank You

Lawrence & Mayo Pvt Ltd

50