Date post: | 22-Jul-2015 |
Category: |
Documents |
Upload: | kamal-birdi |
View: | 247 times |
Download: | 1 times |
.
THE TAXONOMY OF TRAINING AND DEVELOPMENT OUTCOMES (TOTADO)
EVALUATION FRAMEWORK
Dr Kamal BirdiInstitute of Work Psychology
Management SchoolUniversity of Sheffield
Sheffield S10 2TNUnited Kingdom
Email: [email protected]: +44 (0)114 222 3288
This document contains a description of the TOTADO training and development evaluation framework and two tools to help with its implementation and data collection.
1. A paper summarising the TOTADO framework
2. A TOTADO evaluation strategy planning template
3. A TOTADO evaluation interview schedule
1
.
THE TAXONOMY OF TRAINING AND DEVELOPMENT OUTCOMES (TOTADO): A NEW MODEL OF TRAINING EVALUATION
Kamal BirdiInstitute of Work Psychology, University of Sheffield
Paper presented at the Annual BPS Division of Occupational Psychology Conference, 13th-15th
January 2010, Brighton, UK.
INTRODUCTIONThe effectiveness of training and development activities can be assessed by the extent to which those
activities produce desirable outcomes and reduce undesirable ones. Despite organisations expending a
great deal of effort setting up training and development programmes, comparatively less attention is
paid to systematically evaluating their effectiveness (Sadler-Smith, 2006). This lack of assessment
may be partly due to the extra cost and effort involved but it also seems that many Human Resource
practitioners are unaware of how and what to evaluate (Bates, 2004). The objectives of this paper are
therefore to briefly review and critique past models of evaluation and then offer a new model of
evaluation (the Taxonomy of Training and Development Outcomes – TOTADO) which addresses
these shortcomings.
A review of the literature indicates that three types of approaches can be described when considering
how outcome evaluation has been previously addressed. First, the predominant traditional training
evaluation approach (e.g. Kirkpatrick, 1959; Warr et al., 1970) offers a practical framework for
assessing outcomes in terms of trainee reactions, learning of knowledge and skills, on-the-job
behaviour and organisational performance. Although popular in organisations, these models tend to
be theoretically and practically vague in their specification of different types of learning outcomes,
work behaviours and organisational performance criteria (Phillips & Phillips, 2001). A second
approach is illustrated by Kraiger et al. (1993), who have proposed a classification scheme for
individual learning outcomes (cognitive, skill-based or affective) based on psychological theory. This
framework is useful for defining the specific types of criteria which indicate learning and it allows
theoretical rationales to be developed when choosing predictors of learning outcomes. However, this
second approach has rarely been used in subsequent research studies (Beech & Leather, 2006) and,
since it only focuses on learning outcomes, it also needs to be incorporated into the wider frameworks
offered by Kirkpatrick (1959) and Warr et al. (1970). Finally, the employee development literature
(e.g. Benson et al., 2004; Maurer et al., 2002) has empirically assessed experienced or perceived
development outcomes from different types of learning and development activities. The findings from
these studies indicates that such activities can lead to changes in a variety of areas not considered by
traditional training evaluation approaches, such as changing employees’ status, relationship to
colleagues or even health. A broader perspective on types of outcomes beyond individual learning of
knowledge and skills, work behaviour and organisational performance is therefore needed if training
and development effectiveness is to be accurately assessed. The Taxonomy of Training and
Development Outcomes (TOTADO) attempts to do this (see Figure 1) and will be outlined.
2
.
DESCRIPTION OF THE TAXONOMY OF TRAINING AND DEVELOPMENT OUTCOMES
TOTADO integrates and elaborates on previous evaluation approaches to define a taxonomy of
outcomes that allows us to capture and describe the diverse range of outcomes experienced from
participation in both training and other development activities. The research described earlier revealed
that although the impact of development activities can be assessed at differing levels, much of the
work has concentrated on the individual. Kirkpatrick’s model of evaluation has been mainly used to
consider the individual level of analysis from Level 1 to Level 3; Kraiger et al.’s (1993) taxonomy
considers solely individual level learning outcomes; most studies from the employee development
literature have used surveys to assess the impact of training and development activities on their
participants. Consequently, the outcomes described in the previous section refer predominantly to the
individual, whereas other levels of outcomes are equally important. Within the TOTADO framework,
it is therefore said that outcomes can be measured at four basic levels: individual, team (or work
group), organisational and societal.
Figure 1. The Taxonomy of Training and Development Outcomes (TOTADO) Framework.
1. Individual-level outcomes
3
.
With TOTADO we extend previous evaluation models by elaborating five different types of
individual-level outcomes where the trainee is the unit of analysis: Affective (e.g. participant reactions
to training, motivation, self-efficacy, attitudes, mental well-being); Cognitive (verbal knowledge,
knowledge representation); Behavioural (off-the job task performance, on-the-job task performance);
Physical (e.g. health, fitness, injuries): and Instrumental (events, actions or status changes resulting
from participation in training and development activities that are work intrinsic (increased job control),
work extrinsic (pay rise, promotion) or work relational (forming new relationships or networks)).
2. Team-level outcomes
With the increasing implementation of teamworking in the workplace, there has been a concurrent rise
in team training and development activities (Wilson et al., 2007). For example, many organisations
send work groups on external “outward-bound” courses where team members are required to co-
operate and work closely to achieve given objectives. Evidence to date of the impact of such activities
tends to be anecdotal, unsystematic and, in particular, ignores whether there is any change in team
functioning back in the workplace (Salas et al., 2004). Hence, we need to consider using the team or
work group as the unit of analysis for assessing the impact of these types of opportunities. The
majority of the outcomes which will be described for individuals can also be applied by aggregation to
the team context, although there is more focus on measuring interpersonal behaviours and group
cognition and affect. The four dimensions of impact here where the team or group is the unit of
analysis are therefore: Affective (e.g. changes in average team identity or trust); Cognitive (verbal
knowledge, knowledge representation in terms of shared cognition); Behavioural (team task processes,
intra-team processes); and Instrumental (events, actions or status changes for the team as a whole
such as increased team autonomy or gaining team bonuses).
3. Organisational-level outcomes
The next level takes the unit of analysis to be the organisation, or major functional components of it
(e.g. departments) and outcomes relate to those directly involved with the organization in a
transactional format (e.g. employees, customers, shareholders). Although individual- or team-level
outcomes can be aggregated to provide organisational-level outcomes (e.g. proportion of company
employees with a certain technical qualification), some variables can only be measured at the
organisational level (e.g. company profitability, share values). This is the domain that Kirkpatrick
(1959) refers to as Level 4 or Business Results, although he remains vague in terms of what aspects
should be included in this level. It would be conceptually very helpful to identify key categories into
which we can place organisational-level outcomes, much as was done with the individual and team
levels. The organisational effectiveness literature highlights the complex nature of measuring
performance but does attempt to provide some broad criteria. We therefore draw on the work of
management researchers (Cameron, 1980; Kaplan & Norton, 1996; and Richard et al., 2009) to specify
four dimensions of organisational performance outcomes: Financial (e.g. turnover, profit, share price);
Outputs (quantity, quality, variety of components, products or services); Processes (how well the
4
.
organisation functions e.g. time to complete tasks, stoppages, communication system efficiency); and
Resources (human and non-human).
4. Societal-level outcomes
The final level is rarely considered but relates to the impact of training beyond the organisational level
where the area or group outside the organisation is the unit of analysis (Watkins et al, 1998). The
majority of past models focus solely on impacts within the organisation, yet a wider perspective is
sometimes needed. For example, training employees in carbon-friendly practices within a
manufacturing company may translate to decreased waste and pollution levels for surrounding
communities. At this level, we can therefore also assess the impact of training on a sectoral, regional
or even national bases. For instance, does training all Job Centre Plus staff in coaching techniques to
help the long-term unemployed significantly improve regional employment rates? The five broad
dimensions of societal impact we can consider are therefore: Economic (e.g. £ investment in region);
Health and Welfare (e.g. mortality rates from heart disease); Educational (e.g. level of qualifications of
local populations); Law and Order (e.g. local crime rates): and Environmental (e.g. pollution levels).
DISCUSSION
The TOTADO framework therefore presents an approach which integrates and enhances past models
of evaluation by presenting multiple levels of impact and specifying within each level the
key dimensions of outcome change. Key features worth noting include:
• It acknowledges that the effectiveness of training and development participation should be
assessed on a number of levels and using a range of relevant criteria within each level. Conceptually,
it integrates the focused view of the traditional Kirkpatrick training evaluation approach with the wider
evaluation perspective offered by other literatures. By using the taxonomy in the early stages of
developing an evaluation strategy, researchers and practitioners can become fully aware of the variety
of salient outcome criteria that may need to be measured.
• The taxonomy clearly distinguishes between individual, team, organisational and societal
measures of organisational effectiveness. Approaches such as Kirkpatrick’s have tended to remain
vague about measures of individual work behaviour and organisational results and seemed to have
ignored team-level analysis altogether.
• The nature of the taxonomy also allows detailed and comparative examination of the types of
outcomes produced by different learning activities. For instance, how much impact does participation
in formal training compared to work-based development activities have on employees’ general job
satisfaction? In situations where several studies have examined the same development activity but
assessed different types of outcomes, the taxonomy provides a framework to integrate the findings in
order to provide a more complete overview of the impact of that activity. It would also highlight
which types of outcomes had not yet been assessed and hence drive future research to address those
gaps.
5
.
• Since the framework is a taxonomy, it does not make any claims of causality between outcomes.
Although Kirkpatrick (1996) claims that his framework is solely a taxonomy, it has widely been
interpreted as a training effectiveness model which specifies causal linkages between individual and
organisational outcomes (Alliger et al., 1997). If we are truly to identify the factors that influence
training outcomes, just considering other outcomes is clearly insufficient. Previous research has
consistently shown that a wide range of individual, training-related and organisational factors can also
have an impact on different types of outcomes (Noe, 2008). Such factors need to be included in any
good training effectiveness model. The evaluation taxonomy presented here, however, does provide
the building blocks for creating models which propose causal linkages between different types of
outcomes and other types of variables.
• Finally, the evaluation taxonomy provides a common format for assessing learning and
development impact from a diverse range of theoretical approaches. For example, cognitive scientists
conducting laboratory studies on learning complex tasks can be regarded as examining learning
effectiveness in terms of individual-level cognitive or behavioural outcomes that are measured in an
off-the-job context. Field studies conducted by occupational psychologists examining the transfer of
training to the workplace may focus on individual work behavioural outcomes. Team researchers can
be considered as assessing the impact of development activities in terms of changes in team
behaviours such as communication and co-operation. Organisational researchers go beyond individual
and team levels and relate the amount of training offered by companies to organisational-level goal-
directed outcomes such as sales or turnover (Wright et al., 2005). The framework should therefore be
flexible enough to cater for a wide variety of paradigms where learning effectiveness is to be assessed.
The presentation will illustrate the utility of TOTADO by providing examples from research
conducted by the author including:
• An evaluation study which shows how three types of creativity training programme run within a
government department can generate different levels of individual-level affective, cognitive and
behavioural outcomes for participants.
• A study of 500 UK organisations showing the relationship between organisations’ use of
individual training and development practices and different organisational performance outcomes.
REFERENCES
Alliger, G. et al. (1997). A meta-analysis of the relations among training criteria. Personnel
Psychology, 50, 341-358.
Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the
principle of beneficence. Evaluation and Program Planning, 27, 371-347.
6
.
Beech, B., & Leather, P. (2006). Workplace violence in the health care sector: A review of
staff training and integration of training evaluation models. Aggression and Violent
Behavior, 11(1), 27-43.
Benson, G. S., et al. (2004). You paid for the skills, now keep them: Tuition reimbursement
and voluntary turnover. Academy of Management Journal, 47(3), 315-331.
Cameron, K. (1980). Critical questions in assessing organizational effectiveness.
Organizational Dynamics, Autumn, 66-80.
Kaplan, R., & Norton, D. (1996). Linking the balanced scorecard to strategy. California
Management Review, 39, 53-79.
Kirkpatrick, D. (1959). Techniques for evaluating training programs. Journal for the
American Society for Training and Development, 13, 3-9.
Kraiger, K., et al. (1993). Application of cognitive, skill-based and affective theories of
learning outcomes to new methods of training evaluation. Journal of Applied
Psychology, 78(2), 311-328.
Maurer, T. e al. (2002). Perceived beneficiary of employee development activity:A three-
dimensional social exchange model. . Academy of Management Review, 27(3), 432-
444.
Noe, R. A. (2008). Employee Training and Development (Fourth Edition). New York:
McGraw-Hill.
Phillips, P. & Phillips, J. (2001). Symposium on the Evaluation of Training: Editorial.
International Journal of Training and Development, 5(4), 240-247.
Richard, P. et al. (2009). Measuring Organizational Performance: Towards Methodological
Best Practice. Journal of Management, 35(3), 718-804.
Sadler-Smith, E. (2006). Learning and Development for Managers. Oxford: Blackwell
Publishing.
Salas, E. et al. (2004). 25 years of team effectiveness in organizations: Research themes and
emerging needs. In C. Cooper & I. Robertson (Eds.), International Review of
Industrial and Organizational Psychology (Vol. 19, pp. 47-91). Chichester: John
Wiley & Sons.
Warr, P. et al. (1970). Evaluation of management training. London: Gower Press.
Watkins, R et al. (1998). Kirkpatrick plus: Evaluation and continuous improvement with a
community focus. Educational Technology, Research & Development, 46(4), 90-95.
Wilson, J. et al (2007). Group Learning. Academy of Management Review, 32(4), 1041-1059.
Wright, P. et al. (2005). The relationship between HR practices and firm performance:
Examining causal order. Personnel Psychology, 58, 409-446.
7
.
TOTADO EVALUATION STRATEGY FRAMEWORKNAME OF TRAINING / DEVELOPMENT PROGRAMME:
Level Dimension of effectiveness
Type of information Methods of collecting information
Who provides the information
When to collect information
TIME 0 TIME 1 TIME 2 TIME 3
period before the course
at the start of the course
at the end of the course
period after the course
IND
IVID
UA
L Affective Reactions, attitudes, motivation, self-efficacy, mental well-being
Cognitive Verbal Knowledge,Knowledge Representation
Physical Health, Fitness, Injuries
Behavioural Off-the-job, on-the-job task performance
Instrumental Work intrinsic, work extrinsic, work relational, nonwork
TE
AM Affective Shared team attitudes
Cognitive Shared cognition
Behavioural Team performance, team interaction processes
Instrumental Work intrinsic, work extrinsic, work relational,
8
.
NAME OF TRAINING / DEVELOPMENT PROGRAMME:O
RG
AN
ISA
TIO
NA
L Outputs Quantity, Quality, Variety of products / services
Resources Material, Personnel
Processes Efficiency, effectiveness of work systems
Financial Turnover, budget deficits, profit
SOC
IET
AL Economic e.g. £ investment in geographic
region, unemployment rates in area
Health and Welfare e.g. no. of deaths from heart disease, alcohol-related illnesses in local population
Educational e.g. level of qualifications of local population, % social inclusion of minorities
Law and order no. of robberies in the area, % reduction in drug crime
Environmental pollution and waste levels in the region
© K.Birdi, University of Sheffield 2006
Level Dimension of effectiveness
Type of information Methods of collecting information
Who provides the information
When to collect information
TIME 0 TIME 1 TIME 2 TIME 3
period before the course
at the start of the course
at the end of the course
period after the course
9
.
TOTADO INTERVIEW SCHEDULE
“In the following interview, I shall be asking you to describe any changes that you think have arisen because of programme X.”
A. INDIVIDUAL OUTCOMES
GENERAL LEAD INAs a result of taking part in X, how do you think you have changed?
AFFECTIVEHow has taking part in the X changed the way you FEEL about things?
COGNITIVEWhat type of things have you LEARNED as a result of taking part in X?
PHYSICALHow has taking part in the X changed your LEVELS OF HEALTH AND FITNESS?
BEHAVIOURAL What can you DO now that you couldn’t do before taking part in X?How has taking part in X changed your JOB PERFORMANCE?
INSTRUMENTALHas taking part in X led to any changes in the type of work you do?Has taking part in X led to any changes in your job status or conditions of employment?Has taking part in X helped you form new relationships at work or changed existing relationships?Has taking part in X led to any other changes in your work that we haven’t discussed yet?Has taking part in X led to any changes for you outside work?
IDENTIFYING BARRIERS TO CHANGE IN INDIVIDUAL OUTCOMESHas anything in particular stopped or hindered you changing personally the way you wanted after the training / taking part in Programme X?
Is there anything that could be changed about Programme X, your organisation or other aspects that could help you achieve these desired changes?
10
.
B. TEAM / GROUP OUTCOMES
GENERAL LEAD INAs a result of taking part in the X programme, how do you think your team has changed?
AFFECTIVEHow has taking part in the X changed the way your team FEELS about things?
COGNITIVEWhat type of things has your team LEARNED as a result of taking part in X?
BEHAVIOURAL What can your team DO now that you couldn’t do before taking part in X?How has taking part in X changed your team’s WORK PERFORMANCE?
INSTRUMENTALHas taking part in X led to any changes in the type of work your team does?Has taking part in X led to any other changes in the type of work your team does?Has taking part in X helped your team form new relationships at work or changed existing relationships?Has taking part in X led to any other changes in your team’s work that we haven’t discussed yet?
IDENTIFYING BARRIERS TO CHANGE IN TEAM/GROUP OUTCOMESHas anything in particular stopped or hindered your team or teams changing the way that was wanted after the training / taking part in Programme X?
Is there anything that could be changed about Programme X, your organisation or other aspects that could help you achieve these desired changes?
11
.
C. ORGANISATIONAL OUTCOMES
GENERAL LEAD INAs a result of running the X programme, how do you think your organisation has changed?
OUTPUTSHas running X led to any changes in your organisation’s outputs e.g. its productivity, quality levels, variety of products/sevices offered etc.?
RESOURCESHas running X led to any changes in your organisation’s employees e.g. their performance, morale, absenteeism?Has running X led to any changes in your organisation’s material resources e.g. reductions in wastage?
PROCESSES / OPERATIONSHas running X led to any changes in your organisation’s efficiency or the way work is carried out e.g. speed of production, turnaround on project times?
FINANCIALHas running X led to any changes in your organisation’s financial performance e.g. its profitability, balancing of budgets?
Has taking part in X led to any other changes in your organisation that we haven’t discussed yet?
IDENTIFYING BARRIERS TO CHANGE IN ORGANISATIONAL OUTCOMES
Has anything in particular stopped your department or organisation changing the way that was wanted after the training / taking part in Programme X?
Is there anything that could be changed about Programme X, your organisation or other aspects that could help you achieve these desired changes?
12
.
D. SOCIETAL OUTCOMES
GENERAL LEAD INWhat types of general impact would you say the X programme has had on communities or groups outside the organisation?
ECONOMICWhat types of general impact would you say the X programme has had on the economy in your region ? e.g. £ investment in geographic region, unemployment rates in area
HEALTH AND WELFAREWhat types of general impact would you say the X programme has had on the health or welfare of the community in your region? e.g. no. of deaths from heart disease, alcohol-related illnesses in local population
EDUCATIONALWhat types of general impact would you say the X programme has had on the education levels of the community in your region ? e.g. level of qualifications of local population, % social inclusion of minorities
LAW AND ORDERWhat types of general impact would you say the X programme has had on law and order issues in the community in your region? e.g. no. of robberies in the area, % reduction in drug crime
ENVIRONMENTALWhat types of general impact would you say the X programme has had on the geographical environment of your region? e.g. pollution and waste levels in the region
Has taking part in X had any other impacts on the local or national community that we haven’t discussed yet?
IDENTIFYING BARRIERS TO CHANGE IN SOCIETAL OUTCOMES
Has anything in particular stopped or hindered your organisation’s impact on the community or society in the way you wanted after the training / taking part in Programme X?
Is there anything that could be changed about Programme X, your organisation or other aspects that could help you achieve these desired changes?
13
.
NOTE 1: The above questions can be tailored to different stakeholders (trainees, line
managers, senior managers, customers, regulatory bodies etc) and different types of
programmes.
NOTE 2: Where desired changes have not occurred or undesirable changes have occurred,
then probe for the reasons for this situation (see BARRIERS question). Identifying
barriers to desired change in outcomes can help improve the effectiveness of future
activities.
© K.Birdi, University of Sheffield 2006
14