Managing Learning and Knowledge CapitalHuman Resource Development:

Chapter 11

Evaluation

Copyright © 2010 Tilde University Press

Misconceptions about evaluation • Evaluation is a control system

– Likely pay-off balanced against the cost

– Costs unlikely to surface until a time in the future

– Such risks will eventually translate into facts, and the dues will then have to be paid

• Most of the evaluation techniques discussed in this chapter apply to the legitimate system

• Some effort to evaluate development in the shadow system has to be made

2

Need for evaluation• All HRD interventions will be evaluated anyway,

either formally or informally • Legal imperatives • Essential to the survival of the HRD function• Role of the evaluation

– Measure what change has occurred– Improve the other three stages of the HRD system– See if the change is attributable to the learning episode– See if the change was worthwhile

3

Assessment of learning• Gathering, interpreting and describing of information

about learner achievement• Constructive alignment – see Figure 11.1• Cannot directly observe what learning has occurred:

– So, most assessment procedures ask the learner to perform a behaviour

– It is this behaviour that is then measured.

• Several potential weaknesses. For example: – The behaviour is only a sample of the total learning – The behaviour represents only explicit knowledge– Measurement vs Indicator

4

Types of assessment

• From easiest to most difficult– skills testing

– objective written

– subjective written

– performance tests

– learning diaries

– analytical critiques

– portfolio assessment

5

Assessment and the HLO• Constructive alignment

– Relationship between the learning outcomes, the learning strategies of choice, and the assessment types of choice

• Figure 11.3 provides a logical starting point for deciding on appropriate assessment types.

• Initial decision may be adjusted – Can be a range of complexity within each of the assessment

types – Tend to use a number of assessments in combination– Authentic assessment – The characteristics of the learner – Any issues raised in the HRDNI

6

Self-assessment

• Self-assessment is the best and richest form of assessment

• Need to develop the learners to such an extent that they value their own self-assessment above all else

• The learner finally becomes an independent learner

7

The HR developer’s dilemma • Skills tests

• Reflect a simple situation • Easy to show reliability and• However, the observed behaviours of skills are blurred shadows

of the quality that is being assessed

• Portfolio assessment, depends on the honesty of the learner• Major strength when the development of the learner is the main

objective (i.e., encourages self-assessment)– When assessment is for other purposes (e.g. monetary gain for the

learner), the use of the more complex assessment types becomes more problematical

8

The meaning of scores• A raw score - when assessment is gauged using a

quantitative figure– What does the raw score mean?– Criterion‑referenced scoring– Norm-referenced scoring

• Use of scores– Formative assessment

– Summative assessment

9

Use of assessment

• Feedback– To the learner

– To the HR developer

• Evaluation– One of the levels of evaluation

10

Kirkpatrick’s four levels• Reaction

– Reactions of the learners to the learning episode– Usually measured with a questionnaire - happy sheets?

• Learning – discussed previously

• Behaviour– Change in on-the-job behaviour– Use performance appraisal process

• Results – Impact of the learning episode on the organisation as a whole– Tangible indicators (e.g. improved profits)– Success Case Method

11

The presage factors• Stage I: evaluate needs and goals• Stage II: evaluate HRD design• Stage III: evaluate implementation• Stage IV: evaluate learning - similar to Kirkpatrick

• Stage V: evaluate usage and endurance of learning - similar to Kirkpatrick

• Stage VI: evaluate payoff - similar to Kirkpatrick

• Could also add another – evaluate the HR developer• SEE Table 11.1 for a discussion on the eight (8) levels

of evaluation

12

Review of the four roles of evaluation

• The 8 levels of evaluation achieves first two:– Identify what change has occurred

– Improve the other three stages—investigation, design and implementation—of the HRD system

• Need to go further to achieve the last two:– See whether the change is attributable to the

learning episode – scientific model

– See whether the amount of change was worthwhile – cost benefit analysis

13

Scientific Model

• Based on the experimental methods • In order from the simple and less costly to

the complex and more costly– Post-test

– Pre-test–post-test

– Time series

– Control group

– Solomon four

• A complex decision

14

Cost benefit Analysis• Identify the costs, in dollar terms

– Relatively straight forward– Cut off points? – e.g. apportioning electricity charges

• Identify the benefits accruing in dollar terms– Sometimes easy – e.g. reduction in accidents

– Often difficult – e.g. value of change at the meta-ability level in a learner

– Often use a ‘shadow value’ – e.g. promotions individual achieves

– Cut off points?

• The ratio of costs/benefits – Should be in favour of the benefits

15

The evaluation plan• Starts during the design stage• Should include

– Develop the assessment of learning first– Add further assessment of learning as required for the evaluation– Decide what presage variables will be evaluated and when– Review the investigating instruments and incorporate into plan– Design daily and overall course (or workshop) reaction sheets – Plan the scientific pre-test and post-test instruments, if needed. – Identify the methods for the behaviour and results levels – Decide if a cost–benefit analysis is to be used and plan for it– Prepare a budget for the evaluation plan – Send the evaluation plan to staff who are affected

16

The evaluation report• Should at least include the following

– Executive summary– Findings/recommendations section– Table of contents– Main body

• reasons for the evaluation• list of the personnel involved • discussion of - types of evaluations used and how the data

was collected and analysed• discussion of the findings, • list and a discussion of the recommendations

– Appendices

17