Manual Training Needs Assessment and Training Outcome … · 2020. 11. 19. · raining Needs...

ManualTraining Needs Assessment and Training Outcome EvaluationIn an Urban Context

1

5

ASSESSMENT

EVALUATION

• Needs• Effectiveentrypoints• Buildonprevious

lessonslearned

• Event,learningandperformance

• Documentlessonslearned

• Trainee’smotivation• Barriers&support

Partners

TRAINING NEEDS ASSESSMENT

Beneficiaries

ManualTraining Needs Assessment and Training Outcome EvaluationIn an Urban Context

Manual Training Needs Assessment and Training Outcome Evaluation In an Urban Context

FirstpublishedinNairobiin2012byUN-Habitat.Copyright©UnitedNationsHumanSettlementsProgramme2012

AllrightsreservedUnitedNationsHumanSettlementsProgramme(UN-Habitat)P.O.Box30030,00100NairobiGPOKENYATel:254-020-7623120(CentralOffice)www.unhabitat.org

HS/083/12EISBN:978-92-1-132497-6

Disclaimer

Thedesignationsemployedandthepresentationofthematerialinthispublicationdonotimplythe expressionof anyopinionwhatsoever on thepart of the Secretariat of theUnitedNationsconcerningthelegalstatusofanycountry,territory,cityorareaorofitsauthorities,orconcerningthedelimitationofitsfrontiersofboundaries.

ViewsexpressedinthispublicationdonotnecessarilyreflectthoseoftheUnitedNationsHumanSettlementsProgramme,theUnitedNations,oritsMemberStates.

Excerptsmaybereproducedwithoutauthorization,onconditionthatthesourceisindicated.

Coverphoto©UN-Habitat

Acknowledgements

TheCapacityDevelopmentUnitofUN-Habitat isgratefultoalltherepresentativesfromtraininginstitutionsreviewinganearlydraftofthismanualduringaworkshopinNairobi,April2011,andHansTeerlinkattheInstituteforHousingandUrbanDevelopmentStudies(IHS)whocontributedsubstantivelytothesamedraft.

Mainauthors: HannaHasselqvist,LauraThomas

Projectcoordinator: CynthiaRadert

Contributors: ClaudioAcioly,BernhardBarth,FernandoCabreraDiaz, JoeHooper,BridgetOballa,HansTeerlink

Peerreviewers: BenjaminAddo(DeutscheGesellschaftfürInternationaleZusammenarbeit,GIZ),AymanEl-Hefnawi(HousingandBuildingResearchCenter,HBRC,Egypt),SamuelGithaiga(Training ManagementConsultant,Kenya),ReinhardGoethert(MassachusettsInstituteofTechnology,MIT),AndreHerzog(WorldBankInstitute),YuHungHong(LincolnInstituteofLandPolicy),OheneSarfoh(InstituteofLocalGovernmentStudies,Ghana)

Designandlayout: HannaHasselqvist

Contact Information

URL:http://www.unhabitat.org/tcbb

CONTENTSList of Appendices ...................................................................6

List of Figures ..........................................................................6

Glossary ...................................................................................7

Preface.....................................................................................8

Introduction .............................................................................9

Training and Capacity Development .........................................9

The Training Cycle ....................................................................9

Determining the Extent of the Assessment and Evaluation ....10

Engaging the Stakeholders .....................................................10

5. Introduction to Training Outcome Evaluation ...................25

5.1. Proving the Value of Training ...........................................25

5.2. Levels of Evaluation ........................................................25

5.3. Meeting the Challenges of Evaluations ...........................26

6. Managing the Evaluation Process .....................................27

6.1. Planning the Evaluation ..................................................27

6.2. Mobilizing Resources ......................................................28

6.3. The Evaluation Team ........................................................28

6.4. Reality Check: Do Benefits Outweigh Costs? ...................29

7. Evaluating Events and Participant Reactions ....................30

7.1. Participant Reaction Affects Learning ..............................30

7.2. Training Content .............................................................31

7.3. Training Delivery .............................................................31

7.4. Training Support ..............................................................32

8. Evaluating Individual Learning .........................................33

8.1. Why Evaluate Individual Learning? .................................33

8.2. What is Learning? ............................................................34

8.3. Methods for Evaluating Individual Learning ...................34

9. Job Performance Outcomes and Organizational Performance ..........................................................................36

9.1. Why Evaluate Job Performance? ......................................36

9.2. Methods for Evaluating Job Performance ......................37

9.3. Individual and Organizational Level Performance Barriers ...................................................................................37

9.4. Evaluating Organizational Performance and Change .....37

10. Learning from the Training Outcome Evaluation .............39

1. Introduction to Training Needs Assessment .....................13

1.1. Assessing Needs and Capacities ......................................13

1.2. Approaches to Assessment ..............................................14

1.3. Making Sure Training is Relevant ....................................14

2. The Training Needs Assessment Process ............................15

2.1. Verifying Demand and Client Commitment .....................16

2.2. Identifying Key Stakeholders ...........................................16

2.3. Identifying Desired Capacity ............................................16

2.4. Identifying Current Capacity ............................................16

2.5. From Current to Desired Capacity ....................................17

2.6. The Key Question: Can Training Contribute? ....................17

2.7. Specifying Training Needs ................................................18

3. Collecting and Analyzing Data ..........................................19

3.1. Sources of Information ....................................................19

3.2. Data Collection Methods .................................................20

3.3. Analyzing the Data ..........................................................21

4. Bridging the Assessment and Evaluation through Learning Objectives ..............................................................................22

4.1. What are Learning Objectives? ........................................22

4.2. Writing Effective Learning Objectives ..............................23

4.3. Learning and Behavioural Change ..................................23

4.4. Translating the Learning Objectives into Training Design ....................................................................................23

PART 1: TRAINING NEEDS ASSESSMENT PART 2: TRAINING OUTCOME EVALUATION

MANUAL TRAINING NEEDS ASSESSMENT AND TRAINING OUTCOME EVALUATION6

LIST OF APPENDICES

DATA COLLECTION METHODS & TOOLS

APPENDIX 1A: Interview Guidelines ...................................................................................................41

APPENDIX 1B: Competency Interview Template .................................................................................43

APPENDIX 2A: Group Interview Guidelines (Focus Groups) ...............................................................45

APPENDIX 2B: Example of Focus Group Discussion ............................................................................46

APPENDIX 3: Observation Guidelines .................................................................................................47

APPENDIX 4A: Questionnaire Guidelines ............................................................................................48

APPENDIX 4B: Example of Pre-Training Questionnaire .......................................................................49

APPENDIX 4C: Post-Training Evaluation Sheet....................................................................................53

ANALYSIS METHODS & TOOLS

APPENDIX 5: Problem Tree Analysis ....................................................................................................55

APPENDIX 6: Five Whys .......................................................................................................................56

APPENDIX 7: Stakeholder Analysis .....................................................................................................57

APPENDIX 8: SWOT Analysis ...............................................................................................................58

APPENDIX 9: KJ Analysis .....................................................................................................................59

EVALUATION METHODS & TOOLS

APPENDIX 10: Learning Contract .......................................................................................................61

APPENDIX 11: Performance Contract .................................................................................................62

APPENDIX 12: Results Chain ..............................................................................................................64

APPENDIX 13: Follow-Up Plan ............................................................................................................65

APPENDIX 14: Five “W” Interview .....................................................................................................67

APPENDIX 15: Learning Improvement Plan .......................................................................................67

APPENDIX 16: Job Shadow Performance Testing ...............................................................................69

APPENDIX 17: Action Plan ..................................................................................................................69

APPENDIX 18: Outcome Mapping ......................................................................................................71

LIST OF FIGURES

FIGURE 1: The training cycle .................................................................................................................9

FIGURE 2: The main stakeholders in training .....................................................................................11

FIGURE 3: The training needs assessment process .............................................................................15

FIGURE 4: Possible sources for data collection ..................................................................................19

FIGURE 5: The learning objectives are key in all stages of the training process following the assessment ........................................................................................................................................22

FIGURE 6: Four levels of evaluation ...................................................................................................25

FIGURE 7: The training outcome evaluation process .........................................................................27

FIGURE 8: Who to involve in the evaluation team?............................................................................28

FIGURE 9: Checklist to support decision on whether or not to conduct a training outcome evaluation..........................................................................................................................................29

FIGURE 10: The role of evaluating participant reaction in a training outcome evaluation ...............30

FIGURE 11: The role of evaluating individual learning in a training outcome evaluation .................33

FIGURE 12: The role of evaluating job performance in a training outcome evaluation.....................36

FIGURE 13: The role of evaluating organizational performance in a training outcome evaluation..........................................................................................................................................38

FIGURE 14: The action learning cycle .................................................................................................39

7GLOSSARY

GLOSSARY

Capacity Development: Theprocesswherebypeople,organizationsandsocietyas a whole unleash, strengthen, create, adapt, andmaintaincapacityovertime.1

Capacity Assessment: Acapacityassessmentidentifiescapacityonthreelevels;individual,organizational andenablingenvironment. Itisamorecomprehensiveapproachthantrainingneedsassessmentasitalsolooksattheinterrelationbetweeneachoftheselevelsofcapacity.

Impact: Thepositiveandnegative,primaryandsecondarylong-term effects produced by a development intervention,directlyorindirectly,intendedorunintended.2

Learning: Learningisdirectlyrelatedtoincreasedknowledgeandunderstanding; improved technical, mental or socialskills;orchangesinattitudesorvalues.

Learning Objective: Learningobjectivesclearlydefinewhatthegoalof thetrainingis:whatthetraineesareexpectedtolearnfromthetrainingandbeabletoachieveafterthetraining.

Outcomes: The likely or achieved short-term and medium-termeffectsofanintervention’soutputs.3

Outputs: The products, capital goods and services which resultfrom a development intervention; may also includechanges resulting from the intervention which arerelevanttotheachievementofoutcomes.4

Training: Thepassingofknowledge,skillsorattitudes fromonepartyorindividualtoanother.

Training Needs Assessment: A training needs assessment looks specifically at theskills, knowledgeandattitudesofpotential trainees. Itusesthisinformationtodetermineifandhowtheissuecanbeimprovedbytraining.

Training Outcome Evaluation: A training outcome evaluation is carried out to see iftheobjectivesof the traininghavebeenmetandhowthe training process can be improved. Results can bemeasuredonfourlevels:theeventandtheparticipants’immediate reactions, the participants’ learning, theparticipants’ job performance, and the organizationalperformance.

Transfer of Learning: Theapplicationoflearningtotheworkplace.

1 Organisation for Economic Co-operation and Development/The Development Assistance Committee (OECD/DAC), The Challenge of Capacity Development: Working Towards Good Practice, DAC Guidelines and Reference Series, Paris, 2006. Available at http://www.oecd.org/document/7/0,3746,en_2649_33721_38549470_1_1_1_1,00.html (accessed 2012-05-23)2 OECD/DAC, Glossary of Key Terms in Evaluation and Results Based Management, 2002, p. 24. Available at http://www.oecd.org/dataoecd/29/21/2754804.pdf (accessed 2012-05-23)3 ibid, p. 28.4 ibid, p. 28.


PREFACE

Half of theworld’s population already lives in cities.Manycities in developing countries will continue to experiencerapidgrowthrates. Inordertomanageurbandevelopmentchallenges – which will be exacerbated by this growingpopulationpressure,climatechangeandongoingdevolutionof power – the need to further develop the capacities ofcities to address these will also grow. Within its mandateto promote socially and environmentally sustainable townsand cities UN-Habitat collaborateswith and supports LocalGovernment Training Institutions. These institutions arekeyactors indeveloping thecapacitiesof thecities in theircountriesorregions.

Training is one of the most common types of capacitydevelopment activities, with millions of dollars spent on iteach year. These interventions aim to change and improvetheindividualandtheirorganization’sperformance.Trainingneedsassessmentandtrainingoutcomeevaluationarecrucialstepsof the training cycle toensure this change.However,whenfinancialresourcesarescarceassessmentandevaluationareoftenamongthefirststepstobecutback,inparticularastheyareperceivedasrathercomplicated.Theintentionis,ofcourse,tosavemoney,buttheeffectmightverywellbethe opposite, that resources arewasted on training that isnotrelevantandprovidesnoverifiableresults.Additionally,atthistimeofglobalfinancialcrisis it is increasinglyimportantforallstakeholders intrainingtobeabletoprovevalueformoneybyshowingthatthetraininginvestmentsarepayingoff.Wecannotaffordtospendtimeandresourcesontrainingthatdoesnot improvetheworkof theorganizationsbeingtrained.Consequently, all actorswishing to contribute toasustainable urban development should engage in trainingneedsassessmentandtrainingoutcomeevaluation.

This manual provides the basic theory of, and affordablemethods for, conducting training needs assessment andtraining outcome evaluation along with concrete tips and

tools to support their practical application. It builds on theUN-Habitat practice note Training for better cities, whichpresents cutting edge theory and useful tools on trainingfor capacity development. While Training for better citiesconcernstheentiretrainingprocess,thismanualfocusesandexpands on needs assessment and outcome evaluation oftraining.Inaddition,furtherelaboration,detailsandpracticalexamplesconcerningdifferentapproachestotrainingneedsassessment, modes of training, prioritization of traininginterventionsandexamplesoftrainingoutcomeevaluationscanbefoundonthewebsiteoftheInstituteforHousingandUrbanDevelopmentStudies(IHS),http://www.ihs.nl/research/ihs_publications/ihs_manuals/.

Therationalebehindcreatingonemanualforbothassessmentandevaluation,insteadoftwoseparatemanuals,isthatthetwo steps are closely linked and consequently the qualityof onegreatly affects theother. Trainingneeds assessmentproduces learningobjectives for the training,while trainingevaluationmeasureshowwelltheoutcomesofthetrainingmatchthesesameobjectives.Ideallytheevaluationalsofeedsbackintothenexttrainingcycletoimprovetheprocess.

This manual is primarily intended for use by traininginstitutionsprovidingcapacitydevelopmentactivitiesrelatedto sustainable urban development, although other actorsmay find it useful as well. Local governments and otherorganizations who are recipients of training could benefitfrom using this manual to assess internal needs beforeengagingwithtraininginstitutions.

This manual is an output of the project: Enhancing the Contribution of Local Authorities and their Partners towards achieving the MDGs through Capacity Building of Local Government Training InstitutesfundedbytheUnitedNationsDevelopmentAccount(7thTranche).

9INTRODUCTION

training cannot not make any substantial change, such aswhen organizational performance is constrained by thestructureoftheorganization,localandnationalpriorities,orlegislationandpolicies.

The Training CycleWhentraining isconsidered tobeaneffectiveapproachtotheidentifiedproblemadequateassessment,design,delivery,follow-up and evaluation of the training are all crucial toensure positive outcomes. These are the five basic stepsof the training cycle. Although this manual covers mainlyneedsassessmentandevaluationof training it is importanttounderstandhowtheyarelinkedtotheotherstepsinthetrainingcycle.

Duringtheassessmentthecontextandneedsofthetraineesand their organization are studied and objectives for thetrainingaresetaccordingtowhattheorganizationwishestoachieve.Thedesignofthetrainingbuildsonthefindingsoftheassessmentandthedefinedlearningobjectives.Trainingdelivery is the conduct of training, usually in the form ofoneormoreevents.Follow-upisconductedtomonitorandassist trainees post-training as they implement what theyhave learned. Finally, evaluation is carriedout to see if theobjectivesofthetraininghavebeenmetandhowthetrainingprocesscanbeimproved.

INTRODUCTION

Training and Capacity DevelopmentTo understand the important roles of training needsassessment and training outcome evaluation in capacitydevelopment,theroleoftrainingfirsthastobeclear.Trainingin the context of capacity development is the passing ofknowledge,skillsorattitudesfromonepartyorindividualtoanother. It isaplanned learningprocessaiming to improvethe competency of the trainees; so as to positively changetheirbehavioursandprovidebenefits to theirorganizationsandotherstakeholders.

Most positive examples of training involve the learning oftechnicalskills,where it iseasytoseeandexperimentwithhow the learning can be applied. However, it is importanttonotethattraining isnotsuitable inallsituations. Infact,therearemanycaseswheretrainingisnotatalleffectiveinbuildingcapacity.Oneofthebestskillsatrainercanhaveisunderstandingthelimitationsoftraining.

Successfultrainingrequiresthesupportofthetraineesaswellastheirorganization,andtimeinvestedbyboth.Inaddition,the training needs assessment may identify capacity issuesthattrainingalonecannotaffect.Forexample,conductingaleadershiptrainingforemployeesofalocalgovernmentmayresultinapooruseofresourcesandlittlechangeifemployeeslackmotivationbecausetheyfeelundervalued.Toaddresssucha situationwould require a different capacity developmentapproach. There are also more complex situations where

FIGURE1:The training cycle

1

2

34

5

ASSESSMENT

DESIGN

DELIVERYFOLLOW-UP

EVALUATION


lessonslearned

• Actionorientedobjectives

• Involvemanagers• Performanceaids

• Approaches&tools• Process• Fittoclient’sneeds



• Targeting&relevance• Addressingbarriers• Follow-upactivities

Activitiesforproviders,traineesandmanagers

Elementsforeffectivetraining

Elementsfortransferoflearning

Coveralltransferfactors,e.g.howtoovercomebarriers


• Quality• Relevance


However,forthetrainingtopositivelyaffectthetrainee’sworkanotherdimensionhastobeaddedtothetrainingprocess.Theapplicationoflearningtotheworkplaceisusuallyreferredtoas transfer of learning5, andeach stepof the trainingprocessshouldbecarriedoutinawaythatsupportstransferof learning. This includes, for example, identification ofpotentialbarriersandenhancerstotransferoflearningwithinthe organization during the assessment, addressing thesebarriers and enhancers in thedesign, building in follow-upactivities,keepingthetraineesmotivatedduringthedelivery,andinvolvingthetrainees’managersintheprocess.

The traditional linearor cyclicprocessof traininghas tobeappliedwithflexibility.Itmightbenecessarytoloopbacktothe design phase during the delivery to adapt the trainingto changed circumstances, follow-up activities may run inparallelwiththeevaluation,andtheevaluationshouldhaveitsstartingpointalreadyintheassessment.

Manydifferentmethods,theoriesandtoolsexisttoassistineachstepofthetrainingprocess.Whilethismanualfocuseson assessment and evaluation of training, there are manyother sources of information for those who wish to digdeeperintothedesignanddeliveryoftrainingprogrammes.Formore informationonsomeofthese,pleaserefertotheUN-HabitatpracticenoteTraining for better cities6.

Determining the Extent of the Assessment and EvaluationThe extent of the needs assessment and evaluation has tobe determined for each training programme. For a short,low-budgettrainingprogrammeitmaynotbenecessaryorfeasibletostartwithabroadinitialinformationgatheringorengaginginevaluationonanorganizationallevel.Themorelarge-scalethetrainingprogrammeisthemoreextensivetheassessment and evaluation have to be. As aminimum theassessment should identify barriers to transfer of learningand factors that might prevent the training from havinganyresultatall.Thepotential trainingparticipants’existingknowledge,skillsorattitudesalsoneedtobeidentified.Forevaluationeverytrainingshouldincludeatleastevaluationoftheparticipants’reactionstotheeventandwhattheylearnedfromthetraining.

Thereisanongoingdebateonhowfarevaluationscanandshouldbetaken.Thismanualfocusesonhowtodeterminetheoutputsandoutcomesof training, insteadofgrapplingwiththetheoryofwhetherornotitresultsinimpactsatacity,regionalornationallevelinthelong-term.

Impact is, according to the OECD/DAC glossary of 2002,the“positiveandnegative,primaryandsecondarylong-termeffectsproducedbyadevelopment intervention,directlyorindirectly, intended or unintended.”7 Many agencies anddisciplines struggle to find effective ways to evaluate theimpactofcapacitydevelopmentprocesses,includingtrainings,at such grand levels. Furthermore, the potential impact ofa training programme can be affected by many externalvariables,whichoftenmakesitimpossibletodistinguishwhatisadirectresultofthetraining.

5 For a summary and guide to transfer of learning, see M. Taylor, Transfer of Learning: Planning Workplace Education Programs, Human Resources and Social Development Canada, 1997. Available at http://www.nald.ca/library/research/nls/inpub/transfer/cover.htm (accessed 2012-05-23)6 Available at http://www.unhabitat.org/pmss/listItemDetails.aspx?publicationID=3350 (accessed 2012-07-10)7 OECD/DAC, 2002, op. cit., p. 24.

Outputs are defined as “the products, capital goods andserviceswhichresult fromadevelopment intervention;mayalso include changes resulting from the interventionwhicharerelevanttotheachievementofoutcomes.”8

Outcomes are “the likely or achieved short-term andmedium-termeffectsofanintervention’soutputs.”9

Therearealwaysoutputsandoutcomesoftraining,intendedor not, and evaluating these provides useful insights fortraininginstitutionsaswellastheirclients.

Engaging the StakeholdersThetrainingprocesshasseveralstakeholders.Itisimportanttomakecleartheroleofeachofthem,whattheirinterestinthetrainingis,andthereasonsforengagingtheminboththeassessment and evaluation. For example, each stakeholdercontributing financially to the training will benefit fromtakingpartintheevaluationbyknowingwhattheinvestmentresultedin.

8 ibid, p. 28.9 ibid, p. 28.

A local government is aiming to reduceunemployment and key individuals representingthe local government are sent to participate in ajob creation trainingwhere they learn ofmethodsto stimulate the labourmarket. This results in thefoundation of a business incubator, supportingstart-up companies to grow. After three years theunemployment inthecity issignificantly lower,butwhetherornotthisisaresultoftheparticulartrainingis difficult, costly and perhaps even impossibleto determine. There are many other factors thatmight have had an impact on employment, suchas increased global financial stability, new nationallawsmaking it easier for companies tohire,andamultinationalcompanyestablishingabranchinthecity. However, the outputs and outcomes of thetraining interventions,whatthetrainingparticipantlearnedaboutjobcreationandhowitaffectedtheirwork,canstillbemeasured.

AN EXAMPLE OF THE DIFFICULTIES IN EVALUATING IMPACT ON A CITY LEVEL

11INTRODUCTION

Training Institutions: Although training institutions areproviders of training their role should not be seen as one-way suppliers.Assessment andevaluation are important tocreatedemanddriventrainingprogrammesthatarerelevantandofhighquality,andtheyalsocontributetoimprovingtheservicesofthetraininginstitutioninthelong-term.

Client: Depending on the type of training institution,clients can be national or local governments, NGOs, otherorganizations, private sector actors, or a combination ofthese.Theclientinvestsintraining,financially,throughideasandintheformofstafftime.Theywillneedtheassessmenttoknowiftrainingistherightinvestmentandwhattypeoftrainingwouldbevaluable.Evaluationiscrucialfortheclienttoknow if the traininghadthedesiredresultandtoknowhowtoinvestinfuturetraining.Althoughthewordclientisbeingusedinthismanual,theclientshouldnotbeapassiverecipientintheprocessbuthavethemoresuperiorroleofapartnerinvolvedinallsteps.

Trainer:Thetrainerbasesthetrainingdesignontheresultsoftheneedsassessmentandcan,bywayofevaluation,receivefeedbackonhowto improvethedesignof future trainingsandtrainingtools.

Training Participants/Trainees: Since the participants aredirectly involved in the training they are core stakeholders.Theyhave anunderstandingof theneeds for training, canassist in defining training objectives, can provide feedbackontrainingdeliverytoboththetraininginstitutionandtheirorganization,theclient.Theyarealsoanimportantsourceofinformationtoevaluatethelong-termresultsofthetraining.

Funding Agencies:Ifthetrainingiswhollyorpartlyfundedbyexternalparties,suchasdevelopmentpartnersornationalor local government bodies, they will need assessmentresults to be convinced that training is a wise investment.Theevaluationisnecessarytoshowresults.Iftheoutcomeispositive,evaluationscanalsobeusefultoconvinceagenciestoinvestinfuturetrainingprogrammes.

Beneficiaries of the Client’s Services: In the context oftrainingalocalgovernment,thecitizensarethebeneficiaries,astheyutilizetheservicesprovidedbythelocalgovernment.Beneficiariesarerarelyconsideredintheplanning,designandevaluationoftrainingprogrammesbutcanbeausefulsourceofinformationbothforassessmentandevaluation.Theycanprovide their opinions on the quality of the services, bothbeforeandafterthetraining,althoughitisimportanttonotethattheycannotdeterminewhetheranychangeinqualityistheresultoftrainingorotherfactors.

FIGURE2:The main stakeholders in training

$ $

$

TRAINER

TRAINING INSTITUTION (Training provider)

BENEFICIARIES (Citizens)

CLIENT

TRAINEES

FUNDING AGENCY

Training Services

PART 1:Training Needs

Assessment1

13INTRODUCTIONTOTRAININGNEEDSASSESSMENT

1INTRODUCTION TO TRAINING NEEDS ASSESSMENT

1.1. Assessing Needs and CapacitiesConductingatrainingneedsassessment isanessential firststep in theplanningofaneffective trainingprogramme.Atraining needs assessment usually focuses on current anddesired skills, knowledge and attitudes of the trainees.However, before specifying the training needs on anindividualandorganizationlevelitmaybenecessarytocarryout broader initial information gathering on the capacitiesof the organization and its environment. Analyzing thecurrent capacities, strengths, and weaknesses of each,the needs assessment is able to provide a solid base ofinformationwhichcandirectthefocusofthetrainingneedsassessment. It is commonto focusonlyon theweaknessesandgapsincapacity.However,byalsoidentifyingstrengthsand opportunities of the organization, its employees, andenvironment,amoreinformeddecisioncanbemadeonhowtrainingcanstrengthencapacityorifotherapproacheswouldbemoreeffective.

Capacity assessmentreferstoidentifyingcapacityon three levels; individual, organizational andenabling environment. It is a more comprehensiveapproach than a training needs assessment as italso looks at the interrelations between each ofthese levels of capacity. Capacity assessments arecommon when conducting many activities, suchas project design as well as training. Training needs assessment looks specifically at the skills,knowledge and attitudes of potential trainees. Ituses this information todetermine if andhow theissue can be improved by training. In thismanual,whichfocusesontrainingneedsassessment,crucialconcepts of capacity assessment are incorporatedto ensure that the training needs assessment iseffective. For example, it is suggested to carry outinitial information gathering onoverall capacityand to identify the potential barriers to trainingeffectiveness,beforemovingon tospecifying the training needs. This ensures that the trainingwillnotbeawasteofresourcesandthatitwillcontributetotheworkoftheorganization.

THE NEEDS ASSESSMENT TERMINOLOGY USED IN THIS MANUAL


1.3. Making Sure Training is RelevantTrainingisoftenseenasthesolutiontoallcapacityproblems,eventhoughtherearemanysituationstrainingalonecannotimprove. Hence, the first priority of a needs assessmentshould be conducting some initial information gatheringwhichisusedtodeterminewhethertrainingisanappropriateapproach;preventinginvestmentsintotrainingsthatwillnothelptheorganization.Whentrainingisdecideduponasthebest way to develop capacity, the specification of trainingneedsisessentialforidentifyingwhoshouldbetrained,whatthefocusofthetrainingshouldbe,andhowtodeliverthetraining.Thetrainerthenusesthisinformationtodevelopatraining programme that is relevant both to the individualparticipantsandtheirorganizations.

An understanding of the individual and organizationalconstraints and capabilities enables the trainer to devisepractical steps to get the trainee from their current stateto the desired post-training state. By identifying theseconstraints,capabilities,andpotentialenhancersandbarrierstothetransferoflearninginthebeginning,atrainingdesignanddeliverymethod tomaximize theeffectof the trainingcanbedeveloped.

1.2. Approaches to AssessmentTherearemanyapproaches toassessing thecapacityofanorganizationwithinitscontext.Theseapproachesmayincludespecific assessments, such as the Needs Assessment, GapAnalysis and Broad Capacity Assessment which have beenoutlinedbymanyinternationalactors,bothlargeandsmall.Otherapproaches,suchastheCapacityDevelopmentResultsFrameworkpromotedbytheWorldBankInstitute10,includedetailedstepsfromtheidentificationandassessmentphasesto the evaluation phase. These and additional approacheshavebeenwrittenaboutby theOrganisation forEconomicCo-operation and Development (OECD) in a DevelopmentAssistance Committee reference document11. Regardless ofwhichapproachischosen,itiscrucialthatitisappliedwithflexibilitytowardstheeconomical,cultural,politicalandsocialcontext. There isno singlemethod for capacityassessmentthatcanbefollowedstrictlystepbystep,withoutadaptationtothespecificsituation.

10 The World Bank , The Capacity Development Results Framework - A strategic and results-oriented approach to learning for capacity development, 2009. Available at http://siteresources.worldbank.org/EXTCDRC/Resources/CDRF_Paper.pdf?resourceurlname=CDRF_Paper.pdf (accessed 2012-05-23)11 OECD/DAC, 2006, op. cit.

2Training needs assessment as part of a national training strategy. © UN-Habitat

15

THE TRAINING NEEDS ASSESSMENT PROCESS

THETRAININGNEEDSASSESSMENTPROCESS

2Alltrainingactivitiesshouldbeinitiatedbyademand,whichforexamplecanbeproblemsorpotentialopportunitiesidentifiedby an organization or individuals within an organization.Demand can also be triggered by issues expressed bypoliticians or beneficiaries of the organization’s services.Training needs assessment has to be a flexible process butageneralguidanceistostartwithabroadassessmentthatgraduallybecomesmorespecific.Figure3showsthedifferentstepsofassessmentdescribedinthefollowingsections.Some

ofthestepscanbecarriedoutinparallel;however,itiscrucialtodetermineif,andtowhatextent,trainingcancontributetoachievingthedesiredcapacity(step6)beforespecifyingthetrainingneeds(step7).

Throughouttheassessmentprocessdatahastobecollectedandanalyzed.For an overview of the methods and tools, please see section 3, Data Collection and Analysis.

FIGURE3:The training needs assessment process

1 2 3

4

5

6

7

Starting point: Demand

Can training contribute?

Initial information gathering

Verify demand & client

commitment:

Identify key stakeholders:

Identify desired capacity

Specify training needs:

Identify current capacity

From current to desired capacity:

• Identifyrootofproblem

• Internal(client)• External

• What,howandwhomtotrain?• Barriersandenhancerstotransfer

oflearning

• Discuss&prioritizewithclient

IF YES


2.1. Verifying Demand and Client Commitment

Toensurethattheinitialinformationgatheringtakestherightdirectionfromtheverybeginning,thefirststepshouldbetoverify thedemandof the client.Has the client expressedawishtoaddressacertain issue? Iswhattheclientperceivesasbeingtheissuealsotheactualproblem?AProblem Tree Analysismaybehelpful toanswerthesecondquestion. Itillustrateslinkagesbetweenaseriesofproblems,andcanbeusedforidentifyingunderlyingproblemsandtheirextendedeffects,aswellasaidintheidentifyingofwaysinwhichtoaddresstheproblem.Anothermethodforfindingtherootofproblems is theFive Whys method,where it isaskedfivedifferenttimes,infivedifferentways,whytheproblemexists.See Appendix 5 for more details on how to conduct a Problem Tree Analysis and Appendix 6 for the Five Whys method.

Partoftheinitialinformationgatheringistodeterminehowcommitted the clientele are to the desired organizationalchangesrelatingtotheproblemorpotentialnewdirection:

• Willmanagementsupportandengageintheassessmenttoidentifypotentialtrainingneeds?

• Aretheywillingtocommitthenecessarytimeandresourcesforthetraining,andtosupportthetraineesinimplementingwhattheyhavelearned?

Effective training is similar to a partnership. Resources canbespentandtraineescanbetrained,butunlessthetraineeshavethespaceandmandatenecessarytoadoptthelearningintotheirorganizations, itcanbecomeverydifficultforanytraining-related results to be evident. Thus, the role of thetrainerandtheclientelecomplimenteachotherinsupportingthetraineesunderstanding,andlatertheapplication,ofnewskillsorknowledge.

In order to determine if the clientele, specifically themanagement, support the training, it is necessary thatthey thoroughly understand the problem or potential newdirectionoftheorganization.Notonlymusttheyunderstandthechangesthatarenecessary,buttheymustalsounderstandthe role that training has in contributing to achieving thenecessarychanges.

2.2. Identifying Key StakeholdersStakeholders are the individuals, groups or organizationsthatareinfluencedbyorhaveaninfluenceontheparticularsubject of the training needs assessment. They may haveimportantinformationfortheassessment,affecttheresultofthetraining,orbepotentialtrainees.Considerthefollowingindividuals, groups and organizations when identifying thekeystakeholders:

• Specificunitsoremployeeswithintheorganization

• Fundingagencies(e.g.internationalorganizationsorthenationalgovernment)

• Politicians

• Privatesectoractors(e.g.competitors)

• Partners

• Contractorsoftheorganization

• Beneficiariesoftheservicesoftheorganization

• Thetraininginstitutionandtrainers

A Stakeholder Analysis can help in deciding on whichstakeholders are most important to involve in the trainingneeds assessment. By considering what influence thestakeholders have on the subject of the assessment andtheirinterestinthetopicitcanbedeterminedwhothekeystakeholdersare.See Appendix 7 for further instructions on Stakeholder Analysis and an example of a completed analysis.

2.3. Identifying Desired CapacityItiscrucialthattheclientunderstandswhattheywant.Whyhavetheyapproachedthetraining institution,andwhatdotheywantas theultimategoalof the training?The trainermust not rely on assumptions. It is important to discuss,ask questions, discern, and define what results the clientis expecting. It is imperative that the trainer understandswhat theclientdesires toachieve.Howdoes theclient seeitself operating in an ideal state?What would the client’sidealordesiredcapacitybe?Whatistheclient’svision?Theanswerstothesequestionsmayberelatedtotheemployees,the organizational structure, various internal and externalprocesses,partnershipsorcommunication.

Havingaclearunderstandingoftheclient’sdesiredcapacitywill serve as a guiding framework enabling the trainer toensureall effortsof the specificationof trainingneedsandeventual training programme will contribute to the clientrealizingtheirdesiredcapacity.

2.4. Identifying Current CapacityTherearevariousdifferentapproachestoassessingcapacitywithinatrainingcontext.Therehasbeensomedebateaboutthe best way, particularly between use of a gap analysisorcapacity analysis.Whenconductingagapanalysis,thefocusisonidentifyingspecificallywhatlacksbetweenhowtheorganizationandemployeescurrentlyoperateasopposedtohowtheywouldoperateintheiridealstate.Whenconductinga capacity analysis, the focus is to identify every capacitythatcurrentlyexistsbyassessingwhattheorganizationandemployees already have and do (includingwhich processesare already in place). Thismanual acknowledges that both

Mapping of stakeholders and how they influence or are influenced by a project. © UN-Habitat/Hanna Hasselqvist

17THETRAININGNEEDSASSESSMENTPROCESS

approaches have valuable attributes and has includedelementsofbothinthetrainingneedsassessment.

The trainer must understand what knowledge the client,the employees, and their organization, already have aboutthetopicneedingtobeaddressed.Tobetterunderstandtheclientanditscontextthefollowingquestionscanbeasked:

• Istheclientaleadingorganizationwhomothersarelearningfromholdingspecificexpertisewithinacertainsector,oraretheystrugglingtoimprovetheirservices?

• Howdoestheclientperformrelativetotheircounterpartsinothercitiesandnations?

• Dotheemployeesknowwhytheorganizationconductsthingsthewaytheydo?Dotheyagreewiththewaythingsareconducted?

• Doesanoverarchingframeworkoflawsandpoliciesexistinrelationtotheworkoftheclient?Isitclearlyunderstoodbytheclient?

A SWOT Analysis can also be helpful at this stage. Themethodisusedtomapstrengths,weaknesses,opportunitiesand threats of anorganization. This information is used tounderstand how these factors play a role in the way theorganization performs. For instructions on how to carry out a SWOT analysis, see appendix 8.

2.5. From Current to Desired CapacityWith a definition of the problem, the initial informationgathered, and an understanding of both the current anddesiredcapacity,thetrainerhasasolidunderstandingofthecontext. It is now possible for the trainer to identify whatmustbechangedanddevelopedinordertobringtheclientfromitscurrentcapacitytothedesiredcapacity.Atthispoint,thetraineralsoknowswhatcapacitiesalreadyexistthatcansupportthisprocessofchange.

The desired capacity serves to guide the specification oftrainingneeds.Keepingthedesiredcapacityinmindwillhelpidentify the necessary areas of focus and decide onwhichdatatocollectandhowtocollectit.

It is important that the desired capacity is discussed andprioritizedwiththeclientbeforemovingontospecifyingthetrainingneeds.Thiswillensurethatthedatacollected,undertheguidanceofthedesiredcapacity,willcontributetowardsthe understanding of the problem or new direction of theclient’swork.

2.6. The Key Question: Can Training Contribute?

Withthe informationfromthe initial informationgatheringthe training institution and client have to determine iftraining is the best option to move the organization fromtheircurrenttodesiredcapacityandtowhatextenttrainingcancontribute.Istheissueoneofindividualknowledgeandskillsoroneofstructures,processes,andotherorganizationalelements such as the legal and political framework? If thecaseistheformer,anissueofindividualknowledgeorskills,thentrainingmaybeagoodinvestment.However,ifitisanissueoforganizationalstructuresorprocesses,itisverylikelythattrainingalonewillnotimprovethesituation.Askingtheabove question relatively early in the process prevents theclientfromwastingresourcesontrainingthatwillnothavea

positiveeffectontheorganizationandenablestheclienttoinvestwiselyintrainingactivities.

Training itself cannot change the way an organizationfunctions. Rather training may reveal new methods toindividuals, but it is up to the individual to apply themethods.Collaborated efforts are necessary for training tobe effective. This particularly includes having managementwhich iswilling to support the trainees as they implementnewmethods,andanorganizationalstructurewhichisopento change. As previouslymentioned, environmental factorssuchaspoliciesandlegislation,whichtheclientcannotaffect,have an influential role in establishingandmaintaining theorganization’s processes a certain way. These factors can,whenacting asbarriers,make it difficult for the trainee toapplytheirlearning;renderingthemunabletooperatewithintheirnewcapacity,andthetrainingawasteofresources.

This is not to say that training is useless. There is a timeaplace for training, and if it is understood properly trainingcanbeauseful tool for initiatingchange,particularly if thetrainingisdesignedtosupportthetraineesinbeingflexibleandinnovativeindealingwithinstitutionalconstraints.

Itisimportanttodiscussthelimitationsoftrainingwiththeclient and together decide on towhat degree training cansupport and help the client meet its desired capacity. Thisensuresthattheclientdoesnothaveunrealexpectationsofthe training,and that theyunderstand their role inmakingthe training effective in the long-term. Furthermore, it isimportanttounderstandwhatdesiredchangethetrainingcancontribute towards,asa single trainingmaynotcontributetowardsallthedesiredchangesorcapacities.

• Doestheclient,specificallymanagement,supportthetraining?

• Doestheclient,includingallaffectedemployees,understandthereasonsforthenecessarychanges?

• Ingeneral,istheclient,includingallaffectedemployees,supportiveofthechanges?

• Doestheclient’sorganizationalstructurehavethecapacitytochange?

• Doesthechangeaffectaworkingrelationshipwiththeclient’spartners?Ifyes,dothepartnerssupportthechange?

• Aretherefinancialresourcesforthetrainingandnecessarychangesrelatedtothetraining?

• Haveyouconsideredexternalfactors,suchasnationalregulations,corruption,orrapidchangesintheareaofwhichtheclientisoperating,thatwouldhinderthedesiredchangeandtrainingcannotaffect?

• Istrainingthemosteffectivewaytoacquirethedesiredcapacity?(Comparedtoe.g.redistributionofhumanresourcesinternally,hirestaffwiththedesiredcompetences,engageinaninternalorexternalmentorprogramme,orgoonstudyvisits.)

QUESTIONS TO DETERMINE WHETHER OR NOT TRAINING CAN CONTRIBUTE


2.7. Specifying Training NeedsIftheinitialinformationgatheringindicatesthattrainingcancontributetotheidentifiedcapacityneeds,thenextstepistocollectandanalysedataonthespecifictrainingneeds:what,howandwhom to train. The selection of participants iscrucialforthesuccessofthetraining.Consideraspectssuchas:

• Currentskills,knowledgeandattitudesofthepotentialtrainees

• Positionandmandatetoimplementthelearningintotheworkoftheirorganizations

• Motivationtoactivelyengageinthetraining

Foratrainingwithparticipantsfromseveraldifferentclientsitisofteneffectivetohaveatleasttwoparticipantsfromeachorganization–mixingcompetenciesandroles.Thiswaythetraineescansupportand learnfromeachotherwhenbackatwork,leadingtoatrainingthatwilleitheraffectdifferentparts of the organization or affect the entire organizationmoreholistically.

It shouldalsobeexploredwhatmode of training ismostsuitablefortheparticipantsandtheirorganizations:

• Whatdepthoftrainingisrequired–isaworkshopenoughordoesitrequireaMaster’sprogramme?

• Howdoesitaffecttheorganizationifkeypersonnelleavetheofficetogoontrainingforaweek?Isitbetterforeveryonetodedicateoneafternooneveryweek,duringonemonth,fortraining?

• Howwilltheparticipantsbestlearn?Whatpedagogyandmethodsshouldbeusedfortrainingdelivery?

• Howcantheparticipantsbesupportedinapplyingwhattheyhavelearnedintheirworkafterthetraining?

When specifying training needs it also important to detectpotential barriers and enhancers to transfer of learning.The following factors can act either as barriers (if they arelacking)orenhancers(iftheyarepresent):

• Managementinterestandunderstanding

• Supportandresourcestoimplementlearning

• Motivationandaspirationsoftrainees

• Basicknowledgeorskillsoftrainees(e.g.languageskills,computerskills)

Certain barriers can be overcome by good training designor completely avoided by taking preventive actions. Forexample, if the training requires employees with workingknowledgeinEnglishtoattend,thetrainermustspeakwiththeorganization to reinforce the importanceof this as thetrainingdategets closer inorder toavoidotheremployeesbeingsentintheirplace.

The need for urban health services in a certainmunicipalityhasgrownfastduetoasuddenincreasein population. Recent data published by an NGOshowsthatthemajorityofthepopulationinthepoorareas,intheoutskirtsofthecity,donothaveaccesstohealthcare,despitetheservicesbeingsubsidizedbythenationalgovernment.

Thehealthserviceprovidersknowthatthey,at themoment,cannotaffordtoexpandtheirfacilitiesandemploynewdoctors,andthattheirpatientscannotafford to paymore. They have united and chosentospeaktoatrainingprovidertodiscussideasthatmay effectively help them meet demand withoutnecessarily having to make extensive investments.Theypresenttheir ideasasanemployeetrainingintimemanagement,aninvestmentandtrainingwitha new electronic system to maximize efficiency inhandling patent histories, or training some of thedoctors to work as part of a mobile health clinicenablingthemtotreatpatientswhileidentifyingandreducingcommoncausesofsicknesses.

Allthese ideas initiallyseemstobegoodsolutions,however upon conducting a training needsassessment it is discovered that the main reasonwhy thepoorestpeopledonot receivehealthcareis that they do not seek help at the clinics. Thereis no public transport outside the city centre andthey cannot afford to pay for the transportationto the city centre where the hospital and clinicsare located. Consequently, only improving theefficiencybytrainingthehealthserviceprovidersintimemanagementwillnotimprovethesituation.Acombination of training to increase the efficiencywiththeestablishmentofamobilehealthclinicisamoreeffectivesolution.

HOW TRAINING CAN AND CANNOT CONTRIBUTE TO THE DESIRED CAPACITY

3

19COLLECTANDANALYZEDATA

COLLECTING AND ANALYZING DATA3

Information has to be collected and analyzed throughoutthe assessment. First, the initial information gatheringfocuses on identifying current and desired capacity usuallyin a broader organizational perspective. Later, the specifictrainingneeds are analysed. There are a rangeofmethodsavailable fordatacollectionandanalysis.Whatmethods tochoosedependsonthestageoftheassessment,whattypeofdata is tobecollected,and theavailable resourcesbothwithintheorganizationstudiedandtheinstituteconductingtheassessment.Mostlikelyamixofmethodsandsourcesofinformationwillberequired.

3.1. Sources of InformationWhencollectingdatatherearemanypossiblesources,suchasinternaldocumentation(forexamplepreviousassessmentsor annual reports), media, partners, beneficiaries andemployees. To maximize the accuracy of the assessment,it is recommended to use several of these options as theyall provide different perspectives on the work of theorganization. Internally, it is important to look at differentlevelsof theorganization,asmanagersand their staffmaynotagreeonwhattheneedsare.Alsounitsandindividualsthatarehighperforming,aswellasthosewithperformanceproblems,shouldbestudied.Itiscommontoonlyfocuson

FIGURE4:Possible sources for data collection

Media

External sources

Internal sources (the client)

Documentation & reports Managers & employees High & low performing

Partners


Beneficiaries


thepartsof theorganizationwherecapacity is lacking,butindividuals and unitswith high capacity can provide usefulinformationonwhatspecificcapacityiscontributingtotheirperformance.

3.2. Data Collection MethodsAssessing the needs of an organization inevitably meansconsulting people for information,which is a skill in itself.The main methods for studying people’s opinions, skills,attitudesandbehavioursare interviews,questionnairesandobservations,seeTable1foranoverviewofthemethods.

Interviewscanbecarriedoutindividuallyoringroupsandbestructured, with pre-determined questions, or unstructuredtodiscussa topicmore freely. Interviewingdifferentpeoplewithin an organization is a relatively quick way of gettingan overview of for example a problem situation or thecurrentcapacityoftheorganization.See Appendix 1a for interview guidelines and Appendix 1b for a template of a Competency Interview to be carried out with high performing employees.

Group interviews gathersanumberofpeopletodiscussapre-determinedtopic.Ifcarriedoutsuccessfully,themethodresults in answers built on many people’s opinions, whichis suitable for attaining an overview of the issues and thecontext.See Appendix 2a for group interview guidelinesand Appendix 2b for an example of a focus group discussion.

Observations are useful for understanding the work oftheorganization, suchas through the studyofbehaviours;particularly regarding technical skills. It is often difficult todescribehowataskisperformedandobservationscanhelpinidentifyingpeople’sactualbehaviours.However,observationsonlygiveinformationonwhatisdone,butitdoesnotexplainwhyitisdonethatway.Tobetterunderstandthesituationitisthereforerecommendedtocombineobservationswithotherdatacollectionmethods.See Appendix 3 for observation guidelines.

Questionnaires are a time efficient way of obtainingquantitative data and the method is particularly usefulwhen the respondents are geographically wide spread.Questionnairesareoftencarriedouttovalidate informationfrom interviews or observations and to collect quantitativedata on attitudes or knowledge of trainees, for exampleto determine how many employees have knowledge ona specific topic. See Appendix 4a for questionnaire guidelines andAppendix 4b for an example of a pre-training questionnaire.

Nomatterwhichmethodsareusedfordatacollection,theyhavetobecarefullyprepared.Howquestionsareformulatedin an interviewor questionnaire affects the responses. It isimportanttoavoidaskingleadingorguidingquestions.Also,rememberthatquestionsmaybemisinterpreted,especiallyinaninterculturalcontext,oravoidedbecausetherespondentfeels uncomfortable. It may prove helpful to not only askabouttherespondent’sownexperiencesbutalsoaskaboutskills,behavioursandattitudesofcolleagues.

DATA COLLECTION METHOD

MAIN TYPE OF DATA

STRENGTHS CHALLENGES

Interviews Qualitative/Quantitative

• Allowsfordeepunderstanding

• Flexible

• Interviewermightbiasanswers

Group Interviews/ Focus Groups

Qualitative • Participantstriggereachother

• Resultdependsongroupdynamics(dominantpersonalities)

• Interviewermightbiasanswers

• timeconsumingtoanalyze

Questionnaires Quantitative • Easytodistribute

• Anonymousresponsespossible

• Onlinetoolsavailableforautomaticcompilationofdata

• Openandclosedquestionstructurespossible

• Usuallylowresponsefrequency

• Questionsmightbemisinterpreted

• Openquestionsaretimeconsumingtoanalyze

Observations Qualitative/Quantitative

• Goodforstudyingactualbehavioursofpeople

• Providesinformationinitscontext(usefulwhenthecontextisunfamiliartotheassessors)

• Peoplemightchangebehaviourwhenbeingobserved

• Timeconsumingtoanalyze

TABLE1:Strengths and challenges of different data collection methods

21COLLECTANDANALYZEDATA

KJ Analysis is a method for analyzing qualitative datafrommanydifferentsources,and identifyingthekeypointswithoutloosingtheholisticperspective.Importantpiecesofinformationfrominterviews,questionnaires,observationsorprevious analyses are written on for example post-it notesandsortedaccordingtohowtheyrelatetoeachother.For more information on how to conduct a KJ Analysis, see Appendix 9.

TheanalysismethodsreferredtoinSection2,aswellastheKJanalysis,areoftenusefultodoingroupstodiscusshowresponsesorobservationsshouldbeinterpretedandwhatitmeans, forexample in termsofcapacityneedsorpotentialbarriers.Manyof themethods, suchasSWOTanalysisandproblemtreeanalysis,workswelltodoincollaborationwiththeclient.

Tofacilitatetheanalysisofquantitativedatatheresultscanbe compiled in a table format. Most online survey toolsautomaticallydothis.Itishelpfultovisualizethedataingraphsordiagrams,bothtobetterunderstandtheinformationandtobeabletopresentclearandconvincingresultstotheclientandotherinvolvedstakeholders,suchasfunders.

Furthermore,itisimportanttorememberthatbeingcarefullyprepared does not only refer to the structure of the datacollection method itself, but also about being culturallyappropriate. Thismaymean that there is a certainway ofaskingatypeofquestion,thatinterviewsshouldbeconductedby someone of the same gender as the interviewee, andthat cultural idioms andphysical behaviour are understoodproperly (such as whether to make or avoid direct eyecontact). In order to carry out an effective data collectionit is of utmost importance to understand and respect thelocalculture.Tosuccessfullyachievethis,itisoftenusefultoengagelocaltrainers,whoarefamiliarwiththeculture,bothfortheassessment,designanddeliveryoftraining.

3.3. Analyzing the DataTo transfer thecollecteddata intovaluable information thedatahastobeanalyzed.Somedatacollectionmethods,suchasvideo-recordedobservations,requiremoretimetoanalyzethantoconduct,whichhastobetakenintoaccountwhenplanningtheassessmentandbudgeting.

Using post-its to organize and analyze qualitative data. © Hanna Hasselqvist


BRIDGING THE ASSESSMENT AND EVALUATION THROUGH LEARNING OBJECTIVES4

4.1. What are Learning Objectives?Whendatahasbeencollectedandanalyzed,it isimportanttoestablishlearningobjectivesbeforemovingontotrainingdesign. Learning objectives clearly definewhat the goal ofthe training is. They definewhat it is that the trainees areexpected to learn throughout the training, and what thetraineeswillbeabletoachievepost-trainingthattheycouldnotbefore.Thelearningobjectivesusuallyrefertoincreasedknowledge, improved skills or changed attitudes, andmayreflectoneoracombinationofthese.

• Knowledge objectivesrefertolearningnewconcepts,rules,principles,andstrategies,suchasconceptsforurbanplanningorlandreadjustmentstrategies.

• Skill objectivescanbeeithertechnical,mentalorsocial.Technicalskillsincludeconstructingaroadorlearninghowtouseacomputerizedreportingsystem.Problemsolvinganddecisionmakingareexamplesofmentalskills,whileleadershipandcommunicationareexamplesofsocialskills.

• Attitude objectives referstothewaythetraineesperceiveandrespondemotionally,physically,verballyandintellectuallytoothers.

Duringthetrainingdesign,thelearningobjectiveswillguidetheprocessandmakesurethatallactivitiesarecontributingto learning that is relevant to the trainees. The learningobjectivesarealsocrucialwhendeliveringandevaluatingthetraining.Bycomparingtheoutcomesofthetrainingwiththelearningobjectivesitcanbedeterminedwhetherornotthetrainingwassuccessful.

FIGURE5:The learning objectives are key in all stages of the training process following the assessment

1.

2.

3.


TRAINING DELIVERY

TRAINING DESIGN

TRAINING OUTCOME EVALUATION

LEARNING OBJECTIVES

23BRIDGINGTHEASSESSMENTANDEVALUATIONTHROUGHLEARNINGOBJECTIVES

4.2. Writing Effective Learning ObjectivesIt is of utmost importance that the learning objectiveis established in conjunction with the client. They are asignificantstakeholderinthetrainingastheyaretheonesthatmust enable and support the traineesupon the conclusionof the training, to adopt the learning into the workplace.Thus, the objective must have the client’s full support;specificallyofthosewithinmanagement.Toensurethatthetrainees, theirmanagers and the training institution all areawareofandagreeon theobjectivesa learning contractandaperformance contractmaybehelpful.The learningcontract is a simpledocument, drawnupby the employeebeingtrainedandthetrainee’smanager.Itincludesasignedstatementbyeachpersonthatdescribescertainbehavioursthey commit themselves to in regard to the forthcomingtraining.Aperformancecontract isasimilardocument,butsignedbythetraineeandthetrainer.See Appendix 10 for an example of a learning contractandAppendix 11 for a performance contract.

The best learning objectives are clear and can easily bedeterminedwhetherornottheyhavebeenmet.ThismanualrecommendsthattheobjectivesfollowtheSMARTpattern:

• Specific

• Measurable

• Appropriate(intermsofculture:social,economical,political,etc.)

• Realistic(size,scope,etc.)

• Timeframed

Having clear objectiveswill enable the trainer, trainees andtheir workplace to knowwhether or not the training wassuccessful inachievingtheobjectives.Anexampleofawellwrittenlearningobjectivewhichisrelatedtoimprovedpublicrelationsskills,specificallyincommunicatingwiththemedia,isthefollowing:

“After the training, the participant will be able to answer probing questions by the media, on behalf of their respective employer, by responding with facts or simple and smart politically correct answers.”

For more examples of learning objectives, see section 8.3.

Objectivesineffectivetrainingarenotabouthowmanypeopleor who attended the training. Rather, they are specificallyrelatedtothematerialthetraineesarebeingtaught;meetingtheoverallpurposewhichistodeveloptheircapacitytobeabletoaccomplishtheirtasksinamoreeffectiveway.Havinga specific objectivewill not only guide thedirectionof thetraining design, but itwill also provide valuable criteria fortheevaluation.

4.3. Learning and Behavioural ChangeAclearandagreeduponunderstandingoflearningobjectivesbetween the training institution and client,will enable thetraining institution to develop a training programme thateffectively focuses on achieving the objectives within theclient’scontext.Thetrainingwillbedesignedspecificallyforthe trainees to clearly understand the concepts and theirapplication,aswellastosupporttheminchangingbehavioursnecessarytoincreasedjobperformance.

Behaviourisawordthatoftencausesdiscomfortwhenutilizedwithinthecontextofdevelopmentandtraining.Itisatermthatseeminglydoesnotfitwithintheworldofresultsbasedmanagement, outputs, activities, and outcomes. Trainingobjectives often shy away from using the term ‘behaviourchange’becauseitisdifficulttochangeapersons’behaviour;however,thisisexactlywhattraininginstitutionsaimtodo.Ifthereisnochangeinthebehaviouroftraineespost-training,thetrainingisawasteofresources.

If the learning from the training has been useful, relevant,understoodandpractical,itshouldbeeasyandbeneficialforthe traineetoadopt the learning into theirworkplace.Thiswouldresultinchangesnotonlyintheindividuals’behaviour,but also in their organizations’ performance provided thattherearenoorganizationalorexternalbarriers.

4.4. Translating the Learning Objectives into Training Design

The training is specifically designed to meet the learningobjectives. It may be useful to understand training in theperspective of a Results Chain. This process ensures thatthetrainerhasaclearunderstandingofwhateachtrainingactivitywillresult in.Eacheffortoractivityonbehalftothetrainer should lead to the trainees learning a specific skillwhichwillhelpthemreachtheirultimatelearningobjective.For more information on what a Results Chain is and how to use it, please go to Appendix 12.

Remember to ask the simple questions when planning atrainingprogram.Whatwilltraineeslearnfromeachelementof training and howwill it help them achieve the learningobjectives? Is spending 15 minutes on a certain diagramnecessary? Will it result in helping the trainees meet thelearningobjectives?Isthereabetterwayoftransferringthisknowledge?Thetraineeswillhavedifferentlearningstylessousingvariouspedagogicalmethodsisimportanttoensurealltraineeshavetheopportunitytofullygrasptheconceptsandbenefitfromthetraining.

Learning objectives are most likely to be met when thetrainersfollow-upandmonitorthetraineesastheyimplementtheir learning.Furthermore, trainersshouldbecontinuouslyimprovingtheirownknowledgebyconductingevaluationandsharingthelessonslearnedwithco-workersandcolleagues.Manypracticaltoolsexisttoensurethatfollow-up,monitoringandevaluationareconductedeffectively.Usingavarietyoftools ishelpful. Itensures there isaholisticviewordiverseperspectivesconsultedaboutthelearningobjectivesastheyare implemented and followed-up on. Using various toolsalsoaidsinunderstandingandmaximizingtheeffectsofthetraining.ConsiderusingtoolssuchastheFollow-Up Plan,Five “W” Interview,orLearning Improvement Plan.Forfurtherinformationonwhatthesetoolsare,see Appendix 13 for the Follow-Up Plan,Appendix 14 for the Five “W” InterviewandAppendix 15 for a Learning Improvement Plan.

PART 2: Training Outcome

Evaluation

5

25INTRODUCTIONTOTRAININGOUTCOMEEVALUATION

INTRODUCTION TO TRAINING OUTCOME EVALUATION5

5.1. Proving the Value of TrainingWithoutanevaluationorunderstandingoftheresultofthetraining it is impossible to determine whether or not theinvestmentinthetrainingprogrammewassuccessful.Neverbeforehasknowing thevalueofan investmentbeenmoreimportantthanatthistimeofcurrentfinancialcrisis,whichisincreasingtheglobalcompetitionforinvestments.Thereforeorganizations,includingtraininginstitutionsandbeneficiariesoftraining,mustbeincreasinglyaccountableforeverydollar.Trainingevaluations,provingthefinaloutcomeofthetraininginvestment,aremorecrucialthaneverbefore.

Through a training outcome evaluation the traininginstitutioncanprovetheeffectivenessofitsservices;thattheinstitute delivers training programmesthat improve the performance of theparticipantsandtheirorganizations. Inthesamewayevaluationsarecriticaltothe organization investing in training,theclient,toprovetheyarenotwastingresources on activities which do notleadtopositive results.Theevaluationalso provides important feedback thatcan improve future training cycles ofboth the training institution and theclient. By learning from successesand mistakes of previous trainingexperiences the training effectivenesscancontinuouslyimprove.

5.2. Levels of EvaluationEvaluationscanbecarriedoutondifferentlevels:evaluatingtraining events and participants’ immediate reactions,evaluatingparticipants’learning,evaluatingjobperformanceoutcomes, and evaluating organizational performanceand change. These levels are directly related to DonaldKirkpatrick’s “hierarchy of learning”, which describes howthe result of each level depends on the previous levels12.For example, learning is unlikely to occur if the immediateparticipantreactiontothetrainingisnegativeandimprovedjob performance is highly unlikely if the participants havelearnedlittlefromthetraining.

12 Kirkpatrick, Donald L., “Techniques for Evaluating Training Programmes”, Training and Development Journal, June 1979, pp. 78 - 92.

FIGURE6:Four levels of evaluation

PARTICIPANT REACTION

INDIVIDUAL LEARNING

JOB PERFORMANCE

ORGANIZATIONAL PERFORMANCE

Results that can be observed some time

after the training event

Results that can be observed during, or in immediate

connection to, the training event


It is common to evaluate the participants’ reactions in theendofanevent,but theother levelsareunfortunatelynotas often considered. Compared to evaluating participantreactions it is harder to evaluate learning, jobperformanceand organizational performance. However, only evaluatingtheparticipants’reactions intheendoftheeventdoesnotprovidethetraininginstitution,northeclient,withsufficientinformationabout the results. Thehigher levelshave tobestudiedtoseeiflearninghasoccurredandwhetherornotithasbeentransferredtotheworkplace.Onlythenitispossibleto know if the training has had the desired outcome onthe trainees, their jobperformance, and theorganizationalperformance, and ultimately if the training investment haspaidoff.

To evaluate all four levels is particularly important forextensive training programmes and for completely newtraining programmes that have not been delivered before.For all trainingprogrammes the first two levels, participantreactionandlearning,shouldbeevaluated.

5.3. Meeting the Challenges of Evaluations

It is certainly more challenging to evaluate participants’learning,theirjobperformance,andorganizationaloutcomes,especiallybecausetheremaybealackofinterestfrommanyofthestakeholderstoengageinevaluationsaftertheevent.Reasonsincludeevaluationsbeingperceivedastoocostlyoras takingtoomuchtimefromthepeople involved.Despiteall the benefits to evaluations, sometimes the cost of aneffective evaluationdoesoutweigh the costof the trainingprogramme. If the training participants are from differentorganizationsorgeographicalregionsitcouldalsobedifficulttokeepcontactwithallof themonce the trainingevent isover.Anotherbarrier toconductingevaluations is that theyoftendonotfitwithinthetimeframeoftypicalshortprojectcycles.Itmaybearelativelylongtimeafterthetrainingeventbefore any substantive result on a job performance levelcanbeseen.Furthermore,somepeoplefinditimpossibletoisolatetheeffectofthetrainingprogrammeonthetrainees’jobperformancefromotherfactorswithintheorganization.Lastly,sometrainingactivitiesarepoliticalinnatureandwhenthe primary interest is a visible activity training evaluationsmaybeoflowpriority.

Thesechallengesareimportanttobeawareofandshouldbetakenintoconsiderationattheplanningstageofatrainingprogrammeinordertobeabletorespondeffectively.Thiscanbedonebyincludingevaluationinthetrainingprogrammes’budget and time plan, deciding on a suitable extent ofevaluation,and informingtheclientandotherstakeholdersabout the benefits of investing in a training outcomeevaluation.

6

27MANAGINGTHEEVALUATIONPROCESS

66.1. Planning the EvaluationThetrainingoutcomeevaluationhasitsstartingpointalreadyinthetrainingneedsassessmentandthelearningobjectives.It isplanned inparallelwith thedesignanddeliveryof thetraining and carried out both during training events andpost-training.Figure7 illustrates thedifferent stagesof theevaluationprocess andhow they contribute toproving theresultsoftrainingandcollecting“lessonslearned”toimprovefuturetrainingprogrammes.

While comprehensive evaluations areuseful, it is importantnotmaketheprocesstoocomplicated,timeconsumingandcostly.Trainingoutcomeevaluationisamanagementtoolandshouldbedrivenbytheneedtoprovideenoughinformationfor theclient tomakegooddecisionsabout future traininginvestments,andforthetraining institutiontocontinuouslyimprove their trainingprogrammes.However,only focusingonthetraininginterventionitself,withoutnotingiflearninghasoccurredandbeentransferredtotheworkplace,willnotprovideenoughinformationforfutureimprovements.

FIGURE7:The training outcome evaluation process

1

2

3

4

5

PROOF OF RESULTS

LESSONS LEARNED (to improve future training programmes)

Starting point: Training Needs Assessment &

Learning Objectives

Manage the evaluation:

Evaluate event & reactions:

Evaluate indi-vidual learning:

Evaluate job performance & organizational

change:

Learn from the evaluation

• Planinparallelwithdesign&delivery

• Mobilizeresources

• Setupevaluationteam

• Thetraining’scontent,delivery&support

• Refertolearningobjectives

• Knowledge,skills&attitudes


• Usevarioustools&sourcesofinformation


• Bothtrainingprovider&client

• Usee.g.theActionLearningCycle

MANAGING THE EVALUATION PROCESS


6.2. Mobilizing ResourcesEvaluations are an integral part of the trainingprogrammeand the resources for training outcome evaluations shouldbe allocated already when setting up the training budget.If evaluation resources are not mobilized by the time thetraining is complete there are less opportunities to findfunding in time to conduct an effective evaluation. Whensetting up the training budget it may be necessary tonegotiate with funders, making the value of investing inevaluations explicit, to convince them to include funds forpost-training evaluations. Re-prioritizing activities can alsobe considered tomake funds available for evaluation. Lessinterventionsorfewerparticipants,butagoodfollow-upandevaluation,islikelytohavegreaterimpactthancarryingoutmanyinterventionsthatareneverevaluated.

Resourcemobilizationmightworkslightlydifferentforthosetraininginstitutionswhichonlyconductevaluations.However,itisequallyimportanttobeabletoconvincepotentialfundersof the importance of evaluations. Previously conductedevaluations can be used to prove the value of evaluations,aswell as the valueandqualityof the training institution’straining programmes, and consequently making resourcemobilizationeasier.

Themostcommonsourcesforfundingoftrainingprogrammesare the training institution itself, the client, the nationalgovernment, and various typesof international agencies. Itis wise to not rely solely on one source of funding but toworkwithmultiple sources. There are different benefits ofevaluationstothedifferenttypesoffunders,andthespecificbenefitsareusefultostressinthenegotiationsforfunding.

Training institutions should invest inevaluations toprovetheeffectivenessof their trainingprogrammesandto learnfromtheprocessinordertoimprovefuturetrainingactivities.Proofofwhatlearningandperformancechangethetrainingresults in is valuable to promote the institute’s trainingprogrammes,especiallywhenmobilizingresources.

The clientbenefitsfromevaluationsbygainingknowledgeofwhetherornottheirtraininginvestmentspaidoff,andwhyitdidordidnot.Byevaluatingthetrainingtheclientcanmakebetterinformeddecisionsonfuturetraininginvestments.

National governmentsoftenprovidefinancialsupportfortrainingoflocalgovernments.Evaluationsareanopportunityto be accountable to the tax payers by proving that theirmoney is investedwisely in activitieswhichhave a positiveeffectontheperformanceoflocalauthoritiesandarelikelyto improve the services provided to citizens. In addition,the national government can improve the overall trainingeffectivenessofthecountrybymakingsurethatthetrainingstheysupportareevaluatedandthatthelessonslearnedaredocumentedandshared.

International agenciesandotherfundersneedevaluationsto provide accountability and to show transparency of thetraininginvestmentaswellastodemonstrateresults.Thelatteris important asmany international agencies, governments,foundations,corporations,andotherfundersareincreasinglylooking for partners which operate comparably to Results-BasedManagement.Likeallinvestors,fundersofdevelopmentprojectsappreciateknowingtheresultsof their investment,astheytoo,areoftenheldaccountabletootherstakeholders.

6.3. The Evaluation TeamA training outcome evaluation requires engagement fromseveralstakeholdersandideallytheyshouldallberepresentedintheevaluationteam.Atalltimesaninternalevaluationhastobeconductedbymembersofthetraininginstitution.Thisshould include, but not exclusively, the trainer or trainingfacilitator.

Whileitisimportantforthetraininginstitutionandparticulartrainer to evaluate their own work, it is also advisable touseexternalevaluators.Thismeans someonewhowasnotdirectly involved in the training, in the organization of thetraining institution or client, or in the organization whofunded the training. An external evaluator is less likely tobe biased by personal interests, which adds credibility tothe outcome evaluation. It also increases the likelihood ofaccuratelyidentifiedresults,whichiscrucialfortheevaluationtobeofvalue.

Training institutionsusuallyhaveaccess toa largenetwork.Drawuponthiswhenformingtheevaluationteam.Itmaybeadvisabletoincludepeers,technicalexperts,orcivilsocietyintheevaluationteam.Considerincludinganinternalmemberofthetraininginstitution,ortrainee,ifpossible.Oftenamixedteamcomprisedofindividualsfromvariousrelatedfields,willprovideusefulinsightsfromtheirmixofdifferentstrengths,perspectives,vestedinterestsandmotivations.

FIGURE8:Who to involve in the evaluation team?

THE EVALUATION TEAM

From the client:From the training institution:

The external evaluator Representative from funding agency

• Oneormoretrainingparticipants• Thetrainees’manager• TheHRDmanager

• Thetrainer• Thetrainingdirector

29MANAGINGTHEEVALUATIONPROCESS

6.4. Reality Check: Do Benefits Outweigh Costs?

Before conductinga trainingoutcomeevaluation,potentialbarriers should be considered and the benefits of theevaluation weighed against the costs. There may be caseswhere lack of engagement from the stakeholders wouldimpedetheevaluationormaketheresultslessuseful.Insomesituationsitmaybenecessarytostressthevalueofevaluationsto the client or other stakeholders, before conducting theevaluation.

Whilethismanualsuggeststhatevaluationsshouldbebuiltintoalltrainingprogrammes,therearesomesituationswhereconductinganevaluationmaynotmakesenseas thecostssimplyoutweighthebenefits.Thedecisiontonotconductanevaluationshouldbeconsciouslymadeandthereasonswellunderstoodbythetraininginstitutionandclient.Infigure8isachecklistonwhatneedstobeinplacebeforeconductingatrainingoutcomeevaluation.Thisisusefulinsupportingthedecisiononwhetherornottoproceed.

FIGURE9:Checklist to support decision on whether or not to conduct a training outcome evaluation

Doyouknowwhotoinvolveintheevaluation–whothemainstakeholdersare?

YES

NO

NO

NO

NO

NO

NO

YES

YES

YES

YES

YES

Doestheclientwanttoundertaketrainingoutcomeevaluation?Iftheydo,doyouknowwhy/therationalebehind?

Doyouknowwhattoevaluate?Arethereclearlearningobjectivesforthetraining?

Isthereacommitment,bothfromthetrainingproviderandtheclient,tousetheevaluationresultstoimprovefuturetrainingandperformance?

Youmayneedtoworkmorewiththeclienttoensurecommitment,orinformthemwhyevaluationisnotfeasibleor

advisableinthiscase.

Arethereresourcesavailabletoconductanoutcomeevaluationthatisthoroughenoughtoprovidereliableandrelevantinformation?

Willthebenefitsoftheevaluationoutweighthecostsintime,energy,moneyandotherresourcesinvestedintheevaluation?

Conduct the Training Outcome Evaluation

Do not conduct the Training Outcome Evaluation


EVALUATING EVENTS AND PARTICIPANT REACTIONS7

7.1. Participant Reaction Affects LearningEvaluating training events and the participants’ reactionstowardstheseeventsarethemostcommontypeofevaluationconducted and unfortunately most evaluations end at thisstage. Evaluations that only assess the training events andthe participants’ reactions are inadequate and insufficientin providing information to determine if the training wassuccessful. From this type of evaluation it is not possibleto draw any conclusions on whether or not the learningobjectiveshavebeenmet and the training investmentpaidoff.

However, it is still important to evaluate training eventsandparticipants’ reactions. Evaluating events and reactionscontributesto:

• Anunderstandingoflaterevaluationresults,suchaswhyparticipantsdidordidnotlearnorincreasetheirjobperformanceafterthetraining.

• Determiningwhetherornotthetrainingwasculturallyrelevant.Thetrainercanoftentellimmediatelyfromtheparticipants’reactionswhetherornotthetrainingwasrelevant.

• Ideasonhowtoimprovethespecificeventordecideifitshouldorshouldnotbeincludedinsimilarfuturetrainingprogrammes.

• Providingusefulinformationforadaptationoffollow-upactivitiesandpreparationoffuturetrainingofsimilargroups.

FIGURE10:The role of evaluating participant reaction in a training outcome evaluation

Training content





JOB PERFORMANCE



Training delivery

Training content

Training support

INDIVIDUAL LEARNING

31EVALUATINGEVENTSANDPARTICIPANTREACTIONS

Participants’reactionstothetrainingeventaredirectlyrelatedtotheeventitself,whichismadeupofthreekeycomponents:content,delivery,andsupport.Theseareasmustallresultinfairlygoodreactionsfromtheparticipantsinordertosupportindividuallearning.

Avery importantsourceof informationontheparticipants’reactions is the trainer’s observations of the participants’reactionsduringtheevent.Otherwaysofevaluatingeventsarequestionnairesandinterviews.

End-of-event questionnairescanincludeclosedandopenquestionsonwhetherornotthetraineesfoundthecontentappropriate, enjoyed the delivery, or wished to commentonthetrainingsupport.Tomaximisetheresponserate it isrecommendedtoincludethequestionnaireasacompulsorypartintheendoftheevent.See Appendix 4c for an example of a post-training evaluation sheet andAppendix 4a for general information on questionnaires.

Group interviews orsemi-structured interviewsmayalsoserveasgoodways tocollectdataat theendofanevent.For more information and examples on how to use thesemethods,see Appendix 1a for interviews andAppendix 2a for group interviews.

Evaluating the specific training events and the reactions orresponsestheyreceivewillensurethattheeventsincludedinthetrainingareusefulandeffective.Thebettertheanalysisofthese,thebetterthetrainingwillbecomeforfuturetrainingprogrammes,therebymaximizingtheclient’sinvestmentandencouragingfurtherinvestmentsintotraining.

7.2. Training Content Thetrainingcontent isthematerialthat ispresentedatthetraining, not how the training is presented. The trainingcontentshouldbedrivenbywhatthetrainingisexpectedtoachieve;thelearningobjectives. Itmustberelevantbothtothetraineesoftheleastandgreatestcapacities.Thetraineesneed to be able to relate to the content so that they candigest it andunderstandhow to apply it to their lives andwork.

In addition to the learning objectives, training content isdecided upon based on the client and trainees’ currentcapacities.Althoughthecontentreflectswhatispossibleforthetrainingtoachievethetrainingdeliverymethodsgreatlyinfluence the trainees ability to understand and apply thecontent.Therefore,havingexceptionaltrainingcontentdoesnotnecessarilyimplythatalllessonsandskillswillbelearnedandapplicablebyeachofthetrainees.

Trainingcontentusuallyfocusesonthefollowingkeyareas:increasing knowledge and understanding; improving skills(technical,mentalandsocial);changingattitudesandvalues;andpromotingcreativityandingenuity.

Evaluating the trainingcontentwillprovidevaluable insightabouthowthecontentof similar trainingprogrammescanbemodified to bemore effective in the future. It is oftenbeneficialtoaskquestionssuchas:

• Wasthecontentrelevanttothetraineeswork?

• Wasthecontentappropriatetothetrainees’currentknowledgeorskilllevel?

• Whichcontentdidthetraineesfindmostvaluable?

• Didthecontentalignwiththelearningobjectives?

7.3. Training Delivery The training delivery, how the training is conducted, willinfluence the trainees level of learning and affect theirreactiontothetrainingexperience.Itisofutmostimportancethatthetraininginstitutionandtrainerhavetakenthetimetounderstandwhatneedstobecommunicatedandhowtodoitinanappropriatewaythatwillensureitbenefitsboththetraineesandclient.Afterthetrainingtheseeffortshavetobeevaluated;wasthetrainingcontentdeliveredinwaythatsupportedallparticipants’learning?

Trainees will have many different learning styles, thereforeit is key that training institutions can adapt the trainingcontentandshareitthroughavarietyofdifferentmethods.Thetrainingdeliverymustbesuitedfortraineesrepresentingboththeleastandgreatestcapacitypresentatthetraining.Experimentaltactics,alsoknownaslearningbydoingmethods,canreallybenefitthetransferoflearning,particularlyamongadulttrainees.Thisisnottosaythattraditionalexpertlecturesareineffective,butvarioustypesoftrainingmethodsshouldbeexplored.Havinganexperttalkforhoursaboutanytopiccouldquickly lend itself to loose the traineesattention if itdid not include case studies. Better yet, the training couldincludesitevisitsandstakeholderdiscussionsgivingtraineesfirst-handexposureofthetopicpresented.

However,itisimportanttorecognizethattrainingdelivery,nomatterhowingeniousandengagingitmaybe,couldresultinverylittleindividuallearningandnochangeinjobperformanceif the methods used are not culturally appropriate. Thetraining institution needs to know the trainees’ context.Different cultures, societies and communities operate with

An urban mobility workshop is held in a rapidlygrowing, sprawled city. Theworkshop is facilitatedbytransportationengineersandfocusesonhowtouselow-cost,timeefficienttrainsasthesolutiontomasspublictransportationduringcommuterhours.This would free up the roadways for commercialpurposes. They also speak about the potentialchallenges and of engineering accomplishments inothersimilarly-challengedcities.

Thecityofficialsareexcitedaboutthisandwishtostartbuildingthecommutertrainsimmediately.Itissoonevident to the trainingprovider that theydidnotrealizetheofficials invitedfromtheMinistryofTransportation,werenottheoneswithengineeringbackground as they thought. This explained whythe officials’ eyes glazed over when the detailedengineeringblueprintsoftheirandothercitieswerediscussed. It became evident that it was unlikelythe technical aspects of implementation and thepotential challenges were understood in the levelof detail they needed to be. The training contentwas relevant to theMinistryof Transportation,butnot relevant to the trainees’ capacities. Inorder topursue their new visionary transportation goal,havingacommutertrain,thecitywillneedtobringin engineers to analyze the pros and cons for thisspecificcityaswellaslearnhowtobuildthesystem.

WHEN TRAINING CONTENT DOES NOT MATCH THE CAPACITIES OF THE TRAINEES


differentculturalnorms.Levelofliteracyalsohastobetakenintoaccount.Thesefactorscandictatewhomayattendthetraining, who should facilitate the training, which deliverymethods and examples are appropriate,what the trainees’expectationof the training is, etc. For example, sometimesit is necessary to design the training to specifically targetwomenas their learningmaybehindered if the training isdominatedbymen.

It is difficult to evaluate the pedagogy without evaluatingthecontent;howeveritisessentialdoinordertodeterminehow and if the training delivery has contributed to thelearningresults.Herearesomewaystodeterminethelevelofsatisfactionwiththetrainingdelivery:

• Simplyaskandprovideopportunitiesfortraineestorespond,eitheranonymously(iftheleveloftrustandopennessseemslow,whichcouldbeacluethattheprocessisnotworking),oropenly.

• Trackthetardinessorabsenteeismwhichisnottiedtosomelegitimateexcuse.

• Trackthepresenceofdisruptiveandotherresistantbehaviour.

• Holdpost-sessioncritiques.

Observethefollowing:

• Inmixedgenderedtrainings,didmenandwomenparticipate?

• Withparticipantsofallagesinthesametraining,didallparticipate?

• Didmanagersandsubordinatesattendthesametrainings;didallparticipate?

• Weretheirdominantpersonalitiesinthetraining?Ifso,weretheirstatementschallengedbynon-dominantpersonalities?

It is important to also evaluate the cultural aspects of thetraining:

• Howdidtheage,gender,socialstatusofthetrainerandtraineesaffectthetraining?

• Wasthetrainingdeliveredinsuchawaythatalltraineescouldfullyparticipate?

• Werethereanyissuesthatmayberelatedtoculturalinappropriatenessofthetrainingdelivery?

7.4. Training SupportTraining support is what happens before, during and afterthe training,oftenbehindthescenes, tomakethe trainingpossible. Effective administrative and logistical supportenablesthetrainingprogramtorunsmoothly,whichallowsthe trainees to fully concentrate on the training activities.Poor training support, on the other hand, will distract thetrainingparticipantsandhindertheirlearning.Consequently,lacking trainingsupportcanundermine thewhole training,preventinganypositiveresults.

Administrativesupportincludesfactorssuchastheapplicationprocess, information sent out before the training, beingavailable to answer questions, making material accessibleduring and after the training, and distributing contactinformation. Some logistical factors include the trainingvenue,meals,transportation,trainingschedule,soundsystemandvisualaids,andinterpretationfacilities.

Trainingsupportmustbeincludedwhenevaluatingtheeventandtheparticipants’reactions. It isalsohelpfultomonitorandadjust the supportduring the training. If, forexample,theroomforthetrainingistoobigforalltheparticipantstohearthespeakerandtheotherparticipants,actionshouldbetakenimmediatelytosolvetheissue.

Participants filling in post-training evaluation sheets. © UN-Habitat/Bridget Oballa

8

33EVALUATINGINDIVIDUALLEARNING

EVALUATING INDIVIDUAL LEARNING 8

8.1. Why Evaluate Individual Learning? Toevaluateindividuallearningisdifferentfromevaluatingthetrainingeventandparticipants’reactions,despitebothbeingevaluations on an individual level. Itmight be tempting toassume that if the participants find the training interestinganduseful to theirwork, theyalso learneda lot.However,there is no guarantee of learning taking place, no matterhowmuch the participants appreciated the training. Thereis indeeda linkagebetween the two;apositiveparticipantreactiontothetrainingisrequiredforthetrainingtoresultinlearning. If the trainingparticipants forexampleconsidered

the training content to be irrelevant to their context andthescheduleforthetrainingtootight,thelikelihoodofanyusefullearningisverypoor.

Evaluating what individual learning the training resultedin, is an important step in evaluating the outcomes of thetraining.Itproveshoweffectivethetrainingactuallywas,nothow itwasperceived, in termsof relevance to the traineesanddelivery of the training content. If the trainees do notlearn from the training, the desired changes in the clients’organizationwillnottakeplace.

FIGURE11:The role of evaluating individual learning in a training outcome evaluation


INDIVIDUAL LEARNING

JOB PERFORMANCE







8.2. What is Learning?Learning is directly related to increased knowledge andunderstanding; improved technical, mental or social skills;orchangesinattitudesorvalues.Theknowledge,skillsandattitudesevaluated shouldbe the sameas specifiedby thelearningobjectives setupbefore the training.AsdescribedinSection4,thelearningobjectiveshavetobespecificandmeasurable,otherwiselearningresultscannotbeevaluated.

Learningisnotequivalenttobehaviouralchange,butiftheconditionsareright,learningcanleadtochangedbehavioursofthetrainingparticipants.Thishighlydependsonthetrainees’motivation,andtrainingprogrammesshouldbedesignedtomotivatetheparticipantstoadoptnewbehavioursrequiredto implement the learning into their work. Consequently,behavioural change is what links individual learning toimproved job performance and ultimately organizationalchange.

8.3. Methods for Evaluating Individual Learning

Therearemanywaysofevaluatingindividuallearning,whatmethodisbestdependsonwhattypeoflearning(knowledge,skills or attitudes) is to be evaluated. For all methods, thepost-traininglevelofknowledge,skillsorattitudesshouldbecomparedwiththelevelidentifiedduringassessment,aswellaswiththelearningobjectives.Refertothelearningcontractset up before the training in order to find the learningobjectivesandcommitmentsoftraineesandtheirmanagers.See Appendix 10 for an example of a learning contract.

Itissometimesarguedthattheparticipantsintrainingwouldhave learned on the job, without taking part in training.Thisshouldbeconsideredalreadyduringthetrainingneedsassessment.Agoodtrainingneedsassessmentexploresiftheparticipantsare likely to learnon theirownor if training isrequired.However,inasituationwherethevalueoftrainingis uncertain, despite a thorough assessment, it may beuseful to have control groups. Ideally, the control groupshould be similar to the training participants in terms ofjob, age, experience, skill/knowledge level, etc. The controlgroup isgiventhesamepre-andpost-trainingtasksasthetrainingparticipants,buttheydonotreceiveanytraining.Bycomparing the results of the training participantswith theresultsof thecontrolgroup itcanbeestablished if trainingcontributed to the learning, which can also guide futuredecisionsonwhentoconducttraining.

Competency tests canbeusedorallyor inwriting to testprinciples, facts and other knowledge-based objectives.Developingacompetencytestusuallyinvolvesthefollowingsteps:

1. Outlinewhatistobecoveredbythetestbasedonthelearningobjectives.

2. Decideontestformatandtimetocompletethetest.Thequestionscanbeopen,multiple-choice,true/false,anessay,oracombinationofalltypes.

3. Constructquestionsortasksforthetest.Thequestionsshouldbeclearlywrittenandonareadinglevelappropriateforthetesttakers.

4. Checkthatquestionsarejob-related,coverallareasofcompetencyincludedinthetraining,andthatthereareenoughquestionstoensurereliabilityofthetest.

Demonstrations of skills are particularly useful forevaluatingphysical(technical)skills.Itisatypeofobservationconductedinacontrolledenvironment,assimilartotherealwork situation as possible. See Appendix 3 for general information on observations.

A training of government officials in renewableenergy technologies has the following learningobjective:

“After the training the participants will have knowledge of the main types of renewable energy; bio gas, wind power, hydro power, tidal power, wave power, solar energy, and geothermal energy. They know the pros and cons of each type, particularly in relation to their local context. Additionally, the participants have knowledge of the technology behind each type of energy production, including costs and environmental aspects.”

Theparticipants’learningisevaluatedbygivingthemawritten test, both before and after the training,withquestionsontheaspectsofrenewableenergymentionedinthelearningobjective.

EXAMPLE OF COMPETENCY TEST

A training of urban planners in using a computerprogram to create drawings of city plans has thefollowinglearningobjective:

“After the training the participants will be able to use the software to produce clear drawings of a current city plan as well as visualizing proposals for future changes, both on a city level and in detail (scale 1:500).”

The software is new to all the clients and theiremployees.Afterthetrainingthetraineesareaskedtodemonstrate their skillsbyproducingadrawingofaspecificpartoftheircityusingthesoftware.Toseewhetherornot the improved skillsarea resultof training, both employees who took part in thetraining,andemployeeswhohaveusedthesoftwarethesameamountoftimebutwithouttraining,areasked to demonstrate their skills. The quality andaccuracyofthedrawings,aswellasthetimerequiredtoproducethem,arecomparedbetweenthegroups.

EXAMPLE OF DEMONSTRATION OF SKILLS

35EVALUATINGINDIVIDUALLEARNING

Questionnaires canbeusedbefore andafter the trainingtoevaluatechangesinattitudes.Theresultoftheevaluationvery much depends on the accuracy of the assessment ofthe trainees’ attitudes. To avoid the bias and subjectivityassociated with self-assessments, a questionnaire on thetrainee’sattitudescanbeadministeredtootherssuchasthetrainee’ssupervisor,subordinates,andcolleagues.Please see Appendix 4a for general information on questionnaires.

Assessment centre techniques are used for evaluatingknowledge, skills or attitudes of a small group of people.Scenariosorcasesmakethestartingpointoftheevaluationand the participants receive a task to solve related to thescenario or case. The session is video recorded and laterstudied,toevaluatetheskills,knowledge,orattitudesoftheindividualparticipants.Mentalor social skills,whichcannoteasilybeevaluatedinatraditionaldemonstration,arepossibleto studywith assessment centre techniques. Scenarios andcases can also be used to evaluate learning of individualsdirectly,notalwaysingroups.

A training of local government leaders in gendersensitivityhasthefollowinglearningobjective:

“After the training the participants will have increased awareness of issues regarding gender sensitivity, as well as apply appropriate measures in their work to ensure gender equality. This includes paying equal attention to male and female colleagues’ opinions, avoiding language and behaviours that can be considered sexual harassment, and considering both men and women when engaging with the local community.”

Toevaluatethechangesinattitudesofthetrainingparticipant,aquestionnaireissenttotheparticipants’subordinates and colleagues both before andafter the training. The questionnaire is addressingthe issues described in the learning objective andthe respondents are asked to rate the attitudes ofthe training participants on a five-step scale. Bycomparing the ratingbeforeandafter the trainingthechangesinattitudescanbeestablished.

EXAMPLE OF WHEN TO USE QUESTIONNAIRES

Atrainingofengineersemployedbypublicsectorinproblemsolvinghasthefollowinglearningobjective:

“After the training the participants will have improved problem solving skills. They will be able to explore the problem to find the root and contributing factors, creatively find several solutions, and select and argue for the most suitable solution.”

Beforethetrainingtheengineerstobetrainedtookpartinanassessmentcentrewherethey,ingroupsoffourtofive,werepresentedwithacriticalsituationrelatedtotheirjobandaskedtodecideonasolution.Asimilartaskwaspresentedtothesamegroupsafterthetraining.Bothsessionswerefilmedandanalyzed.Foreachparticipantthesessionsbeforeandafterthetrainingwerecomparedinregardstohowwelltheyreasoned around the cause of the problem, howmany ideas for solutions they came up with, andhowwelltheymotivatedtheproposedsolution.

EXAMPLE OF ASSESSMENT CENTRE TECHNIQUES

Multiple choice test. © Antonio G.


JOB PERFORMANCE OUTCOMES AND ORGANIZATIONAL PERFORMANCE9

9.1. Why Evaluate Job Performance?Evaluating job performance is different from evaluatingindividuallearning.Whathastobeevaluatedishowthetraineeapplies the knowledge or skills from the training. Traineesmay or may not apply everything they have learned; it iseasiertolearnsomethingthantochangeexistingbehaviouralandstructuralpatterns.Furthermore,when learningcanbeobserveditmaystilltakeawhilebeforeithasanyeffectonthejobperformance.Evaluatingjobperformanceistheonlywaytoevaluatewhethertheindividuals’learninghasresultedinbehaviouralchangeandimprovedperformanceatwork.

Training outcome evaluations are specifically designedto determine the cause and effect of relationships; fromthe individual contact with the training institutions andidentification of performance discrepancies, to determiningwhether the investment into training is causingany impactonjobperformance.

Whether or not the training has successfully achieved itsoutcome,itmayhaveachievedunexpectedandunpredictedresults.Thiscanbebothapleasantordisconcertingsurprise.Training outcome evaluations are a good mechanism tounderstandthecausesofthechanges,orlackofchanges,aswellascausesofanyunintendedconsequences.

FIGURE12:The role of evaluating job performance in a training outcome evaluation


INDIVIDUAL LEARNING

JOB PERFORMANCE






37JOBPERFORMANCEOUTCOMESANDORGANIZATIONALPERFORMANCE

9.2. Methods for Evaluating Job Performance

Evaluating job performance will reveal if there has beenbehaviouralchangeinthetraineespost-training,andifitcanbedirectlyrelatedtothetraining.Thisisonlypossibleusingspecificevaluationcriteria.Evaluatorscanneverbe100percentcertainthatthechangedbehaviourisdirectlyrelatedtothetraining.Howeverwithsmarttrainingoutcomeevaluationprocesses (beginning during the training needs assessmentandtrainingdesignphases),itispossibletoascertain,withacertaindegreeofconfidence,whichbehaviouralchangesarearesultofthetraining.

Tobeabletoseechangesinperformanceitisusefultoapplythesamemethodswhenevaluatingoutcomesasduringtheassessment. If, for example, the trainees’ managers wereinterviewedaboutthetrainees’ jobperformancebeforethetrainingthesamequestionsonlevelofperformanceshouldbeaskedduringtheevaluation.Interviewsandquestionnairescanbothbeusedtoevaluatejobperformance.However,itisimportanttokeepinmindthattheyarelimited.Forexample,interviewsonlyprovidetheopinionsoftheinterviewee,whichisnotnecessarilythesameastheactualjobperformanceofthetrainee.Thedesign,structureandcontentof interviewsandquestionnairesstronglydeterminetheresultsandshouldbecarefullythoughtthrough.Please see Appendix 1a for more information on and examples of interviews andAppendix 4a for questionnaires. Regular conversationscanalsocontributetotheevaluation.Questionsaskedinaninformalandrelaxedatmosphereisagoodwayofgettingafirstideaofthejobperformanceoutcomes.

AnothermethodtoassessifindividualbehaviouralchangehasoccurredattheworkplaceistheJob Shadow Performance Test. This tool is made up of two parts. The first part isconductedduringtheassessmentwhenspecifyingthetrainingneeds. At this point, personnel conducting the assessmentwill job shadow some of the client’s employees who holdkeypositionsrelatingtotheclient’sproblemorimprovementareas.Whattheydoandhowtheydoitarewatched.Notesandobservationsarerecorded.Thesecondpartisconductedafterthetrainingandoncethetraineeshavehadopportunitytoapplytheir learningtotheirworkplace.Whilemoretimeconsuming,withdiscernment,thechangesinbehaviourcanbeaccreditedtothetraining.Please see Appendix 16 for more details on Job Shadow Performance Testing.

9.3. Individual and Organizational Level Performance Barriers

Asmentionedearlier,therearemanybarrierstothetransferof learning. Somemightnotbepossible to identify duringthe assessment, and evaluationsmake a good opportunityto identify them and prevent them from hindering futuretraining programmes. Factors that may affect the trainee’sapplication or lack of application of the learning are oftenorganizational structures, or legal and political frameworksbeyond the trainees’control.Didmanagement support thetraining?Didtheclienthaveunrealisticexpectations?Didtheclienthaveenoughresourcestoimplementthelearning?

However,iftherehavebeenindividualAction Plansdevelopedatthetrainingevent,outliningspecificwaysthetraineecanincorporate the learning,andnoneor fewareapplied, thelackofapplicationislikelyonanindividuallevel.Asatraininginstitution,donotfeeldefeatediflittlechangehasoccurredinindividualjobperformance,oriflearningfromthetrainingnot been applied. Instead, investigate further. Perhaps thetraineedidnotagreethatanewprocesswasnecessaryfortheorganizationandwantedtomaintainthestatusquo;thattheydonotbelievethelearningwillbenefittheirorganizationorthemselves;or,thattheydonotunderstandhowtoapplythe training.Their responsescanbeused tobetterprepareforfuturetrainings,andtobetterunderstandhowtointeractwith the client so that all employees support the training.Please see Appendix 17 for more details on Action Plans.

9.4. Evaluating Organizational Performance and Change

Evaluating organizational performance and change ischallenging. The greater the influence and reach of thetrainee’s jobacross theorganization,and themoregeneralthelearningobjectives,themoredifficultitistoseparatetheoutcomeofatraininginterventionfromalltheotherfactorsthat are influencing the organization’s performance. Thisdoesn’tmean the evaluator should not try to evaluate theorganizationalimpactofthesekindsoftraininginterventions.Rather,itdoessuggestthatthemagnitudeofthechallengeisworthyofmorerigorousattentionbytheevaluator.

Some types of organizational performance are easier toevaluate than others. For example, the organizationalperformance related to the training of a clerical staffmemberon theclient’snewword-processorcanbedirectlymeasuredbyhis or her ability to produceprintedmaterial.Theorganization’sperformanceinproducingprintedmaterialis directly related to the performance of the employee.With production increasing, organizational performancealso increases. However, evaluating the organizationalperformance as a result from sending the chief executiveofficer to an executivedevelopment seminar ismuchmoredifficulttotrackandmeasure.

Organizationsaremultifacetedandaremadeupofindividualsworkingindifferentareas,eachwiththeirownpurposeandspecific role. The best way to conduct an organizationaltrainingoutcomeevaluationistoutilizeacombinationofthetoolsalreadydiscussedbecausedifferentorganizationalrolesarebestevaluatedbydifferentevaluationtools.ConductingavarietyofevaluationtoolsasdiscussedinthismanualandfoundintheAppendix,suchasinterviews,groupinterviews,job shadow performance testing and questionnaires,would strengthen an evaluation. If the particular aspect of

A clerical employee is sent away to learn how tooperate the agency’s new word processor. Shecomes back and begins to produce volumes ofprintedmaterial.Thetrainingoutcomes,inthiscase,canbemeasured fairly easily.However,heroutputintimidatesothersinthesecretarialpoolofworkers.As a result, they begin to “sabotage” her workefforts,which,inturn,affectthenewlytrainedclerk’smotivationandworkperformance.Training impactrecedes.Somemightattributethepoorperformancetothetraining.

UNINTENDED CONSEQUENCES OF TRAINING


Forexample,interviewingonlythetraineeswillnotproducea balanced evaluation. Theymay have applied the trainingandfounditquiteuseful,havinglearnedanewframeworkinwhichtoseeandperformtheirtasks.However,whilethetraineesmayhaveseengreatimprovementsintheirworkandfeelthattheyaccomplishtheirtasksbetter,theircolleaguesandmanagersmayhavenoticedlittletonochangeintheiroverall productivity of the organization. A true evaluationof the training’s effect at an organizational level can onlybe measured and understood when speaking with variousmembersoftheorganization.Changingthewaysomethingis done, such as howa report iswritten, is not necessarilythesameasincreasingindividualororganizationalefficiency,effectiveness or productivity. Often, these skills still rely onhowothers,inadditiontothetrainee,willimplementthem.

In order to truly understand how influential the trainingprocess is, from assessment to evaluation, some use anevaluationtoolcalledOutcome Mapping.Thistoolevaluatesthe training programme based on the change in trainees’behaviour, activities and actions from before and after thetraining. It determines if traineesdidor didnot implementthelessonslearnedintotheirworkplace.Inordertodothisbest, there needs to be a thorough understanding of, andinformation gathered on, how things operated before thetraining and enough time passed to provide trainees theopportunitytoimplementtheirlearningintotheirworkplaces,beforethecomparisonmaytakeplace.Itmaybehelpfultoconsider the OutcomeMapping as two job shadows, onebefore the trainingandthesecondafter,wheredifferencesarecompared,ifthereareany.For more information on Outcome Mapping, see Appendix 18.

organizationalperformancethatthetrainingaimedtoimproveis regularlydocumentedby theclient, these recordings canalsobeasourceofinformationfortheevaluation.

Notonlydomultipletoolsenabletheresponsestobecross-checked, ensuring that the data collected is not skewed,theyalsoensurethatnosingularvoiceorbiaswilloverpowerthe opinions of others. Part of conducting a good trainingoutcome evaluation is giving many individuals opportunityto respond. However, it is not about how many differentevaluation tools are used. More importantly, the emphasismustremainoncarefulselectionintowhichevaluationtoolsareusedandwhichstakeholdersareinvolved.

An evaluator conducted a training outcomeevaluation followinga skills up-grading trainingbyconductinginterviewsandsendingquestionnairestotraineesandadditionalemployeeswithkey relatedpositions. Upon receiving the results, he chose tospecifically focus an additional evaluation tool onthe organization’s internal communication. Whilethis was not something he expected to evaluate,the original results of the large scale evaluationhas indicated that improved communication mayhave been an unexpected benefit of the training.He decides to conduct a group interview withintervieweesfromvariousrelatedpositionswithintheorganization.Hechoseagroupinterviewbecauseitwillquicklyprovidehimwiththorough informationonthenewareaidentified,withouttakinguptimefrommanyoftheclient’semployees.

EVALUATING ORGANIZATIONAL PERFORMANCE

FIGURE13:The role of evaluating organizational performance in a training outcome evaluation


INDIVIDUAL LEARNING

JOB PERFORMANCE





connection to, the training event 10

39LEARNINGFROMTHETRAININGOUTCOMEEVALUATION

LEARNING FROM THE TRAINING OUTCOME EVALUATION10

Toensurethatallpotential lessonsarelearnedandapplied,training institutionsmust continuously apply themselves tolearningand self-reflection.Theneed tocontinuously learnisofcritical importanceas training institutionsmustalwaysbe developing in order to remain on the cutting edge oftheory and practice. Staying relevant to the issues thatgovernmentsface,andhavingtheknowledgetoeffectivelykeepgovernments informedaboutbestpractices,willkeeptraininginstitutionsavaluableresource.

A popular tool for reflection and learning is the ActionLearningCycle,whichcanbeadaptedandappliedtomanydifferentcontexts.Usingthebasicquestionsinthediagrambelow,theactionlearningcyclecanbeusedasaquickreviewofhowthe trainingprogramme, trainingneedsassessmentor training outcome evaluation were delivered, either as awholeorindividually.Withmoretimeallowedandquestionsadaptedtoreflectspecificconsiderations,theactionlearningcyclecanbeusedforsimpleaswellascomplexsituations.Itcanalsobeusedforbothbroadordeepanalyticalassessmentsoftrainingprogrammes’multi-dimensionaloperations.

FIGURE14:The action learning cycle

1

2

3

4 PLANNING ACTION

LEARNING REFLECTION

Howcanweincorporatethelearningintoournextround

ofactivities?

Whathappened?Whatwentwell?Notsowell?

Whatnewknowledge,ideas,understandingor

conceptscanwedrawfromtheexperience?

Whathavewedone?

Appendices

41APPENDICES

APPENDIX 1A: Interview Guidelines

What is it?

Aninterviewismadeupofquestionsaskedverbally(asopposedtoinwriting),eitherinperson,viaphone,oronline(e.g.usingSkype).Notesaretakenoftheanswersandtheinterviewcanalsobeaudioorvideorecordedwithpermissionofthepersonbeinginterviewed,theinterviewee.Interviewscanbestructured,withonlypreparedquestions,orunstructured,todiscussatopicmorefreely.

When to use it?

Interviewsareusefulthroughouttheassessmentprocessandwhenevaluatingindividuallearningaswellas the learning’s impacton jobperformance.Themethodcanbeusedboth toobtainqualitativeandquantitativedata.However,touseinterviewssolelyforquantitativedatacollectionisnotverytimeefficientcomparedtoquestionnaires,unlessthereisreasontobelievethatquestionnaireswouldbemisunderstoodor not sufficiently responded to. Structured interviews are suitable for obtaining qualitative data, forexampletocompareattitudesorknowledgeofpeople,andunstructuredinterviewsforexploringanareaofinterestorfindoutaboutreasonsbehindproblems.Itisoftenusefultocombinetheapproachesinasemi-structuredinterview,withbothpreparedquestionsandroomfordiscussions.See section 3.2 of the manual for the use of interviews in assessmentand section 7.1 and 9.2 for evaluation.

How to use it?

1. Define purpose of interview. Whatinformationissoughtandhowwouldthatcontributetotheassessmentorevaluation?

2. Decide on approach.Isastructured,anunstructuredorsemi-structuredapproachthebestwaytogettothedesiredinformation?Unstructuredinterviewsprovidedeeperinformationonatopicbutaremoretimeconsumingtoanalyze.

3. Select interviewees and make appointments.Mosttimesitisnotpracticaltointervieweveryoneataworkplaceandtheselectionofintervieweesisimportanttogetafairrepresentationofdifferentopinionsandexperiences.

4. Prepare questions. The interview should begin with general questions that are easy for theintervieweetoanswer,andgraduallygetmorespecific.Questionsregardingthepresentareusuallyeasiertoreplytothanquestionsaboutthepastorprojectionsofthefuture.Avoidleadingorguidingquestions such as “Has the staff performance decreased during the last year?”. A betterway toformulatethequestionwouldbe “Has there been any change in staff performance during the last year? If so, how has it changed?”.Togettherequiredinformationdirectquestionsmightnotalwaysbepossible.Forsensitivetopics,suchaswhatmistakesarebeingmadeorwhatrulesareignored,itmightbebettertoaskaboutbehavioursofcolleaguesingeneral.

5. Conduct interview. When conducting the interview it is important to create an environmentwheretheintervieweefeelscomfortable.Ifpossibletheinterviewshouldbecarriedoutinaprivatespacewhere,forexample,theinterviewee’sbossorcolleagueswillnotunexpectedlywalkin.Foranunstructuredorsemi-structuredinterviewitisimportanttofollowuptheanswersbyprobing–askingquestionsonwho,what,why,where,whenandhow–tobetterunderstandthesituation.Itmightbenecessarytohaveanotetakerotherthantheinterviewerortoconsidermakinganaudiorecordingoftheinterview.Remember,beforerecordingsaremade,theintervieweemustfirstgivepermissionfortherecordinganditsuse.

6. Analyze answers. Iftheinterviewhasbeenaudioorvideorecordedatranscriptionfirsthastobemade.Keepinmindfortheplanningthattranscribingtakesatleasttwiceasmuchtimeasthelengthoftherecording.Forstructuredinterviewstheanswerscanbecompiledinatableforcomparisonandidentificationofsimilaritiesanddifferences.ForunstructuredinterviewstheKJ Analysis (appendix 9)isasuitablemethod.

Please see example interview on the following page.


Example Interview

What follows isan illustrationofprobingbyanevaluator toexpandorclarifyapoint.Noticehow,atonepointearly intheinterview,theevaluatorrepeatstheemployee’sstatementwithoutaskinganewquestion.

Employee Let’ssee.ThemainthingIhavebeendoingdifferentlysincecompletingthetrainingiswritingbetterreports.

Evaluator Youfeelyouhavebeenwritingbetterreportssincethetraining?

Employee That’sright.

Evaluator Doyouwritealotofreportsinyourjob?

Employee I’llsayIdo!Atleastonemajorreportaweek.

Evaluator Canyoutellmeexactlywhatmakesyourreportsbetternowthanbeforethetraining?

Employee Yes.Myreportsaremoreconciseandbetterorganizedthanbeforethetraining.

Evaluator Howdoyouaccountfortheimprovementinconcisenessandorganization?

Employee IthinkithadtodowithsomeideasIgotduringthetraining.Theinstructorshowedushowtodiagramourmaterial.Sheputalotofemphasisonshortsentencesandsimplewords.Itmadesensetomethatsayingmorewithfewerwordssaveseverybodyalotoftime.

Evaluator LetmeseeifI’mfollowingyou-youfeelthatyourimprovedreportsaredirectlyrelatedtoideasyougotduringthetraining.Betterorganizationandamoreconcisestyleofwritingsavesyoutime.Hasthisaffectedyourproductiveness?

Employee Noquestionaboutit,Iammuchmoreproductivenow.Icandoareportinabouthalfthetimeitusedtotake.Myreportsareeasiertoreadnowandprobablygetbetterresults.

43APPENDICES

APPENDIX 1B: Competency Interview Template13

See section 3.2 of the manual for the use of competency interviews.

Interviewee information

Name:__________________________________________

Jobtitle:________________________________________

Organization:____________________________________

Telephone:______________________________________

Read this before starting the interview

Puttheintervieweeateasebyengaginghim/herinconversationandprovidinginformationonthepurposeandnatureoftheinterview.Asktheroundofquestiononetofiveuptofivetimesoruntiltheintervieweehasnomorerelevantexamplesofaccomplishments.Askquestionsix,sevenandeightonlyonce;dothisaftercompletingalloftheprecedingfive-questionrounds.

Reminder:Makeliberaluseofprobingquestions,suchaswhy,howandwhen,oruseanyotherfollow-upquestionsthatcometomind.Practicebeinganactivelistener.Makenotesontheinterviewee’skeypointsusingthespaceprovidedundereachquestionandonseparatepaperifnecessary.

Questions

1. Whenyouthinkaboutthethingsyouhaveaccomplishedintheroleof(bespecific)whichoneofthesedoyoufeelthebestabout?

2. Yousayyoufeelbestabout…(summarizeinterviewee’saccomplishment).Tellmewhatspecificknowledge,skillsorattitudes,orcombinationofthese,accountforyoursuccessinthisarea?

3. Asyouseeit,yoursuccessdependson(summarize).Now,explainwhatyouhavehadtolearninordertobeassuccessfulasyouareinthisarea?

13 Adapted from the UN-Habitat manual A Guide to National Training Needs Assessment for Human Settlements: A Competency Based Approach.


4. In your opinion, how can others in the same role learn to be as successful in this area ofaccomplishmentasyouare?

5. Youhavegivenmeaverycompletepictureofthisareaofaccomplishment.Now,Iwantyoutodescribeanotheraccomplishmentinthisrolethatyoufeelgoodabout(continuewithquestion2).

6. Whatchallengesdoyoufaceinyourrole,inparticularregardingmatterswhereyourorganizationcouldprovidebettersupport?

7. Whatnewknowledge,skillsorattitudinalchangesdoyouthinksomeonewillneedinordertoperformsuccessfullyinthisrolefiveyearsfromnow?

8. Whatdoyouthinktraininginstitutionscouldbedoingtoprepareotherstobemorecompetentinthisrole?

45APPENDICES

APPENDIX 2A: Group Interview Guidelines (Focus Groups)

What is it?

Inagroupinterview,orafocusgroup,around6-10peoplearegatheredtodiscussquestionsonaspecifictheme.Amoderator isasking thequestionsandmaking sure thediscussions staywithin theareasofinterest.Theaimisadiscussioninwhicheveryoneinthegroupcanparticipateandwheretheparticipantsbuildoneachothers’answers.Groupsarearrangedinculturallyappropriateways,oftenbygenderandage,toensurethatparticipantscanspeakfreely.

When to use it?

Group interviews are useful during the assessment,when exploring the problemor demand and thecurrent capacityof the client. If carriedout successfully, themethod results in answersbuiltonmanypeople’sopinions,whichissuitableforattaininganoverviewoftheissuesandthecontext.See section 3.2 of the manual for the use of group interviews in assessmentandsection 7.1 for evaluation.

How to use it?

1. Define purpose of discussion.Whattopic is tobediscussedandhowdoes itcontributetotheassessment?

2. Prepare questions. Thequestionsshouldbe“open”,soastoencouragetheparticipantstoelaborateontheanswers,notjustreplywithyesorno.Startwithgeneralquestionsonthetopic,thatareeasytoanswer,andgraduallymakethequestionsmorespecific

3. Select and invite participants. When selecting participants for the group interview both theexperienceofpeople and thegroupsdynamicshave tobe taken into account. There is always ariskofafewpeopledominatingthediscussionandtheresultonlyreflectingtheiropinions.Tokeepthe discussions focused and everyone participating, itmay help to select a rather homogeneousgroup.Culturalaspectsshouldalsobeconsidered.Areemployeescomfortablediscussingwiththeirmanagers?Is itpossibletohavemixedgroups,withbothmenandwomen,oris itbettertohaveseparateinterviewgroups?

4. Conduct interview.Askilledmoderatorisrequiredtokeepthediscussionsfocusedandtoencourageinteractionbetweentheparticipants.Interactioniswhatgivesthemethoditsvalue,thusthegroupinterviewshouldnotbeasetofindividualinterviews.Itmaybenecessarytomakeaudioorvideorecordingsofthegroupinterviewsand/orhaveapersonassignedtotakenotes.Remember,beforerecordingsaremade,thegroupmustfirstgivepermissionfortherecordingsanditsuse.

5. Analyze result.Iftheinterviewhasbeenaudioorvideorecordedatranscriptionfirsthastobemade.Keep inmindfortheplanningthattranscribingtakesat lesttwiceasmuchtimeasthe lengthoftherecording.TheKJ Analysis (appendix 9)isasuitablemethodforanalyzingtheresultsofgroupinterviews.GroupinterviewscanalsoserveasagoodsourceofinformationforaSWOT Analysis (appendix 8).


APPENDIX 2B: Example of Focus Group Discussion14

See section 3.2 of the manual for the use of focus group discussions in assessment.

MinistryFocusGroupDiscussionCapacityNeedsAssessmentLiberia,2009

The discussions brought together in total 20-25 key staff of the four departments in theMinistry ofInternalAffairs.ItwasfacilitatedbyajointteamoftheMinistryandtheUnitedNationsCountySupportTeam,andthedurationofthesessionwastwohours.

InthecapacityneedsassessmentprocessfocusgroupdiscussionswerealsocarriedoutinthreeCountiesinLiberia,accompaniedbyquestionnairesdistributedtostaffintheMinistryaswellasintheCounties.

Objectives of the Ministry of Internal Affaires Focus Group Discussion:

• IdentifyperceptionsofMinistrystaffregardingtheMinistry’stasksandfunctions.

• IdentifyperformancelevelandperformancegapsoftheMinistryinrelationtoitsenvisionedmission.

• IdentifymajorcausesforperformancelevelandperformancegapswithintheMinistryinrelationtoitsenvisionedmandate.

• Identifyoptionsforaddressingidentifiedcapacitygapsandcommitmentstowardsaddressingthose.

Agenda of Ministry of Internal Affaires Focus Group Discussion:

1. OpeningremarksbyDeputyMinister

2. Warm-upexercises

3. Plenary exercise: “What in your opinion are the tasks and functions of the Ministry of InternalAffaires?”

4. Individualexercise:“HowdoyouranktheperformanceofyourMinistryinrelationtotheenvisionedMission?”

5. Groupwork(6-7participantspergroup):“Whatarethekeyfactorscontributingtoperformanceandnon-performanceinyourMinistryinrelationtotheenvisionedmission?”

6. Plenarydiscussion:“Howcantheidentifiedcapacitygapsbebridged?”

7. Follow-upandclosing

14 The focus group discussion was part of the activities to review individual, organizational and institutional capacities influencing local service delivery in Liberia conducted by Ministry of Internal Affairs and UN-Habitat in 2009. The assessment is documented in the Liberia Local Government Capacity Building Assessment and resulted in a 2 year Liberia Local Government Capacity Building Plan. These activities and outputs were possible under the UN County Support Team Joint Programme: Increasing Administrative, Technical and Institutional Capacity for Sub-National Service Delivery within the Republic of Liberia.

47APPENDICES

APPENDIX 3: Observation Guidelines

What is it?

Observationisamethodtostudyaspecificsituationbywatchingitwhiletakingnotesormakingvideorecordings.Itcaneitherbeareallifesituationorasituationstagedforthepurposeoftheobservation.

When to use it?

Observationsareusefulbothforinitialinformationgathering,whenlittleisknownabouttheproblemandcontext,andforevaluationoflearningandjobperformance.Itistheonlywaytostudyactualbehaviours–whatpeopledooftendiffersfromwhattheysaytheydo,andinaddition,manyskillsorproceduresaredifficulttoexplainbuteasiertoshow.However,thismaybeatimeconsumingmethodandthereisnoguarantee that theobservedbehaviour is thesameas the typicalbehaviour. It isalso important torememberthatobservationsgiveinformationonwhatisdone,butitdoesnotexplainwhyitisdonethatway.Tobetterunderstandthesituationitisthereforerecommendedtocombineobservationswithotherdatacollectionmethods.See section 3.2 of the manual for the use of observations in assessmentandsection 7.1and8.3 for evaluation.

How to use it?

1. Define purpose of observation.Whatactivitiesorsituationsaretobestudied?Listthingstolookforandquestionstobeansweredbytheobservation.

2. Book date and time.Makesurethattheobservationisscheduledforadaywhentheactivitiesofinterestcanbestudied.Seekpermissionforvideorecordingsinadvance.

3. Carry out observation.Takenotesduringtheobservationsormakevideorecordings.Thisisarathertimeconsumingmethod.Inaddition,itmaytakeabitoftimeforpeopletofeelcomfortablewithbeingobserved,particularlyiftheyarealsobeingfilmed.

4. Analyze result.Analyzingvideorecordingsisverytimeconsuming,andtosavetimeitisgoodtotakenotesofthekeyfindingsduringtheobservations.Thevideorecordingcanratherbeusedasareferencewhen it isuncertainwhathappened,whenstudyingdetailsofaspecificactivityor tocommunicatewithpeoplenotfamiliarwiththeactivity.


APPENDIX 4A: Questionnaire Guidelines

What is it?

Aquestionnaireisasetofwrittenquestions,eitheronpaperoracomputer.Thequestionscanbe“closed”,withpredefinedanswersthattherespondentscanchoosefrom,or“open”,wheretherespondentscanformulateanswersinthewaytheylike.

When to use it?

Questionnairesaremainlyusedtocollectdatafromalargenumberofpeopleandisparticularlyusefulwhentherespondentsaregeographicallywidespread.Theyareoftencarriedouttovalidateinformationfrominterviewsorobservationsandtocollectquantitativedataonattitudesorknowledgeoftrainees.Byincludingclosedquestions,forexampleaskingtherespondentstoratetheirknowledgeofaspecifictopiconapredefinedscale, it iseasytocomparetheknowledgelevelduringtheassessmentwiththepost-training knowledge level. Hence, questionnaires are useful for assessment aswell as evaluation.Questionnairesalsoprovidethepossibilityofanonymousresponses,whichmaybehelpfulespeciallywhenthetopicissensitive.See section 3.2 of the manual for the use of questionnaires in assessmentandsection 7.1,8.3and9.2 for evaluation.

How to use it?

1. Define purpose of questionnaire.Whatdataistobecollected?Isthispossiblewithoutmeetingtherespondentsinperson?

2. Create and formulate questions.Followthesamestructurewhencreatingaquestionnaireaswhenpreparinganinterview–startwithgeneralquestionsandgraduallymakethemmorespecific.Openquestionsrequiremoreeffortoftherespondentthanquestionswithpredefinedanswers,andalargenumberofopenquestionsmightlowertheresponserate.Ifusingscalesforthequestions,considerwhethertohaveanevenoroddnumberofsteps.Anevennumberofstepsforcestherespondenttotakeastandwhileanoddnumbergivestheoptionofaneutralanswer.

3. Decide on way of distribution. Questionnaires can either be handed out during an event ordistributedbypost, emailorasanonline form.Tohandoutquestionnairesaspartofanactivityusuallyresultsinahigherresponseratethanquestionnairessentbypostoronline.Thereareseveralonlinetoolsforcreatinganddistributingquestionnaires,forexampleGoogledocsforms(docs.google.com), Kwik Surveys (www.kwiksureveys.com), and SurveyMonkey (www.sureveymonkey.com). Anadvantageofusingthesetoolsisthattheyautomaticallycompilethedataaspeoplerespond.

4. Pilot test the questionnaire.Itisdifficulttopredicthowtherespondentswillinterpretthequestions,andpilottestingaquestionnaireiscrucialtomakesurethequestionsareunderstoodcorrectly.

5. Distribute questionnaire. Ifthequestionnaireishandedoutduringanevent,remembertoprintenoughcopiesforeveryone.Forpostaloronlinedistribution,itishelpfultoaddacontactaddressorphonenumberincasetherespondentshaveanyquestions.

6. Send out reminder.Itmaybenecessarytosendoutareminderiftheresponserateislow.

7. Compile data and analyze.Resultsfromthequestionnairescanbecompiledintables.Itishelpfultovisualizequantitativedataingraphsordiagrams.Remembertoplanadditionaltimetocompile,analyzeandevaluateresponsestoopenquestions.

49APPENDICES

APPENDIX 4B: Example of Pre-Training Questionnaire15

See section 3.2 of the manual for the use of pre-training questionnaires.

HowtostrengthengenderaspectsinyourdailyworkGendertrainingforROAPstaffFukuoka,Japan,15-16June2009 Guidelines

15 This pre-training questionnaire was used for a gender training of UN-Habitat Regional Office of Asia and the Pacific in Fukuoka, 2009. It served as training needs assessment for the facilitators to tailor-make the training, and was mandatory for all course participants to complete in order to attend the training.

A. HOW ARE YOU ADDRESSING GENDER IN YOUR DAILY WORK AT THE MOMENT?

1. What are you doing in your projects and programmes in terms of gender mainstreaming, women’s empowerment and gender balance? (Please tick themostappropriateboxbelow intermsofhowoftenactionsaretaken.Foranyextraclarifications,pleaseusethespacebelowthetabletoelaborate.)

Never Rarely Sometimes Often/regularly Always

Inmyofficewedoathoroughgenderanalysisbeforeprojectformulation

Inmyoffice,thereisa‘’gendercheck’’doneonceaprojectdocumentiscompletetomakesurethatgenderissufficientlyaddressed

Inmyoffice,datarelatedtoaproject/programmeissex-disaggregated(male/female)wheneverdataisavailable

Inmyoffice,thereisaregularmonitoringandevaluationofongoingprojects/programmesfromagenderperspective


2. In your office/programme portfolio how many projects include the following: (Pleasetickthemostappropriatebox)

All More than half Less than half Less than a quarter

None

Project/programmeincludestheresultsofgenderanalysis

Project/programmehaveexplicitgenderequalityobjectives

Project/programmetargetingwomen/girlsspecifically

Project/programmetargetingmen/boysspecifically

Project/programmeclearlydefinegenderequalityoutputsandoutcomes

Project/programmereportsusesex-disaggregateddata

Project/programmeincluderesourcesdedicatedtoincreasegenderequality

B. WHAT CAPACITY AND INSTITUTIONAL SUPPORT DO YOU REQUIRE?

3. Have you ever received gender training?(Pleasetickone)

Yes No

4. If yes, please state for how many days the gender training was for, and the main topics covered.

51APPENDICES

5. I at present feel equipped to: (pleasetickthemostappropriateboxforeachquestion)

Fully equipped Somewhat equipped

Not fully equipped

Conductagenderanalysis

Incorporategenderaspectsinaprojectproposal

Ensurethatprojectimplementationisconsistentlygendersensitive(suchasindesigningtraining,selectinginterventionsetc)

Identifyqualifiedgender-sensitiveconsultantsforrecruitment

EasilyincorporategenderdimensionsintheHabitatCountryProgrammeDocuments(HCPDs)andprojectdocuments

EstablishcontactwithrelevantNGOsandotherpartnersintheregionworkingongender

FindrelevanttoolsthatareavailablewithinUN-HABITATtofacilitateworkongender

ExplainwhattheGenderEqualityActionPlan(GEAP)involvesandmyroleasUN-HABITATstaff

Deliverapublicpresentationand/orspeaktocounterparts/partnersongender-relatedmattersinhumansettlements

Evaluateaproject/programmefromagenderperspectiveincludingtheuseofappropriateindicators


6. Do you feel that you can get regular and sufficient backstopping support on work related to gender from the UN-HABITAT Gender Mainstreaming Unit at HQ? Pleaseexplainwhatkindofsupportyoumayhavereceived,andanyadditionalsupportyouwouldliketohave.

7. In your view, are there considerable institutional incentives within UN-HABITAT for you to work on gender (such as promotion opportunities, recognition, budget allocations)?

8. To promote more effective integration and promotion of gender equality in my daily work I need the following knowledge and/or support:

9. In your view, what is the main thing UN-HABITAT needs to do in order to more effectively work on gender?

Thank you for taking the time to complete this questionnaire and we look forward to seeing you at the gender course in June!

53APPENDICES

APPENDIX 4C: Post-Training Evaluation Sheet

What is it?

Thismeasureshowtraineesreacttothetrainingprogramme.Thisisimportanttolearnifthetraineeslikethetrainingandiftheyfounditrelevant.Thisisalsotheiropportunitytoprovidefeedbackforimprovementstothetrainingprogramme.

When to use it?

Usethistoolthroughoutthetrainingprogrammeorattheconclusionofkeysessions,andwhenwishingtosolicitthetrainees’responsestothetraining.See section 7.1ofthemanual.

How to use it?

Deliverthistothetraineesuponcompletionofthetraining.Addasmanytrainersasnecessary.Changetheheadingsandself-assessmenttopicsasneededsothattheyrelatedirectlytothetrainingprogramme.Remembertoadaptthequestionstothespecifictrainingevent.

Forexample,ifoperatingonlinetraining,ortrainingwithanonlinecomponent,makesuretoaskabouttheeaseofthetraining.Wasitdifficultoreasy?Inwhichways?

Please see template on the following page.


Sample Post-Training Evaluation Sheet16

1. Pleaseratethistrainingaccordingtotheheadingsandprovideadditionalfeedbackinthecommentssection.Pleasecircletheappropriatenumbers.

RATINGSCALE: 1=LOW 3=MEDIUM 5=HIGH

2. Givesomethoughttowhatyouknewbeforethistrainingandwhatyoulearnedduringthetraining.Pleasecirclethenumberthatbestrepresentsyourknowledgeandskillsbeforeandafterthetraining.

RATINGSCALE: 1=LOW 3=MEDIUM 5=HIGH

3. Wasthetrainingrelevant?

4. Doyoufeelconfidentintheissues?

5. Howwillyouapplythelessonsfromthetrainingtoyourwork?

6. Whatwerethestrengthsofthetraining?

7. Whatweretheweaknessesofthetraining?

8. Howcanweimprovethetraining?

9. Wasthetraininganappropriatelength?Ifno,wasittoolongortooshort?

10. Whatdidyouthinkoftheinteractiveexercises?

11. Howcanthetrainingsupport(logistics,administration,etc.)beimproved?

16 I-TECH, Training Evaluation - Forms and Questionnaires: Training Evaluation and Learning Self Assessment, 2006. Available at http://www.go2itech.org/HTML/TT06/toolkit/evaluation/forms.html (accessed 2012-05-23). Question 11 is in addition to the original tool as used by I-TECH.

Trainer Name(s) Expertise ClarityCulturally Appropriate

Time Management

Responsiveness to Your Needs

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Comments:

Before Training Self-assessment of Knowledge and Skills Related to: After Training

1 2 3 4 5 Understandingofthekeytopics. 1 2 3 4 5

1 2 3 4 5 Applicationofthekeytopicsintrainee’sjobandworkplace. 1 2 3 4 5

1 2 3 4 5Designinganappropriate (includingproper timeline)plan to implement thekeytopic.

1 2 3 4 5

1 2 3 4 5 … 1 2 3 4 5

1 2 3 4 5 … 1 2 3 4 5

1 2 3 4 5 … 1 2 3 4 5

55APPENDICES

APPENDIX 5: Problem Tree Analysis

What is it?

AProblemTreeAnalysisillustratesthelinkagesbetweenaseriesofproblems.Itisusedtounderstandtheunderlyingproblemsandidentifykeyareastoaddresstheproblem.

When to use it?

Thistoolshouldbeconductedbeforedesigningaprojectorconducingtraining.Whenanissueorproblemispresented, theanalysiswill helpbetterunderstand the context,potentialproblemareas toaddress,andhowaddressingspecifiedproblemareaswillcontributetoreducingtheroutecauseoftheproblemanditseffects.Remember,thebestmethodmaybeapproachesotherthantraining,suchassupportingcommunity-developedandledsolutions.See section 2.1. of the manual.

How to use it?

CreatingaProblemTreeAnalysisislikesolvingapuzzle.UsethefollowingkeystepsasguidelinestoconductaneffectiveProblemTreeAnalysis.

12. Determining the Problem-Areas. Mark down allrelated problems. It is important to be inclusive acrossvarious sectors asmany problems can be cross-cuttingandareoftenrelated.Dothisoversometimetoensurethatallkeypiecesofinformationaremarked.Workwithothers;themoreperspectivesthebetter.

13. Classify Each Problem. It is important to determinewhattherootcauseoftheproblemisandwhichfactorscanbeaddressed.Problemscanbeclassifiedthreeways:Environment,Causes,andEffects.

14. Arranging the Problem Tree.ArrangetheProblemTreetomake themost sensewith environmental factors atthebottom,causes in themiddleandeffectsat the top.TodeterminetheCoreProblem,askwhichproblemisaresultoftheenvironment,yetaffectsallcauses?

15. Determining How & Where to Address the Problem.Identifykeyproblemareasthattheorganizationcanbestaddress.Askthefollowingquestions:

• Howcanthisorganizationreducetheproblemsandtheireffects?

• Istrainingthebestmethodormostappropriatemethod?

• Whatotheractivitiescanbetakenreducetheproblem?

• Which problems do this organization’s strengths relateto?

• Whichproblemscanthisorganizationeffectivelyaddress?

• Isitcertainthatthisorganization’sstrengthswillproducepositiveimpacts?

• Whataretheassumptionsandrisksinvolved?

Core Problem

Visualizing the Classification of Problems

Effects

Existasaresultofthecauses.

Someeffectsarealsocauses,influencingoraffectingfurtherproblems.Ex.Uncleandrinkingwater.

Causes

Thesecauseotherproblemstoexist,andoftendirectlycorrelatetotheenvironment.Ex.Poorinfrastructure.

Environment

Underlyingfactorsthatarebeyondcontrolandcannotchange.Ex.Politicalconflict,poverty,coloniallegacy.


APPENDIX 6: Five Whys

What is it?

This is a process of asking five times, why the problem exists. Think of the problem from differentperspectives,andgatheralltheinformationyoucantodeterminewhattherootoftheproblemis.

When to use it?

Usethistooltodetermineiftrainingisthemosteffectiveapproachtosolvingtheproblem.Itwilldetermineexistingknowledgeoftheproblem,andwhatneedstobefurtherresearched. See section 2.1. of the manual.

How to use it?

Askwhy the problem exists, five different times in five differentways. Each answer leads to a newquestionsothatthequestionsbuildsuccessivelyontheanswerstothepreviousquestion.

Intheexamplebelow,theFiveWhystool isbeingusedtohelpdeterminewhyacertainpopulation isincreasinglyill.

1. Ask a question to identify the initial problem. Question:Why is the population increasingly ill?Answer:...Because they are drinking unclean water.

2. Use this answer to build your next question. Question:Why are they drinking unclean water? Answer:…Because they are drinking from the same pond as their livestock.

3. Use this answer to build your next question.Question:Why are they drinking from the pond?Answer:…Because the borehole is not accessible.

4. Use this answer to build your next question.Question:Why is the borehole not accessible?

Continueuntilyouhavediscoveredtherootoftheproblem.Gointoasmuchdetailasnecessary.

57APPENDICES

APPENDIX 7: Stakeholder Analysis17

What is it?

Stakeholder Analysis is a method of mapping the stakeholders of a project and identifying the keystakeholderstoinvolve.

When to use it?

A Stakeholder Analysis is useful to carry out in the beginning of the training needs assessment. See section 2.2. of the manual.

How to use it?

Firstalistofstakeholdersisbrainstormed.Stakeholdersareallindividuals,groupsororganizationsthatinfluencethetopicoftheassessmentorareinfluencedbyit,includingforexample:

• Specificunitsoremployeeswithintheorganization

• Fundingagencies(e.g.internationalorganizationsorthenationalgovernment)

• Politicians

• Privatesectoractors(e.g.competitors)

• Partners

• Contractorsoftheorganization

• Beneficiariesoftheservicesoftheorganization

• Thetraininginstitutionandtrainers

The second step is to structure the stakeholders in a grid according towhat influence they have ontheprojectandhowinterestedtheyare,seetheexamplebelow.Stakeholderswithbothhighinfluenceandhighinterestaremostimportanttoinvolveintheassessment,butstakeholderswhohaveveryhighinfluenceandlowinterest,orveryhighinterestbutlowinfluencemightalsobeimportanttoinvolve.Howmanystakeholderstoinvolvedependsontheextentoftheassessmentandavailableresources.

Example of stakeholders in a Participatory Budgeting training,who in differentways influence or areinfluencedbythebudgetingprocessofthelocalgovernment:

17 Adapted from mindtools.com. For more information on how to analyze stakeholders, see: http://www.mindtools.com/pages/article/newPPM_07.htm

Mayor

Low

High

Low High

INFLUENCE

INTEREST

Involve in assessment

Head of Finance Department

Trainees’ colleagues

Potential trainees

Funder

CBOs

Youth groups

Other local government departments (health, infrastructure etc.)

Contracted accounting firm

Ministry of Finance


APPENDIX 8: SWOT Analysis

What is it?

Usefultoscreenthecontextsurroundingtheproblem,theSWOTAnalysiscanhelpdeterminewhetherornot trainingcan solve the issue.This tooldetermines the internalStrengthsandWeaknessesofanorganization, while analyzing external Opportunities and Threats effecting the organization. Thisinformationisusedtounderstandhowthesefactorsplayaroleinthewaytheorganizationperforms.

When to use it?

Whenpresentedwithaproblemor trainingopportunity, conductaSWOTAnalysis tounderstand thecontextsurroundingtheorganizationandhowtorespond.Theknowledgegatheredwillenabletrainerstomake appropriate decisions thatwill build upon the strengths andopportunities of trainees,whileaddressingtheirweaknessesandfocusingontheirabilitytoperforminlightofthethreatstheyface.Mostimportantly,theSWOTAnalysiswillhelpdeterminewhetherornottrainingwillsolvetheproblematall.See section 2.3. of the manual.

How to use it?

Work through the key questions in the diagrambelow tomake sure all the necessary information iscollectedtoproduceaneffectivetrainingstrategy.Askadditionalquestionsasneeded.

Opportunities• What good opportunities are

opentotheorganization?

• Whattrendscantheorganizationbecomeinvolvedin?

• How can the organization’s andtrainees’strengthsbeturnedintoopportunities?

Ex. The organization can:

• Meetnewpotentialpartners

• Meetnewpotentialdonors

Internal

Characteristics of the

Organization

Strengths• What do the trainees already

know?

• Whatdothetraineesdowell?

• What unique resources areavailable?

Ex. Trainee’s organization has:

• Goodinternalcommunication

• Respectinthefield

• Manylocalpartners

Weaknesses• Whatcouldtraineesimprove?

• Where do trainees have fewestresources?

• What do the trainees see asweaknesses?

Ex. Trainee’s organization has:

• Irregularannualfunding

• Lackofresources

• Fewpersonnel

Threats• What trends could harm the

organization?

• Whatisthecompetitionorotherorganizationsdoing?

• What threats do theorganization’s weaknessesexposeitto?

Ex. The organization may face:

• Securitythreats

• Increasingcostofoperation

• Changingdonorpreferences

Positive Effect

Towards achieving the objective

External

Characteristics of the Environment

Negative Effect

Towards achieving the objective

59APPENDICES

APPENDIX 9: KJ Analysis18

What is it?

KJAnalysisisamethodtostructureandgetanoverviewofqualitativedata.PreferablytheKJAnalysisisconductedbyagroupof3-7people.Itisnotnecessarythateveryonehastakenpartincollectingthedata.

When to use it?

Themethodcanbeusedduringtheassessment,forexampletooverviewtheproblemandtheclient’scurrentcapacities.TheKJAnalysisisparticularlyusefulwhenanalyzingdatafrommanydifferentsourcesofinformation.See section 3.3 of the manual.

How to use it?

1. Select data.Writestatementsorfactsfromthedatacollection.Anexamplewouldbetousepost-itnotes (one factperpost-it).Most likely there isa largeamountofdataavailable,and toavoidmakingtheanalysisunnecessarilytimeconsuming,onlyincludethefactsthataremostrelevanttotheinvestigatedissues.

2. Organize data.Inthegroup,reviewthenotesonebyone.Placethefirstonabigpieceofpaper,ideallyonthewallsothateveryonecanseeit.Continuewithmorefactsandplacethemaccordingtohowtheyareperceivedtoberelated.Theperceptionofrelationsmightchangewhennewfactsareadded.Rearrangethenoteswhennecessaryandgraduallygroupsandsub-groupsoffactswillbeformed.Thenotescanalsobearrangedhierarchically,forexamplewhenanalyzingtherootsofaproblem(asisoftendonewhenconductingaProblemTreeAnalysis).

3. Identify themes.Namethegroupsoffactsaccordingtowhatthemestheyarerelatedto.

4. Document result.Photographthenotestomakeadigitalsummaryofthegroupsandkeyfacts.Forexample,onemightusetextboxesinWordorPowerPoint.Thesummaryisusefulwhensharingtheresultswiththeclientorpeopledesigninganddeliveringthetraining.

Please see examples on the following page.

18 The KJ analysis, or the KJ method, is named after the Japanese professor Jiro Kawakita (KJ are his initials, starting with his last name). He developed the method in the 1950s as a way of analyzing large amounts of qualitative data, mainly collected during interviews and conversations. Read more about Jiro Kawakita on the Ramon Magsaysay Award Foundation’s website: http://www.rmaf.org.ph/Awardees/Biography/BiographyKawakitaJir.htm


Example of KJ Analysis

ExampleoffactnotesstructuredinaKJAnalysis:

ExampleofaKJAnalysissummarycreatedinPowerPoint:

61APPENDICES

APPENDIX 10: Learning Contract

What is it?

Learning contracts can make explicit the important learning to be pursued during training and thebehaviourstobeachievedasaresultoftraining.Itisusuallyanagreementbetweenthetraineeandhisorhersupervisor,statingtherequiredcommitmentsofbothinorderforthetraineetoachievethelearningobjectives.See section 4.2and8.3 of the manual.

When to use it?

Thelearningcontractshouldbesetupbeforethetraining,whenthelearningobjectiveshavebeendefined.

How to use it?

The trainee and supervisor prepare the learning contract together, in collaboration with the trainingprovider.Itcanlater,bothbeforeandafterthetraining,supportmeetingswherethetraineeandsupervisordiscussthetrainingandprogressinachievingthelearningobjectives.

Exampleofapre-traininglearningcontractbetweenatraineeandhis/hersupervisor:

Employee: I (name) _____________________________ request approval to participate in a trainingprogrammeon(topic)_________________________________________________.

Thelearningobjective(s)Iwillachievebyattendingthisprogrammeis/are:_____________________________________________________________________________________________________________________________________________________________________________________________________________

Ifacceptedforthetraining,Iwill:

• Attendallscheduledsessionsandactivities

• Completeallpre-work,readingandotherassignments

• Participateactivelyinalltrainingactivities

• CreateadetailedactionplandescribingwhatIintendtodotomakeeffectiveuseof…..workshoplearningtoachievemylearningobjective(s)

• Discussthisactionplanwithmysupervisor

• Discusshighlightsoftheworkshopwithmyco-workers

Signed__________________________Date_________________________

Supervisor:I,(name)________________________________,supervisorofthisemployee,willsupporttheemployeeinachievinghis/herlearningobjectivesinthefollowingways:

• Releasetheemployeefromworkassignmentstoallowcompletepreparationforandattendanceatallscheduledworkshopsessionsandactivities

• Meetwiththeemployeeaftertheworkshoptoreceiveinformationonworkshopcontentandreviewtheemployee’splanforusingworkshopresultstoachievetheemployee’slearningobjective(s)

• Reinforceandencouragetheapplicationofnewtechniquesandbehaviourslearnedattheworkshop

• Provideopportunitiesfortheemployeetoapplynewtechniquesandbehaviours

Signed__________________________Date_________________________

Learning Contract


APPENDIX 11: Performance Contract

What is it?

Thistoolisusedtohelpthetraineeandtraineragreeuponlearningobjectivesandspecificoutcomesofthetrainingprogrammebeforeitbegins.Itisusedbeforeconductingthetraininginordertodevelopandensurethatbothpartieshaveacommonunderstandingofthetrainingprogrammegoals,eachother’sexpectationsandresponsibilities.Afterthetrainingprogrammeconcludes,thetraineeandtrainerseeifandhowthetrainingachieveditsgoals.

When to use it?

Thistoolisusedbeforethestartofthetrainingprogramme,duringthedesignphase,sothattheevaluationcanbebasedonthecontractoragreementbetweenthetraineeandtrainer.Thesecondpartmustbedoneafterthetrainingprogrammeends.See section 4.2 of the manual.

How to use it?

Developacontract,similartotheexampleonthefollowingpage,beforethetrainingbegins.Haveboththetraineeandtraineragreetotheirresponsibilitiesandtheobjective.

Please see example template on the following page.

63APPENDICES

Sample Performance Contract

Before Conducting the Training

What does the trainee and their workplace expect the trainee to learn from the training programme?

How will the training programme improve the trainee’s performance at their workplace?

Why does the trainee’s participation in this training programme seem to be a good investment of time and money for their organization?

How will the trainee’s participation contribute to the training programme?

Stated Agreement:

I(thetrainer)promisetodomybestwithinthetrainingprogrammeandidentifiedtrainingthemestohelpthetraineeattaintheabovestatedexpectations.

TrainerSignature Date

I(thetrainee)promisetoattend,participateandapplytothebestofmyabilitiestheknowledgegainedinthetrainingprogrammetoattaintheabovestatedexpectations.

TraineeSignature Date

After Conducting the Training

How has the trainee’s specific areas of knowledge, skill, attitude and behaviour changed as a result of the training programme?

What did the trainee learn that they did not know before?

In what specific ways did the trainee’s workplace benefit from the trainee’s participation in the training programme?

What are the trainee’s challenges to implementing their new learning, into their workplace?

In what specific way can the trainer further support the trainee?


APPENDIX 12: Results Chain

What is it?

TheResultsChainisaboutdeterminingthespecifictrainingactionsrequiredtomeettheobjectivesofthetraining.

When to use it?

UsetheResultsChainwhendesigningthetrainingprogramme.Thistoolwillhelpdeterminehoweachstepcontributestoachievingthedesiredimpactontrainees.See section 4.4 of the manual.

How to use it?

Followthebelowexampletodeterminetheimpacttrainingwillhave,andwhatactivitiesarenecessarytoachievethem.

IMPACT*

Example:

Thereisincreasedawarenessandimprovementinthewaythetrainees’organizationworks.

*Remember,itisverydifficulttoproveandmeasuretheimpactoftrainingbecausethe ability to apply training andwhat has been learnedwill be effected by otherpositiveandnegativevariables.

OUTCOME (Learning Objective)

Example:

Traineesareeffectivelyapplyingthelessonslearnedtotheirworkplace.

OUTPUTS

Example:

• Traineeshaveincreasedknowledgeandunderstandingofthekeyissues

• TraineeshaveActionPlansthatwillenablethemtoeffectivelyapplythelessonslearnedtotheirworkplace.

ACTIVITIES/RESOURCES

Example:

• Educatethetraineesonkeyissues

• Providematerialtosupportthelearning(graphs,documents,visuals,PowerPoint,etc.)

• Reinforcethelessonslearnedthroughexerciseswherethetraineescantangiblyexperiencetheirlearnedactivities(sitevisits,participantdemonstrations,etc).

• Supportandencouragetraineestodeterminehowtheywillapplythelessonslearned (ex. trainees write an Action Plan demonstrating how they willincorporatetheirnewlearningintotheirworkplace).

Start here!Readthechartfromthebottomtothetop.

65APPENDICES

APPENDIX 13: Follow-Up Plan

What is it?

Thistoolisusedinconjunctionwiththetoolsusedtodesignthetrainingprogramme.Itcomparestheexpectationsofeachlearningexerciseandtheirspecificlearninglessonswithitsperformanceandwhattraineesactuallylearned.

When to use it?

Develop the Follow-Up Plan before implementing the training.Use this toolwhen determining if thetrainingmethodswereeffectiveandbenefitedthetrainees,helpingthemmeettheirlearningobjectives.See section 4.4 of the manual.

How to use it?

Whendeterminingwhat exerciseswill be used during the training period to teach key lessons,writedown the expectation of each exercise into the Follow-Up Plan. After conducting each exercise, seehowitcomparestowhatwasexpected.Didtheexerciserunsmoothly?Didtraineesrespondpositively?Didtraineesrememberthelessonbytheendofthetraining?Didtraineesincorporatethelearningintoadditional elements of the training programme? How can the trainers further enhance the learningexercises?

FAO (2011), “Learning Module 3, Toolbox 6: Provide follow up support.” In Good Learning Practices for Effective Capacity Development. 8 August 2011 – draft.



Sample Follow-Up Plan

Learning Initiative:

Ex.TrainingforCommunityEntryintocommunitieswithdifferentcustomsandvalues.

Delivery Date:

Ex.7-April-2011

Identified Learning Exercises

1.OpenDiscussion 2.SiteVisits 3.RolePlay

Lessons to be Learned

1.Therearemanydifferentdefinitionsofcommunityentryandvalues,etc.

2.Themanydifferentwaysofapplication(communityentry),hencethedifferentdefinitions.

3.Tryingtoimplementsomeofthedifferenttechniquesbasedonrandomlydrawncommunitycharacteristics,oroncasestudies.

Trainees Response to Learning Exercise

1.ModeratelyEngaged 2.ActivelyEngaged,verypositivefeedbackinthemoment.

3.Mosttraineeswereactivelyengaged,lessthanonequarteroftraineesrespondednegatively.

What was the Culminating Activity?

TraineeswroteindividualActionPlansforhowtheycouldsharethelessonslearnedintotheirworkplace.

Did Trainees Remember and Correctly Apply the Lessons Learned from each Learning Exercise in the Culminating Activity?

Explainkeystrengthsandweaknesses.

How can the Learning Exercises be Further Enhanced?

Explainwheretraineesspecificallystruggledandwhattacticsweresuccessfulinhelpingthemunderstand.

Timeframe: When Is Additional Follow-Up Provided to Trainees at Their Workplace?

Follow-upattwotimes:6monthsand9months.Finalevaluation:1year.

Budget: Have the Appropriate Allocations Been Approved?

Checkforapprovalofdivisionandofdonors.

67APPENDICES

APPENDIX 14: Five “W“ Interview

What is it?

Semi-structuredFive“W”Interviewisusedtogatherinformationfromtrainees.Theindividualconductingthefollow-upaskstrainees,oneonone,toanswersimple(non-leadingandunbiased)questionsabouttheeffectivenessandusefulnessofthetraining.

When to use it?

Usethistoolafterthetraininghastakenplaceandoncetraineeshavehadachancetobeingimplementingthelessonslearned.See section 4.4 of the manual.

How to use it?

AskquestionsaboutthetrainingandlessonslearnedusingtheFive“W”Interviewstructure.RememberthattheFive“W”sareoftencomplimentedbyasking“How?”.Bepatientandaskforhonestanswers.

Someexamplequestionsinclude:

1. WHYdidyouattendthetrainingevent?

2. Didyoudolearnatthetrainingevent?Ifyes,WHATdidyoulearn?

3. WHENdidyouunderstandthelessonsbeingtaught?(Thewholetime/mostofthetime/halfofthetime/partofthetime?Duringwhichactivities?)

4. WHObenefitedfromyourbeingtrained?

5. Didyouapplywhatyoulearnedtoyourworkplace?Ifyes,HOWdidyouapplyis?Diditchangethewayyouand/oryourcolleagueswork?

APPENDIX 15: Learning Improvement Plan

What is it?

TheLearningImprovementPlanisanagreementbetweenthetraineesandtheirsupervisors,aimingtosupportthetraineesinapplyingtheirlearningtotheirwork,inordertoimproveperformance.Itstatestheperformancegoalofthetraineesaswellasresourcesneeded,potentialbarriers,andactionstotaketoovercomethebarriers.

When to use it?

TheLearningImprovementPlanisformulatedafterthetrainingandservesassupportduringthefollow-upphaseofthetraining.Itcanalsobevaluabletostudythelearningimprovementplanwhenevaluatingthetrainingprogramme.See section 4.4 of the manual.

How to use it?

Thetraineeandhisorhersupervisorshouldpreparethelearningimprovementplantogether,withsupportfromthetrainingprovider.Refertothetrainingneedsassessmentandlearningobjectiveswhenwritingthegoalofthetrainee,indicatorsofsuccess,outcome,barrierstolearningbeingimplementedinthejobetc.



Template of a post-training learning improvement plan between a trainee and his/her supervisor

Area of Improvement

WhattasksorfunctionsamIdoingnowthatIwanttododifferentlyorbetterbasedonwhatIhavelearnedintraining?(Describetasksorfunctions)

Goal

WhatspecificchangesinperformingthesetasksorfunctionsdoIhaveinmind?BywhendoIplantoaccomplishthis?(Describewithdate)

Success Indicators

HowwillIknowwhetherornotandhowwellIhavesucceededinachievingthisgoal?(Explain)

Resources

Whatassistance(fromwhom)andresources(time,money,equipment)doIneedtoimplementmyimprovementplan?(Specify)

Barriers

WhatobstaclesorbarriersdoIexpecttoencounter(frominsidemeorfromexternalsources)inimplementingmyimprovementplan?(Explain)

Action Plan

WhatspecificstepsmustItaketoavoidordealwitheachoftheseobstaclesorbarriers?(Specify)

Outcome

WhatcontributionswillImaketomyjob,theorganization,andtheorganization’sservicesbyachievementofmylearningimprovementplan?(Describe)

Commitment

Bysigningthiscontract,weagreetomakeacommitmentoftimeandmoneytocarryoutthislearningimplementationplan.Weagreetomeetevery__________monthstoreviewprogressandmakechangesasneeded.

Nametrainee____________________________________Signature____________________________________________

Namesupervisor_________________________________Signature____________________________________________

69APPENDICES

APPENDIX 16: Job Shadow Performance Testing

What is it?

Thistoolisatestthatallowstheindividualconductingthefollow-uptodetermineifthetraineeremembersandappliesthelessonsfromthetrainingintotheirdailylifeattheworkplace.

When to use it?

Usethistoolafterthetraininghastakenplaceandoncetraineeshavehadachancetobeginimplementingthelessonslearned.See section 9.2 of the manual.

How to use it?

Unlikewrittentests,theindividualconductingthefollow-updoesnotrevealwhattheyarelookingfortothetrainee.Instead,theyjointhetraineeonthejob,andobservehowtheywork.Remembertolookatbehaviourandforapplicationofthelessonslearned,notjustwhetherornotthetraineeunderstandstheconcepts.

Lookforanswerstoquestionssuchas:

• Doesthetraineerememberthelessonslearned?

• Doesthetraineeapplythelessonslearnedfromthetrainingprogrammeintotheirworkforce?

• Hasthetraineesharedaboutthelessonslearnedfromthetrainingprogrammetocolleagues?

• Ifthetraineewroteanactionplan,hastheactionplanbeenimplemented?Ifso,hasitbeensuccessful?Ifnot,hasthetraineemodifiedit,orgivenupontheconcepts?

APPENDIX 17: Action Plane

What is it?

AnActionPlaniscreatedbyindividualtrainees,orateamoftraineesfromthesameorganization,outliningspecificstepsthattheywillimplementwhenreturningtotheirrespectiveworkplace.Thesestepsleadtoaparticulargoalandmayincludeconcretethingswithinthetrainee’smandate,suchasimplementationofspecificchanges,advocacytoparticularsupervisors,communicatingknowledgeonanewprocesstoapartner,etc.

When to use it?

UsethistooltocreatetheActionPlanduringthetraining.TraineesbegintoimplementtheActionPlanoncereturningtotheirworkplace.TrainerswillusetheActionPlantomonitortheprogressofthetrainees’implementationaswellastofollow-upandevaluateonwhichspecificitemsthetraineeshaveandhavenotimplemented.See section 9.3 of the manual.

How to use it?

Havethetraineeidentifywhatchangesneedtooccurforthelessonslearnedtobeimplementedintotheirworkplace.Outlinespecificstepsnecessarytomakethesechangesoccur.Considerthefollowingpoints:

• Whoneedstosupportthechanges?

• Howwillthechangesbecommunicated?

• Whatwillthenewprocesslooklike?

• Will current processes, logistics and administration procedures be disrupted? How can a smoothtransitiontothenewprocessoccur?

Please see example Action Plan on the following page.


Example Action Plan

Overall Intervention Strategy

Statement by trainee: Uponreturningtomyworkplace,Iwilldiscusswiththeappropriateindividualsandinstitutionsthefollowingstrategyforimprovingthequalityandavailabilityoflocalgovernmenttrainingopportunitieswithinthesphereofmyinstitution’sinfluence:

1. Statement of overall strategy

A.Implement,inmyowndepartment,methodsusedtoconductthiscourse.

B.ContinueworkonthethreeproblemswhichIpresentedatthestartofthecourse:(1)maintenanceofflats;(2)marketproblem;and(3)publicrelationsinthecityhall.

C.Continueworkonanurbanplanningandhousingstrategywhichcouldhaveaprofound,long-termimpactonfutureplanninginRomania.

2. Identify the results expected to be achieved through strategy implementation

A.Moreefficientteamwork

B.Involvemydepartment’semployeesinresolutionofthethreeproblems

C.Developandrunatrainingcoursespecificallyforpeopleinvolvedinurbanplanningandmanagement

3. List the stakeholders who need to be involved if the strategies are to be implemented successfully

A.Mayor;managerofurbanplanningdepartment

B.Allthosewhoareinterestedinthesesubjectsandofficialsoftheresponsiblecityhalldepartments

C.TheMinistryofPublicWorksandregionalplanningoffices;architectswhohaveparticipatedinthiscourse

4. Identify the stakeholder (individual or organization) who will take the lead responsibility to ensure

A.ChiefoftheUrbanPlanningDepartment(thetraineewritingthisActionPlan)

B.Threecoordinatinggroupsorganizedfromappropriatedepartmentsatthecityhall

C.PerhapstheMinistryofPublicWorks,regionalplanningand/ortheassociationofmayors

5. Propose a timetable for implementation

A.Immediatelyonreturntowork

B.January/February(3-4months)

C.Byendofnextyear(14months)

6. List the resources needed to carry out the strategy

A.Peoplewhowanttolearnnewmethodsofwork

B.Peopletotakepartinproblem-solvingsessions

C.Sourcesoffinancing(sponsors);peopleforthetrainingteam;peoplewhowanttotakepartinthetrainingprogramme

7. List some initial tasks that need to be undertaken if the strategy is to be implemented successfully

A.Explainmethodsofwork;involvepeopleinbrainstormingsessions(problemsolving);questionpeopleaboutwhattheylike/donotlikeabouttheirwork.

B.Trainthekeystakeholderstoimprovetheirabilitytoresolvethethreeproblems;offerthemalternativestoconsider;facilitatetheiruseofproblemsolvingmethods.

C.Sendaproposaltovariouskeycityhallsaskingfortheirinterestinpursuingtheproposedcourseofaction;collectresponsesandreviewthem;sendasummaryoftheresponsestotheMinistriesofRegionalPlanningandPublicFinancewitharequestforfinancial/planningassistance;sendinvitationstoparticipate(eventually)toformercommunist-blockcountries;locateavenueforthetrainingcourse;identifysponsors;organizethetrainingteam.

71APPENDICES

APPENDIX 18: Outcome Mapping

What is it?

Thistoolevaluatesthetrainingprogrammebasedonthechangeintrainees’behaviour,activitiesoractionsposttraining.Itdeterminesiftraineesdidordidnotimplementthelessonslearnedintotheirworkplace.

When to use it?

Usethistoolaftertheconclusionofthetrainingprogrammeandafterenoughtimehaspassedtodetermineifthetraineesdidordidnotimplementthelessonslearned.See section 9.4 of the manual.

How to use it?

Check iftraineesdidordidnot implementthe lessons learned. Iftheydidnot implementthe lessonslearned,trytodetermineifitwasbecausetraineesdidnotunderstandthelessons,oriftheydidnotknowhowtoapplyit.Moreinformationisavailablefromthesourcesbelow.

Overseas Development Institution (ODI): Research and Policy in Development: http://www.odi.org.uk/rapid/tools/toolkits/Outcome_Mapping/Index.html

InternationalDevelopmentandResearchCommittee(IDRC).VariousdocumentsaboutIDRC’sOutcomeMappingmodelareavailableat:http://web.idrc.ca/en/ev-26586-201-1-DO_TOPIC.html

BuildingLearningandReflectionsintoDevelopmentPrograms:http://web.idrc.ca/openebooks/959-3/

United Nations Human Settlements ProgrammeP.O. Box 30030, GPO Nairobi, 00100, KenyaTelephone: +254 20 762 [email protected] www.unhabitat.org/publications

Manual Training Needs Assessment and Training Outcome Evaluation In an Urban Context

Assessing training needs and evaluating the outcomes of training are crucial, both for trainingprovidersandrecipientsoftraining,toensurethattrainingiseffectiveandresourcesarewiselyused.Despitetheimportanceofassessmentandevaluation,thesestepsareoftentheleastprioritizedofthetrainingcycleastheyareperceivedascomplicatedandcostly.Thismanualsupportstrainingproviders with practical and affordable methods and tools for training needs assessment andtrainingoutcomeevaluation,ultimatelyaimingtoincreasethevalueoftraining.Themethodsandtoolsareaccompaniedbybasictheoryonassessmentandevaluationaswellasideasonhowandwhytoengagefundersandotherstakeholdersintheprocess.

1

5

ASSESSMENT

EVALUATION


lessonslearned




Partners


Beneficiaries

HS/083/12EISBN:978-92-1-132497-6

Date post:	25-Feb-2021
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times