36
Will Thalheimer, PhD Capturing Learning Data: Secrets to Improvement and Achieving Maximum Impact Slides available at: www.is.gd/will999stuff
Will Thalheimer, PhD
Capturing Learning Data: Secrets to Improvement and Achieving Maximum Impact
Slides available at: www.is.gd/will999stuff
Consult
Educate
Evaluate
Bridging Gap between Research and Practice
Bridging Gap between Research and Practice
Research Translation
Bridging Gap between Research and Practice
https://WorkLearning.com/catalog/
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan University
& Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
34‐Page Report
QuestionWhen designing learning, should we base our learning
methods on the preferences of our learners?
A. Learners are FAIRLY GOOD JUDGES of their own learning, so we should trust their learning‐design intuitions.
B. Learners are NOT OFTEN GOOD JUDGES of their own learning, so we should utilize other learning factors.
C. Learners are SOMETIMES GOOD JUDGES and SOMETIMES POOR JUDGES of their own learning.
There is no perfect measurement tool!
“Our measures are not perfect, but they should instead be thought of as approximations.”
Deborah L. Bandalos, Author of the 2018 book:
Measurement Theory and Applications for the Social Sciences
How would you rate your learner‐feedback questions?
Terrible
1
Bad
2
Okay
3
Good
4Given, that there is NO perfect
measurement…
Smile Sheetsto
Learning
r=.09
Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997).
A meta-analysis of the relations among training criteria.
Personnel Psychology, 50, 341-357.
Smile Sheetsto
Learning
r=.09
Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D.
(2008). A review and meta-analysis of the nomological network of
trainee reactions. Journal of Applied Psychology, 93, 280-295.
1990’s 2000’s
Weak Relationship is below .30 and .09 is VERY WEAK
So…SMILE SHEETS tell us VERY LITTLE about Learning
2017 – University Teaching
r = .20
“Despite more than 75 years of sustained effort, there is presently no evidence supporting the widespread belief that students learn more from professors who receive higher… ratings.”
Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree
54321
Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:
On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):
“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)
Research shows that learnersdon’t always know their own learning…
Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).
Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).
Learners Fail to Properly Use ExamplesRenkl (1997).
Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).
Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).
Performance-FocusedSmile Sheets
http://is.gd/TrainingResearch2012
http://is.gd/ddResearch
1. Red‐flagging training programs that are not sufficiently effective.
2. Gathering ideas for ongoing updates and revision of a learning program.
3. Judging strengths and weaknesses of a pilot program to enable revision.
4. Providing instructors with feedback to aid their development.
5. Helping learners reflect on and reinforce what they learned.
6. Helping learners determine what (if anything) they plan to do with their learning.
7. Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the training and/or the instructors.
8. Upholding the spirit of common courtesy by giving learners a chance for feedback.
9. Enabling learner frustrations to be vented—to limit damage from negative back‐channel communications.
Modified based on work by Robert Brinkerhoff
Reasons for Smile Sheets
Effectiveness
of the
Learning
Reputation
of the
Learning
Supporting Learners
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.
A. I am NOT AT ALL ready to use the skills taught.
B. I have GENERAL AWARENESS but will NEED MORE GUIDANCE to put the skills into practice.
C. I need MORE HANDS‐ON EXPERIENCE to be GOOD at using these skills.
D. I am FULLY COMPETENT in using these skills.
E. I am CAPABLE at an EXPERT LEVEL in using these skills.
A Better Smile Sheet Question
0 10 20 30 40 50 60
HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.
Unacceptable
Acceptable
Alarming
Superior
Percentage of Respondents
Superior/Overconfident?
NOT AT ALL READY
HAVE AWARENESS,NEED MORE GUIDANCE
NEED MOREHANDS ON EXPERIENCE
FULLY COMPETENT
COMPETENTAT EXPERT LEVEL
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
After the course, when you begin to apply your new knowledge at your worksite, which of the following supports are likely to be in place for you? Select as many items as are likely to be true.
A. I will have my PROGRESS MONITORED BY MY SUPERVISOR in applying the learning.
B. I will have someone available TO COACH OR MENTOR ME in applying the learning.
C. I will have easy access to a COURSE INSTRUCTOR to contact for guidance and support.
D. I will have JOB AIDS to guide me in applying the learning to real job tasks.
E. I will be PERIODICALLY REMINDED of key learning concepts/skills over the next few months.
F. I will NOT get much direct support, but will rely on my own initiative.
A Question About Follow-Through…
But won’t learners dislike
these new type of
questions?
Percentage saying NEW QUESTIONS EQUAL OR BETTER:
90%
Percentage saying NEW QUESTIONS
BETTER:
80%
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
What aspects of the training made it MOST EFFECTIVE FOR YOU? What should WE DEFINITELY KEEP as part of the training?
Three Open-Ended Questions to End Your Survey
What aspects of the training COULD BE IMPROVED?
Remember, your feedback is critical, especially in providing us with constructive ideas for improvement.
Is there anything else we should have asked about? Is there anything you want to tell us?
Smile Sheets should be ONLY ONE PARTof our learning evaluation efforts
Smile Sheets
Understanding
Remembering
Motivation to Apply
After Supports
Meeting Target Goals?
Job Performance
Organizational Results
Learner Expectations
Other Expectations
Supports:
Management Support?
Workplace Obstacles?
Reinforcement?
Reminders?
Learners able to:
Understand?
Remember?
Make Decisions?
Apply the Learning?
What Messages Are Being Sent?
What Messages Are Being Sent to the Online Trainer?
Compared to most webinars (online meetings), how well did the session keep
YOUR attention? Select ONE choice.
‐‐‐
A. I had a HARD TIME STAYING FOCUSED.
B. My attention WANDERED AT A NORMAL LEVEL.
C. My attention RARELY WANDERED.
D. I was very much SPELLBOUND throughout the session.
RESEARCH/EVIDENCE
For each column,
SELECT ONE CHOICE.
HOW MUCH RESEARCH/EVIDENCE
WAS SHARED?
⃝ ZERO OR VERY LITTLE RESEARCH/EVIDENCE
was shared.
⃝ A SMALL AMOUNT OF RESEARCH/EVIDENCE
was shared.
⃝ A SUBSTANTIAL AMOUNT OF RESEARCH/
EVIDENCE was shared.
QUALITY OF RESEARCH/EVIDENCE SHARED?
⃝ I DON’T KNOW THE RESEARCH well enough to
answer this question.
⃝ Too often, the concepts presented SEEMED TO
CONFLICT WITH THE MOST‐TRUSTED RESEARCH.
⃝ The concepts presented WERE GENERALLY
SUPPORTED BY SOLID RESEARCH.
What Message Is Being Sent?
eLearning
Self‐StudyLearning Program
Employee Exit Survey
I am clear about what is expected of me as a result of going through this training.
A. Strongly DisagreeB. DisagreeC. NeutralD.AgreeE. Strongly Agree
Good
Okay
Bad
Things
No Time For
Use Fewer
Questions
You can Rotate
Questions
You can do A‐B Testing
Give half new
questions, half old
Tailor Your Questions
Get Stakeholder Feedback
Iterate!Get Expert Feedback
BETA
Hendrick had been CLO for three months and had begun to wrest control of the thrashing octopus that was his company’s learning‐and‐development department. He started by getting learning evaluation under control, based on the dictum “what gets measured, gets managed.”
The first pilot was with the strategically‐important Leadership for High‐Potentials course. The new evaluation strategy involved the following:
• Improving the evaluation questions asked of learners by getting rid of the old questions and only asking questions that target Level 3 (work improvement) and Level 4 (organizational impact).
• Adding a Net Promoter Score (NPS) question which asks learners to rate the likelihood that they would recommend the learning intervention to their coworkers.
A. NOT AT ALL effective.
B. SOMEWHAT effective.
C. MOSTLY effective.
D. VERY effective.
What do you think Sandra, a world‐class learning evaluation expert, will tell Hendrick about the effectiveness of his evaluation design?
Hendrick had been CLO for three months and had begun to wrest control of the thrashing octopus that was his company’s learning‐and‐development department. He started by getting learning evaluation under control, based on the dictum “what gets measured, gets managed.”
The first pilot was with the strategically‐important Leadership for High‐Potentials course. The new evaluation strategy involved the following:
• Improving the evaluation questions asked of learners by getting rid of the old questions and only asking questions that target Level 3 (work improvement) and Level 4 (organizational impact).
• Adding a Net Promoter Score (NPS) question which asks learners to rate the likelihood that they would recommend the learning intervention to their coworkers.
A. NOT AT ALL effective.
B. SOMEWHAT effective.
C. MOSTLY effective.
D. VERY effective.
What do you think Sandra, a world‐class learning evaluation expert, will tell Hendrick about the effectiveness of his evaluation design?
Level 1
Reaction
Level 2
Learning
Level 3
Behavior
Level 4
Results
When we ask learners questions!!!
One of the Biggest LiesIn Learning Evaluation
https://is.gd/BiggestLies
Performance-Focused Smile Sheets:
What Questions Do You Have?
34‐Page Report
Why Do We Need a New Evaluation Model?
Only 20% were able to do the learning measurement they wanted to do…
2007
In general, are you able to do the learning measurement you want to?
Dixon, G., & Overton, L (2017). Making an impact: How L&D leaders can demonstrate value. Available at: www.towardsmaturity.org/impact2017
“The evaluation of training programs in terms of ‘results’ is progressing at a very slow rate.”
Donald Kirkpatrick1960
“With some notable exceptions… relatively little work has been devoted to making evaluation more useful and worthwhile.”
Robert Brinkerhoff1981
54 CommonEvaluation Mistakes
www.is.gd/evaluation54mistakes
The Kirkpatrick Four-Level Model
Level 1
ReactionLevel 2
LearningLevel 3
BehaviorLevel 4
Results
The Kirkpatrick-Katzell Four-Level Model
Read about Raymond Katzell’s role:
https://is.gd/originator
Level 1
ReactionLevel 2
LearningLevel 3
BehaviorLevel 4
Results
“The Kirkpatrick framework has a number of theoretical and practical shortcomings.”
“[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders…” (p. 91)
https://is.gd/research22review
“Kirkpatrick's framework is not grounded in theory and the assumptions of the model have been repeatedly disproven over the past 25 years…
…(Alliger & Janak, 1989; Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997; Holton, 1996; Sitzmann, Brown, Casper, Ely, & Zimmerman, 2008; see Kraiger, 2002, pp. 333–335 for a critical review of Kirkpatrick's theory).”
Sitzmann and Weinhardt (in press from 2017)
Special Thanks:
• Julie Dirksen• Clark Quinn• Roy Pollock• Adam Neaman• Yvon Dalat• Emma Weber• Scott Weersing• Mark Jenkins• Ingrid Guerra‐Lopez• Rob Brinkerhoff• Trudy Mandeville• Mike Rustici
A
D
D
I
E
Assessment
Design
Development
Implementation
Evaluation
Messaging of Four-Level Model of Learning Evaluation
Harmful Messages (Sent or Missed)
• Does Not Warn Us Against Ineffective Evaluation Practices
• Ignores the Role of Remembering
• Level 2 Learning is Mashed into One Bucket
Beneficial Messages
• Don’t Just Focus on Learning! Focus on Results too!
• Learner Opinions Are Not Most Important
Understanding‐‐‐‐‐‐‐‐
Remembering
Performance‐Focused Smile Sheets
TraditionalSmile Sheets
Understanding‐‐‐‐‐‐‐‐
Remembering
Understanding‐‐‐‐‐‐‐‐
Remembering
Work Performance
Performance In Learning
Four Ways to Use LTEM
1. Assessing Your Evaluations2. Learning Design & Development:
Working Backward from Your Goals3. Credentialing Courses/Learners4. Spurring Improved Learning Designs
Assessing Your Evaluations
2019
2019
2020
2020
Two Ways to Use LTEM
1. Sales Increase by 5%
2. Managers Coach Better
3. Simulated Coaching
4. Scenario Questions
5. IF‐THEN Decisions
6. Perf‐Focused Questions
Learning Design & Development:
Working Backward from Your Goals
Credentialing Courses/Learners
Training Vendors Trade Associations
Universities Organizations
Learners
“Those who met the requirements performed successfully on an LTEM
Tier‐5 Assessment, showing competence in making decisions in challenging realistic scenarios.”
http://bit.ly/certification888
Spurring Improved Learning Designs
Introduce LTEM
Set Evaluation Objectives
Set Learning Objectives
Design Learning
Deploy Learning
Key LTEM Messages1. Goal of learning is to create transfer and positive learning benefits.
2. We should consider the many effects of learning transfer, not just organizational results.
3. Measuring attendance or learner activity is inadequate to validate success.
4. Measuring learner perceptions is inadequate to validate success. Focusing on effectiveness is better than focusing on satisfaction or reputation.
5. We should evaluate our success in supporting remembering, not just comprehension.
6. Measuring knowledge is generally inadequate to validate success.
7. Measuring decision‐making or task competence during learning is better than measuring knowledge.
Same asFour‐Level
A More Muscular Approach to EvaluationSending Messages to Nudge Improvements
Evaluation that helps us make our most important decisionsand helps us get the resources and support we need!
• LTEM Report and Model — https://www.worklearning.com/ltem/
Subscribe to my Newsletter: — https://www.worklearning.com/sign‐up/
• Katzell’s Contribution (Kirkpatrick NOT Originator) — https://is.gd/Katzell
• Updated Smile‐Sheet Questions 2018 — https://is.gd/Questions2018
• A Better Net Promoter Question — https://is.gd/replaceNPS
• Be Careful When Benchmarking — https://is.gd/DoNotBenchmark
• Debate About Kirkpatrick Model — https://is.gd/epicbattle
• Better Responses on Smile Sheets — https://is.gd/betterresponses
• Be Careful When Benchmarking — https://is.gd/DoNotBenchmark
Learning Technologies – London 2019www.youtube.com/watch?v=iq2n9srYnVw
Will Thalheimer, PhD
Phone: +1‐617‐718‐0767Email: [email protected]: WorkLearning.comBook: SmileSheets.comTwitter: @WillWorkLearn
THANK YOU!
Follow‐up: www.is.gd/will999stuff
LTEM, Gold‐Certification Workshop, SlidesSign up for Beta, Sign‐up for Will’s Newsletter
