11
STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Michelle Nolin, CPLPSr. Director, Product Strategy & Solutions
Informa Training Partners
Dr. B. Price KerfootAssociate Professor
Harvard Medical School
2STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
LEARNING OBJECTIVES
Identify pedagogically sound strategies for mLearning design
Appreciate findings from the latest instructional design research for mLearning
Identify techniques for mLearning that ensure knowledge and skills are gained, retained, and transferred
Apply 10 Tips for Knowledge Retention with mLearning
3STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
What your learners really want on their tablets and phones…
4STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
What your learners don’t want…
TAP PINCH-ZOOM
And then… READ,SCROLL,
READ, SCROLL, READ, SCROLL, READ,
SCROLL, READ, SCROLL, READ SCROLL
5STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Are you (and your learners) a…
Digital native OR digital immigrant?
7STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
What is mlearning?
M = Mobile
DEVICE (Hardware)Smartphones, iPads, other
tablets
TECHNOLOGY (Software)
OS: Apple and DroidDozens of authoring tools
CONNECTIVITYCellular networksInternet/Intranet
8STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
eLearnng OR mLearning?
eLearning
• Real-time or self-paced• Synchronous or
asynchronous• Tethered (via Web,
Internet)• Structured, formal
(GUI, architecture, etc.)
mLearning
• Self-paced• Un-tethered• Informal• Unstructured, just-in-
time, field-based• Anywhere, anytime
9STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Polling Question
Are you using mLearning in your organization today?
YES
Not yet, but actively working on mLearning initiative
Maybe next year
No plans to use mLearning
11STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Challenges of mLearning
Device variability
Connectivity
Screen size and readability
Text entry
Limited memory
It’s MOBILE (and we work in a regulated industry)
REF: Elias T. Universal Instructional Design Principles for Mobile Learning. IRRODL.12, No 2, 2011.
12STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Poll
What are some of the challenges you face with mLearning in your organization?
14STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Unique opportunities
Mobile devices are inexpensive and pervasive
Handle media well
Continuous, just-in-time • In-the-field training and support• “Situational dependent knowledge”
Rapid and urgent
15STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Best uses of mlearning
Designed correctly, mLearning IS a knowledge retention tool.
• LCOs: think “topics” NOT “courses”• Job aids/support tools (quick “look ups”)• Just-in-time training• Rapid deployment of timely content• Continuous touch points with your learners• Evaluations, assessments, and remediation
16STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Drive Retention with the mLearning NINE
Start (so you can end with) your desired
outcomeBring it close to task Consider Bloom’s
taxonomy
Set clear objectives—and meet them
Match the objective to the device Engage learners
Establish logical architecture and
navigation—and stick to it
Establish and follow good design standards
Meet the learner where they live
17STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Start with the desired outcome.
Instructional technology is great, BUT… it’s not about the tool, or the platform. What outcome do you need to achieve?
Content
Calendar
Culture Cost
4Cs of Training
18STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Bring it close to task.
Examples
19STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Consider Bloom’s taxonomy.
Evaluation
Synthesis
Analysis
Application
Comprehension
Knowledge
Lower order thinking skills
Higher order thinking skills
20STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Set clear objectives (and meet them).
Outcome
PerformanceLearning
21STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Match the objective to the platform.
Not all objectives can be achieved on a mobile device
Some learning/performance objectives are better suited for a tablet, others on a smartphone
22STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Engage learners.
Draw the learner in… limit passive “push down” of content
Use media where appropriate• Audio• Video• Animations
Link mLearning with live interaction and collaboration
Rethink assessments: gamify it
23STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Establish logical architecture and navigation (and stick to it).
Design an intuitive GUI with multiple navigation paths through the content• Let the learner drive and control access
Design for cross-platform, cross-device use• Use responsive design• Optimize for touch screen environments
24STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Establish and follow good design standards.
Micro chunk the content
More images and media, less text
Use large, easy to read fonts
Avoid “porting” existing text/elearning content—design and develop from the ground up for mobile devices
Design according to the aspect ratio of the device (not your computer monitor)
25STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Meet the learner where they live.
mLearning allows us to meet learners where they live – on their PHONE and TABLETS.
• Cell phones = global population • ½ the US population has smartphones• Survey: 4 out of 5 people 18-44 check their
phone within 15 minutes of waking• 84% time spent on email, texting, social
media, only 16% on voice calls
Sticky mLearning: How to Ensure Knowledge Retention
in mLearning Programs
B. Price Kerfoot, MD EdM
VA Boston Healthcare SystemHarvard Medical School
Conflict of Interest Disclosure
• Harvard submitted patent on spaced education methodology.
• Harvard has launched a start-up company Qstream which hosts this methodology outside of the Harvard firewalls.
• As the “inventor”, I am an equity owner and director of this company.
Multi-institutional randomized controlled trial of 351 medical students at 4 northeastern medical schools
Acad Med. 2006;81(3):224-30.
Multi-institutional randomized controlled trial of 351 medical students at 4 northeastern medical schools
Acad Med. 2006;81(3):224-30.
Retention of BLS and ACLS Training Among Nurses
Resusc 2008; 78: 59-65
n=133
Lammers (Michigan State Univ) Acad Emerg Med 2008; 15: 1181-89
Hermann Ebbinghaus (1850-1909)
In 1885, he published "On Memory”
Forgetting curves
Learning curves
Spacing effect
Spaced Education: the basics
• Spacing Effect– spaced– repeated– repeated at increasing intervals
- Increased efficiency of learning- Reduction in the slope of the forgetting curve
Sisti et al, Learning & Memory 2007.
Pagani et al, Cell 2009.
Xue et al, J Cognitive Neuroscience 2010.
Psychol Sci, 2006; 17: 249
Science 2011; 331: 772-5.
“Testing effect”Retrieval practice
Psychol Sci, 2006; 17: 249
Spaced Education Delivery System:harnesses both the spacing & testing effects
Spaced Education Delivery System
VA PSA Screening Spaced Education Trial
VA PSA Screening Spaced Education Trial
VA PSA Screening Spaced Education Trial
It Works!
• Increases knowledge & retention- Medical Education 2007: 41:23-31 --- UGME- Journal of General Internal Medicine 2008; 23(7):973-8 --- UGME - Academic Medicine 2011 (in press) --- UGME - Journal of Urology 2007; 177, 1481-1487 --- GME– Journal of Urology 2009; 181, 2671-2673 --- GME– Journal of the American College of Surgeons 2010; 211: 331-72673 --- GME– Annals of Surgery 2009; 249: 744–749 --- CME
• Improves self-assessment of knowledge– American Journal of Surgery 2009; 197(1):89-95
• Changes behavior– American Journal of Surgery 2009: 197(2), 252-257– American Journal of Preventive Medicine 2010; 39: 472-8
• Is well-accepted by learners- Demonstrated in all trials to date
Interactive Spaced Education Versus Web-based Modules for Teaching Histopathology Diagnostic Skills:
a Randomized Controlled Trial
B. Price Kerfoot MD EdM, Yineng Fu MD, Michael Ritchey MD, Donna Connelly MS & Elizabeth Genega MD
VA Boston Healthcare SystemHarvard Medical School
Beth Israel Deaconess Medical CenterAmerican Urological Association
J Am Coll Surg 2010;211:331-7
ISE Delivery System: (2) the question
Spaced Education on GU Pathology
Weeks
0 10 20 30 40 50
Pe
rce
nta
ge
Sco
res
50
60
70
80
90
100
Web-based Teaching Modules
Module 1
Module 2
Module 3
OC-1
OC-2OC-3 OC-4
J Am Coll Surg 2010; 211: 331-7
Spaced Education on GU Pathology *
Weeks0 10 20 30 40 50
Pe
rcen
tage
Sco
res
50
55
60
65
70
75
80
Interactive Spaced Education
Cycle 1
Cycle 2
Cycle 3
OC-1
OC-4OC-3
OC-2
78% preferred SE over WBT 580/583 requested more SE
J Am Coll Surg 2010; 211: 331-7
Spaced Education Progress Testing (SEPT) of Medical Students:a Randomized Controlled Trial
B. Price Kerfoot, MD EdM, Kitt Shaffer MD PhD, Graham T. McMahon MD MMSc, Harley Baker EdD,
Edward Krupat PhD, Jamil Kirdar, MBChB MRCP, & Elizabeth G. Armstrong EdD
Harvard Medical School
Acad Med 2011, 86(3): 300-306
SEPT: Study Structure
• University of Virginia School of Medicine• University of Pittsburgh School of Medicine• Harvard Medical School• University of Alabama School of Medicine
Acad Med 2011, 86(3): 300-306
SEPT: 120-item assessment tool
Year of Training0 1 2 3 4 5
Perc
enta
ge o
f It
em
s C
orr
ect
35
40
45
50
55
60
65
All 120 Items
Acad Med 2011, 86(3): 300-306
SEPT: 40-item assessment of retention
Acad Med 2011, 86(3): 300-306
Weeks0 10 20 30 40
Per
cent
age
Sco
re
30
40
50
60
70
80
90
LPTASEPT
p<0.001effect size=0.95
2-weekcycled-review
6-weekcycled-review
Items 1-40initial presentation
Final cycle to assess
learning retention
SEPT: 40-item assessment of retention
J Cont Educ Health Prof, 31(2):103–108, 2011
74% of participants (181/246) completed the SE program. Of these, 97% (176/181) submitted the behavior change survey.
This randomized controlled trial was conducted from March 2009 to April 2010, immediately following the PriMed live CME conference in Houston, Texas.
J Cont Educ Health Prof, 2011; 31(2):103–8
86% agreed or strongly agreed that the SE program enhanced the impact of the live CME conference.
97% requested to participate in future SE supplements to live CME courses.
J Cont Educ Health Prof, 2011; 31(2):103–8
Interactive Spaced Education to Improve Prostate-Specific
Antigen Screening:a Randomized Controlled Trial
B. Price Kerfoot MD EdM, Galina Sokolovskaya MS
Elizabeth Lawler MPH ScD, David Gagnon MD PhD,
& Paul R. Conlin, MD
VA Boston Healthcare System
Am J Prev Med 2010; 39: 472-8
VA PSA Screening Spaced Education Trial
Am J Prev Med 2010; 39: 472-8
VA PSA Screening Spaced Education Trial
93% requested to participate in future spaced education programs
Spacing
Weeks0 10 20 30 40
Per
cent
age
Sco
re
30
40
50
60
70
80
90
LPTASEPT
p<0.001effect size=0.95
2-weekcycled-review
6-weekcycled-review
Items 1-40initial presentation
Final cycle to assess
learning retention
Weeks0 10 20 30 40 50
Pe
rce
nta
ge
Sco
res
50
55
60
65
70
75
80
Interactive Spaced Education
Cycle 1
Cycle 2
Cycle 3
OC-1
OC-4OC-3
OC-2
Reinforcement
Effective after a face-to-face eventDurable behavior change
VA PSA Screening Spaced Education Trial
Spaced Education Delivery System
Lessons from Harvard: Using Gamification to Juice
Your Sales Training
68STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
CHOPPED ACTIVITY
Break into small groups
Ingredients you need are in your “inbox”
Cook up an mLearning intervention
Present Your DISH to the larger group
69STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
10 Tips for Knowledge Retention with mLearning
1. Select the right content at the right time for the right reason in ALL mLearning.
2. Don’t let technology be the tail that wags the dog… desired outcome should drive your mLearning initiatives.
3. Couple engagement through mobile devices with the principles of spaced education to boost knowledge retention.
70STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
10 Tips for Knowledge Retention with mLearning
4. Investigate DIY, easy-to-learn, and cost-effective mLearning tools to create continuous touch points with your learners.
5. Think beyond “just-in-time” training and performance support for mobile learning… designed well, mLearning boosts knowledge retention.
6. Use mLearning to meet learners “where they live,” providing them with opportunities to access content when they need it, confirm what they know (and don’t know), and retain their knowledge.
71STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
10 Tips for Knowledge Retention with mLearning
7. Integrate mLearning tools with virtual instructor-led training (VILT) to make distance learning intimate.
8. Integrate mLearning onsite at live meetings for learner engagement.
9. mLearning is not reading on your phone… deliver learning interactions on mobile devices to engage learners, foster learning transfer, and sustain desired outcomes.
10. Recognize how digital natives and digital immigrants, like to learn, and deliver training on mobile devices that balances the needs of these two learner types.
72STICKY MLEARNING: HOW TO ENSURE KNOWLEDGE RETENTION IN MLEARNING PROGRAMS
Questions?
SPEAKER EMAILS:
Michelle Nolin: [email protected]
Dr. Kerfoot: [email protected]