1. 1. Measuring the Impact ofSocial and Informal LearningJohn Kleeman / Chairman, Questionmarkjohn@questionmark.com Thursday, 26th January 2012Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2. 2. Learning is Changing Academic modelworked for 2500 yearsSign up Internet changingthingsLearnGoogle itChanging work rolesTestDeeper understanding of learningDo workCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
3. 3. Only ~10% of learning Learning from your comes from studymentors, conversations (reading, researching, with peers and meetings courses, etc.) (if you pay attention and ask questions)70+20+10 Formal Learningmodel oflearningLearning From Others10% 20%Learning By Doing On-The-Job learning by making mistakes, taking 70% time for retrospectivesand learning from the experience. Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
4. 4. 70+20+10 model for learning If you invest in~70% from 70+20+10, how do you Experience measure the impact? ~20% from Need to measure Others Are people learning?Are people performing better? ~10% from StudyCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
5. 5. Assessments? Assessments are a critical part of learning Quizzes Surveys Tests Exams How can assessments Measure informal learning? Measure improved performance?Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
6. 6. Assessing Impact of Social/InformalLearning Initiatives Did they learn it? Did theyAre theylike it? doing it?ResultsCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
7. 7. Did they like it? Use surveys to evaluateattitudes/reactions Specific content / initiatives Whole 70+20+10 initiative Use surveys to identifysuccess cases using theBrinkerhoff method Surveys Side-by-side with learning Delivered on mobile devices More conventional routesCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 7is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
8. 8. Mobile Assessments Questionmark assessments: Can run via browser Or on mobile devices via iPhone, iPad or AndroidCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
9. 9. QR codes can be used to launch surveyseasily BooksPowerPoint Signage Benefits Ease of access for increased response rates Demographic data easily captured Better data accuracy Easier analysis of survey results by demographicCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 9is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
10. 10. Survey & Course Evaluation Analytics Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 10 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
11. 11. Did they learn it? Formative assessments Help direct learning Strengthen memory Engage learners In 70+20+10 user-generated quizzes provide extra dimension Measure knowledge and skills Compare before to after Compare to benchmark Compare to others Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 11 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
12. 12. Quiz attached to video in SharePoint Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 12 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
13. 13. Embed Youtube and other hosted video inQuestionmark Live items!Embed YouTube or other hosted videodirectly in questionstimulusCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 13is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
14. 14. Questions Through the Learning ProcessReduce Forgetting CreateIntrigue StrengthenKnowledge/SkillsMemory QuestionsQuestionsQuestions Questions Questions TimeCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
15. 15. Quizzes and tests slow forgetting curve Taking a quiz or test Knowledge Retained gives retrieval90%80% practice 70% Reinforces mental60% After 5 pathways to make 50%56% mins40% future recall more 30%42%After 1 likely 20%week10% 0%Data from Test-Enhanced Learning,Roediger and Karpicke 2006. Study only Study, TestPsychological Science Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
16. 16. Embed in a Wiki Embedded in a Wiki Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 16 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
17. 17. Embedding in Blogs, Portals, Wikis For Learners For Instructors Focus the learners attention Easily use assessments Practice memory recallthroughout the learning Identify what was learned process Provide seamless user Provide feedback to assist the experience learner to: Centralised management of Correct misconceptions assessment content and Instill confidence Point to relevant material results Impact data on social learning Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 17 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
18. 18. Compare to Benchmarks BenchmarkParticipantTopiclevelOverallCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 18is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
19. 19. Compare to Others Participant Topiclevel Overall Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 19 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
20. 20. Are they doing it? In 70+20+10, people are learning In multiple places At different times In different ways Measuring performance more meaningful than measuring learning Observational assessments can do this Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 20 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
21. 21. Observational assessment vs. traditional quiz, test or exam Traditional assessment Participant is presented with questions for direct response Observational assessment Observer is presented with questions to rate participant to measure knowledge, skills and abilitiesCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 21is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
22. 22. What are Observational Assessments? Observational assessments provide a mechanism for an observer to assess a participant For example, a participant might be observed and rated on Performance of a specific task or procedure Knowledge acquisition and application Demonstration of skills/abilities that traditional assessment may be less reliable in measuringCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 22is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
23. 23. If assessing performance of a task, observertypically rates participant in three areas:Performing thePreparationWrap Up task Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
24. 24. Typical workflow for observer Monitor/observer logs in Selects assessmentSelects participant to be observedAnswers/submits responses to questions Results/feedback reported to stakeholdersCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 24is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
25. 25. Monitor/observer logs in
26. 26. Selects assessment
27. 27. Selects participant tobe observed
28. 28. Answers/submitsresponses toquestions
29. 29. Example applications of Observational AssessmentCertificationObservational assessments can be used as critical components of the certification process in some cases are required by regulatory authorities.ProcessPerformanceAn organization has delivered a training course to its employees and is now evaluating its effectiveness. By observing and monitoring behavioral changes inor level 3 the workplace before and after the training, the organization will be able to assessassessment:the changes caused by the training and evaluate the return on investment gained.EquipmentObservational assessment can be used to record and rate operators skills, abilities, safety practices and adherence to required proceduresoperation:Customer Workplace Assessments can also be used to evaluate retail sales and customer service staff on how well they cope in different situations to assess whereServiceadditional training is required or where personnel excel.evaluation:Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
30. 30. Feedback to Participant on PerformanceResults/feedback reported tostakeholders
31. 31. Mobile devices Mobile devices such as tablets and smart phonesideal for delivering observational assessments Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
32. 32. The same assessmentthat is delivered on traditional laptop can be delivered via ahandheld mobile devicesuch as iPad, iPhone or Android phone/tablet
33. 33. How to measure 70+20+10 impact?Did theylearn it?Did theyAre they like it? doing it? Results Copyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
34. 34. Visit Questionmarkat Stand 60 Questions? Twitter: @questionmark and @johnkleemanMore info: www.questionmark.com/go/observationalCopyright 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.