Post on 27-Dec-2015
transcript
1
Storytelling as Instruction Nov 06 Anacapa Sciences
7 November 2006
Presentation to: Storytelling as an Instructional Method Conference
Mesa, Arizona
Presentation by:Alan Spiker
Anacapa Sciences, Inc.Santa Barbara, CA
Scenario-Based Training: Lessons from Development and Implementation
2
Storytelling as Instruction Nov 06 Anacapa Sciences
Topics
What’s a Scenario and why is it needed?
Challenges Facing a Small Business Implementing SBT
Evolution of Scenarios as an Element of Training
Useful ISD Principles of Scenario Based Training
Measuring SBT Effectiveness
Lessons Learned: Navy, Commercial Aviation, Army
3
Storytelling as Instruction Nov 06 Anacapa Sciences
Topics (details)
What’s a Scenario and why is it needed for training? - look to the Thesaurus- many good reasons and some not-so good reasons for scenarios
Challenges Facing a Small Business in Implementing SBT- cajole not coerce, evince not convince, insinuate not dictate- implementing various types of SBIR-funded technologies
Evolution of Scenarios as an Element of Training- Geospecific DB --- > Framework ---- > Wizard ----- > Serious Games?
Useful ISD Principles of Scenario Based Training - eight that REALLY matter
Measuring SBT Effectiveness- Kirkpatrick’s 4-Level formal method- informal method in the real world
Lessons from Implementing SBIRs w/ Navy, Commercial Aviation, Army- Sometimes a great notion- Our scorecard: wins, losses, draws
4
Storytelling as Instruction Nov 06 Anacapa Sciences
What’s a Scenario?
1. An outline of the plot of a dramatic work, giving particulars to scenes, characters, situations, etc.
2. The outline or manuscript of a motion picture or television program, giving the action in the order in which it takes place, description of scenes and characters
3. An imagined or projected sequence of events, esp. any of several detailed plans or possibilities
4. Setting for a work of art or literature
5. Postulated sequence of possible events
6. An outline or synopsis of a play
Syn: scheme, plan, concept, sketch
Sources: Dictionary.com WordNet
5
Storytelling as Instruction Nov 06 Anacapa Sciences
What’s a Scenario really?
1. An outlineoutline of the plot of a dramatic work, giving particularsparticulars to scenes, characters, situations, etc.
2. The outline or manuscript of a motion picture or television program, giving the actionaction in the order in which it takes place, description of scenes and characterscharacters
3. An imaginedimagined or projected sequencesequence of eventsevents, esp. any of several detailed plans or possibilitiespossibilities
4.4. SettingSetting for a work of art or literature
5. Postulated sequence of possiblepossible events
6. An outline or synopsissynopsis of a play
7. Syn: scheme, plan, concept, sketch
Sources: Dictionary.com WordNet
6
Storytelling as Instruction Nov 06 Anacapa Sciences
Scenarios are good because they:
• Can be a script or a blueprint of what will happen
• Define roles of non-trainees
• Constrain allowable actions, decision points
• Help establish cognitive realism and trainee buy-in
• Lay out sequence of action to unfold
• They define which events occur, and when
• Give the storyline of actions
• Serve as communication conduit for training team
• And thus ……
7
Storytelling as Instruction Nov 06 Anacapa Sciences
Scenarios:
• Prevent chaos in the training environment
• Provide an antidote to unscripted free play in the sims
• Make training more interesting to the trainee
• Make training more believable – more face valid
• Make training more personal, customizable
• Are more consistent with our Experiential Learning Model approach to training adults
• Can’t substitute for measurable learning objectives
• Don’t ensure criterion-based training
• Don’t guarantee positive transfer of training
BUT Scenarios --
8
Storytelling as Instruction Nov 06 Anacapa Sciences
Our Challenge as a Small Business
• Anacapa Sciences – small business est. 1969• Implement SBT as part of SBIR products• Solicit claimants through insinuation, use of data-
driven approach, value-added consulting on the side
- so no organizational clout but that’s hardly unique to us
• Since 2000, attempted SBT in several SBIRs:– SamePage: team training in Shared Understanding for ARI (Ph. II)
– SimDATT: scenario-generation tool for NavAir (Ph. II+)
– (CT)2: web-based critical thinking training modules for ARI (Ph. III)
– TARGET: serious game based trainer (GBT) for ONR (Ph. I)
– Stalwart: Wiki-based team trainer for JFCOM (Ph. I)
9
Storytelling as Instruction Nov 06 Anacapa Sciences
Common Approach
• Identifying training need – why SBT needed in first place
• Finding suitable SBT sponsor – how SBT will help him/her
• Defining training baseline – what SBT is replacing
• Finding training objectives – usually a real scavenger hunt
• Selecting form of SBT – one size does not fit all
• Managing expectations – what SBT can and can’t do
• Measuring SBT effectiveness – how to know if it’s working
• Implementing SBT – what’s the exit strategy
• Assessing feasibility– who’s planning the birthday party
SBT is still training - Typical Steps in SBT Projects
10
Storytelling as Instruction Nov 06 Anacapa Sciences
Evolution of SBT Small Business Perspective
• Scenarios WEREWERE: Big, physical, fixed, geo-specific, 1-1 physical fidelity with criterion environment, created by SMEs, little measurement
• Scenarios ARE NOWARE NOW: Smaller, cognitive, dynamic, regional, 1-1 cognitive fidelity with criterion environment, created by project team, stopping points for measurement
• Scenarios WILL BECOMEWILL BECOME: Micro, meta-cognitive, adaptive, generic, 1-1 higher order fidelity with criterion environment, created by anyone, embedded measurement
11
Storytelling as Instruction Nov 06 Anacapa Sciences
Evolution of SBT Mission Scenario
trainee’s role
12
Storytelling as Instruction Nov 06 Anacapa Sciences
Evolution of SBT Instructor Script
instructor’sactions
13
Storytelling as Instruction Nov 06 Anacapa Sciences
Evolution of SBT Characteristic Features Now
• Synopsis – Storyline method of organization
• Trainee role specified through multiple means
• Timeline – used as scaffold on which to specify sequence of action
• Events – are specified, selected, and inserted as basic unit of action
14
Storytelling as Instruction Nov 06 Anacapa Sciences
Evolution of SBT Synopses - Storylines
Select scenario fromSelect scenario froma library of possiblea library of possiblesynopsessynopses
15
Storytelling as Instruction Nov 06 Anacapa Sciences
Evolution of SBT Trainee Roles
Conveyed through instructions, tables, special OPORDs,maps, immersion exercises, vignettes
(1) Evaluation of possible enemy courses of action that might disrupt our purpose in
occupying Fwd 1. The enemy’s primary methods of disruption are to promote fear and distrust – fear of them and distrust of US forces. Determine whether US occupation of Fwd 1 will diminish, enhance, or remain unaffected the enemy’s capacity to:
Conduct criminal actions against the general populace to induce fear in them. Conduct criminal actions against local leadership to induce fear in them. Conduct criminal actions against the general populace and local leadership to
demonstrate lack of capacity of US forces to protect them. Spread rumors that the US forces lack resolve to occupy Fwd 1 over a long period
of time.
Responsibilities on team
Estimate proactive and reactive enemy activities Predict the general enemy reaction to the positioning of our forces in Fwd 1. Build and interpret a link diagram that will describe the relations among and
between civil and religious leaders. Estimate the threat to logistics routes when asked by the Logistics Analyst.
Information/knowledge available to you in this mission 1. The Bn S-2 and the commander have pondered the nature of the enemy over the past
several months. What is listed on the OPORD, paragraph 1a is only a summary of their thoughts. Here is a more detailed general estimate:
Compare this photo to the OPORD map. Where is the train station in relation to Fwd 1? Who else should know about this? Why or why not?
17
Storytelling as Instruction Nov 06 Anacapa Sciences
Evolution of SBT Event Selection and Specification
18
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTOverview
• Navair funded add-on to SimDATT project• Identified 69 principles from ISD realm that could be applied
to guide SBT development• User version incorporated into SimDATT to aid designer
Design Principle 1-4: Promote student motivation by instilling mental conflict.
Mental conflict can be provoked in students through several methods, including: 1) using facts that contradict their past experience, 2) involving a sense of mystery by presenting unresolved problems, and 3) providing scenario events that “baffle” their perceptions.
Explanation: Provoking mental conflict can stimulate a student's curiosity about the KSA he/she is about to learn and hence their motivation to learn. These “garden path” experiences, where students are essentially lured into an initial failure experience, help to form more accurate mental models and help them unlearn previously erroneous methods of looking at situa-tions and problems.
Example: There are a number of effective ways to trick a student to follow down a “garden path” in order to get their attention and impress upon them the difficulty of the to-be-learned KSA. Examples of five effective “garden path” tech-niques are shown in Figure 1-4.
Reference: Keller & Burkman (1993); Ross, Phillips, Klein, & Cohn (2005)
GARDEN PATH EXPERIENCE THAT PROMOTES MENTAL
CONFLICTTYPE OF KSA
Progressive increase in task difficulty until STUD’s skill level is exceeded
Prelude to learning to make precision approaches
Presentation of ambiguous stimulus that defies accurate ID
Prelude to learning VFR to IFR transition
Presentation of conflicting information that makes unambiguous decision difficult
Prelude to learning advanced aviation decision making skills (e.g., divert airfield)
Presentation of benign stimulus that is followed by abrupt malevolent change
Prelude to learning to recover from unusual attitude
Presentation of message that contains clauses which are logically or factually inconsistent
Prelude to learning ATC communication protocols
Figure 1-4. Five ways to promote mental conflict.
19
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTEight Really Useful Ones
ISD Design Principle
1-1 Use 5-part scenario structure
1-5 Link scenario objectives to student learning goals
2-1 Link scenario content to training objectives
2-8 Include meta-cognitive skills in scenario KSAs
3-2 Organize training by varying setup conditions
4-2 Use ‘telescoping’ approach to design scenario content
6-7 Use analogous cases as training scaffold
8-4 Stress critical thinking in debrief questions
20
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTFive-Part Scenario Structure
Storyline/Synopsis Players Events Content Timeline
Basic daytime flight from Dulles toNashville. During start, severalinterrupted start problems areencountered and discussed. NormalSID using autopilot. Practice areaentry for steep turns and stalls. Inpractice area, low hydraulic pressure(1) light comes on. After isolationand correction, descent is initiatedback to Dulles, using autopilot. Fullstop landing is completed.
ATC
ATIS
Dispatch
Interrupted Start
IFR Standard Departure
HYD 1 Low Pressure
Steep Turns
Stalls
SID Chart
CFM (company flightmanual)
Checklists
Start & Taxi — Interrupted Start
Departure– IFR
Enroute– Turns Stalls HYD
Approach– ILS
21
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTLink Objectives to Student Goals
The scenario will:
Expose you to different types of
Hydraulic Systems failures
Know bold-faced items by heart and engage correct emergency systems within
sec. of detection
After completing the scenario you will be able to
Subject you to different levels of low-vis conditions
Initiate VFR-IFR transition actions w/o hesitation
while remaining spatially oriented
Require you to practice multiple
approaches and make landings on runways
of various lengths
Internalize decision height actions and opt for
missed approach w/o hesitation
Expose you to high volume ATC traffic
under different amounts of radio
noise
“Tune” your attention to particular voices on the
radio and to assertively ask for a repeat transmission
when messages are inaudible
22
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTLink Content to Objectives
23
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTInclude Meta-Cognitive Skills
Generate mental simulations to troubleshoot emergenciesGenerate mental simulations to what-if a planGenerate mental simulations to ID route weaknessesEngage in counterfactual thinking (assume plan is wrong)Learn to size up a situationGenerate story to explain away inconvenient dataDiagnose cause of some flight disturbanceControlling attention through eye-scanning techniquesStress resistance trainingShared situational awareness
24
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTVary Setup Conditions
Variables that Promote Transfer of Training
High Level of Initial Learning
High Perceived Similarity between Training Task and Criterion Task
Tasks Learned in Relevant Context
Tasks Learned with Similar Cognitive Processes to Criterion Task
Example of Application in SBT
Variability in Training Conditions
Increase Difficulty of Instruction through Contextual Interference
Gradually Fade out Feedback and Knowledge of Results
Ensure that all core tasks receive many practice trials
Ensure that students are made aware of the task elements in training that will be present on the job (e.g., time pressure, equipment malfunctions)
Factors like time of day, amount of rest, time to prepare, etc. should be similar in training to that expected on the job
Ensure that tasks require a comparable level of critical thinking, decision making, problem solving, etc. as that expected on the job
Conduct training where environmental, aircraft, and flight conditions vary widely from session to session
Include contextual factors that make learning difficult, like frequent malfunctions, radio traffic, garbled comms, etc.
After initial sessions, don’t give student feedback after every takeoff/landing, but only after several
25
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTUse Telescoping Design to Create Content
(1)
(2)
(3)
(5)
(4)
26
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTUse Analogous Cases as a Scaffold
2 scenarios most likeyour present scenarioin their emphasis onCRM
27
Storytelling as Instruction Nov 06 Anacapa Sciences
ISD Principles of SBTStress Critical Thinking in the Debrief
28
Storytelling as Instruction Nov 06 Anacapa Sciences
Measuring SBT EffectivenessThe Formal Way
Evaluation Level Methodology
I User Acceptance Survey
II User Learning Comprehension Test
III Transfer to Job Gaining Unit Survey
IV Organizational Impact Longitudinal Study
Kirkpatrick’s 4-Level Approach to Evaluation
29
Storytelling as Instruction Nov 06 Anacapa Sciences
Measuring SBT EffectivenessReal World Metrics - Checklist
Does system have its own room?
Does system have a real-person as champion?
Does using community pony up some of its own money to keep system going?
Are they establishing training objectives for each scenario?
Do they call us (for tech support) to keep system going?
Do they return our phone calls and e-mails?
Do they invite us to their parties?
30
Storytelling as Instruction Nov 06 Anacapa Sciences
Lessons LearnedAnacapa’s Scorecard
• Why did SimDATT version for Navy fail?
• Why did SimDATT version for ASU-Mesa succeed?
• Why did SamePage come to a draw with the Army?
• Why might (CT)2 win/lose/draw with Army? - too close to call now
Our Record is 1-1-1-1Our Record is 1-1-1-1
31
Storytelling as Instruction Nov 06 Anacapa Sciences
Why SimDATT Failed with Navy
• Users liked the aviator interface and content• But didn’t need scenarios for their part-task training• Either FRS or refresher• So all the marks appear high in Level I-II evaluation• But our real-world metrics are very poor
32
Storytelling as Instruction Nov 06 Anacapa Sciences
Why SimDATT Succeeded at ASU-Mesa
• Same basic system as for the Navy• ASU-Mesa embraced scenarios for their students• Make even initial training LOFT-like• Have intimate relationship with the “gaining unit”• Supporting a four-level Kirkpatrick evaluation
33
Storytelling as Instruction Nov 06 Anacapa Sciences
Why SamePage Stalled with Army
• Users liked the concept• Users liked the immersion experience• Users liked the challenge, great for leader development• But need domain changes to cover multiple MOSs• Needed a wizard, not a framework
36
Storytelling as Instruction Nov 06 Anacapa Sciences
Why (CT)2 Still in Limbo with Army
Have conducted multiple evaluations, all very positive
Contains multitude of cognitive vignettes
But where to put into curriculum?
What has to drop out to put (CT)2 in?
Yet our inputs reached other parts of CGSC curriculum
37
Storytelling as Instruction Nov 06 Anacapa Sciences
Trends in SBTWrap-up
• SBT becoming more adaptive• SBT could become more meta-cognitively oriented• SBT will become more SCORMED• SBT scenarios will be updated often, perhaps daily• SBT will become more personalized (e.g., wikis,
avatars)• SBT will become “jazzed up” to support a market-
based approach to training (how many hits did you get today Mr. Training Manager?)
• SBT will become SBD, scenario based development
38
Storytelling as Instruction Nov 06 Anacapa Sciences
Meta-Cognitive Skills in ScenariosEffects Based Thinking Tool