Post on 27-Dec-2021
transcript
Purpose of the PresentationExplain the title, “Assessment for Improved Transfer and Long Term Outcome”Identify assessment strategies and guiding principles for enhancing transfer of learningto new situationsIdentify assessment strategies and guiding principles for enhancing transfer of assessment results to new situations Provide a ‘report card’ on the effectiveness of current assessment practices in contributing to transfer and long term outcomesIdentify where we should focus our future efforts to achieve better transfer and long term outcomes from our assessments
Purpose of the PresentationExplain the title, “Assessment for Improved Transfer and Long Term Outcome”Identify assessment strategies and guiding principles for enhancing transfer of learningto new situationsIdentify assessment strategies and guiding principles for enhancing transfer of assessment results to new situations Provide a ‘report card’ on the effectiveness of current assessment practices in contributing to transfer and long term outcomesIdentify where we should focus our future efforts to achieve better transfer and long term outcomes from our assessments
“Assessment for Improved Transfer and Long Term Outcome”
Transfer: the skill of applying what was learned to new situations (Mayer, 2004)*
In medical education, there is:Short-term transfer – applying what was learned earlier to what is currently being learned, and Long-term transfer – applying what was learned to the ‘real world’ care of patients (‘Long term outcome’)
*Mayer, Richard E., “Should There Be a Three‐Strikes Rule Against Pure Discovery Learning? “, American Psychologist, Vol 59(1), Jan 2004, 14‐19.
But, ‘transfer’ has an additional meaning in the context of assessment …The ‘transfer’ of assessment results to new situations – the degree to which assessment results are indicators of students’ performance in new situations.New situations include further training, and ultimately the independent care of patients.
So …, there are two aspects to ‘transfer’ in the context of assessment
Transfer of learning to new situationsTransfer of assessment results to new situations
To address the title of this presentation, our challenge as medical educators is …
How can we design our assessments to:
better assist students to transfer their previous learning to new situations?enable more accurate inferences about the future performance of students in new situations
Purpose of the PresentationExplain the title, “Assessment for Improved Transfer and Long Term Outcome”Identify assessment strategies and guiding principles for enhancing transfer of learning to new situationsIdentify assessment strategies and guiding principles for enhancing transfer of assessment results to new situations Provide a ‘report card’ on the effectiveness of current assessment practices in contributing to transfer and long term outcomesIdentify where we should focus our future efforts to achieve better transfer and long term outcomes from our assessments
The Challenge
How can we design our assessment strategies to enhance students’ short and long term transfer of learning?
What conditions must be met to enable transfer of learning?
The student must be able to recall the knowledge or skill to be transferred.The student must understand the knowledge or be proficient in the skill to be transferredThe student must see the relevance of the knowledge or skill to the new situation
How do we design assessments to enhance recall, understanding/proficiency, and
recognition of relevance?
Key instructional strategies used to enhance the transfer of learning …
Learning in context (e.g., via integrated systems-based curricula, case/problem based curricula, learning while caring for patients)Practice in using the knowledge/skillFeedback on performance
How can transfer of learning be effectively addressed in our assessment strategies?
1. Take advantage of the adage, ‘assessment drives learning’
Assessment is the most powerful tool we possess for conveying to students
what is important to learn. Our challenge therefore is to test what is
important to learn!
Utility (U) of an Assessment Strategy
R = ReliabilityV = Validity(E = Educational impact)A = AcceptabilityC = Cost
U = R x V x E x A x C
How can transfer of learning be effectively addressed in our assessment strategies?
2. Test students’ ability to apply their knowledge in contrast to recalling their knowledge
Test ‘in context’
Why test ‘application’?
Clinical Reasoning and Decision-Making
“How do clinicians use their knowledge to arrive at diagnostic and management decisions?”
Clinical Reasoning and Decision-Making
(Bordage, G. Academic Medicine, 1994, 1999)
Reduced DispersedElaborated Compiled
Clinical Reasoning and Decision-Making
(Bordage, G. Academic Medicine, 1994, 1999)
Reduced (“empty mind”)Dispersed (“cluttered mind”)Elaborated (“deductive thinking”)Compiled (“recall/recognition”)
Factual RecallWhat is the most likely renal abnormality in children with nephrotic syndrome and normal renal function?
(A) acute poststreptococcal glomerulonephritis
(B) hemolytic-uremic syndrome*(C) minimal change nephrotic syndrome(D) nephrotic syndrome due to focal and
segmental glomerulosclerosis(E) Schönlein-Henoch purpura with nephritis
ApplicationA 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. What is the most likely diagnosis?
Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, pp 58-9.
What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function?
A B *C* D E Overall P-value
1 0 99 0 08 1 90 1 0 94
A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. What is the most likely diagnosis?
0 1 98 1 010 9 66 10 5 84
Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58‐9.
How can transfer of learning be effectively addressed in our assessment strategies?
3. By providing students with feedback on their performance
Feedback is most effective if it is timely (immediate, a ‘teachable moment’), specific, descriptive, honest, and
challenges the student to self-assess their own performance
Summary -- transfer of learning can be effectively addressed in our assessment strategies by …
1. By taking advantage of the adage, ‘assessment drives learning’
2. By testing students’ ability to apply their knowledge in contrast to recalling their knowledge
3. By providing students with feedback on their performance
Context Practice Feedback
Purpose of the Presentation
Explain the title, “Assessment for Improved Transfer and Long Term Outcome”Identify assessment strategies and guiding principles for enhancing transfer of learning to new situationsIdentify assessment strategies and guiding principles for enhancing transfer of assessment results to new situations Provide a ‘report card’ on the effectiveness of current assessment practices in contributing to transfer and long term outcomesIdentify where we should focus our future efforts to achieve better transfer and long term outcomes from our assessments
Transfer: The Missing Link inMedical Education
Geoff Norman, Ph.D.McMaster University
norman@mcmaster.caMay 2010
Transfer and Assessment
Every test is a simulation of reality
Multiple choice (MCQ)
Key Features Problems
OSCE
Human Patient Simulator (HPS)
The move from performance on simulation to performance in reality = Transfer
Authenticity/fidelity
Assumptions underlying Test Scores
Score on a test is correlated with a “Score” on reality
The more authentic (high fidelity) the test, the higher the correlation of the test score with a ‘score’ on reality
Is there evidence supporting the validity of these assumptions?
David Swanson, NBMEMay 2010
Does greater fidelity = Greater transfer?
“How much authenticity is enough?”
“It is possible to have too much authenticity”
Meet “HARVEY”
“Harvey” combines a life-sized manikin with JV, carotoid and peripheral pulses and realistic heart sounds with computer modeling of cardio physiology… able to produce realistic symptom complexes.”
Geoff Norman’s meta analysis summary of findings from studies of transfer of learning from hi fidelity simulators
Hi fi simulators show small, non-significant learning gains over lo-fidelity simulation or didactic instruction
Geoff Norman: Relationship of Licensing Exam Scores to Complaints, Peer Assessments and Chart Reviews in Canada
Lower fidelity simulations (specifically key features examinations and MCQs) appear to provide better transfer than OSCEsRelation between assessment fidelity/authenticity and transfer of test results is questionable, and uncertain at bestThe missing elements interfering with transfer are not self-evident
??? Prototypical presentations ??? Noise (breath sounds, etc.)??? Anxiety, responsibility, (situated cognition)
Transfer from MCQ and OSCE Test Results to Performance in Clinical Practice (Mini-CEX)
Correlations between
Mini-CEX and MCQ scores .335Mini-CEX and OSCE scores .372
Meta analysis summary of the studies of transfer of assessment results to new situations
The transfer of assessment results to clinical practice situations is generally distressingly poor!Assessment results from low fidelity simulations appear to provide the best prediction of (tranfer to) future performance
Summary: Transfer of assessment results to new situations
In general, the transfer of assessment results to new situations is distressingly poor!Studies do not support the use of hi-fidelity simulations in improving transferThere is some evidence that assessments using lo fidelity simulation produce more ‘transferable’assessment results.
Purpose of the PresentationExplain the title, “Assessment for Improved Transfer and Long Term Outcome”Identify assessment strategies and guiding principles for enhancing transfer of learningto new situationsIdentify assessment strategies and guiding principles for enhancing transfer of assessment results to new situations Provide a ‘report card’ on the effectiveness of current assessment practices in contributing to transfer and long term outcomesIdentify where we should focus our future efforts to achieve better transfer and long term outcomes from our assessments
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge
Provide feedback on performance
Use of ICT/computers
Use of ‘lo fi’ simulation
Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge
Provide feedback on performance
Use of ICT/computers
Use of ‘lo fi’ simulation
Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge
Provide feedback on performance
Use of ICT/computers
Use of ‘lo fi’ simulation
Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge
Provide feedback on performance
Use of ICT/computers ☺Use of ‘lo fi’ simulation
Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge
Provide feedback on performance
Use of ICT/computers ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment strategy Current Trend
Future Direction
Test application of knowledge
Provide feedback on performance
Use of ICT/computers ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge
Provide feedback on performance
Use of ICT/computers ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge
Provide feedback on performance
Use of ICT/computers ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge ☺ ☺ ☺Provide feedback on performance
Use of ICT/computers ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge ☺ ☺ ☺Provide feedback on performance ☺ ☺ ☺Use of ICT/computers ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge ☺ ☺ ☺Provide feedback on performance ☺ ☺ ☺Use of ICT/computers ☺ ☺ ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge ☺ ☺ ☺Provide feedback on performance ☺ ☺ ☺Use of ICT/computers ☺ ☺ ☺Use of ‘lo fi’ simulation ☺ ☺ ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge ☺ ☺ ☺Provide feedback on performance ☺ ☺ ☺Use of ICT/computers ☺ ☺ ☺Use of ‘lo fi’ simulation ☺ ☺ ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients
Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge ☺ ☺ ☺Provide feedback on performance ☺ ☺ ☺Use of ICT/computers ☺ ☺ ☺Use of ‘lo fi’ simulation ☺ ☺ ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients ☺ ☺ ☺Review clinical performance,provide feedback
Enhancing Transfer and Long Term Outcomes -- Current Trends & Future
Directions
Assessment Strategy Current Trend
Future Direction
Test application of knowledge ☺ ☺ ☺Provide feedback on performance ☺ ☺ ☺Use of ICT/computers ☺ ☺ ☺Use of ‘lo fi’ simulation ☺ ☺ ☺ ☺Use of ‘hi-fi’ simulators
Observe students with patients ☺ ☺ ☺Review clinical performance,provide feedback
☺ ☺ ☺