71
A Two Decade Investigation Work Based Assessments (WBA) in Hong Kong’s Medical Internships C. B. Hazlett, Ph.D. Adjunct Professor (CUHK), Professor Emeritus (UAlberta)
A Two Decade Investigation
Work Based Assessments (WBA)
in
Hong Kong’s Medical Internships
C. B. Hazlett, Ph.D.
Adjunct Professor (CUHK), Professor Emeritus (UAlberta)
Purpose
Describe some of the underlying research & advantages in
developing & maintaining a valid database
consisting of work based assessments during the internship
Illustrate how a continuum of evidence-based R&D with WBA,
over the period 1996 – 2016
has informed licensing decisions as well as medical education
Reason Intern WBA can be Used to Inform Medical Education
Clinical Ability
Clinical Ability
An Internship
Key time for measuring clinical ability that still remains
attributable, primarily, to undergrad medical education
Key learning outcome in undergrad medical education
Key attribute to measure in an internship using WBA
Agenda
Introducing HK’s
Internship & WBA
Validity of
WBA Data
WBA Tool
Using the
Assessments
Q&A Quality of
Internship
Introducing HK’s Internship & WBA
Using WBA to inform
undergraduate medical education
Hong Kong Interns are placed in the 2 medical school’s teaching hospitals
and/or various other teaching affiliated hospitals throughout the SAR
53 clinical units utilised
Internship for CUHK & HKU medical graduates has
common requirements that comply with HK Med Council’s policy re internships
common overseeing committee (Deans (reps) / Hospitals COS (reps) / HKHA)
Hong Kong’s One-Year Internship
Interns specify preference for placement in hospital units approved to take interns
relative standing in related discipline is the weight for prioritising preference
assignment of rotations for CUHK & HKU is done jointly
Four 3-month rotations
1 must be in each of medicine & surgery (2 in either or both are allowed)
if only 1 is in medicine: 1 must be in a medical related specialty (e.g., ped.)
if only 1 is in surgery: 1 must be in a surgical related specialty (e.g., orthop.)
within a rotation up to 6 weeks can be in related subspeciality (e.g., oncol.)
Goal
Why was work base assessment
(WBA) tool developed?
Goal
. . . before they are granted a license
to practice medicine in Hong Kong
Develop & maintain valid mechanism for
determining if Hong Kong’s interns are
sufficiently competent clinically . . .
WBA Tool (for Hong Kong’s Internships)
Preceptors assess intern performance
in terms of specific criteria
WBA Tool
(see handout)
Professional Knowledge
Clinical Skill
Attitude to Patients
Attitude to Staff
Willingness to Learn
Organizational Ability
Clinical Judgement
Attendance at Educ. Activities
Use of Medical Language
Communication Skills
Sense of Responsibility
Unacceptable Needed Help Average Good Excellent
Likehert Rating Scale 11 Criteria
Likert Rating Scale (definitions for each category provided)
• Excellent
Clearly among the best of all interns; consistently exhibited an
unexpectedly high performance, commitment, and/or ability;
outperformed 90% or more of his/her peers at this stage of
development in clinical competencies, i.e., 90th to 99th percentile
rank (P90-99).
• Good
• Average
• Needed Help & Counseling
• Unacceptable
Qualitative Data Also Obtained
• Excellent
• Good
• Average
• Needed Help & Counseling
• Unacceptable
After rating an intern in terms of specified criteria,
commentary is solicited to complement the ratings
When completed, preceptor’s assessment is discussed with intern
Both sign the form to confirm this discussion has occurred & that the
intern has understood the underlying rationale for the preceptor’s ratings
Validity of WBA Data
Are data collected with WBA
tool acceptably accurate?
Face Validity
Content Validity
Construct Validity
Concurrent Validity
Predictive Validity
Types of Validity Investigated
Face Validity Established
Iterative reviews: 11 criteria for defining clinical ability for interns
endorsed by 80% to 100% by samples of
1. Teaching faculty (from Hong Kong’s two medical schools)
2. Chiefs of Service (in hospitals providing internships)
3. Interns (who were initially rated with the tool)
Content Validity Established
Goal in developing WBA tool was to measure clinical ability (of
interns)
Literature defines clinical ability as multi-dimensional, involving
relevant
1. Knowledge
2. Skill
3. Attitude
11 Performance Criteria are Representative of the 3 Dimensions (in terms of clinical ability wanted in an intern)
Professional knowledge (knowledge)
Clinical judgement (knowledge)
Medical language (knowledge)
Clinical skill (skill)
Organizational ability (skill)
Communication skills (skill)
Attendance at educational activities (skill)
Willingness to learn (attitude)
Attitude to patients (attitude)
Attitude to staff (attitude)
Sense of responsibility (attitude)
Construct Validity Established
Factor Analysis (FA) investigations done each year since 1996
FA is an appropriate statistical model for detecting if there is one or more
factors (constructs) underlying all responses to items in an assessment
FA of preceptors’ ratings always produced a dominant factor accounting for
more than half & up to three quarters of the total variance
In lieu of measuring 11 different abilities, WBA tool primarily measures 1 factor
(i.e., a construct that is common to (i.e., underlies) all 11 abilities)
- given tool’s face & content validity and analyses of the factor structures,
construct is very likely strengths in clinical ability
Concurrent & Predictive Validity Criteria
Concurrent validity
WBA data need to be consistent with other independent measures of
the same construct, taken at about same time period (i.e., concurrently)
Predictive validity
WBA data need to be consistent with other independent measures of
the same construct, taken at an earlier or later point in time
To Establish Concurrent & Predictive Validity
Assumption
If a medical school significantly modifies its curriculum & teaching,
changes in learning outcomes (i.e., clinical abilities) are wanted/intended
If there’s evidence curriculum & teaching modifications produced
improvements in undergraduate performance
then improvements in clinical ability during internship should occur
& be measureable by a WBA tool that has predictive validity
If there’s evidence a WBA tool & other independent measures of the
interns’ clinical abilities all detect similar changes
then WBA tool has concurrent validity
NonPBL integrated curriculum based on design developed at Cardiff University
(increased active learning, enhanced horizontal & vertical integration,
early clinical experience, high fidelity assessments to drive learning, etc.)
Passive Active Passive Active
Estimating Concurrent & Predictive Validity (cont’d)
Background
CUHK medical students admitted in 2001 & subsequently were
exposed to a major modification in curriculum & teaching design
%
To Establish Concurrent & Predictive Validity
Differences in performance (in undergraduate studies & in the internships)
were determined between CUHK cohorts admitted to medical school from
- 1996 to 2000 (i.e., educated by former curriculum)
- 2001 to 2002 (i.e., first 2 cohorts educated by modified curriculum)
Possible Confounding Variables Controlled
(Analysis of Covariance)
Gender
- male or female interns
Inflation of marks
- possible trend for preceptors to increase their ratings of interns over
years
Specialty
- Med. / Surg. / Ped. / OBGyne. / Psych. / Orthop.
Type of Hospital
- teaching vs teaching affiliated hospitals
Effect Sizes Defined (J. Cohen,1997)
- small effect can be important but rigorous research designs &
large samples are needed to detect this size of effect
(e.g., average height difference between 16 & 18 yr old girls)
- medium effect is visible to naked eye
(e.g., average height difference between 14 & 18 yr old girls)
- large effect is grossly perceptible
(e.g., average height difference between 13 & 18 yr old girls)
Effect Size Determined if Ho Rejected
95% confidence interval for effect size determined
- lower end of interval reported: most conservative estimate if change positive
most liberal estimate if change is negative
Established Concurrent Validity of WBA Tool
Effect Size
Large + Medium +
Small +
No Change
Small -
Medium -
Large -
11 Clinical
Skill Ratings
COS’
comments
Changed intern performance after modifying CUHK curriculum
Patients’
comments
Medical
Incident
Reports
independent measures
taken at similar point in time
WBA
tool
Preceptor comments content analyzed (by two independent raters)
53% increase in number of very positive comments for the newer interns
Character of Plotted Large Effects
Letters of appreciation from patients cared for by interns
100% increase in number of letters of appreciation for the newer interns
Medical incident reports involving interns
48% reduction by the newer interns
Established Predictive Validity of WBA Tool
Effect Size
Large + Medium +
Small +
No Change
Small -
Medium -
Large -
Student
Engagement
Graduates’
Generic Skills
10 Educ.Trials
4th & 5th Yr
Student
Concerns:
Content
Student
Concerns:
Clinical Skills
11 Clinical
Skill Ratings
COS’
comments
changed undergraduate performance
after modifying CUHK curriculum
changed intern performance
after modifying CUHK curriculum
independent measures taken at earlier
point in time
WBA Tool
Conclusion: Validity of WBA Tool
Given its level of face, content, construct, concurrent & predictive
validity, WBA tool provides an adequate measure of an intern’s
overall clinical ability
Quality of Internship
WBA need to be informed by
complementary data regarding
quality of internship experience
Complementary Data: Quality of the Intern’s Rotation
Log book
nature & number of required clinical tasks specified
intern enters date(s) for each task completed & verified by supervisor
Faculty Visit
interviews with intern & preceptor during rotation
Assessment Tool for Intern to Rate the Rotation
see handout
Interns’ Assessment of the Quality of their Rotation (hardcopy) Page 1
additional questions on back of page
On-line
version
Proportional Time Expended in Assigned Duties
Supervised Patient Care (e.g., diagnostic workup) % ____
Patient-related Clerical Work (e.g., clerk admissions) % ____
Routine Clerical Work (e.g., book appointments) % ____
Blood Collection % ____
Educational Development (e.g., grand rounds) % ____
Who Guided & Supervised?
Prof / COS / Consultant/ Reader/ Senior Lecturer Yes / No
SMO/Lecturer Yes / No
MO graduated > 2 yrs ago Yes / No
MO graduated within last 2 yrs Yes / No
Adequacy of Supervision
If a particular type of supervisor provided guidance/supervision . . . .
. .
How adequate was the quality of this guidance in terms of its
effect on your development as a clinician?
Good / Satisfactory / Poor
How adequate was the amount of supervision provided to you?
Good / Satisfactory / Poor
Additional Information re Quality of Rotation
Frequency
on-calls per month
opportunities to manage patients under supervision
opportunities to interact with staff in
- diagnostic services
- nursing
- allied health services (excluding nursing)
Ministries of Education & Health endorsement of & assistance in
project
For other information collected see your handout
The assessment data collected from interns also were reasonably valid
As with WBA tool, the form for soliciting feedback information from interns had
face & content validity
- excellent endorsement by faculty, preceptors & interns
- solicited info. consistent with literature’s definition for quality internship
Interns’ feedback re. rotations demonstrated convergent & divergent validity
(the most rigorous requirement for construct validity, Campbell & Fiske, 1959)
attributes that should converge theoretically, did converge in their feedback
attributes that should diverge theoretically, did diverge in their feedback
Interns’ rating of their rotation probably was not influenced by the amount of
work that they were assigned (i.e., Ho not rejected)
but was influenced (i.e., Ho rejected) by how educationally useful
the work was in terms of their development as a clinician
An Example of Divergent & Convergent Validity
in the Interns’ Assessments of Rotation Data
2011 - 2013
Character of Above Average vs Below Average Rotations
No difference in patient related clerical work
No difference in # of on-calls
* Above Average ~ 25% less routine clerical work
* Above Average ~ 33% less blood collection
* Above Average ~ 90% more supervised pat. care
* Above Average ~ 60% more opportunities in educ. development
0 20 40 60 80 100 Percentage
On call frequency
(per month)
Patient related
clerical work
Routine clerical work
Blood collection
Supervised patient
care
Educational
development
* Statistically Significant
Percentage of Time
* Excellent rotations have ~ 40% less routine clerical work
* Excellent rotations have ~ 40% less blood collection
* Excellent rotations have ~ 100% more superv. pat. care
* Excellent rotations have ~ 120% more educational development
2014 - 2016
Character of Excellent Rotations vs. Inadequate Rotations
* Statistically significant
No difference
No difference
On call freq.
(per month)
Patient related
clerical work
Routine clerical
work
Blood
collection
Supervised
patient care
Educational
development
Conclusion re Validity of Intern Feedback Tool
Given its level of face, content, convergent & divergent validity
the intern feedback tool also provides adequately valid data for
complementing information provided by the WBA tool
- i.e., in regards to level of quality for each of the interns’ rotations
Using the Assessments
The data inform 7 important
exercises in medical education
Usefulness of Intern WBA Data
1.
inform faculty’s curriculum design
(i.e., are desired learning outcomes achieved as medical
curriculum, teaching & learning evolve over time)
Keeping mind that interns’ clinical ability is the key learning outcome
for a medical programme
Long term, valid measures of learning outcomes (via WBA tool) . . .
Effect Size
Large + Medium +
Small +
No Change
Small -
Medium -
Large -
11 Clinical
Skill Ratings
COS’
comments
Changes in clinical ability among interns
after modifying curriculum
improvement is
visible to naked eye grossly perceptible
Differences in Performances Between Former & Modified Curricula
Work Based Assessment Tool (WBA)
Usefulness of Intern WBA Data
inform faculty’s teaching design (i.e., when benchmarked
with other designs used in medical education)
Keeping mind that interns’ clinical ability is the key learning outcome for
a medical programme
Long term, valid measures of learning outcomes (via WBA tool) . . .
2.
Clinical Skill
Benchmarking NonPBL vs PBL programmes based on WBA internship data
Macro View (magnification factor 1X)
Benchmarking nonPBL vs PBL programmes based on WBA internship data
Micro View (magnification factor 10X)
Clinical Skill
Benchmarking NonPBL vs PBL programmes based on WBA internship data
Micro View (magnification factor 10X)
~ 1%
difference
maintained
over years
Benchmarking NonPBL vs PBL (Magnification Factor: 10X)
Benchmarking NonPBL vs PBL (Magnification Factor: 10X)
Benchmarking NonPBL vs PBL (Magnification Factor: 10X)
Benchmarking NonPBL vs PBL
But as noted before . . .
with a macro view there is little to
distinguish the learning outcomes
between NonPBL & PBL programmes
on any of the 11 benchmarks
Usefulness of Intern WBA Data
inform teaching & learning reviews by university & governmental agencies
(i.e., are teaching & learning aligned with intended learning outcomes)
Keeping in mind that interns’ clinical ability is the key learning outcome for
a medical programme
Long term, valid measures of learning outcomes (via WBA tool) . . .
3.
CUHK Integrated Framework for Aligned Curriculum
Desired
learning
outcomes
Learning
activities Feedback
Content /
concepts
Assessment
Actual
learning
outcomes
high fidelity tests to drive learning
enhanced
active learning
valid WBA data
key learning outcome:
clinical ability
enhanced integration:
vertically & horizontally
Usefulness of Intern WBA Data
inform a faculty’s Quality Assurance (QA) exercises for the internship
Keeping in mind that interns’ clinical ability is the key learning outcome
for a medical programme
Long term, valid measures of learning outcomes (via WBA tool)
in conjunction with valid intern feedback . . .
4.
(i.e., which duty-assignments / clinical units / hospitals / supervisors
meet expectations)
Proportional Time in Assigned Duties
(2011 - 2013)
Patient Care
24%
Routine Clerical
17.8%
Educational Develop.
8.9 %
CUHK & HKU Interns
Clerical Patient Related
30.3 %
Blood Collection
19 %
Medicine
Oncology
Pediatrics
Psychiatry
Obs &Gyne
Orthopedics
Surgery
SD
Overall Quality of 53 Units Providing Internships
CUHK & HKU Interns
Good Education 2
SD
Good Interaction
Good Patient Management
0
-2
2
2
2
0
0
0
-2
-2
-2
Distinguishing Hospitals by Various Strengths
Hospital:
A B C D E F G H I J K L M N O
Good Supervision
No Supervision
Inadequate Amount
Poor Quality
Combined Shortfall
Prof/COS
SMO
MO >2 yr
MO ≤ 2 yr
0% 20% 40% 60% 80% 100%
Supervisory Limitations (2011 – 2013)
(CUHK & HKU Interns)
Usefulness of Intern WBA Data
inform if faculty’s steps in CQI (continuous quality improvement) for the
internship have achieved desired changes
Keeping in mind that interns’ clinical ability is key learning outcome
for a medical programme
Long term, valid measures of learning outcomes (via WBA tool)
in conjunction with valid intern feedback . . .
5.
Some Examples of CQI Steps Taken (given information gleaned from WBA data & interns feedback)
• All hospitals providing internships should employ phlebotomists
• To be an intern supervisor, one must now also teach in medical programme (to do latter one must be at least a SMO)
• Logbook needs to specify a required number of clinical duties involving management of patients under supervision
Changes in Proportional Time for Duties: 2011-2013 & 2014 - 2016
Patient Care
24%
25%
Routine Clerical
17.8%
17.8%
Educational Develop.
8.9 %
9.3 %
CUHK & HKU Interns
Clerical Patient Related
30.3 %
30.9 %
Blood Collection
19 %
17 %
CUHK & HKU Interns
Chronic 1%/1%
Acute
Both
77% / 74%
How Often ? What Type of Patient ?
2010-2013 & 2014 -2016
Changes in Management of Patients Under Supervision
Infrequently
26% / 20%
Never/Rarely
3.3 % / 2.7 %
Frequently
70.6 % / 77 %
22% / 25%
CUHK & HKU Interns
Changes in Supervisory Limitations: 2014-16 vs 2011-13
Down to 23% from 35%
Down to 14% from 18%
Prof /
COS
SMO
MO >2 yr
MO ≤ 2 yr
No Supervision
Inadequate Amount
Poor Quality
Combined Shortfall
0% 20% 40% 60% 80% 100%
Usefulness of Intern WBA Data
inform professional accreditation reviews in regard to 2 main questions
- is programme informed by ongoing monitoring & evaluation?
- are graduates adequately prepared for entry level clinical practice?
Keeping in mind that interns’ clinical ability is the key learning outcome
for a medical programme
Long term, valid measures of learning outcomes (via WBA tool) . . .
6.
HK’s long term, valid WBA database has been an integral
component for helping to answer these two queries
Most Importantly the Key Usefulness for Intern WBA Data
inform Hong Kong’s public if & how well its medical schools are
preparing graduates to provide quality care for its citizens
Interns’ clinical ability is key learning outcome for a medical programme
Long term, valid measures of learning outcomes (via WBA tool). . .
7.
Recall the Goal for Developing the WBA Tool
. . . before they are granted a license
to practice medicine in Hong Kong
Develop & maintain valid mechanism for
determining if Hong Kong’s interns are
sufficiently competent clinically . . .
1.000
1.500
2.000
2.500
3.000
3.500
4.000
4.500
5.000
1 2 3 4 5 6 7 8 9 10
Rating
Year 2005 ‘06 ‘07 ‘08 ‘09 ‘10 ‘11 ‘12 ‘13 ‘14
Clinical Skill
Excellent
Good
Average
Needs Help
Unacceptable
Based on WBA data developed adhering to evidence based protocols,
it’s apparent this goal is being achieved
“No endeavor worth doing can be achieved without cooperative effort”
D. Johnson (2002)
This achievement has depended on preceptors’ expertise & contributions
for whom both faculty members & interns have much respect & gratitude
Questions & Answers
What questions do you have?
