Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 1
Integrated Entry-to-Practice Assessment Strategies
Integrated Entry-to-Practice AssessmentStrategies
Glenys Wilkinson, Australian Pharmacy Council,
and
Diana Spizzirri, Ontario College of Pharmacists, Toronto, Canada
Annie Lee, Faculty of Pharmacy, University of Toronto, Canada
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 2
Objectives
• Describe the collaborative process between independentpharmacy organizations on common assessment goals
• Describe assessment strategies developed in Australia andOntario, Canada for entry to practice (licensure)
• Review the outcome of the process to develop theseassessment strategies and give examples of lessonslearned
Integrated Entry-to-Practice AssessmentStrategies
Common Themes:
• Collaboration
• Implementation across different organizations
• Entry-to-practice competency assessment
• Evaluation of outcomes
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 3
The Australian Pharmacy Council Ltd (APC)
Is the independent accreditation authority for pharmacy educationand training. We:
• Accredit pharmacy schools and programs; intern trainingprograms; and accrediting agencies to accredit continuingprofessional development (CPD) activities
• Conduct examinations on behalf of the Pharmacy Board ofAustralia (PharmBA)
• Assess the qualifications and skills of overseas trained pharmacists
Australian Pharmacy Council
Registration requirements for pharmacists in Australia:
Graduation from an accredited approved degree program
Completion of 1824 hours of supervised practice
Successful completion of an accredited intern training program
Successful completion of registration examinations
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 4
Australian Pharmacy Council• Two registration examinations
– Intern Written Examination (IWE)
– National oral examination (Oral)
• The IWE is a competency-based 125 Multiple-Choice Questioncomputer-delivered exam
• The Oral examination is an assessment that requirescandidates to demonstrate that they can competently applytheir knowledge and skills
The Intern Year Assessment BlueprintA joint APC and PharmBA project that followed the revision andupdate of National Competency Standards Framework for Pharmacistsin Australia
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 5
The project
• To optimise the assessment of interns during their trainingyear against the newly developed competencies
• To ensure the appropriate assessment methodologydevelops a direct link between learning objectives andassessment methods
• To provide transparency and clarity for ITPs, preceptorsand interns regarding how competence is measured andby whom
Collaborative partners
• Australian Pharmacy Council (APC)
• Pharmacy Board of Australia (PharmBA)
• International Regulator colleagues – UK, USA, Canada,New Zealand, Ireland
• Other professions within Australia
• International Expert Reviewer – reviewed final documents
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 6
Governance – Steering Committee
ConsultationThree consultations were conducted during the project with:
Intern Training Providers
Examiners
Preceptors
Interns
International Expert Reviewer
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 7
The project outcomes1. Literature review of intern competency assessment (including
domestic and international processes and assessment methods) forInternational jurisdictions
2. Documentation and analysis of the current intern year assessmentstructures (internal use)
3. Development of an Assessment Blueprint for the intern year,mapped against the National Competency Standards Framework forPharmacists in Australia 2016
International literature review Identified strengths and concerns with applications of competency standards
Examined the use of Miller’s pyramid to guide the development of assessmentprocesses
Evaluated advantages, disadvantages, feasibility, validity, reliability, and educationalimpact of commonly used assessments processes including MCQ’s, viva voce/oralexams, OSCE’s and workplace based assessments
Concluded each method of assessment has its own advantages and disadvantages andno single method can adequately assess all aspects of clinical competence
Important to use multiple modes of assessment to ensure knowledge, skills andattitudes are demonstrated to a pre defined level
Criteria for good assessment described as validity, reproducibility, equivalence,feasibility, educational effect, catalytic effect and acceptability
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 8
Review of current intern year assessment
An internal review for APC and PharmBA purposes only
Qualitative review that considered the three component of internyear assessments as discrete activities
Recommended strategies to build on the strength of eachcomponent to enable integrated, holistic and comprehensiveassessments to ensure public safety and confidence in recentlyregistered pharmacists
Intern year blueprintAn assessment blueprint that utilized the following methodologies
• Multiple choice questions
• Oral assessment
• Objective structured clinical examinations (OSCE)
• Mini clinical evaluation exercise (mini-CEX)
• Case based discussion (CbD)
• Multisource feedback (MSF)
• In-training assessment (ITA)
• Portfolio
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 9
Ontario Pharmacy Patient Care Assessment Tool(OPPCAT)
Common pharmacy practice based assessment tool
Collaborative Partners
Ontario College of Pharmacists
Faculty of Pharmacy, University of Toronto
School of Pharmacy, University of Waterloo
Hospital Pharmacy Residency Forum of Ontario
Ontario College of PharmacistsRegulatory body for the practice of pharmacy in Ontario, Canada.
Registration requirements for pharmacists in Ontario:
Graduation from an approved accredited pharmacy degree program
Successful completion of Canadian pharmacy licensing exam
Successful completion of Ontario pharmaceutical jurisprudence exam
Successful completion of a practice based assessment (PACE)
Others (e.g. language proficiency, good standing…)
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 10
Ontario College of Pharmacists Practice Assessment of Competence at Entry (PACE)
Competency-based assessment in a pharmacy practice site by anappointed and trained College assessor
PACE is assessment then, if needed, individualized self-directeddevelopment with a coaching pharmacist to address any performanceareas
PACE Process
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 11
Collaboration
Ontario College of Pharmacists collaborated with other organizations
At the right time and place
Sharing of ideas and human resources
Similar goal and outcomes
Common need lead to moving forward quickly with less formalcollaboration
Working group established as natural evolution
More formality and structure introduced after tool implemented
Implementation in Programs• Advanced Pharmacy Practice Experience
(APPE) in PharmD program• PharmD for Pharmacists program
University ofToronto
University ofToronto
• Patient Care Rotations in 4th year PharmD• PharmD Bridging Program
University ofWaterloo
University ofWaterloo
• Patient Care Rotations in OntarioHospital PharmacyResidency ProgramsHospital PharmacyResidency Programs
• Practice Assessment of Competence at Entry(PACE) for pharmacists
Ontario College ofPharmacists
Ontario College ofPharmacists
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 12
OPPCAT Development TimelineSpring2014
Residencyprograms
anduniversities
Fall 2014Ontario
College ofPharmacists
(OCP)
Fall2015
v.5focusgroups
May 2016
U of Torontolaunch v.6
July2016
Residencylaunch
January2017
U ofWaterlooand OCPlaunch
January2017
SatisfactionSurvey
January2018
Dataanalysis/revision
OPPCAT
NAPRA Entry-to-Practice
Competencies
AFPCEducationalOutcomes
CanadianPharmacy
Residency BoardStandards
Literaturereview
Competencybased
frameworkSolo taxonomy
Five point Likertscale
Domains,elements andbehaviouraldescriptors
Companionglossary
Framework:
Standards:
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 13
OPPCAT – 5 Domains and 20 Elements
Patient Care(8)
Communication andEducation (4)
Professionalism (2)Professional
Collaboration (3)
PracticeManagement (3)
Domain
Element
BehaviouralDescriptor
Rating Scale
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 14
Behavioural Descriptors
Demonstrate progressionfrom levels 1 to 5
Each descriptor has 3 components:• Guidance/support required• Effectiveness/quality• Consistency
Guidance
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 15
OPPCAT Satisfaction Survey ResultsAnonymous on-line survey to assess the practicality, usability andsatisfaction with OPPCAT: Preceptors/Assessors (n=171), Students/Residents (n=69)
Survey Results – For Consideration
Adding ‘not applicable’ (some programs)
Too long and difficult to fill out tool (programs using fillable pdf)
Site freezes; hard to log on (OCP assessors)
Tweaking of some behaviour descriptors in Patient Care domain
Assessing specific knowledge of behaviours (residency preceptors)
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 16
Next StepsValidation Study (planned)
Data from all 4 programs to be considered for validation
1 – 2 cohorts of data available for the study
OPPCAT Updates and Training:
Revision of OPPCAT to v.7
Dissemination of OPPCAT to other programs in Canada
Collaborate to standardize OPPCAT training
OPPCAT Training Organizations have own OPPCAT training programs for
assessors/preceptors
Preceptor Development Working Group (2018) to collaborate
Taking inventory of resources used by each program
Evaluating a platform for delivering online programming
Reviewing video resources
Assessing need for resources and modules
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 17
Lessons Learned: Ontario
• Collaboration possible due to the shared commitment and ‘investment’
• Timing was important – i.e. right time and place
• Need patience and perseverance
– Different organizations need to ensure buy-in across all leadership teams
• Funding/financial support for work as it evolves
Lessons Learned: Ontario
• Online assessment tool
– Consider technology solution/platform across all organizations
• Standardization of tool through combined preceptor trainingand shared resources
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 18
Lessons Learned: APC• Collaboration challenging but rewarding
• Need authorising mandate from senior management
• First step in a change process i.e. Kotters ‘burning platform’
• Enables broad commitment to the final product
• Time consuming
• Requires focusing on the end goal but flexibility may be required
• ‘From the balcony’ principles of Adaptive Leadership (Heifer,Grashow &Linsky. 2009)
Lessons Learned: APC
• Elicits and highlights different ‘political’ considerations
• Engagement across organisational boundaries requires persistence,awareness of others and sensitivity
• Need to pay attention to the emotional side of collaboration andnegotiation
• Need effective communication – don’t assume
Integrated Entry-to-Practice AssessmentStrategies
Philadelphia, PennsylvaniaSeptember 26 – 29, 2018
Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 19
Speakers Contact Information
Glenys Wilkinson B.Soc Sci., MSW, M.App Sci.
Diana Spizzirri R.Ph., BSc.Phm., ACPR, [email protected]
Annie Lee R.Ph., BScPhm., MSc(T), [email protected]