All Rights Reserved, Duke Medicine 2007. Clinical Competency Committees: Transitioning our Learners...

All Rights Reserved, Duke Medicine 2007


Clinical Competency Committees: Transitioning our Learners in GME

2013 Graduate Medical Education Conference

Predicting Effective Transitions to the NAS Palm Desert, CA – October 24, 2013

Kathryn Andolsek MD MPH

[email protected]


An Institutional Question?

Are trainees at Loma Linda achieving the competency required for unsupervised

practice that includes the delivery of safe, timely, equitable, effective and patient-

centered care?


“So how do you know?”


Objectives

1. Optimize your CCC for learners, your program and institution

2. Use evidence from medical education research to make best choices for CCC

3. Conduct a gap analysis and determine your Next Steps to strengthen your CCC



Competency Based Education


Dreyfus and Dreyfus Model

Dreyfus SE and Dreyfus HL. A 1980Carraccio CL et al. Acad Med 2008;83:761-7

Time, Practice, Experience

NoviceAdvanced Beginner

Competent

Proficient

Expert/Master


Milestones and Trajectories “D” “E”

Start PGY1

Finish PGY3

Fully Competent

A B

C

MilestonesLucey and

Boote

A’

D

E


Clinical Competency Committee (CCC)

• A group of faculty (and others) trained in knowing learner expectations and the understanding of evaluations

• Uniformly monitor trainee progression through the continuum of education using objective measures

• assist the PD with the assessment of competency, need for remediation, and feedback

Page 10



Participation

CCC Quiz


Accreditation Council for Graduate Medical Education (ACGME) Requirements/Guidance

Frequency Asked Questions

Published December 2012

Access at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf

FAQ Addresses:1. Role of CCC in resident assessment

2. Qualifications for CCC membership

3. Preparation for CCC members

4. Milestones and the CCC

5. CCC and Remediation

Proposed Common Program Requirements Section V

Comments accepted thru May 2013

Access at http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPR_Impact.pdf

ACGME NAS Webinar

New CPR 2016http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs_07012016_TCC.pdf

http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf


http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPR_Impact.pdf




1. The ACGME expects each program to form a CCC and develop its’ members by:

A. 2013

B. 2014

C. When the specialty begins to require milestones

D. The time of the first C.L.E.R. visit

E. Before the next Program Site visit

F. Before the next “Internal Review”

G. None of the above


1. The ACGME expects each program to form a CCC and develop its’ members by:

A. 2013

B. 2014

C. When the specialty begins to require milestones

D. The time of the first C.L.E.R. visit

E. Before the next Program Site visit

F. Before the next “Internal Review”

G. None of the above


CCC implementation required by…..

Proposed CPR Impact Statement: “The proposed effective date for these revisions is July 1, 2013. However, programs in Phase II of NAS will not be expected to be in full compliance with new requirement until July 1, 2014.”

FAQ:

ACGME Programs expected to form a CCC by June 2013

FAQ Addresses:

1. Role of CCC in resident assessment

2. Qualifications for CCC membership

3. Preparation for CCC members

4. Milestones and the CCC

5. CCC and Remediation

Accessed February 20th 2013 at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf




2. The minimum number of CCC members should be

A. 2

B. 3

C. 4

D. 5


2. The minimum number of CCC members should be

A. 2

B. 3

C. 4

D. 5


Common Program Requirements 7-1-2013


http://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/CCC_PEC_FAQs.pdf


3. The CCC members….

A. Vote on each resident’s performance

B. Provide a consensus on each resident

C. Only consider residents who need remediation plans

D. Only review residents in their last year of training

E. None of the above


3. The CCC members….

A. Vote on each resident’s performance

B. Provide a consensus on each resident

C. Only consider residents who needs remediation plans

D. Only review residents in their last year of training

E. None of the above






Role of CCC in resident assessment

• Members make consensus decision on the progress of each resident

• CCC utilizes:– Existing resident assessment data– Faculty observations– Data from milestone assessment (beginning July 2013)




4. The CCC must include:

A. Patients

B. Nurses

C. Peer selected residents

D. Core faculty members

E. Program director

F. All of the above

G. C, D, and E


4. The CCC must include:

A. Patients

B. Nurses

C. Peer selected residents

D. Core faculty members

E. Program director

F. All of the above

G. C, D, and E




Qualifications for CCC members

• faculty members

• Those who observe and evaluate residents; willing to make hard decisions

• Others such as assessment specialists, non-MD educators, resident (?)

• Webinar: Advisors/mentors excluded from discussion


http://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/CCC_PEC_FAQs.pdf 7-19-2013




5. According to the ACGME, faculty development needed by the CCC members includes:

A. Knowing their potential legal liability

B. Giving “bad news” to the resident after the milestone determination has been reached

C. Reaching a common agreement of milestones narrative meaning

D. Determining how many assessments are needed for any given milestones

E. Applying QI improvement principles to the evaluation process

F. Knowing the best remediation strategies for certain milestone suboptimal performance

G. All of the above

H. None of the above


5. According to the ACGME, faculty development needed by the CCC members includes:

A. Knowing their potential legal liability

B. Giving “bad news” to the resident after the milestone determination has been reached

C. Reaching a common agreement of milestones narrative meaning

D. Determining how many assessments are needed for any given milestones

E. Applying QI improvement principles to the evaluation process

F. Knowing the best remediation strategies for certain milestone suboptimal performance

G. All of the above

H. None of the above


Faculty Development for CCC members(…plans to develop training resources for CCCs)

• Evaluation is a core faculty competency but most will need additional training in evaluation process

• Discuss milestone narratives & reach common agreement of meaning

• Evaluation process training:– How to aggregate data– How to interpret evaluation data

• Venue for faculty member discussion about resident evaluation: – How many assessment needed for each milestone– Data quality– Application of QI principles as part of evaluation process

Accessed February 20th, 2013 at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf



6. A resident rotates on another specialty service. That specialist evaluates them as performing poorly. The CCC should….

A. Use the grade provided by the specialist as their grade in assessing resident performance against the milestones

B. Not consider the evaluation as it came from a different specialty than the program

C. Take the evaluation and apply it with other data to the resident’s program milestones

D. Vote whether the evaluation seems accurate and should be included in the overall review of the resident’s performance


6. A resident rotates on another specialty service. That specialist evaluates them as performing poorly. The CCC should….

A. Use the grade provided by the specialist as their grade in assessing resident performance against the milestones

B. Not consider the evaluation as it came from a different specialty than the program

C. Take the evaluation and apply it with other data to the resident’s program milestones

D. Vote whether the evaluation seems accurate and should be included in the overall review of the resident’s performance


Milestones and the CCC

• CCC will take data from evaluations and apply them to the milestones to mark progress of residents.

• The CCC will have the advantage of knowing how each of the specialists evaluated residents and can apply that knowledge as it marks the residents’ progress on the milestones.




7. Pilot assessments on the milestones have found that the first time the evaluation is done, it takes approximately _____for each resident.

A. 5 minutes

B. 10 minutes

C. 20 minutes

D. 30 minutes

E. 45 minutes

F. Up to an hour


7. Pilot assessments on the milestones have found that the first time the evaluation is done, it takes approximately ____ for each resident.

A. 5 minutes

B. 10 minutes

C. 20 minutes

D. 30 minutes

E. 45 minutes

F. Up to an hour


Time required: milestones and the CCC

• It takes a significant amount of time (up to an hour) to conduct the milestones assessment for each resident the first time the evaluation is done.

• Subsequent evaluations take less time.




Remediation and the CCC• Goal: identify residents who are not progressing

• CCC sets performance thresholds (until national benchmark data available)

• For early years - compare resident performance on milestones to peers in program

• When deficiency identified, interventions considered (may or may not be role of CCC?)– Mentor– Additional readings– Skill lab experiences– Added rotations– Extend education– Counseling to consider another specialty/profession




Benefits of CCC• For the resident

– Offers insight and perspectives of a GROUP of faculty

– Serves as an early warning system if resident fails to progress (opportunity for remediation)

• For the program director– CCC outcomes shared with resident so Program Director can be

advocate for resident learning

• For the faculty/program– Utilizes objective assessment measures making it easier for

evaluators– Chance to simplify assessment tools; evaluation system– Opportunities to improve programs - curriculum, rotation schedules,

supervision, mentorship (educational QI)


Evidence based: What the literature says….

• Group conversations more likely to uncover deficiencies in professionalism for students (Hemmer et al 2000)

• Group assessments improve inter-rater reliability and reduced range restriction in multiple domains (Thomas et al 2011)

• CCC helps ensure decisions are not arbitrary or capricious AND serves as checks and balances (especially identifying marginal resident) (Wu et al 2010, Schwind et al 2004, Williams et al 2005)

• No evidence (that) individuals dominate discussions or gang up (Williams et al 2005)

• 18% of resident deficiencies requiring remediation only became apparent through group discussion; average discussion 5 minutes per resident, range 1-30 minutes (Schwind et al 2001)

• Amount of time spent discussing a resident is frequently a measure of the severity of the problem/concern (Sanfrey et al 2012)


CCC: The Legal & Practical

Legal precedent suggests Courts will support educational institutions decisions about resident progression and graduation when:

• Decisions are reasonable (e.g. made at routine faculty meetings)• There is a shared understanding of expectations• Residents with performance issues are given Notice (informed of deficiency)

and Opportunity to Cure (improve)

(Missouri v. Horowitz 1978)

• Decisions are made “conscientiously and with careful deliberation”• Taking into account learners entire record

(Univ. of Michigan v. Ewing 1985)

Are your CCC proceedings and documentation “peer review protected”?


CCC Components, Practical Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future) could be used?

How are findings determined?

How will findings be conveyed to Program Director? (who makes ultimate decision)

How/by whom will findings be conveyed to learners?

What happens when CCC determines learner is performing sub-optimally?

What faculty development is required for CCC members?

How will you describe this process to your program?



Pre-Conference Survey Results

• Electronic survey link sent on February 19th

• 123/270 responded (46%)

• THANK YOU!



Think about…CCC Components, Processes & Procedures






Think about…CCC Components, Processes & Procedures Membership: How many members? Who is eligible/optimal? For how long?

Evidence Based: What the literature says…..

No single individual should make judgments about the competence of a trainee in isolation (Swing et al 2010)

assessment in competency based medical education benefits from the “wisdom of a group process in making judgment about trainee progress” (Holmboe et al 2010)

Group conversations more likely to uncover deficiencies in professionalism (Hemmer 2001)

Group assessment improved inter-rater reliability and reduced range restriction in multiple domains in an internal medicine residency (Thomas 2011)

5-7 members most effective – “balances out idiosyncrasies in judges ratings”(Williams et al 2012)

Maybe at least 3….








Consider: Maybe NOT the program director ??? Possibly the length of the training program










Evidence Based: What the literature says…..

The role of assessment in competency based medical education benefits from processes that are more continuous and frequent (Holmboe et al 2010)

So, what does “that mean?”











Consider: Complete confidentiality – nothing discussed outside of CCC No delusions of power Committed to objective assessment of each learner













What are the shared set of standards/thresholds? How will learner expectations be determined?

Evidence Based: What the literature says….• Competency based medical education requires developmental criteria = milestones,

learning objectives or benchmarks• Faculty should acquire shared mental models and understanding on what

competence should look like at various developmental stages (Holmboe et al 2011)










Think about…… CCC Components, Processes & Procedures







Evidence Based: What the literature says….Assessment must:• Be based on authentic encounters and frequent direct observation (Holmboe et al 2010; Carraccio et al 2002; Williams et al 2003; Govaerts et al 2007)

• Move away from just numbers; greater reliance on qualitative assessment (Govaerts et al 2007)


Potential Assessment Measures

• Use a variety• Include direct observation• Possibly 10? What “10” do you use?


Potential Assessment Measures• Use a variety• Possibly 10?

Rotation evaluations (by attending - direct observation for adequate time) Multi-rater (360) peer Multi-rater (360) nurse Multi-rater (360) patient Research project project Teaching skills evaluation Self-assessment Procedure/patient log summary Patient care outcomes (more difficult - maybe chart audit of outcomes or performance services

data) projecct Handoff evaluation Standardized patient video Simulation observation Milestones achievement Performance improvement project Compliance items




Consider a Dashboard


















Consider: It’s a consensus - not a vote It’s a RECOMMENDATION to the program director










How will findings be conveyed to PROGRAM DIRECTOR? (who makes the ultimate decision)

How are findings conveyed to ACGME?

How are findings “retained” by the Program (or should they?)

How will findings be conveyed to the LEARNERS?








How are findings determined? How will findings be conveyed to Program Director? (who makes ultimate decision)

How will findings be conveyed to the resident(s)?Evidence Based: What the literature says…..

Formative and summative assessment is critical to resident success (Carraccio et al 2002)

Faculty must become less fearful of providing meaningful (truthful) performance data (Holmboe et al 2011)











What happens when CCC determines resident is performing sub-optimally?











What happens when CCC determines resident is performing sub-optimally?

Consider:Collection of performance data for high-stakes use could adversely affect the accuracy of resident assessment and the usefulness of the results for formative purposes within a program (Swing et al 2009)

CCC may NOT be the venue for remediation and corrective action plans
















Evidence Based: Literature says…..• “assessor training is a key component of high quality assessment in residency

programs…evidence to suggest that faculty can be trained to improve the quality of their assessments”

(Dudak et al 2012)

• Faculty must build competency in teaching and assessment (Holmboe et al 2011)

• Faculty need: Feedback on range of scores compared with co-evaluators Basic knowledge of core psychometric concepts Time to learn and practice new teaching/assessment skills

(Holmboe et al 2011)













How will you describe this process to your program?













How will you describe this process to your program? Description in program manual (for faculty and learners) Discussed at GMEC, new resident/fellow orientation, faculty meetings


CCC: Next Steps1. Select members of CCC

2. Develop common expectations of CCC Role

3. Learn, develop or borrow milestones (resident expectations)

4. Define/refine (simplify!) assessment process– Evaluation tools (attempt to identify at least 10 assessments for each learner)– Who are evaluators (ample opportunity to observe learner performance)– Guidance for evaluators

5. Identify if anything missing and fill in the gaps

6. Write a CCC description so transparent to all (who, what, when, why)

7. CCC Members train/learn together (faculty development)

8. Use experience to improve the process (It’s educational CQI!)


Survey Results: Reported Best Practices

• CCC "assigned" to specific PGY, so they become "experts" at that level

• Everyone on committee works with residents with first hand knowledge of resident abilities

• generous amount of qualitative and quantitative multi-sourced evaluations for all fellows to allow us to compare across training level & by the program as a whole

• Each member of the committee has expertise in separate subspecialties and recognizes skill or deficiencies in their respective fields of interest

• Each resident knows the outcome within a day of the meeting and the advisers and PD are aware of any problems or concerns

• Combine small programs with CCC of the core

• Make a faculty development opportunity

• Use RMT


Potential Enhancements To Your CCC• Use CCC forum as opportunity for faculty development (content in addition to

resident evaluation). Read and discuss medical education journal article, latest ACGME Common Program Requirements, GME Program Goals.

• Identify thresholds for acceptable performance in areas such as compliance. (e.g. < 10% duty hour violation; < 95% hospital web module completion, < 85% conference attendance)

• For high performing residents with no questionable evaluations, the CCC can simply confirm acceptable performance and instead spend most of its’ time reviewing and discussing those residents with questionable performance who are potentially not meeting expectations. (but include all)

• Identify critical times for CCC to meet. e.g. Jan-February if contract issues and scheduling must be determined for the next academic year; prior to Program Director’s semi-annual review with residents.


Potential Enhancements To Your CCC• Frequency? Can the CCC can meet more frequently for a brief review of all residents

and review in detail the performance of one PGY level cohort each time.

• Combined programs (e.g. med-psych and med-peds) consider having their own CCC and use information from each Core CCC on resident performance.

• Include a description of CCC in the resident manual (and distribute to faculty)

• Include a professional staff healthcare team member who works closely with residents on CCC (e.g. social worker, nurse practitioner)

• Use CCC members to revise and simplify evaluation system (every evaluation form does not need to cover every competency, every rotation does not need to cover every milestone, etc)

• Have resident advisors (where applicable) prepare a presentation for the CCC on their advisee’s performance utilizing a variety of assessment tools.

• Evaluate it!


Other Best Practices?


From a list serve…

“I am beginning the process of negotiating the budget for our CCC committees I’m trying to get a sense of what programs have received….

Our ask to the hospital so far We have asked for 0.006 Faculty FTE’s and 0.003 coordinator FTE’s per resident plus a TBD startup bolus of hours. These numbers were arrived at based on the NIH type model we plan to use and a best-guess at time for faculty to review data.”XXXXXProgram Director


Participation

Given what you know now, what is the first thing you will do to enhance your CCC?


References• Albanese MA. Challenges in using rater judgments in medical education. J Eval Clin Pract. 2000;6:305–19.• Baker K. Determining resident clinical performance: Getting beyond the noise. Anesthesiology.

2011;115(4):862-878.• Carr SJ. Assessing clinical competency in medical houseofficers: how and why should we do it? Postgrad

Med J. 2004;80(940):63-6.• Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: From Flexner to

competencies. Acad Med. 2002; 77(5):361-367.• David DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-

assessment compared with observed measures of competence: A systematic review. JAMA. 2006;296(9):1094-102.

• Downing SM. Threats to the validity of clinical teaching assessments: What about rater error? Med Educ. 2005;39:353–5

• Dudek NL, Marks MB, Wood TJ, Dojeiji S, Bandiera G, Hatala R, Cooke L, Sadownik L. Quality evaluation reports: Can a faculty development program make a difference? Med Teach. 2012;34:e725-e731.

• Ginsburg S, McIlroy J, Oulanova O, Eva K, Regehr G. Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Acad Med. 2010;85:780–6.

• Greaves JD, Grant J: Watching anesthetists work: Using the professional judgment of consultants to assess the developing clinical competence of trainees. Br J Anaesth. 2000;84:525–33

• Govaerts MJ, van der Vleuten CP, Schuwirth LW, Muijtiens AM. Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Adv Health Sc Educ Theory Pract. 2007;12(2):239-260.

• Hamdy H, Prasad K, Anderson MB, Scherpbier A, Williams R, Zwierstra R, Cuddihy H. BEME systematic review: Predictive values of measurements obtained in medical schools and future performance in medical practice. Med Teach. 2006;28:103–16

• May 16.


References• Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81-112.• Haur KE, Mazotti L, O’Brien B, Hemmer PA, Tong L. Faculty verbal evaluations reveal strategies used to

promote medical student performance. Med Educ Online. 2011; 10.3402/meo.v16i0.6354. Epub 2011 Hemmer PA, Hawkins R, Jackson JL. Assessing how well three evaluation methods detect deficiencies in medical students' professionalism in two settings of an internal medicine clerkship. Acad Med. 2000;75(2):167-73

• Herbers JE Jr, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ. How accurate are faculty evaluations of clinical competence? J Gen Intern Med. 1989;4:202–8

• Holmboe ES: Faculty and the observation of trainees' clinical skills: Problems and opportunities. Acad Med. 2004;79:16–22

• Holmboe ES, Hawkins RE. Methods for revaluating the clinical competence of residents in internal medicine: a review. Ann Intern Med. 1998;129(1):42-8.

• Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessing in competency-based medical education. Med Teach. 2010;32:676-682.

• Holmboe ES, Ward DS, Reznick RK, Katsufrakis PJ, Leslie KM, Patel VL, Ray DD, Nelson EA. Faculty Development in Assessment: Noel GL, The missing link in competency-based medical education. Acad Med. 2011;86(4):460-467.

• Herbers JE Jr, Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do internal medicine faculty members evaluate the clinical skills of residents? Ann Intern Med. 1992;117:757–65.

• Issenberg SB, McGaghie WC, Waugh RA. Computers and evaluation of clinical competence. Ann Intern Med. 1999;130(3):244-5.

• Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA. 2009;302:1316–26..


References• Langsley DG. Medical competence and performance assessment. A new era. JAMA. 1991;266(7):977-80.• Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for

graduate medical education: A systematic review. Acad Med. 2009;84:301–9Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2010; 341:c5064.

• Nasca TJ, Philibert I, Brigham T, Flynn TC. The Next Accreditation System – Rationale and Benefits. N Engl J Med. 2012;366(11):1051-1056.

• Pangaro L. A new volcabulary and other innovations for improving descriptive in-training evaluations. Acad Med.1999;74(11):1203-7.

• Scavone BM, Sproviero MT, McCarthy RJ, Wong CA, Sullivan JT, Siddall VJ, Wade LD. Development of an objective scoring system for measurement of resident performance on the human patient simulator. Anesthesiology. 2006;105:260–6.

• Stillman PL, Swanson DB, Smee S, Stillman AE, Ebert TH, Emmel VS, Caslowitz J, Greene HL, Hamolsky M, Hatem C, Levenson DJ, Levin R, Levinson G, Ley B, Morgan GJ, Parrino T, Robinson S, Willms J. Assessing clinical skills of residents with standardized patients. Ann Intern Med. 1986;105:762–71.

• Swing SR, Clyman SG, Holmboe ES, Williams RG. Advancing Resident Assessment in Graduate Medical Education. J Grad Med Educ. 2009;1(2):278-86.

• Swing SR; International CBME Collaborators. Perspectives on competency-based medical education from the learning sciences. Med Teach. 2010;32(8):663-8.

• Wilkinson JR, Crossley JG, Wragg A, Mills P, Cowan G, Wade W. Implementing workplace-based assessment across the medical specialties in the United Kingdom. Med Educ. 2008;42(4):364-73.

• Williams RG, Klamen DA, McGaghie WC. Cognitive, social and environmental sources of bias in clinical performance ratings. Learn Med. 2003;15(4):270-292.

• Williams RG, Sanfey H, Chen X, Dunnington GL. A controlled study to determine measurement conditions necessary for a reliable and valid operative performance assessment. Annals of Surg. 2012;256(1):177-187.

Date post:	12-Jan-2016
Category:	Documents
Upload:	erin-rice
View:	214 times
Download:	0 times

All Rights Reserved, Duke Medicine 2007. Clinical Competency Committees: Transitioning our Learners...

Documents