The MiniCEX and the The MiniCEX and the Evaluation of Clinical SkillsEvaluation of Clinical Skills

National Health Service

Foundation Training Program

Patient Care CompetenciesPatient Care Competencies

Gather accurate, essential information from all sources, including medical interviews, physical examinations, medical records and diagnostic/therapeutic procedures

Make informed recommendations about preventive, diagnostic and therapeutic options and interventions that are based on clinical judgment, scientific evidence, and patient preference.

Patient Care: ThemesPatient Care: Themes

Clinical skills essential to patient careCannot make “good” decisions unless you

work with good and accurate information– GIGO principle

Evaluation of clinical skills requires direct observation

Workshop ObjectivesWorkshop Objectives

Review current state of – Physician clinical skills– Faculty evaluation skills

Understand the importance of direct observation by faculty for assessment of clinical skills

Workshop ObjectivesWorkshop Objectives

Discuss practical strategies for focused direct observation

Review rater training methods

– Direct Observation of Competence (DOC) training

Workshop ElementsWorkshop Elements

Mini-Lectures: – State of clinical skills– Quality of faculty ratings

Direct observation exercises– Performance dimension exercise– Frame of reference training

Video ExerciseVideo Exercise

Situation: A foundation trainee performs a medical interview in the outpatient setting. Using the MiniCEX form provided, please rate the performance of this trainee.

Key “Basic” Clinical SkillsKey “Basic” Clinical Skills

Medical interviewingPhysical examinationsCounseling/patient educationClinical judgment/reasoningReflective practice

– Self-directed learning– Professional growth and improvement– Medical errors

Are Clinical Skills Important?Are Clinical Skills Important?

Where do clinical skills fall into the hierarchy of physician competencies and mastery in an era of advanced technology?

Diagnosis and Medical InterviewDiagnosis and Medical Interview

Hampton (BMJ, 1975):– Medical interview: 82%– Physical exam: 9%– Laboratory: 9%

Kirch (Medicine, 1996)– Medical interview (+PE): 70%– Imaging: 35%

Importance of Sound Clinical SkillsImportance of Sound Clinical Skills

Diagnostic errors– Inaccurate/ incomplete medical interview

one of leading causes (Bordage)• Wrong information leads to wrong decisions

Patient satisfaction– Higher with better communication skills

Patient self-care– Better adherence and outcomes associated with

better physician communication skills

Clinical Reasoning: A PrimerClinical Reasoning: A Primer

Patient/situation

characteristicsPrior knowledge

Problem Representation*

Information GatheringContext

Evaluation Action

Gruppen and Frohna, International Handbook on Research, 2002

Clinical Skills: U.K. TraineesClinical Skills: U.K. Trainees

Fox (2000)– Voluntary study of 22 PRHOs using OSCE– Only 45% with passing score on drug advice

communication station– 0% passed locomotor system examination

Evans (2004)– 26 new PHROs

– All had passed 22 station OSCE in medical school

– Majority failed skill stations in blood pressure measurement and cannulation

Clinical Skills: U.S. TraineesClinical Skills: U.S. Trainees

Stillman (1990)– OSCE: wide variability in graduating

medical student clinical skills Mangione (1997)

– Deficient cardiac and pulmonary auscultatory skills

– Medical students, FP and IM residents– Replicated findings in Canada and U.K.

Clinical Skills: Practicing MDsClinical Skills: Practicing MDs

Ramsey (1998)– Incomplete history-taking / preventive health

screening among Primary care physicians in Northwest Unites States

Braddock (1999)– Study of informed decision making (IDM) and

counseling– Simple analysis of presence or absence of 7 key

elements– 1058 outpatient visits: only 9% of visits met minimal

criteria for IDM

Importance of Faculty: U.K. StudiesImportance of Faculty: U.K. Studies

Grant (Med Educ, 2003)– Inadequate coverage and frequency of

supervision activities– Discordance between specialist registrars and

attendings Kilminster (Med Educ, 2000)

– Systematic review of supervision– Better supervision associated with improved

patient safety and quality of care.

Importance of Faculty: U.S. StudiesImportance of Faculty: U.S. Studies

Inpatient Study (Lancet, 2003)– Reviewed 100 consecutive admissions to

teaching service in U.S.– Faculty detected 26 PExam findings missed by

house officer that impacted patient’s care Outpatient Studies

– Two separate studies showed that faculty assessment disagreed with that of house officer in up to 30% of patients

Clinical Skills: Themes Clinical Skills: Themes

Deficiencies exist across continuumSpecific skills more “error-prone”

– Eg: musculoskeletal and neuro exams

Not detected by other evaluation methods

– Performance of basic clinic skills does not correlate with performance in other dimensions of competence

Clinical Skills: ThemesClinical Skills: Themes

House officers:

– Aware of importance

– Recognize under-emphasisWithout detection deficiencies in

clinical skills cannot be corrected

Miller’s PyramidMiller’s Pyramid

KNOWS

KNOWS HOW

SHOWS

HOW

DOES

MCQ EXAM

Extended matching / CRQ

OSCE

Portfolios

Faculty ObservationFaculty Observation

Faculty Observation / Rating SkillsFaculty Observation / Rating Skills

Patient care settings– Ratings based mostly on perceived

knowledge and personality– Little evidence of direct observation– Significant “Halo” effect

Gray, Thompson, Haber, Grant, etc.

Faculty Observation / Rating SkillsFaculty Observation / Rating Skills

Research settings– Poor inter-rater reliability– Brief rater training methods ineffective

• Didactic instructions

• Demonstration videos without practice

– Accuracy: structured > open-ended forms– Increased accuracy discriminative ability

Kalet, Herbers, Noel, Kroboth

Faculty as Raters – Key IssuesFaculty as Raters – Key Issues

Faculty do not observe actual performance

Faculty ratings lack:– Reliability– Accuracy/validity

Content specificity– How comfortable are you with own skills?

Improving Faculty Ratings: SolutionsImproving Faculty Ratings: Solutions

Step 1: Getting faculty to observe– Required a part of Foundation Program– Focused observations are logistically possible– 5 to 10 minute observations are valuable– Build on faculty “epiphany”

• The “you will not believe what I saw today” experience

– Provide “usable tool”

Foundation Mini - CEX ToolFoundation Mini - CEX Tool

Simple rating scale using 6 dimensions and overall rating

“Structured” approach to direct observationDirect assessment of actual patient careIncorporation of CEX into daily activitiesEvidenced-based

Research: Mini - CEX ToolResearch: Mini - CEX Tool

Two large scale U.S. studies involving 36 total residency programs

Logistically feasible to incorporate miniCEX into daily activities

High satisfaction among house staff Good to excellent reliability characteristics Overall scores and interpersonal scores

correlated with trainee’s ECFMG OSCE scores

Logistics: Outpatient ClinicLogistics: Outpatient Clinic

One mini-CEX per trainee per day per week– One attending observes portion of first visit

of the day– Minimizes disruption of clinic– Perform over course of academic year– Easy to obtain 6-8 Mini-CEX’s per year per

trainee in single setting

The Patient EncounterThe Patient Encounter

Sampling “parts” of the encounter:

INTERVIEWPHYSICAL

EXAMCOUNSELING

Solutions: Step 2Solutions: Step 2

Improve reliability– Multiple brief observations– Perform over time: outpatient setting allows

for longitudinal observation– Involve multiple faculty– MiniCEX: sufficient reliability for pass/fail

determinations after just 4 observations

Solutions: Step 3Solutions: Step 3

Improve accuracy and validity

– Most difficult step

– Use structured rating forms

– Rater training (faculty development)

• Caveat: brief “one time” interventions do not work

Does Faculty Training Work?Does Faculty Training Work?

Performance Appraisal Literature:Can reduce rating errorsCan improve discriminative abilityCan improve accuracy

Approaches to Rater TrainingApproaches to Rater Training

Behavioral Observation TrainingPerformance Dimension TrainingFrame of Reference TrainingDirect Observation of Competence

Training

Videotape Exercise: BOTVideotape Exercise: BOT

Situation: An attending is performing a miniCEX of a house officer performing a physical exam.

Questions:– How well did this attending evaluate the

house officer?– How was the house officer-patient

interaction affected?

Basic Faculty Observation SkillsBasic Faculty Observation Skills

Prepare for the observation– Faculty: Know what you’re looking for

– Resident: Let them know what to expect

– Patient: Let them know why you are there

Minimize intrusiveness – correct positioning Minimize interference with the house officer-

patient interaction Avoid distractions

TriangulationTriangulation

DESK

Resident

Patient

Attending

Basic Observation StrategiesBasic Observation StrategiesIncrease the amount of “sampling”

– More observations lead to more accurate evaluations (“practice makes perfect”)

Use of observational “aides”

– Behavioral diary to record observed performance.

– U.S. study: simple 3X5 card diary lead to increased comments on forms

Performance Dimension TrainingPerformance Dimension Training

Group exercises designed to familiarize faculty with the specific elements of a competency

Should involve discussion of the criteria required for each element

Use defined, agreed upon elements of a competency to calibrate faculty– Playing from the “same sheet of music”

PDT ExercisePDT Exercise

In your small group, discuss what should be the basic components of an effective medical interview for a foundation trainee performing an outpatient consultation

Frame of Reference TrainingFrame of Reference Training

Goal is to improve “judgment” and accuracySteps in FOR training:1. Group performance dimension training (PDT)

exercise

2. Review clinical vignettes that describe critical incidents of performance: unsatisfactory to average to superior

Frame of Reference TrainingFrame of Reference Training

3. Faculty, using framework developed in PDT exercise, provide ratings on a behaviorally anchored rating scale (BARS)

4. Session trainer provides feedback on what “true” ratings should be for each vignette along with rationale

5. Group finishes by discussing discrepancies between trainer’s ratings and the participants’ ratings

Frame of Reference TrainingFrame of Reference Training

Most difficult aspect of FOR:

– Setting the actual standards that distinguish between levels of performance

– Reaching agreement and/or consensus among teaching faculty

DOC TrainingDOC Training

Combination of:– Behavioral observation training– Performance dimension training– Frame of reference training– “Live” practice in observation with

standardized residents/patients• Individual evaluation and feedback• Group debrief with Eval and FB

DOC Training TrialDOC Training Trial

Randomized controlled trial of 40 faculty from 16 residency programs

DOC training:– High satisfaction (favorite aspect of course)– Increased comfort in observation– Changed rating behavior at 8 months– Increased accuracy in identifying

unsatisfactory performance

Direct Observation: ChallengesDirect Observation: Challenges

Like all skills, requires training and practice

Faculty “calibration” important– Agreeing on “metrics” of performance– Faculty comfort with own skills

Faculty training– Brief interventions mostly ineffective

Observation: Helpful HintsObservation: Helpful Hints

Sample “parts” of the visit:– History-taking– Physical examination– Counseling

Perform longitudinally– No need to do it all at once

Agree on performance metrics with faculty

SummarySummary

Basic clinical skills are important: so is the need to observe them!

Observation is a complex skill that requires training and practice

Direct observation by educators will remain a critical component of both evaluation and feedback