Do the National Board of Medical Examiners (NBME) subject examinations predict success on the MCCQE Part I?

Author: Dr. Carolina Escudero

Date: September 24, 2011

Introduction

• Evaluation of medical students is an important but challenging process

• Essential to identify students who are not meeting the requisite minimum level of knowledge and clinical competence

• Many different tools have been used in the evaluation process

• several tools shown to be poor at identifying students who are below average (Colleti, 2000; Speer et al., 2000)

2

Introduction

• National Board of Medical Examiners (NBME) subject examinations

• Discipline-based, standardized multiple choice tests • Provide normally distributed scores and have correlated

with overall clerkship performance (Nahum, 2004) • Security of the examination • Widely used in clerkship evaluation

- In 2001, 67% of medical schools in Canada and US used the NBME subject examination in pediatrics (Kumar et al., 2004)

3

Introduction

• NBME subject examinations are available in the following 6 subjects:

• Pediatrics • Internal medicine • Surgery • Obstetrics/gynecology • Psychiatry • Family Medicine

• Memorial University uses 5 NBME examinations to

evaluate students during their clerkship rotations

4

5

NBME subject examination use by English-speaking Canadian medical schools * From personal correspondence in April, 2011

MUN Queen’ s U. U. of Manitoba UBC

Dalhousie U. McGill U. U. of Western Ont. U. of Alberta McMaster U. U. of Toronto

U. of Ottawa NOSM U. of Saskatchewan U. of Calgary

No NBME exams used

≥ 5 NBME exams used

Some NBME exams used

NBME and Licensing Examinations

• NBME exams correlate well with scores on the USMLE Step 2

• Correlations ranging from r = 0.53 - 0.74 (Ripkey et al., 1999; Myles et al., 2002)

• Suggested that low scores on the NBME could predict students at risk for low scores on the USMLE Step 2

• Canadian licensing examination (MCCQE Part I) • No previously published research on using the NBME to

predict MCCQE Part I performance despite wide uptake in Canada

6

Purpose

• To compare Memorial University medical students’ scores on the NBME subject examinations to their scores on the MCCQE Part I

• To help determine if NBME scores can identify students at risk of MCCQE Part I failure

7

Methods

• Obtained NBME scores from 4 academic years

• Total MCCQE Part I, MCQ, CDM, and discipline-specific MCQ scores collected

• To be involved in the analysis, NBME and the MCCQE scores were required

• Only the results of the first attempt for each examination were included

• Pearson’s correlation, chi square, and ROC analysis were performed

8

Confidentiality

• Student names and randomly generated numerical identifiers sent to the Medical Council of Canada (MCC)

• MCC matched MCCQE Part I scores to name and numerical code

• Student names removed and the remainder sent the Canadian Post-MD Education Registry (CAPER)

• List of identifying numbers and NBME scores sent to CAPER

• CAPER matched the MCCQE Part I scores and the NBME scores

• After matching, the numerical codes were removed and the de-identified list was sent to the authors

• Student confidentiality was ensured as reverse de-identification could not be performed

9

Results

• 224 student scores were analyzed

• Mean NBME scores: • Pediatrics: 73.8 (SD 7.0) • Internal Medicine: 75.9 (SD 6.7) • Surgery: 70.9 (SD 7.5) • Obstetrics/Gynecology: 73.6 (SD 6.6) • Psychiatry: 78.0 (SD 7.0)

• Mean MCCQE Part 1 score: 522.8 (SD 63)

10

Results

• Correlations between the examinations • NBME exams and total MCCQE Part I score (r=0.519-

0.613, p=0.000) • NBME exams and the MCCQE part I MCQ score (r=0.506-

0.605, p=0.000) - Similar to those reported between the NBME exams and the

USMLE Part II (0.53-0.74)

• NBME exams and their respective subject-specific MCQ scores (r=0.333-0.489, p=0.000)

11

• 6 students failed the MCCQE Part 1 • 3 passed all of the NBME exams • 3 failed 2 or more examinations

• 12 students failed at least one NBME examination (score <60)

• 25% of these students failed the MCCQE Part 1 - 1 NBME exam failed: 0% - 2 or more NBME exams failed: 75%

• Failing at least 1 NBME exam gives an odds ratio of failing the MCCQE Part I of 23.2 (95% CI = 4.1-131.5)

Results

• ROC Analysis • To determine the sensitivity and specificity of NBME

threshold scores in predicting MCCQE Part 1 failure

Results

Average NBME score

ROC Curve Comparing Average NBME Score to MCCQE Part 1 Performance

ROC Curve Comparing Average NBME Score to MCCQE Part 1 Performance

Average NBME score of ≤69 gives 83% sensitivity and 83% specificity

Average NBME score of ≤70 gives 100% sensitivity and 76% specificity

Average NBME score of ≤68 gives 67% sensitivity and 90% specificity

NBME Exam

Area under the Curve

P value

Overall Average

0.922 0.000

Pediatrics 0.932 0.000

Internal Med.

0.898 0.001

Surgery 0.886 0.001

Obs/Gyne 0.830 0.006

Psychiatry 0.819 0.008

• ROC Analysis for average NBME score • Average NBME score of <70 is a good

predictor of students at risk of MCCQE Part 1 failure

• Students with an average NBME score <70 are significantly more likely to fail the MCCQE Part 1 (χ2 = 19.9, p = 0.000)

Results

• NBME subject examinations show moderate correlations with MCCQE Part I scores

• Failure of at least one of the NBME subject examinations is associated with an increased risk of MCCQE Part I failure

• When looking at the average score on the NBME

examinations, a cut-off score of <70 gives a good sensitivity and specificity for identifying students at increased risk of MCCQE Part I failure

Overall Findings

• Results from only one medical school • Were not able to correct for the time at

which students wrote the examination during their clerkship year

• Studies show improvement in scores when NBME written later in the clerkship year (Reteguiz et al, 2002; Manley et al, 2006)

• Failure of either the NBME subject

examinations or the MCCQE Part I is an uncommon event

• Small number of students who failed the MCCQE Part I, it is difficult to draw firm conclusions from these results

Study Limitations

• Dr. Anne Drover (Memorial University)

• Medical Council of Canada (Dr. Tim Wood)

• CAPER

Acknowledgements

References

1. Nahum, G.G. (2004). Evaluating medical student obstetrics and gynecology clerkship performance: Which assessment tools are most reliable? American Journal of Obstetrics and Gynecology, 191, 1762-71.

2. Colletti, L.M. (2000). Difficulty with negative feedback: face-to-face evaluation of junior medical student clinical performance results in grade inflation. Journal of Surgery Research, 90, 82-7.

3. Speer, A.J., Solomon, D.J., and Fincher, R.M.. (2000). Grade inflation in internal medicine clerkships: results of a national survey. Teaching and Learning in Medicine, 12, 112-6.

4. Reteguiz, J.A. and Crosson, J. (2002). Clerkship order and performance on family medicine and internal medicine national board of medical examiners exams. Family Medicine, 34(8), 604-8.

5. Manley, M. and Heiss, G. (2006). Timing bias in the psychiatry subject examination of the national board of medical examiners. Academic Psychiatry, 30, 116–119.

6. Levine, R.E., Carlson, D.L., Rosenthal, R.H., Clegg, K.A., and Crosby, R.D. (2005). Usage of the national board of medical examiners subject test in psychiatry by U.S. and Canadian clerkships. Academic Psychiatry, 29, 52–57.

7. Barzansky, B., and Etzel, S.I. (2001). Education programs in US medical schools, 2000-2001. Journal of the American Medical Association, 286, 1040-1055.

8. Kumar, A., Gera, R., Shah, G., Godambe, S. and Kallen, D.J. (2004). Student Evaluation Practices in Pediatric Clerkships: A Survey of the Medical Schools in the United States and Canada. Clinical Pediatrics, 43, 729-735.

9. Ogunyemi, D. and Taylor-Harris, D. (2005). Factors that correlate with the U.S. Medical Licensure Examination Step-2 scores in a diverse medical student population. Journal of the National Medical Association, 97(9), 1258-62.

10. Myles, T.D. and Henderson, R.C. (2002). Medical licensure examination scores: relationship to obstetrics and gynecology examination scores. Obstetrics and Gynecology, 100(5 Pt 1), 955-8.

11. Ripkey, D.R., Case, S.M, and Swanson, D.B. (1999). Identifying students as risk for poor performance on the USMLE Step 2. Academic Medicine, 74 (10), S45-8.

12. Hemmer, P.A. and Pangaro, L.N. (2002). Natural history of knowledge deficiencies following clerkships. Academic Medicine, 77, 350–3..

13. Minion, D.J., Donnelly, M.B., Quick, R.C., Pulito, A., and Schwartz, R. (2002). Are multiple objective measures of student performance necessary? The American Journal of Surgery, 183, 663–5.

14. Kuo, A.A and Slavin, S.J. (1999). Clerkship curricular revision based on the Ambulatory Pediatric Association and the Council on Medical Student Education in Pediatrics guidelines: does it make a difference? Pediatrics, 103(4 Part 2), 898-901.

15. Rockney, R.M. and Allister, R.G. (2005). Dropping the shelf examination: does it affect student performance on the United States Medical Licensure Examination Step 2? Ambulatory Pediatrics, 5, 240 243.

16. Potts, M.J. and Phelan, K.W. (1997). A drop in pediatric subject examination scores after curriculum changes that emphasize general pediatric topics. Archives of Pediatrics and Adolescent Medicine, 151 (9), 938-42.

References

Questions?

NBME Exam

Cut-off for 83%

Sensitivity

Associated Specificity

Overall Average

≤69 83%

Pediatrics <65 93%

Internal Med. <71 78%

Surgery <66 78%

Obs/Gyne <70 73%

Psychiatry <78 55%