1
Undergraduate Medical Education
Program Evaluation Committee
Medical Council of Canada Qualifying Examinations – Part I
University of Toronto Results Summary
December 2009
APPENDIX 39APPENDIX 41
2
1.0 Introduction In the spring of their graduating year, the majority of Canadian medical students take Part I of the Medical Council of Canada’s qualifying/licensing examination (MCCQE). As soon as 18 months later, during the fall of their second year of postgraduate (i.e., residency) training, Canadian medical graduates then complete Part II of the MCCQE. Part I of the examination is a written examination. In addition to an overall score, domain scores are determined for the multiple choice and clinical decision making components of the examination as well as discipline-specific domain scores for Medicine, Obstetrics and Gynaecology, Pediatrics, Psychiatry, Population Health and Preventive Medicine, Surgery and, since 2006, Family Medicine. There is also a sub-score provided for the “Consideration of the Legal, Ethical and Organizational Aspects of the Practice of Medicine” (CLEO) domain of the examination. A comparison of performance on both parts of the MCCQE—overall and for each of the various domain mean scores—by University of Toronto medical graduates and all Canadian medical graduates, serve as key evaluation measures of Toronto’s Undergraduate Medical program. The Faculty’s Undergraduate Medical Education Program Evaluation Committee (UMEPEC), a faculty-student committee, was formed to oversee the evaluation of the performance of Toronto’s undergraduate medical curriculum (Note: UMEPEC’s current membership is listed in the Appendix below). In 2004, in collaboration with the Associate Dean, Undergraduate Medical Education and in preparation for Toronto’s most recent Accreditation Survey, UMEPEC identified two MCCQE performance targets: 1. That the mean standardized scores for University of Toronto medical graduates would exceed
the Canadian national average (which also includes Toronto’s graduates) overall and for all domains of Part I and Part II of the MCCQE; and,
2. That Toronto would be ranked within the top half of Canadian Medical schools according to
the mean standardized score overall and for all domainss of Part I and Part II of the MCCQE. The results of Part I of the MCCQE are reported here (the results of Part II of the MCCQE are reported separately). 2.0 Medical Council of Canada’s Qualifying Examination (MCCQE) – Part 1 Results On an annual basis, the Medical Council of Canada provides each medical school with reports that summarize the mean performance of its students as well as those of all Canadian medical graduates on the various aspects of the MCCQE - Part I examination. A comparison the Toronto and Canadian medical student mean scores and Toronto’s ranking among Canadian Medical schools for each of the Part I examination domains are presented below. 2.1 Analyses of Mean Scores Overall - Over the past 18 years (1992-2009), Toronto’s undergraduate medical students have generally achieved a slightly higher mean score in comparison to all Canadian undergraduate medical students on the MCCQE - Part I’s overall mean score (Figure 1).
3
Domains
In terms of the various domains of the MCCQE - Part I, Toronto students’ most recent year’s average mean scores equalled or exceeded that of all Canadian medical graduates on all but two domains of the examination: Clinical Decision Making and Medicine. A review of the 18-year trends for the various examination domains reveals that Toronto students have generally achieved mean scores comparable to all Canadian medical graduates for the examination’s Clinical Decision Making and Multiple Choice domains (Figures 2 and 3), although in the past 2 years the Toronto mean score for the Clinical Decision Making domain has been lower than the corresponding Canadian medical graduates’ mean score. Since the CLEO (Consideration of the Legal, Ethical and Organizational aspects of the Practice of Medicine) domain was introduced into the MCCQE - Part I in 2000, Toronto students have consistently achieved mean scores that clearly exceed those achieved by all Canadian medical graduates (Figure 4). Over the past 18 years (1992-2009), Toronto has had variable results on the various discipline-specific domains of the MCCQE - Part I examination when compared to the corresponding mean scores achieved by all Canadian medical graduates. For much of the past decade, Toronto students’ mean score on the Population Health and Preventive Medicine domain of the examination has clearly exceeded that for all Canadian medical graduates (Figure 5). Over that same period, the mean scores for Toronto graduates on the Medicine (Figure 6) and Surgery (Figure 7) domains have been comparable to, and often exceeding, the corresponding Canadian averages. However, this year, for the second time in the past 5 years, Toronto students’ mean score on the Medicine domain was lower than the corresponding Canadian medical graduates’ mean score. Up until the past 4 to 5 years, Toronto students’ mean scores on the Obstetrics and Gynaecology (Figure 8), Pediatrics (Figure 9) and Psychiatry (Figure 10) domains of the examination have generally been lower than those for all Canadian medical graduates. However, Toronto students’ mean scores on the Obstetrics and Gynaecology (Figure 8) has equalled or exceeded the corresponding Canadian mean scores for each of the past 5 years. Finally, since 2006, the first year that a mean score for the Family Medicine component was provided, Toronto students have achieved a mean score that exceeded the corresponding Canadian medical graduate mean score (Figure 11). As suggested above, the mean scores achieved by Toronto’s medical graduates exceeded those of all Canadian medical graduates on every aspect of the 2009 MCCQE - Part I examination except for three. As such, the program evaluation performance target that the mean standardized scores for University of Toronto medical graduates exceed the Canadian average on all aspects of the MCCQE-Part 1 was not achieved for Clinical Decision Making, Pediatrics, and Medicine by the graduating class of 0T9. 2.2 Analyses of School Rank Overall - Over the past 17 years (1993-2009), Toronto medical students’ MCCQE - Part I overall mean scores have consistently been ranked in the middle to upper middle rankings—mostly in the 2nd quartile (Figure 12). In terms of the various domains of the MCCQE - Part I, results from the 2009 examination ranked Toronto students’ average mean scores for all but 2 domains—Clinical Decision Making and Medicine—of the examination in the upper half of the mean
4
scores achieved by each of Canada’s 17 medical schools. Details concerning Toronto students’ rankings in each of the various MCCQE - Part I domains over the past 17 years are described in turn below. Multiple Choice Question Domain – Over the past several years, Toronto graduates’ mean scores have consistently been ranked in the 2nd quartile, and in 3 of the most recent 5 years in the 1st quartile (Figure 13). Clinical Decision Making Domain – In the past, Toronto graduates’ mean scores have consistently been ranked in the 2nd quartile. However, in the last 3 years Toronto’s graduates’ ranking has fallen into the 3rd quartile, being ranked 10th in 2007, 13th in 2008, and 12th in 2009 (Figure 14). Medicine Domain – For the majority of the past several years, Toronto graduates’ mean scores have consistently been ranked in the 2nd quartile. However, results over the past 5 years have been quite varied—being ranked in the 1st quartile from 2006 to 2008, and in the 3rd quartile (ranked 11th) in 2005 and 2009 (Figure 15). Obstetrics and Gynaecology Domain – Up until the past 5 years, Toronto graduates’ mean scores had been consistently ranked in the 3rd and 4th quartiles. However, in the past 5 years Toronto’s mean scores have been ranked in the upper half of mean scores achieved by graduates from Canada’s 17 medical schools—being ranked 4th in 3 of those years (Figure 16). Pediatrics Domain – Over the past several years, Toronto graduates’ mean scores have consistently been ranked in the 3rd quartile. However, in 2009 this ranking rose into the 2nd quartile (Figure 17). Preventive Medicine And Community Health (PMCH) and Population Health, Ethical, Legal and Organizational (PHELO) Domain – Over the past several years, Toronto graduates’ mean scores have consistently been ranked in the in the upper half of the mean scores achieved by graduates from Canada’s 17 medical schools—being ranked 1st overall in 5 of the last 7 years (Figure 18). Psychiatry Domain – Over the past several years, Toronto graduates’ mean scores have generally been ranked in the 3rd quartile. However, in 3 of the past 4 years, a ranking within the 2nd quartile has been achieved (Figure 19). Surgery Domain – Over the past several years, Toronto students’ mean scores have generally been ranked in the 2nd quartile (Figure 20). Family Medicine Domain – Since 2006, when the Family Medicine domain score was first reported, Toronto students’ mean score has been consistently ranked in the upper half of scores achieved by graduates from Canada’s 17 medical schools, being ranked 2nd from 2006 to 2008 and 8th in 2009 (Figure 21).
5
Consideration of the Legal, Ethical and Organizational Aspects of the Practice of Medicine (CLEO) Domain – Since the introduction of the CLEO component in 2000, Toronto graduates’ mean scores have consistently been ranked in the 1st quartile (Figure 22). As suggested above, in 2009 the mean scores achieved by Toronto graduates on two domains of the MCCQE - Part I resulted in Toronto graduates being ranked in the lower half of Canada’s 17 medical schools. As such, the program evaluation performance target that Toronto be ranked within the top half of Canadian Medical schools according to the mean standardized score for all aspects of the MCCQE - Part I was not achieved for Clinical Decision Making and Medicine by the graduating class of 0T9. 3.0 Summary As summarized in the table below, Toronto’s performance relative to all Canadian medical graduates overall and on the 10 domains of the MCCQE - Part I in 2009 is quite consistent with the relative performance of Toronto medical graduates on that examination over the past 5 years. These results are also depicted graphically in Figures 23 and 24. Toronto – Canadian Toronto’s Ranking Mean Score Difference Median (and range) Median (and range) Population Health 27 (20 to 41) 1 (1 to 2) CLEO 27 (17 to 41) 2 (1 to 4) Family Medicine* 12 (7 to 23) 2 (2 to 8) MCQs 12 (8 to 22) 4 (2 to 6) Medicine 11 (-4 to 14) 4 (3 to 11) Overall 7 (6 to 18) 5 (3 to 7) Surgery 8 (6 to 20) 6 (4 to 7) Ob/Gyn 6 (0 to 8) 6 (4 to 8) Psychiatry 3 (-6 to 7) 7 (6 to 12) Pediatrics 0 (-9 to 11) 9 (3 to 13) CDM 0 (-7 to 4) 10 (7 to 13)
* only reported for past 4 years (since 2006) Given this year’s results and similar trends over the past several years, the Committee will continue to closely monitor upcoming years’ results, particularly the Clinical Decision Making and Medicine domain scores. In addition, given that Toronto has not met the performance targets on the Clinical Decision Making domain for each of the past 3 years, the Committee recommends that this be further explored.
6
Appendix
Undergraduate Medical Education Program Evaluation Committee 2009/2010 Membership
Professor Geoff Anderson, Department of Health Policy, Management, and Evaluation
Andrew Brown, 3rd Year (1T1) Medical Student
Professor Mark Hanson, Associate Dean – Admissions and Department of Psychiatry
Professor Bart Harvey, Department of Family and Community Medicine (Chair)
Samira Jeimy, 2nd Year (1T2) Medical Student
Kayi Li, 3rd Year (1T1) Medical Student
Professor Katherine MacRury, Department of Obstetrics and Gynaecology
Waed Mallah, 1st Year (1T3) Medical Student
Professor Jodi McIlroy, Department of Medicine and Wilson Centre for Research in Education
Ali Okhowat, 4th Year (1T0) Medical Student
Professor Richard Pittini, Department of Obstetrics and Gynaecology
Professor Anita Rachlis, Department of Medicine
Professor Jay Rosenfield, Vice-Dean, UME and Department of Pediatrics
Mr. Karl Torbicki, 4th Year (1T0) Medical Student
7
Figure 1. Medical Council of Canada Qualifying Examination - Part I:University of Toronto, 1992-2009 OVERALL FINAL SCORE
500
510
520
530
540
550
560
570
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
University of TorontoCanadian
8
Figure 2. Medical Council of Canada Qualifying Examination - Part I: University of Toronto, 1992-2009 CLINICAL DECISION MAKING
410
430
450
470
490
510
530
550
570
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of TorontoCanadian
9
Figure 3. Medical Council of Canada Qualifying Examination - Part I:University of Toronto, 1992-2009 MULTIPLE-CHOICE QUESTIONS
510
520
530
540
550
560
570
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of TorontoCanadian
10
Figure 4. Medical Council of Canada Qualifying Examination - Part I: University of Toronto, 2000-2009 CLEO
(Considerations of Legal, Ethical and Organizational Principles)
500
510
520
530
540
550
560
570
580
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of TorontoCanadian
11
Figure 5. Medical Council of Canada Qualifying Examination - Part I:University of Toronto, 1992-2009 PREVENTIVE MEDICINE & COMMUNITY HEALTH
470
480
490
500
510
520
530
540
550
560
570
580
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of Toronto
Canadian
12
Figure 6. Medical Council of Canada Qualifying Examination - Part I: University of Toronto, 1992-2009 MEDICINE
490
500
510
520
530
540
550
560
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of Toronto
Canadian
13
Figure 7. Medical Council of Canada Qualifying Examination - Part I: University of Toronto Results, 1992-2009 SURGERY
480
490
500
510
520
530
540
550
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of TorontoCanadian
14
Figure 8. Medical Council of Canada Qualifying Examination - Part I: University of Toronto, 1992-2009 OBSTETRICS & GYNAECOLOGY
480
490
500
510
520
530
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of TorontoCanadian
15
Figure 9. Medical Council of Canada Qualifying Examination - Part I: University of Toronto, 1992-2009 PAEDIATRICS
480
490
500
510
520
530
540
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of Toronto
Canadian
16
Figure 10. Medical Council of Canada Qualifying Examination - Part I: University of Toronto Results, 1992-2009 PSYCHIATRY
490
500
510
520
530
540
550
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of TorontoCanadian
17
Figure 11. Medical Council of Canada Qualifying Examination - Part I: University of Toronto, 2006-2009 FAMILY MEDICINE
515
520
525
530
535
540
545
550
555
560
2006 2007 2008 2009
Graduating Class
Mea
n St
anda
rd S
core
U of Toronto
Canadian
18
Figure 12. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 1993-2009 National Ranking – OVERALL FINAL SCORE
National Ranking: 1
2 3 3 4 4 4 5 5 5 5 5 5 5 5 6 6 6 7 7 7 7 8 8______________________________________________________________ _ 9 9 10 11 12 13 14 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: 6 5 8 5 7 5 5 5 5 9 7 4 5 3 4 6 7
19
Figure 13. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 1993-2009 National Ranking – MULTIPLE CHOICE QUESTIONS
National Ranking: 1
2 2 3 3 4 4_____ ________ 4 5 5 5 6 6 6 6 6 7 7 7 7 8 8_____________________________________________________ ___ 9 9 10 11 11 12 12 _____ 13 14 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: 12 5 9 7 8 6 6 7 7 11 6 4 5 2 3 4 6
20
Figure 14. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 1993-2009 National Ranking – CLINICAL DECISION MAKING
National Ranking: 1 1
2 2 2 3 3 4 4 _ __ 5 5 5 5 5 6 7 7 7 7 8 __________________________________________________________________ __ 9 9 9 10 10 11 12 12___ 13 13 14 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: 2 2 7 1 3 5 5 4 5 9 9 5 7 7 10 13 12
21
Figure 15. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 1993-2009 National Ranking – MEDICINE
National Ranking: 1 1
2 2 3 3 3 3 4 4_ _ _ 5 5 5 5 6 6 6 6 7 7 7 8 8_________________________________________________________________ __ 9 10 11 11 11 12 _____ 13 14 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: 7 3 8 2 5 6 5 6 6 7 5 4 11 3 3 1 11
22
Figure 16. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto Results, 1993-2009 National Ranking – OBSTETRICS & GYNAECOLOGY
National Ranking: 1
2 3 4 _ 4 4 4 5 6 6 6 6 7 8 _______________________________________________________________ _8 9 9 10 11 11 11 11 12 12 12 _______ 13 13 14 14 15 15 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: 11 6 6 9 15 11 12 11 13 15 14 12 4 6 4 8 4
23
Figure 17. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 1993-2009 National Ranking – PAEDIATRICS
National Ranking: 1
2 3 3 4 _ _ 5 5 6 6 7 7 7 8 8 _____ ___________________ 8__________________________________ _ 9 9 9 9 10 10 11 11 12 12 12 12 _______ 13 13 14 14 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: 5 6 8 10 12 12 7 9 8 12 14 9 11 3 9 13 7
24
Figure 18. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 1993-2009 National Ranking – PREVENTIVE MEDICINE / COMMUNITY HEALTH
National Ranking: 1 1 1 1 1 1
2 2 2 2 3 3 3 3 4 4 _ _ _ 5 6 6 7 8 8__ ___________________________ __8____________________________ _ 9 10 10 11 12 _____ 13 13 14 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: 10 13 8 2 4 3 3 3 6 8 1 1 1 1 1 2 2
25
Figure 19. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 1993-2009 National Ranking – PSYCHIATRY
National Ranking: 1
2 3 4 _ _____ 5 6 6 6 7 7 8 ____________________8__________________________________________________ 9 9 9 10 10 11 11 11 11 11 12 12 12 12 12 _______ 13 13 14 15 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: 15 12 11 13 12 12 8 11 11 9 10 9 12 6 11 6 7
26
Figure 20. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 1993-2009 National Ranking – SURGERY
National Ranking: 1
2 3 3 3 3 3 4 _ __ 4 5 5 5 6 6 6 6 6 6 7 7 8 ___________________________________________8_____________________________ 9 9 9 10 11 11 12 _____ 13 14 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: 9 3 11 9 3 3 3 6 5 6 8 6 7 6 5 4 6
27
Figure 21. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 2006-2009 National Ranking – FAMILY MEDICINE
National Ranking: 1
2 2 2 2 3 4 _____ 5 6 7 8 _ 8 9 10 11 12 _____ 13 14 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a 2 2 2 8
28
Figure 22. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 2000-2009 National Ranking – CLEO
National Ranking: 1 1 1 1
2 2 2 2 3 3 4 ______ 4 5 6 6 7 7 8 _____ 9 10 11 12 _____ 13 14 15 16
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 National Rank: n/a n/a n/a n/a n/a n/a n/a 2 7 6 1 1 3 1 2 4 2
29
Figure 23. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 5-Year (2005-2009) Difference from Canadian Mean Score by MCCQE-Part I Component
Population Health [! ! !* !]
CLEO [! ! ! ! !]
Family Medicine† [! ! ! !]
MCQs [! !* ! !]
Surgery [! ! !!!]
Overall [! ! !!*]
Medicine [! !* ! !]
Pediatrics [! !* !!]
Ob/Gyn [!* ! ! !]
Psychiatry [! ! ! ! !]
CDM [! ! ! ! !]
40 35 30 25 20 15 10 5 0 -5 -10 -15 -20 -25 Difference Between Toronto and Canadian Mean Scores (Toronto minus Canadian Mean Score) * multiple years represented † only reported for past 4 years (since 2006)
30
Figure 24. Medical Council of Canada Qualifying Examination – Part I:
University of Toronto, 5-Year (2005-2009) National Ranking Summary by MCCQE-Part I Component
Population Health [!* !*]
CLEO [! !* ! !]
Family Medicine† [!* !]
MCQs [! ! ! ! !]
Overall [! ! ! ! !]
Medicine [!* ! !*]
Pediatrics [! ! ! ! !]
Surgery [! ! !* !]
Ob/Gyn [!* ! !]
Psychiatry [!* ! ! !]
CDM [!* ! ! !]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 University of Toronto National Ranking * multiple years represented † only reported for past 4 years (since 2006)
