MAKING SENSE OF THE ACGME SURVEY
OPDO MEETING 2015
STACEY T. GRAY, MDPROGRAM DIRECTOR,HARVARD MEDICAL SCHOOL
PANELISTS
STEVEN PLETCHER, MD Program Director, UCSF
VYVY YOUNG, MD Associate Program Director, University of
Pittsburgh
IRAM AHMAD, MD PGY-7 Resident, University of Iowa
ACGME SURVEYWhat is it?
RESIDENT SURVEY Questions about the
clinical and educational experience in the training program
Questions about duty hours
FACULTY SURVEY Questions about the
experience working in the training program
Interactions with residents
ACGME SURVEY What is it?
Administered annually for residents, fellows and core faculty
NOTIFICATION Program is responsible
for notifying and reminding trainees and faculty about the survey
TIMING Mid-January thru early
June 5 week window to
respond
ACGME SURVEY What is it?
REQUIRED RESPONSE RATE Resident/Fellow
Survey ▪ 70% completion rate
Faculty Survey ▪ 60% completion rate
SURVEY CATEGORIES
RESIDENT SURVEY
Duty HoursFacultyEvaluation SystemsEducational ContentResourcesPatient Safety/TeamworkOverall Evaluation of Program
FACULTY SURVEY
Supervision and TeachingEducational ContentResourcesPatient SafetyTeamworkOverall Evaluation of Program
SURVEY QUESTIONS
Sample questions are not available Major categories
remain the same Questions change
each year – picked from a question bank
Not all survey takers will see the same questions
THE RESULTS
THE RESULTS
ACGME SURVEYWhy is it important? ANNUAL PROGRAM
REVIEW DATA ENTRY ANNUAL ADS
UPDATE▪ Program Changes▪ Program Attrition▪ Scholarly Activity – faculty and
residents▪ Major changes ▪ Response to citations
CASE LOGS RESIDENT SURVEY FACULTY SURVEY MILESTONES BOARD PASS RATE
HOW ARE THE RESULTS USED?
“The ACGME’s Resident/Fellow and Faculty Surveys are an additional method used to monitor graduate medical clinical education and provide early warning of potential non-compliance with ACGME accreditation standards”
RED FLAGS
RESPONSE RATE Resident survey – 70% Faculty survey – 60%
RED FLAGS
RESIDENT SURVEY Duty hour violations Negative responses compared to
national ORL program results in 3 major categories
Negative responses compared to national ORL program results in 20% of the overall survey
Negative response to final summary question
RED FLAGS
FACULTY SURVEY Negative responses compared to
national ORL program results in 2 major categories
Negative responses compared to national ORL program results in 20% of the overall survey
HOW DO YOU PREPARE YOUR PROGRAM?
COMPILED OTOLARYNGOLOGY DATA2014-2015
AGGREGATE DATA2014-2015
106 programs Resident Response
1442/1506 96% response rate
Faculty Response 1406/1625 87% response rate
AREAS FOR IMPROVEMENT
Focus on areas of less than 95% compliance
Potential areas for improvement National level – how are
we doing as a specialty? Program level –
compare to national ORL standards
RESIDENT SURVEY:Duty Hours
% Program Compliant
% National Compliant
80 hours 96% 94%
1 day free in 7 99% 97%
In-house call every 3rd night
100% 99%
Night float no more than 6 nights
100% 99%
8 hours between duty periods
96% 97%
Continuous hours scheduled
98% 96%
RESIDENT SURVEY:Duty Hours
% Program Compliant
% National Compliant
80 hours 96% 94%
1 day free in 7 99% 97%
In-house call every 3rd night
100% 99%
Night float no more than 6 nights
100% 99%
8 hours between duty periods
96% 97%
Continuous hours scheduled
98% 96%
Duty Hours
Do you review duty hour rules with residents before the survey?
RESIDENT SURVEY:Faculty
% Program Compliant
% National Compliant
Sufficient supervision
98% 92%
Appropriate level of supervision
98% 96%
Sufficient instruction
94% 85%
Faculty and staff interested in residency education
94% 84%
Faculty and staff create environment of inquiry
91% 78%
RESIDENT SURVEY:Faculty
% Program Compliant
% National Compliant
Sufficient supervision
98% 92%
Appropriate level of supervision
98% 96%
Sufficient instruction
94% 85%
Faculty and staff interested in residency education
94% 84%
Faculty and staff create environment of inquiry
91% 78%
RESIDENT SURVEY:Faculty
% Program Compliant
% National Compliant
Sufficient supervision
98% 92%
Appropriate level of supervision
98% 96%
Sufficient instruction
94% 85%
Faculty and staff interested in residency education
94% 84%
Faculty and staff create environment of inquiry
91% 78%
FACULTY SURVEY:Faculty supervision & teaching
% Program Compliant
% National Compliant
Sufficient time to supervise residents/fellows
95% 94%
Residents/fellows seek supervisory guidance
92% 92%
Interest of faculty and PD in education
98% 96%
Rotation and educational assignment evaluation
99% 99%
Faculty performance evaluated 99% 98%
Faculty satisfied with personal performance feedback
89% 86%
FACULTY SURVEY:Faculty supervision & teaching
% Program Compliant
% National Compliant
Sufficient time to supervise residents/fellows
95% 94%
Residents/fellows seek supervisory guidance
92% 92%
Interest of faculty and PD in education
98% 96%
Rotation and educational assignment evaluation
99% 99%
Faculty performance evaluated 99% 98%
Faculty satisfied with personal performance feedback
89% 86%
Faculty Issues
Resident Supervision vs. Independence Different perception
Learning Environment Faculty Development Mentorship Teaching techniques
Faculty Feedback How to provide it?
RESIDENT SURVEY:Evaluation
% Program Compliant
% National Compliant
Able to access evaluations 99% 99%
Opportunity to evaluate faculty members
98% 96%
Satisfied that evaluations of faculty are confidential
92% 84%
Opportunity to evaluate program
99% 98%
Satisfied that evaluations of program are confidential
91% 78%
Satisfied that program uses evaluations to improve
88% 73%
Satisfied with feedback after assignments
84% 70%
RESIDENT SURVEY:Evaluation
% Program Compliant
% National Compliant
Able to access evaluations 99% 99%
Opportunity to evaluate faculty members
98% 96%
Satisfied that evaluations of faculty are confidential
92% 84%
Opportunity to evaluate program
99% 98%
Satisfied that evaluations of program are confidential
91% 78%
Satisfied that program uses evaluations to improve
88% 73%
Satisfied with feedback after assignments
84% 70%
Evaluation issues
Resident Feedback Not just a semi-annual
review In the moment
Confidentiality Reassurance for residents
Evaluations need to be meaningful Issues identified by residents
should be addressed Improvements should be
documented and reviewed with the residents
RESIDENT SURVEY:Educational content
% Program Compliant
% National Compliant
Provided goals and objectives for assignments
96% 95%
Instructed how to manage fatigue 97% 92%
Satisfied with opportunities for scholarly activities
92% 76%
Appropriate balance for education 89% 79%
Education (not) compromised by service
81% 69%
Supervisors delegate appropriately 100% 99%
Provided data about practice habits 73% 68%
See patients across variety of settings
98% 95%
RESIDENT SURVEY:Educational content
% Program Compliant
% National Compliant
Provided goals and objectives for assignments
96% 95%
Instructed how to manage fatigue 97% 92%
Satisfied with opportunities for scholarly activities
92% 76%
Appropriate balance for education 89% 79%
Education (not) compromised by service
81% 69%
Supervisors delegate appropriately 100% 99%
Provided data about practice habits 73% 68%
See patients across variety of settings
98% 95%
FACULTY SURVEY:Educational content
% Program Compliant
% National Compliant
Worked on scholarly project with residents/fellows
86% 76%
Residents/fellows see patients across a variety of settings
99% 98%
Residents/fellows receive education to manage fatigue
99% 99%
Effectiveness of graduating residents/fellows
99% 97%
Outcome achievement of graduating residents/fellows
99% 99%
FACULTY SURVEY:Educational content
% Program Compliant
% National Compliant
Worked on scholarly project with residents/fellows
86% 76%
Residents/fellows see patients across a variety of settings
99% 98%
Residents/fellows receive education to manage fatigue
99% 99%
Effectiveness of graduating residents/fellows
99% 97%
Outcome achievement of graduating residents/fellows
99% 99%
Educational Content Issues Providing data
about practice habits? Evaluations? Patient outcomes?
Educational Content Issues Time for research and
scholarship How protected is the time? How do you expose
residents to all of the potential research projects and interested faculty?
Balance between education and service How protected is the time? How do you evaluate the
quality of didactics? How involved are the
residents in the didactics?
RESIDENT SURVEY:Resources
% Program Compliant
% National Compliant
Access to reference materials 100% 99%
Use EMR in hospital 98% 97%
Use EMR in ambulatory setting 100% 97%
EMR integrated across settings 89% 82%
EMR effective 96% 94%
Provided a way to transition care when fatigued
91% 80%
Satisfied with process to deal with problems and concerns
91% 80%
Education (not) compromised by other trainees
96% 91%
Residents can raise concerns without fear
91% 80%
RESIDENT SURVEY:Resources
% Program Compliant
% National Compliant
Access to reference materials 100% 99%
Use EMR in hospital 98% 97%
Use EMR in ambulatory setting 100% 97%
EMR integrated across settings 89% 82%
EMR effective 96% 94%
Provided a way to transition care when fatigued
91% 80%
Satisfied with process to deal with problems and concerns
91% 80%
Education (not) compromised by other trainees
96% 91%
Residents can raise concerns without fear
91% 80%
FACULTY SURVEY:Resources
% Program Compliant
% National Compliant
Program provides a way for residents/fellows to transition care when fatigued
100% 99%
Residents/fellows workload does not exceed capacity to do the work
98% 99%
Satisfied with faculty development to supervise and educate residents/fellows
96% 95%
Satisfied with process to deal with residents/fellows problems and concerns
95% 93%
Prevent excessive reliance on residents/fellows to provide clinical service
98% 98%
Resource Issues
Fatigue Management Care Transitions
Resident Concerns Confidential
reporting Free of fear of
intimidation or retaliation
RESIDENT SURVEY:Patient safety/teamwork
% Program Compliant
% National Compliant
Tell patients of respective roles of faculty and residents
99% 99%
Culture reinforces patient safety responsibility
100% 99%
Participated in quality improvement 88% 86%
Information not lost during shift changes or patient transfers
98% 97%
Work in interprofessional teams 99% 98%
Effectively work in interprofessional teams
100% 99%
RESIDENT SURVEY:Patient safety/teamwork
% Program Compliant
% National Compliant
Tell patients of respective roles of faculty and residents
99% 99%
Culture reinforces patient safety responsibility
100% 99%
Participated in quality improvement 88% 86%
Information not lost during shift changes or patient transfers
98% 97%
Work in interprofessional teams 99% 98%
Effectively work in interprofessional teams
100% 99%
RESIDENT SURVEY:Patient safety/teamwork
% Program Compliant
% National Compliant
Tell patients of respective roles of faculty and residents
99% 99%
Culture reinforces patient safety responsibility
100% 99%
Participated in quality improvement 88% 86%
Information not lost during shift changes or patient transfers
98% 97%
Work in interprofessional teams 99% 98%
Effectively work in interprofessional teams
100% 99%
FACULTY SURVEY:Patient safety
% Program Compliant
% National Compliant
Information not lost during shift changes or patient transfers
93% 89%
Tell patients of respective roles of faculty and residents/fellows
88% 90%
Culture reinforces patient safety responsibility
97% 96%
Residents/fellows participate in quality improvement or patient safety activities
92% 90%
FACULTY SURVEY:Patient safety
% Program Compliant
% National Compliant
Information not lost during shift changes or patient transfers
93% 89%
Tell patients of respective roles of faculty and residents/fellows
88% 90%
Culture reinforces patient safety responsibility
97% 96%
Residents/fellows participate in quality improvement or patient safety activities
92% 90%
f
FACULTY SURVEY:Teamwork
% Program Compliant
% National Compliant
Residents/fellows communicate effectively when transferring clinical care
99% 98%
Residents/fellows effectively work in interprofessional teams
100% 99%
Program effective in teaching teamwork skills
100% 99%
Patient safety/teamwork
Patient Care Transitions
Quality Improvement
Interprofessional teams
OVERALL EXPERIENCEResident perspective
Which of the following best summarizes your opinion of your residency program?
Very positive
Positive
Neutral
Negative
Very negative
0 10 20 30 40 50 60 70 80
OVERALL EXPERIENCEFaculty perspective
Which of the following best summarizes your opinion of your residency program?
Very positive
Positive
Neutral
Negative
Very negative
0 10 20 30 40 50 60 70 80 90 100
AS A SPECIALTY
Residents and Faculty generally have a positive view of our training programs
High compliance with most major categories
New questions highlight ACGME concerns with training
FINAL THOUGHTS
PREPARE YOUR PROGRAM
Remind the residents AND faculty before the survey Duty hours Confidentiality of your evaluation system Availability/accessibility of Goals & Objectives Resources & systems for communication/support▪ Faculty mentors▪ Chief or senior residents▪ Institutional
Sleep deprivation education Quality Improvement
USE YOUR RESULTS
Assess what the program is doing well
Assess opportunities for improvement
Topics for national discussions on training
Gain insight into what the ACGME is looking for in training programs