MAKING SENSE OF THE ACGME SURVEY

OPDO MEETING 2015

STACEY T. GRAY, MDPROGRAM DIRECTOR,HARVARD MEDICAL SCHOOL

PANELISTS

STEVEN PLETCHER, MD Program Director, UCSF

VYVY YOUNG, MD Associate Program Director, University of

Pittsburgh

IRAM AHMAD, MD PGY-7 Resident, University of Iowa

ACGME SURVEYWhat is it?

RESIDENT SURVEY Questions about the

clinical and educational experience in the training program

Questions about duty hours

FACULTY SURVEY Questions about the

experience working in the training program

Interactions with residents

ACGME SURVEY What is it?

Administered annually for residents, fellows and core faculty

NOTIFICATION Program is responsible

for notifying and reminding trainees and faculty about the survey

TIMING Mid-January thru early

June 5 week window to

respond

ACGME SURVEY What is it?

REQUIRED RESPONSE RATE Resident/Fellow

Survey ▪ 70% completion rate

Faculty Survey ▪ 60% completion rate

SURVEY CATEGORIES

RESIDENT SURVEY

Duty HoursFacultyEvaluation SystemsEducational ContentResourcesPatient Safety/TeamworkOverall Evaluation of Program

FACULTY SURVEY

Supervision and TeachingEducational ContentResourcesPatient SafetyTeamworkOverall Evaluation of Program

SURVEY QUESTIONS

Sample questions are not available Major categories

remain the same Questions change

each year – picked from a question bank

Not all survey takers will see the same questions

THE RESULTS

THE RESULTS

ACGME SURVEYWhy is it important? ANNUAL PROGRAM

REVIEW DATA ENTRY ANNUAL ADS

UPDATE▪ Program Changes▪ Program Attrition▪ Scholarly Activity – faculty and

residents▪ Major changes ▪ Response to citations

CASE LOGS RESIDENT SURVEY FACULTY SURVEY MILESTONES BOARD PASS RATE

HOW ARE THE RESULTS USED?

“The ACGME’s Resident/Fellow and Faculty Surveys are an additional method used to monitor graduate medical clinical education and provide early warning of potential non-compliance with ACGME accreditation standards”

RED FLAGS

RESPONSE RATE Resident survey – 70% Faculty survey – 60%

RED FLAGS

RESIDENT SURVEY Duty hour violations Negative responses compared to

national ORL program results in 3 major categories

Negative responses compared to national ORL program results in 20% of the overall survey

Negative response to final summary question

RED FLAGS

FACULTY SURVEY Negative responses compared to

national ORL program results in 2 major categories

Negative responses compared to national ORL program results in 20% of the overall survey

HOW DO YOU PREPARE YOUR PROGRAM?

COMPILED OTOLARYNGOLOGY DATA2014-2015

AGGREGATE DATA2014-2015

106 programs Resident Response

1442/1506 96% response rate

Faculty Response 1406/1625 87% response rate

AREAS FOR IMPROVEMENT

Focus on areas of less than 95% compliance

Potential areas for improvement National level – how are

we doing as a specialty? Program level –

compare to national ORL standards

RESIDENT SURVEY:Duty Hours

% Program Compliant

% National Compliant

80 hours 96% 94%

1 day free in 7 99% 97%

In-house call every 3rd night

100% 99%

Night float no more than 6 nights

100% 99%

8 hours between duty periods

96% 97%

Continuous hours scheduled

98% 96%

RESIDENT SURVEY:Duty Hours

% Program Compliant

% National Compliant

80 hours 96% 94%

1 day free in 7 99% 97%

In-house call every 3rd night

100% 99%

Night float no more than 6 nights

100% 99%

8 hours between duty periods

96% 97%

Continuous hours scheduled

98% 96%

Duty Hours

Do you review duty hour rules with residents before the survey?

RESIDENT SURVEY:Faculty

% Program Compliant

% National Compliant

Sufficient supervision

98% 92%

Appropriate level of supervision

98% 96%

Sufficient instruction

94% 85%

Faculty and staff interested in residency education

94% 84%

Faculty and staff create environment of inquiry

91% 78%

RESIDENT SURVEY:Faculty

% Program Compliant

% National Compliant

Sufficient supervision

98% 92%

Appropriate level of supervision

98% 96%

Sufficient instruction

94% 85%

Faculty and staff interested in residency education

94% 84%

Faculty and staff create environment of inquiry

91% 78%

RESIDENT SURVEY:Faculty

% Program Compliant

% National Compliant

Sufficient supervision

98% 92%

Appropriate level of supervision

98% 96%

Sufficient instruction

94% 85%

Faculty and staff interested in residency education

94% 84%

Faculty and staff create environment of inquiry

91% 78%

FACULTY SURVEY:Faculty supervision & teaching

% Program Compliant

% National Compliant

Sufficient time to supervise residents/fellows

95% 94%

Residents/fellows seek supervisory guidance

92% 92%

Interest of faculty and PD in education

98% 96%

Rotation and educational assignment evaluation

99% 99%

Faculty performance evaluated 99% 98%

Faculty satisfied with personal performance feedback

89% 86%

FACULTY SURVEY:Faculty supervision & teaching

% Program Compliant

% National Compliant

Sufficient time to supervise residents/fellows

95% 94%

Residents/fellows seek supervisory guidance

92% 92%

Interest of faculty and PD in education

98% 96%

Rotation and educational assignment evaluation

99% 99%

Faculty performance evaluated 99% 98%

Faculty satisfied with personal performance feedback

89% 86%

Faculty Issues

Resident Supervision vs. Independence Different perception

Learning Environment Faculty Development Mentorship Teaching techniques

Faculty Feedback How to provide it?

RESIDENT SURVEY:Evaluation

% Program Compliant

% National Compliant

Able to access evaluations 99% 99%

Opportunity to evaluate faculty members

98% 96%

Satisfied that evaluations of faculty are confidential

92% 84%

Opportunity to evaluate program

99% 98%

Satisfied that evaluations of program are confidential

91% 78%

Satisfied that program uses evaluations to improve

88% 73%

Satisfied with feedback after assignments

84% 70%

RESIDENT SURVEY:Evaluation

% Program Compliant

% National Compliant

Able to access evaluations 99% 99%

Opportunity to evaluate faculty members

98% 96%

Satisfied that evaluations of faculty are confidential

92% 84%

Opportunity to evaluate program

99% 98%

Satisfied that evaluations of program are confidential

91% 78%

Satisfied that program uses evaluations to improve

88% 73%

Satisfied with feedback after assignments

84% 70%

Evaluation issues

Resident Feedback Not just a semi-annual

review In the moment

Confidentiality Reassurance for residents

Evaluations need to be meaningful Issues identified by residents

should be addressed Improvements should be

documented and reviewed with the residents

RESIDENT SURVEY:Educational content

% Program Compliant

% National Compliant

Provided goals and objectives for assignments

96% 95%

Instructed how to manage fatigue 97% 92%

Satisfied with opportunities for scholarly activities

92% 76%

Appropriate balance for education 89% 79%

Education (not) compromised by service

81% 69%

Supervisors delegate appropriately 100% 99%

Provided data about practice habits 73% 68%

See patients across variety of settings

98% 95%

RESIDENT SURVEY:Educational content

% Program Compliant

% National Compliant

Provided goals and objectives for assignments

96% 95%

Instructed how to manage fatigue 97% 92%

Satisfied with opportunities for scholarly activities

92% 76%

Appropriate balance for education 89% 79%

Education (not) compromised by service

81% 69%

Supervisors delegate appropriately 100% 99%

Provided data about practice habits 73% 68%

See patients across variety of settings

98% 95%

FACULTY SURVEY:Educational content

% Program Compliant

% National Compliant

Worked on scholarly project with residents/fellows

86% 76%

Residents/fellows see patients across a variety of settings

99% 98%

Residents/fellows receive education to manage fatigue

99% 99%

Effectiveness of graduating residents/fellows

99% 97%

Outcome achievement of graduating residents/fellows

99% 99%

FACULTY SURVEY:Educational content

% Program Compliant

% National Compliant

Worked on scholarly project with residents/fellows

86% 76%

Residents/fellows see patients across a variety of settings

99% 98%

Residents/fellows receive education to manage fatigue

99% 99%

Effectiveness of graduating residents/fellows

99% 97%

Outcome achievement of graduating residents/fellows

99% 99%

Educational Content Issues Providing data

about practice habits? Evaluations? Patient outcomes?

Educational Content Issues Time for research and

scholarship How protected is the time? How do you expose

residents to all of the potential research projects and interested faculty?

Balance between education and service How protected is the time? How do you evaluate the

quality of didactics? How involved are the

residents in the didactics?

RESIDENT SURVEY:Resources

% Program Compliant

% National Compliant

Access to reference materials 100% 99%

Use EMR in hospital 98% 97%

Use EMR in ambulatory setting 100% 97%

EMR integrated across settings 89% 82%

EMR effective 96% 94%

Provided a way to transition care when fatigued

91% 80%

Satisfied with process to deal with problems and concerns

91% 80%

Education (not) compromised by other trainees

96% 91%

Residents can raise concerns without fear

91% 80%

RESIDENT SURVEY:Resources

% Program Compliant

% National Compliant

Access to reference materials 100% 99%

Use EMR in hospital 98% 97%

Use EMR in ambulatory setting 100% 97%

EMR integrated across settings 89% 82%

EMR effective 96% 94%

Provided a way to transition care when fatigued

91% 80%

Satisfied with process to deal with problems and concerns

91% 80%

Education (not) compromised by other trainees

96% 91%

Residents can raise concerns without fear

91% 80%

FACULTY SURVEY:Resources

% Program Compliant

% National Compliant

Program provides a way for residents/fellows to transition care when fatigued

100% 99%

Residents/fellows workload does not exceed capacity to do the work

98% 99%

Satisfied with faculty development to supervise and educate residents/fellows

96% 95%

Satisfied with process to deal with residents/fellows problems and concerns

95% 93%

Prevent excessive reliance on residents/fellows to provide clinical service

98% 98%

Resource Issues

Fatigue Management Care Transitions

Resident Concerns Confidential

reporting Free of fear of

intimidation or retaliation

RESIDENT SURVEY:Patient safety/teamwork

% Program Compliant

% National Compliant

Tell patients of respective roles of faculty and residents

99% 99%

Culture reinforces patient safety responsibility

100% 99%

Participated in quality improvement 88% 86%

Information not lost during shift changes or patient transfers

98% 97%

Work in interprofessional teams 99% 98%

Effectively work in interprofessional teams

100% 99%

RESIDENT SURVEY:Patient safety/teamwork

% Program Compliant

% National Compliant

Tell patients of respective roles of faculty and residents

99% 99%

Culture reinforces patient safety responsibility

100% 99%

Participated in quality improvement 88% 86%

Information not lost during shift changes or patient transfers

98% 97%

Work in interprofessional teams 99% 98%

Effectively work in interprofessional teams

100% 99%

RESIDENT SURVEY:Patient safety/teamwork

% Program Compliant

% National Compliant

Tell patients of respective roles of faculty and residents

99% 99%

Culture reinforces patient safety responsibility

100% 99%

Participated in quality improvement 88% 86%

Information not lost during shift changes or patient transfers

98% 97%

Work in interprofessional teams 99% 98%

Effectively work in interprofessional teams

100% 99%

FACULTY SURVEY:Patient safety

% Program Compliant

% National Compliant

Information not lost during shift changes or patient transfers

93% 89%

Tell patients of respective roles of faculty and residents/fellows

88% 90%

Culture reinforces patient safety responsibility

97% 96%

Residents/fellows participate in quality improvement or patient safety activities

92% 90%

FACULTY SURVEY:Patient safety

% Program Compliant

% National Compliant

Information not lost during shift changes or patient transfers

93% 89%

Tell patients of respective roles of faculty and residents/fellows

88% 90%

Culture reinforces patient safety responsibility

97% 96%

Residents/fellows participate in quality improvement or patient safety activities

92% 90%

f

FACULTY SURVEY:Teamwork

% Program Compliant

% National Compliant

Residents/fellows communicate effectively when transferring clinical care

99% 98%

Residents/fellows effectively work in interprofessional teams

100% 99%

Program effective in teaching teamwork skills

100% 99%

Patient safety/teamwork

Patient Care Transitions

Quality Improvement

Interprofessional teams

OVERALL EXPERIENCEResident perspective

Which of the following best summarizes your opinion of your residency program?

Very positive

Positive

Neutral

Negative

Very negative

0 10 20 30 40 50 60 70 80

OVERALL EXPERIENCEFaculty perspective

Which of the following best summarizes your opinion of your residency program?

Very positive

Positive

Neutral

Negative

Very negative

0 10 20 30 40 50 60 70 80 90 100

AS A SPECIALTY

Residents and Faculty generally have a positive view of our training programs

High compliance with most major categories

New questions highlight ACGME concerns with training

FINAL THOUGHTS

PREPARE YOUR PROGRAM

Remind the residents AND faculty before the survey Duty hours Confidentiality of your evaluation system Availability/accessibility of Goals & Objectives Resources & systems for communication/support▪ Faculty mentors▪ Chief or senior residents▪ Institutional

Sleep deprivation education Quality Improvement

USE YOUR RESULTS

Assess what the program is doing well

Assess opportunities for improvement

Topics for national discussions on training

Gain insight into what the ACGME is looking for in training programs