William Carey UniversityWilliam Carey UniversityFaculty DevelopmentFaculty Development
Staying ahead in an era of Staying ahead in an era of assessmentassessment
Robert G. Cuzzolino, Ed.D.Robert G. Cuzzolino, Ed.D.Vice President for Graduate Programs and Planning, Vice President for Graduate Programs and Planning,
PCOMPCOM
Osteopathic Medical Education in an environment of planning and assessment
Objectives
Be able to distinguish among mission, goals and outcomes
Describe how a continuous quality improvement program works.
Organize goals or competencies with related outcomes data in an understandable way
Describe the conditions necessary for strategic planning to yield organizational or instructional change.
Objectives, continued
Describe methods of longitudinal analysis
Differentiate between the measurement of student achievement and the assessment of institutional or program effectiveness
Mission, Goals, Competencies, Outcomes External reviewers focus on mission as the
foundation for institutional assessment and planning.
Mission statements are just a starting point in the assessment/planning process chain.
Mission
Goals
Competencies
Student Outcomes
Program Assessment
Planning
Mission demands goals Goals demand learning objectives Objectives demand measurement Measurement demands analysis Analysis demands planning Planning demands change Change demands review
in context of mission
Be kind to your outside reviewers. The linkages among institutional goals,
program competencies and outcomes are not intuitive to most visitors, so why not map them out?
Institutional Learning Goal
Academic Program Competency Group
Outcomes Data
II. Communication The student acquires interpersonal and communications skills that result in effective information exchange and teaming with patients, their families, and other health professionals.
Postgraduate assessment
(% above comparative low quartile)
Patient Instruction Skills: 98% (x=3.22/4)
Standardized Patient Communications Assessment (end of year)
[1-9 point assessment scale:
1= poor, low level/ 9= superior skill]
1st Year Standardized Patient
Communications Assessment (end of year)
Rapport 6.8
Empathy 7.2
Confidence 5.3
Info 7.0
Listen 6.6
Feedback 6.1
PE 7.3
2nd Year Standardized Patient
Communications Assessment (end of year)
Rapport 7.4
Empathy 7.3
Confidence 7.9
Info 8.0
Listen 7.4
Feedback 7.3
PE 8.1
Seniors rating of instruction in
physician/patient relationship:
89% appropriate
3.7% inadequate
7.2% excessive
A look at a hypothetical WCUCOM GCO Grid
On Change 1.3. “The COM must connect its strategic
plan to learning outcomes assessment in order to continuously improve the educational quality of its osteopathic medical education program.”
In other words, “How does my COM make changes?”
“Have the changes we’ve made been based on outcomes data?”
Assessment Data / SWOT Analysis /Departmental Plans /Community Input /
Committee Actions/ Accreditor Recommendations/ Budget Requests
Planning
Resource Allocation Responsibility for Change
Progress Reporting
Annual Plan Renewal
Outcomes, outcomes and more outcomes
“Constant testing will no more address the problems with our educational system than constantly putting an overweight person on the scale will cure obesity.”
-Anna Quindlen
Agreeing on the terminology There is a difference between the
measurement of individual student achievement and the assessment of institutional or program effectiveness.
Unfortunately, accreditation standards often refer to both of these as“learning assessment” or “outcomes.”
COCA Standard 6.6:
“The COM must define, publish and implement educational outcomes, based on its own educational objectives that will prepare students for osteopathic graduate medical education.”
Your COM can not measure learning outcomes if it has not articulated its learning objectives.
Standard 6.5 Documentation guideline for proving that your COM has stated learning objectives and assessment methods:
“Course objectives and course syllabi, evaluation process for all courses, and outcomes from the evaluation process.”
Standard 6.6.1“The COM must establish clinical competencies and a methodology to ensure they are being met.”
Learning Assessment Plans Most institutions have or are developing
learning assessment plans.
These plans tend to focus on the individual student level and define the expected competencies, assessment methods, timing and (sometimes) explain how these outcomes are reported back to the college.
Learning Assessment Plans are valuable at the individual student assessment level.
They document the measurement of “learning” up front for students, and also compel faculty to clearly define objectives and develop measures of student achievement - - as a system.
While they are a good basis for developing objectives and corresponding measures of learning, assessment plans do not produce program effectiveness data. They illustrate how you evaluate students… not the program.
Question for the group
What have you learned from Comlex I results?
200 Comlex scores mean nothing in and of themselves.
But if the lowest 50 scores are your PBL track students, you have a program effectiveness issue
When does a set of learning assessment data help in planning? When it is aggregated When it is augmented by other data When it illustrates a trend When it is benchmarked or comparative When it suggests a course of action, a
change, or the need for a resource.
What learning outcomes data sources can help inform institutional change?
Comlex scores are a mandated element.
Others?
Student clerkship ratings Course grade trends Standardized patient encounters Student survey data End of clerkship exams Alumni surveys AOA, state board data Freshman surveys & Graduation Surveys Match data and residency selection data Certification
Council for Higher Education Accreditation: “Effectiveness is a broad concept that refers
to the overall attainment of the mission and goals of a particular institution or program. As such, it may embrace various kinds of behavioral outcomes for students that go beyond student learning, such as employment, economic mobility, and contributions to personal and civic life…”
Summary on outcomes data In the self study, we have two needs for
academic performance data:
First – Student Assessment WCUOM must show that individual
student achievement is regularly measured against the expected competencies in a planned manner. It is also important that we indicate when and how these expectations and their assessments are communicated.
Second – Program Evaluation
The self study must coalesce individual student achievement results into meaningful program assessment data sets, and then show that the COM analyzed the outcomes data and took the appropriate action. Link curricular, instructional, organizational, and resource changes to data.
Example of an institutional effectiveness process – COMLEX and DO student
success This brief presentation will share information that
was developed by the PCOM learning resource committee and shared with the DO Curriculum Committee and the DO Admissions Committee.
It departs from analysis of grades, clinical observation ratings and mean Comlex scores and treats retrospective data to inferential analysis, with conclusions as to meaning,
but leaves the action plan to the faculty.
Tracking Change
This is most difficult aspect of continuous quality improvement through strategic planning.
Longitudinal data
What do you (will you) know about practice characteristics (geographic distribution, specialty, residency match, certifications) of your graduates ?
Think ahead – define your questions before you graduate a class.
Please rate the graduate, «FIRST_NAME» «LAST_NAME», on each of the following items by circling the appropriate number. In making the ratings please compare the intern with all interns you have supervised, not just those in your recent group:
Top Upper Lower Bottom Unable to Quarter Middle Middle Quarter Judge Quarter Quarter
Osteopathic Principles and Practice, which are established and incorporated in the development of skills. ____ ____ ____ ____ ____
I17
4.03.02.01.0
Competence in Osteopathic Principles and Practice40
30
20
10
0
Std. Dev = .69
Mean = 3.2
N = 65.00
Question 17: Diagnostic SkillValue of Diagnostic Skill
129 5.1 5.1 5.1
14 .6 .6 5.7
2378 94.3 94.3 100.0
2521 100.0 100.0
"blank"
"weak"
"strong"
Total
ValidFrequency Percent Valid Percent
CumulativePercent
Preparation of Diagnostic Skill
153 6.1 6.1 6.1
71 2.8 2.8 8.9
2297 91.1 91.1 100.0
2521 100.0 100.0
"blank"
"weak"
"strong"
Total
ValidFrequency Percent Valid Percent
CumulativePercent
0
10
20
30
40
50
60
70
80
90
100
Blank Weak Strong
General Rating
Per
cent
Value of DiagnosticSkill
Preparation ofDiagnostic Skill
Question 13: Understanding the Payor/ Reimbursement System
Value of Understanding the Payor/Reimbursement System
141 5.6 5.6 5.6
375 14.9 14.9 20.5
2005 79.5 79.5 100.0
2521 100.0 100.0
"blank"
"weak"
"strong"
Total
ValidFrequency Percent Valid Percent
CumulativePercent
Preparation of Understanding the Payor/Reimbursement Syaytem
162 6.4 6.4 6.4
1890 75.0 75.0 81.4
469 18.6 18.6 100.0
2521 100.0 100.0
"blank"
"weak"
"strong"
Total
ValidFrequency Percent Valid Percent
CumulativePercent
0
10
20
30
40
50
60
70
80
90
Blank Weak Strong
General Rating
Per
cent
Value Understanding theSystem
Preparation ofUnderstanding theSystem
Strategic Plan
Goal #2: Enhance academic quality, faculty development, scholarly activity and clinical service.
k) Pursue state approval of interdisciplinary PhD research degree or pursue partnership with PhD-granting institution (see Goal 5 a)
l) DO curriculum committee to assess the adequacy of business in medicine content and encourage co-curricular presentations on business, management and medical economics.
m) Convert geriatric lecture content to on-line format; reformat course to focus on interactive discussion
Summary on assessmentDemonstrating the connection of learning
outcomes to strategic planning is important.
Student assessment and program assessment are not the same thing, but both are important to the self study, and both run on their own respective form of learning outcomes data.
The COCA standards carry an expectation of longitudinal data.
A look at the PCOM strategic plan
