Southeastern Association
for Community College Research
2008 Conference
Beyond Compliance…Using a Model of Comprehensive Academic Program Review
to Improve Education
August 2008
Southeastern Association for Community College Research
2
Comprehensive Academic Program Review
Department of Institutional research and Effectiveness
St. Petersburg College P.O. Box 13489,
St. Petersburg, FL 33733 (727) 341-3059
FAX (727) 341-5411
Presenters
•James Coraggio, Director of Institutional Research and Effectiveness
•Maggie Tymms, Assessment Coordinator for Academic Programs
August 2008
Southeastern Association for Community College Research
3
SPC Background SPC, established in 1927, is the
oldest 2-year college in Florida First Community College in Florida to
offer 4 year degrees (2002) 9 Campuses throughout the county 2007-08 FTE: 16,086 (LD), 1,338
(UD) Opening Spring 2008 headcount:
25,754 Annual 2007-08 enrollment: 62,200
August 2008
Southeastern Association for Community College Research
4
Compliance Issues with IE SACS’s Peer Review Research
Project revealed that in the area of Institutional Effectiveness (IE) 62% of institutions in off-site reviews
were deemed non-compliant and 27% of institutions in on-site visits
were deemed non-compliant IE was the second most identified
area behind faculty qualifications
August 2008
Southeastern Association for Community College Research
5
SPC Experience with IE As part of the recent SACS
reaccreditation process, SPC spent a considerable amount of time and effort documenting and detailing their IE processes
The result…SPC had “zero” compliance issues in the area of IE during the off-site reviews
August 2008
Southeastern Association for Community College Research
6
SACS Requirements for IE Core Requirement 2.5: Integrated
Planning & Evaluation Processes “…ongoing, integrated, and institution-wide
research-based planning and evaluation processes…”
Comprehensive Standard 3.3.1: Institutional Effectiveness “…identifies expected outcomes…; assesses
whether it achieves these outcomes; and provides evidence of improvement…
Comprehensive Standard 3.5.1: General Education Core “… identifies college-level competencies
within the general education core and provides evidence that graduates have attained those competencies.”
August 2008
Southeastern Association for Community College Research
7
SACS Requirements for IE Additional IE related requirements
include: Comprehensive Standard 3.1.1:
Institutional Mission Federal Requirement 4.1: Evaluating
Success Federal Requirement 4.2: Curriculum
August 2008
Southeastern Association for Community College Research
8
IE Evidence SACS Suggested Documentation for CR 2.5
Evidence of linkage of IE to institutional mission
Institutional plans and budgets that demonstrate linkage of assessment findings to planning at all levels
Minutes of appropriate (IE related) unit, committee, task force meetings…
Documentation that relates to IE, such as budget preparation instructions, minutes of budget presentation meetings, annual reports, annual assessment updates, IE reports
Samples of specific actions taken to improve the IE process and/or results from that process
August 2008
Southeastern Association for Community College Research
9
Performance Improvement From Compliance to Performance
Improvement
Data Information ImprovementKnowledge
Improved OutcomesSPCReportsAssessment Data
CommunityCollege LeadershipProgram Directors
FacultyStudents
August 2008
Southeastern Association for Community College Research
10
IE at SPC
“Institutional Effectiveness is the integrated, systematic, explicit, and documented processes of measuring performance against the SPC mission for purposes of continuous improvement of academic programs, administrative services, and educational services offered by the college.”
Operating ResultsBudget PlanningStrategic & Unit
PlanningEvaluations/ AssessmentsMission
“Closing the Loop”
August 2008
Southeastern Association for Community College Research
11
Evaluation & Assessment Evaluation and Assessment
are about ‘Choices, Direction, Focus…Quality Improvement’
• Instructional Effectiveness Processes at SPC were recently evaluated as part of the College’s on-going continuous performance improvement
August 2008
Southeastern Association for Community College Research
12
Educational Assessment• One IE process that experienced
extensive revisions was educational assessment
• Individual assessments in the areas of academic programs, Gen Ed, and program review were evaluated and enhancements were made to the quality and integration of assessments as well as the dissemination of assessments results
August 2008
Southeastern Association for Community College Research
13
Changes & Improvements Evaluated individual assessment
instruments and current assessment processes
Found new ways to integrate assessment with the educational process Focus on fewer, yet meaningful improvements (action items) Use multiple assessment methods
(direct & indirect) Involve faculty in process Integrate timing of assessments (3-year cycle)
August 2008
Southeastern Association for Community College Research
14
Three-year Assessment Cycle
Academic Program Viability Reviews
(APVRs) conducted yearly
APARs
CAPRs
August 2008
Southeastern Association for Community College Research
15
Changes and Improvements Now looking at the ‘Big Picture’ Assessment focus is on CHANGE
through Quality Improvement Faculty-driven assessment
process; IRE serves in consultant capacity
Focus on critical major learning outcomes (MLOs) with fewer, more focused action plan items
Improve access and awareness of assessment information
August 2008
Southeastern Association for Community College Research
16
SPC Assessments
1. On-going
2. On-going
3. Annual
1. Environmental Scans
2. Trends Analysis
3. WorkNet Surveys & Exit Interviews
Economic & Workforce Development
AS Degrees & PSAV Programs
WorkNet One-Stop Ops.
1. Annual
2. Annual
3. Annual
4. Annual
5. Annual
6. Annual
7. Annual
8. Annual
9. Annual
10. Annual
11. Annual
1. Entering Students Survey
2. Enrolled Students Survey
3. Graduating Students Survey
4. Employer Survey
5. Recent Alumni Survey
6. Budget. Planning, IR Survey
7. Technology Resources Allocation Survey
8. Other Department-specific Surveys
9. Administrative Services Oversight (ASOG) Reports
10. Student Services Oversight (SSO) Report
11. Education Support Services (ESS) Reports (Libraries, Learning Support Centers)
Partnerships / Services Student Services
Community Services
Other Institutions
1. Annual
2. Every 3-years / program
3. Annual
4. Every 3-years / program
5. Annual
1. Education Oversight Group (EOG) Reports
2. Academic Program Assessment Reports (APARs)
3. Academic Program Viability Reviews (APVRs)
4. Comprehensive Academic Program Reviews (CAPRs)
5. End-of-year SD-IOs Assessment Report
Accessible Education BS, BAS, AA, AS Degrees
Technical Certificates
Applied Tech. Diplomas
Distance Education
Developmental Education
Continuing Education
Recurring CycleEvaluations / AssessmentsSPC Mission Components
1. On-going
2. On-going
3. Annual
1. Environmental Scans
2. Trends Analysis
3. WorkNet Surveys & Exit Interviews
Economic & Workforce Development
AS Degrees & PSAV Programs
WorkNet One-Stop Ops.
1. Annual
2. Annual
3. Annual
4. Annual
5. Annual
6. Annual
7. Annual
8. Annual
9. Annual
10. Annual
11. Annual
1. Entering Students Survey
2. Enrolled Students Survey
3. Graduating Students Survey
4. Employer Survey
5. Recent Alumni Survey
6. Budget. Planning, IR Survey
7. Technology Resources Allocation Survey
8. Other Department-specific Surveys
9. Administrative Services Oversight (ASOG) Reports
10. Student Services Oversight (SSO) Report
11. Education Support Services (ESS) Reports (Libraries, Learning Support Centers)
Partnerships / Services Student Services
Community Services
Other Institutions
1. Annual
2. Every 3-years / program
3. Annual
4. Every 3-years / program
5. Annual
1. Education Oversight Group (EOG) Reports
2. Academic Program Assessment Reports (APARs)
3. Academic Program Viability Reviews (APVRs)
4. Comprehensive Academic Program Reviews (CAPRs)
5. End-of-year SD-IOs Assessment Report
Accessible Education BS, BAS, AA, AS Degrees
Technical Certificates
Applied Tech. Diplomas
Distance Education
Developmental Education
Continuing Education
Recurring CycleEvaluations / AssessmentsSPC Mission Components
SPC Mission – Assessments - Closing the Loop
August 2008
Southeastern Association for Community College Research
17
Program Review Process
• New enhanced process is a summative evaluation that includes two reports with multiple measures such as:– program-specific performance
measures– profitability measures– economic data
August 2008
Southeastern Association for Community College Research
18
Academic Program Viability Review
Published yearly Included measures:
Program Graduates Course Enrollment Student Semester Hour
(SSH) productivity Relative Profitability Index
(RPI-T) Economic Trend Data
August 2008
Southeastern Association for Community College Research
19
CAPR Objectives• Comprehensive Academic Program
Review (CAPR) developed to meet three objectives within the academic assessment process:– To provide a comprehensive report that
summarizes all elements of the program’s viability and productivity from a 360-degree perspective
– To provide comprehensive and relevant program-specific information to key College stakeholders, such as the President’s Cabinet members, in order to make critical decisions regarding the continued sustainability of a program
– To provide program leadership a vehicle to support and document actionable change for the purposes of performance improvement
August 2008
Southeastern Association for Community College Research
20
Previous Program Review Model• Traditionally, program reviews at
SPC consisted primarily of a community focus group and a few occupational growth measures
• This information was presented to the President’s Cabinet for evaluation
• CAPR was designed to be more representative of a program’s quality and, as such, contains measures involving a number of stakeholder perspectives
August 2008
Southeastern Association for Community College Research
21
Elements of the CAPR• Specific CAPR measures include the
– program description with recent program accreditation information
– program performance measures including enrollment, productivity, grade distributions, and fulltime/adjunct faculty ratios
– program profitability measures– academic outcomes from recent end-of-
program assessments– stakeholder perceptions including student
surveys of instruction results, advisory committee minutes, and employer and recent alumni survey results
– occupation trends and information– state graduate outcomes information– the program director’s perspective of
program issues, trends, and recent success
August 2008
Southeastern Association for Community College Research
22
Measures CAPR
• In order to ensure that the information presented in the CAPR is used appropriately, each measure includes: – a standardized definition of the
performance measure– a reference to the source
information used in the calculation
August 2008
Southeastern Association for Community College Research
23
Program Performance Example
Figure 4: Program GraduatesSource: SPC Factbook, Table 31
Program Graduates
Program Graduates
6
911
14
17
14
0
2
4
6
8
10
12
14
16
18
20
2001-02 2002-03 2003-04 2004-05 2005-06 2006-07
Gra
du
ate
s
August 2008
Southeastern Association for Community College Research
24
Program Profitability
Example
Figure 8: Fiscal SummarySource: PeopleSoft Financial Production database, report ID: ORGBUDSI
Relative Profitability Index (RPI-T) is calculated by dividing a program’s income by the sum of its personnel costs and current expenses.
Fiscal Summary
0.430.31
0.38
0.61
0.40
0.000.100.200.300.400.50
0.600.700.800.901.00
2002-03 2003-04 2004-05 2005-06 2006-07
RP
I-T
August 2008
Southeastern Association for Community College Research
25
Academic Outcomes Example
Table 3 Summary of 2004-2005 Assessment Findings
Evaluation Area Summary of Assessment Findings - Portfolio Evaluations
Number of Students
Sufficient Documentation
(Mean)
Acceptable to Florida Courts &
Attorneys (Mean)
Timely (Mean)
2004-2005 Session I 9
MLO 1 9 3.83 3.85 3.89
MLO 2 9 3.59 3.78 3.78
MLO 3 9 3.70 3.78 3.81
MLO 4 9 3.78 3.81 3.81
MLO 5 9 3.70 3.81 3.81
MLO 6 9 3.52 3.81 3.70
MLO 7 9 3.59 3.81 3.81
MLO 8 9 3.81 3.85 3.85
Paralegal program was evaluated through an Academic Program Assessment Report (APAR) in 2004-2005. Each of the program’s eight major learning outcomes (MLOs) was evaluated.
August 2008
Southeastern Association for Community College Research
26
Stakeholder Perceptions Example
Figure 9: SSI Lecture CoursesSource: PeopleSoft Query S_SSI_CHRT_QRY_CAMPUS
Purpose of the SSI survey is to acquire information on student perception of the quality of courses, faculty, and instruction, and to provide feedback information for improvement. SSI: Lecture Courses
6.76.9 6.8 6.8
6.76.8 6.9 6.8
6.7
6.9 6.9 6.8
6.6
6.8 6.8 6.8
5.05.2
5.45.6
5.86.0
6.26.4
6.66.8
7.0S
pring 2
006
Fall
2006
Spring 2
007
Fall
2007
Spring 2
006
Fall
2006
Spring 2
007
Fall
2007
Spring 2
006
Fall
2006
Spring 2
007
Fall
2007
Spring 2
006
Fall
2006
Spring 2
007
Fall
2007
F/S Interaction Organization Presentation Evaluation
Content Area
SS
I S
ati
sfa
cti
on
Level (S
cale
1-7
)
August 2008
Southeastern Association for Community College Research
27
Occupation Profile Example
2005 Location
Pay Period 10% 25% Median 75% 90%
Hourly $12.63 $15.61 $19.79 $25.22 $31.41 United States
Yearly $26,300 $32,500 $41,200 $52,500 $65,300
Hourly $12.36 $15.33 $19.59 $23.95 $27.42 Florida
Yearly $25,700 $31,900 $40,700 $49,800 $57,000
Hourly $14.25 $16.15 $19.09 $21.91 $26.25 Tampa-St. Petersburg-Clearwater, FL MSA
Yearly $29,600 $33,600 $39,700 $45,600 $54,600
Table 5Wage Information for Paralegals
Source: Bureau of Labor Statistics, Occupational Employment Statistics Survey; Florida Agency for Workforce Innovation
The distribution of 2005 wage information is divided by percentiles for hourly and yearly wages.
August 2008
Southeastern Association for Community College Research
28
Use of Results• To encourage the use of results, the
program director and provost are required to provide an action plan for improving the performance of the program
• Follow-up report on these results is required the following year
• CAPR process also includes a review of the CAPR documentation by the technical advisory committee and the President’s Cabinet
August 2008
Southeastern Association for Community College Research
29
Action Plan Example
Action ItemCompleti
on Date
Responsible Party
1
Rework website for two-year program Include pdf worksheet for students. Add information on ABA approval Video clips of instructors discussing their
classes and how to be successful in that class
Add instructions on working with FACTS.org
August 2007Program Coordinator
2Construct a survey to be administered to all
PLA 1003 students to determine their goals in the program/class
May 2007Program Coordinator
3
Develop one semester upper division classes which will allow AA students to gain basic paralegal information in Real Estate, Probate, Litigation, and Family Law
June 2007Program Coordinator
Special Resources Needed:Specialized data regarding library and textbook usageArea(s) of Concern/Improvement:The advisory committee and the instructors have expressed their wish that the portfolio in its current form continue to be supported by the college’s technology team…
August 2008
Southeastern Association for Community College Research
30
Stakeholder Perceptions• CAPR has been well received by
stakeholders (program directors, provost, and members of the President’s Cabinet)
– “…provides a more representative picture of the overall quality and sustainability of an individual academic program.” - SVP, Baccalaureate Programs and University Partnerships
– “…greatly enhanced SPC’s capacity to evaluate its lower division academic programs and to encourage the use of assessment results to improve College programs.” – VP, Information Systems, Business Services, Budgets, Planning
August 2008
Southeastern Association for Community College Research
31
Ed Outcomes Site• To provide a medium for completing the
educational assessment reports as well as establishing a repository for program specific information, SPC developed an Educational Assessment Web site (https://it.spcollege.edu/edoutcomes/)
• College administration and instructional staff are provided access to “completed” assessment reports including the CAPR
• Online access further encourages the use of assessment data as well as highlighting “best practices” across the college
August 2008
Southeastern Association for Community College Research
32
Ed Outcomes Site – College Access
August 2008
Southeastern Association for Community College Research
33
Ed Outcomes Site – PD Access
August 2008
Southeastern Association for Community College Research
34
Ed Outcomes Site – PD Access
August 2008
Southeastern Association for Community College Research
35
Ed Outcomes Site – CAPR
• Program Review is just one of the educational materials included in the Ed Outcomes site
• CAPRs are uploaded into the site as a complete document (PDF)
August 2008
Southeastern Association for Community College Research
36
Ed Outcomes Site - CAPR
August 2008
Southeastern Association for Community College Research
37
Ed Outcomes Site – APAR• Academic Program Assessment
Reports (APARs) are created and integrated within the Ed Outcomes site allowing:– Easier aggregate reporting – Automated creation of follow-up
reports– Automated Email approval
process
August 2008
Southeastern Association for Community College Research
38
Ed Outcomes Site - APAR
August 2008
Southeastern Association for Community College Research
39
Ed Outcomes Site - APAR
August 2008
Southeastern Association for Community College Research
40
IE Website
August 2008
Southeastern Association for Community College Research
41
Future Direction Our ultimate goal is to provide
stakeholders ‘timely’, ‘relevant’, ‘accurate’, and ‘interpretable’ data through: Formatted (dashboard) style reports,
and On-demand customizable reporting,
with Valid, reliable, and standardized
measures.
August 2008
Southeastern Association for Community College Research
42
QuestionsDepartment of Institutional research and Effectiveness
St. Petersburg College P.O. Box 13489,
St. Petersburg, FL 33733 (727) 341-3059
FAX (727) 341-5411
Southeastern Association
for Community College Research
2008 Conference
Beyond Compliance…Using a Model of Comprehensive Academic Program Review
to Improve Education