Southeastern Association for Community College Research 2008 Conference Beyond Compliance…Using a...

Southeastern Association

for Community College Research

2008 Conference

Beyond Compliance…Using a Model of Comprehensive Academic Program Review

to Improve Education

August 2008

Southeastern Association for Community College Research

2

Comprehensive Academic Program Review

Department of Institutional research and Effectiveness

St. Petersburg College P.O. Box 13489,

St. Petersburg, FL 33733 (727) 341-3059

FAX (727) 341-5411

Presenters

•James Coraggio, Director of Institutional Research and Effectiveness

•Maggie Tymms, Assessment Coordinator for Academic Programs

August 2008


3

SPC Background SPC, established in 1927, is the

oldest 2-year college in Florida First Community College in Florida to

offer 4 year degrees (2002) 9 Campuses throughout the county 2007-08 FTE: 16,086 (LD), 1,338

(UD) Opening Spring 2008 headcount:

25,754 Annual 2007-08 enrollment: 62,200

August 2008


4

Compliance Issues with IE SACS’s Peer Review Research

Project revealed that in the area of Institutional Effectiveness (IE) 62% of institutions in off-site reviews

were deemed non-compliant and 27% of institutions in on-site visits

were deemed non-compliant IE was the second most identified

area behind faculty qualifications

August 2008


5

SPC Experience with IE As part of the recent SACS

reaccreditation process, SPC spent a considerable amount of time and effort documenting and detailing their IE processes

The result…SPC had “zero” compliance issues in the area of IE during the off-site reviews

August 2008


6

SACS Requirements for IE Core Requirement 2.5: Integrated

Planning & Evaluation Processes “…ongoing, integrated, and institution-wide

research-based planning and evaluation processes…”

Comprehensive Standard 3.3.1: Institutional Effectiveness “…identifies expected outcomes…; assesses

whether it achieves these outcomes; and provides evidence of improvement…

Comprehensive Standard 3.5.1: General Education Core “… identifies college-level competencies

within the general education core and provides evidence that graduates have attained those competencies.”

August 2008


7

SACS Requirements for IE Additional IE related requirements

include: Comprehensive Standard 3.1.1:

Institutional Mission Federal Requirement 4.1: Evaluating

Success Federal Requirement 4.2: Curriculum

August 2008


8

IE Evidence SACS Suggested Documentation for CR 2.5

Evidence of linkage of IE to institutional mission

Institutional plans and budgets that demonstrate linkage of assessment findings to planning at all levels

Minutes of appropriate (IE related) unit, committee, task force meetings…

Documentation that relates to IE, such as budget preparation instructions, minutes of budget presentation meetings, annual reports, annual assessment updates, IE reports

Samples of specific actions taken to improve the IE process and/or results from that process

August 2008


9

Performance Improvement From Compliance to Performance

Improvement

Data Information ImprovementKnowledge

Improved OutcomesSPCReportsAssessment Data

CommunityCollege LeadershipProgram Directors

FacultyStudents

August 2008


10

IE at SPC

“Institutional Effectiveness is the integrated, systematic, explicit, and documented processes of measuring performance against the SPC mission for purposes of continuous improvement of academic programs, administrative services, and educational services offered by the college.”

Operating ResultsBudget PlanningStrategic & Unit

PlanningEvaluations/ AssessmentsMission

“Closing the Loop”

August 2008


11

Evaluation & Assessment Evaluation and Assessment

are about ‘Choices, Direction, Focus…Quality Improvement’

• Instructional Effectiveness Processes at SPC were recently evaluated as part of the College’s on-going continuous performance improvement

August 2008


12

Educational Assessment• One IE process that experienced

extensive revisions was educational assessment

• Individual assessments in the areas of academic programs, Gen Ed, and program review were evaluated and enhancements were made to the quality and integration of assessments as well as the dissemination of assessments results

August 2008


13

Changes & Improvements Evaluated individual assessment

instruments and current assessment processes

Found new ways to integrate assessment with the educational process Focus on fewer, yet meaningful improvements (action items) Use multiple assessment methods

(direct & indirect) Involve faculty in process Integrate timing of assessments (3-year cycle)

August 2008


14

Three-year Assessment Cycle

Academic Program Viability Reviews

(APVRs) conducted yearly

APARs

CAPRs

August 2008


15

Changes and Improvements Now looking at the ‘Big Picture’ Assessment focus is on CHANGE

through Quality Improvement Faculty-driven assessment

process; IRE serves in consultant capacity

Focus on critical major learning outcomes (MLOs) with fewer, more focused action plan items

Improve access and awareness of assessment information

August 2008


16

SPC Assessments

1. On-going

2. On-going

3. Annual

1. Environmental Scans

2. Trends Analysis

3. WorkNet Surveys & Exit Interviews

Economic & Workforce Development

AS Degrees & PSAV Programs

WorkNet One-Stop Ops.

1. Annual

2. Annual

3. Annual

4. Annual

5. Annual

6. Annual

7. Annual

8. Annual

9. Annual

10. Annual

11. Annual

1. Entering Students Survey

2. Enrolled Students Survey

3. Graduating Students Survey

4. Employer Survey

5. Recent Alumni Survey

6. Budget. Planning, IR Survey

7. Technology Resources Allocation Survey

8. Other Department-specific Surveys

9. Administrative Services Oversight (ASOG) Reports

10. Student Services Oversight (SSO) Report

11. Education Support Services (ESS) Reports (Libraries, Learning Support Centers)

Partnerships / Services Student Services

Community Services

Other Institutions

1. Annual

2. Every 3-years / program

3. Annual


5. Annual

1. Education Oversight Group (EOG) Reports

2. Academic Program Assessment Reports (APARs)

3. Academic Program Viability Reviews (APVRs)

4. Comprehensive Academic Program Reviews (CAPRs)

5. End-of-year SD-IOs Assessment Report

Accessible Education BS, BAS, AA, AS Degrees

Technical Certificates

Applied Tech. Diplomas

Distance Education

Developmental Education

Continuing Education

Recurring CycleEvaluations / AssessmentsSPC Mission Components

1. On-going

2. On-going

3. Annual

1. Environmental Scans

2. Trends Analysis

3. WorkNet Surveys & Exit Interviews

Economic & Workforce Development

AS Degrees & PSAV Programs

WorkNet One-Stop Ops.

1. Annual

2. Annual

3. Annual

4. Annual

5. Annual

6. Annual

7. Annual

8. Annual

9. Annual

10. Annual

11. Annual

1. Entering Students Survey

2. Enrolled Students Survey

3. Graduating Students Survey

4. Employer Survey

5. Recent Alumni Survey

6. Budget. Planning, IR Survey

7. Technology Resources Allocation Survey

8. Other Department-specific Surveys

9. Administrative Services Oversight (ASOG) Reports

10. Student Services Oversight (SSO) Report

11. Education Support Services (ESS) Reports (Libraries, Learning Support Centers)

Partnerships / Services Student Services

Community Services

Other Institutions

1. Annual


3. Annual


5. Annual

1. Education Oversight Group (EOG) Reports

2. Academic Program Assessment Reports (APARs)

3. Academic Program Viability Reviews (APVRs)

4. Comprehensive Academic Program Reviews (CAPRs)

5. End-of-year SD-IOs Assessment Report

Accessible Education BS, BAS, AA, AS Degrees

Technical Certificates

Applied Tech. Diplomas

Distance Education

Developmental Education

Continuing Education

Recurring CycleEvaluations / AssessmentsSPC Mission Components

SPC Mission – Assessments - Closing the Loop

August 2008


17

Program Review Process

• New enhanced process is a summative evaluation that includes two reports with multiple measures such as:– program-specific performance

measures– profitability measures– economic data

August 2008


18

Academic Program Viability Review

Published yearly Included measures:

Program Graduates Course Enrollment Student Semester Hour

(SSH) productivity Relative Profitability Index

(RPI-T) Economic Trend Data

August 2008


19

CAPR Objectives• Comprehensive Academic Program

Review (CAPR) developed to meet three objectives within the academic assessment process:– To provide a comprehensive report that

summarizes all elements of the program’s viability and productivity from a 360-degree perspective

– To provide comprehensive and relevant program-specific information to key College stakeholders, such as the President’s Cabinet members, in order to make critical decisions regarding the continued sustainability of a program

– To provide program leadership a vehicle to support and document actionable change for the purposes of performance improvement

August 2008


20

Previous Program Review Model• Traditionally, program reviews at

SPC consisted primarily of a community focus group and a few occupational growth measures

• This information was presented to the President’s Cabinet for evaluation

• CAPR was designed to be more representative of a program’s quality and, as such, contains measures involving a number of stakeholder perspectives

August 2008


21

Elements of the CAPR• Specific CAPR measures include the

– program description with recent program accreditation information

– program performance measures including enrollment, productivity, grade distributions, and fulltime/adjunct faculty ratios

– program profitability measures– academic outcomes from recent end-of-

program assessments– stakeholder perceptions including student

surveys of instruction results, advisory committee minutes, and employer and recent alumni survey results

– occupation trends and information– state graduate outcomes information– the program director’s perspective of

program issues, trends, and recent success

August 2008


22

Measures CAPR

• In order to ensure that the information presented in the CAPR is used appropriately, each measure includes: – a standardized definition of the

performance measure– a reference to the source

information used in the calculation

August 2008


23

Program Performance Example

Figure 4: Program GraduatesSource: SPC Factbook, Table 31

Program Graduates

Program Graduates

6

911

14

17

14

0

2

4

6

8

10

12

14

16

18

20

2001-02 2002-03 2003-04 2004-05 2005-06 2006-07

Gra

du

ate

s

August 2008


24

Program Profitability

Example

Figure 8: Fiscal SummarySource: PeopleSoft Financial Production database, report ID: ORGBUDSI

Relative Profitability Index (RPI-T) is calculated by dividing a program’s income by the sum of its personnel costs and current expenses.

Fiscal Summary

0.430.31

0.38

0.61

0.40

0.000.100.200.300.400.50

0.600.700.800.901.00

2002-03 2003-04 2004-05 2005-06 2006-07

RP

I-T

August 2008


25

Academic Outcomes Example

Table 3 Summary of 2004-2005 Assessment Findings

Evaluation Area Summary of Assessment Findings - Portfolio Evaluations

Number of Students

Sufficient Documentation

(Mean)

Acceptable to Florida Courts &

Attorneys (Mean)

Timely (Mean)

2004-2005 Session I 9

MLO 1 9 3.83 3.85 3.89

MLO 2 9 3.59 3.78 3.78

MLO 3 9 3.70 3.78 3.81

MLO 4 9 3.78 3.81 3.81

MLO 5 9 3.70 3.81 3.81

MLO 6 9 3.52 3.81 3.70

MLO 7 9 3.59 3.81 3.81

MLO 8 9 3.81 3.85 3.85

Paralegal program was evaluated through an Academic Program Assessment Report (APAR) in 2004-2005. Each of the program’s eight major learning outcomes (MLOs) was evaluated.

August 2008


26

Stakeholder Perceptions Example

Figure 9: SSI Lecture CoursesSource: PeopleSoft Query S_SSI_CHRT_QRY_CAMPUS

Purpose of the SSI survey is to acquire information on student perception of the quality of courses, faculty, and instruction, and to provide feedback information for improvement. SSI: Lecture Courses

6.76.9 6.8 6.8

6.76.8 6.9 6.8

6.7

6.9 6.9 6.8

6.6

6.8 6.8 6.8

5.05.2

5.45.6

5.86.0

6.26.4

6.66.8

7.0S

pring 2

006

Fall

2006

Spring 2

007

Fall

2007

Spring 2

006

Fall

2006

Spring 2

007

Fall

2007

Spring 2

006

Fall

2006

Spring 2

007

Fall

2007

Spring 2

006

Fall

2006

Spring 2

007

Fall

2007

F/S Interaction Organization Presentation Evaluation

Content Area

SS

I S

ati

sfa

cti

on

Level (S

cale

1-7

)

August 2008


27

Occupation Profile Example

2005 Location

Pay Period 10% 25% Median 75% 90%

Hourly $12.63 $15.61 $19.79 $25.22 $31.41 United States

Yearly $26,300 $32,500 $41,200 $52,500 $65,300

Hourly $12.36 $15.33 $19.59 $23.95 $27.42 Florida

Yearly $25,700 $31,900 $40,700 $49,800 $57,000

Hourly $14.25 $16.15 $19.09 $21.91 $26.25 Tampa-St. Petersburg-Clearwater, FL MSA

Yearly $29,600 $33,600 $39,700 $45,600 $54,600

Table 5Wage Information for Paralegals

Source: Bureau of Labor Statistics, Occupational Employment Statistics Survey; Florida Agency for Workforce Innovation

The distribution of 2005 wage information is divided by percentiles for hourly and yearly wages.

http://www.bls.gov/oes

http://www.bls.gov/oes

http://www.labormarketinfo.com/

August 2008


28

Use of Results• To encourage the use of results, the

program director and provost are required to provide an action plan for improving the performance of the program

• Follow-up report on these results is required the following year

• CAPR process also includes a review of the CAPR documentation by the technical advisory committee and the President’s Cabinet

August 2008


29

Action Plan Example

Action ItemCompleti

on Date

Responsible Party

1

Rework website for two-year program Include pdf worksheet for students. Add information on ABA approval Video clips of instructors discussing their

classes and how to be successful in that class

Add instructions on working with FACTS.org

August 2007Program Coordinator

2Construct a survey to be administered to all

PLA 1003 students to determine their goals in the program/class

May 2007Program Coordinator

3

Develop one semester upper division classes which will allow AA students to gain basic paralegal information in Real Estate, Probate, Litigation, and Family Law

June 2007Program Coordinator

Special Resources Needed:Specialized data regarding library and textbook usageArea(s) of Concern/Improvement:The advisory committee and the instructors have expressed their wish that the portfolio in its current form continue to be supported by the college’s technology team…

August 2008


30

Stakeholder Perceptions• CAPR has been well received by

stakeholders (program directors, provost, and members of the President’s Cabinet)

– “…provides a more representative picture of the overall quality and sustainability of an individual academic program.” - SVP, Baccalaureate Programs and University Partnerships

– “…greatly enhanced SPC’s capacity to evaluate its lower division academic programs and to encourage the use of assessment results to improve College programs.” – VP, Information Systems, Business Services, Budgets, Planning

August 2008


31

Ed Outcomes Site• To provide a medium for completing the

educational assessment reports as well as establishing a repository for program specific information, SPC developed an Educational Assessment Web site (https://it.spcollege.edu/edoutcomes/)

• College administration and instructional staff are provided access to “completed” assessment reports including the CAPR

• Online access further encourages the use of assessment data as well as highlighting “best practices” across the college

https://it.spcollege.edu/edoutcomes/

August 2008


32

Ed Outcomes Site – College Access

August 2008


33

Ed Outcomes Site – PD Access

August 2008


34

Ed Outcomes Site – PD Access

August 2008


35

Ed Outcomes Site – CAPR

• Program Review is just one of the educational materials included in the Ed Outcomes site

• CAPRs are uploaded into the site as a complete document (PDF)

August 2008


36

Ed Outcomes Site - CAPR

August 2008


37

Ed Outcomes Site – APAR• Academic Program Assessment

Reports (APARs) are created and integrated within the Ed Outcomes site allowing:– Easier aggregate reporting – Automated creation of follow-up

reports– Automated Email approval

process

August 2008


38

Ed Outcomes Site - APAR

August 2008


39

Ed Outcomes Site - APAR

August 2008


40

IE Website

August 2008


41

Future Direction Our ultimate goal is to provide

stakeholders ‘timely’, ‘relevant’, ‘accurate’, and ‘interpretable’ data through: Formatted (dashboard) style reports,

and On-demand customizable reporting,

with Valid, reliable, and standardized

measures.

August 2008


42

QuestionsDepartment of Institutional research and Effectiveness

St. Petersburg College P.O. Box 13489,

St. Petersburg, FL 33733 (727) 341-3059

FAX (727) 341-5411

Southeastern Association

for Community College Research

2008 Conference

Beyond Compliance…Using a Model of Comprehensive Academic Program Review

to Improve Education

Date post:	14-Jan-2016
Category:	Documents
Upload:	charlotte-norman
View:	214 times
Download:	0 times

Southeastern Association for Community College Research 2008 Conference Beyond Compliance…Using a...

Documents