Program Review by p.mery, dean of institutional effectiveness march 2015
Visiting Team Report – Word Counts
1+ 4001
100 130
200 37
300 9
400 6
500 2
1000 1
Grand Total 4186
number of
words
times they
appear
Visiting Team Report – Word Counts
1+ 4001
100 130
200 37
300 9
400 6
500 2
1000 1
Grand Total 4186
number of
words
times they
appear
“evidence” = 405
“student” = 472 “planning” = 306
“Plan” = 335
“college” = 985
“program review” = 159
“data” = 160
“institution” = 521
Most words occur very infrequently 1496 occur once 2300 occur < 20x
~Top 50 words used to create the “Wordle” (see prior slide) appear far more often than the 4000+ words not included in the Wordle.
“Program Review” is prominent among the ~Top 50.
Quick Quiz
What is the purpose of program review?
Quick Quiz
The purpose of program review is...
◦ Unit level improvement
◦ Collectively achieving institutional goals
◦ Fulfilling our mission
◦ Reaching toward our vision
◦ Meeting educational needs of students
◦ And yes, connection to budget… but as a means not an end.
Self Evaluation
Where are we really?
What examples of fully looped CQI can you cite
based on this year’s program reviews?
Self Evaluation
Where are we really?
Improvements Made as a Result of Programmatic Assessments (I.B.1. excerpt, pages 44-45)
DEPARTMENT/UNIT IMPROVEMENTS
Engineering & Technology As a result of SLO assessment, the Engineering and Technology degree was
revised into two distinct majors with refined major descriptions and course
requirements. It now offers an Electronic Engineering AS and a Mechanical
Engineering AS.
Mathematics Assessment of different delivery modes highlighted that a hybrid, team
teaching method of instruction resulted in higher pass rates. More hybrid
sections are now being scheduled.
Automotive/ Motorcycle
Technology, Construction
& Building Maintenance
Now offers six stackable certificates building up to an AS degree that allows
students to complete a section and return to a work site with recognized
national certification skill sets.
Culinary Arts & Hospitality
Studies Improved and better aligned student expectations and skill building by turning
a 3-unit class into 3 one-unit classes. These one-unit courses drilled down
subject matter and allowed students to tailor skills and build a specialized
culinary portfolio.
Speech Communication After direct assessment of student speeches, students struggled to articulate
their own credibility through research and experience. Custom instructional
materials were created using highly relevant, localized, and current subject
matter.
VTR Evaluation
Where are we really?
Overall, we need to…
make better use of data
maintain a learning outcome focus
get more employees involved
clarify some of the criteria
yes... connect it to budget
overall… build it into our DNA
VTR Evaluation
Where are we really?
I.B.1. The college will be able to maintain compliance assuming it continues its ongoing dialogue about continuous improvement of student learning and institutional processes; and the current and planned staffing levels in the Office of Research and Planning are achieved and maintained so that plans to enhance access to disaggregated program-level data are implemented, the use of data in program review is more consistent across all of the programs, and the assessment question in program review is further clarified.
VTR Evaluation
Where are we really?
I.B.4. In order to remain in compliance, the
college needs to continue efforts to engage more employees in department and college planning activities, determine and implement processes to reallocate existing resources as appropriate.
VTR Evaluation
Where are we really?
II.A.2.c
While the college's program review process is sufficient, a process that made more robust use of data would lead to better-informed continuous improvement.
VTR Evaluation
Where are we really?
II.A.2.e
There is evidence that program review in general at the college is just starting to take shape and that faculty are only starting to fully understand the relevance of the process and importance of planning.
VTR Evaluation
Where are we really? II.B.3. SLO assessments and program review analysis
are taking place, but it is not clear that robust discussions and collaboration between the Ocean Campus and the centers are taking place. As mentioned in Standard II.B.1, student services programs have developed SLOs and program reviews which require the inclusion of SLOs in order to be submitted, but the assessments of these SLOs are in different phases depending on services and locations.
VTR Evaluation
Where are we really? III.D.4. The College’s departments and units develop budget
plans through the use of program review documents completed each year during cycles that integrate with the College’s financial planning calendar. Although the College has been conducting Program Review for many years now, the college is still making strides in ensuring full participation by all programs and completion of all requested elements of the Program Review. The first year to allocate resources using program review was FY 2013-14. This allocation was only of additional funding or staffing and did not tie base allocations to program review.
VTR Evaluation
Where are we really?
III.D.4., continued
During 2014-15, the College will assess the allocations as to the effectiveness and improvements the College achieved. The College is conducting these assessments during the Fall 2014 semester.
VTR Evaluation
For additional analysis…
Members of the Academic Senate including the
Senate President mined the VTR and developed Program Review Processes Consideration, an extensive document called to create dialogue around our Program Review processes.
https://docs.google.com/document/d/1GObJKoqBjatgOMXQnxSRen-8naU_IHKPwS8vhKoGw0U
Your next steps
Your top six Ways to increase (focus on) institutional
effectiveness and CQI: 1. 2. 3. 4. 5. 6.
Your next steps
Your top six Some possible ideas: 1. Become more data-informed 2. Seed some future “looped” examples 3. Collect evidence (follow the loop) 4. Encourage involvement 5. Do nuts & bolts, e.g., budget workshops 6. Be the culture (A-Team models!)
Keep Moving & Modeling
How can you inspire others? List some things you do every day to model
the principles of CQI: 1. 2. 3. 4. 5. 6.
Keep Moving & Modeling
What gets in the way? Some common issues:
◦ Silos ◦ Distrust ◦ Everyday tasks ◦ You name it! ◦ The impediments won’t vanish but we need to keep coaxing, pushing, and modeling CQI
Integrated Assessment, Planning, and Budgeting
A presentation on Program Review is never complete without the flowchart!
Shown on next slide. Note the multiple,
interconnected CQI and assessment wheels beginning with SLOs which inform program review and annual planning, all the way up through long-range, institutional planning.
Quick Reminder