50
Developing an Outcomes Assessment Program: The CCBC Approach Dr. Irving Pressley McPhail, Chancellor The Community College of Baltimore County Delgado Community College – February 18, 2005
Developing an Outcomes Assessment Program:
The CCBC Approach Dr. Irving Pressley McPhail, Chancellor
The Community College of Baltimore County
Delgado Community College – February 18, 2005
LearningFIRST 2.0
The Strategic Plan For CCBC
FY 2004 to FY 2008
StudentLearning
Learning College
InfusingTechnology
LearningSupport
EffectiveCommunication
Organizational Excellence
ValuingDiversity
Community& InstitutionalAdvancement
EnrollmentManagement
Learning First 2.0
CCBC: A Vanguard Learning College
One of twelve community colleges from across the U.S. and Canada chosen to participate in the League’s Vanguard Learning College Project
Five areas of concentration: Organizational Culture Staff Recruitment and Development Learning Outcomes Assessment Technology Underprepared Students
Middle States Standard 14: Assessment of Student Learning
Assessment of student learning
demonstrates that the institution’s
students have knowledge, skills, and
competencies consistent with
institutional goals and that students at
graduation have achieved appropriate
higher education goals.
Middle States Standard 14: Assessment of Student Learning
In order to carry out meaningful assessment activities, institutions must articulate statements of expected student learning at the institutional, program, and individual course levels, although the level of specificity will be greater at the course level. Course syllabi or guidelines should include expected learning outcomes.
SACS – Principles of Accreditation
3.3 Institutional Effectiveness 3.3.1 The institution identifies expected
outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.
Council on Innovation and Student Learning
(CISL)
How it all began!!
Chancellor’s Charge to CISL
Serve as a college-wide think tank
Lead the transformation of the CCBC into
a premier, learning centered college
Serve as change agents
Help to frame policies, procedures and
infrastructure needed to become a
learning college
Getting Started - Fall 1998
Council on Innovation and Student Learning Membership included trustees, chancellor, faculty,
professional staff and classified staff (35-40 members)
Assessment subcommittee created the Learning Outcomes Assessment Plan for the College
(8 members) Guide to Learning Outcomes Assessment and
Classroom Assessment (updated Spring 2003) www.ccbcmd.edu/loa/index.html
Measuring Student Learning
Measuring Student Learning
Non-measures Student satisfaction
surveys Program evaluation MHEC reporting Student grades Retention rates Graduation rates Transfer rates
Indirect measures Exit interviews of
graduates Employer surveys Transfer studies
Measuring Student Learning
Direct measures include: Standardized tests Portfolio assessment Capstone experience Locally developed tests Externally reviewed exhibitions and
performances
CCBC’s Learning Outcomes Assessment Program
CCBC’s Assessment Plan Principles
Primary reason for assessment is to improve and expand student learning
Development of an effective program is a long-term, dynamic process
Must involve a multi-method approach Must include training and support for faculty
and staff Results are not used punitively for students or
faculty
CCBC’s Assessment Plan Principles
Seek to use the most reliable, valid methods and instruments available
Never an end in itself, only a means to an end – the improvement of student learning
Continuous Quality Improvement
DesignImpleme
ntAssess
CCBC’s Outcomes AssessmentPhilosophy
Assessment is: A natural and on-going part of instruction Consistent with traditional instructional
practices Designed to meet specific objectives Conducted in a risk-free environment
LOA Project Collaboration
Learning Outcomes Associate Planning, Research and Evaluation Liaison Learning Outcomes Assessment Advisory Board Deans’ Council General Education Review Board Developmental Education Advisory Committee
LOA Project Collaboration Learning Outcomes Associate
Development of design
Survey/tool development
Staff development
Logistical plan
Institutional Research Data entry Data analysis Data interpretation
Types of Designs
Portfolio assessment Standardized tests External graders/experts Pre- and post-tests Cooperation with other schools Creativity Abounds!
Project Elements
Stage 1: Design Course Project
Determine Measurable Objectives Select Assessment Instrument Include External Validation Control Important Variables
Develop Full Proposal
Stage 2: Implement Design Administer Assessment Instrument(s) Collect Data Analyze Data
Project Elements
Stage 3: Test Objectives
Met? > Write Final Report
Not Met? > Design Course Improvements Write Interim Report
Project Elements
Stage 4: Implement Course Improvements Collect Data Analyze Data
Stage 5: Final Report
Discipline/Campus College
Project Elements
In Summary
Establish Climate of Continuous Improvement
Empower Faculty Provide Substantial Administrative and
Fiscal Support Share Results - Celebrate Successes
Course Level Assessment
High Impact LOAs
CINS 101 HLTH 101 PEFT 101 SDEV 101 ENGL 101
MGMT 101 SOCL 101 PHIL 101 BIOL 110 SPCM 101
ARTS 104 ENVS 101 PSYC 105 MATH 082
FY ‘04FY ‘03 FY ‘05
HLTH 101: Health and Wellness
High impact project Pre-test/post-test design 100 item, four option multiple choice format exam Topics included subjects commonly covered in
introductory health textbooks Draft instrument was reviewed by a health/curriculum
expert at a four-year university Two and four-year colleges and universities have been
invited to share our instrument so that we might develop external comparisons
Health 101 LOAResults
Matched pair t-test analysis Students’ scores improved significantly from pre-
to post-test At least 75% of students scored a 75% or higher
on the post-test Scores varied by campus; small difference in
scores based on race Reassessment will be conducted after
recommendations have been implemented
Health 101 LOA Recommendations
Three topic areas were identified as needing improvement: nutrition, heart disease, and human sexuality
The campus with students who consistently scored higher on the pre-test will offer more honors’ sections
A faculty guide, “Strategies for Teaching Health Education” has been developed and shared with full-and part-time faculty
RDNG 052: College Reading
Instruments Used the Nelson-Denny Reading Test (Forms G
and H) pre and post for the 2001 and the 2003 assessments
Used the Learning Attitudes Study Strategies Inventory (LASSI) for the two semesters of pilot projects and the 2001 assessment
Design Pre-post project design Matched pair data analysis
RDNG 052: College Reading - Fall 2001 Results
Mean pre-post difference of 3.5 (raw points) was statistically significant at the .001 level
Post-test score was greater than pre-test score in 74% of cases
Resulting grade level increase of .7 grade level (in a 15 week semester)
RDNG 052: College Reading - Fall 2001 Results
Exiting students read at 9.6 mean grade level
Differences among campuses—learned from best practices on each campus
LASSI results showed significant improvement in 10 of 12 scales from pre to post
RDNG 052: College Reading – Fall 2001 Interventions
Interventions included: Using more Internet-based learning
opportunities Enhancing instructional activities that focused
on literal and inferential comprehension Greater focus on working with adjunct faculty
RDNG 052: College Reading – Fall 2003 Results
Mean pre-post difference of 3.7 was statistically significant at the .001 level
Post-test score was greater than pre-test score in 75% of cases (larger sample)
Resulting grade level increase of 2.0! (in a 15 week semester)
Exiting students read at 10.1 mean grade level Differences among campuses; every campus
improved
RDNG 052 - 2001 to 2003: Improvements in Student Learning
Significant improvements in students’ reading levels from 2001 to 2003 indicate increased learning in RDNG 052.
RDNG 052 College Reading - Final Recommendations
The Cycle of Continuous Improvement Research Accuplacer cut scores between
RDNG 051 and RDNG 052 Continue to close the performance gap Continue to share best practices Continue to study post-Reading success rates
(success in subsequent courses)
Making the Transition from Course to Program Level Assessment
Program Review General Education Developmental Education
General Education
The Academic Profile
And
The “GREATs”
The Academic Profile
Norm-referenced, externally developed test of general knowledge
Widely used Sample size: 1,017 Students Purpose - to establish baseline data upon
implementation of new General Education Program in Fall 2001
Academic Profile - Findings
The Academic Profile provides CCBC with a baseline measure of how our students are acquiring academic skills developed through General Education courses compared to national norms. CCBC students scored at or close to the national
norms on the total mean score and all sub-scores the first time the test was administered.
The Academic Profile will be administered again in Fall 2004 (sample size 2,000+).
Academic Profile –Data Implications
Work on Critical Thinking skills Reinforce skills learned in one class in
other classes Provide Culturally Mediated Instruction Possible statewide initiatives
GREAT Projects
General Education Assessment Teams (GREAT) Projects for all Maryland Higher Education Commission general education categories
Common Graded Assignments (CGA) and accompanying Scoring Rubrics Created by faculty teams
External Consultant Faculty Training Three semesters of Pilot Projects Full implementation began in Fall 2003 Every course assessed once every three years
GREAT Data (Fall, 2003) Implications Increased awareness of faculty regarding how
General Education courses are defined by the six criteria
Midrange to higher scores in Content may indicate a traditional approach in how “Content” is conveyed with little expectation for higher level use of content in application, critical thinking, analysis, and synthesis activities
Workshops on how to use the GREAT data to improve student learning
LOA Successes
Creating a “culture of assessment” with increased faculty participation and buy-in
Using outcomes assessment for Program-level assessment
Forging new partnerships between faculty teams, institutional research staff, the Vice Chancellor for Learning and Student Development’s office, and the Outcomes Associate in analyzing data and making curricular and pedagogical recommendations for change
LOA Successes
Creating a newly updated Guide for Learning Outcomes Assessment and Classroom Learning Assessment, a model guide for assessment
www.ccbcmd.edu/loa/index.html
Establishing the Learning Outcomes Assessment Advisory Board
Recognition by the League for Innovation in the Community College and Middle States Commission on Higher Education as a national leader in Learning Outcomes Assessment
For Further Information:
Dr. Irving Pressley McPhail
Chancellor
The Community College of Baltimore County
800 South Rolling Road
Baltimore, Maryland 21228-5317
Telephone: 410-869-1220
Fax: 410-869-1224
Email: [email protected]
Website: www.ccbcmd.edu
For Further Information:
Dr. Alvin Starr, Acting Vice Chancellor for Learning and Student Development
Email: [email protected]
Dr. Rose Mince, Assistant to the VCLSDEmail: [email protected]
Professor Tara Ebersole, Outcomes AssociateEmail: [email protected]
Questions /Comments
