Post on 18-Dec-2015
transcript
The Creation of Two Comparison Groups to Facilitate Evidence-Based Decision Making
Prepared and Presented by Keith WurtzDean, Institutional Effectiveness, Research and Planning
Crafton Hills College
Session Objectives
• Provide a brief introduction to the Left Lane Program
• Explain where the idea for two comparison groups came from
• Describe the two comparison groups and the limitations of the study
• Describe the findings• Review the process for Continuous Quality
Improvement and how the results were used to inform decision-making
The Left Lane Program (LLP)
• Purpose of the LLP is to reduce the amount of time it takes students to earn an AA/AS Degree at Crafton from 5.3 years to 4 years
• Program funded by the San Bernardino Community College District Chancellor
• First Cohort in Fall 2012
The Left Lane Program (LLP)
• Students who participate in Left Lane receive Priority Registration B
• To keep the Priority Registration B status students are required to do the following– Attend a Summer Bridge Program– Participate in SOA3R – Student Orientation, Application,
Assessment, Advisement, and Registration– Complete and closely follow their Student Education Plan– Enroll in math and English as stated in their SEP– Attend all tutoring meetings and check-in appointments– Maintain a 2.0 GPA– Enroll in 12 or more units per semester (depending on SEP)
The Left Lane Program (LLP)
• How are students selected for the LLP?– Need to place into one level below transfer English
or one level below transfer level math– Need to apply
The Idea for Two Comparison GroupsCollaborative Model of Institutional Research
Faculty/Staff/Student
Researcher
Joint Activity
Primary Responsibility
Key Features:• Dialogue-rich• Jointly-driven processes
Source: Dr. Rob Johnstone and Priya Chaplot, RP Group Conference Presentation
Raju HegdeDean of Math, English, Reading, and Instructional Support
Kirsten ColveyDean of Student Services, Counseling, and Matriculation
The Two Comparison Groups
• Left Lane Students– 258 out of 276 (93%) earned a GOR in Fall 2012– 82% graduated high school in 2012– 63% Female– 49% Hispanic
• First Comparison Group (Last Year High School was 2012)
• Second Comparison Group (Last Year High School was 2012 and did not participate in LLS)
The Two Comparison Groups
Characteristic Left Lane Students
1st Comparison Group2012 High School
Graduates
2nd Comparison Group2012 HS Graduate and Did not
Participate in LL Strategy
N 258 583 357
Graduated HS 82% 100% 100%
Female 63% 48% 49%
Hispanic 49% 44% 48%
SOA3R Yes Varied No
LC Yes Varied No
SI Varied Varied No
The Two Comparison Groups
Characteristic Left Lane Students
1st Comparison Group2012 High School
Graduates
2nd Comparison Group2012 HS Graduate and Did not
Participate in LL StrategyN 258 583 357
Placed Basic Skills Math 22% 26% 30%
Placed Developmental Math 77% 64% 62%
Placed Transfer Level Math 1% 10% 8%
Placed Basic Skills English 36% 37% 40%
Placed Developmental English 40% 34% 32%
Placed Transfer Level English 24% 29% 28%
Limitations
• The success rate comparisons do not control for instructor or by when the section was offered
• The comparison groups do not control for skill level (i.e. math and English placements)
Percent of Students Earning a Grade on Record in English and Math
GOR in English GOR in Math0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
77%
89%
45% 48%
26%34%
Left LaneFirst GroupSecond Group
Left Lane students were substantially (Effect Sizes were > .67) and statistically significantly (p values were < .001) more likely to earn GOR in English and Math.
Percent of Students Successfully Completing English and Math
Success in English Success in Math0%
10%20%30%40%50%60%70%80%90%
100%
82%
64%
78%
59%69%
51%
Left Lane First Group Second Group
• Left Lane students were slightly more likely to successfully complete English (ES = .10, p = .308) and math (ES = .11, p = .196) than students in the first comparison group
• Left Lane students were substantially and statistically significantly more likely to successfully complete English (ES = .32, p = .020) and math (ES = .27, p = .017) than students in the second comparison group
Percent of Students Successfully Completing English, Math, and/or Reading
Female African American Hispanic0%
10%20%30%40%50%60%70%80%90%
100%
81% 81% 79%73%
61%70%69%
45%
66%
Left Lane First Group Second Group
• Left Lane female (ES = .20, p < .001, 81% to 73%), African American (ES = .43, p = .007, 81% to 61%), and Hispanic (ES = .19, p < .001, 79% to 70%) students were substantially and statistically significantly more likely to successfully complete courses than students in the first comparison group
• Left Lane female (ES = .31, p < .001, 81% to 68%), African American (ES = .77, p < .001, 81% to 45%), and Hispanic (ES = .27, p < .001, 79% to 66%) students were substantially and statistically significantly more likely to successfully complete courses than students in the second comparison group
English, Math, and Overall Retention (i.e. Persistence – earning GOR) from Fall to Spring
English Math Overall Retention0%
10%20%30%40%50%60%70%80%90%
100%
65%56%
87%
52%46%
80%
46% 43%
75%
Left Lane First Group Second Group
• Left Lane English (ES = .25, p = .007), Math(ES = .19, p = .034), and overall (ES = .19, p < .012) retention was substantially and statistically significantly higher for Left Lane students than students in the first comparison group
• Left Lane English (ES = .39, p = .002), Math(ES = .27, p = .018), and overall (ES = .31, p < .001) retention was substantially and statistically significantly higher for Left Lane students than students in the second comparison group
English and Math Improvement Rates from Fall to Spring
Success in Subsequent English
Success in Subsequent Math
Overall Success in Subsequent Course
0%10%20%30%40%50%60%70%80%90%
100%
74%
61%67%
76%66%
71%79%
64%71%
Left Lane First Group Second Group
Left Lane students did not perform as well in their subsequent English and/or math course after successfully completing an English and/or math course in their first semester
1. Gather Data / Assess
Process for Continuous Quality Improvement
ContinuousQuality
Improvement
1. Gather Data / Assess
2. Review, Discuss, & Gather
Feedback
1. Gather Data / Assess
2. Review, Discuss, & Gather
Feedback
3. Identify Strategies for Improvement
1. Gather Data / Assess
2. Review, Discuss, & Gather
Feedback
3. Identify Strategies for Improvement
4. Implement Improvements
Communicate Results
Communicate Feedback
Communicate Strategies
Communicate Improvements
Left Lane Program
Summer Bridge Evaluation Surveys
Initial Research after the first Fall semester
More in-depth research after the first year
1. Gather Data / Assess
2. Review, Discuss, &
Gather Feedback
3. Identify Strategies for Improvement
4. Implement Improvements
Left Lane Program
Summer Bridge Evaluation Surveys
Initial Research after the first Fall semester
More in-depth research after the first year
Evaluation surveys and interviews indicated that students wanted both English and math during Summer Bridge
39% of non-Left Lane students participated in SOA3R, a learning community, or supplemental instruction. As a result, first comparison group had success rates 70% or higher.
LLP students are more likely to enroll in math and English in the first term and they were less likely to enroll in and successfully complete math and English in subsequent spring semester.
1. Gather Data / Assess
2. Review, Discuss, &
Gather Feedback
3. Identify Strategies for Improvement
4. Implement Improvements
A Note on Facilitating Continuous Quality Improvement
• Traditional approach to facilitating evidence-based decision making– Provide the research results to constituents only in reports, briefs, and
dashboards– Discuss the possible implications and interpretations verbally
• Example– Left Lane students were substantially (ES = 1.10) and statistically
significantly (p < .001) more likely to earn a GOR in a math course their first semester (89%) at Crafton than non-Left Lane students (34%)
– Left Lane students were substantially (ES = 1.03) and statistically significantly (p < .001) more likely to earn a GOR in an English course their first semester (77%) at Crafton than non-Left Lane students (26%)
A Note on Facilitating Continuous Quality Improvement
New approach to facilitating evidence-based decision making• Provide the research results to constituents in reports, briefs, and dashboards, and…• Provide the research results with an interpretation that focuses on the possible implications of the research
Left Lane Comparison Groups and Possible Improvement AreasWhen preparing the comparison groups to examine the relationship between Left Lane Students and student outcomes one of the most interesting findings was the number of Fall 2012 first-time Crafton students who graduated from high school in 2012 who did not participate in Left Lane, but had participated in one of the Left Lane Strategies (LLS). Thirty-nine percent (n = 226) of these students participated in SOA3R, a learning community, or in supplemental instruction. As a result, the first comparison group of incoming students had success rates that were 70% or higher. Moreover, the Left Lane students often did not have a statistically significantly higher likelihood of achieving the outcomes examined than the first comparison group. Suggesting the possibility that one method for improving the LLP is to focus on ensuring that as many students as possible participate in the LLS rather than solely on LLP.
One of the strategies for decreasing the average amount of time it takes students to complete an AA/AS Degree involves increasing how quickly students are able to successfully complete transfer level English and math. The initial results of the research illustrated here indicates that Left Lane students are more likely to enroll in math and English courses in the fall, but in some instances they were not as likely to enroll in math and English in the subsequent semester and/or successfully complete the subsequent English and/or math course in the summer semester. LLP may need to require students to participate in the LLS in the spring semester and devote additional resources to services in the spring semester.
Left Lane Program
Summer Bridge Evaluation Surveys
Initial Research after the first Fall semester
More in-depth research after the first year
Evaluation surveys and interviews indicated that students wanted both English and math during Summer Bridge
39% of non-Left Lane students participated in SOA3R, a learning community, or supplemental instruction. As a result, first comparison group had success rates 70% or higher.
LLP students are more likely to enroll in math and English in the first term and they were less likely to enroll in and successfully complete math and English in subsequent spring semester.
Combine Summer Bridge so that students are prepared for both English and math
One method for improving the LLP is to focus on ensuring that as many students as possible participate in the LL strategies.
Require students to participate in LL strategies in the spring along with enrolling in math and English in the spring.
1. Gather Data / Assess
2. Review, Discuss, &
Gather Feedback
3. Identify Strategies for Improvement
4. Implement Improvements
Left Lane Program
Summer Bridge Evaluation Surveys
Initial Research after the first Fall semester
More in-depth research after the first year
Evaluation surveys and interviews indicated that students wanted both English and math during Summer Bridge
39% of non-Left Lane students participated in SOA3R, a learning community, or supplemental instruction. As a result, first comparison group had success rates 70% or higher.
LLP students are more likely to enroll in math and English in the first term and they were less likely to enroll in and successfully complete math and English in subsequent spring semester.
Combine Summer Bridge so that students are prepared for both English and math
One method for improving the LLP is to focus on ensuring that as many students as possible participate in the LL strategies.
Require students to participate in LL strategies in the spring along with enrolling in math and English in the spring.
Combined the Summer Bridge Program to include both English and Math
Required Students to complete a 2 semester SEP to stay enrolled in math and English
New LLP proposal to SBCCD Chancellor emphasized increasing the capacity of the LLP and its strategies: counseling, SEPs , SOA3R and SI. In addition, also emphasized creation of DLAs in math and English, additional sections in math and English in both fall and spring.
LL strategies are also being included in new Title COOP grant proposal.
1. Gather Data / Assess
2. Review, Discuss, &
Gather Feedback
3. Identify Strategies for Improvement
4. Implement Improvements