Examining Research from a First-Year Student Math Early Warning Pilot
Dr. Greg Budzban, Chair and Professor Math Department Amber Manning-‐Ouelle?e, Director of Enrollment Management,
College of Business Southern Illinois University Carbondale
Annual First Year Experience Conference February 7-‐10, 2015
Dallas, TX
Session Agenda
• Ins$tu$onal Profile • SIU Gateway Math Course Structure • Math Data and Predic$ve Value of Week 8
Vector Analysis and Markov models • Predic$ve Analysis • Pilot data and Outreach Efforts • Lesson Learned • Q & A
InsQtuQonal Profile • 4-‐year Public Research University and
Open Access • Undergraduate 13,461 • Graduate 4,485 • 48% First-‐genera$on • Over 85% on some type of financial
aid • Average 22.2 ACT • Minority Enrollment 28% • Female 46% Male 54% • 103 Bachelors degrees, 78 Masters, 34
Doctoral programs
Gateway Math Courses Trends
• 22% of students require mathematic s remediation
• Remedial courses not always effective
• Research indicates that the more required developmental courses students take, the less likely they are to do so.
SIU Math Courses
• Math 101 (Non-STEM majors, satisfies Core Curriculum requirement)
• Math 107 (Includes STEM/Business majors, no credit towards degree, “remedial” course)
• Math 108 (Includes STEM/Business majors, credit towards degree, satisfies UCC)
Early Warning IntervenQon PlaUorm
Week 3 25%* (Preparation) + 25%* (Motivation) + 50%* (Demonstration) • RED: 0% to 55% • ORANGE: 56% to 65% • YELLOW: 65% to 75% • GREEN: 76% to 100%
• Students also receive an intervention score in week 8 and week 12 that is simply their course grade at that time.
Intermediate Algebra Data
Fall 2013
Warning'levelWeek'3'Totals
Green'at'week'8
Yellow'at'week'8
Orange'at'week'8
Red'at'week'8
Not'enrolled'week'8
#ABC %ABC
Green'at'wk'3 176 122 38 6 8 2 137 77.84%Yellow'at'wk'3 67 17 27 16 6 1 36 53.73%Orange'at'wk'3 36 5 11 10 9 1 9 25.00%Red'at'wk'3 52 1 2 4 40 5 3 5.77%Not'on'Wk'3'list 4 2 0 1 1 0 1 25.00%'Week'8'totals 335 147 78 37 64 9 186 55.52%%ABC 55.52% 90.47% 57.69% 10.80% 6.25% 0%
Week'3'to'Week'8''Math'107'FALL'2013
College Algebra Data Fall 2013
Warning'levelWeek'3'Totals
Green'at'week'8
Yellow'at'week'8
Orange'at'week'8
Red'at'week'8
Not'enrolled'week'8
#ABC %ABC
Green'at'wk'3 303 245 32 16 10 0 263 86.80%Yellow'at'wk'3 112 36 43 21 12 0 73 65.18%Orange'at'wk'3 79 11 24 22 22 0 29 36.71%Red'at'wk'3 144 4 12 34 91 3 25 17.36%Not'on'Wk'3'list 9 2 3 1 3 0 4 44.44%'Week'8'totals 647 298 114 94 138 3 394 60.90%%ABC 60.90% 94.90% 60.53% 30.85% 9.35% 0%
Week'3'to'Week'8''Math'108'FALL'2013
Intermediate Algebra Data Spring 2014
Warning'levelWeek'3'Totals
Green'at'week'8
Yellow'at'week'8
Orange'at'week'8
Red'at'week'8
Withdrew'by'week'12
#ABC %ABC
Green'at'wk'3 55 26 16 6 7 0 37 67.27%Yellow'at'wk'3 33 5 17 8 3 3 18 54.55%Orange'at'wk'3 29 0 4 17 8 2 9 31.03%Red'at'wk'3 49 0 2 5 42 13 4 8.16%'Week'8'totals 166 31 39 36 60 18 68 40.96%#ABC 68 29 21 14 4%ABC 40.96% 93.55% 53.85% 38.89% 6.67% 0%
Week'3'to'Week'8''Math'107'SPRING'2014
College Algebra Data Spring 2014
Warning'levelWeek'3'Totals
Green'at'week'8
Yellow'at'week'8
Orange'at'week'8
Red'at'week'8
Withdrew'by'week'12
#ABC %ABC
Green'at'wk'3 177 116 41 15 5 3 141 79.66%Yellow'at'wk'3 79 15 31 21 11 8 38 48.10%Orange'at'wk'3 55 1 14 19 20 10 16 29.09%Red'at'wk'3 87 5 6 8 59 33 13 14.94%Not'on'Wk'3'list 3 2 0 0 0 1 2 66.67%'Week'8'totals 401 139 92 63 95 54 210 52.37%#ABC 210 134 53 16 7 0%%ABC 52.37% 96.40% 57.61% 25.40% 7.31% 0%
Week'3'to'Week'8''Math'108'Spring'2014
PredicQve Value of Week 8 Grades
• Intermediate Algebra : Success rate of Week 8 metric (C or be?er) Fall 2013 – Red 4/64 (6.25%) – Orange 4/37 (10.8%) – Yellow 45/78 (57.7%) – Green 133/147 (95.3%)
• College Algebra : Success rate of Week 8 metric (C or be?er) – Red 13/138 (9.3%) – Orange 29/94 (30.8%) – Yellow 70/114 (60.5%) – Green 283/298 (94.9%)
Markov Models of Student Performance
0.81 0.11
0.03 0.05
0.38
0.11 0.19
0.32
0.28
0.14
0.28
0.24
0.02
0.08
0.03
0.63
Markov Models of SIU College Algebra Fall 2013 – Week 8 to Final grade 0.95
0.05
0.39
0.61
0.31
0.69
0.09
0.91
Feature Vector Data Analysis • The structure of the performance data permits a fine grain analysis to
optimize student support resources.
• Example: Analyze the transition behavior of two “yellow” students in College Algebra in Spring 2014.
• Student 1 :
• .25*(68) + .25*(68) + .5*(80) = 17 + 17 + 40 = 74
• Final Grade:F
• Student 2 : • .25*(55) + .25*(98) + .5*(72) = 13.75 + 24.5 + 36 = 74. 25
• Final Grade: B
• Prediction? Which student succeeded?
MOTIVATION!
Markov Models of SIU College Algebra Fall 2013 – Week 8 to Final grade 0.95
0.05
0.39
0.61
0.31
0.69
0.09
0.91
• The EW data suggests that “geeng green” by week 8 is the pathway to success.
• Colleges of Science/Business pilot will target “yellow alert” students in Fall 14 – Goal: 50% “Yellow-‐to-‐Green” by week 8.
EW Pilot Fall 2014-‐COS/COB
Academic Affairs • Upper level Administra$on • Support and understanding of goal
• Faculty involvement • IntenQonal le?ers to students on effort • Reframe to posiQve • Invest in student Experience in classroom
Outreach Efforts
Student Affairs • College-‐level Reten$on Staff
• Devised protocol for Qmeline of intervenQons
• Set tracking methods to collaborate across departments
• Shared common data with key consQtuents at the university
• Followed up with feedback survey at the beginning of the spring semester
Outreach Efforts
• First-‐year advisors • Contacted the students via
phone, email • Tracked responses in EAS
• Housing staff • RA involvement with study
sessions
[ ] [ ]14499118299
63.24.08.03.28.28.3.14.11.19.38.32.03.05.11.81.
15872132298 =
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
∗
Actual Week 3 distribuQon (Fall 2014) [ Gr Y Or R]
Predicted Week 8 distribuQon [ Gr Y Or R]
Week 3 to 8 transiQon matrix From Fall 2013
Using the transiQon matrix to make predicQons
0.38→0.425
0.11→0.06 0.19→0.09
0.32→0.425 Increased the students in the top two categories from 70% to 85% . An increase of 21.5% in one semester!
Markov Models of SIU College Algebra Fall 2014 Pilot
0.94
0.06
0.31
0.69
0.29
0.71
0.08
0.92
Markov Models of SIU College Algebra Fall 2014 – Week 8 to Final grade
Warning levelWeek 3 Totals
Green at week 8
Yellow at week 8
Orange at week 8
Red at week 8
Withdrew #ABC %ABC
Green at wk 3 297 256 22 9 10 0 271 91.2%Yellow at wk 3 132 57 49 16 10 0 95 72.0%Orange at wk 3 68 17 23 15 13 0 33 48.5%Red at wk 3 132 7 21 24 76 4 26 19.7%Not on Wk 3 list 0 0 0 0 0 0 0 0.0% Week 8 totals 629 337 115 64 109 4 425 67.6%%ABC 67.57% 94.10% 69.57% 29.70% 8.25% 0%
Week 3 to Week 8 Math 108 FALL 2014
College Algebra Data Fall 2014
[ ] [ ]14499118299
63.24.08.03.28.28.3.14.11.19.38.32.03.05.11.81.
15872132298 =
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
∗
Actual Week 3 distribuQon (Fall 2014) [ Gr Y Or R]
Predicted Week 8 distribuQon [ Gr Y Or R]
Week 3 to 8 transiQon matrix From Fall 2013
[ 340 121 70 129] Actual Week 8 distribuQon
(Fall 2014)
Using the transiQon matrix to make predicQons
RecommendaQons
• Campus-‐wide direc$on and communica$on • Success because of so many partnerships (Advising Council, math department, EAS, retenQon staff, Deans, Chairs, and student affairs staff
• Iden$fy the math course “needs” of your campus • What are DWF rates? • Demographic of students? • Does “remedial” math work? • Major-‐specific math courses • Prerequisite courses • Tenured vs. adjunct faculty
RecommendaQons
• Seek data as support for curriculum changes • NaQonal trends • Campus advising • InsQtuQonal data on course failures, drop, and
repeats • Department collabora$on • Assess what departments are already doing early
warning • Set protocol for outreach and Qmeline • Frequent and consistent meeQngs
RecommendaQons
• Select pla_orm that supports your students and faculty pedagogy • One system that a course coordinator oversees • User-‐friendly and pulls the data into manageable informaQon
• Assess your outreach efforts • Target what worked • Feedback from students • Reframe to posiQve – moQvaQon is already there • Track tutoring and instructor office hours • Early intervenQon means students find the correct major!
