Predicting Success in Job Corps Students
What Variables Create a Completer?Chicago Job Corps Longitudinal Study
Alexander Adam Eschbach, Ph.D.Center Mental Health Consultant
Paul Simon Chicago Job Corps CenterOperated by Management and Training Corporation
FuturePrese
ntPast
Predicting SuccessPresentation Structure Inspecting the data first…
Sample size Gender, race Entry drug testing Educational characteristics Approach to psychological symptom
checklist
Past
Predicting SuccessPresentation Structure How we are using the data…PastPresent Future
Past
Present
Future
Predicting SuccessPresentation StructureHow we are using the data… Past
Examining trends in student characteristics
Identifying factors that appear to predict success
Past
Predicting SuccessPresentation StructureHow we are using the data… Present
Using data collected today to help students today
Interventionally relevant data (e.g., suicide risk assessment for students endorsing suicidal thoughts)
Present
Predicting SuccessPresentation StructureHow we are using the data… Future
Regularly using mathematical formula to identify “at-risk” students
Developing “tailor-made” interventions to increase student retention
Employing “evidence-based” methods to increase success
Future
Predicting SuccessPresentation StructureWarnings/caveats about using/interpreting the
data… Reflects data from Chicago Job Corps only Every center has its own unique geographical
location (urban vs. rural) Unique racial, ethnic background Differences in academic and vocational
programs Cannot assume data presented here will apply
to another Job Corps center Necessary to test this prediction model at
other centers
Past
Predicting SuccessAcknowledgements
Many thanks to… MTC Research Institute
Carl Nink, Director of the MTC Research Institute Rob Olding, Ph.D., Statistician and Dean of University of
Phoenix Management Training Corporation (MTC)
Christina Hunter , MTC Director of Operations Anita Sharp, MTC Vice President, Central Region
Chicago Job Corps Administration and Colleagues Bryan Mason, Center Director, Paul Simon Chicago Job Corps Gemma Ross, Health and Wellness Manager, Chicago Job Corps
National and Regional Mental Health Consultants Valerie Cherry, Ph.D., Lead National Mental Health Consultant Helena Mackenzie, Ph.D., Region V Mental Health Consultant
Longitudinal Study
Data from nearly 3,000 CJC students
Separated from March 2004 – December 2009
Only students who completed SCL-90-R
Only students with complete data
Past
Longitudinal StudySCL-90-R Information 90-item checklist for physical, behavioral,
and psychological complaints Every new student completes SCL-90-R on
second day People with 6th grade reading level can
understand questions Literature that shows this checklist has been
used with various ethnic and racial groups “Face-valid,” “What you see is what you
get”
Past
Longitudinal StudyWhy look at this data? Problem with “revolving door” noticed
in the early 2000s 350 – 375 positions, requiring 1 – 2
years to “complete” Averaging 600 new students a year Huge attrition (loss of students) due to
Drug use Violence Resignation AWOL Medical problems
Past
Longitudinal StudyWhy look at this data? Huge cost to Department of Labor
secondary to “revolving door” Considerable stress on CJC staff
Much effort to orient new students Much disappointment when student
does not finish program Minimizing negative experiences for
students due to failure Ultimately identifying “at-risk”
students early to improve retention and chance for success
Past
Longitudinal StudyBeginning Assumptions(A Priori Hypotheses) Probably a combination of factors
would predict success Math, reading ability Prior drug use (“positive on entry”) Psychological issues (depression, etc.) Behavioral issues (self-discipline,
persistence) Age Gender
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
14
Longitudinal StudySample Information
1388; 48%1499; 52%
Gender
MaleFemale
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
15
Longitudinal StudySample Information
101; 4%359; 14%
522; 21%
443; 18%
376; 15%
267; 11%
169; 7%
115; 5%
102; 4% 33; 1%3; 0%1; 0%
Age at Separation
16 1718 1920 2122 2324 2526 27
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
16
Longitudinal StudySample Information
1845; 64%
713; 25%
288; 10% 24; 1%
Racial African-Amer-icanLatinoCaucasianOther
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
17
Longitudinal StudySample Information
602; 24%124; 5%
544; 22%1043; 42%
35; 1%93; 4% 27; 1% 16; 1%1; 0%
Separation Type
Disciplinary Medical AWOLCompleter Parental Withdrawal ResignationRegular Transfer AT Transfer Adminis. Separation
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
18
Longitudinal StudySample Information
599; 21%
2287; 79%
Drugs on Entry to Program
PositiveNegative
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
19
Longitudinal StudySample Information
351; 59%
248; 41%
Male vs. Female Positive on Entry
MaleFemale
Z = 5.78, p < 0.01 (tw0-tailed t-test). Females significantly lesslikely to enter the center with a “positive” on drug testing.
Past
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul
Simon MTC Chicago Job Corps 20
01020304050607080
Disciplinary, 50.8
Disc Viol Mand, 34.7
Disc Viol, 23.1Reg Trans,
22.2Disc Viol
Nonmand, 18.7AWOL, 15.6
Resignation, 15.1
Ordinary Separation,
12.4Med Sep w/ Reinstate, 9.4
Parent Withdraw,
8.6MSWR Final Close, 5.6
Fraudulent, 0
Positive on Entry % by Separation Type
Past
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul
Simon MTC Chicago Job Corps 21
Reading Math480490500510520530540550560
Ordinary SeparationDisciplinaryParent WithdrawMed Sep ReinMSWR FinalResignationAWOLDisc NonmandDisc MandatoryDisc Drugs
Past
TABE Scores by Separation Type
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
Weird, consistent trends from SCL-90-R data Reflected Chicago Job Corps students’
unique response pattern A pattern noticed before during dissertation
research A subset of male --- but sometimes female ---
students who selected zero or just a few SCL-90-R items were often found to be “positive on entry” for drug use. These were “Deniers” (students who selected 0 – 3 SCL-90 items) “There’s nothin’ wrong with me, but, oh, I
use weed a lot.” These students were often separated for
drug use or violence.
Past
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
Generally speaking, it appeared that Ordinary separation students (“Completers”) endorsed a moderate number of items as a group. They were called “Reporters” (Students who selected 4 - 37 SCL-90-R items)
Past
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
Students who endorsed many SCL-90-R items were often separated for medical or psychiatric reasons. These were called “Exaggerators” (Students who selected 38 – 90 SCL-90-R items.)
Past
Longitudinal StudyUnivariate Look at the Data(“One Thing at a Time”)
Using the SCL-90-R to help students in the “Here and Now.”
SCL-90-R “Top 10” Items Endorsed by Chicago Job Corps students…
Past
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
10. Trouble concentrating. 9. Awakening in the early
morning. 8. Pains in lower back. 7. Trouble remembering
things. 6. Having to check and
double-check what you do.
Past
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
5. Trouble falling asleep. 4. Feeling easily annoyed or
irritated. 3. Worrying too much about
things. 2. Headaches.
Past
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
1. Feeling that most people cannot be trusted.
Past
29
Item Content En-dorsed
%
Not bother
-ed
A little bit
Moder-ately
Quite a bit
Extreme-ly
Sample N
En-dorsed
1. Trust 64% 1033 751 367 345 384 2880 1847
2. Headache 61% 1123 995 387 291 84 2880 1757
3. Worry 54% 1331 780 341 226 205 2883 1552
4. Annoyed 52% 1377 804 321 231 142 2875 1498
5. Memory 48% 1479 905 237 198 60 2879 1400
6. Double-check 47% 1521 810 287 158 105 2881 1360
7. Sleep trouble 41% 1705 561 224 207 186 2883 1178
8. Trouble concentrating 39% 1764 675 222 125 98 2884 1120
9. Back pain 38% 1786 522 246 204 121 2879 1093
10. Early awakening 38% 1778 575 246 151 122 2872 1094
Top 10, Most Frequently Endorsed SCL-90-R Items
Chicago Job Corps Students 3/2004 – 12/2009
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul
Simon MTC Chicago Job Corps
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
30
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
Curious about the least selected items?
SCL-90-R “Least Popular” Items Endorsed by Chicago Job Corps students…
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
31
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
81. Shouting or throwing things.
82. Trembling. 83. Heavy feelings in your
arms or legs. 84. Having thoughts that are
not your own. 85. Feeling afraid to go out of
your house alone.
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
32
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
86. Thoughts of ending your life.
87. The idea that someone else can control your thoughts.
88. Feeling afraid you will faint in public.
89. A lump in your throat.
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
33
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
90. Hearing voices that others do not hear.Only 128/2883 (4%) of
students selected this item.
Past
34
Bottom 10, Least Frequently Endorsed SCL-90-R ItemsChicago Job Corps Students 3/2004 – 12/2009
Item Content En-dorsed %
Not bother
-ed
A little bit
Moder-ately
Quite a bit
Extreme-ly
Sample N
En-dorse
d
1. Shouting, throwing
10% 2592 202 42 27 19 2882 227
2. Trembling 10% 2591 197 43 28 16 2875 284
3. Heavy extremities
10% 2602 186 46 30 14 2878 276
4. Thoughts not your own 9% 2625 168 47 24 15 2879 254
5. Afraid to leave house 8% 2654 142 41 29 15 2881 227
6. Thoughts of ending life 7% 2670 140 41 17 18 2886 216
7. Thoughts controlled 7% 2680 118 33 26 18 2875 195
8. Afraid of fainting 7% 2680 122 39 21 12 2874 194
9. Lump in throat 6% 2714 106 31 16 16 2883 169
10. Hearing voices 4% 2755 81 26 9 12 2883 128
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul
Simon MTC Chicago Job Corps
Past
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
35
Longitudinal Study
Suicide Info (Center for Disease Control, Atlanta) In 2009, 13.8% of U.S. high school
students reported that they had seriously considered attempting suicide during the 12 months preceding the survey; 6.3% of students reported that they had actually attempted suicide one or more times during the same period.
Present
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
36
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
SCL-90-R Data in the “Here and Now” Students endorsing Item #15 on the SCL-90-R
(Thoughts of ending your life) are seen within 24 hours of arriving on center
Suicide Risk Assessment conducted by CMHC and/or Psychology Extern (Practicum Student)
Base rate of suicidal ideation reported on SCL-90-R lower at CJC than reported base rate by CDC for same age group (13.8% U.S. high school population; 216/2887 [7.0 %] CJC students endorsed Item #15)
Present
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps
37
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
SCL-90-R Data in the “Here and Now” In our sample, Item #15 (Thoughts of
ending your life) was ranked 86th out of 90 items.
One of the least frequently endorsed items.
When it was endorsed, more than 2/3 of the students reported they misread the item, made a mistake, or they were reporting a former problem and not current suicidal thinking.
Present
Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)
Summary of the Univariate Findings “Positive on Entry” is important TABE reading and math scores are
Important Comparing “Deniers” to “Exaggerators”
to “Reporters” is important Is gender important? Other variables to predict success?
Past
Future
Preliminary Conclusions
Mathematical model confirms our original scientific guess (hypothesis) that there are identifiable factors predict success
Past
Future
Next StepsWhere did we go from here? Further data analyses Use logistic regression to introduce different
variables that could predict “group membership” “Plug in” TABE scores, Positive on Entry, SCL-90-R
data, and other possible variables into a formula to calculate a “number”
The greater that “number” is more than 0 (zero), the greater the odds that the student will be in the Successful Completer group
If the calculated “number” is less than 0 (zero), the lower the number, the more likely that the student will be in the Unsuccessful/Noncompleter group
Future
Past
Next StepsWhere did we go from here? Various separation types divided into
Unsuccessful (Noncompleters), Successful (Completers), and Neutral outcomes. Completion, Ordinary Transfers = Successful Resignation, Disciplinary, AWOL = Unsuccessful Medical separation = Neutral
Use univariate observations to “plug into” multivariate (“Logistic Regression”) analysis
Concept is that outcomes are explained by the complex interaction of several variables
Future
Next StepsWhere did we go from here?Variable Coefficien
tApprox.
SEWald Chi-
Sq.P - value
Intercept -7.212000 0.67 116.3 < .001Pos. on Entry -1.183000 0.13 80.4 < .001Highest Grade .2477700 0.03 52.5 < .001TABE Math/100 .4427600 0.13 12.5 < .001SCL-90-R D, R, E
.2914200 0.09 10.0 0.002TABE Reading/100
.3003800 0.13 5.3 0.022Gender .2164100 0.11 4.2 0.04
Building the Logistic Regression (Mathematical) Model
Future
Next StepsWhere did we go from here?Building the Logistic Regression (Mathematical) Model Positive on entry Highest grade achieved Initial TABE math score Initial TABE reading score SCL-90-R category
Denier (0 – 3 items) Reporter (4 – 37 items) Exaggerator (38 – 90 items)
Gender
Future
44
Next StepsWhere did we go from here?Building the Logistic Regression
(Mathematical) Model Predicted Outcome
Observed Outcome
Noncompleters
Completers
Noncompleters
666 276 942
Completers 322 481 803
988 757 1745Correctly predicted percent = 65.73% ≈ 66%. False positives = 36.46%.False negatives = 32.59%. Area under the ROC curve, c = 0.7158. Sensitivity = 59.9%. Specificity = 70.7%. (Random samples of larger data yielded same results.)
Future
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul
Simon MTC Chicago Job Corps
Next StepsWhere did we go from here?Building the Logistic Regression (Mathematical) Model The Logistic Regression Model (the equation
[formula] produces a number. If we want to predict better than “chance,” we started by using 0.5 as the “cut-off” score. The greater than 0.5 that a student scores, the greater the odds that he or she will be a Successful Completer. If the student scores much less than 0.5, the greater the odds that he or she will be an Unsuccessful Noncompleter.
Future
The logistic function is useful because it can take as an input any value from negative infinity to positive infinity, whereas the output is confined to values between 0 and 1. The variable z represents the exposure to some set of independent (predictor) variables, while ƒ(z) represents the probability of a particular outcome, given that set of explanatory variables. The variable z is a measure of the total contribution of all the independent variables used in the model and is known as the logit.
The variable z is usually defined as:
48
So a student who was positive on entry (= 1), was male (= 1), had reached the 9th grade, received an Initial TABE Reading score of 490, an Initial TABE Math score of 507, and was in SCL-90-R Category 3 would have the following regression prediction equation beginning with the Intercept (mathematical constant):-7.212 + 1(-1.1830) + 1(.21641) + 9(.24777) + (490/100[.30038]) + (507/100[(.44276]) + 3(.29142) = -1.3577448. If -1.3577448 is plotted on the graph, it becomes apparent that the student would definitely have less than a 50-50 chance of completing the Job Corps program if we do not intervene and help him.
Future
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul
Simon MTC Chicago Job Corps
Z
Plotting Chance for SuccessMale Student Example ( -1.3577448
)Less Than 50-50 Chance for
Success
Z = -1.3577448
50-50 Chance
50
In contrast, a student who was negative on entry (= 0), was female (= 0), had reached the 12th grade, received an Initial TABE Reading score of 650, an Initial TABE Math score of 700, and was in SCL-90-R Category 2 would have the following regression prediction equation beginning with the Intercept (mathematical constant):
-7.212 + 0(-1.1830) + 0(.21641) + 12(.24777) + (650/100[.30038]) + (700/100[(.44276]) + 2(.29142) = +1.39587. If +1.39587 is plotted on the graph, it becomes immediately apparent that the student has considerably more than a 50-50 chance of being a successful completer from Job Corps.
Future
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul
Simon MTC Chicago Job Corps
Z
Plotting Chance for SuccessFemale Student Example
( +1.39587 )Greater Than 50-50 Chance for
Success
Z = + 1.39587
50-50 Chance
52
An extreme case of success would be a student who was negative on entry (= 0), was female (= 0), had reached the 14th grade, received an Initial TABE Reading score of 700, an Initial TABE Math score of 750, and was in SCL-90-R Category 2 would have the following regression prediction equation beginning with the Intercept (mathematical constant):-7.212 + 0(-1.1830) + 0(.21641) + 14(.24777) + (700/100[.30038]) + (750/100[(.44276]) + 2(.29142) = +2.26298. See the plot on the next page to see the marked increase in probability of success.
Future
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul
Simon MTC Chicago Job Corps
Z
Plotting Chance for SuccessExtreme Female Student Success Example ( + 2.26298 )
Over 90% Chance for Success
Z = + 2.26298
50-50 Chance
2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon Chicago Job Corps
54
An extreme case of non-success would be a student who was positive on entry (= 1), was male (= 1), had reached the only the 8th grade, received an Initial TABE Reading score of 400, an Initial TABE Math score of 380, and was in SCL-90-R Category 1 would have the following regression prediction equation beginning with the Intercept (mathematical constant):-7.212 + 1(-1.1830) + 1(.21641) + 8(.24777) + (400/100[.30038]) + (380/100[(.44276]) + 1(.29142) = - 3.021. See the plot on the next page to see the marked decrease in probability of success.
Future
Z
Plotting Chance for SuccessExtreme Male Non-success Example ( - 3.021002 )
Less Than 5% Chance for Success
Z = -3.021002
50-50 Chance
Next StepsHow will the data help our students? What does the data mean?
Is ability to be “Negative on Entry” a sign of self-discipline?
Is “Highest Grade Achieved” a sign of persistence?
Is “Initial TABE Math score” a sign of problem-solving skills?
These factors appear most predictive of success.
FutureFuture
Next StepsHow will the data help our students? Early prediction of group
membership helps staff intervene In consultation with the CMHC and
Wellness personnel, academic, vocational, and residential staff can work together to develop a plan to support/intervene with a student who presents the risk factors of students prone to leave before completion of the program.
FutureFuture