1
Neal Schmitt Michigan State University
Presented at College Board, ETS, AERA Conference
December 10, 2010
Combining Cognitive and Non-Cognitive Measures: Expanding the Domain of
College Performance and its Prediction
2
Acknowledgements Jessica Fandre Tim Pleskac Alyssa Friede Abigail Quinn Michael Gillespie Lauren Ramsay Anna Imus Smriti Shivpuri Brian Kim Ruchi Sinha Stephanie Merritt Tae-Young Yoo Fred Oswald Mark Zorzie Matt Reeder Juliya Golubovich College Board (Wayne Camara and Krista
Mattern)
3
Outline History and Background “Job analysis” or conceptualization Instrumentation (description of noncognitive
measures used and outcomes examined) Validity data Subgroup Differences and Implications Faking issues Acceptability Research (profiles, fit, goal orientation, dif) Limitations and future research
4
Developing Alternative Measures
of Student PotentialMotivation for our work Broaden the scope of student outcomes and
capabilities considered in college admissions.
Reduce adverse impact.
Test the feasibility of developing “noncognitive” measures that are
valid practical in terms of time and effort required to assess less susceptible to faking
5
Identify a broader domain of college student performance:
Review university mission statements and department objectives
Interview with university staff responsible for student life at Michigan State University
Review of the education literature on student outcomes
Our systematic search resulted in 12 dimensions of student performance…
Developing Alternative Measures of Student Potential (Oswald et al, 2004,
JAP)
6
1. Knowledge and mastery of general principles
2. Continuous learning, intellectual interest and curiosity
3. Artistic and cultural appreciation
4. Appreciation for diversity
5. Leadership
6. Interpersonal skills
7. Social responsibility and citizenship
8. Physical and psychological health
9. Career orientation
10. Adaptability and life skills
11. Perseverance
12. Ethics and integrity
Dimensions of College Student
Performanceintellectual
interpersonal
intrapersonal
7
Two “Noncognitive” Measures
1. Situational judgment inventory A situation is presented along with several
alternative courses of action. The respondent is asked to indicate what she/he
would be most likely and least likely to do.
2. Biodata Short, multiple choice reports of
past experience/background and/or interests/preferences.
8
Situational Judgment Inventory (SJI)
Developed situational judgment items for each of the 12 performance dimensions
Student generated critical incidents (CIs) for each dimension
Translated CIs to stems for each item
Other students generated solutions to these questions
Researchers edited the options
Re-sorting back into 12 dimensions
3 Answer Keys: (see Motowidlo, Dunnette, & Carter, 1990) Expert student (junior and senior students) scoring Resident Advisor scoring African American key
9
Sample SJI Item for Leadership
You are assigned to a group to work on a particular project. When you sit down together as a group, no one says anything.
a) -1 Look at them until someone eventually says something
b) Start the conversation yourself by introducing yourself
c) +1 Get to know everyone first and see what they are thinking about the project to make sure the project’s goals are clear to everyone
d) Try to start working on the project by asking everyone’s opinion about the nature of the project
e) You would take the leadership role by assigning people to do things or ask questions to get things rolling
10
Sample SJI Item forInterpersonal Skills
You and some other students in your dorm area feel that a small group of students are highly disruptive during times when you would like to study or sleep. What would you do?
a) Talk to the resident assistant about it, as that is one of the responsibilities of their job.
b) -1 That’s part of life in the dorms. Let it go.c) Bring it up at the next floor meeting.d) +1 Politely talk to the disruptive students and ask them to
be more considerate. If the problem persists, talk to the resident assistant.
e) Wear earplugs or headphones when necessary.f) Develop and implement appropriate rules to address the
problem.
Sample Item for Knowledge
You decided early in the term to do a paper on a topic very interesting to you. However, you have found it difficult to find information on your topic, your job has taken more time than you wanted, and you have had more work in your other courses than you anticipated. Now it seems like you may have to engage in several "all-nighters" to complete your paper on time. What would you do?
a. Seek help from other students who may have had a similar experience. b. (-1)Pick a topic that can be completed quicker. An “A” is an “A”. c. (1)Set up a schedule on which you can complete all of the other work
you need to do, spend as much time on the paper as possible, and meet with the instructor to discuss what you have so far and get suggestions.
d. Do whatever it takes to complete the paper, including “all-nighters”. e. Talk to the instructor about the situation and ask for advice. f. Make the paper a priority, but take into account how much the paper is
worth in the class.
11
12
Biodata MeasureDeveloped biodata items for each dimension
Reviewed biodata item pools and adapted items related to each major performance dimension.
Resorted items into dimensions. Asked a pilot sample to respond to open-ended
versions of quantitative response options to determine appropriate scale anchors.
We used a rational scoring approach to these items, but also developed empirical keys against 3 criteria
13
Sample Biodata Items for Leadership
1. The number of high school clubs and organized activities (such as band, sports, newspapers, etc.) in which I took a leadership role was:
a) 4 or moreb) 3c) 2d) 1e) I did not take a leadership role
2. How often do you talk your friends into doing what you want to do during the evening?
a) most of the timeb) sometimes (about half the time)c) occasionally (about as often as others in my groupd) seldom or infrequentlye) never
14
Sample Biodata Items for Multicultural Appreciation
1. How often have you participated in social service or charity organizations?
a) Four or more timesb) Three timesc) Two timesd) Oncee) Never
2. If given a choice at a restaurant, would you order any food with which you are unfamiliar?
a) Never, I would always order foods that I know and enjoyb) Sometimes I might try a new food if someone else ordered itc) Occasionally I will order something new provided I can also
order familiar food at the same timed) If given a chance, I will always order a new food and try it
15
Sample Biodata Items for Social Responsibility
1. In the past year how many times have you considered the environment when purchasing a product (for example hairspray, or a car?)
a) Neverb) Oncec) Twiced) Three or four timese) Five times or more
2. How often do you work with not for profit groups?a) Neverb) Not very oftenc) Sometimesd) Oftene) Always
16
Outcomes Examined Self Ratings on behaviorally anchored
rating scales built around the 12 dimensions
Self rated class attendance University archives (grades) Organizational citizenship behavior Deviance Continuation in school and graduation
17
Interpersonal skills—Self Rated Performance Outcome 1 2 3 4 5 6 7 | | | | | | | Before you make your rating, please read these two examples: Example 1 Your roommate, usually a tidy person, has recently experienced some personal difficulties . As a result, the roommate has become quite distracted and has left much of the household responsibilities to you. You have talked to the roommate about your concerns, and empathetically requested that the roommate resume his/her share of the responsibilities as soon as possible. A month passes and you are still doing too much of the roommate’s work. What do you expect you would do? Example 2 You have been standing in line for the restroom for some time after a campus event, and someone cut s into the line ahead of you. What do you expect you would do?
Unsatisfactory Fulfills
Expectations Exceptional
Exceptional
You talk with the roommate again and explain that you are suffering as a result of the roommate’s behavior. You attempt to come up with a mutually acceptable plan of action.
Unsatisfactory
You ask to change rooms.
Unsatisfactory
You comment loudly to someone nearby how rude it is that people cut in line.
Fulfills Expectations
You tell the person that there is a line.
Exceptional
You calmly and politely inform the person that there is a line and ask that they move to the back.
Fulfills Expectations
You do his/her share of the work, and put anything of the roommate’s that affects you in the roommate’s area of the room.
Definition: Communicating and dealing well with others, whether in informal social situations or more formal school-related situations. Being aware of the social dynamics of a situation and responding appropriately.
18
Validity Data: College GPA 2001-02 2003-04 2004-8 2009-10 (N=614) (N=568) (N>1900) (N>550)
Knowledge .22 .22 .26 .22 Learning .05 -.02 .13 .10 Art Appr. .01 -.03 .18 .16 Mltcult.Appr. .07 -.04 .11 .12 Lead .14 -.01 .09 .07 Int. Skl. .04 Soc. Resp. .08 .07 .13 .08 Health .23 .14 .11 .04 Cr. Ornt. -.02 -.06 -.14 -.11 Adapt .21 .13 .05 .01 Perser .15 .07 .07 .10 Ethics .14 .22 .17 .13 SJI .16 .11 .22 .09 HSGPA .39 .53 .29 ACT/SAT .33 .34 .53 .44
19
Validity Data: Class Absences
2001-02 2005-06 2008 2009-10 (N>630) (N>900) (N >556) (N>600)
Knowledge -.18 -.15 -.14 -.03 Learning .00 -.07 -.06 .04 Art Appr. .07 -.03 -.08 -.10 Mltcult.Appr. .02 -.04 -.10 -.06 Lead -.03 -.06 -.02 -.04 Int. Skl. -.09 Soc. Resp. -.09 -.08 -.08 -.11 Health -.23 -.17 -.12 -.05 Cr. Ornt. -.06 -.05 -.08 -.05 Adapt -.15 -.10 -.10 .07 Perser -.20 -.18 -.16 -.01 Ethics -.31 -.17 -.24 -.11 SJI -.27 -.16 -.17 -.14 HSGPA -.04 -.01 -.02 ACT/SAT .11 .17 .14 .07
20
Validity Data: Self Report-BARS
2001-02 2003-04 2008 2009-10(N=614) (N=568) (N>547) (N>600)
Knowledge .33 .46 .21 .32 Learning .27 .37 .24 .39 Art Appr. .51 .33 .20 .33 Mltcult.Appr. .38 .34 .29 .30 Lead .43 .34 .29 .38 Int. Skl. .15 Soc. Resp. .35 .32 .26 .30 Health .28 .16 .26 .29 Cr. Ornt. .22 .36 .22 .31 Adapt .24 .26 .28 .30 Perser .34 .46 .36 .37 Ethics .11 .39 .23 .22 SJI .53 .54 .23 .25 HSGPA .08 .08 .02 ACT/SAT -.01 -.01 .01 .08
21
Incremental Validity: College GPA
01-02 03-04 08 09-10Step 1: Act/SAT, HSGPA ∆R2 .103 .179 .398 .200Step 2: Biodata, SJI ∆R2 .089 .070 .029 .069 Adjusted R .438 .499 .419 .480N 610 331 1155 296
Noncog. Vars. Significant Know Know Lrng. Cr. Ornt. Lrng. Lrng. Health Know Health Ethics Cr.Ornt. Adapt SJI SJI
22
Incremental Validity: Absenteeism and BARS
05-06 08 follow-upAbsenteeism BARS Absenteeism BARS
Step 1: Act/SAT, HSGPA ∆R2 .028 .019 .033 .008Step 2: Biodata,
SJI ∆R2 .060 .215 .116 .240
Adjusted R .297 .484 .386 .497N 800 801 556 547Noncog. Vars. Health Health Lead Multic
Ethics Ethics Health Health SJI SJI Ethics Persv Adapt SJT Ethics
Cr.Ornt.Persev.
23
Subgroup Differences in Standardized Units (04sample)
Male-Female Cauc-Afr.Am. Cauc-Hisp.Am.SJT -.15 .13 .10Know .00 .36 .23 Cont.Lrn. .10 .05 .00Artistic -.17 .31 .01Multi.Appr. -.16 -.08 -.46Leader -.15 .14 .10Respons. -.25 .16 .01 Health .52 .44 .30Car.Ornt. -.17 -.55 -.08Adapt. .00 .11 .16Persev. -.23 -.19 .00Ethics -.19 .22 .29HSGPA -.04 -.61 .42SAT/ACT .43 -1.46 1.01
24
Percent of subgroups admitted under various strategies
Hispanic Asian African Caucasian
Cog Cog+ Cog Cog+ Cog Cog+ Cog Cog+
Top 15% 4.3 6.4 17.8 14.9 .9 4.1 77.0 74.6
Top 50% 4.0 4.6 10.5 10.1 8.3 10.0 77.1 75.3
Top 85% 4.5 4.7 7.6 7.7 18.4 18.7 69.5 69.0
All 3.7 9.0 19.4 67.8
Cog=equally weighted composite of HSGPA and SAT/ACT
Cog+=equally weighted composite of HSGPA, SAT/ACT, and Non-cognitive measures.
25
Average Cumulative GPA for Subgroup Members who graduated at various levels of selectivity Hispanic Asian African Caucasian N Cog N Cog+ N Cog N Cog+ N Cog N Cog+ N Cog N Cog+ Top15 4 3.67 6 3.57 23 3.88 18 3.84 0 1 3.66 208 3.88 197 3.87 Top50 17 3.51 17 3 .42 55 3.65 54 3.63 15 3.46 15 3.38 651 3.66 638 3.66
Top85 23 3.44 22 3.47 64 3.62 64 3.62 49 3.05 47 3.08 929 3.52 905 3.54
26
Proportion of Subgroups Graduating in Four Years under Different Levels of Selectivity
Hispanic Asian African Caucasian
Cog Cog+ Cog Cog+ Cog Cog+ Cog Cog+Top15 93 90 89 91 100 92 72 70Top50 79 78 72 75 87 87 59 60Top85 77 76 68 68 79 78 51 52
27
Admits versus Applicants: Standardized mean differences (d)
2004 2006 Knowledge .55 .54 Continuous Learning .43 .49 Artistic Appreciation .35 .30 Multicultural Appreciation .48 .40 Leadership .43 .35 Social Responsibility .46 .46 Health .26 .30 Career Orientation .25 .21 Adaptability .22 .25 Perseverance .39 .32 Ethics .62 .53 Situational Judgment .50 .52
28
Conclusions on Faking Research (represents summary of a series of studies Applicants score higher than current students
(d=.2 to .6) Coaching has a significant impact on the degree
of score inflation Elaboration can minimize score inflation, but…
feasibility is an issue and its effects do not appear to generalize to nonelaborated items
Elaboration has no impact on validity Warnings do not appear to have much effect,
but…generalizability to an applicant situation has not been evaluated.
Are they any less fakable than essays or less inflated than letters of recommendation???
29
How often in the past year have you programmed in AJMR?
never once twice three or four times five times or more
How often, in the past three years, have you operated a rhetaguard?
never once twice three or four times five times or more
Sample Bogus Items: Carelessness
Profiling Subgroups of Students based on HSGPA, SAT/ACT and Noncognitive Variables
30
31
32
33
34
35
36
37
Outcomes and Implications There were predicted outcome differences (GPA,
absenteeism, satisfaction) across profile groups Highly motivated, career-oriented group is most
likely to respond to remedial efforts particularly if they relate to their career objectives
Marginal group without career objectives is a high risk group. Career counseling and intensive remediation may be necessary
Efforts to broaden the scope of interests of the high ability, culturally limited may be desirable in some universities
Students in the high ability well rounded group might be identified as potential student leaders and peer mentors/tutors.
38
Overall Conclusions We can develop valid noncognitive measures that
relate to GPA and other important student outcomes
Faking of the biodata and SJI remains a problem. Reactions to their use are not significantly
different than reactions to the ACT/SAT Subgroup differences are minimal and certainly
much less than those we find for cognitive ability measures
There may be useful other ways to employ these instruments; that is, to identify subgroups for whom interventions designed to retain them will be needed.
39
Thank you for your attention!