24
Assessment & Evaluation NSF CISE REU Site PI Meeting February, 2012
Assessment & Evaluation
NSF CISE REU Site PI MeetingFebruary, 2012
Project Supporters2009 Working Group Members
Guy Alain AmousouChris Aberson Wendy Cooper Teresa Dahlberg Andy Fagg Stephen Gilbert Manfred Huber Niels Lobo
Sanjay Madrias Joan Peckham Eric Wong Yu-Dong YoaKevin Zeng
2010 Implementation MembersBrooklyn College, CUNY Iowa State University University of Alabama University of Massachusetts AmherstColorado Springs Jackson State University University of Central Florida UNC CharlotteDakota State University Louisiana State University University of Texas Austin University of South Carolina Depaul University Montclaire State University University of Houston University of Wisconsin Oshkosh Depauw University Marshall University University of Illinois Urbana-ChampaignHope College Oklahoma University
2010 Participants: 13 Common Application20 CISE REU Pre-Post Survey
NSF CISE REU PI Meeting, Monterey, February 2012
Project Supporters2011 Implementation MembersAuburn University University of Alabama University of South FloridaDakota State University of California- Berkley University of Texas AustinIowa State University University of Central Arkansas University of Wisconsin OshkoshLouisiana State University of Central Florida Virginia TechMarquette University of Houston Washington State UniversityMarshall University University of Illinois at Urbana-Champaign Rutgers UniversityMontclaire University University of Massachusetts Amherst University of Texas San AntonioNorth Dakota State University University of Missouri University of MarylandTexas State San Marcos UNC Charlotte
2011 Participants: 20 Common Application18 CISE REU Pre-Post Survey
NSF CISE REU PI Meeting, Monterey, February 2012
Assessment Project • Common Application • a la carte Survey • Evaluation Toolkit
– Coitweb.uncc.edu/reu/toolkit
• Future Directions– Toolkit and Survey Refinements– Continued Longitudinal Follow up– Explore Common Application tools
NSF CISE REU PI Meeting, Monterey, February 2012
Common Application2010 (13 sites)Most applied to 1 siteApplication Range: 29-152Average Number of Applications per Site: 77~ 130 accepted = 18% approximate acceptance rate
Unique Applicants Applied to 1+Sites
697
309
2011 (19 sites) (only 14 with apps)Most applied to 1 siteApplication Range: 4-176Average Number of Applications per Site: 79
NSF CISE REU PI Meeting, Monterey, February 2012
Unique Applicant Gender
72%
27% 1%
Males 500
Females 187
Didn't Specify 10
2010 72%
26% 2%
Males 615
Females 220
Didn't Specify 16
2011
Unique Applicant Ethnicity
10%18%7%
1%
61%
2%1%
Asian 68
African American 124
Hispanic/Latino 52
Pacific Islander 4
Caucasian 426
Native American 14
Multi-ethnic/Other 9
2010
14%16%
10%
1%52%
1%
7%
Asian 117
African American 133
Hispanic/Latino 81
Pacific Islander 4
Caucasian 440
Native American 7
Multi-Ethnic/Other 63
2011
Applicant Level in School12%
28%
42%
18%
Freshman 86
Sophomore 191
Junior 294
Senior 123
2010
4%16%
22%
8%
50%
Freshman 64
Sophomore 264
Junior 273
Senior 142
2011
Degrees Considering
69%
26%
0% 4%
Master's Programs 479
PhD Programs 180
Undecided 3
Not Considering Grad Program 5
2010
66%
28%
1% 5%
Master's Programs 552
PhD Programs 235
Undecided 9
Not Considering Grad Program 45
2011
Post-Graduation Plans - Industry36%
24%6%
2%
33%
Agree 157
Strongly Agree 104
Disagree 25
Strongly Disagree 9
Uncertain 146
2010
33%
27%5%1%
34%
Agree 281Strongly Agree 228Disagree 45Strongly Disagree 6Uncertain 287
2011
Post-Graduation Plans - Academia24%
13%
11%2%
50%
Agree 104
Strongly Agree 57
Disagree 49
Strongly Disagree 9
Uncertain 222
2010
24%
15%12%1%
48%
Agree 201
Strongly Agree 128
Disagree 100
Strongly Disagree 11
Uncertain 407
2011
First Person in Family to Attend College
80%
19%1%
No 679
Yes 165
No selection 5
2011
81%
19%
<1%
No 356Yes 85No selection 1
2010
Immediate Family Involved in Research
86%
14%<1%
No 380Yes 61No selection 1
2010
85%
14%1%
No 725
Yes 117
No selection 7
2011
CISE REU Pre-Post Survey• Pre survey: May-June and Post survey: July-August• Total N: 339 REU Students; 20 Sites in 2010 + 18 in 2011
What is measured:• Self Efficacy
– I can formulate a research problem• Intent to attend graduate school
– I plan to apply to graduate school in a computing discipline• Attitudes towards computing
– I like to use computer science to solve problems• Help seeking/coping skills
– When I do poorly on an exam, typically I….skip class
Positive Impact has been demonstrated
NSF CISE REU PI Meeting, Monterey, February 2012
Methodology• Items
– 4 point Likert type scale, 4 being positive– Some items were reverse scored– Collapsed into construct means representing 4 variables– Ethnicity collapsed into URM status
• Reliability– Coefficient alphas above .547
• MANOVA – To test hypothesis that there would be differences between means
based on time, gender, URM status
NSF CISE REU PI Meeting, Monterey, February 2012
Survey Participant ProfilesPre Demographics (N=219)
NSF CISE REU PI Meeting, Monterey, February 2012
Male68%
Female32%
Gender
9%
10%
14%
1%58%
2% 3% 3%
EthnicityAsian African AmericanHispanic/Latino Native Hawaiian/Pacific IslanderCaucasian Native AmericanPrefer not to specify Multi-ethnic
14% (28) reported having had 1 prior REU
1%
6%
33%
61%
Level in school Level in school in Fall
freshman sophomore junior senior
Survey Participant ProfilesPost Demographics (N=160)
NSF CISE REU PI Meeting, Monterey, February 2012
5%
39%56%
Level in School in Fall 2011
sophomore junior senior
7%
10%
11%
2%
55%
4%
7%4%
Ethnicity at Post SurveyAsian African AmericanHispanic/Latino Native Hawaiian/Pacific IslanderCaucasian Native AmericanPrefer not to specify Multi-ethnic
59%
37%
4%
Gender Male FemalePrefer not to specify
Combined Outcomes 2010+2011• 2010
– Increases at Post Assessment were not significant– No main effects, no interactions
• 2011– Significant increase at Post for Self-Efficacy
NSF CISE REU PI Meeting, Monterey, February 2012
Outcomes2010No significant increases
2011*Significant increase (p=.00)
Construct Time Mean SD
Self- Efficacy
pre 3.05 0.47
post 3.43 0.40
Intent pre 3.10 0.46
post 3.19 0.47
Attitude pre 3.59 0.32
post 3.65 0.29
Help Seeking- Coping
pre 2.94 0.33
post 2.98 0.37
Construct Time Mean SD
Self- Efficacy
pre 3.11 0.47
post 3.48* 0.39
Intent pre 3.22 0.59
post 3.23 0.66
Attitude pre 3.67 0.36
post 3.70 0.38
Help Seeking- Coping
pre 2.98 0.39
post 2.98 0.36
NSF CISE REU PI Meeting, Monterey, February 2012
Post Program EvaluationMajority of REU Students were satisfied with their programFaculty advisor 89%Housing 90%Program in general 90%Research experience 91%Interaction with staff 88%Interaction with students 90%
NSF CISE REU PI Meeting, Monterey, February 2012
What does this mean?We know there is positive impact from the REU
experienceIncreases between pre and post surveysSlightly higher interest in graduate school among
applicants that did participate to those who didn’tIndividual sites and students have powerful stories
The ‘surprise’ factor can negatively impact attitudes in the short term
Look at individual storiesConsider the learning outcomes
NSF CISE REU PI Meeting, Monterey, February 2012
Implications & Limitations• Effects are small given the
time frame• Effects may be delayed
• Self-report• Possible ceiling effect• Consider time series design
What else should we measure?•Mentoring•Ethics•Knowledge gains
NSF CISE REU PI Meeting, Monterey, February 2012
Next StepsCommon Application• Track offers• Follow up with
applicants• Compare
accepted/declined on key indicators
Survey• Deeper analysis• Larger sample• More variables, or
different variables• Control groups?• Learning outcomes?
NSF CISE REU PI Meeting, Monterey, February 2012
Discussion
• Collaborations for Research & Writing Circles• Case Studies• Develop and pilot new survey modules (i.e. ethics)
CISE REU Evaluation Toolkithttp://www.coitweb.uncc.edu/reu/toolkit/
NSF CISE REU PI Meeting, Monterey, February 2012
