10/12/2010
1
Assessment of the First Year Assessment of the First Year ExperienceExperience-- Covering All the BasesCovering All the Bases
Dr. Mary Lou D’Allegro: Senior Director, Planning, Research & AssessmentPenn State BerksPenn State Berks
Dr. Tami Mysliwiec: Associate Professor, Biology & First Year Seminar CoordinatorPenn State Berks
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010 1
Objectives for PresentationObjectives for Presentation
(1) Identify the critical first year experiences that should be assessed,
(2) Identify the assessments that will aptly capture these critical first year experiences,
(3) Develop a framework in which the assessment data will be analyzed,
(4) Explore the implications of both the improvement of the first year experience and subsequently student success, and
2
(5) Introduce a multi-dimensional model that enables assessment professionals to ensure that all that should be considered is assessed.
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
10/12/2010
2
What this Presentation is NOTWhat this Presentation is NOT
Creative methods to assess the First Year Experience
Innovative use of existing technology or adaption of new faculty
New ways to engage faculty & staff
3
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
Admittedly,Admittedly,This Presentation is Just Plain Boring!
4
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
10/12/2010
3
BackgroundBackground
Penn State Berks is a college campus of the Pennsylvania State University (PSU) system
2800 students926 New Baccalaureate in fall 2008 741 New Baccalaureate in fall 2009
SAT-Critical Reading + SAT-Quantitative hovers around+/- 1000
30% of students are in residence halls Almost all are full-time
5
Require First Year Seminar (FYS) for all first time students
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
YouTube Video: http://www.youtube.com/watch?v=DaMX-QmdSE0
BackgroundBackground By Spring 2009, University required all campuses (19) to
have a First Year Experience (FYE) plan in place
Must address PSU goals for FYE (set in stone)• G l 1 d i l i d i h h h l l i f • Goal 1: to engage students in learning and orient them to the scholarly community from
the outset of their undergraduate studies in a way that will bridge to later experiences in their chosen majors, and
• Goal 2: to facilitate students’ adjustment to the high expectations, demanding workload, increased liberties, and other aspects of the transition to college life.
Objective 1: to introduce students to University study
Objective 2: to introduce students to Penn State as an academic community, including fields of study and areas of interest available to students
6
g y
Objective 3: to acquaint students with the learning tools and resources available
Objective 4: to provide an opportunity for students to develop relationships with full-time faculty and other students in an academic area of interest to them
Objective 5: to introduce students to their responsibilities as part of the University community.
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
10/12/2010
4
BackgroundBackgroundAssessment of First Year Engagement Plan
Semester
Assessment
Assessment Activity
When Administered
Who is Responsible?
Cohort
Purpose
Spring 2008
NSSE Survey Feb 08-Apr 08
SR PRA* First Year/ Seniors
Baseline
Summer 2009
Survey of FTCAP, Survey Student
Orientation Experience
Last week in June, July, 3rd
week in August
SR PRA, DR AA**, AS CL***
All new incoming students
Baseline
rdFall 2009 Survey of FYEP Learning Outcomes
3rd Week,12th Week
FYS Coordinator,FYS Faculty,
SR PRA,
First Year Seminar Students
Assess student gains in outcomes between
pre/postSpring 2010
Survey of FYEP Learning Outcomes
3rd Week, 12th Week
FYS Coordinator, FYS Faculty,
SR PRA
First Year Seminar Students
Assess student gains in outcomes between
pre/postSummer
2010 Survey of FTCAP,
Survey Student Orientation Experience
Make changes to FTCAP, Student
Orientation, and FYS based on results of
Fall 2009/Spring 2010 Surveys
AS CL, DR AA,
FYS Coordinator, Associate Dean
All new incoming students
Assess improvement of FTCAP and Student Orientation Experience with regards to FYEP
student outcomes
Fall 2010 Survey of FYEP Learning Outcomes
3rd Week, 12th Week
FYS Coordinator, FYS Faculty,
SR PRA
First Year Seminar Students
Assess student gains in outcomes between
pre/post
7
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
Spring 2011
NSSE Survey
Survey of FYEP Learning Outcomes
Feb 11-Apr 11
3rd Week, 12th Week
SR PRA
FYS Coordinator, FYS Faculty,
SR PRA
First Year/ Seniors
First Year Seminar Students
Ascertain gains in composite scores
between 2008-2011 scores for BOTH
Cohorts
Assess student gains in outcomes between
pre/postSummer
2011 Look to see if changes
to FYS improved student outcomes
FYS Coordinator, SR PRA,
Associate Dean
First Year Assessment ComponentsFirst Year Assessment Components
(0) Assessment of Placement Testing Experience (FTCAP),
(1) Assessment of the New Student Orientation,(1) Assessment of the New Student Orientation,
(2) Survey of the First Year Seminar,
(3) Student success indicators, and
(4) Examination of NSSE results for first year students.
8
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
10/12/2010
5
First Year Assessment ComponentsFirst Year Assessment ComponentsConceptual ModelConceptual Model
Orientation (1)
Placement testing
(0)
FYS (2)
Extent of College
Engagement (3)
( )
9
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
Student Success Indicators
(4) Ultrinstic, 2010
FTCAP SurveyFTCAP Survey
•Ask students & parents questions about FTCAP activities
•Results VERY positive so there has been no drill-down results to:
Orientation, FYS, NSSE, Student Success Indicators
10
• Comparison to previous FTCAP Survey administrations
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
10/12/2010
6
New Student Orientation SurveyNew Student Orientation Survey•Asks questions about orientationactivities
•Asks questions that connects the orientation activities to the FYE orientation activities to the FYE goals & objectives
• If students don’t respond to “connection” questions, revamp orientation
11
•Working on student learning (ex. Common Reading FYS score on Common Reading assessments)
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
New Student Orientation SurveyNew Student Orientation Survey
12
•Results focused around orientation activities
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
•Results focused around FYE goals
10/12/2010
7
New Student OrientationNew Student Orientation(New International Students Survey)(New International Students Survey)
13
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
Pre/Post Survey of the FYSPre/Post Survey of the FYS
• Asks questions about the FYSactivities How often engage in “good behaviors” How important good behaviors arep g Behavior categories
Classroom (ex. attentive in class)Academic (ex. think critically)
Personal (ex. network with other majors)
• Asks questions that connect theFYS and FYE goals & objectives
14
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
• If responses are poor on POST or minimal positive movement from PRE POST revise FYS
10/12/2010
8
NSSE Results for First Year StudentsNSSE Results for First Year StudentsStep 1: Look at five composites for First Year vs SeniorsStep 2: Look at (-) comparisons to three peer sets
(1) PSU college campuses(2) “Local Competitors”-regional area schools that Berks compete(3) “National Benchmark ” schools with better retention graduation rates than Berks
First Year (FY) vs. Seniors
(SR)First Year
(FY)Seniors
(SR)TOTAL
PLUS (+)
2003-2006-2008
NSSE 2008 Results
PSU
Peer Comparisons
National BenchmarkLocal Competitive Set
(3) National Benchmark- schools with better retention graduation rates than Berks
Step 3: Look at previous administrations
15
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
Composite FY FY SR FY SRLevel of Academic Challenge (LAC) - + - - - + - - 2
Active & Collaborative Learning (ACL) + + - + - + + - 6Student-Faculty Interaction (SFI) + + + + + + + - 8
Enriching Educational Experience (EEE) - - - - - + + - 2Supportive Campus Environment (SCE) + + - + - - - - 3
p
NSSE Results for First Year StudentsNSSE Results for First Year StudentsStep 4: Look at individual item effect sizes (with other PSU Campuses)
both (-) & (+) within composites that peers outperformed Berks
Composite Item #First Year Senior First Year Senior First Year Senior
Item Effect Size (ES)^ ES Significant?MeanFirst Year Senior First Year Senior First Year Senior
1l.
2.58 2.71 -0.08 -0.28 *
1u. 2.48 2.57 -0.07 -0.04
1v. 2.56 2.47 -0.13 -0.25 * *
7a. 0.10 0.64 0.25 0.01 **
7b. 0.25 0.48 -0.18 -0.28 *** *
7c. 0.11 0.26 -0.01 0.047e. 0.18 0.37 -0.23 -0.22 *** *
7f. 0.02 0.04 0.01 -0.31 ***7g. 0.04 0.21 0.03 0.08
Enrich
ing E
duca
tional E
xperie
nc
Foreign language coursework
Independent study or self-designed majorStudy abroad
Had serious conversations with students of a different race or ethnicity than your own
Had serious conversations with students who are very different from you in terms of their religious beliefs, political opinions, or personal values
Used an electronic medium (listserv, chat group, Internet, instant messaging, etc.) to discuss or complete an assignment
Practicum, internship, field experience, co-op experience, or clinical assignment
Community service or volunteer worka t c pate a ea g co u ty o so e ot eformal program where groups of students take two or more classes together
16
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
7h. 0.04 0.31 0.16 -0.07
9d. 2.13 1.53 -0.15 -0.56 * ***
10c. 2.79 2.45 0.11 0.07
ce (E
EE
)
Encouraging contact among students from different economic, social, and racial or ethnic backgrounds
Culminating senior experience (capstone course, senior project or thesis, comprehensive exam, etc.)
Participating in co-curricular activities (organizations, campus publications, student government, fraternity or sorority, intercollegiate or intramural sports, etc.)
10/12/2010
9
NSSE Results for First Year StudentsNSSE Results for First Year Students
Step 5: Should be looking at NSSE results by(1) Orientation participation(2) FYS type (Linked or not linked)( ) yp ( )
(3) FYS grade categories (A-C, D-F, W)
17
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
Student Success IndicatorsStudent Success Indicators
•Student Success IndicatorsFirst Semester Credits EnrolledFirst Semester Credits EarnedFirst Year Credits Earned First Year GPA One-year retention rateSix-year graduation rate
18
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
10/12/2010
10
Student Success IndicatorsStudent Success Indicators
• Over 40 cohorts identified by Enrollment Management Team & First Year Experience Committee
Group *Club/activities/recreation
*Commuters and off-campus housing students *Veterans *STEM *Intercollegiate athletics Ethnicity Gender Committee
• Also examine student success indicators (SSI) by: (1) Orientation participation(2) FYS type (Linked or not linked)(3) FYS grade categories
(A-C, D-F, W)
• Comparison of SSI of cohort
Gender Parent education level Non-traditional Learning Community Discovery Learning Center Aspire students ESL Apart from the Start Student “leaders” Remedial courses Gateway courses Honors students Hybrid classes 1st Year assessment Judicial issues Financial Aid
19
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
• Comparison of SSI of cohort to overall baccalaureate or associate cohort
Financial Aid Division/Majors Instate/Out of State Service area Alumni CHOA Clearing house “Crossing the Finish Line” Early Intervention Students
Student Success IndicatorsStudent Success IndicatorsStandard Report of Student Success Indicators for overall, & each of the eight that were completed this summer
Veterans Count Fall 2003 Level
1st Semester Credits Enrolled
1st Semester Credits Earned
1st Semester
GPAOne Year Retention
3 (Asc)/ 6 (Bac) Year Graduation
ALL Veterans
CountALL B h l 14 68 14 35 2 68 83 2% 63 6%ALL Bachelors 14.68 14.35 2.68 83.2% 63.6%ALL Associate 13.05 11.87 2.02 82.8% 12.6%
17 Veterans Bachelors 12.56 12.56 3.12 82.4% 82.4% 898 Veterans Associate 9.86 9.86 2.72 75.0% 37.5%
20
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
10/12/2010
11
RecommendationsRecommendations
Sometimes letting the 400 lb gorilla stay in the room is preferable to reporting assessment results
That said…
21
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
RecommendationsRecommendationsBefore you start the assessment• Have articulated goals or objectives or both in place
Assessment• Assess to those goals,
Assess the intangibles, “the means,” but
Do not loose site of the “ends”- student success
22
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
10/12/2010
12
RecommendationsRecommendationsData Analysis Considerations:• Disaggregate results by specific student groups
• Use comparative data if available (historic, peer institutions)p ( , p )
Use of Assessment• Be able to connect the “ends” to the “means” (slide 9)to make assessment results meaningful and valuable
23
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010
to make assessment results meaningful and valuable
• Use assessment of the “means” to identify milestones or activities that affirm student success or fall short
QuestionsQuestions
24
The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010