Assessing SAGES with NSSE Assessing SAGES with NSSE datadataOffice of Institutional ResearchSeptember 25th, 2007

IntroductionIntroductionToday’s talk presents an analysis of

the impact of SAGES participation on a number of NSSE survey items

Focus on first-year responses only as no full-implementation classes have graduated

Report will first focus on NSSE benchmark scales followed by a discussion of individual NSSE items related to SAGES learning goals

AnalysisAnalysisThe seven years of NSSE data collected

by Case has been divided into four groups: ◦ Pre-SAGES (2000, 2001)◦ Students in the SAGES pilot (2002-2004)◦ Students not in the SAGES pilot (2002-2004)◦ Full-Implementation Classes (2005-2006)

An analysis of covariance (ANCOVA) was conducted to determine group differences on NSSE benchmarks and survey items

All analyses control for first-year reported major

NSSE BenchmarksNSSE Benchmarks In order to condense NSSE’s 80+ survey

items into easily discussed and analyzed scales, NSSE has developed five conceptually and statistically valid “benchmark” scales.

Scales include “Active and Collaborative Learning,” “Academic Challenge,” “Student-Faculty Interactions,” and “Supportive Campus Environment.”

In 2005 NSSE changed the calculation of the fifth benchmark, “Enriching Educational Experiences,” making pre-2005 longitudinal comparisons of this scale unreliable.

Active and Collaborative Active and Collaborative LearningLearningMeasures the extent to which

students engage in classroom activities and collaborate with others to solve problems.

Items include: asked questions in class, made class presentations, worked with other students on projects during class, worked with other students outside of class.

Active and Collaborative Learning

• Results indicated that those in the SAGES pilot had significantly higher scores than all other students.

• Additionally, those in the full-implementation of SAGES had, on average, significantly higher scores than pre-SAGES students.

Active and Collaborative Learning

• Results indicated that those in the SAGES pilot had significantly higher scores than all other students.

• Additionally, those in the full-implementation of SAGES had significantly higher scores than pre-SAGES students.

Academic ChallengeAcademic ChallengeMeasures the extent to which

students exert—and institutions demand—academic effort.

Items include: time spent preparing for class, number of assigned textbooks, and the extent to which the campus environment is perceived to emphasize academics.

Academic Challenge

• Results revealed no group differences on the Academic Challenge measure

Student-Faculty Student-Faculty InteractionInteractionMeasures the extent to which

students learn and solve problems by interacting with faculty members.

Items include: the extent to which students discussed grades with faculty, worked with faculty members on activities other than coursework, and discussed career plans with faculty.

Student-Faculty Interaction

• Results indicated that those in the SAGES pilot had significantly higher scores than those in the full-implementation class

• There were no significant differences among the remaining three groups

Student-Faculty Interaction

• Results indicated that those in the SAGES pilot had significantly higher scores than those in the full-implementation classes

• There were no significant differences among the remaining three groups

Supportive Campus Supportive Campus EnvironmentEnvironmentMeasures the extent to which students

believe that the institution is committed to their success and cultivates positive relations among different groups on campus.

Items include: quality of relationships with faculty, quality of relationships with peers, and the extent to which the campus environment is perceived to provide support to succeed socially and academically.

Supportive Campus Environment

• Results indicated that those in the SAGES pilot had significantly higher scores than pre-SAGES students and students in the full-implementation classes

• There were no significant differences among the remaining three groups

Supportive Campus Environment

• Results indicated that those in the SAGES pilot had significantly higher scores than pre-SAGES students and students in the full-implementation classes

• There were no significant differences among the remaining three groups

Individual ItemsIndividual ItemsSAGES goals focus on classroom

participation, developing writing and speaking skills, and academic advising.

The following slides examine 6 individual NSSE items relevant to these goals:◦ Asked questions in class◦ Gave a class presentation◦ My experience at Case has contributed to my

ability to write clearly and effectively◦ My experience at Case has contributed to my

ability to speak clearly and effectively◦ Satisfaction with University administration◦ Satisfaction with academic advising

Individual ItemsIndividual Items

• For both of these items, those in the SAGES pilot outperformed all other groups; however, students in the full-implementation classes had significantly higher scores than pre-SAGES students.

Individual ItemsIndividual Items

• For both of these items, there were no differences between those in the SAGES pilot and those in the full-implementation classes.

• Those in the SAGES pilot and the full-implementation classes outperformed the other two groups.

Individual ItemsIndividual Items

• For “Satisfaction with Advising” there were no year-to-year differences.

• For “Satisfaction with Administration,” those in the SAGES pilot had significantly higher scores than all other groups. Those in the full-implementation classes had significantly lower scores than all other groups

Conclusions – BenchmarksConclusions – BenchmarksFor three of the four benchmark

scales, students in the SAGES pilot significantly outperformed at least one other group.

Of the four benchmark scales, only one—Active and Collaborative Learning—significantly increased from pre-SAGES to the full-implementation class.

Conclusions – Individual Conclusions – Individual itemsitemsStudents in the full-implementation

classes scored significantly higher than pre-SAGES students on a number of items:◦ Asked questions in class◦ Gave a class presentation◦ My experience at Case has contributed to

my ability to write clearly and effectively◦ My experience at Case has contributed to

my ability to speak clearly and effectively

There were no group differences on ratings of satisfaction with advising

There was a significant drop in ratings of satisfaction with administration from pre-SAGES to the full-implementation classes.

LimitationsLimitationsNSSE is a survey of the entire

experience at Case, not an assessment of SAGES.

These analyses statistically control for first-year expected major only.

Significant differences from pre- to post-full-implementation of SAGES can be inferred to be due in part to the change in curriculum, but may also be due to unmeasured—or un-measurable—influences.

Thank You!Thank You!Questions? Concerns?

Contact: Tom Geaghantrg9@case.edu368-1500