California State University First Year Experience Assessment Presenter: Joseph Pica, Ed.D. CEO...

transcript

California State UniversityFirst Year Experience

Assessment

Presenter:Joseph Pica, Ed.D.

CEO Educational Benchmarking (EBI)

June 25, 2004

Fundamentals of Structuring and

Implementing Successful First Year Seminars

The statistical analysis for this presentation was based on the Educational

Benchmarking (EBI) First Year Initiative (FYI) Assessment

and conducted by

Randy L. Swing, Ph.D.Co-Director, Policy Center on the First

Year of College &Fellow, The National Resource Center

on The First-Year Experience and Students in Transition

in conjunction with Educational Benchmarking

Value of a Collaborative

• Shared mission and purpose

• Common problems and barriers

• Varied expertise

• Benchmarks for performance

10%15%20%25%30%35%

California State UniversityFirst Year Experience Survey Question:What is the MOST important topic you would like to see

addressed at future First Year Experience Meetings?

California State UniversityFirst Year Experience Survey Question:What is the MOST important topic you would like to see

addressed at future First Year Experience Meetings?

Direct Quote:

“The elements that research indicates are the most important

to have as components of an effective first year program.”

Presentation Overview

• The elements that research indicates are the most important to have as components of an effective first year program.

• Principles for leveraging assessment to initiate and sustain improvement

FYS – Goals• Retention 37%• Academic Achievement 37%• Participation in College Life 11%

The Policy Center on the First College Year

• Ease transition to college• Increase “student skills”• Increase persistence rates• Increase graduation rates

First-Year Initiative Survey

FYI• Developed by the Policy Center on the First

College Year and Educational Benchmarking (EBI)

• 62 institutions (limited to 4-year institutions*)• Over 30,000 students• 7- point scale• Learning Outcome Factors• Administered in the last week of fall 2001

*4-year or 2-year regional campuses - a 2-year version is in development.

Issues Addresses byFYI Findings

Course Theme

Credit/Contact Hours

Required/ Not Required

Letter Graded/Pass-Fail

Linked Course Format

Undergraduate Teaching Assistants

FYI Assessment and Research

ContextThe following principles

and practices provide the foundation to the

research and the First Year Initiative Assessment

Research/Assessment Context

1.Factors: A more powerful indicator than individual question results.

2. Regression: Allows you to identify predictors of performance

3.Predictor Factors: Factors, if improved, have the greatest impact on improving overall effectiveness

Factors

• Factors (also called “constructs”) are groupings of related questions that share a relationship.

• The basic assumption of factor analysis is that underlying dimensions, or factors, can be used to explain more complex phenomena.

Factor MeansCourse Learning Outcomes mean

Course Improved Knowledge of Campus Services . . . . . 4.72

Course Improved Knowledge of Campus Policies. . . . . 4.58

Course Improved Connections with Peers. . . . . 4.54

Course Improved Connections with Faculty. . . . 4.50

Course Improved Managing Time/Priorities. . . . 4.50

Course Improved Critical Thinking. . . . 4.40

Course Improved Study Strategies. . . .4.19

Course Improved Knowledge of Wellness Issues. . . . 3.90

Course Improved Academic/Cognitive Skills. . . . 3.74

Course Increased Out-of-Class Engagement. . . .3.63 1.0 2.0 3.0 4.0 5.0 6.0 7.0

1/2 3/41/4

Factor Reliability

Once it has been determined that a certain set of questions

do share a relationship and therefore constitute a factor,

there is an additional (and necessary) statistical test to

assess the psychometric soundness of the factor.

• A Cronbach’s Alpha of zero would mean that there is no internal consistency at all, i.e., subjects are likely to respond with anything from 1 to 7 on any of the questions in a factor with no discernable pattern.

• An Alpha of 1 would mean that every subject answered every question comprising the factor consistently (e.g., all subjects answered with all 7’s, or all 1’s). This is a highly unlikely event.

Factor Reliability

FYI Factor Reliabilities Factor Descriptions ReliabilityCourse Improved Study Strategies 0.90Course Improved Academic and Cognitive Skills 0.89Course Improved Critical Thinking 0.91Course Improved Connections with Faculty 0.84Course Improved Connections with Peers 0.90Course Increased Out-of-Class Engagement 0.90Course Improved Knowledge of Campus Policies 0.90Course Improved Knowledge of Academic Services 0.87Course Improved Managing Time and Priorities 0.92Course Improved Knowledge of Wellness 0.90Sense of Belonging and Acceptance 0.90Usefulness of Course Readings 0.90Satisfaction with College/University 0.90Course Included Engaging Pedagogy 0.92Overall Course Effectiveness 0.92

Regression• A correlation establishes the

relationship between two variables.

• Regression analysis, by contrast, allows us to determine the relationship between some dependent variable (e.g. overall effectiveness) and multiple independent variables (e.g., engaging pedagogy, etc.).

• Major predictors identify, in order of descending importance, the factors that have the greatest impact on overall effectiveness.

• While minor predictors are statistically significant, individually and combined they add little to the predictability of overall effectiveness.

Regression

Predictors

Therefore, the return on the investment for

improving the major predictors would far

outweigh the return on improving the minor

predictors.

Horizontal Cross Barset at Mean = 5.50 (75%

satisfaction level)

Vertical Cross Barsets the line between Major and

Minor Predictors of Overall Satisfaction

Top PriorityHigh Impact, Low

Performance

MonitorLow Impact,

Low Performance

MaintainLow Impact,

High Performance

Maintain or

Improve High Impact, High Performance

Impact: Predictors of Overall Satisfaction

4 Quadrants

Priority Matrix

Top PriorityHigh Impact, Low

Performance

MonitorLow Impact,

Low Performance

MaintainLow Impact,

High Performance

Maintain or

Improve High Impact, High Performance

Impact: Predictors of Overall Satisfaction

Priority Matrix

Engaging Pedagogy #1Predictor of Course

Effectiveness

Engaging Pedagogy Factor

• To what degree did the course include:– A variety of teaching methods– Meaningful class discussions– Challenging assignments– Productive use of classroom time– Encouragement to speak in class– Encouragement for students to work

together– Meaningful homework

Course Theme

Four Types of Academic Seminars

1. COLLEGE TRANSITION THEMEorientation/study skills/managing transitions

2. SPECIAL ACADEMIC THEMEinterdisciplinary/problem-focused/selected topic

3. DISCIPLINE BASEDintroduction to major/department/discipline

4. REMEDIAL/STUDY SKILLSstudy skills for a high risk population

Transition-Theme

What is a First-Year Seminar?

A course offered specially for new students to assist with the transition into college.

Four Types of Seminars

Remedial2%

Discipline-based10%

Special Academic

College Transition Theme courses deal directly with orientation to college and academic skills. This is the University 101 (USC) model.

Special Academic Theme courses deal with a selected topic other than college transition. These are often taught as interdisciplinary seminars where a small group of students and a model learner/teacher use a variety of methods to investigate an important theme.

Discipline based are often an introduction to a major or department. They are based in individual academic departments.

Remedial/Study Skills, was considered, but there were too few cases to draw any meaningful conclusions.

Study StrategiesAcademic Skills

Critical ThinkingFaculty Connections

Peer ConnectionsOut-of-Class

Policies/ProceduresCampus Services

Time/PrioritiesWellness/Spirituality

BelongingCourse SatisfactionEngaging Pedagogy

19.213.327.9

28.234.418.034.139.426.423.058.533.630.5

17.316.031.127.732.112.719.222.5

23.115.5

55.137.036.5

9.79.116.819.717.210.0

35.932.215.310.7

52.821.318.3

Theme Format:Percent of students with mean 5.50 or greater

SpecialTransition Academic Discipline

FYI Finding

Transition-theme and Special Academic-theme courses were about equal on learning outcomes and student satisfaction.

Discipline-theme courses produced lower learning outcomes and student satisfaction.

Remedial courses – too few in the study to draw valid conclusions.

Why?Engaging Pedagogy explains the

difference.

Transition 30.5%Special Academic 36.5%Discipline 18.3%

Course Theme

Credit Hours2001 Pilot Administration

% of institutions (N=62)

No credit

4 or more credits

1 credit

2 credits

3 credits

Percent of students mean 5.50 or greater 15.9 20.6 19.4

10.9 12.7 17.323.1 26.1 34.224.9 28.3 29.626.1 35.5 37.314.8 18.5 18.633.3 35.3 30.435.9 41.0 35.424.6 27.7 25.218.5 24.0 23.055.3 58.5 60.228.1 34.9 35.424.1 30.1 37.0

Contact Hour(s)1 Hr 2 Hrs 3

1 contact hour courses

Orientation to Services2 contact hour courses

Study Strategies Peer Connections Faculty

Connections3 contact hour courses

Academic SkillsCritical Thinking

most effective with learning outcomes associated with a basic orientation to campus services.

perform well at producing effective study strategies, peer connections, and faculty connections

significantly impact academic skills (reading, writing, oral presentation skills) and critical thinking skills

FYI Finding

The number of contact hours should match the

intended goals of a first-year seminar.

Course Theme

Required/Not Required

2001 Pilot Administration% of institutions (N = 62)

35%17%

Cal FYI42% 37% No students in any section required to enroll32% 11% Some students in some sections required to enroll16% 35% All/most students in every section required to enroll10% 17% Mixed formats – no one format constitutes 80%

16%10%

Percent of students mean 5.50 or greater 16.1

12.727.824.827.412.925.927.722.616.756.027.929.0

19.514.028.129.438.017.834.442.426.724.058.237.731.9

Required Not Required

Study Strategies 4.18 4.43Academic Skills 3.47 3.76

Critical Thinking 4.36 4.56Faculty Connections 4.35 4.66Peer Connections 4.31 4.84

Out-of-Class 3.36 3.76Policies/Procedures 4.28 4.66

Campus Services 4.35 4.93Time/Priorities 4.27 4.57

Wellness/Spirituality 3.59 4.14Belonging 5.41 5.51

Course Satisfaction 4.29 4.88Engaging Pedagogy 4.43 4.74

Factor Means for Required/not requiredControlled for Contact Hrs, Theme-types, and Grade-formats

Required Not Required

Study Strategies 4.18 4.43Academic Skills 3.51 3.72

Critical Thinking 4.39 4.55Faculty Connections 4.37 4.65Peer Connections 4.34 4.83

Out-of-Class 3.36 3.77Policies/Procedures 4.21 4.72

Campus Services 4.29 4.98Time/Priorities 4.26 4.59

Wellness/Spirituality 3.55 4.18Belonging 5.43 5.51

Course Satisfaction 4.32 4.86Engaging Pedagogy 4.47 4.70

Factor Means for Required/not requiredControlled for student characteristics (gender, race/ethnic, HS Grades, and commuter/residential)

FYI FindingCourses that are NOT REQUIRED

. . . produced greater outcomes than required courses.

Engaging Pedagogy

Required Not Required 4.48* 4.71*

* after controlling for differences in grading and contact hours

Course Theme

Grading 2001 Pilot Administration

% of institutions (N = 62)

82% Letter Graded16% Pass/Fail

2% Mixed

Graded

Mean Scores *controlling for Contact Hrs, Required, UGTAs, & Themes

4.25 4.333.63 3.454.40 4.394.49 4.544.56 4.453.68 3.444.51 4.794.70 4.794.38 4.503.93 3.815.48 5.414.56 4.624.58 4.43

FYI FindingOverall, grading format produces

mixed results, even when controlled for Contact Hours, Theme-types, Required, and

UGTAs.Key Finding:

Graded courses are associated with higher

scores on Engaging Pedagogy

Course Theme

Linked Courses2001 Pilot Administration

% of institutions (N = 62)

11%16%

Few or no sections linked

Some Linked Most Linked

16%43%

Mean Scores controlling for Required, Grading, Contact, & Theme

4.46 4.383.78 3.644.62 4.504.60 4.655.02 4.633.63 3.814.55 4.704.75 4.914.60 4.574.02 4.155.55 5.504.89 4.724.76 4.66

FYI FindingLinking the seminar to other

courses produced greater learning outcomes for:

Academic SkillsStudy Skills

Critical ThinkingEngaging Pedagogy

Limitations in this study reduce the capability to make definitive statements about the impact of

linking courses.

Course Theme

Use of Undergraduate Teaching Assistants

Less than 80% of sections have UTAs

21%16%

No sections

have UTAs

More than 80%of sections have

Mean Scores controlling for Required, Grading, Contact, & Theme

4.28 4.313.56 3.744.40 4.584.54 4.444.66 4.423.73 3.234.51 4.214.83 4.234.47 4.304.04 3.495.48 5.414.57 4.514.57 4.60

FYI FindingUndergraduate Teaching Assistants are associated with higher mean learning outcomes - except for academic skills, critical thinking skills, and engaging pedagogy.

.... even after controlling for:

Required/NotGrading formatThemeContact Hours

Summary:

Engaging Pedagogy best predicts the learning

outcomes and student satisfaction with the

seminar

Student rating of Engaging Pedagogy:

• No significant difference by gender

• African-Americans, Latino, and Native Americans gave higher ratings than Whites and Asians

• “A” high school students gave lower ratings

Right Reason• Help staff calibrate

performance

• Identify where to focus effort

• Identify where training is needed

• Provide motivation to improve

• Use assessment for Continuous Improvement, not Evaluation

• Don’t punish staff for past performance they are not capable of changing.

• Join with staff to use information to improve performance

Right Reason

Right Method

• Credible

• Confidential

• Comparative

• Comprehensive

• Continuous

Right Reason

Right Info

• reallocate budgets

• focus resources on a limited number of initiatives with the greatest probability of producing improvement

Results should be of research quality, but analyzed and presented to provide decision-makers with the information they need to:

Right Method

Right Reason

Right Info

Right People

• Put the information directly in the hands of the people in the best position to initiate and implement change

• Put information into the hands of the people responsible for results

Right Method

Right Reason

Right Info

Right Time

Right People

• Immediacy of feedback brings relevancy and impact to results

• Provide results at a time staff can evaluate performance, focus on impact factors to create innovative initiatives to improve.

• The shortest assessment cycle leads to the most rapid improvement

Right Method

Right Reason

Right Info

Repeatedly

Right Time

Right People

• Essential to establish expectation in staff they will be able to determine the impact of their efforts to improve through future assessments

• Assessment is a management tool not an isolated event

• Continuous assessment is paramount to continuous improvement

Right Method

Right Reason

California State University First Year Experience Assessment Presenter: Joseph Pica, Ed.D. CEO...

Documents