George D. Kuh

Rankings and the Visibility of

Quality Outcomes in the EHEA

Dublin, Ireland

January 31, 2013

Measuring What Matters:

Forging the Right Tools

to Assess Student

Learning Outcomes

Overview

The U.S. context

A word about NILOA

Assessment: Purposes and approaches

The kind of learning we need today

The measurement tools we need

Concluding thoughts

The U.S. Context

Unprecedented numbers of increasingly diverse students matriculating

Many underprepared students

Rising college costs

Continuing shift of cost from government to students

Increasing numbers of part-time instructors

Worries about university quality, global competitiveness

NILOA

NILOA’s mission is to discover and

disseminate effective use of assessment data

to strengthen undergraduate education and

support institutions in their assessment

efforts. SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL

PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ● LISTSERV ●

PRESENTATIONS ● TRANSPARENCY FRAMEWORK ● FEATURED WEBSITES ●

ACCREDITATION RESOURCES ● ASSESSMENT EVENT CALENDAR ● ASSESSMENT

NEWS ● MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL

SCAN ● DEGREE QUALIFICATIONS PROFILE

www.learningoutcomesassessment.org

www.learningoutcomeassessment.org

Assessment 2013

Greater emphasis on student learning

outcomes and evidence that student

performance measures up

Assessment 2013

Greater emphasis on student learning outcomes and evidence that student performance measures up

Demands for comparative measures

Increased calls for transparency ---public disclosure of student and institutional performance

Assessment “technology” has improved markedly, but still is insufficient to document learning outcomes most institutions claim

Measuring Quality in Higher Education

(Vic Borden & Brandi Kernel, 2010)

Web-based inventory hosted by AIR of assessment

resources. Key words can be used to search the four

categories:

instruments (examinations, surveys, questionnaires,

etc.);

software tools and platforms;

benchmarking systems and data resources;

projects, initiatives and services.

http://applications.airweb.org/surveys/Default.aspx

Assessment Purposes

Improvement

Accountability

Continuous

Improvement

Accountability

Strategic dimensions

Purpose Formative (improvement) Summative (judgment)

Orientation Internal External

Motivation Engagement Compliance

Implementation

Instrumentation Multiple/triangulation Standardized

Nature of evidence Quantitative and qualitative Quantitative

Reference points Over time, comparative,

established goal

Comparative or fixed

standard

Communication of

results

Multiple internal channels Public communication,

media

Use of results Multiple feedback loops Reporting

Two Paradigms of Assessment

Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.

Assessment Tools

Direct (outcomes) measures

-- Evidence of what students have learned or can do

Indirect (process) measures

-- Evidence of effective educational activity by students and institutions

Direct Measures

ETS Proficiency Profile & Major Field Tests

ACT Collegiate Assessment of Academic Proficiency (CAAP)

Collegiate Learning Assessment (CLA) – the AHELO measure of general skills

Competency tests (e.g., nursing, education)

Portfolios (authentic student work such as writing samples)

Performances, demonstrations

Indirect Measures

National Surveys of Student Engagement (NSSE/CCSSE/AUSSE/SASSE)

Beginning College Survey of Student Engagement (BCSSE)

Faculty Survey of Student Engagement (FSSE)

Cooperative Institutional Research Program (CIRP)

Your First College Year (YFCY)

College Student Experiences Questionnaire (CSEQ)

Noel Levitz Student Satisfaction Inventory

Institution-level assessments of learning outcomes for all institutions

Program-level assessments of learning outcomes for all institutions

What Really Matters in University:

Student Engagement

Because individual effort and involvement are the critical determinants of college impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular offerings to encourage student engagement.

Pascarella & Terenzini, 2005, p. 602

The quality of student experiences varies more within than between institutions.

Supportive Campus Environment: 4th-Year Students at Master's Institutions

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Master's Institutions

Percentile 10

Percentile 50

Percentile 90

% of Variance Between Institutions

0% 20% 40% 60% 80% 100%

Academic Challenge

Active/Collab Learning

Stu-Fac Interaction

Enriching Experiences

Supportive Campus

Deep Learning

Higher Order Thinking

Integrative Learning

Reflective Learning

Satisfaction

Practical Competence

Personal/Social Devel

General Education

First-Year Senior

Simple

Generally Applicable

Accurate

Commensurate Complexity (Thorngate, 1976; Weick, 1985)

US Economy Defined by Greater

Workplace Challenges and Dynamism

More than 1/3 of the entire US labor force

changes jobs ANNUALLY.

Today's students will have 10-14 jobs by age

38.

Half of workers have been with their

company less than 5 years.

Every year, more than 30 million Americans

are working in jobs that did not exist in the

previous year.

DOL-BLS

The World Wants More From Us

and Our Graduates

…more college-educated

workers.

…more educated workers with

higher levels of learning and

knowledge.

Raising The Bar – October/November 2009 – Hart Research for 23

Employer expectations of employees have increased

88%

88%

90%

91%

% who agree with each statement

Our company is asking employees to take on more responsibilities and to use a broader set of skills than in the past

Employees are expected to work harder to coordinate with other departments than in the past

The challenges employees face within our company are more complex today than they were in the past

To succeed in our company, employees need higher levels of learning and knowledge today than they did in the past

Key Capabilities Open the Door for

Career Success and Earnings

“Irrespective of college major or institutional selectivity, what matters to career success is students’ development of a broad set of cross-cutting capacities…”

Anthony Carnevale, Georgetown University

Center on Education and the Workforce

Narrow Learning is Not Enough:

The Essential Learning Outcomes

Knowledge of Human Cultures and the Physical & Natural World

Intellectual and Practical Skills

Personal and Social Responsibility

“Deep” Integrative Learning

Deep, Integrative Learning

Attend to the underlying meaning of information as well as content

Integrate and synthesize different ideas, sources of information

Discern patterns in evidence or phenomena

Apply knowledge in different situations

View issues from multiple perspectives

Do we measure what we value? or

Do we value what we measure?

Wise decisions are needed about what and how to measure the proficiencies demanded by the 21st century

We need – and are poised for – a

“sea change” in what counts as

meaningful evidence of student

progress and accomplishment.

Employers On Accountability Challenge – December 2007 – Hart Research for

7%

33%

35%

46%

69%

Supervised internship/community-based project

83%

79%

60%

56%

32%

Senior project (e.g., thesis, project)

Essay tests

Electronic portfolio & faculty assessments

Multiple-choice tests

Evidence of College Graduates Skills and Knowledge

Very effective Fairly effective

To Get the Right Kind of Evidence…

We need assessment approaches sensitive to a wide variety of knowledge, abilities, proficiencies, and dispositions

Employers On Accountability Challenge – December 2007 – Hart Research for

Promising Experiments

http://www.learningoutcomesassessment.org/DQPCorner.html

Shift the national conversation from what is taught to what is learned by providing HEIs with a template of widely agreed-upon competencies required for the award of degrees.

http://www.learningoutcomesassessment.org/DQPCorner.html

3 levels: Associate, Bachelor, Masters

5 areas: •Broad, Integrative Knowledge •Specialized Knowledge •Intellectual Skills •Applied Learning •Civic Learning

Occasional Paper #16

The Degree Qualifications

Profile: Implications for

Assessment

Peter T. Ewell & Carol Geary

Schneider

This paper offers guidance for how to

gather evidence about the extent to

which the competencies described in the

DQP are mastered at the levels claimed.

The challenges associated with

assessing DQP proficiencies are

outlined.

www.learningoutcomeassessment.org/OccasionalPapers.htm

Promising Experiments

A dozen two- and four-year HEIs are using the VALUE Rubrics in a “proof of concept at scale” with an eye toward building a national vehicle for using common rubrics

Valid Assessment of Learning in Undergraduate Education (VALUE) Rubrics

Inquiry and analysis Critical thinking Creative thinking Written communication Oral communication Reading Quantitative literacy Information literacy Teamwork Problem solving Civic knowledge and engagement Intercultural knowledge and competence Ethical reasoning and action Foundations and skills for lifelong learning Integrative learning

AAC&U VALUE Project – 15 Rubrics

Promising Experiments

Massachusetts effort led by the

Commissioner for Higher Education

is enlisting additional states to use

assessment of authentic student

learning work to compare

performance and monitor progress

Moving Quality Assurance Forward

Shift the motivation for assessment work from compliance mentality to institutional responsibility

Experiment with ways to “roll up” program level results into meaningful institution-level profiles of student accomplishment

Reconcile or ameliorate the tensions between the accountability and improvement purposes and uses of assessment

Moving Quality Assurance Forward

Show how assessment results are

being used to modify curriculum and

teaching and learning approaches and

enhance student learning

Transparency

Voluntary System of Accountability (APLU/AASCU)

VSA Student Learning

Outcomes Pilot

Four-year experiment

Value-added approach

Institutional level evidence

Administer and publicly post results:

– Collegiate Assessment of Academic

Proficiency

– Collegiate Learning Assessment

– ETS Proficiency Profile

Templates

Voluntary System of Accountability (APLU/AASCU)

U-CAN /Building Blocks for 2020 (NAICU)

College Navigator (NCES)

Transparency by Design/College Choices for Adults (WCET)

Voluntary Framework of Accountability (AACC)

Transparency Framework (NILOA)

Transparency Reports

Transparency Framework

Providing Evidence of

Student Learning:

A Framework for

Transparency

Based on an examination of

about 1000 institutional

websites, the Transparency

Framework provides

guidance to institutions for

effectively presenting

learning outcomes

assessment information on

their websites.

http://planning.iupui.edu/assessment/

The things we have to learn

before we do them, we learn

by doing them.

Aristotle, Nicomachean Ethics

What Matters in University: A Data-Based Narrative…

We need a campaign led by university

leaders, staff, students, and quality

assurance agencies that features

students’ best work, buttressed by

evidence that all students meet

established proficiencies in response to

staff-designed assignments that require

students to show they can use and apply

what they are learning to concrete

situations on and off the campus.

Questions &

Discussion