Academic Program AssessmentNovember 18-19, 2014

Ryan J. McLawhon, Ed.D.DirectorInstitutional AssessmentRyan.McLawhon@tamu.edu

Elizabeth C. Bledsoe, M.A.Program CoordinatorInstitutional Assessmentebledsoe@tamu.edu

Kimberlee PottbergSr. Admin CoordinatorInstitutional AssessmentK-pottberg@tamu.edu

assessment@tamu.edu979.862.2918

assessment.tamu.edu

• Components of the WEAVEonline Assessment Plan & expectations of each

• Question and Answer Session

Agenda

SACS Comprehensive Standard 3.3.1

3.3 Institutional Effectiveness

3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness)

3.3.1.1 educational programs, to include student learning outcomes

3.3.1.2 administrative support services

3.3.1.3 educational support services

3.3.1.4 research within its educational mission, if appropriate

3.3.1.5 community/public service within its educational mission, if appropriate

SACS Expectations

SACS Comprehensive Standard 3.3.1

3.3 Institutional Effectiveness

3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness)

3.3.1.1 educational programs, to include student learning outcomes

3.3.1.2 administrative support services

3.3.1.3 educational support services

3.3.1.4 research within its educational mission, if appropriate

3.3.1.5 community/public service within its educational mission, if appropriate

SACS Expectations

and provides evidence of improvement based on analysis of the results…

The Assessment Circle

Develop Program

Mission & Outcomes

Design an Assessment

Plan

Implement the Plan &

GatherInformation

Interpret/Evaluate

Information

Modify & Improve

Adapted from: Trudy Banta, IUPUI

Develop Mission and Outcomes

Develop Program

Mission & Outcomes

• The mission statement links the functions of your unit to the overall mission of the institution.

• A few questions to consider in formulating the mission of your unit:

– What is the primary function of your degree?

– What should students graduating from your program know or be able to do?

Mission Statement

• Brief, concise, distinctive

• Clearly identifies the program’s purpose and larger impact

• Clearly aligns with the mission of the division and the University

• Clearly identifies the primary stakeholders of the program: i.e., students, faculty, parents, etc.

Characteristics of a Well-Defined Mission Statement

There are two categories of outcomes:

Learning Outcomes

Program Objectives

Outcomes/Objectives

• Encompass a discipline-specific body of knowledge

• Limited in number (manageable)

• Specific, measurable and/or observable

• Meaningful

• Describe a level of mastery appropriate to the degree

• Use Action Verbs

Outcomes/Objectives should…

Cognitive Learning

Knowledge - to recall or remember facts without necessarily understanding them

articulate, define, indicate, name, order, recognize, relate, recall, reproduce, list, tell, describe, identify, show, label, tabulate, quote

Comprehensive - to understand and interpret learned information

classify, describe, discuss, explain, express, interpret, contrast, associate, differentiate, extend, translate, review, suggest, restate

Application - to put ideas and concepts to work in solving problems

apply, compute, give examples, investigate, experiment, solve, choose, predict, translate, employ, operate, practice, schedule

Analysis - to break information into its components to see interrelationships

analyze, appraise, calculate, categorize, compare, contrast, criticize, differentiate, distinguish, examine, investigate, interpret

Synthesis - to use creativity to compose and design something original

arrange, assemble, collect, compose, construct, create, design, formulate, manage, organize, plan, prepare, propose, set up

Evaluation - to judge the value of information based on established criteria

appraise, assess, defend, judge, predict, rate, support, evaluate, recommend, convince, conclude, compare, summarize

Affective Learningappreciate, accept, attempt, challenge, defend, dispute, join, judge, praise, question, share, support

• Graduates will be able to design a comprehensive research project from research question to interpretation of results.

• Students will identify and discuss various aspects of architectural diversity in their design projects.

Examples of Learning Outcomes

• Process statements– Relate to what the unit intends to accomplish

• Level or volume of activity (publishing, etc.)• Compliance with external standards of “good practice in the field” or

regulations (accreditation requirements, etc.)

• Satisfaction statements– Describe how those you serve rate their satisfaction with your

degree program or activities

Program Objectives

• Process statements

– Students will contribute to the field by publishing in peer-reviewed journals and national conferences.

• Satisfaction statements– Students will report they are satisfied with the

quality of their education.

Examples of Program Objectives

15

Student Learning Outcome Example:

Student Learning Outcome A Student Learning Outcome B

• Students receiving a degree from this program will be effective communicators.

• Students receiving a degree from this program will be able to effectively communicate their research findings both verbally and in writing.

Design an Assessment Plan

Design an Assessment

Plan

After establishing your outcomes…

• Define and identify the sources of evidence you will use to determine whether you are achieving your outcomes.

• Detail what will be measured and how

• Identify or create measures which can inform decisions about your program’s processes and services.

Assessment Measures

• Measurable and/or observable– You can observe it, count it, quantify, etc.– Specifically defined with enough context to understand

how it is observable

• Meaningful– It captures enough of the essential components of the

objective to represent it adequately– It will yield vital information about your program

• Triangulates data– Multiple measures for each outcome– Direct and Indirect Measures

Measures should be…

There are two basic types of assessment measures:

Direct Measures

Indirect Measures

Types of Assessment Measures(Palomba and Banta, 1999)

Direct measures are those designed to directly measure:

what a stakeholder knows or is able to do (i.e., requires a stakeholder to actually demonstrate the skill or knowledge)

Direct Measures

• A rubric used to assess a collection of work samples (student work)

• Observation of behavior

Common Direct Measures

Indirect measures focus on:

stakeholders’ perception of their level of learning

stakeholders’ perception of the benefit of programming or intervention

completion of requirements or activities stakeholders’ satisfaction with some aspect of the

program or service

Indirect Measures

• Surveys

• Exit interviews

• Count of publications/presentations

• Retention/graduation data

• Demographics

• Focus groups

Common Indirect Measures

Some things to think about:

– How would you describe the end result of the outcome? OR How will you know if this outcome is being accomplished?

– Where are you currently delivering the outcome?

• Are there any naturally occurring assessment opportunities?

– Will the resulting data provide information that could lead to an improvement of your curriculum and/or pedagogy?

Choosing Assessment Measures

• An achievement target is the result, target, benchmark, or value that will represent success at achieving a given outcome.

• Achievement targets should be specific numbers or trends representing a reasonable level of success for the given measure/outcome relationship.

Achievement Targets

• Students will score a 2.5 out of 4 on the writing rubric.

• 75% of graduating students will achieve an average of 3.5 on the dissertation rubric.

Examples of Achievement Targets

Implement & Gather Information

Implement the Plan &

GatherInformation

• The results of the application of the measure to the collected data

• The language of this statement should parallel the corresponding achievement target.

• Results should be described in enough detail to prove you have met, partially met, or not met the achievement target.

Findings

• Example 1:

– Achievement Target

• The mean score on the state mandated certification exam of students graduating from the program will meet or exceed the state average score of 79.

– Findings

• The achievement target was met. The overall mean score of students from the Teaching, Learning, and Culture program exceeded that of the state average score of the state certification exam. Results: Program overall mean scaled score—91.50, State overall mean scaled score—79.13.

Examples of Findings Statements

Interpret/Evaluate Information

Interpret/Evaluate

Information

Analyzing Findings

• Three key questions at the heart of the analysis:

– What did you find and learn?

– So What does that mean for your academic program or support unit?

– Now What will you do as a result of the first two answers?

Analysis Question Responses should…

• Demonstrate thorough analysis of the given findings

• Provide additional context to the action plan (why this approach was selected, why it is expected to make a difference, etc.)

• Update previous action plans – results of implementation

Modify/Improve

Modify & Improve

• After reflecting on the findings, you and your colleagues should determine appropriate action to improve the program.

• Actions outlined in the Action Plan should be specific and relate to the outcome and the results of assessment.

– Action Plans should not be related to the assessment process itself

Action Plans

An Action Plan will…

• Clearly communicate how the evidence of student learning achievement or other Findings inform a change or improvement to curriculum/pedagogy.

• This DOES NOT include:– Changes to assessment processes– Continued monitoring of information– Changes to the program not informed by the data

collected through the assessment process

Assessment Review

Mission Statement

Outcomes/Objectives

Measures

Targets

Findings

Action Plans

Analysis Questions

• Assess what is important

• Use your findings to inform actions

• You do not have to assess everything every year

Take-Home Messages

OIA Consultations

• WEAVEonline support and training• Assessment plan design, clean-up, and

re-design

– And we can come to you!

• New Website: assessment.tamu.edu

Questions?

http://assessment.tamu.edu/conference

The Principles of Accreditation: Foundations for Quality Enhancement. SACS COC. 2008 Edition.

Banta, Trudy W., & Palomba, C. (1999). Assessment Essentials. San Francisco: Jossey-Bass.

Banta, Trudy W. (2004). Hallmarks of Effective Outcomes Assessment. San Francisco: John Wiley and Sons.

Walvoord, Barbara E. (2004). Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education. San Francisco: Jossey-Bass.

Assessment manuals from Western Carolina University, Texas Christian University, the University of Central Florida were very helpful in developing this presentation.

Putting It All Together examples adapted from Georgia State University, the University of North Texas, and the University of Central Florida’s Assessment Plans

References