Outcomes Assessment 2

Program Assessment

Joseph A. ShaeiwitzWest Virginia

Universityjoseph.shaeiwitz@mail.wvu.ed

u

Daina M. BriedisMichigan State

Universitybriedis@egr.msu.edu

Outline

ABET and engineering criteria Program objectives Program outcomes Assessment

performance criteria assessment measures – direct and

indirect rubrics

Review

Outline

ABET and engineering criteria Program objectives Program outcomes Assessment

performance criteria assessment measures – direct and

indirect rubrics

Review

Initial Quiz

What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,

what is the difference between outcomes and objectives?

Identify four assessment methods. Classify each as direct or indirect.

Initial Quiz

What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,

what is the difference between outcomes and objectives?

Identify four assessment methods. Classify each as direct or indirect.

Initial Quiz

What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,

what is the difference between outcomes and objectives?

Identify four assessment methods. Classify each as direct or indirect.

Initial Quiz

What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,

what is the difference between outcomes and objectives?

Identify four assessment methods. Classify each as direct or indirect.

Initial Quiz

What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,

what is the difference between outcomes and objectives?

Identify four assessment methods. Classify each as direct or indirect.

ABET and Engineering Criteria

ABET = Accreditation Board for Engineering and Technology

Engineering criteria changed to assessment basis (TQI) around 2000

Must prove that students are achieving objectives and outcomes measure output, feedback model previously, feed forward model

input outputassumed

curriculum

Feed Forward Model

oneclass

oneclass

education process

one course

one course

enteringcollege

graduate alumnus

Feedback Model

The Two Loops of the Engineering Criteria

Establish Indicatorsthat Objectives are

Being Achieved

Determine HowOutcomes will be

Assessed

Determine HowOutcomes will be

Achieved

Determine OutcomesRequired to Achieve

Objectives

Determine educational objectives

Evaluate/Assess

Input fromConstituencies

Formal InstructionStudent Activities

ABET and Engineering Criteria 8 criteria

students program educational objectives program outcomes and assessment professional component faculty facilities institutional support and financial

resources program criteria

ABET and Engineering Criteria

Focus of this workshop program educational objectives

definition how to establish how to assess

program outcomes and assessment definition how to establish how to assess

Minute PaperClearest vs. Muddiest Point

What have you just learned about ABET and assessment?

What points are the clearest? What points are the “muddiest?”

Outline

ABET and engineering criteria Program objectives Program outcomes Assessment

performance criteria assessment measures – direct and

indirect rubrics

Review

Program Objectives*

“…broad statements that describe the career and professional accomplishments that the program is preparing the graduates to achieve.”*

“Our graduates will be successful …” must define “successful”

*Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org

Program Objectives*

must be detailed and published include constituencies/periodically evaluated educational program to achieve outcomes

(defined later) and to prepare graduates for accomplishments that achieve objectives

ongoing evaluation to determine extent objectives attained, use results for program improvement

*Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org

Exercise

New program in nanobiomolecular engineering

Define two program objectives.

Outline

ABET and engineering criteria Program objectives Program outcomes Assessment

performance criteria assessment measures – direct and

indirect rubrics

Review

Program Outcomes* “…statements of what students are expected to

know and be able to do by the time of their graduation.”

Outcomes must “…foster attainment of the program objectives…”

Process to produce outcomes Assessment process, with documented results

demonstrating measurement demonstrating degree of achievement

Evidence results used for program improvement

*Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org

Program Outcomes Minimum outcomes “a-k” plus

program criteria Opportunity to be unique, i.e., define

unique outcomes, not just repeat “a-k”

Helpful to map program-defined outcomes into a-k

Helpful to map outcomes into classes Necessary to map outcomes into

objectives

WVUABET

1chemicalprocess

2communicate

3 computers

4 learn

independently and group

work

5lab and data

analysis

6continuing education

7safety,

societal,environmental

8ethics

9broad

education

a. apply math, sci, engr b. expts - design, conduct, analyze, interpret data c. design system d. multidisciplinary teams e. identify, formulate, solve engineering problems

f. professional and ethics g. communication h. broad education - global impact i. life-long learning j. contemporary issues k. use techniques, skills, modern engineering tools

WVU outcomeWVU class

1chemicalprocess

2communicate

3 computers

4 learn

independently and group

work

5lab and data

analysis

6continuing education

7safety,

societal,environmenta

l

8ethics

9broad

education

CHE 201 CHE 202 CHE 230 CHE 310 CHE 311 CHE 312 CHE 315 CHE 320 CHE 325 CHE 450/451 CHE 455/456

Exercise

Define three unique program outcomes for the nanobiomolecular engineering program.

Outline

ABET and engineering criteria Program objectives Program outcomes Assessment

performance criteria assessment measures – direct and

indirect rubrics

Review

Performance Criteria

What will students do to demonstrate achievement of outcome

Example: “An ability to communicate effectively.” (3g in ABET list)

What are attributes of effective communication?

Performance Criteria for Effective Communication When making an oral presentation,

students will maintain eye contact …

A written report will follow a prescribed format demonstrate proper grammar and

punctuation adhere to commonly accepted word

usage

Education

al

Objective

Learning

Outcome

Contempo-raryIssues

Professionally responsible

1.Has knowledge of current technological issues related to XXX engineering and society 2. Is able to discuss major political and societal issues and their pertinence to XXX engineering

Performan

ce Criteria

Exercise

Define two or three performance criteria for one of the outcomes previously defined.

Outline

ABET and engineering criteria Program objectives Program outcomes Assessment

performance criteria assessment measures – direct and

indirect rubrics

Review

Indirect vs. Direct Assessment Indirect – based mostly on student self-

evaluation surveys interviews focus groups

Direct – by faculty or some other means of evaluation of student performance (advisory boards evaluate design projects)

Indirect vs. Direct Assessment Indirect

necessary, but not sufficient provides uncalibrated snapshot

self-assessment not necessarily reliable Terminology may be unfamiliar

Direct necessary for quality assessment plan not now for all, but new Program

Evaluators are being trained to look for this feature

relies on faculty experience, expertise, and judgment

How do we know if the students have the requisite outcomes?

When the students think they do, based on student survey results

With direct evidence from student workEvidence is the key to accreditation

Faculty evaluation of student work is the key to providing evidence

Primary assessment of student outcomes should be based on student work (direct) e.g., student portfolios, student projects,

assignments and exams, some employer surveys where skill is observed

Secondary evidence Senior exit surveys, alumni surveys, employer

surveys (qualitative evidence based on opinion), other

Combination of both methods – triangulation

Assessment Measures

Indirect Assessment Measures

Surveys Interviews Course satisfaction surveys

Direct Assessment Measures

End-of-Course Assessments Targeted Assignments Capstone Experiences Capstone Exams Portfolios

End-of-Course Assessments

Course should have objectives perhaps set by department committee

Related to program outcomes Assigned problems (assignments,

exams, projects) each related to course objectives

Evaluation/reflection by instructor

End-of-Course Assessments

Advantages quick and easy assessment can be done in parallel with

grading Disadvantage

not comprehensive no big picture

Opinion a component of assessment plan

Targeted Assignments/Problems

Key assignments that relate to specific program outcomes

Multiple assignments per outcome recommended

Integrate through curriculum can demonstrate progress toward

achievement of program outcome

Targeted Assignments/Problems

Advantages quick and easy assessment can be done in parallel

with grading Disadvantages

none really need consistent evaluation method

with reliable inter-rater reliability

Capstone Experiences

Can be design, laboratory, research All programs have them Where students are supposed to

demonstrate and synthesize what learned

Usually includes teamwork, communication

Capstone Experiences

Advantages already required in program assessment can be done in parallel

with grading Disadvantages

none really need consistent evaluation method

with reliable inter-rater reliability

Capstone Exams

FE Exam detailed, subject-related results

available Department-generated

Capstone Exams

Advantages FE is standardized the most direct measure possible

Disadvantages FE may not set bar as high as some want students may not take department-

generated exam seriously if results do not impact grades or graduation

Portfolios

Collection of student work demonstrating outcomes

Must also evaluate the portfolio Can also have students reflect on

work in portfolio

Portfolios

Advantage comprehensive

Disadvantages portfolio evaluation is additional work need consistent evaluation method

with reliable inter-rater reliability

Other Direct Assessment Methods

Journals Concept maps Oral presentations with follow-up

questions (like M.S./Ph.D. defense)

Exercise

Select an assessment method to be used for direct assessment of program outcomes previously defined.

Outline

ABET and engineering criteria Program objectives Program outcomes Assessment

performance criteria assessment measures – direct and

indirect rubrics

Review

Rubrics

A set of categories developed from the performance criteria that define extent to which the performance criteria are met (progression towards attaining).

Developing Rubrics

Define levels of performance for each performance criterion best first to define top and bottom

levels of performance – then fill in middle ground

3-5 levels of performance recommended

Developing Rubrics

Standardized method to ensure inter-rater reliability

Specific definitions of terms like excellent, understand, not acceptable, exceeds expectations

Initial effort and periodic review required

Developing Rubrics If have five levels of performance, can

make each level a grade if have three levels, can make each level

A/C/F, and interpolate for B/D Use rubric for assessment and grading Advantage: Students and faculty have

clearly defined criteria for grading of what appears to be subjective (lab and design reports, oral reports, etc.)

“Application of math & science”

Problem Assessment Form Connects physical model with math model Able to write unsteady state mass balance Makes appropriate substitution for flow terms Makes appropriate simplification for flow terms Converts differential equation into Laplace form

correctly Understands idea for output/input form of

transfer function Correctly combines Laplace transforms in series Correct answer

Rubrics – Assessment Scale“Application of math & science” (5=high)

Level 5 Formulates models

correctly Applies calculus or linear

algebra to solve problems Correct calculations Correct statistical analysis . . .

Level 3 Formulates models with

some trouble Some understanding of

calc/linear algebra applications

Minor calc errors Minor statistical errors . . .

Level 1 Not able to model Cannot apply

calc/linear algebra Incorrect calculations Does not apply

statistics . . .

Education

al

Objective

Learning

Outcome

Contempo-raryIssues

Professionally responsible

1.Has knowledge of current technological issues related to XXX engineering and society 2. Is able to discuss major political and societal issues and their pertinence to XXX engineering

Performan

ce Criteria

Outcome: A knowledge of contemporary issues Performance Criteria Scoring Rubrics

Rating Scale & Element

Needs Improvement 1 2 Met Expectations 3 4Exceeded

Expectations5 N/A

Has knowledge of current technological issues related to chemical engineering and society (global warming, resource depletion, waste proliferation, etc.)

Has minimal knowledge of technological issues and their relevance to chemical engineering; has weak connection between the issue and scientific principles for analysis and has trouble developing solutions.

Has reasonable knowledge of technological issues; some may not be directly relevant to chemical engineering; can apply scientific principles to analysis and suggest solutions when guided.

Has thorough knowledge of current technology issues related to chemical engineering and is able to analyze them and propose solutions using scientific principles

Has knowledge of and is able to discuss major societal and political issues and their pertinence to chemical engineering

Has minimal knowledge of societal and political issues; if given an issue, does not see its connection to engineering without instruction; is minimally effective in discussion and presentation of such issues

Has reasonable knowledge of societal and political issues; recognizes some connection to chemical engineering, but misses the details; is somewhat effective in discussing and presenting such issues when prompted

Has thorough knowledge of societal and political issues related to chemical engineering; recognizes the “big picture” and the details; presents strong discussion of such issues

Rubric for Oral Presentations

Attribute 1-Not proficient 2-Progressing to proficiency

3-Proficient 4-Superior proficiency

Score

Effective use of Visual Aids (VA) Clarity and readability not clear or

readable difficulty reading clear and readable superior clarity and

readability

Use of space on VA VA unreadable because too crowded

too little or too much information of VA

appropriate amount of information on VA

VAs very well laid out

Lettering readable font unreadable font too small font readable Color, over- or under-use (if used) colors too hard to

distinguish, colors do not project well

poor choice and use of colors

primary/easily distinguishable colors

use of color enhances clarity of presentation

Wording concise slides full of text slides too wordy slides appropriate Appropriate amount of information per VA

so much information per VA or so much missing information to make VA useless

too much information per VA, missing information such as size of total pie

appropriate level of information per slide

Presentation Organization Logical order of topics totally disjointed,

no organization some items presented out of order

organization as per guidelines

superior organization enhances communication

Appropriate use of time: Not too long /short

far too long or far too short

somewhat too long or too short

appropriate length

Complete "story" told story missing, no story told

story incomplete complete story told

Introduction: Problem stated problem not stated, problem poorly stated

problem clearly stated

problem clearly stated, good perspective on problem shown

Rubric for Written Presentations

Attribute 1-Not proficient 2-Progressing to proficiency

3-Proficient 4-Superior proficiency

Score

Report Mechanics Organization inappropriate

content of most sections of report

some content in inappropriate section of report

content appropriate to all sections of report

unique organization enhances readability and/or understandability of report

Complete Story Told no story told, very incomplete

aspects of complete story missing

complete story told additional material enhances quality of report

Aesthetics unacceptable – e.g., tables and figures cannot be read/understood, fonts difficult to read

some portions are sloppy and difficult to read

text, tables, figures readable and understandable

text, tables, figures so clear and understandable as to enhance report impact

Format so many format errors as to make report useless

some format errors format followed unique format aspects that enhance report impact

Spelling any spelling errors only spelling errors are different spellings for same pronunciation

no spelling errors

Grammar and Punctuation too many grammar and punctuation errors

grammar and punctuation errors

only a very few minor grammar or punctuation errors

no grammar or punctuation errors

Length far too long or too short

too long or too short

appropriate length

Rubric for Majors and Design Projects

Attribute 1-Not proficient 2-Progressing to proficiency

3-Proficient 4-Superior proficiency

Score

Design of equipment, Analysis of performance of existing equipment, Understand interrelationship between equipment in process

Design of individual equipment major errors in individual equipment design

some errors in equipment design

equipment designed correctly

unique aspects of equipment design enhance result

Understand interrelationship between equipment on flowsheet

no understanding of equipment interrelationship

minimum understanding of equipment interrelationship

clear understanding of equipment interrelationship

exploitation of equipment interrelationship to enhance result

Constraints/limitations of individual equipment and flowsheet understood

constraints/ limitations not understood

not all constraints/ limitations understood

constraints/ limitations clearly understood

exploitation of constraints/ limitations to enhance result

Response to questions indicates understanding of ChE principles

response to questions demonstrates lack of understanding

response to questions shows gaps in understanding

response to questions shows clear understanding

response to questions shows superior understanding

Significance of conclusions understood

lack of understanding, no explanations

gaps in understanding, few explanations

clear understanding and explanations

superior understanding with in-depth explanations

Apply chemistry, math, physics, life science, engineering science

Apply engineering science inability to apply principles

a few basic principles applied

most principles applied, demonstration of effect on design

all principles applied and interwoven with engineering to complete design

Rubrics

Additional rubrics athttp://www.che.cemr.wvu.edu/ugrad/outcomes/rubrics/index.php

Instructions on rubrics

http://webquest.sdsu.edu/rubrics/weblessons.htm

Exercise

For the two or three performance criteria for one of the outcomes previously defined, and assuming the assessment method previously defined, begin to develop an evaluation rubric.

Outline

ABET and engineering criteria Program objectives Program outcomes Assessment

performance criteria assessment measures – direct and

indirect rubrics

Review

Recommendation

Strongly recommend adopting direct assessment measures into assessment activities

Will be expected in the near future!

Exercise

What are the two most important things that you learned in this workshop?

What is still unclear to you about program assessment?

Questions

?