Developing an Effective Developing an Effective Assessment Plan Assessment Plan Colorado School of Mines Colorado School of Mines Prof. Ronald Miller Prof. Ronald Miller www.mines.edu/academic/chemeng/assess www.mines.edu/academic/chemeng/assess Prepared by Prepared by Dr. Aiman El-Maleh Dr. Aiman El-Maleh King Fahd University of Petroleum & King Fahd University of Petroleum & Minerals, Dept. of Computer Eng., Saudi Minerals, Dept. of Computer Eng., Saudi Arabia Arabia
Transcript
Developing an Effective Developing an Effective Assessment PlanAssessment Plan
Developing an Effective Developing an Effective Assessment PlanAssessment Plan
Colorado School of MinesColorado School of MinesProf. Ronald MillerProf. Ronald Miller
King Fahd University of Petroleum & Minerals, King Fahd University of Petroleum & Minerals, Dept. of Computer Eng., Saudi ArabiaDept. of Computer Eng., Saudi Arabia
Colorado School of MinesColorado School of MinesProf. Ronald MillerProf. Ronald Miller
King Fahd University of Petroleum & Minerals, King Fahd University of Petroleum & Minerals, Dept. of Computer Eng., Saudi ArabiaDept. of Computer Eng., Saudi Arabia
Components of a successful assessment planComponents of a successful assessment plan Aligning program-level outcomes with course-level Aligning program-level outcomes with course-level
outcomesoutcomes Some assessment Do’s and Don’tsSome assessment Do’s and Don’ts Observations about assessmentObservations about assessment RubricsRubrics Focus on efficiencyFocus on efficiency Program objectives & outcomesProgram objectives & outcomes Program assessment and evaluation plan exampleProgram assessment and evaluation plan example Assessment metrics matrixAssessment metrics matrix Oral presentation assessment rubricOral presentation assessment rubric
Components of a successful assessment planComponents of a successful assessment plan Aligning program-level outcomes with course-level Aligning program-level outcomes with course-level
outcomesoutcomes Some assessment Do’s and Don’tsSome assessment Do’s and Don’ts Observations about assessmentObservations about assessment RubricsRubrics Focus on efficiencyFocus on efficiency Program objectives & outcomesProgram objectives & outcomes Program assessment and evaluation plan exampleProgram assessment and evaluation plan example Assessment metrics matrixAssessment metrics matrix Oral presentation assessment rubricOral presentation assessment rubric
3
Components of a Successful Assessment Components of a Successful Assessment PlanPlanComponents of a Successful Assessment Components of a Successful Assessment PlanPlan
Program objectives and program outcomesProgram objectives and program outcomes Performance criteriaPerformance criteria Implementation strategyImplementation strategy Evaluation methodsEvaluation methods Timeline/logisticsTimeline/logistics Feedback loopFeedback loop
Program objectives and program outcomesProgram objectives and program outcomes Performance criteriaPerformance criteria Implementation strategyImplementation strategy Evaluation methodsEvaluation methods Timeline/logisticsTimeline/logistics Feedback loopFeedback loop
Program ObjectivesProgram Objectives• What are the overall goals of the program?What are the overall goals of the program?
• What are the expected accomplishments of program What are the expected accomplishments of program graduates in the first few years after graduation?graduates in the first few years after graduation?
• How do the program goals complement institional and How do the program goals complement institional and accreditation expectationsaccreditation expectations
Program OutcomesProgram Outcomes• What should your students know and should be able by time What should your students know and should be able by time
of graduation?of graduation?
• Are you using measurable verbs to describe the outcomes?Are you using measurable verbs to describe the outcomes?
Program ObjectivesProgram Objectives• What are the overall goals of the program?What are the overall goals of the program?
• What are the expected accomplishments of program What are the expected accomplishments of program graduates in the first few years after graduation?graduates in the first few years after graduation?
• How do the program goals complement institional and How do the program goals complement institional and accreditation expectationsaccreditation expectations
Program OutcomesProgram Outcomes• What should your students know and should be able by time What should your students know and should be able by time
of graduation?of graduation?
• Are you using measurable verbs to describe the outcomes?Are you using measurable verbs to describe the outcomes?
Performance CriteriaPerformance Criteria• How will you know the outcomes have been achieved?How will you know the outcomes have been achieved?
• What level of student performance will indicate students What level of student performance will indicate students posses the outcome?posses the outcome?
• Criteria must be related to the method(s) used for assessing Criteria must be related to the method(s) used for assessing each outcomeeach outcome
Implementation StrategyImplementation Strategy• How will the outcomes e met?How will the outcomes e met?
• What program activities (curricular and co-curricular) help What program activities (curricular and co-curricular) help your students meet each outcome?your students meet each outcome?
Performance CriteriaPerformance Criteria• How will you know the outcomes have been achieved?How will you know the outcomes have been achieved?
• What level of student performance will indicate students What level of student performance will indicate students posses the outcome?posses the outcome?
• Criteria must be related to the method(s) used for assessing Criteria must be related to the method(s) used for assessing each outcomeeach outcome
Implementation StrategyImplementation Strategy• How will the outcomes e met?How will the outcomes e met?
• What program activities (curricular and co-curricular) help What program activities (curricular and co-curricular) help your students meet each outcome?your students meet each outcome?
Evaluation MethodsEvaluation Methods• What assessment methods will you use to collect data?What assessment methods will you use to collect data?
• How will you interpret and evaluate the data?How will you interpret and evaluate the data?
LogisticsLogistics• When will you measure?When will you measure?
• How often will you measure?How often will you measure?
• Who will collect and interpret the data and report the Who will collect and interpret the data and report the results?results?
Evaluation MethodsEvaluation Methods• What assessment methods will you use to collect data?What assessment methods will you use to collect data?
• How will you interpret and evaluate the data?How will you interpret and evaluate the data?
LogisticsLogistics• When will you measure?When will you measure?
• How often will you measure?How often will you measure?
• Who will collect and interpret the data and report the Who will collect and interpret the data and report the results?results?
7
FeedbackFeedbackFeedbackFeedback
Who needs to know the results?Who needs to know the results? How can you convince them the outcomes were met?How can you convince them the outcomes were met? How can you improve your program and your How can you improve your program and your
assessment process?assessment process? Discrepancies between performance criteria and Discrepancies between performance criteria and
student outcome results form the basis for program student outcome results form the basis for program improvement.improvement.
Who needs to know the results?Who needs to know the results? How can you convince them the outcomes were met?How can you convince them the outcomes were met? How can you improve your program and your How can you improve your program and your
assessment process?assessment process? Discrepancies between performance criteria and Discrepancies between performance criteria and
student outcome results form the basis for program student outcome results form the basis for program improvement.improvement.
8
Aligning Program-level Outcomes with Aligning Program-level Outcomes with Course-level OutcomesCourse-level OutcomesAligning Program-level Outcomes with Aligning Program-level Outcomes with Course-level OutcomesCourse-level Outcomes
Once program objectives and outcomes are Once program objectives and outcomes are established, course-level outcomes should be established, course-level outcomes should be prepared.prepared.
Map the course-level outcomes onto the program-Map the course-level outcomes onto the program-level outcomes to be sure coverage is complete and level outcomes to be sure coverage is complete and students are given enough opportunity to meet each students are given enough opportunity to meet each outcome.outcome.
Each course is given a score to indicate its coverage Each course is given a score to indicate its coverage of the outcomeof the outcome• 1: little emphasis1: little emphasis
• 2: some emphasis2: some emphasis
• 3: significant emphasis3: significant emphasis
Once program objectives and outcomes are Once program objectives and outcomes are established, course-level outcomes should be established, course-level outcomes should be prepared.prepared.
Map the course-level outcomes onto the program-Map the course-level outcomes onto the program-level outcomes to be sure coverage is complete and level outcomes to be sure coverage is complete and students are given enough opportunity to meet each students are given enough opportunity to meet each outcome.outcome.
Each course is given a score to indicate its coverage Each course is given a score to indicate its coverage of the outcomeof the outcome• 1: little emphasis1: little emphasis
• 2: some emphasis2: some emphasis
• 3: significant emphasis3: significant emphasis
9
Some Assessment Do’s and Don’tsSome Assessment Do’s and Don’tsSome Assessment Do’s and Don’tsSome Assessment Do’s and Don’ts
Don’t start collecting data before developing clear Don’t start collecting data before developing clear objectives, outcomes, and a process.objectives, outcomes, and a process.
Do promote stakeholder buy-in by involving as many Do promote stakeholder buy-in by involving as many constituencies in the process as possible.constituencies in the process as possible.
Do involve as many faculty members as possible; Do involve as many faculty members as possible; balance day-to-day assessment tasks (one person?) balance day-to-day assessment tasks (one person?) with periodic input from program faculty.with periodic input from program faculty.
Don’t forget to look for campus resources to help Don’t forget to look for campus resources to help supplement program assessment efforts.supplement program assessment efforts.
Don’t start collecting data before developing clear Don’t start collecting data before developing clear objectives, outcomes, and a process.objectives, outcomes, and a process.
Do promote stakeholder buy-in by involving as many Do promote stakeholder buy-in by involving as many constituencies in the process as possible.constituencies in the process as possible.
Do involve as many faculty members as possible; Do involve as many faculty members as possible; balance day-to-day assessment tasks (one person?) balance day-to-day assessment tasks (one person?) with periodic input from program faculty.with periodic input from program faculty.
Don’t forget to look for campus resources to help Don’t forget to look for campus resources to help supplement program assessment efforts.supplement program assessment efforts.
10
Some Assessment Do’s and Don’tsSome Assessment Do’s and Don’tsSome Assessment Do’s and Don’tsSome Assessment Do’s and Don’ts
Don’t forget that quality of results is more important Don’t forget that quality of results is more important than quantity .than quantity .
Not every outcome needs to be measured for every Not every outcome needs to be measured for every student every semester.student every semester.
Do collect and interpret data that will be of most value Do collect and interpret data that will be of most value in improving learning and teaching.in improving learning and teaching.
Don’t forget that quality of results is more important Don’t forget that quality of results is more important than quantity .than quantity .
Not every outcome needs to be measured for every Not every outcome needs to be measured for every student every semester.student every semester.
Do collect and interpret data that will be of most value Do collect and interpret data that will be of most value in improving learning and teaching.in improving learning and teaching.
11
Observations about AssessmentObservations about AssessmentObservations about AssessmentObservations about Assessment
There will always be more than one way to measure There will always be more than one way to measure any outcome.any outcome.
There is no “magic bullet” that will measure all There is no “magic bullet” that will measure all student abilities.student abilities.
There is a consistently There is a consistently inverseinverse correlation between correlation between quality of measurement methods and their quality of measurement methods and their expediencyexpediency• The best methods usually take longer and cost moreThe best methods usually take longer and cost more
There will always be more than one way to measure There will always be more than one way to measure any outcome.any outcome.
There is no “magic bullet” that will measure all There is no “magic bullet” that will measure all student abilities.student abilities.
There is a consistently There is a consistently inverseinverse correlation between correlation between quality of measurement methods and their quality of measurement methods and their expediencyexpediency• The best methods usually take longer and cost moreThe best methods usually take longer and cost more
Use of multiple assessment methods to assess and Use of multiple assessment methods to assess and evaluate an outcome.evaluate an outcome.
Improves reliability (repeatability) and validity Improves reliability (repeatability) and validity (accuracy) of assessment data.(accuracy) of assessment data.
Recommendations for program improvement are Recommendations for program improvement are more credible.more credible.
Use of multiple assessment methods to assess and Use of multiple assessment methods to assess and evaluate an outcome.evaluate an outcome.
Improves reliability (repeatability) and validity Improves reliability (repeatability) and validity (accuracy) of assessment data.(accuracy) of assessment data.
Recommendations for program improvement are Recommendations for program improvement are more credible.more credible.
13
RubricsRubricsRubricsRubrics
A detailed guide to scoring student work products or A detailed guide to scoring student work products or performancesperformances• Analytic: scores individual components or characteristics of Analytic: scores individual components or characteristics of
a work producta work product
• Holistic: scores work product as a wholeHolistic: scores work product as a whole
Can be used for assigning grades to individual Can be used for assigning grades to individual students or for overall program assessmentstudents or for overall program assessment
A detailed guide to scoring student work products or A detailed guide to scoring student work products or performancesperformances• Analytic: scores individual components or characteristics of Analytic: scores individual components or characteristics of
a work producta work product
• Holistic: scores work product as a wholeHolistic: scores work product as a whole
Can be used for assigning grades to individual Can be used for assigning grades to individual students or for overall program assessmentstudents or for overall program assessment
14
Creating a RubricCreating a RubricCreating a RubricCreating a Rubric
Determine what learning outcomes to assessDetermine what learning outcomes to assess Determine what student work will be assessedDetermine what student work will be assessed Determine an appropriate rating scaleDetermine an appropriate rating scale Draft the rubric using sample workDraft the rubric using sample work Test the rubric on student work with several Test the rubric on student work with several
evaluatorsevaluators Revise rubric language until inter-rater reliability Revise rubric language until inter-rater reliability
meets expectationsmeets expectations
Determine what learning outcomes to assessDetermine what learning outcomes to assess Determine what student work will be assessedDetermine what student work will be assessed Determine an appropriate rating scaleDetermine an appropriate rating scale Draft the rubric using sample workDraft the rubric using sample work Test the rubric on student work with several Test the rubric on student work with several
evaluatorsevaluators Revise rubric language until inter-rater reliability Revise rubric language until inter-rater reliability
meets expectationsmeets expectations
15
Sustaining the EffortSustaining the EffortSustaining the EffortSustaining the Effort
How will your program and institution sustain the How will your program and institution sustain the assessment efforts you design and implement?assessment efforts you design and implement?• Issues of faculty time and effort?Issues of faculty time and effort?
• Faculty rewards and recognition?Faculty rewards and recognition?
• Will programs and the institution be willing to improve ased Will programs and the institution be willing to improve ased on assessment results?on assessment results?
• How can you make the process more efficient?How can you make the process more efficient?
How will your program and institution sustain the How will your program and institution sustain the assessment efforts you design and implement?assessment efforts you design and implement?• Issues of faculty time and effort?Issues of faculty time and effort?
• Faculty rewards and recognition?Faculty rewards and recognition?
• Will programs and the institution be willing to improve ased Will programs and the institution be willing to improve ased on assessment results?on assessment results?
• How can you make the process more efficient?How can you make the process more efficient?
16
Focus on EfficiencyFocus on EfficiencyFocus on EfficiencyFocus on Efficiency
Assessment is designed to provide data to Assessment is designed to provide data to improve improve the programthe program, not certify students for gradation, not certify students for gradation
This impliesThis implies• SamplingSampling of student work and other assessment data of student work and other assessment data
collectioncollection
• Less frequent assessment of some outcomesLess frequent assessment of some outcomes
Consider use of scoring rubrics as a mechanism for Consider use of scoring rubrics as a mechanism for rapid, holistic scoring of student work productsrapid, holistic scoring of student work products
Select student work products that ca be used to Select student work products that ca be used to assess more than one outcomeassess more than one outcome
Assessment is designed to provide data to Assessment is designed to provide data to improve improve the programthe program, not certify students for gradation, not certify students for gradation
This impliesThis implies• SamplingSampling of student work and other assessment data of student work and other assessment data
collectioncollection
• Less frequent assessment of some outcomesLess frequent assessment of some outcomes
Consider use of scoring rubrics as a mechanism for Consider use of scoring rubrics as a mechanism for rapid, holistic scoring of student work productsrapid, holistic scoring of student work products
Select student work products that ca be used to Select student work products that ca be used to assess more than one outcomeassess more than one outcome
17
Focus on EfficiencyFocus on EfficiencyFocus on EfficiencyFocus on Efficiency
Don’t measure an outcome if you don’t want to know Don’t measure an outcome if you don’t want to know the resultsthe results
Don’t collect data that won’t be usedDon’t collect data that won’t be used Develop a long-range plan to measure outcomes Develop a long-range plan to measure outcomes
periodically but not necessarily every semesterperiodically but not necessarily every semester Focus on key data that will help make program Focus on key data that will help make program
improvementsimprovements
Don’t measure an outcome if you don’t want to know Don’t measure an outcome if you don’t want to know the resultsthe results
Don’t collect data that won’t be usedDon’t collect data that won’t be used Develop a long-range plan to measure outcomes Develop a long-range plan to measure outcomes
periodically but not necessarily every semesterperiodically but not necessarily every semester Focus on key data that will help make program Focus on key data that will help make program
improvementsimprovements
18
Program ObjectivesProgram ObjectivesProgram ObjectivesProgram Objectives
Objective 1Our graduates will enter the workforce with a high-quality basic education in chemical engineering fundamentals.
Objective 2
Our graduates will possess the knowledge and skills required to apply engineering fundamentals to the analysis, synthesis, and evaluation of chemical engineering processes and systems.
Objective 3
Our graduates will develop personally to ensure a lifetime of professional success and an appreciation for the ethical and social responsibilities of chemical engineering and world citizen.
3.1Our graduates will demonstrate an awareness of professional
and ethical responsibility. f
3.2Our graduates will demonstrate an ability to communicate in
oral and written formats.g
3.3Our graduates will demonstrate an awareness of the impact of
engineering solutions in a global and societal context.h
3.4Our graduates will demonstrate an ability to engage in life-
long learning and self-education.i
3.5Our graduates will demonstrate an awareness of contemporary issues in chemical engineering.
j
22
Program Assessment and Evaluation Plan Program Assessment and Evaluation Plan ExampleExampleProgram Assessment and Evaluation Plan Program Assessment and Evaluation Plan ExampleExample
