Date post: | 21-Dec-2015 |
Category: |
Documents |
Upload: | edwina-barrett |
View: | 216 times |
Download: | 2 times |
+
Academic Program Workshop for Creating an Assessment Plan
Facilitated by Pamela Baker, PhD, Director, CET&L Laurah Turner, PhD, Assistant Director, CET&L
University of Cincinnati, Center for the Enhancement of Teaching & Learning
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Session Outcomes
Identify and explain to colleagues the purpose and process of assessment.
Identify and evaluate the components of an assessment plan.
Develop a framework for the ideal graduate of your program.
Draft strong program-level learning outcomes.
Understand alignment of outcomes at the program, course, and unit-level.
Conduct a preliminary audit of program-level assessments.
Draft a curriculum map for your program that identifies specific courses where program learning outcomes are addressed.
Develop a plan to gather & analyze assessment data to determine how well student learning matches expectations (with a view to identify action to be taken based on findings).
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Session Timeline
9:00-9:15 Introductions and Program Overview 9:15-9:30 Identify & Explain the Purpose of an Assessment
Plan 9:30-9:45 Identify & Evaluate the Components of an
Assessment Plan 9:45-10:05 Develop a framework for the Ideal Graduate 10:05-10:45 Identify Strong Program Outcomes 10:45-10:55 BREAK 10:55-12:00 Review Program Outcomes 12:00-12:30 LUNCH 12:30- 1:15 Conduct a Preliminary Audit of Program Level
Assessment 1:15-2:15 Draft a Curriculum Map 2:15:2:25 BREAK 2:25-3:25 Develop a plan to gather and analyze assessment
data (with a view to identify action to be taken based on findings).
3:25-3:30 Q & A
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Overview: Identify & Explain the Purpose & Process of Assessment What is the purpose of program assessment within the context of
Semester Conversion & Higher Ed?
What is assessment of a program? Program Effectiveness (e.g. the program will graduate 90% of
students, & 70% of those students will earn degrees within five years)
Program Outcomes (i.e. emphasis on demonstrating what students can do as a result of completing a program)
What is the common purpose of assessment? A tool to make meaningful improvements and appropriate
interventions A tool to ensure efficient allocation of resources A way to highlight strengths within a curriculum Reaccreditation
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Assessment, Continued
SHORTER TERM Ascertain whether what our students have learned is good enough
Identify assessment methods & measures Identify whether these assessment tools are assessing what it is
we expect our students to learn Identify whether required courses/experiences within the
curriculum align with the program level outcomes Determine whether our students are learning at the appropriate
scope, depth, and level
LONGER TERM Use long-term findings to make more significant curriculum revisions
(beyond simple fine-tuning of programs) Use assessment evidence to make informed decisions about your
program within the context of college and institutional priorities Identify external benchmarks to evaluate the learning of our students
Sharing evidence & best practicesAdapted from Linda Suskie, “Why Are We Assessing?,” Inside Higher Ed (October 26, 2010), adapted from her talk at the 2010 Assessment Institute. (Dr. Suskie serves as Vice President of the Middle States Commission on Higher Education.) & Susan Hatfield, “Assessing Your Program-Level Assessment Plan,” Idea Paper #45, The Idea Center.
+
University of Cincinnati, Center for the Enhancement of Teaching & Learning
Anatomy of an Assessment Plan
Program Outcomes that can be measured, assessed, or observed in some way
Curriculum/Program Map connections identified between courses and program-level outcomes as well as clear levels of learning at each level (emerging, developing, achieved)
Methods/Measured articulated, which include direct & indirect examples of student learning
Program Outcomes
Curriculum/Program
Map
Methods/Measures
Assessment Infrastructur
e
Findings
Use of Findings
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Anatomy of an Assessment Plan, Continued
Assessment Infrastructure comprised of program faculty charged with overseeing the execution of the assessment plan, with efforts linked to college and institutional assessment efforts & goals
Findings for several years explained, patterns and trends identified (data collected & reviewed on an annual basis)
Use of Findings discussed among faculty; learning outcomes and pedagogical strategies reviewed and revised based on assessment data
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+ACTIVITY 1: Envisioning the Ideal Graduate
What does an “ideal” graduate of your program look like?
How will your graduates think and behave?
What are 3-4 important things your students will learn to do in your program?
What do students in your program need to learn to prepare them for a job?
What would you like your students to be doing consistently 5 years from now?
The Task: Brainstorm the “ideal” graduate of your program. Be sure to describe how you imagine your ideal graduates once they become professionals in their field.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+ACTIVITY 1: Continued
If these category listings are not applicable to your program, please insert alternative headings.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Program Outcomes
Outcomes are achieved results or consequences of what was learned (i.e. evidence that learning took place).
Outcomes should be framed in terms of the program and not individual courses or students.
Outcomes are student-centered and describe what a student should be able to do or understand upon graduation.
Outcomes should describe the abilities, knowledge, values and attitudes expected of students after completion of the program.
Outcomes must be stated in such a way that they can be measured or observed by more than one assessment method.
+
University of Cincinnati, Center for the Enhancement of Teaching & Learning
Program Outcomes
Course Outcomes
Unit Outcomes
Class Outcomes
How Program Outcomes Fit Within a Curriculum
Represent broad statements incorporating areas of inter-related knowledge and skills developed throughout a program, across courses and experiences.
Represent the measurable and observable knowledge, skills, abilities, or attitudes students should have by the end of the course. Course outcomes should align with the program outcomes.
Represent the measurable and observable knowledge, skills, abilities, or attitudes students should have by the end of the unit. Unit outcomes should align with the course outcomes.
Represent the knowledge, skills, abilities, or attitudes that students should achieve by the end of an individual class session. Class outcomes should align with the unit outcomes.
+
University of Cincinnati, Center for the Enhancement of Teaching & Learning
Course Outcomes
How Program Outcomes Fit Within a Curriculum
Represent the measurable and observable knowledge, skills, abilities, or attitudes students should have by the end of the course. Course outcomes should align with the program outcomes.
+
Unit Outcomes
Class Outcomes
How Program Outcomes Fit Within a Curriculum
Represent the measurable and observable knowledge, skills, abilities, or attitudes students should have by the end of the unit. Unit outcomes should align with the course outcomes.
Represent the knowledge, skills, abilities, or attitudes that students should achieve by the end of an individual class session. Class outcomes should align with the unit outcomes.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Unit & Class-Level Outcomes
Move from content covered to student action
Consider how in-class activities can model or reinforce learning outcomes
Make course calendar more explicitly supportive of learning outcomes
+
University of Cincinnati, Center for the Enhancement of Teaching & Learning
Program Outcomes
How Program Outcomes Fit Within a Curriculum
Represent broad statements incorporating areas of inter-related knowledge and skills developed throughout a program, across courses and experiences.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+What are Program Outcomes?
Program outcomes represent broad statements that incorporate many areas of inter-related knowledge and skills developed over the duration of the program through a wide range of courses and experiences.
They represent the big picture, describe broad aspects of behavior, and encompass multiple learning experiences.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Program Outcomes…
are achieved results or consequences of what was learned (i.e. evidence that learning took place)
should be framed in terms of the program and not individual courses or students
are student-centered and describe what a student should be able to do or understand upon graduation
should describe the abilities, knowledge, values and attitudes expected of students after completion of the program
must be measurable or observable by more than one assessment method
+
Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning
Bloom’s Taxonomy Action Verbs
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Program Outcomes, Examples: Undergraduate
Examples: Too generic: Students completing the Engineering program
will be practiced in design skills. Specific/Assessable: Engineering graduates will
demonstrate knowledge of math, science, and engineering fundamentals. Specifically, the student will have the ability to:• Demonstrate general design principles.• Use fundamental engineering techniques, skills and tools
for engineering practice.• Analyze and interpret data to produce meaningful
conclusions and recommendations.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+
Bachelor Degree in Business (BBA) Upon completion of the BBA Program, students will be able to:
1. Apply functional and cross-functional knowledge to critically assess business problems
2. Use analyses to inform and develop integrative solutions that improve business outcomes
3. Express ideas clearly, logically and persuasively in both oral and written formats
4. Recognize ethical and social responsibility issues in a business environment and know how to apply a process of ethical inquiry
5. Show how operating in a global market creates business opportunities and challenges
Program Outcomes: Undergraduate
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Program Outcomes: Undergraduate
Bachelor Degree in Health Science (HLSC):
Upon completion of the HLSC Program, students will be able to:
1. Plan, execute, evaluate, adjust physical activities and programs appropriate for self and clients
2. Provide complete, understandable and accurate information within content areas to clients and community
3. Develop and/or disseminate scientific information to the general public and the academic community
4. Apply information from various basic and applied science disciplines in a manner that provides for efficient, effective and safe physical activities
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Program Level Outcomes Categories for Graduate Level
Need to be progressively more advanced in academic content
Need to address discipline knowledge and research or professional practice
Need to address technology skill development (if appropriate)
Need to be informed & aligned with professional organization standards, licensing bodies, or specialized accreditors
Professional development of graduate students
Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning
+Example Student Learning Outcomes at the Graduate LevelBEFORE:
Develop knowledge across the subdisciplines of physics beyond that expected for a baccalaureate degree in physics.
Develop expertise in an area of physics that is appropriate to the doctoral level.
Develop and demonstrate the ability to carry through a substantial research project, with independent intellectual contributions to the conduct and direction of the research.
Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning
+Example Student Learning Outcomes at the Graduate LevelAFTER:
Upon completion of the physics PhD program, students will be able to: identify and describe the broad fundamental concepts of physics; analyze physics problems using fundamental physics concepts and
appropriate mathematical models; solve physics problems using analytic, computational, observational, or
experimental methods; present clearly organized oral explanations of physics topics at a level
appropriate to the audience at hand; explain and analyze at an expert level the concepts and the state of
current research in a subfield of physics; compose and defend written presentations of physics research in a
format and at a level appropriate for journal publication; conceptualize, design, and implement new approaches for solving
research problems in physics, including analytic, computational, observational, and experimental methods, as appropriate.
Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning
+Example Student Learning Outcomes at the Graduate Level Professional Development
Ethics
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Activity 2: Define Student Learning Outcomes
Focused on the learner, rather than on the content covered
Attainable, rather than aspirational
Describe observable and demonstrable skills and knowledge
Reflect how the learner will use course content now and in the future
Indicate specific and measurable elements that will be assessed using multiple assessment methods
Identify in the curriculum where the PO will be introduced, developed and achieved
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+ACTIVITY 2b: Peer Review of Program Outcomes
Gallery Walk
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Program Outcomes and Your Program Assessment Plan PO’s are the foundation of an assessment plan
Program Outcomes
Course Outcomes
Unit Outcomes
Class Outcomes
Program Outcomes
Curriculum/Program
Map
Methods/Measures
Assessment Infrastructur
e
Findings
Use of Findings
The Process of
Assessment
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Activity 3: Curriculum Mapping Matrix
To identify where key learning outcomes are addressed in the curriculum
To identify whether courses are aligned with the curriculum
Specifically, to determine/identify whether course-level student learning outcomes align with program-level outcomes
Note: You’ve already completed the process of mapping course-specific student learning outcomes to program outcomes, when you completed the P-1.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Activity 3: ContinuedInstructions
List the program learning outcomes identified in the P-1 in the column labeled “Program Learning Outcomes.”
List the required courses and experiences identified in the P-1 across the top row.
NOTE: Do not list courses or experiences controlled by external academic units. Only list courses and experiences that
are controlled by your program. If you identify a program outcome, you must be able to assess it.
Identify which course(s) and/or experiences utilize each program learning outcome and at what level the program learning outcome is addressed (e.g. emerging, developing, achieved)
Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning
Curriculum Mapping Matrix: Linking Program Outcomes to Curriculum
Key Required Courses and Experiences Identified in P-1
E: EmergingD: DevelopingA: Achieved
A101 A202
A502 …. … …
OUTCOMES
1 Outcome 1 E D A
2 Outcome 2 E D A
3 Outcome 3 EDA
4
5
6
7
8
9
Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Identify & Describe Corresponding Program Outcomes & Assessment Methods: Conduct a Preliminary Audit
Circle any of the measures used in your academic program
+Why Not Just Use Course Grades?
+Why aren’t course grades adequate indicators for program assessment?
Course letter grades are insufficient for program assessment because: Letter grades are nominal values reflecting a summary of a
student’s performance. Grades are not an exact measurement of what students
have and have not learned. Grades do not measure achievement of program outcomes,
learning of specific skills, or the capacity of the curriculum to achieve program outcomes.
+Example: Outcomes for an Undergraduate Economics Program
Students should be able to identify, explain, and use economic concepts, theories, models; and data-analytic techniques.
Students should be able to deliver effective presentations in which they combine visual communication design with oral arguments and/or the written word.
Students should use investigative skills necessary for conducting original economic research and participating effectively in project teams.
Students should be able to apply their economic tools to formulate positions on a wide range of social and economic problems and engage effectively in policy debates.
+Example Curriculum Map for an Undergraduate Economics Program
E101
E202
E303
E404
1. Students should be able to identify, explain, and use economic concepts, theories, models; and data-analytic techniques
E D A
2. Students should be able to deliver effective presentations in which they combine visual communication design with oral arguments and/or the written word.
E D A
3. Students should use investigative skills necessary for conducting original economic research and participating effectively in project teams.
E D A
4. Students should be able to apply their economic tools to formulate positions on a wide range of social and economic problems and engage effectively in policy debates
E D D A
+Letter grades are nominal values reflecting a summary of a student’s performance
Measurements of Learning in E303
Grade Points Possible
Exam 78 100
Assignment 18 20
Quiz 10 50
Oral Presentation 98 100
Exam 88 100
Attendance 10 20
Total 302 390
Course Grade: 77% C
+PO 1. Students should be able to identify, explain, and use economic concepts, theories, models; and data-analytic techniques
Measurements of Learning in E303
Grade Points Possible
PO 1 PO 2
Exam 70 100
Assignment 15 20
Quiz 10 50
Oral Presentation 98 100
Exam 82 100
Attendance 10 20
Total 285 390 167/220
98/100
Course Grade: 73% C 76% 98%
PO 2. Students should be able to deliver effective presentations in which they combine visual communication design with oral arguments and/or the written word.
+PO 2: Students should be able to deliver effective presentations in which they combine visual communication design with oral arguments and/or the written word.
The oral presentation assignment is worth 25% of the course grade.
It is possible for one student to get an exceptional grade on that assignment, and another to fail, with both students receiving a “C” for the final course grade.
The OVERALL course grade does not measure: students’ demonstrated
oral presentation skills the capacity of the
curriculum to achieve PO2
Measurements of Learning
in E303
Student 1
Student 2
Points Possible
Exam 70 70 100
Assignment 15 12 20
Quiz 10 40 50
Oral Presentation
98 50 100
Exam 82 98 100
Attendance 10 15 20
Total 285 285 390
Course Grade:
73% 73% C
+Why aren’t course grades adequate indicators for program assessment?
Course letter grades are insufficient for program assessment because: Grades reflect the evaluation practices, policies, and
criteria of individual instructors. Faculty teaching the same course may teach different
material. Faculty teaching the same course may emphasize different
course outcomes.
+Grades reflect the evaluation practices, policies, and criteria of individual instructors
Measurements of Learning in E303
Points Possible
Exam 100
Assignment 20
Quiz 50
Oral Presentation 100
Exam 100
Participation 20
Total 390
Measurements of Learning in E303
Points Possible
Exam 100
Assignment 20
Quiz 50
Oral Presentation 100
Exam 100
Attendance 20
Participation 20
Quiz 50
Total 460
Instructor 1 Instructor 2
Difference of 70 points between two sections of the
same course!
+Why aren’t course grades adequate indicators for program assessment?
Course letter grades are insufficient for program assessment because: Letter grades are nominal values reflecting a summary of a
student’s performance. Grades are not an exact measurement of what students have
and have not learned. Grades do not measure achievement Program outcomes,
learning of specific skills or the capacity of the curriculum to achieve PO2
Grades reflect the evaluation practices, policies, and criteria of individual instructors.
Faculty teaching the same course may teach different material. Faculty teaching the same course may emphasize different
course outcomes.
+What CAN Be Used for Assessment?
Specific grades that reflect a specific program outcome An exam Assignment Oral presentation
Then the corresponding scores can be used to represent students' learning achievement on the program outcome, which can serve as data for program assessment.
+Conditions for Using Course Products as Assessment Measures:1. All sections of the required course must produce the course
product (e.g. an oral presentation)
2. Faculty must agree on product structure (e.g. directions, requirements)
3. Faculty must agree on how the products should be evaluated: Minimum performance standards Rubric
Sect
ion
1
Sect
ion
2
Sect
ion
3
Sect
ion
4
Sect
ion
5
Sect
ion
6
Sect
ion
7
Sect
ion
80
40
8077 80 71
8869 75 85 77
Average Grade on Standard Oral Presentation
Average Program Score: 78/100
+Final Message
Course products (cases, papers, presentations, exercises) may be used as program assessment measures, but course grades, by
themselves, cannot.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Activity 4: Conduct a Preliminary Audit of Assessment Methods Utilized in Your Program
Using the “Program Assessment Inventory” (also available on the CET&L Website), read through the assessment measures and their associated descriptions. Check off any of the measures currently used in your academic program.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Activity 5: Complete the Assessment Measures Alignment Matrix
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Example Assessment Measures Alignment Matrix
Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning
+Activity 5: Complete the Assessment Measures Alignment Matrix1. List your revised program learning outcomes in Assessment
Measures Alignment Matrix on the following page in the column labeled “Program Learning Outcomes”.
2. Using the Program assessment measures identified in the previous step, match the assessment tool to the Program outcome (please note that that same assessment may be used for multiple program outcomes).
3. Enter these in the table on the following page under the column labeled “Assessment Methods” and next to the program outcome to which they apply.
4. For each assessment measure listed describe in the adjacent columns where (what courses), when/how often the data will be collected (timeline), and who is responsible for collecting, reviewing and making recommendations about the data.
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Activity 6: Developing Your Assessment Infrastructure
Assessment Personnel Who will chair/serve on the assessment committee? What is the responsibility of the committee?
Collecting data, submission process and form Review of data (recommended annually) Recommendation of findings and decimation/discussion of findings/recommendations Collection of course syllabus/discussion with individual faculty regarding required courses in
the curriculum Recognition of service-load
Assessment Schedule How often will data be collected When will faculty be reminded that data will be collected (recommended beginning of term) Remind faculty to submit data at end of term Schedule of data compilation, summary and review by PC Timeline for when these findings with be shared & discussed with department faculty Follow-up discussion with faculty teaching required courses and/or overseeing required
experiences Timeline for improvement strategies identified (both short and long-term)
Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning
+Looking Forward: An Example Implementation of an Assessment Plan
University of Cincinnati, Center for the Enhancement of Teaching & Learning
+Homework: Revise your plan to gather, analyze, and interpret assessment data
Complete the “Academic Program Assessment Template,” using the following resources: Program outcomes Curriculum mapping matrix Assessment Measures Alignment Matrix Developing your assessment infrastructure
Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning
+Tomorrow
Activity 7: Peer evaluation of drafted program assessment plan