Refining Your Assessment Plan: Session 2
Direct Evidence of Student LearningRyan Smith and Derek HerrmannUniversity Assessment Services (UAS)
IntroductionsRefining Your Assessment Plan: Session 2
Introductions• Ryan Smith, Director, UAS• Derek Herrmann, Coordinator, UAS• Assessment Advisory Council (AAC)• Assessment Academy Team
Introductions• Who you are• In what Department/School you work• Why you are here
Overview of PRAAPRefining Your Assessment Plan: Session 2
Overview of PRAAP• Process for the Review of Academic
Assessment Plans• Two years before Program Review (and
one year before Self-Study)• AAC members and UAS staff• Use an established rubric
Overview of PRAAPElements Undeveloped Developing Established Exemplary
Goals and Learning OutcomesDirect Evidence of Student LearningIndirect Evidence of Student LearningUse of the Results
Direct Evidence of Student Learning
Refining Your Assessment Plan: Session 2
Direct Evidence of Student Learning• Tangible, visible, self-explanatory, and
compelling evidence of exactly what students have and have not learned
Direct Evidence of Student Learning• Performance assessments and
traditional assessments– Traditional assessments
• Multiple-choice tests, essay tests, and oral examinations
• Usually designed only to collect assessment information, not give students a learning opportunity
Direct Evidence of Student Learning• Performance assessments and
traditional assessments– Performance assessments
• Ask students to demonstrate their skills rather than relate what they have learned through traditional tests
• Increasingly popular because they merge learning and assessment
Direct Evidence of Student Learning• Embedded and add-on assessments
– Embedded assessments – program, general education, or institutional assessments that are embedded into course work
– Add-on assessments – beyond course requirements
Direct Evidence of Student Learning• Quantitative and qualitative assessments
– Quantitative assessments – use structured, predetermined response options that can be summarized into meaningful numbers and analyzed statistically
– Qualitative assessment – use flexible, naturalistic methods and are usually analyzed by looking for recurring patterns and themes
Direct Evidence of Student Learning• Effective assessment formats
– Paper/thesis – students develop an argument and support it with information or data they have gathered
– Project – students complete an assignment over a prolonged period of time
– Development of a product – project whose focus is on the development of a tangible product
Direct Evidence of Student Learning• Effective assessment formats (cont’d)
– Performance – students prepare and present a performance of a valued activity
– Exhibition – a project, product, or performance that is presented to judges and defended or debated with them
– Case study/critical incident – students are given a realistic example of an application in their field
Direct Evidence of Student Learning• Effective assessment formats (cont’d)
– Clinical evaluation – students perform a professional service in a real-life setting
– Oral exam – students answer spontaneous questions put to them by experts
– Interview – similar to an oral exam but the forum in which it is carried out may not be as public or involve as many questioners as an oral exam
Direct Evidence of Student Learning• Effective assessment formats (cont’d)
– Comprehensive exam – students complete a time-limited essay test that requires them to organize and present central ideas, facts, and concepts in response to questions
– Portfolio – students gather examples of their work, write about aspects of their learning and achievement, and include their written reflections
Direct Evidence of Student Learning• Effective assessment formats (cont’d)
– Portfolios• Showcase portfolio – document the extent of learning by
featuring the student’s best work• Developmental portfolio – designed to show student
progress and includes evidence of growth by comparing products from early and late stages of the student’s academic career
• Collective portfolio – collections of student work that are created by faculty for assessment purposes
Curriculum Mapping (Part 2)Refining Your Assessment Plan: Session 2
Curriculum Mapping• Analyze learning opportunities by
mapping learning goals against courses
Curriculum MappingCourse Goal 1 Goal 2 Goal 3 Goal 4
111 X X
138 X X
200 X X
231 X X X
331 X X
340 X X
392 X X X X
Curriculum MappingCourse Goal 1 Goal 2 Goal 3 Goal 4
111 Paper/thesis Paper/thesis
138 Paper/thesis Paper/thesis
200 Project Project
231 Performance Performance Performance
331 Exhibition Exhibition
340 Case study/critical incident
Case study/critical incident
392 Clinical evaluation Clinical evaluation Clinical evaluation Clinical evaluation
Curriculum Mapping• Transcript analysis – review a sample of
transcripts of graduating students to learn courses they choose and when they take them
Curriculum Mapping• Syllabus analysis – can help determine
if students have enough assignments and classwork to achieve each of the intended learning goals of the course, including the program, general education, or institutional goals that course is intended to support
RubricsRefining Your Assessment Plan: Session 2
Rubrics• A rubric is a scoring guide: a list or chart
that describes the criteria you and perhaps your colleagues will use to evaluate or grade completed student assignments
Rubrics• Using a rubric to grade student
assignments…– Makes your life easier– Improves student learning
Rubrics• A checklist rubric is
a simple list indicating the presence of the things you’re looking for in a completed assignment
Criteria PresentCriterion 1
Criterion 2
Criterion 3
Criterion 4
Criterion 5
Rubrics• A rating scale rubric
is a checklist with a rating scale added to show the degree to which the things you’re looking for are present in completed assignments
Criteria Not Present Developing Established Advanced
Criterion 1
Criterion 2
Criterion 3
Criterion 4
Criterion 5
Rubrics• Descriptive rubrics
replace the checkboxes of rating scale rubrics with brief descriptions of the performances that merit each possible rating
Criteria Not Present Developing Established Advanced
Criterion 1Description of
the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Criterion 2Description of
the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Criterion 3Description of
the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Criterion 4Description of
the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Criterion 5Description of
the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Description of the performance that merits this
rating
Rubrics• Creating effective rubrics
– Look for models– List the things you’re looking for– Leave room for the ineffables and the
unexpected
Rubrics• Creating effective rubrics (cont’d)
– Create the rating scale• Create at least three levels• Label each level with names, not just numbers• If you are developing a descriptive rubric, fill in
its boxes– Try out the rubric
Questions, comments, and discussion
Refining Your Assessment Plan: Session 2