Post on 26-Dec-2015
transcript
THE ROLE OF CALIBRATION IN ADVANCING FACULTY LEARNING
ABOUT STUDENT LEARNING
Terry Rhodes & Ashley Finley
AAC&U
Institute on Integrative Learning and the Departments
Portland, OR
“OUR” COMMON GROUND:
Criteria
The Anatomy of a VALUE RubricThe Anatomy of a VALUE Rubric
Levels
Performance Descriptors
THE CALIBRATION TRAINING PROCESS Scoring Steps:
Review rubric to familiarize yourself with structure, language, performance levels
Ask questions about the rubric for clarification or to get input from others regarding interpretation
Read student work sample Connect specific points of evidence in work sample
with each criterion at the appropriate performance level (if applicable)
Calibration Steps: Review scores Determine common score(s) Hear from outliers Discuss Determine final score
THE GROUND RULES
This is not grading. Think globally about student work and about
the learning skill. Think beyond specific disciplinary lenses or content.
We are not changing the rubric (today). Our work is time sensitive. Go with your
instinct. Start with 4 and work backwards. Pick one
performance benchmark per criterion. Avoid “.5”.
Zero does exist. Assign “0” if work does not meet benchmark (cell one) performance level. N/A exists. Assign “not applicable” if the student work is not intended to meet a particular criterion.
SIGNATURE ASSIGNMENTS Assignment should enable attainment of
criteria Break down criteria to determine key components for
assignment
What should students do with content to meet criteria? E.g. What are the pieces to be analyzed, compared,
integrated?
Will the assignment be used for more than one outcome?
What are the types of assignments that will be most helpful for allowing students to demonstrate competency?
EXAMPLE OF PROCESS
Step 1: All Gen Ed Courses reported as addressing and assessing Info. Tech. Literacy identified as potential courses from which to request artifacts. (54 courses)
Step 2: Of courses identified, approx. 20% were randomly selected for sample (10 courses, 36 total sections)
Step 5: Artifacts submitted to Director of Learning Outcomes for scoring. (66 artifacts)
Step 4: Start of semester, department chairs notified of courses in from which artifacts were to be requested. Chairs worked with individual faculty to fulfill request.
Step 3: Within each selected course, 2 students randomly selected by roster # to submit artifacts (74 artifacts)
Step 6: Faculty scoring team met at the close of spring semester for a norming session and scoring. (62 artifacts)
From: Carroll Community College
Flow chart of sequential steps in the request, submission, and scoring of student artifacts for Learning Goal 4: Information and technology literacy.
CAMPUS EXAMPLE OF OUTCOMES ASSESSMENT USING RUBRIC DATA
Dimension % of students who scored 2 or higher
% of students who scored 3 of higher
Explanation of Issues 68.3 35.5
Interpreting & Analysis
65.0 28.2
Influence of Context and Assumptions
48.8 21.2
Student’s position 54.5 24.0
Conclusions and related outcomes
47.7 17.0
From: UNC-Wilmington, Critical Thinking Rubric
BUILDING THE EVIDENTIARY BASE: UNIVERSITY OF KANSAS
Per
cen
t o
f R
atin
gs
Critical Thinking: Issues, Analysis, and Conclusions
Inter-rater reliability = >.8
BUILDING THE EVIDENTIARY BASE: UNIVERSITY OF KANSAS
Per
cen
t o
f R
atin
gs
Critical Thinking: Evaluation of Sources and Evidence
BUILDING THE EVIDENTIARY BASE: UNIVERSITY OF KANSAS
Per
cen
t o
f R
atin
gs
“VALUE added” for 4 years - writing