Rubrics and feedback in a diverse classroom Major Research Paper – Major Research Paper – M.A in Applied M.A in Applied Linguistics – York U. Linguistics – York U. Jessica King Jessica King Copyright Jessica King 2010. This work is the intellectual property Copyright Jessica King 2010. This work is the intellectual property of the author. Permission is granted for this material to be of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. notice is given that the copying is by permission of the author.
Transcript
Rubrics and feedback in a diverse classroom
Major Research Paper – M.A in Major Research Paper – M.A in Applied Linguistics – York U.Applied Linguistics – York U.
Jessica KingJessica KingCopyright Jessica King 2010. This work is the intellectual property of the author. Permission Copyright Jessica King 2010. This work is the intellectual property of the author. Permission
is granted for this material to be shared for non-commercial, educational purposes, is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to is given that the copying is by permission of the author. To disseminate otherwise or to
republish requires written permission from the authorrepublish requires written permission from the author
Presentation outline
1.1. Motivation for projectMotivation for project
2.2. Context for researchContext for research
3.3. Research QuestionsResearch Questions
4.4. Theoretical BackgroundTheoretical Background
5.5. MethodologyMethodology
6.6. ResultsResults
7.7. RecommendationsRecommendations
8.8. BibliographyBibliography
Why did I do this project?
Context for research – C101 Foundation level writing course for most diploma programs Foundation level writing course for most diploma programs
and prerequisite for more advanced communications and prerequisite for more advanced communications coursescourses
Duration – 42 hours (usually 3.5 months)Duration – 42 hours (usually 3.5 months) Both NES and NNESBoth NES and NNES Revision of C101 started in 2006 – creation of “Standard-Revision of C101 started in 2006 – creation of “Standard-
Varied College English curriculum”Varied College English curriculum” 2 major assignments with 2 different rubrics2 major assignments with 2 different rubrics Process approach to writing instructionProcess approach to writing instruction Implemented across the College in Fall 2007Implemented across the College in Fall 2007
Context for research – the rubrics• A1: 1000 word opinion response to a readingA1: 1000 word opinion response to a reading
• A2: 1250 word research essay using APA refA2: 1250 word research essay using APA ref– Criteria: textual analysis replaced with Criteria: textual analysis replaced with
development & supporting evidence; grammar & development & supporting evidence; grammar & mech expanded to include referencing mech expanded to include referencing
• Analytic method with holistic scoringAnalytic method with holistic scoring
Research questions
Teacher Perceptions of Rubrics and FeedbackTeacher Perceptions of Rubrics and Feedback• Do the teachers find these rubrics useful? If Do the teachers find these rubrics useful? If
so, in what ways are they useful?so, in what ways are they useful?• What are the functions of the rubrics for the What are the functions of the rubrics for the
teachers?teachers?
Theoretical background and relevant research The mature writing process: The mature writing process:
knowledge telling vs. knowledge telling vs. Knowledge Knowledge transformingtransforming. . (Scardamalia & Bereiter 1987)(Scardamalia & Bereiter 1987)
Hayes et al’s (1987) Process model of Hayes et al’s (1987) Process model of revisionrevision
Theoretical Background and Relevant Research The teacher’s role in student revisionThe teacher’s role in student revision
Students DO read feedback Students DO read feedback (Ferris, 2003)(Ferris, 2003)
Varied feedback works best * Varied feedback works best * (Bitchener et al (Bitchener et al 2005; Goldstein, 2004; Ferris, 2003)2005; Goldstein, 2004; Ferris, 2003)
Debate still exists about usefulness of Debate still exists about usefulness of grammar correction grammar correction (Truscott, 2007; Ferris, 2004)(Truscott, 2007; Ferris, 2004)
Theoretical Background and Relevant Research Assessment Assessment
Rubrics can be a source of feedbackRubrics can be a source of feedbackBasis for student self-assessmentBasis for student self-assessmentProvides clear explanation of what Provides clear explanation of what
makes a good paper makes a good paper (Andrade, 2000)(Andrade, 2000)
Theoretical Background and Relevant ResearchAdvantagesAdvantages
More robust profile of student achievement More robust profile of student achievement (Andrade, (Andrade, 2000)2000)
Acknowledges unequal development of sub-skillsAcknowledges unequal development of sub-skills
DisadvantagesDisadvantages
May slow down marking process May slow down marking process (Hughes, 2003)(Hughes, 2003)
Interaction with teacher’s internal criteria Interaction with teacher’s internal criteria (Barkaoui, 2007; (Barkaoui, 2007; Hamp-Lyons, 1995)Hamp-Lyons, 1995)
Not easily transferable to different assignments or Not easily transferable to different assignments or students students (Hamp-Lyons & Henning, 1991)(Hamp-Lyons & Henning, 1991)
Methodology
Participants:Participants: N = 8N = 8 4 contract, 4 full-time status4 contract, 4 full-time status All had taught new C101 for at least one semesterAll had taught new C101 for at least one semester 3 different departments: CSHS, GEA, HOSP3 different departments: CSHS, GEA, HOSP
Methodology
Interview structure:Interview structure: IntroductionIntroduction Teacher feedbackTeacher feedback Teacher perception of rubricsTeacher perception of rubrics
Methodology
Study Limitations:Study Limitations: No analysis of actual feedback given on student texts No analysis of actual feedback given on student texts
– only had teachers’ description of their practice– only had teachers’ description of their practice But focus of interview was not on “best practices”But focus of interview was not on “best practices”
Different teacher schedules in summerDifferent teacher schedules in summer But all teachers had taught C101 beforeBut all teachers had taught C101 before
Smaller classes in summerSmaller classes in summer Result: sample of students (N=5) not Result: sample of students (N=5) not
representative of actual student experiencerepresentative of actual student experience
Results
Tina Steve Pam Lucy Ken Kathy Karen Brian
Yrs teach 7 5 5 16 12 9 20 12
Yrs teach C101
2.5 5 5 0.5 1 9 2 0.5
ESL exp.
EFL, ESL no no
ESL in coll.
ESL lit., LINC
ESL Writ. class
ESL writ. In college
TESL cert. ES/FL
Yrs using rubrics
7 5 2“on and off”
2 Used criteria sheets
20 5
Results - Feedback
Teachers used 5 different types of feedback: Teachers used 5 different types of feedback: between-the-lines edit, marginal comments, end between-the-lines edit, marginal comments, end comments, rubrics, oral conferencescomments, rubrics, oral conferences
Time spent on first and final drafts: mean = 20 mins Time spent on first and final drafts: mean = 20 mins for each.for each.
One teacher did not give feedback for first draftOne teacher did not give feedback for first draft Half the teachers used rubrics to provide feedback to Half the teachers used rubrics to provide feedback to
students between drafts (in conjunction with other students between drafts (in conjunction with other kinds of feedback)kinds of feedback)
Results – Rubric use
Most (n=7) teachers explain rubrics to students. Most (n=7) teachers explain rubrics to students. All explained rubrics by leading class through an All explained rubrics by leading class through an
explanation of the rubrics. Some gave a short explanation of the rubrics. Some gave a short description, some a more involved overviewdescription, some a more involved overview
Results – Rubric use
All teachers stated rubrics assisted them when All teachers stated rubrics assisted them when giving feedback. Rubrics assisted them in 4 ways:giving feedback. Rubrics assisted them in 4 ways:
as guidelines or lists of benchmarks (n=3)as guidelines or lists of benchmarks (n=3) as justification or evidence for grades given as justification or evidence for grades given
(n=2)(n=2) as communication tools between teachers as communication tools between teachers
and students (n=3)and students (n=3) as increasing objectivity or reducing as increasing objectivity or reducing
subjectivity in evaluation (n=2)subjectivity in evaluation (n=2)
Results – Rubric Use
Did the teachers feel the rubrics restricted their written Did the teachers feel the rubrics restricted their written feedback? No (N=4), Yes (N=4)feedback? No (N=4), Yes (N=4)
Did the teachers feel that the rubric standards were Did the teachers feel that the rubric standards were appropriate for their students in general? Yes (N=5), No appropriate for their students in general? Yes (N=5), No (N=3)(N=3)
Did the teachers find standards appropriate for NNES Did the teachers find standards appropriate for NNES students? Yes (N=2), No (N=3), unable to answer students? Yes (N=2), No (N=3), unable to answer (N=3)(N=3)
Did teachers think there were specific L2 writing Did teachers think there were specific L2 writing problems that were not dealt with in the rubric? Yes problems that were not dealt with in the rubric? Yes (N=5) – grammar & mechanics, style sections(N=5) – grammar & mechanics, style sections
Results – Rubric Use
Did teachers make modifications to the rubrics? Yes Did teachers make modifications to the rubrics? Yes (N=8)(N=8) (N=3) made minor changes – cosmetic, not (N=3) made minor changes – cosmetic, not
affecting meaningaffecting meaning (N=2) made medium-level changes – vocabulary(N=2) made medium-level changes – vocabulary (N=4) made major changes – altering or adding to (N=4) made major changes – altering or adding to
criteriacriteria
Results - Rubrics
What did teachers feel were the advantages to using rubrics?What did teachers feel were the advantages to using rubrics? Made teacher expectations clear, reinforced college Made teacher expectations clear, reinforced college
standards (N=5)standards (N=5) Increased objectivity and appearance of fairness (N=2)Increased objectivity and appearance of fairness (N=2) Saves time (N=2)Saves time (N=2)
What did teachers feel were the disadvantages to using What did teachers feel were the disadvantages to using rubrics?rubrics? Rubrics differed from their ideas of good essays (N=4)Rubrics differed from their ideas of good essays (N=4) Rubrics are too open to interpretation (N=3)Rubrics are too open to interpretation (N=3) Rubrics are too time consuming (N=1)Rubrics are too time consuming (N=1)
Results
Dominant Themes:Dominant Themes: Student and teacher interpretation of rubric language Student and teacher interpretation of rubric language
– clarity of feedback and objectivity in grading– clarity of feedback and objectivity in grading Accessibility of language for studentsAccessibility of language for students Teacher interpretation of languageTeacher interpretation of language
Discrepancies between teachers’ general Discrepancies between teachers’ general impressions and rubric scoresimpressions and rubric scores
Recommendations
Concentrate feedback on first drafts - formative Concentrate feedback on first drafts - formative feedbackfeedback (Ferris, 2003)(Ferris, 2003)
Consider varied feedback - mix of oral, written, rubric Consider varied feedback - mix of oral, written, rubric useuse (Ferris, 2003; Leki, 2007, Bitchener et al 2005)(Ferris, 2003; Leki, 2007, Bitchener et al 2005)
Promote student understandingPromote student understanding Explain rubrics as early as possible – use Explain rubrics as early as possible – use
Cognitive Modeling Cognitive Modeling (Cumming, 1986; Dobson & Feak, (Cumming, 1986; Dobson & Feak, 2001; Scardamalia et al 1984)2001; Scardamalia et al 1984)
Portfolio assessments as in Wilhelm (1996) use in Portfolio assessments as in Wilhelm (1996) use in EAP classEAP class
Recommendations
Promote teacher understanding (if rubrics are created by Promote teacher understanding (if rubrics are created by college)college)
Teacher training on rubrics Teacher training on rubrics (Barkaoui, 2007)(Barkaoui, 2007) Longer text explanation of rubrics supplied to teachersLonger text explanation of rubrics supplied to teachers Empower teachers to recommend changesEmpower teachers to recommend changes
Promote mutual understanding – review wording of criteria Promote mutual understanding – review wording of criteria with NES and NNES students in mind (grammar/ with NES and NNES students in mind (grammar/ mechanics and style sections)mechanics and style sections)
Revisit “problem” terms: rhetorical mode, diction, Revisit “problem” terms: rhetorical mode, diction, subordination, modifiers, lack of awareness of subordination, modifiers, lack of awareness of audience, obscuring meaning, structures are variedaudience, obscuring meaning, structures are varied
Bibliography Andrade, G. H. (2000). Using rubrics to promote thinking and learning. Andrade, G. H. (2000). Using rubrics to promote thinking and learning. Educational LeadershipEducational Leadership, 13-18., 13-18.
Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study, Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study, Assessing Writing,Assessing Writing, 12, 86-107. 12, 86-107.
Bereiter, C., & Scardamalia M. (1987).Bereiter, C., & Scardamalia M. (1987). The Psychology of Written Composition The Psychology of Written Composition. Hillsdale, NJ: Lawrence Erlbaum . Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.Associates, Inc.
Bitchener, J., Young, S., & Cameron, D. (2005). The effect of different types of corrective feedback on ESL student Bitchener, J., Young, S., & Cameron, D. (2005). The effect of different types of corrective feedback on ESL student writing. writing. Journal of Second Language WritingJournal of Second Language Writing, 14, 191-205., 14, 191-205.
Cumming, A. (1986). Intentional learning as a principle for ESL writing instruction: A case study. In P.M. Lightbown & S.P. Cumming, A. (1986). Intentional learning as a principle for ESL writing instruction: A case study. In P.M. Lightbown & S.P. Firth (Eds.) Firth (Eds.) TESL Canada Journal: Both Sides of the Desk: Roles and Responsibilities in ESL/EFL Teaching and TESL Canada Journal: Both Sides of the Desk: Roles and Responsibilities in ESL/EFL Teaching and LearningLearning. 69-83.. 69-83.
Dobson, B. & Feak, C. (2001). A cognitive modeling approach to teaching critique writing to non-native speakers. In D. Dobson, B. & Feak, C. (2001). A cognitive modeling approach to teaching critique writing to non-native speakers. In D. Belcher & A. Hirvela (Eds.). Belcher & A. Hirvela (Eds.). Linking Literacies: Perspectives on L2 Reading-Writing Connections. Linking Literacies: Perspectives on L2 Reading-Writing Connections. (pp. 186-199). Ann (pp. 186-199). Ann Arbor: The University of Michigan Press.Arbor: The University of Michigan Press.
Ferris, D. R. (2003). Responding to writing. In B. Kroll (Ed.). Ferris, D. R. (2003). Responding to writing. In B. Kroll (Ed.). Exploring the Dynamics of Second Language Writing Exploring the Dynamics of Second Language Writing (pp. (pp. 119-140). Cambridge: Cambridge University Press.119-140). Cambridge: Cambridge University Press.
Ferris, D. R. (2009). Ferris, D. R. (2009). Teaching college writing to diverse student populations.Teaching college writing to diverse student populations. Ann Arbor: University of Michigan Press. Ann Arbor: University of Michigan Press.
Hamp-Lyons, L. (1995). Rating nonnative writing: The trouble with holistic scoring. TESOL Quarterly, 29, 759-762.Hamp-Lyons, L. (1995). Rating nonnative writing: The trouble with holistic scoring. TESOL Quarterly, 29, 759-762.
Hamp-Lyons, L. & Henning, G. (1991). Communicative writing profiles: An investigation of the transferability of a multiple-Hamp-Lyons, L. & Henning, G. (1991). Communicative writing profiles: An investigation of the transferability of a multiple-trait scoring instrument across ESL writing assessment contexts. Language Learning, 41, 337-373.trait scoring instrument across ESL writing assessment contexts. Language Learning, 41, 337-373.
BibliographyHayes, J., Flower, L., Schriver, K.A., Stratman, J.F., Carey, L. (1987). Cognitive processes in revision. In S. Rosenberg Hayes, J., Flower, L., Schriver, K.A., Stratman, J.F., Carey, L. (1987). Cognitive processes in revision. In S. Rosenberg
(Ed.), (Ed.), Advances in Applied Psycholinguistics Advances in Applied Psycholinguistics (Vol. 2) (pp. 176-240). New York: Cambridge University Press.(Vol. 2) (pp. 176-240). New York: Cambridge University Press.
Hughes, A. (2003). Hughes, A. (2003). Testing for language teachersTesting for language teachers (2nd ed.). Cambridge: Cambridge University Press. (2nd ed.). Cambridge: Cambridge University Press.
Leki, I. (2007, June). Research on feedback to L2 writing, Leki, I. (2007, June). Research on feedback to L2 writing, TESL ContactTESL Contact, 33(2), 7-27. , 33(2), 7-27.
Silva, T. (1993). Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications. Silva, T. (1993). Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications. TESOL QuarterlyTESOL Quarterly, 27, 657-677., 27, 657-677.
Wilhelm, K. H. (1996). Combined assessment model for EAP writing workshop: Portfolio decision-making, criterion-Wilhelm, K. H. (1996). Combined assessment model for EAP writing workshop: Portfolio decision-making, criterion-referenced grading, and contract negotiation, referenced grading, and contract negotiation, TESL Canada JournalTESL Canada Journal, 14(1), 21-33., 14(1), 21-33.
