LARC_CALPER Testing and Assessment Webinar - Rubrics W_ JD Brown

7/29/2019 LARC_CALPER Testing and Assessment Webinar - Rubrics W_ JD Brown

1/18

DEVELOPING AND USING

RUBRICS IN LANGUAGEASSESSMENT

James Dean Brown

University of Hawaii atMnoa


2/18

A. What is a rubric?According to Brown (2012): The termrubrichas existed in English for more than 600 years and, duringsome of that time, it has meant a set of printed rules or instructions (Encarta Encyclopedia, 2004).

However, more recently, rubrichas come to be used widely in education. In the classroom, rubric maymean a set of categories, criteria for assessment, and the gradients for presenting and evaluating learning.When grading a students essay, for example, a teacher may apply a rubric for its quality of organization,giving a 3 for Advanced Proficient, 2 for Proficient, and a 1 for Partially Proficient (Cooper & Gargan,2009, p. 54).A rubric in language teaching is typically a grid set up in one of two ways: (a) with scores along oneaxis of the grid and language behavior descriptors inside the grid for what each score means in terms oflanguage performance or (b) with language categories along one axis and scores along the other axisand language behavior descriptors inside the grid for what each score within each category means interms of language performance.

B. What sorts of assessment benefit most from rubrics?C. How can we create an analytic rubric?D. How can we create a holistic rubric?E. Are there useful online tools that can be used to develop rubrics?F. Are there other examples of rubrics in the literature for say group assessment, other types of

writing assessment, reading fluency, etc.?G. What are some useful referencs for further reading?H. If there is time: What are formal steps should be considered in rubric development?I. If there is time: How might task-dependent and task-independent rubrics differ?. If there is time: Are rubrics just a North American thing?


3/18


4/18

Table 2: Advantages, Disadvantages, & ItemWriting Guidelines for Personal-Response Assessments_____________________________________________________________________________________________Overall Advantages- Personal assessment; Directly related to and integrated into curriculum;

Appropriate for assessing learning processesDisadvantages - Difficult to create and structure; Scoring is subjectiveOverall Guidelines- Make sure students have a voice in everything that takes place

Conferences Advantages- Can help students understand learning processes and learning strategies;Help students develop better self-images; Teachers can elicit specific skills or tasks thatneed review; Can be used to inform, observe, mold, and gather information about SsDisadvantages - Requires a lot of time; Difficult to use for grading purposes; Typicallynot scored or rated at allGuidelines1. insure that students feel in control of conference2. focus discussion on the students worries and views about the learning process3. consider focusing students on self-image issues4. consider eliciting performances on specific tasks or skills that need to be reviewed5. hold conferences regularly and frequently consider scoring and grading conferences

by applying rubrics

Portfolios Advantages- Enhances student learning; teachers role; assessment process; curriculumDisadvantages - Design decision problems; Logistical problems; Interpretation problems;Reliability problems; Validity problemsGuidelines1. decide with students in advance who will do what2. introduce and explain portfolios to the students3. have students select and collectmeaningful work4. periodically have students reflect on their portfolios5. periodically have students, teachers, outsiders, etc. review the portfolios6. consider scoring portfolios by using rubrics


5/18

Self-assessments Advantages- Takes less time; Involves students in the assessment process; Encourages

student autonomy; Can increase learner motivationDisadvantages- Accuracy varies depending on skill levels and material involved; Higherlevel students may tend to underestimate their abilities; Prone to subjective errors; May

function differently depending on the consequencesGuidelines1. decide in advance what aspect of their language performance they will be assessing2. where suitable, develop a written rating scale for the learners to use in scoring (and

decide on whether to use a holistic or analytic scoring)3. plan in advance the logistics of how the students will score themselves4. use directions that give the students descriptions of concrete linguistic situations that

they can score in behavioral terms if at all possible

5.

make sure the students understand the self-scoring they will do6. have students score all of the items of one type in one session7. consider having students score themselves on two different occasions or have

another student (or the teacher) do the same scoring_____________________________________________________________________________________________


6/18


7/18


8/18

Table 5: Holistic Version of the Scale for Rating Composition Tasks

SCORESDESCRIPTORS

5 Appropriate title, effective introductory paragraph, topic is stated, leads to body; transitional expressions used; arrangement of material shows plan (could beoutlined by reader); supporting evidence given for generalizations; conclusion logical & complete. Essay addresses the assigned topic; the ideas are concrete andthoroughly developed; no extraneous material; essay reflects thought. Native-like fluency in English grammar; correct use of relative clauses, prepositions, modals,articles, verb forms, and tense sequencing; no fragments or run-on sentences. Correct use of English writing conventions; left & right margins, all needed capitals,paragraphs indented, punctuation & spelling; very neat. Precise vocabulary usage; use of parallel structures; concise; register good.

4Adequate title, introduction, & conclusion; body of essay is acceptable but some evidence may be lacking, some ideas arent fully developed; sequence is logicalbut transitional expressions may be absent or misused. Essay addresses the issues but misses some points; ideas could be more fully developed; some extraneousmaterial is present. Advanced proficiency in English grammar; some grammar problems dont influence communication, although the reader is aware of them; nofragments or run-on sentences. Some problems with writing conventions or punctuation; occasional spelling errors; left margin correct; paper is neat and legible.Attempts variety; good vocabulary; not wordy; register OK; style fairly concise.

3 Mediocre or scant introduction, or conclusion; problems with the order of ideas in body; the generalizations may not be fully supported by the evidence given;problems of organization interfere. Development of ideas not complete or essay is somewhat off the topic; paragraphs arent divided exactly right. Ideas gettingthrough to the reader, but grammar problems are apparent and have a negative effect on communication; run-on sentences or fragments present. Uses generalwriting conventions but has errors; spelling problems distract reader; punctuation errors interfere with ideas. Some vocabulary misused; lacks awareness ofregister; may be too wordy.

2Shaky or minimally recognizable introduction; organization can barely be seen; severe problems with ordering of ideas; lack of supporting evidence; conclusionweak or illogical; inadequate effort at organization. Ideas incomplete; essay does not reflect careful thinking or was hurriedly written; inadequate effort in area ofcontent. Numerous serious grammar problems interfere with communication of the writers ideas; grammar review of some areas clearly needed; difficult to readsentences. Serious problems with format of paper; parts of essay not legible; errors in sentence-final punctuation; unacceptable to educated readers. Poorexpression of ideas; problems in vocabulary; lacks variety of structure.

1 Absence of introduction or conclusion; no apparent organization of body; severe lack of supporting evidence; writer has not made any effort to organize thecomposition (could not be outlined by reader). Essay is completely inadequate and does not reflect college level work; no apparent effort to consider the topiccarefully. Severe grammar problems interfere greatly with the message; reader cant understand what the writer is trying to say; unintelligible sentence structure.Complete disregard for English writing conventions; paper illegible; obvious capitals missing, no margins, severe spelling problems. Inappropriate use ofvocabulary; no concept of register or sentence variety.


9/18

Table 6: Analytic Oral Presentation Rubric: Class presentations

Teacher Name:J D Brown

Student Name: ________________________________________

CATEGORY 4 3 2 1

Enthusiasm Facial expressionsand body languagegenerate a stronginterest andenthusiasm about thetopic in others.

Facial expressionsand body languagesometimes generatea strong interest andenthusiasm about thetopic in others.

Facial expressionsand body languageare used to try togenerate enthusiasm,but seem somewhatfaked.

Very little use offacial expressions orbody language. Didnot generate muchinterest in topicbeing presented.

Preparedness Student iscompletely preparedand has obviouslyrehearsed.

Student seems prettyprepared but mighthave needed acouple morerehearsals.

The student issomewhat prepared,but it is clear thatrehearsal waslacking.

Student does notseem at all preparedto present.

Speaks Clearly Speaks clearly anddistinctly all (100-95%) the time, andmispronounces nowords.

Speaks clearly anddistinctly all (100-95%) the time, butmispronounces oneword.

Speaks clearly anddistinctly most ( 94-85%) of the time.Mispronounces nomore than one word.

Often mumbles orcannot beunderstood ORmispronounces morethan one word.

Vocabulary Uses vocabularyappropriate for theaudience. Extendsaudience vocabularyby defining wordsthat might be new tomost of the

audience.

Uses vocabularyappropriate for theaudience. Includes1-2 words that mightbe new to most ofthe audience, butdoes not define

them.

Uses vocabularyappropriate for theaudience. Does notinclude anyvocabulary thatmight be new to theaudience.

Uses several (5 ormore) words orphrases that are notunderstood by theaudience.

Posture and EyeContact

Stands up straight,looks relaxed andconfident.Establishes eyecontact witheveryone in theroom during thepresentation.

Stands up straightand establishes eyecontact witheveryone in theroom during thepresentation.

Sometimes stands upstraight andestablishes eyecontact.

Slouches and/or doesnot look at peopleduring thepresentation.

Date Created: Jan 08, 2011 11:32 pm (UTC)

Created athttp://rubistar.4teachers.org/


10/18


11/18


12/18


13/18


14/18

SomeUsefulReferencesArter, J., & McTighe, J. (2001). Scoring rubrics in the classroom: Using performance criteria for assessing and

improving student performance. Thousand Oaks, CA: Corwin/Sage.Bachman, L. F., & Savignon, S. J . (1986). The evaluation of communicative language proficiency: A critique of the

ACTFL oral interview.ModernLanguage Journal, 70, 380-390.Brindley, G. (1991). Defining language ability: The criteria for criteria. In S. Anivan (Ed.), Current developments in

language testing (pp. 139-164). Singapore: Regional Language Centre.Brown, J . D. (2005).Testing in language programs: A comprehensive guide to English language assessment (New

edition). New York: McGraw-Hill.Brown, J . D. (Ed.) (2012).Developing, using, and analyzing rubrics in language assessment with case studies in

Asian and Pacific languages.Honolulu, HI: National Foreign Languages Resource Center.Brown, J . D. (Ed.). (2013).New ways of classroomassessment(2nd ed.). Alexandria, VA: Teachers of English to

Speakers of Other Languages.Brown, J . D., & Bailey, K. M. (1984). A categorical instrument for scoring second language writing skills.

LanguageLearning,34(4), 21-42.Brown, J . D., Hudson, T., Norris, J. M., & Bonk, W. (2002). Investigating second language performance

assessments. Honolulu, HI: NFLRC.Buttner, A. (2007). Activities, games, assessment strategies, and rubrics: For the foreign language classroom.

Larchmont, NY: Eye on Education.Campbell Hill, B., & Ekey, C. (2010).The next-step guide to enriching classroomenvironments: Rubrics and

resources for self-evaluation and goal setting for literacy coaches, principals, and teacher study groups, K-6.Portsmouth, NH: Heinemann.Cooper, B. S., & Gargan, A. (2009). Rubrics in education: Old term, new meanings. Phi Beta Kappan, 19(1), 54-55Fulcher, G. (1987). Tests of oral performance: The need for data-based criteria.ELT Journal, 41, 287-291.Glickman-Bond, J ., & Rose, K. (2006).Creating and using rubrics in today's classrooms: A practical guide.

Norwood, MA: Christopher-Gordon.Hutson-Nechkash, P. (2003). Help me write: Frames and rubrics for classroomwriting success.Greenville, SC:

Thinking Publications.acobs, H. L., Zinkgraf, S. A., Wormuth, D. R., Hartfiel, V. F., & Hughey, J . B. (1981).Testing ESL composition: A

practical approach. Rowley, MA: Newbury House.Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment, Research &

Evaluation,7(25). Moskal, B. M. (2000). Scoring rubrics: what, when and how? Practical Assessment, Research

& Evaluation, 7(3).Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities and task difficulty in task-basedsecond language performance assessment. Language Testing, 19(4), 396-418.

Norris, J . M., Brown, J. D., Hudson, T., & Yoshioka, J . (1998). Designing second language performanceassessments. Honolulu, HI: National Foreign Languages Resource Center, University of Hawaii at Mnoa(alsoavailable from ERIC: ED 451 701).

North, B., & Schneider, G. (1998). Scaling descriptors for language proficiency scales. Language Testing, 15(2),217-263.

Popham, W. J . (1997). Whats wrong and whats right with rubrics. Educational Leadership, 55(2), 72-75.Tierney, R., & Simon, M. (2004). Whats still wrong with rubrics: Focusing on the consistency of performance

criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2).Upshur, J . A., & Turner, C. E. (1995). Constructing rating scales for second language tests.ELT Journal, 49(1), 3-

12.


15/18

Appendix ASuggested Stages and Steps in the Rubric Development ProcessStages and Steps

Stage 1 PlanningStep 1.1 Define your goal.Step 1.2 Go to the source material.Step 1.3 Brainstorm.Step 1.4 Analytic or holistic?Step 1.5 Decide the categories.Step 1.6 Decide the range of scores your want to use.

Stage 2 Design the Rubric.Step 2.1 Put scores on one axis.Step 2.2 Put the categories on the other axis.Step 2.3 Fill in the rubric descriptors for each score level.

Stage 3 Planning the Assessment Procedures and Using the RubricStep 3.1 Decide on the stimulus formats.Step 3.2 Decide on the response formats.Step 3.3 Write clear instructions.Step 3.4 Make sure the instructions and stimulus materials are ready.Step 3.5 Arrange for the mechanics of assessment.Step 3.6 Actually do the assessment.Step 3.7 Train raters to use the rubric.

Stage 4 Evaluate the Reliability/Fairness of Your RubricStage 5 Evaluate the Quality of Your Rubric

Step 5.1 Evaluate the validity of your rubric.

Step 5.2 Evaluate the usability of your rubric.Stage 6 - Plan Feedback and Revise for Pedagogically Useful Ratings

Step 6a Plan for student and teacher feedback.Step 6b Set up a cycle of revision and improvement.


16/18


17/18

Appendix CCommon European Framework of Reference for Languageshttp://www.coe.int/T/E/Cultural_Co-operation/education/Languages/Language_Policy/Common_Framework_of_Reference/


18/18

Mahalo!

Date post:	04-Apr-2018
Category:	Documents
Upload:	david-clark
View:	217 times
Download:	0 times

LARC_CALPER Testing and Assessment Webinar - Rubrics W_ JD Brown

Documents