Central Mass. Readiness Center Part

IIMaintaining Quality

Creating Cut Scores

Preparing for June Report

Preparing for 2015 Full Implementation

Dr. Deborah Brady

Ribas Associates

Good Morning!

DO NOWReview today’s Agenda, Handout, and

PowerPointFill out a Name TagNote cards for questions and/or exit slipsHave a cup of coffeeWiki Page has today’s materials as well as

Day 1 Materials

Topic Comments on Exit SlipsDDMs Examples of good DDMs for all areas: science, math, elementary, middle school Assessing quality of specific DDMs Developing rubrics for DDMs; (H-M-L); rubric examples Developing Exemplars PARCC examples Processes June report Roll out to teachers: best practices Sample DDM Plan for one educator Calibration of scores and scorers; assuring growth scores are comparable Developing cut scores Planning for full implementation Team Time Assessing the “holes” in DDM in the district PARCC/Writing to Text/Text Sets

Text sets: finding, developing, assessing the rigor and appropriateness

Parking Lot For unresolved issues

Size of classes that are too small to determine student growth (DESE)

Agenda Day 2February 18. 2014

Where are you on your journey now? (Carousel, Living Likert)

Preparing for June report and for next year Local Curriculum Quality: Checklist for DDM Indirect and Direct Measures

Good, Bad, Questionable Examples from local districts

Calibration Protocols and options

Juried Resources—Assessments and More elaborate Lessons with performance assessemtns

Documenting 2 DDMs per Educator (Documentation Form for 2 DDMs)

Scoring Assessments and Developing Local Cut Scores Pre-Test, Post-Test Cut Score Example

Rubric Scoring in 100% system

Introduction to protocols for assessment: samples

Time to research, work with team, plan for June, plan for next year

The Steps Necessary to Get Ready for June Report and After

• Adapting present assessments

• Creating new assessments

• Writing to text for HSDevelopin

g and Piloting

Assessments

• Alignment of Content

• Rigorous and appropriate expectations

• Approval of assessments

Assessing Quality

and Rigor

• Security• Calibration

of standards and of assessors

• Rubric quality

• Analysis of results: High-M-Low Growth

Piloting

• 2 DDMs per educator

• JUNE REPORT• Directions for

teachers • Directions for

students• Organizing for

the actual assessments

• Storing, tracking the information

2015 Full Implementatio

n

• Data storage

• Data Analysis

• L-M-H Growth

• Roster Verification

• Data team timeInterpreting the results

Student Impact

Carousel Walk and Living Likert Scale

Carousel Walk Take a marker for your team (or yourself if you are on your own)

Visit each of the 5 “stages” of DDM development

Put a check to the right of each item that you have completed

After viewing each stage, return to the stage that most represents you and/or your district

Living Likert Go to your “stage”

What are the barriers? What are the strengths that your district has?

Discuss the barriers and strengths with the others at your stage.

Be prepared to report out as a whole group.

Preparing for the June ReportCurriculum Quality Checklist

Documenting 2 DDMs per educator

Checklist for DDM Direct and Indirect Measures Check all Items that are

completedDefinition Please fill in this column as completely as you

can Course This information is necessary for June Report

Grade(s) of DDM Grade level(s) that this assessment will cover

Check one

Direct Measure Indirect Measure

Direct Measure= Assessment of student growthIndirect Measure=Assessment of something that connects indirectly to student growth (attendance, for example)

Content Area List content area.Indirect measure include the job category/categories of those involved, for example, SLPs or guidance counselors

Alignment to State and/or District Standards

For direct measures: List at least 2 standards that will be assessed so that this is a “substantial” assessment.For indirect measures, 1) What are the substantial, important, essential areas that you are assessing? 2) How does this indirect measure connect with student growth?

Please list the two (or more) standards using standards language.1. 2.

Rigor: Check the levels of Blooms that are assessed

The original Bloom is the first word on the list. The new Bloom (all verbs) is the second. Note, in the new Blooms, Creating is on the highest level, above evaluating.Indirect Measures: NA; skip Types of Questions and Duration (below), but fill in the remaining categories.

Check all that apply: Knowledge, Remembering Comprehension, Understanding Application, Applying Analysis, Analyzing Synthesis, Creating Evaluation, Evaluating

Check all that apply

Type(s) of questions

Multiple Choice, fill in, short answer (recall items from content area) Multiple Choice, fill in, short answer (text dependent questions) Open Response (short answer) Essay (long response). Type:

Narrative Informational Text Argument with claims and proof

One text is read Two texts are read Performance Assessment (CEPA) Other_______ (Fill in at right.)

Indicate the percentage of the assessment for each question type, for example, multiple choice=50%; 2 open responses=50% (25% each) Multiple Choice _____%Open Response _____%Essay _____________%

Performance Assessment: Describe briefly:

Duration of assessment

Assessments can take place in a class period or over a period of days. Indirect Measures please fill this section out.

Necessary for planning for full implementation.

When assessment(s) will take place

Provide approximate month or window for assessment(s), for example, end of first trimester, September.Provide multiple dates if the assessment is a pre-post or is administered more than once.Indirect Measures please fill this out.

Components of assessment that are completed so far.

Directions to teacher for administering Directions to students Graphic organizers (optional) The assessment Scoring guide Rubric Security Calibration protocol if this assessment has a rubricIndirect measures: Describe your proposed process; you will need to create many of these components

Rubric Not Yet Does not apply

How was the rubric created? For example,adapted from DESE’s CEPA rubric, or developed by the department.

Please attach rubric along with the assessment as far as you have gone.

Quality Tools Core Curriculum Objectives: Model Curriculum Units and Curriculum Embedded Performance

Assessment Rubrics: DESE Cognitive Complexity and Common Core: Daggert and Hess Juried assessments: www.EngageNY.org Juried units/assessments: www.achieve.org, MA MCU Juried rubrics: CEPA, Hess (ELA, math, History, Science),

Delaware’s K-12 rubrics for 3 Common Core text types: argument, narrative, informational text

Calibration and Looking at Student Work (LASW) Protocols (on handout) plus http://Nsfharmony.org/protocol/a_z.html

Achieve.org has Common Core aligned and annotated student sample essays at http://www.achievethecore.org/page/507/in-common-effective-writing-for-all-students

Demonstrating Growth(when accuracy of computation may be a concern)

Essay Prompt from Text

Read a primary source about Mohammed based on Muhammad’s Wife’s memories of her husband.

Essay: Identify and describe Mohammed’s most admirable quality based on this excerpt. Select someone from your life who has this quality. Identify who they are and describe how they demonstrate this trait.

What’s wrong with this prompt?

Science Open Response from Text

Again, from a textbook, Is this

acceptable?

Scoring Guides from Text

Lou Vee Air Car built to specs (50 points)

Propeller Spins Freely (60 points)

Distance car travels

1m 70

2m 80

3m 90

4m 100

Best distance (10,8,5)

Best car(10,8,5)

Best all time distance all classes (+5)

235 points total

A scoring guide from a textbook for

building a Lou Vee Air Car. Is it good enough to ensure

inter-rater reliability?

Technology/Media Rubric

A multi-criteria rubric for technology. What is

good, bad, problematical?

PE Rubric in Progress.Grade 2 Overhand

throw.Looks good?

Music: Teacher and Student Instructions

World Language Scoring Guide and Rubric

Table Talk

Curriculum Quality

Local Standards

2 DDMs per EducatorMEPID Last Name First Name Grade/Dept. DDM1 DDM2 DDM3

(optional)  Jones Brigit ELA 6 MCAS 6 Growth

Score ELA (SGP)ELA 6 DDM (writing to

text) 

  Smith Marion 9-12 library Library Search Tools DDM

Indirect: Increase teachers who do

research in library.

  Watson Elsbeth 5 ELA team Fountas and Pinnell DDM

MCAS 5 Growth Score History Unit Exam

DDM  Holmes Sharon Grade 2 Fountas and

Pinnell DDMGalileo DDM

NOTE: Consider developing a system for naming DDMs for storing, locating, assuring consistency as you report in June

F&P.6; M.Gal.6ELA.SGP.6

Math.6.1; Math.6.2; Math.6.GalileoPE.K-5 I, PE.K-5 2

Bio.9-10.1; Bio.9-10.2; Bio.9-10.H1; Bio.9-10.H2

Table Talk

June Reporting

Scheduling for Full Implementation

Calculating Cut ScoresPre- Post- Calculation

Rubric Inaccuracy when translated to a percentage

Cut Scores for L-M-H Growth

Pre-testPost test

Difference

%age growth

Diff/pre Student Scores Sorted low to high

%diff/pre

Teacher score is based on the MEDIAN Score of her class for

each DDM Teacher L-M-H Rank

20 35 15 75% 20% 5Cut score? LOW Growth

25 30 5 20% 42% 15 bottom 20%30 50 20 67% 42% 20

35 60 25 42% 50% 25 Moderate Growth

35 60 25 42% 60% 25 median teacher score

40 70 35 87% 62% 25 median Teacher score 40 65 25 62% 67% 25

50 75 25 50% 70% 30

50 80 30 60% 75% 35 Top 20%?

50 85 35 70% 87% 35 Cut score? HIGH GROWTH

Rubrics Converted to %Roobrix.com

1. Number of levels in Rubric

2. Number of criteria in Rubric

3. Minimum passing grade if student did the work (very poorly)

4. Student’s score

4 Levels

2. Criteria: Topic/Conventions (weighed 2:1=3 criteria)

3. 50

4. Proficient times 2/Advanced

Result=89%

Topic

4 3 2 1

Topic 4 3 2 1

Conv. 4 3 2 1

Total 4 6

Table TalkLocal Cut Scores

High-Moderate-Low Student Growth

Teacher’s Growth Score is the Median Score for each class included in the DDM (not the

average)

Begin with top and bottom 10% then go to the “body of the work,” i.e., the student work

Rubrics and percentages

Calibration ProtocolsInter-Rater Reliability

Singleton Teacher and Fairness

Rhode Island http://www.ride.ri.gov/Portals/0/Uploads/Documents/Teachers-and-Administrators-Excelent-Educators/Educator-Evaluation/Online-Modules/Calibration_Protocol_for_Scoring_Student_Work.pdlf

Select Facilitator Individually

sort your students’ work into

High, Medium, and

Low piles

Individuallydescribe the characteristi

cs of each pile with

rubric or list of criteria.

As a group,share

characteristics.

As a group,develop

rubric or list of criteria

and exemplars(possibly)Score all

Assessments

As a group, develop an action plan (next step)

for each level and for all students

Reflect on process.

Share reflections.

START HERE

PROTOCOLSProtocols for Calibrating Inter-Rater ReliabilitySee handout

Develop local Cut Scores

Assign L-M-H Growth to

each Student

Sort student scores to

each TeacherDetermine

Median scoreAs teacher’s Growth Score

Modification for

DDM Process

Brenda is making tree costumes for a play. The list below shows the amounts of the different colors of cloth Brenda will use to make one tree costume.

yards brown cloth

yards orange cloth

yard yellow cloth

1. What is the difference, in yards, between the amount of orange cloth and the amount of brown cloth that Brenda will use to make one tree costume? Show or explain how you got your answer.

2. Brenda plans to use brown cloth for the trunk and branches of the tree, and orange and yellow cloth for the leaves.

3. What is the total amount of cloth, in yards, Brenda will use to make the leaves of one tree costume? Show or explain how you got your answer.

4. Brenda wants to make two tree costumes.5. What is the total amount of cloth, in yards, Brenda will use to

make two tree costumes? Show or explain how you got your answer.

MCAS Rubrics and Examples at all levelsGrade 6 Math

MA DESE Student Work at http://www.doe.mass.edu/mcas/student/2013/

2

The student response demonstrates a fair understanding of the Number and Operations-Fractions concepts involved in adding and subtracting fractions with unlike denominators, including mixed numbers, by replacing given fractions with equivalent fractions in such a way as to produce an equivalent sum or difference of fractions with like denominators. While some aspects of the task are completed correctly, others are not. The mixed evidence provided by the student merits 2 points

K-5 Same Prompt; 6-12 Same Prompt

http://www.achievethecore.org/page/507/in-common-effective-writing-for-all-students

Grade 4

On-Demand Writing- Uniform Prompt

Which is Better as a Pet, a Dog or a Cat?

Many people have a dog for a pet. Some people have cats. Wich is better? I say dog. Maybe you say cat. I just might be able to persaude you in the following. Dogs are great companions for lonely people. They can go for a rousing walk in the park, or a good long nap. Playing games of catch or fetch every day makes good fun. Even a jog on the hottest day could even be enjoyable too. Dogs don’t just provide fun though. They can also provide protection.

Dogs are very intelligent. They can be trained to find people or save them. Some don’t even need to be trained. For instance, if someone is trying to break in, your dog might bark and scare them off. Dogs are great for many different reasons. Overall, dogs are awesome pets to have. Have I convinced you though? If you are, then great! If your not then thats okay. It's really up to you. So which one is it going to be?

AnnotationsIntroduces a topic clearly  States an opinion  Provides a concluding section related to the opinion presented   Creates an organizational structure in which related ideas are grouped to support the writer’s purpose  Provides reasons that are supported by facts and details

  Links opinion and reasons using words and phrases  

Annotated Exemplar ORQ: How does the author create the mood in the poem?

Answer and explanation in the student’s words

Specific substantiation from the text

4 Texture: Crunchy outside; moist inside cookie and slightly melting chocolateAppearance: Glossy from the use of real butter; fits in the palm of your hand, variation in shapes and modulation of surface perhaps of gourmet or home made origin; see substantial chocolate chunks.Taste: Cookie melts in your mouth and the chocolate is high quality, in chunks, and possibly imported dark chocolateTemperature: Warm, just out of the oven and cooling on a cooling rack

3 Texture: Cake-like, chocolate is at room temperatureAppearance: Flat surface, slightly smaller than the palm of your hand, some variation in sizes and surface modulation indicates it may be home made, see some possible bites without chocolate. Looks like commercial chocolate bits.Taste: Cookie is mostly crunchy. The chocolate tastes like commercial chocolate chips.Temperature: Room temperature.

2 Texture: Somewhat dry.Appearance: Flat surface, small, almost skimpy size; maybe from a cafeteria. Many bites without chips.Taste: Somewhat dry. Chocolate is not of high quality; may be old.Temperature: Cool, maybe from cafeteria refrigerator or freezer.

1 Texture: Dry, cold, crumbling Appearance: Porous, skimpy portion, uniform shapes, perhaps commercially madeTaste: Dry and chocolate has a white coating as if it is old or of poor qualityTemperature: Cold

Chocolate Chip Cookie/Holistic Rubric

copyright 2013 Ribas Associates

“Don’t let perfection get in the way of good.”

Team Time to Plan

The Steps Necessary to Get Ready for June Report and After

• Adapting present assessments

• Creating new assessments

• Writing to text for HSDevelopin

g and Piloting

Assessments

• Alignment of Content

• Rigorous and appropriate expectations

• Approval of assessments

Assessing Quality

and Rigor

• Security• Calibration

of standards and of assessors

• Rubric quality

• Analysis of results: High-M-Low Growth

Piloting

• 2 DDMs per educator

• JUNE REPORT• Directions for

teachers • Directions for

students• Organizing for

the actual assessments

• Storing, tracking the information

2015 Full Implementatio

n

• Data storage

• Data Analysis

• L-M-H Growth

• Roster Verification

• Data team timeInterpreting the results

Student Impact