+ All Categories
Home > Documents > Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public...

Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public...

Date post: 16-Jan-2016
Category:
Upload: laureen-oneal
View: 214 times
Download: 0 times
Share this document with a friend
Popular Tags:
72
Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools
Transcript
Page 1: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Student Learning Objectives

Day Three: Selecting Quality Assessment Instruments

Aurora Public Schools

Page 2: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Re-Introductions Center for Transforming Learning and Teaching

Catalyzing and co-creating the transformation of learning environments through the use of assessment so that all are engaged in learning and empowered to positively contribute in a global society.

www.ctlt.org

Facilitator/Trainer:Julie Oxenford O’Brian

Coach/Trainer:Mary Beth Romke

[email protected] [email protected]

CTLT 2014 www.ctlt.org

Page 3: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

SLO ComponentsLearning Goal 

Learning Goal

Standards Reference

Rationale

Success Criteria

Measures 

Evidence Sources

Alignment of Evidence

Collection and Scoring

Performance Targets 

Baseline Data

Performance Groups

Performance Targets

Rationale for TargetsProgress Monitoring 

Check Points

Progress Monitoring Evidence Sources

Instructional Strategies

SLO Results Student Performance Results

Targets Met

Teacher Performance

Day One

Day Two

Day Three

Day Four

Day Four

Day Five

Day Five

CTLT 2014 www.ctlt.org

Page 4: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Check-In With your table group, discuss:

Have you completed the Learning Goal section of an SLO?

Did you identify appropriate assessment methods for your SLO Learning Goal?

What questions do you now have about SLO Learning Goals?

Capture any remaining questions about SLO Learning Goals or baseline data on a sticky note.

Full-Group Check: Did you identify baseline data sources related to your SLO

Learning Goal?

How many?

CTLT 2014 www.ctlt.org

Page 5: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Identify the “Big Ideas” for the course/class

Start with one big idea and identify associated standards

Develop a statement of intended learning (SLO Learning Goal candidate)

Is the Learning Goal rigorous enough (DOK >=2 for K-2 or DOK >=3 for 3-12)

Does the learning goal correspond with the needs for your students

Select as your SLO Learning Goal

yes

yes

no

no

Defining SLO Learning Goals

Note catcher page 2

CTLT 2014 www.ctlt.org

Page 6: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

SLO Learning Goal Components Learning Goal (statement if intended

learning)

Standards Reference (full text of standard statements)

Rationale for the Learning Goal (why an appropriate focus, appropriate DOK, need for identified student population)

Success Criteria (characteristics of proficient performance on the goal)

CTLT 2014 www.ctlt.org

Page 7: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

What we are hearing from you. . .

Pluses (+) Time to work with

colleagues and get real work done (13)

Assessment methods (10)

Structure of training and materials (5)

SLO Form/Rubric (4)

Changes for my practice Using a greater variety of

assessment methods (10)

Establishing SLO Learning Goals (3)

Using multiple data points (triangulating data) (3)

CTLT 2014 www.ctlt.org

Page 8: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

What we are hearing from you. . .

Changes for Training Color code or make

handouts easier to navigate (6)

Provide more examples (4)

More or less time to process (4)

Clarify terminology (3)

Questions How will this roll-out district

wide? (14)

How do we identify baseline data? What are possible data sources? (5)

How do we address DOK appropriately? (3)

CTLT 2014 www.ctlt.org

Page 9: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

What we are hearing from you. . .Ah Ha’s !! The process of developing, monitoring SLOs is

backwards planning and not that far from what we already do (7).

Assessment is not a test, it’s a process and there are lot’s of options for how to collect data as part of the assessment process (6).

Baseline data is there, it doesn’t have to be a pre-test (5).

SLOs are about how students who start at different places grow, not just about proficiency (3)

CTLT 2014 www.ctlt.org

Page 10: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Post-Session Survey 24 folks responded (out of 70)

Overall understanding of Session 2 content: 62% - need more practice

38% - I’ve got it or I could teach someone else

Finalized SLO Learning Goal: 75%

Seeing connections to what I am already doing in my practice: 100%

CTLT 2014 www.ctlt.org

Page 11: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

More Survey Results: Levels of UnderstandingTopic Some Understanding Much or Great

Understanding

Writing SLO Learning Goal Rationale

29% 71%

Finalizing SLO Learning Goal

75% 19%

Describing how assessment is used

46% 50%

Identifying a variety of assessment methods

17% 79%

Relationship between assessment method and DOK of the learning goal

38% 54%

ID Baseline Data Sources

25% 63%

CTLT 2014 www.ctlt.org

Page 12: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

More Survey Results: Training Training Methods required active participation:

88% agree/strongly agree

Provided structured learning opportunities: 92% agree/strongly agree

Made content relevant: 87% agree/strongly agree

Presented in a coherent manner: 78% agree/strongly agree

CTLT 2014 www.ctlt.org

Page 13: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Posting SLO Learning Goals Getting examples from you is critical to the

pilot!

10 people posted SLO Learning Goals by January 7th

Individualized feedback provided on what was posted.

Updating the form and the web site to make this easier!

CTLT 2014 www.ctlt.org

Page 14: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Focus for Today

Identifying assessment instrument(s) to use in measuring student learning in relationship to the SLO Learning Goals.

CTLT 2014 www.ctlt.org

Page 15: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Norms

The standards of behavior by which we agree to operate while we are engaged in learning together.

CTLT 2014 www.ctlt.org

Page 16: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Materials

CTLT 2014 www.ctlt.org

Page 17: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Session Three: Learning Outcomes

Engage in learning activity during this session.

Complete follow-up tasks.

Understand the information that needs to be provided in the SLO Form about assessment instruments.

Interpret the Quality Assessment Criteria.

Recognize potential sources of bias in assessment tasks.

Evaluate the quality of a variety of assessment tasks.

Specify information gained from assessment instruments (multiple tasks) by creating an assessment map.CTLT 2014 www.ctlt.org

Page 18: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Activity: Monitoring your learning Turn to Progress Monitoring (Note Catcher, p. 3-4)

Re-write today’s learning outcomes in language that has meaning for you.

Create a bar graph which describes where you currently believe you are in relationship to each learning target.

Leave the “reflections” column blank for now.

Learning Target

I don’t know what this Is

I need more

practice

I’ve got It

I can teach others

Reflections

Identify a variety of methods (informal and formal) for collecting data about student learning.

In my words:

CTLT 2014 www.ctlt.org

Page 19: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

SLO Session Three Agenda

The Role of Assessment in

SLOs

Fairness in

Assessment

Evaluating Assessment

Items

Mapping Assessment Instruments

CTLT 2014 www.ctlt.org

Page 20: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Reviewing Terminology Turn to “What Objectives are Not” (Tools, p. 1) and take

out “SLO Terminology” (first section of your notebook).

Read “What Objectives are Not”.

Use both resources to answer these questions with your table group: What is the difference between an objective and an

assessment?

What is the difference between an objective and an activity?

How are these terms similar and how are they different: objective, learning goal, learning target?

Share questions/concerns with the full group.

CTLT 2014 www.ctlt.org

Page 21: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Levels of ObjectivesLevel of Objective

Global Educational Instructional

Scope Broad Moderate Narrow

Time needed to learn Two or more

years (often many)

Weeks, months, or academic year Hours or days

Purpose or function Provide vision Design curriculum Prepare lesson

plans

Example of use Plan a multi-year curriculum (e.g.,

elementary reading)

Plan units of instruction

Plan daily lessons, activities,

experiences and exercises

A Taxonomy for Learning, Teaching, and Assessing: A revision of Bloom’s taxonomy of educational objectives, 2001

SLO

Learn

ing

GoalsLe

arnin

g Ta

rget

s

CTLT 2014 www.ctlt.org

Page 22: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Educational AssessmentEducational Assessment is. . .

A process by which educators use students’ responses to specially created or naturally occurring stimuli to draw inferences about the students’ knowledge and skills.

A process of reasoning from evidence.

Measuring learning.

Pellegrino, J., Chudowsky, N., & Glaser, R., Eds. (2001). Knowing What Students Know: The Science and Design of Educational Assessment. Washington DC: National Academy Press.

CTLT 2014 www.ctlt.org

Page 23: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Assessment includes:

1. The aspect(s) of student learning that are to be assessed (cognition).

2. The tasks used to collect evidence about students’ achievement (observation).

3. The approach used to analyze and interpret the evidence resulting from the tasks (interpretation).

Pellegrino, J. Chudowsky, N. & Glaser, R. Eds. (2001). Knowing What Students Know: The Science and Design of Educational Assessment. Washington DC: National Academy Press.

CTLT 2014 www.ctlt.org

Page 24: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Assessment in the SLO Process

As part of the SLO Process, we use multiple evidence sources (data collected from a variety of assessment instruments) to “reason from evidence” about:

Student learning in relationship to our Learning Goal at the beginning of the instructional interval (baseline data).

Student progress towards the Learning Goal during the instructional interval (progress monitoring/formative assessment).

Student learning in relationship to our Learning Goal at the end of the instructional interval (summative assessment).

Teacher contribution to Student Learning Growth (aggregation of results across students in the class/course).

CTLT 2014 www.ctlt.org

Page 25: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Today. . . Building our toolkit related to quality assessment

practice that can apply across all uses of assessment for SLOs and for our practice more generally.

Identifying and ensuring the quality of the assessment instruments we will use to collect data and reason from evidence about student learning in relationship to our Learning Goal at the end of the instructional interval (summative assessment).

CTLT 2014 www.ctlt.org

Page 26: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Reminder: Quality Assessment Criteria

Turn to the SLO Rating Scale and the Quality Assessment Criteria (first section of your notebook).

Consider: How do these documents work together to provide information about quality assessment as part of SLOs?

Our quality assessment criteria focus for today: Content Focus Fairness and Cultural Sensitivity Sufficiency of Evidence

Review these three sections of the Quality Assessment Criteria, identify clarifying questions about these criteria and capture them on sticky notes.

CTLT 2014 www.ctlt.org

Page 27: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

SLO Session Three Agenda

The Role of Assessment in

SLOs

Fairness in

Assessment

Evaluating Assessment

Items

Mapping Assessment Instruments

CTLT 2014 www.ctlt.org

Page 28: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Quality Assessment Criteria: Fairness and Cultural Sensitivity

The assessment tasks are visually clear and uncluttered (e.g. appropriate white space and or lines for student responses, graphics and/or illustrations are clear and support the test content, the font size seems appropriate for the students).

The vocabulary and context(s) presented by the assessment tasks are free from cultural or other unintended bias.

The directions and responses of the assessment tasks are presented in as straightforward a way as possible for a range of learners.

Accommodations are provided to ensure that English Language Learners and/or students with disabilities can fully access the content represented by the task(s).

CTLT 2014 www.ctlt.org

Page 29: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

We’ve all been students

Recall an assessment experience that was less than optimal…

Take 2 minutes to make notes about your story: What happened?

What didn’t seem “right”?

How did that feel?

Why do you remember the experience?

CTLT 2014 www.ctlt.org

Page 30: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Compare Bad ExperiencesIn your table group…

1. Select a recorder

2. Summarize your stories (Record titles.)

3. Describe “why” it was bad(Record the why’s.)

4. Identify common threads(Recorder circles.)

Bad Assessment Story Titles

Why (common threads)

CTLT 2014 www.ctlt.org

Page 31: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Reducing systematic error

Score = ST + SE

SE = Esystematic + Erandom

One possible source of systematic error = A group of students don’t have the same opportunity to show what they know as others (bias).

CTLT 2014 www.ctlt.org

Page 32: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Dimensions of Cultural Attributes Turn to “Dimensions of Cultural Attributes” (Tools, p. 3).

Individually, locate yourself on the continuum for each of the 5 dimensions. Would it be different personally and professionally?

Consider these questions: Do your responses differ personally from within a school setting? Do you have a sense of where your students would fall on each

of these dimensions? How might they be similar or different from you?

How could these dimensions of cultural attributes help you consider ways bias can be introduced into assessment of students’ learning?

CTLT 2014 www.ctlt.org

Page 33: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Recognizing Sources of Bias Turn to “Assessments in Need of Repair” (Tools, p. 9)

and “Notes about an Assessment in Need of Repair” (Tools, p. 13).

Work with a partner.

Select one of the scenarios of Assessments in Need of Repair on which to focus.

Answer the questions in the “Notes about an Assessment in Need of Repair” table for your scenario.

Prepare to share sources of bias and how you would repair the item.

CTLT 2014 www.ctlt.org

Page 34: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Sources of Bias. . .

The design of an assessment instrument (or task) is only one source of bias.

Other potential sources of bias include:Scoring

Learning Environment

Student Context

CTLT 2014 www.ctlt.org

Page 35: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Reducing Sources of Bias: Activity Turn to “Potential Sources of Bias Checklist” and

“Minimizing Sources of Bias Notes” (Tools, p. 15 & 17). Work with a partner (ensure each group includes at least

one teacher). Activity:

1. Use the Potential Sources of Bias Checklist.

2. Think about your students.

3. Identify the sources of bias that seem to arise most often for your students and check those items.

4. Brainstorm ways you can minimize those challenges.

5. Capture notes in Minimizing Sources of Bias Notes.

CTLT 2014 www.ctlt.org

Page 36: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

About error. . .

There will always be error in assessment.

Error is exacerbated when only one assessment instrument is used to evaluate student performance on the SLO learning goal.

Using multiple measures or a body of evidence for SLOs reduces measurement error.

CTLT 2014 www.ctlt.org

Page 37: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Taking it into Practice: Reducing Bias in Measuring Student Learning Review assessment tasks/instruments with a focus on

potential sources of bias.

Consider: Will the tasks/instrument give all of your students the same opportunity to show what they know and can do?

Adjust tasks/instruments to minimize sources of bias.

Work with a colleague to get/provide an external review of potential sources of bias in tasks/instruments you plan to use for SLOs.

CTLT 2014 www.ctlt.org

Page 38: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

SLO Session Three Agenda

The Role of Assessment in

SLOs

Fairness in

Assessment

Evaluating Assessment

Items

Mapping Assessment Instruments

CTLT 2014 www.ctlt.org

Page 39: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Quality Assessment Criteria: Content Focus The assessment methods and tasks are consistent

with the depth of knowledge and types of cognitive processes represented by the SLO Learning Goal.

The assessment content is age/grade level appropriate.

CTLT 2014 www.ctlt.org

Page 40: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

How do assessment items work? Task/Item. An activity, exercise, or question requiring

students to solve a specific problem or demonstrate knowledge of specific topics or processes.

Assessment items/tasks are the building blocks of tests.

Independently read: “How Test Items Work” (Tools, p. 19). [Note: targeted construct = learning objective(s)]

In your note catcher, answer: What does it mean that “a construct cannot be directly observed”?

What can be observed?

What are the elements of an assessment item/task?

What are critical characteristics of each of these elements?

CTLT 2014 www.ctlt.org

Page 41: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Aggregating to different levels

Multiple Students'

Scores (Teacher Rating)

Multiple Assessment Instruments (SLO Learning Goal)

An Assessment Instrument (evaluates students on part of the SLO Learning Goal)

Items (Learning Targets)

CTLT 2014 www.ctlt.org

Page 42: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Reminder: Assessment Methods Continuum

Observation

Questioning (individual, group, full class)

SelectedResponse

Short ConstructedResponse

Demonstration or Performance

Portfolio

Time

Complexity of information

Informal Formal

ExtendedConstructedResponse

Student Products

CTLT 2014 www.ctlt.org

Page 43: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Types of assessment items Selected Response:

Multiple choice

True-False

Matching

Short Constructed Response

Extended Constructed Response

Performance or Demonstration

CTLT 2014 www.ctlt.org

Page 44: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Evaluating or Critiquing Items What information does this item provide about the learning goal?

Review the content (knowledge and skills represented by the item) and determine whether the cognitive processes demanded are similar to those identified in the learning goal.

Review the content and determine how the item is asking students to interact with the content or to demonstrate their understanding of the content.

Compare the content and DOK level to the SLO learning goal.

Is this a quality item of this type?

Does it meet quality criteria for the type of item?

Does it introduce bias?

Is the language used at an appropriate level?

CTLT 2014 www.ctlt.org

Page 45: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Tools to support Item Critiques Turn to “Assessment Item Critique Checklists”

for different item types (Tools, p. 27).

Item Types: Extended Written Response (Essay), Matching, Multiple Choice, Performance Task, Short Constructed Response, True False.

Components: Learning Goal and Assessment Item Alignment. Quality Checklist (by item type). Bias and Language Considerations.

Russell, M., & Airasian, P. (2012). Classroom Assessment: Concepts and Applications. p. 157-8., NY: McGraw-Hill.

CTLT 2014 www.ctlt.org

Page 46: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Part 1: Does the item provide useful information about the SLO learning goal?1. Review the content (knowledge and skills) represented by the item.

2. Identify the type of thinking (cognitive processes) required to respond to the item. Look at the verbs.

3. Identify the depth of knowledge required to respond to the item. Consider how the item is asking students to interact with the content or demonstrate their understanding of the content.

4. Compare the item to the intended learning goal.

Does the item provide information about at least one aspect of the learning goal?

Does it require the same cognitive processes?

Does it match the DOK range reflected by the goal?

CTLT 2014 www.ctlt.org

Page 47: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Sample Task: 3rd Grade Writing

Sample Writing Task:Describe what lunchtime is like for you on a school day.

Be sure to provide a lot of details about your lunchtime so that someone who has never had lunch with you on a school day can understand where you have lunch and what lunchtime is like.

Sample Learning Goal:Students will write informative/explanatory paragraphs that examine a topic and convey ideas and information clearly.

They will:

Introduce a topic and group related information together; include illustrations when useful to aiding comprehension.

State main ideas and include sufficient details or facts to support their reasoning.

Develop the topic with facts, definitions, and details.

Use linking words and phrases to connect ideas within categories of information.

Provide a concluding statement or section.

CTLT 2014 www.ctlt.org

Page 48: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Scoring Rubric: Informative WritingExcellent Develops ideas well and uses specific, relevant details across the response. Well organized with clear transitions. Sustains varied sentence structure and exhibits specific word choices. Exhibits control over sentence boundaries; errors in grammar, spelling, and mechanics do not interfere with understanding. Skillful Develops ideas with some specific, relevant details. Clearly organized; information is presented in an orderly way but response may lack transitions. Exhibits some variety in sentence structure and exhibits some specific word choices. Generally exhibits control over sentence boundaries; errors in grammar, spelling, and mechanics do not interfere with

understanding.

Sufficient Clear but sparsely developed; may have few details. Provides a clear sequence of information; provides pieces of information that are generally related to each other. Generally has simple sentences and simple word choice; may exhibit uneven control over sentence boundaries. Has sentences that consist mostly of complete, clear, distinct thoughts; errors in grammar, spelling, and mechanics generally

do not interfere with understanding.

Uneven May be characterized by one or more of the following: Provides limited or incomplete information; may be list-like or have the quality of an outline. Disorganized or provides a disjointed sequence of information. Exhibits uneven control over sentence boundaries and may have some inaccurate word choices. Errors in grammar, spelling, and mechanics sometimes interfere with understanding.

Insufficient May be characterized by one or more of the following: Provides little information and makes little attempt at development. Very disorganized or too brief to detect organization. Exhibits little control over sentence boundaries and sentence formation; word choice is inaccurate in much of the response. Characterized by misspellings, missing words, incorrect word order; errors in grammar, spelling, and mechanics are severe

enough to make understanding very difficult in much of the response

CTLT 2014 www.ctlt.org

Page 49: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Sample Item: Group Discussion Is there match between the knowledge and

skills represented by the item and by the learning goal?

Cognitive processes of the item and those required by the learning goal?

DOK level of the item?

DOK level of the learning goal (based on the highest level represented)? Also note the DOK range addressed by the learning goal.

CTLT 2014 www.ctlt.org

Page 50: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Is it aligned with the learning goal?

Learning Goal Assessment Item

Comparison

Operational Verbs

Examine, convey Describe, provide Convey and describe are the same, examine not in item

Cognitive Processes

Apply Analyze

Apply Partial

Knowledge and Skills

Students write informative/ explanatory texts, a topic, ideas and information

Asks students to convey details and information about lunchtime (informative)

Similar

Depth of Knowledge

2-3 2 Does not reach the highest DOK level

CTLT 2014 www.ctlt.org

Page 51: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Part 2: Is it a Quality Item for the type of item?

Evaluate the item against the quality criteria for the particular type of item.

CTLT 2014 www.ctlt.org

Page 52: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Is it a good extended response item?Qualities Met CriteriaThe item does not depend on a great deal of memorization or recalling from memory.The students’ task is explicitly described, and focused with terms such as “state and defend”, “apply the principle to”, “develop a valid conclusion”. The item is clear regarding how long or extensiveness the student response should be.The time required for students to complete the item matches the point value of the item.

Optional items are not included.Possible responses and associated performance level (scoring rubric) are developed in advance.Students are aware of the rubric or criteria used to score their response.The importance of writing mechanics and how it will figure into the students' score is determined in advance in the scoring rubric.

The item connects with the students' real life context where possible.

CTLT 2014 www.ctlt.org

Page 53: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Part 3: Does the item introduce bias into the measurement?

Consider the potential for bias being introduced by the item.

Consider the language of the item

CTLT 2014 www.ctlt.org

Page 54: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Bias and Language

1. Possible sources of systematic bias based on student characteristics.

Can you think of any possible sources of bias?

2. Strategies to mediate bias? None needed

3. Is the item written in language students will understand?

Yes

CTLT 2014 www.ctlt.org

Page 55: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Activity: Practice Critiquing Items Choose a partner. Select two items to critique. Use the “Assessment Item Critique Checklist” for the

appropriate item types (Tools, p. 27). For each item:

1. Identify the content (knowledge and skills) represented.

2. Determine the cognitive processes and complexity required to respond to the item.

3. Make notes comparing the learning content corresponding to your learning objective and the item, how are they similar? how are they different?

4. Evaluate the item against the appropriate checklist for that item type.

5. Identify sources of bias and strategies to mediate.

Group Discussion: How did this work? Sharing examples.

CTLT 2014 www.ctlt.org

Page 56: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Taking it into practice Tools: How test items work (reading), Assessment Item

Critique Checklist(s).

Next Steps:

Practice critiquing items/tasks that you plan to use in the next few months for summative purposes (e.g. end of unit test).

Revise items/tasks as necessary based on your critique.

Work with a colleague to provide/receive external review of items/tasks you are currently using.

Critique items/tasks you plan to use as part of measuring student learning for SLOs.

CTLT 2014 www.ctlt.org

Page 57: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

SLO Session Three Agenda

The Role of Assessment in

SLOs

Fairness in

Assessment

Evaluating Assessment

Items

Mapping Assessment Instruments

CTLT 2014 www.ctlt.org

Page 58: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Quality Assessment Criteria: Content Focus

Collectively the tasks, or items, included in the assessment instrument(s) address the content of the SLO Learning Goal.

CTLT 2014 www.ctlt.org

Page 59: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Assessment Instruments

Assessment instruments (or tests) are used to gather information about students’ achievement or cognitive skills.

They frequently include multiple items or tasks.

For those with many items (e.g., over 30), the assessment typically represents a broad sampling of the domain of interest.

CTLT 2014 www.ctlt.org

Page 60: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Aggregating to different levels

Multiple Students'

Scores (Teacher Rating)

Multiple Assessments (SLO Learning Goal)

An Assessment (Part or all of the SLO Learning Goal)

Items (Learning Targets)

CTLT 2014 www.ctlt.org

Page 61: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Quality Assessments

High quality assessments are made up of high quality items.

Which items are included matters.

An “assessment map” describes which items are included, similar to. . .

test blueprint, or

table of specifications.

CTLT 2014 www.ctlt.org

Page 62: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Why map an assessment?1. Clarify how results will be used (instructional importance and

utility).

2. Determine the accuracy of the information provided about the learning being assessed (validity).

3. Determine how much information an assessment provides about different targets corresponding to your learning goal (reliability).

4. As the basis for scoring the assessment in a way that provides information about the SLO goal and corresponding learning targets.

5. As the template for understanding what information the assessment results can provide about instruction.

6. Note: No need to map an assessment that includes only one item/ task.

CTLT 2014 www.ctlt.org

Page 63: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Assessment Map Example (Tools p.47)Learning Targets Corresponding to

Learning Goal

Item #

Cognitive Processes

Knowledge and Skills

DOK Match Amount of Information

Locate commonly used positive numbers on a number line.

1. 3.

UnderstandUnderstand

Number lineNumber line

11

HighHigh

1 pt1 pt

Use the relationships among fractions, decimals, and percents in problem-solving situations using pictorial or concrete materials.

2.

4.

6.

7.

Apply

Remember

Apply

Remember

Number line, fractions, decimalsFractions

Fractions

Fractions

2

1

3

1

High – apply and DOK2

Low – remember is not use

High - apply and DOK3

Low – remember is not use

4 pt

1 pts

2 pt

1 pt

Demonstrate meanings for numbers using physical materials in problem-solving situations.

5.

8

Understand

Understand

Square numbersFractions

1

1

High

High

1 pt

1 pt

CTLT 2014 www.ctlt.org

Page 64: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Example of an Assessment Map (Tools, p. 48)

Learning Target Total Points Possible on Assessment

Locate commonly used positive numbers on a number line. 

2

Use the relationships among fractions, decimals, and percents in problem-solving situations using pictorial or concrete materials.

8

Demonstrate meanings for numbers using physical materials in problem-solving situations.

2

CTLT 2014 www.ctlt.org

Page 65: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Assessment Mapping Process

Review: Assessment mapping process (Tools, p. 49)

Assessment mapping template (Tools, p. 51)

Assessment mapping job aide (Tools, p. 53)

Do you have questions about the process?

CTLT 2014 www.ctlt.org

Page 66: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Practice: Map an assessment Find a partner (at least one classroom teacher per

partnership).

Take out: “Assessment Mapping Process” (Tools, p. 49), “Assessment Mapping Template” (Tools, p. 51, and note catcher), and an example test.

Try out the process of creating a map of your example test.

Exchange your test and map with another pair.

Provide feedback about the other pair’s test map.

Revise your map (if needed) based on the feedback.

CTLT 2014 www.ctlt.org

Page 67: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Why map assessments? Before you administer an assessment, to

determine if it is worth the time for you and your students.

Determine if the assessment provides adequate information for the intended use of the results.

Revise and improve assessments.

Plan assessments.

Determine if a selected assessment addresses the content of (aligns with) the SLO learning goal.

CTLT 2014 www.ctlt.org

Page 68: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

SLO Form: Measures and Scoring Evidence Sources (Measures or Assessments)

Alignment of Evidence to Learning Goal

Today:

What Measures/Assessment Instruments are appropriate to use to measure student learning at the end of the instructional interval?

How will you know the assessment instruments you use align with the SLO Learning Goal?

CTLT 2014 www.ctlt.org

Page 69: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Next Steps. . .I. Complete the Learning Goal Section of your SLO on the SLO

Form and post to the SLO Pilot web site (success criteria is optional).

II. Practice Evaluating the Quality of an Assessment Resource: Consider an assessment you have planned at the end/during your next instructional unit.

1. Critique the items/tasks (use the appropriate item critiquing checklist for each item, and don’t forget to consider bias)

2. Develop an assessment map (for an instrument with more than one item).

3. Partner with a colleague to give/receive feedback about your critiques and map.

4. Make any revisions to the assessment that you discover are needed.5. Bring artifacts to the next session (February, 10th)

III. Identify assessment instruments/tasks you could use to measure student learning related to your SLO learning goal at the end of the instructional interval.

CTLT 2014 www.ctlt.org

Page 70: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Reflect and Consider your Learning

Return to your Progress Monitoring Form, turn to Quality Assessment part 2.

Did you move to the right in your self-assessment? Add to your graph.

Make any notes about your own learning in the “reflections” column.

CTLT 2014 www.ctlt.org

Page 71: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

Give us Feedback!! Oral: Share one ah ha!

Written: Use sticky notes

+ the aspects of this session that you liked or worked for you.

The things you will change in your practice or that you would change about this session.

? Question that you still have or things we didn’t get to today

Ideas, ah-has, innovations

Leave your written feedback on the parking lot.CTLT 2014 www.ctlt.org

Page 72: Student Learning Objectives Day Three: Selecting Quality Assessment Instruments Aurora Public Schools.

References Heritage, M. (2010). Formative assessment: Making it happen in the

classroom. Thousand Oaks, CA: Corwin Press.

Kusimo, P., Ritter, M. G., Busick, K., Ferguson, C., Trumbull, E., & Solano-Flores, G. (2000). Making assessment work for everyone. Assessment Laboratory Network Project of the Regional Educational Laboratories. p. 249-253 & 257-259. San Francisco, CA: WestEd – Publications Center.

Pellegrino, J., Chudowsky, N., & Glaser, R., Eds. (2001). Knowing What Students Know: The Science and Design of Educational Assessment. Washington DC: National Academy Press.

Russell, M., & Airasian, P. (2012). Classroom Assessment: Concepts and Applications. p. 157-158., NY: McGraw-Hill.

CTLT 2014 www.ctlt.org


Recommended