Basic Assessment Literacy

Everything You Wanted to Know

About Testing But Was Never That Interested

Smartest kid in class

Sticky Ideas

Quality data requires a quality test

Quality tests are not easy to create

Assessment literacy means:

5 Blind Men & an Elephant?

Assessment literacy means:

Gathering dependable information about student achievement

Using the assessment process to promote maximum achievement

What’s a “good test?”

“The most basic and obvious answer to that question is that good tests measure what you want to measure, and bad tests do not.”

Ben ClayIs This a Trick Question?

Questions to ask

Why?Users/Uses

What?Targets

How Much?Sample

How?Method

Accurate?Avoid Bias

Why?Users/Uses

What?Targets

How Much?Sample

How?Method

Accurate?Avoid Bias

Why?

Is the Purpose for the assessment clear?o Why are these targets being assessed? o Who will use the information?o What will the information be used for?

Why?Users/Uses

What?Targets

How Much?Sample

How?Method

Accurate?Avoid Bias

What?

Are the Targets clear and appropriate?o Specific & clearly defined expectationso Center on the core of the disciplineo Aligned to program / course / unit level

curriculum

Four target types

Knowledgeo George Washington / 1787 / US

Constitution

Reasoningo Evaluate differences between government

types

Skillso Analyze a primary source document

Productso Create a reasoned argument for allowing

immigration reform

An assessment that accurately reflects its intended target is Valid

Why?Users/Uses

What?Targets

How Much?Sample

How?Method

Accurate?Avoid Bias

How?

Different targets require different methodso Match appropriate method with intended

purpose and target

General categories

Objectiveo Student selects correct response from

several alternatives / supplies a word or short phrase to answer a question

o Easy to grade / hard to create

Subjectiveo Student organizes / presents an original

answero Hard to grade / easy to create

Objective vs. Subjective o “By definition, no test can be truly objective”

Selected Response o Multiple Choice, True/False, Matching, Fill-in-

the-blank

Essayso Short Answer, Extended Response

Performance Tasks Personal Communication

Specific categories

Mark Pett ©2001

Let’s try it!

Name the parts of the human skeletono Selected response

Appraise a composition on the basis of its organizationo Short answer essay

Demonstrate safe lab skillso Performance task

Cite four examples of satire in Huckleberry Finno Short answer essay

Design a logo for a web pageo Performance task

Describe the impact of a Bull Marketo Extended response essay

Diagnose a physical ailmento Extended response essay

List important mental attributes necessary in an athleteo Short response essay

Categorize great American fiction writerso Selected response

Why?Users/Uses

What?Targets

How Much?Sample

How?Method

Accurate?Avoid Bias

How much?

How can we Sample appropriately?o “All assessments are just a small part of an

‘ideal’ assessment of infinite length.”

“Have I gathered enough information of the right kind so that I can draw confident conclusions about student achievement?”

Why?Users/Uses

What?Targets

How Much?Sample

How?Method

Accurate?Avoid Bias

Mark Pett ©2001

Accurate?

What are possible sources of Interference or Bias?

Assessmento Poorly written, poor target-method match,

culturally biased, inappropriate reading level

Assessoro Incorrectly scored

Studento Test anxiety, cheat, guess, interpretation

different than intent

Environmento Cold, hot, poor light, time restraints, noise,

lunch, recess

Eliminating errors due to bias increase the Reliability of our assessments

What was new?

What was re-new?

Steps to Structured Response test

Prepare a blueprint

Select material to assess

Build test items from propositions

Prepare a blueprint

Step 1

401525Total

1358Rights & responsibilities of citizens

1257Structure of US government

15510Forms of government

TotalProcess & Application

Know & Understand

Content Targets

Select material to assess

Step 2

“It is always tempting to emphasize the parts of the course that are easiest to test, rather than the parts that are important to test.”

Ben ClayIs This a Trick Question?

So what’s worth assessing?

Which test items, if any, need to be changed?

Can you think of a test that included items you felt weren’t worth learning or assessing?

How do you make decisions about what’s worth assessing?

Review the Case Study

A select sub-set of all possible important elements

Professional judgment of relative importance

So . . . ?

Step 3

Build test items from propositionso “Clearly stated sentences that reflect

elements of content aligned to Course / Unit EUs & EQs.”

o Create more than blueprint requires

Examples

In a monarchy, the right to govern is secured through birth.

The executive and legislative branches differ in that the latter is elected directly by the people.

True or False

In a monarchy, the right to govern is secured through birth.

orIn a monarchy, the right to govern is

secured through the approval of those governed.

Fill in the Blank

How is the right to govern secured in a monarchy?

Multiple Choice

How is the right to govern secured in a monarchy?

A. With military power

B. Through birth

C. By popular vote

D. Through purchase

True or False

The executive and legislative branches differ in that the latter is elected directly by the people.

or Members of executive and legislative

branches are both elected directly by the people.

Fill in the Blank

Election of members of the executive and legislative branches differ in what way?

Multiple Choice

Election of members of the executive and legislative branches differ in what way?A. Legislators are restricted by term limits; presidents are notB. Legislators are elected directly; presidents are not.C. One must register to vote for legislators; not for presidentD. One must register to vote for presidents; not for legislators.

Franzipanics Quiz

Specific assessment criteria

Multiple choice

Advantageso Effective measuremento Reduces guessingo Can cover wide range of content

Disadvantageso Difficult to create

Multiple Choice suggestions

Stem is a clear, explicit & singular question

Use at least four alternatives

Avoid grammatical / sentence structural cues

Highlight the negative in negatively stated stems

Alternatives are the same format and length

All alternatives should be plausible

Avoid answer patterns

Is the question any good?

Student who knows the content will know the answer before reading the alternatives

“Your goal is not to trick students or require them to make difficult judgments about two nearly equal options.”

“Your goal is to write questions that students who understand the material will answer correctly and students who do not understand will answer incorrectly.”

Dr. John JohnsonPenn State University

Bill Watterson ©1995

True or false

Advantageso Can measure large amounts of basic

knowledgeo Students can process many questions quickly

Disadvantageso Guessing workso Need many items for reliability

True & false suggestions

Statements are absolutely true or false

Express item as simply & clearly as possible

Measure only one idea per item

Use possible misconceptions as false items

Avoid negatively stated items

Matching

Advantageso Measures large amounts of knowledge level

contento Useful for factual content areas

Disadvantageso Can be time consuming for studentso Does not measure higher learning levels

Matching suggestions

Use items that share same foundation of knowledge

Avoid grammatical cues

Keep stem list to10-15 items

Response list should be larger than stem list

Fill-in-the-Blank

Advantageso Good for 5 Ws and Ho Helps minimize guessing

Disadvantageso Difficult to assess high levels of learningo May have more than one correct answero Time consuming scoring

Fill-in-the-Blank suggestions

Omit only significant items from stem

Insure only one correct answer

Avoid grammatical cues

Blank at stem’s end

Should count for more points

Things to discuss

Possible accommodations

Student evaluations of test

Pilot version / full version

Who takes the test

Who administers the test