Post on 04-Jul-2015
transcript
Basic Assessment Literacy
Everything You Wanted to Know
About Testing But Was Never That Interested
Smartest kid in class
Sticky Ideas
Quality data requires a quality test
Quality tests are not easy to create
Assessment literacy means:
5 Blind Men & an Elephant?
Assessment literacy means:
Gathering dependable information about student achievement
Using the assessment process to promote maximum achievement
What’s a “good test?”
“The most basic and obvious answer to that question is that good tests measure what you want to measure, and bad tests do not.”
Ben ClayIs This a Trick Question?
Questions to ask
Why?Users/Uses
What?Targets
How Much?Sample
How?Method
Accurate?Avoid Bias
Why?Users/Uses
What?Targets
How Much?Sample
How?Method
Accurate?Avoid Bias
Why?
Is the Purpose for the assessment clear?o Why are these targets being assessed? o Who will use the information?o What will the information be used for?
Why?Users/Uses
What?Targets
How Much?Sample
How?Method
Accurate?Avoid Bias
What?
Are the Targets clear and appropriate?o Specific & clearly defined expectationso Center on the core of the disciplineo Aligned to program / course / unit level
curriculum
Four target types
Knowledgeo George Washington / 1787 / US
Constitution
Reasoningo Evaluate differences between government
types
Skillso Analyze a primary source document
Productso Create a reasoned argument for allowing
immigration reform
An assessment that accurately reflects its intended target is Valid
Why?Users/Uses
What?Targets
How Much?Sample
How?Method
Accurate?Avoid Bias
How?
Different targets require different methodso Match appropriate method with intended
purpose and target
General categories
Objectiveo Student selects correct response from
several alternatives / supplies a word or short phrase to answer a question
o Easy to grade / hard to create
Subjectiveo Student organizes / presents an original
answero Hard to grade / easy to create
Objective vs. Subjective o “By definition, no test can be truly objective”
Selected Response o Multiple Choice, True/False, Matching, Fill-in-
the-blank
Essayso Short Answer, Extended Response
Performance Tasks Personal Communication
Specific categories
Mark Pett ©2001
Let’s try it!
Name the parts of the human skeletono Selected response
Appraise a composition on the basis of its organizationo Short answer essay
Demonstrate safe lab skillso Performance task
Cite four examples of satire in Huckleberry Finno Short answer essay
Design a logo for a web pageo Performance task
Describe the impact of a Bull Marketo Extended response essay
Diagnose a physical ailmento Extended response essay
List important mental attributes necessary in an athleteo Short response essay
Categorize great American fiction writerso Selected response
Why?Users/Uses
What?Targets
How Much?Sample
How?Method
Accurate?Avoid Bias
How much?
How can we Sample appropriately?o “All assessments are just a small part of an
‘ideal’ assessment of infinite length.”
“Have I gathered enough information of the right kind so that I can draw confident conclusions about student achievement?”
Why?Users/Uses
What?Targets
How Much?Sample
How?Method
Accurate?Avoid Bias
Mark Pett ©2001
Accurate?
What are possible sources of Interference or Bias?
Assessmento Poorly written, poor target-method match,
culturally biased, inappropriate reading level
Assessoro Incorrectly scored
Studento Test anxiety, cheat, guess, interpretation
different than intent
Environmento Cold, hot, poor light, time restraints, noise,
lunch, recess
Eliminating errors due to bias increase the Reliability of our assessments
What was new?
What was re-new?
Steps to Structured Response test
Prepare a blueprint
Select material to assess
Build test items from propositions
Prepare a blueprint
Step 1
401525Total
1358Rights & responsibilities of citizens
1257Structure of US government
15510Forms of government
TotalProcess & Application
Know & Understand
Content Targets
Select material to assess
Step 2
“It is always tempting to emphasize the parts of the course that are easiest to test, rather than the parts that are important to test.”
Ben ClayIs This a Trick Question?
So what’s worth assessing?
Which test items, if any, need to be changed?
Can you think of a test that included items you felt weren’t worth learning or assessing?
How do you make decisions about what’s worth assessing?
Review the Case Study
A select sub-set of all possible important elements
Professional judgment of relative importance
So . . . ?
Step 3
Build test items from propositionso “Clearly stated sentences that reflect
elements of content aligned to Course / Unit EUs & EQs.”
o Create more than blueprint requires
Examples
In a monarchy, the right to govern is secured through birth.
The executive and legislative branches differ in that the latter is elected directly by the people.
True or False
In a monarchy, the right to govern is secured through birth.
orIn a monarchy, the right to govern is
secured through the approval of those governed.
Fill in the Blank
How is the right to govern secured in a monarchy?
Multiple Choice
How is the right to govern secured in a monarchy?
A. With military power
B. Through birth
C. By popular vote
D. Through purchase
True or False
The executive and legislative branches differ in that the latter is elected directly by the people.
or Members of executive and legislative
branches are both elected directly by the people.
Fill in the Blank
Election of members of the executive and legislative branches differ in what way?
Multiple Choice
Election of members of the executive and legislative branches differ in what way?A. Legislators are restricted by term limits; presidents are notB. Legislators are elected directly; presidents are not.C. One must register to vote for legislators; not for presidentD. One must register to vote for presidents; not for legislators.
Franzipanics Quiz
Specific assessment criteria
Multiple choice
Advantageso Effective measuremento Reduces guessingo Can cover wide range of content
Disadvantageso Difficult to create
Multiple Choice suggestions
Stem is a clear, explicit & singular question
Use at least four alternatives
Avoid grammatical / sentence structural cues
Highlight the negative in negatively stated stems
Alternatives are the same format and length
All alternatives should be plausible
Avoid answer patterns
Is the question any good?
Student who knows the content will know the answer before reading the alternatives
“Your goal is not to trick students or require them to make difficult judgments about two nearly equal options.”
“Your goal is to write questions that students who understand the material will answer correctly and students who do not understand will answer incorrectly.”
Dr. John JohnsonPenn State University
Bill Watterson ©1995
True or false
Advantageso Can measure large amounts of basic
knowledgeo Students can process many questions quickly
Disadvantageso Guessing workso Need many items for reliability
True & false suggestions
Statements are absolutely true or false
Express item as simply & clearly as possible
Measure only one idea per item
Use possible misconceptions as false items
Avoid negatively stated items
Matching
Advantageso Measures large amounts of knowledge level
contento Useful for factual content areas
Disadvantageso Can be time consuming for studentso Does not measure higher learning levels
Matching suggestions
Use items that share same foundation of knowledge
Avoid grammatical cues
Keep stem list to10-15 items
Response list should be larger than stem list
Fill-in-the-Blank
Advantageso Good for 5 Ws and Ho Helps minimize guessing
Disadvantageso Difficult to assess high levels of learningo May have more than one correct answero Time consuming scoring
Fill-in-the-Blank suggestions
Omit only significant items from stem
Insure only one correct answer
Avoid grammatical cues
Blank at stem’s end
Should count for more points
Things to discuss
Possible accommodations
Student evaluations of test
Pilot version / full version
Who takes the test
Who administers the test