Date post: | 23-Jan-2015 |
Category: |
Education |
Upload: | chicago-elearning-technology-showcase |
View: | 303 times |
Download: | 2 times |
WRITING TEST QUESTIONS FOR
ANY TEST ENGINE
Sharon Gander, M.Ed., CPT
Judith Hale, Ph.D., CPT
OVERVIEW
Overview and presenter introductions
Test engine challenges
Conditions & scenarios
Questions and responses
Quality Distractors
Quality checks
Test Engine Debrief
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 6
TAKE TIME TO:1 – OPEN A WORD PROCESSOR
2 – OPEN OR DOWNLOAD A TEST ENGINE
3 – DRAFT 2-3 TEST QUESTIONS ON ANY TOPIC
4 – CHECK THAT YOU HAVE RECEIVED THE HANDOUTS
3 – WORK WITH A PARTNER ON ACTIVITIES – FIND YOUR PARTNER NOW!
Prepare for this session by doing the steps above!
OBJECTIVES1. Write a one-paragraph scenario opening to a question.
2. Write a question with one right answer.
3. Write three plausible answers and one correct answer to the posed question.
4. Identify strategies for validating the preferred response to posed question.
5. Check answers for “tells” that give away the right or wrong answer.
6. Explain the value of using Plain Language and Global English in writing test questions.
7. Leave with three tools to apply in writing test items for learning or certifications.
8. Recognize test engine characteristics that impact item design, delivery and validation.
9. Prototype one or more questions in a free test-engine.
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 8
Immediately after this session, begin building scenario-based questions for courses, certifications and test-outs as opportunities arise.
TEST ENGINE CHALLENGES
What challenges do you experience in writing test items for online test-
engine delivery?
TEST ENGINE CHALLENGES
Developer familiarity with test engine while writing
The need to write clear questions and responses
It is difficult to get valid content
Need to overcome the strong tendency to writerecall (knowledge/facts)
Difficult to write questions that make sense to the test candidate
Continuous error checking dulls the senses
Difficult to use appropriate question type
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 10
CONDITIONS AND SCENARIOS
How do you create performance-based test questions that work
in any test engine?
PERFORMANCE-BASE QUESTIONS
The heart of a performance-based question is a condition.
You could use:
Given a specific situation and resources…
Or
Describe the situation in 1-2 paragraph story(a scenario)
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 12
Given the fact that test engines and survey engines handle question sets and answer evaluation very differently and the fact that you need to produce quality assessment questions as quickly as possible.
Which two question types are most standard across all engines?
a) True/false and multiple-choice.
b) Fill-in-the-blank and ordering.
c) Multi-select and ranking.
d) Essay and matching.
13THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
SOAR – A SCENARIO TECHNIQUE
Situation – essential facts of the setting
Obstacles – the hurdles that will prevent action or require unique actions; problems and challenges
Action – the key decisions and/or action steps are required
Result – the desired outcome
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 14
ACTIVITY #1: PREPARE QUESTION
1. Think of a situation of your own (or use the one below).
2. Describe the conditions using your SOAR template
3. Work with partner – 3 minutes
Write for the action. What action will overcome the obstacle to gain the desired result
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 15
Can’t think of a situation? Try this:
You are tasked with assessing SME’s skills for writing content. Your organization wants SMEs to apply adult learning theory to content that will be used in e-learning solutions. Describe a scenario that will assess this skill.
QUESTIONS AND RESPONSES
Why use multiple-choice questions with only one correct response?
WHY ONE CORRECT ANSWER?
Clarity Decreases time readers spend figuring out what
the question is Decreases the time spent guessing at what the
question writer intended
Defensibility Can be supported by unambiguous documented
references Provides clear evidence of knowledge when
answering knowledge-type questions Provides clear evidence of performance when
answering performance-based questions
Technology Scoring and tracking answers Works with every test engine
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 17
WHAT IS A CLEAR QUESTION? One that gives the reader clear signals about “what to answer” by:
Starting with a question word (5Ws/H) Who, what, when, where, why, or how
Ending with a question mark Avoiding the hanging question that results from
using a colon that leaves the reader trying to read your mind
Using only one verb
Using action verbs Using either the present or past tense
Avoiding the passive “is” “was”
Using either an actor, “you” or implied “you”
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 18
ACTIVITY #2: WRITE A QUESTION
1. Using your SOAR scenario and the item writing template
2. Write a question with ONE right answer
More about distractors and incorrect responses coming soon.
3. Work with partner – 3 minutes
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 19
Write for the action.
Find a way to ask, what action will overcome an obstacle to gain the desired result?
WHAT IS A QUALITY DISTRACTOR
Should one response be obviously wrong?
Should I use the “ none of the above” or “all of the above responses?
Should I use answers like “a and c only”?
QUALITY DISTRACTORS
Are plausible – all of them are plausible Real and real-life
Force discrimination Requires test-taker to evaluate options
Separate the less experienced from the highly experienced
Requires test-taker to have experience (and education) to choose the most effective response
Technically accurate May distract with variance in technical details
Forces accuracy in technical aspects
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 21
MULTIPLE CHOICE RESPONSES
One Correct Response With documented reference of correctness
Not just an experts “feeling” that it’s right
Three Plausible Responses One mistake that most novices make
(a common error)
One mistake that soon-to-be experts tend to make
One mistake that even many experts make
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 22
ANSWER VALIDATION STRATEGIES
Need to validate your preferred response?
Document references that validate the correct response (#1)
Ask an acknowledged expert to write a white paper on the subject
Poll experts and write a white paper about the polled results
Give question to experts and novices; determine which answer the experts choose
Other? Anything that documents a preferred response other than “I believe this to be true.”
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 23
In a multiple choice question, the optimal answer set is four answers with one right answer.
What should the other three answers be?
a) Two wrong answers plus one “all correct” answer.
b) Two plausible answers plus one obviously wrong answer.
c) One plausible answer, one wrong answer, plus an “all correct” answer.
d) Three plausible answers typical of different skill levels.
24THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
ACTIVITY #3: PLAUSIBLE RESPONSES1. Write 3 plausible and 1 correct response to your question
2. Enter your question in your test engine.
3. View your question in the test engine as though you were a test-taker.
4. Work with partner – 5 minutes
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 25
Checkpoint
Is there a difference between what you planned to write and what the test engine allowed or how the question displayed? Why?
QUALITY ASSURANCE
How do you check your work before publishing it?
BASIC QUALITY ASSURANCE
• Grammar, spelling, punctuation
• Looking for “tells” (hints that give away the correct response)• Length of responses
• Change in verb tense
• Change in actor
• Change in technical terms
• Change in punctuation
• Ensure that the question stem• Uses only one action verb
• Starts with question work (5Ws/H)
• Ends in question mark
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 27
Key words, changes in grammar style and verb tense can provide “tells” (hints to the right answer) among the plausible distractor answers.
Which of the following techniques would also prevent a “tell” in your answer set?
a) Use mainly very short one or two word answers.
b) Use very precise technical terminology or phrases.
c) Use answers that are the relatively the same length.
d) Use only present tense verbs in plausible answers.
28THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
In a multiple choice question, the optimal answer set is four answers with one right answer and:
a) 3 wrong answers
b) 2 plausible answers +1 obviously wrong answr
c) Use one plausible answer, onw very wrong answer, and one “all correct” answer.
d) Three plausible answers typical of different skill levels.
29THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
What does the question really test?
WHAT QA ISSUES?
ADVANCED QUALITY ASSURANCEPlain Language
• Active voice (avoid “is” and “was”) – go for the action
• Use strong clear noun – a person, place or thing (avoid “it” “they”)
• Use simple sentence structures – end your thought with a period and limit the number of comma and colons.
• Use “you” or the implied “you” command when the reader is required to take action
• Spell out acronyms
• Avoid jargon
• Organize your points
Global English• Avoid the use of country specific terms (e.g., USDA, Rte 66, M1, Kindergarten)
• Describe common situations rather than using local terms (e.g., tsunamis, School District, US green building code)
Reading Level (approx. 9th grade)• Watch the long names with many capitalized nouns in a row
• Put periods at end of bullet-points
• Avoid jargon
• Limit use of technical terms
• Use short paragraphs
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 30
Your current project must provide an assessment with many questions that will be used in many countries by professionals with many language preferences.
What is the strongest reason for using Plain Language and Global English in writing test questions?
a) Your question items will all be in one verb tense for reading ease.
b) Your questions will be clear to anyone who reads English.
c) Your answers are researched, justifiable and measurable.
d) Your work can be used internationally without translation.
31THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
How did the test engine affect your question?
What happened to your quality assurance when you entered your question into the test engine?
What test engine issues did you experience?
Did you put your questions in a work processor first? If so, how did that impact your results?
TEST ENGINE IMPACT
ERROR PREVENTIONUse a QA Checklist like the one provided Check the writing, spelling and grammar
Review questions (offline) with an expert audience who has not seen them• Do they validate the question, the correct answer and the validation
resource (text, website, document)
Load your items into the online format Does the item display correctly? Does the answer correctly evaluate as right or wrong? Does the answer/score transfer to the LMS correctly?
Run an online pilot with both expert and non-experts candidates Feedback on questions and test process Cut scores
Modify the test as needed
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 33
YOU HAVE HAD THE CHANCE TO:
1. Write a one-paragraph scenario opening to a question.
2. Write a question with one right answer.
3. Write three plausible answers and one correct answer to the posed question.
4. Identify strategies for validating the preferred response to posed question.
5. Check answers for “tells” that give away the right or wrong answer.
6. Explain the value of using Plain Language and Global English in writing test questions.
7. Leave with three tools to apply in writing test items for learning or certifications.
8. Recognize test engine characteristics that impact item design, delivery and validation.
9. Prototype one or more questions in a free test-engine.
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 34
In the next 7 days, write at least 5 scenario-based, multiple-choice questions with 1 correct and 3 plausible responses.
QA them with the checklist and enter them into a test engine.
THE CHALLENGE!
WRITING TEST QUESTIONS FOR
ANY TEST ENGINE
Sharon Gander, M.Ed., CPT
Judith Hale, Ph.D., CPT
Contact us:
Sharon Gander at [email protected]
Judy Hale at [email protected]
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 37
SESSION EVALUATIONPlease take the time to provide us with your session feedback using any of the following methods.
QR Code
1. Download a QR reader to your Smartphoneor tablet
2. Scan the QR code from this slide, yoursession evaluation sign, or your program
3. Launch and complete the evaluation
URL Address
1. Locate the URL for this session from this slide, you session evaluation sign, or your program.
2. Enter the URL Code for this session into the browser of your laptop, Smartphone or tablet
3. Launch and complete the evaluation
URL: https://www.questionmark.com/cets07