Standards for Question Writing

Mary Grantner, MASenior Manager, Physician

Assessment

Fundamentals

What is a test?

An instrument that tells us something about student knowledge, skills, and/or abilities.

Fundamentals

What is a test item?Cue requiring student response with rules for scoring response.

Response can be prompted by an instruction to perform a task, or a question requiring an answer.

FundamentalsBy definition, ARS employs a question requiring an answer.

The default test item is multiple-choice.• Stem• Responses• Single correct response (best-answer

format)

Use those nifty keypads

A multiple-choice test item can measure which of the following student achievements?

A. Critical thinking skillB. High-inference psychomotor skillC. Declarative knowledgeD. High-inference cognitive ability

A multiple-choice test item can measure which of the following student achievements?A. Critical thinking skill*B. High-inference psychomotor skillC. Declarative knowledge*D. High-inference cognitive ability*

Approach to Writing• Identify Key Concepts

– What do you want to measure?• Identify Actions

– What do you want the learner to be able to do?

• Ask for Direct Action/Concept• Provide a Single Correct Answer

Identify Key Concepts• Start with learning objectives• When they walk out of the room, will

learners ask “so what?” What should they now understand?

• Put each concept into a declarative sentence.

Identify Actions• Drill down – what should the learner do

with the concept?• Verbs “do” – consider some verbs that

apply to the Key Concept.• Put each action into a declarative

sentence.

A Word about Verbs• Strong verbs are vital to any good writing.• Be direct and active.

– Form of “to be” + past participle – Mary was given a book by John – is passive.

• Avoid forms of “to be” when possible.• And (drumroll please)…

“When you see an adverb, kill it.”

-- Mark Twain

What does it mean to walk quickly?Briskly?

Urgently?Fast?

Choosing VerbsMeasure Recall or Understanding, use:

CharacterizeDefine

DemonstrateDescribe

FindIllustrate

Choosing VerbsMeasure Problem-solving, use:

ClassifyConcludeDiagnoseDetermine

PlanRecommend

Ask for Direct Action/Concept

• Focus on a single Key Concept at a time.– Less really is more.

• Using an active verb, ask a direct question.

• Ask the learner to do something.• Keep it simple!

Provide a Single Correct Answer• Focus on “real life”.• Be as brief as possible.

– Answer should contain only what is absolutely necessary.

• Keep it simple!

Keypads Up!Which of the following best defines adverb?a. Modifies a noun or other substantive. b. Ob to be executed by item writer.c. Expresses action, existence, or

occurrence.d. Modifies a verb, adjective, or other

adverb.

Which of the following best defines adverb?

a. Modifies a noun or other substantive. b. Obliged to be executed by item writer.c. Expresses action, existence, or

occurrence.d. Modifies a verb, adjective, or other

adverb. *

Try ThisIdentify Key Concept:

Pulmonary embolism (PE) and deep venous thrombosis (DVT) are part of the same complex and common process (i.e. venous thromboembolism), which has a wide spectrum of clinical presentations and consequences, ranging from no symptoms or sequelae to a lethal outcome. DVT usually starts in the deep calf veins and propagates superiorly, and it is associated with symptomatic PE in about one-third of untreated patients. DVT is commonly confused with other conditions such as cellulitis, leg edema, and chronic venous insufficiency, and PE is also notoriously difficult to diagnose.

You Decide• As the content expert, it’s up to you to

decide what is most important.• Let’s choose this:

PE is also notoriously difficult to diagnose

• Write a direct question based on the paragraph.

The QuestionWhich of the following statements best explains the difficulties in diagnosing lower extremity DVT?

Answer:Is clinically similar to other conditions of the lower extremities.

Wrong Answers• Also called distractors, are hard to write.• Try using these:

– Common misconceptions you’ve seen;– Misapply a principle (i.e. add when you should

subtract);– Ways in which things are related (cause or

coincidence?).

Which of the following statements best explains the difficulties in diagnosing lower extremity DVT?

a. Is a distinctly different process than PE.b. Occurs rarely in cases of symptomatic PE.c. Is clinically similar to other conditions of the

lower extremities.*d. Usually originates in difficult to image deep

veins of the thigh.

Other Rules/Hints• Just as in life, a fact is seldom absolutely

true or absolutely false. No T/F questions!• “All / None of the Above” is essentially T/F

– Not allowed.• Avoid any negative question

– Which is false?– Which is incorrect?

Other Rules/Hints• Stem is short and options long – a poor item!

– As much information as possible in the stem and keep the options short

• Avoid absolute terms. – Words like “always” or “never” are rarely true in life

and so are almost never the right answer on a test. • Avoid word repeats (word that appears in both

stem and response); points to a correct answer.• Language in the options must be parallel (i.e. all

diagnoses, or all anatomic entities).

Questions?

Mary Grantnermgrantner@rsna.org

630-368-3733