Standards for Question Writing

Mary Grantner, MASenior Manager, Physician

Assessment

Fundamentals

What is a test?

An instrument that tells us something about student knowledge, skills, and/or abilities.

Fundamentals

What is a test item?Cue requiring student response with rules for scoring response.

Response can be prompted by an instruction to perform a task,

or a question requiring an answer.

Fundamentals

By definition, ARS employs a question requiring an answer.

The default test item is multiple-choice.• Stem• Responses• Single correct response (best-answer

format)

Use those nifty keypads

A multiple-choice test item can measure which of the following student achievements?

A. Critical thinking skill

B. High-inference psychomotor skill

C. Declarative knowledge

D. High-inference cognitive ability

A multiple-choice test item can measure which of the following student achievements?

A. Critical thinking skill*

B. High-inference psychomotor skill

C. Declarative knowledge*

D. High-inference cognitive ability*

Approach to Writing

• Identify Key Concepts– What do you want to measure?

• Identify Actions– What do you want the learner to be able to

do?

• Ask for Direct Action/Concept• Provide a Single Correct Answer

Identify Key Concepts

• Start with learning objectives• When they walk out of the room, will

learners ask “so what?” What should they now understand?

• Put each concept into a declarative sentence.

Identify Actions

• Drill down – what should the learner do with the concept?

• Verbs “do” – consider some verbs that apply to the Key Concept.

• Put each action into a declarative sentence.

A Word about Verbs

• Strong verbs are vital to any good writing.• Be direct and active.

– Form of “to be” + past participle – Mary was given a book by John – is passive.

• Avoid forms of “to be” when possible.• And (drumroll please)…

“When you see an adverb, kill it.”

-- Mark Twain

What does it mean to walk quickly?

Briskly?

Urgently?

Fast?

Choosing Verbs

Measure Recall or Understanding, use:

Characterize

Define

Demonstrate

Describe

Find

Illustrate

Choosing Verbs

Measure Problem-solving, use:

Classify

Conclude

Diagnose

Determine

Plan

Recommend

Ask for Direct Action/Concept

• Focus on a single Key Concept at a time.– Less really is more.

• Using an active verb, ask a direct question.

• Ask the learner to do something.• Keep it simple!

Provide a Single Correct Answer

• Focus on “real life”.• Be as brief as possible.

– Answer should contain only what is absolutely necessary.

• Keep it simple!

Keypads Up!

Which of the following best defines adverb?

a. Modifies a noun or other substantive.

b. Ob to be executed by item writer.

c. Expresses action, existence, or occurrence.

d. Modifies a verb, adjective, or other adverb.

Which of the following best defines adverb?

a. Modifies a noun or other substantive.

b. Obliged to be executed by item writer.

c. Expresses action, existence, or occurrence.

d. Modifies a verb, adjective, or other adverb. *

Try ThisIdentify Key Concept:

Pulmonary embolism (PE) and deep venous thrombosis (DVT) are part of the same complex and common process (i.e. venous thromboembolism), which has a wide spectrum of clinical presentations and consequences, ranging from no symptoms or sequelae to a lethal outcome. DVT usually starts in the deep calf veins and propagates superiorly, and it is associated with symptomatic PE in about one-third of untreated patients. DVT is commonly confused with other conditions such as cellulitis, leg edema, and chronic venous insufficiency, and PE is also notoriously difficult to diagnose.

You Decide

• As the content expert, it’s up to you to decide what is most important.

• Let’s choose this:

PE is also notoriously difficult to diagnose

• Write a direct question based on the paragraph.

The Question

Which of the following statements best explains the difficulties in diagnosing lower extremity DVT?

Answer:

Is clinically similar to other conditions of the lower extremities.

Wrong Answers

• Also called distractors, are hard to write.• Try using these:

– Common misconceptions you’ve seen;– Misapply a principle (i.e. add when you should

subtract);– Ways in which things are related (cause or

coincidence?).

Which of the following statements best explains the difficulties in diagnosing lower extremity DVT?

a. Is a distinctly different process than PE.b. Occurs rarely in cases of symptomatic PE.c. Is clinically similar to other conditions of the

lower extremities.*d. Usually originates in difficult to image deep

veins of the thigh.

Other Rules/Hints

• Just as in life, a fact is seldom absolutely true or absolutely false. No T/F questions!

• “All / None of the Above” is essentially T/F – Not allowed.

• Avoid any negative question – Which is false?– Which is incorrect?

Other Rules/Hints• Stem is short and options long – a poor item!

– As much information as possible in the stem and keep the options short

• Avoid absolute terms. – Words like “always” or “never” are rarely true in life

and so are almost never the right answer on a test.

• Avoid word repeats (word that appears in both stem and response); points to a correct answer.

• Language in the options must be parallel (i.e. all diagnoses, or all anatomic entities).

Questions?

Mary Grantner

mgrantner@rsna.org

630-368-3733