ITEM WRITING
HANDBOOK
2019
Version 6.0
About Inteleos Inteleos is an internationally recognized non-profit global medical certification organization that
manages the American Registry for Diagnostic Medical Sonography® (ARMDS®), Alliance for
Physician Certification & Advancement™ (APCA™) and the Point-of-Care Ultrasound (POCUS)
Certification Academy™.
About ARDMS The American Registry for Diagnostic Medical Sonography®(ARDMS®), incorporated in 1975, is
an independent, nonprofit organization that administers examinations and awards credentials in
areas of ultrasound. Our credentials have been awarded to over 90,000 medical professionals
worldwide and are recognized as the international standard in sonography credentialing.
ARDMS administers examinations and awards credentials for the following specialties:
• Registered Diagnostic Medical Sonographer® (RDMS®)
• Registered Diagnostic Cardiac Sonographer® (RDCS®)
• Registered Vascular Technologist® (RVT®)
• Registered Musculoskeletal™ Sonographer (RMSKS™)
These credentialing programs are accredited by the American National Standards Institute (ANSI)
to the International Organization for Standardization (ISO) 17024 Standard for organizations that
certify personnel.
MISSION AND VISION ARDMS Vision: The American Registry for Diagnostic Medical Sonography (ARDMS) creates the
global standards of professional excellence in sonography.
ARDMS Mission: The American Registry for Diagnostic Medical Sonography (ARDMS) empowers
sonographers to provide exceptional patient care through rigorous assessments and continual
learning.
©2019 American Registry for Diagnostic Medical
Sonography, Inc.
All rights reserved.
2019 Version 6.0
Table of Contents Introduction .............................................................................................................................................. 1
Purpose of this Handbook .................................................................................................................. 1
How to Use this Handbook ................................................................................................................. 1
Background Information.......................................................................................................................... 2
The role of testing in professional credentialing ............................................................................... 2
Why sonography professionals take ARDMS examinations ............................................................. 2
Security of Test Content .......................................................................................................................... 3
Considerations when writing a test item ........................................................................................... 4
Coming up with an idea ................................................................................................................. 4
Thinking about thinking ................................................................................................................. 5
Cognitive demand versus item difficulty ....................................................................................... 7
Putting it all together ...................................................................................................................... 7
Characteristics of High-Quality Items ..................................................................................................... 8
Fairness in Testing ................................................................................................................................... 8
The Item Development Process .............................................................................................................. 9
Item Writing Guidelines ........................................................................................................................ 10
General submission guidelines ....................................................................................................... 10
Guidelines for writing the stem ....................................................................................................... 11
Guidelines for writing the answer options ...................................................................................... 14
Submitting Items ................................................................................................................................... 17
References ............................................................................................................................................ 19
Appendix A – Glossary of Terms .......................................................................................................... 20
Appendix B – Item Writing Checklist ................................................................................................... 22
Appendix C – Guidelines for Submitting Images and Digital Video Clips .......................................... 23
Appendix D – Taxonomy of Media-Enhanced Items ........................................................................... 24
Appendix E – Sample Stem Starters ................................................................................................... 34
Appendix F – Instructions for Submitting Items in the RIW Portal .................................................... 35
1 ARDMS Item Writing Handbook
Introduction
Purpose of this Handbook
Creating exam questions – or items – that are relevant and reflective of current practice can be a
difficult task. Item writing requires not only subject matter expertise but also training and practice
in writing test items to appropriately elicit knowledge from others. The purpose of the American
Registry for Diagnostic Medical Sonography (ARDMS) Handbook, Version 6.0 (the Handbook) is to
help both novice and experienced item writers in creating quality items.
Features of this Handbook include:
❖ A glossary (Appendix A) defining terminology used in the item writing process (terms
defined in the glossary are in italics the first time they are used in the Handbook)
❖ Presentation of item writing guidelines and best practices
❖ Examples of item writing Do’s and Don’ts
❖ Information on how to submit items and related images, digital video clips, and graphics
How to Use this Handbook We recommend that Subject Matter Experts (SMEs) new to item writing read through the
Handbook in its entirety. Following this, the Handbook should be retained as a handy reference
when questions regarding the item writing process arise. To aid in using the Handbook as a
reference, important information is highlighted in call-out boxes and illustrations.
One section of the Handbook that should be frequently referred to when writing test items is
Appendix B, which contains item writing guidelines that should be carefully followed. These
guidelines are based upon best practice and testing research. They are discussed in more detail
in the body of this Handbook.
ARDMS is the leading credentialing body for the sonography
community. Our success is due to our volunteers who share our
mission of quality care and patient safety. Since 1975, our exam
questions have been written by credentialed sonography
professionals active in the field. The knowledge and experience of
our dedicated volunteers are vital to making certain that our exams
are relevant and reflective of current clinical practices.
THANK YOU for your commitment to excellence in your profession.
2 ARDMS Item Writing Handbook
Background Information
The role of testing in professional credentialing
For many professions, examinations are used as the means by which professional competence is
judged. The measure used to make this judgment is reflected in the following question:
“Does the candidate meet or exceed minimum competence?”
When writing test items, the item writer should focus on important job elements and create items
that allow all candidates to demonstrate the knowledge and skills required to practice the
sonography profession competently. Each item should examine:
✓ Knowledge all professionals should know to practice their field safely and competently
✓ Ability to apply what they know in their work environment AND/OR
✓ Various techniques or maneuvers professionals use in their work environment
The content of each ARDMS examination is intended
to address subject matter delineated in the content
outline of the examination. The content outline is
derived from a practice analysis of tasks performed
on the job. Thus, each examination obtains its
pertinence to current practice because each item
written for the examination is directly linked to a task
on the content outline.
Content outlines for ARDMS examinations are
available on the ARDMS website and in the Resource Center of the volunteer portal for item
writers to reference.
Why sonography professionals take ARDMS examinations
ARDMS credentials help patients, healthcare professionals, and healthcare employers identify
sonography professionals who meet or exceed the most rigorous sonographer criteria in the
industry. Additionally, ARDMS credentials help sonography professionals:
Practice analysis
Content outline
Exam items
3 ARDMS Item Writing Handbook
✓ Demonstrate their commitment to the sonography profession
and its performance standards;
✓ Remain current on new trends in ultrasound and ultrasound
technology by requiring professionals to complete annual
training classes; and
✓ Establish their professional credentials by recognizing their
individual professional achievements.
Security of Test Content ARDMS is committed to preserving the integrity of items and credentialing procedures by ensuring
the security of examination materials. All item writers must read, understand, and adhere to a set
of terms which govern their participation in the ARDMS exam development process. These terms
were outlined in the Statement of Service (SOS) that all exam development volunteers must sign.
Your signature will be kept on file in the Inteleos Volunteer Engagement office. ARDMS requires
that you sign the SOS on an annual basis.
The Mechanics of Item Writing
Anatomy of a test item The primary item type used across all ARDMS examinations is the single-select multiple-choice
item (SSMC).
❖ All items submitted must be your own original work and not published
elsewhere.
❖ All items, and any associated media (e.g., images or videos) will become
the intellectual property of Inteleos and may not be used elsewhere.
❖ Items should be submitted through the method agreed upon with
Inteleos staff.
❖ Items should NOT be transmitted via e-mail.
❖ NO copies of items should be retained by item writers.
Security – Your Role as an Item Writer
4 ARDMS Item Writing Handbook
A test item is like a puzzle piece.
SSMCs consist of three parts:
❖ The stem asks a question relating to a specific area of the content outline.
❖ The key is the correct answer to the question asked in the stem.
❖ Three distractors are plausible but incorrect answers to the question asked in the stem.
SSMCs may be media-enhanced, and Appendix C contains guidelines for submitting images.
ARDMS examinations may also use other media-enhanced item types in addition to SSMCs.
Appendix D provides a taxonomy of these other media-enhanced item types and item writing
guidelines specific to each of these are included in Appendix D.
Considerations when writing a test item
Think about a test item as a piece of a puzzle with the entire
examination providing a picture of each candidate’s competency.
Just like each puzzle piece contributes a small piece of
information about what the picture on the puzzle looks like, so
should each test item contribute a small piece of information
about the candidate’s knowledge and skills. A poorly written test item means that it is not
contributing as it should to providing a picture of candidate competence.
Coming up with an idea
Before starting item writing projects, the Exam Program Manager conducts a gap analysis of the
examination’s item bank to determine content areas in which more items are needed. Item
writers then may be directed to write items to these specific areas.
EXAMPLE OF A SINGLE-SELECT MULTIPLE-CHOICE ITEM
Axial resolution can be determined by evaluating which quality control test?
A. A flat interface
B. A round hole in a plastic block
C. Closely-spaced parallel membranes
D. Increasingly closely spaced fine wire
➔ STEM
➔ KEY
DISTRACTORS
5 ARDMS Item Writing Handbook
Any writing exercise – including writing test items – begins with coming up with an idea. What do
you want to write about? Common errors seen in the workplace are good places to start when
trying to think of ideas for test items related to a particular task from the content outline. Test
items may also be developed in relation to a specific image or video.
Thinking about thinking
Performing a job task – or answering a test item – requires thinking. Sometimes this thinking only
involves recalling information. More frequently, however, it requires a higher-level thinking skill,
such as correlating data from different sources.
The type or level of thinking involved in successfully completing a task is termed
cognitive demand. The cognitive demand required by any particular job task or
test item is determined by the level of thinking typically needed to perform the
task or respond to the item. The actual level required is based upon the
individual. Individuals who routinely perform a job task may simply recall how they
have done the task in the past. For example, an experienced sonographer
reviewing an ultrasound may use pattern recognition skills to identify normal structures, whereas
a novice sonographer may require a higher-level thought process. Similarly, if a test taker has
been exposed to the specific content of a test item, through formal training or study, their
response may be based upon simple recall, even if the test item was intended to assess a higher-
level thinking skill.
Despite the subjectivity that may be involved in assigning cognitive demand to test items, test
items should be written to target a specific cognitive level. These cognitive levels can be thought
of as hierarchical, with the simplest level of thinking on the lowest level of the hierarchy and
higher levels on successive levels.
Inteleos classifies items to three cognitive levels:
Evaluation/Synthesis
Application/Analysis
Knowledge/Comprehension
6 ARDMS Item Writing Handbook
Each individual test item should be written to an appropriate cognitive level. The verb contained
in the task statement from the content outline that the item is being written to address provides
an indication of the appropriate level. Items should be written at or below the cognitive level
indicated by the verb in the task statement.
Examinations should include items across all three levels to fully measure the breadth of
knowledge and skills sonography professionals use to perform their work tasks. An exam displays
the best fidelity to practice when it includes items from across the full range of cognitive levels.
Level 1: Knowledge/Comprehension
Description:
Recall a fact, information or a procedure
Sample action verbs
Define, identify, recognize, recall
Sample item
According to ALARA (As Low as Reasonably
Achievable) guidelines, which parameter
should be minimized?
A. Frequency
B. *Transmit power
C. Time gain compensation
D. Overall gain
Level 2: Application/Analysis
Description:
Engage mental processes beyond habitual
response using information or conceptual
knowledge; thinking which requires two or
more steps; use reasoning
Sample action verbs
Apply, use, measure, obtain, compare,
calculate
Sample item
What is the Nyquist limit if a pulsed-wave
Doppler instrument has a pulse repetition
frequency of 16 kHz?
A. 5 kHz
B. *8 kHz
C. 10 kHz
D. 32 kHz
Level 3: Evaluation/Synthesis
Description:
Make judgments based on criteria and
standards; put parts together to make a
whole; use complex reasoning
Sample action verbs
Analyze, assess, evaluate, synthesize,
correlate
Sample item
Which structure is indicated by the arrow on
this image?
A. Esophagus
B. Common carotid artery
C. Longus colli muscle
D. *Internal jugular vein
*indicates the correct response
7 ARDMS Item Writing Handbook
Cognitive demand versus item difficulty
Cognitive demand and item difficulty are often used interchangeably. However, while these
concepts are related, they are not the same thing. Items that require a greater cognitive demand
frequently are more difficult, but low cognitive demand recall items may also be difficult –
particularly if they address content that is more trivial or obscure.
Putting it all together
Now you have all of the ingredients for writing test items:
Voilà! The genesis
of a test item
Next address the appropriate
cognitive level
First, focus on the
content to be tested
Cognitive demand
•The thinking skills used by a typical test taker to arrive at a correct response
Item difficulty
•A statistical property of a test item
Example:
•When was ultrasound first used for medical purposes?
•Recall question but may be difficult because it deals with a trivial fact
8 ARDMS Item Writing Handbook
But Item writing is an art as well as a science. This Handbook is intended to provide ARDMS item
writers with information that will assist you in honing the art of item writing. Once a core item is
drafted, you can use the formatting guidelines presented to refine your item so that it assesses
what it is meant to assess.
Characteristics of High-Quality Items The critical question addressed in any ARDMS examination is, “Is the individual competent
enough to practice sonography?” Test items should be written so they elicit information needed to
answer this question for each candidate. How the item is written should not interfere with
gathering this information. To avoid interference, items should be free of grammatical errors and
consistent in format and style.
Test-wise candidates, even if they are unfamiliar with the test content, can sometimes use clues
in the item to determine the correct answer. Following item writing guidelines can assist in
preventing cueing of correct responses.
Appendix B provides an item development checklist that provides a quick reference guide to use
as you are writing your test items and to serve as a quality check before submitting them. The
item writing guidelines used to create the checklist will be discussed in more detail in the body of
this Handbook. But first, fairness as a testing concept requires special attention.
Fairness in Testing Fairness in a testing context means that test items do not
disadvantage any candidates by unnecessarily including
content that would be more familiar to certain groups (e.g.,
geographic, socioeconomic, ethnic) than to others.
The introduction of bias into an examination raises exam
validity concerns because it means that the exam is
assessing something other than what it is intended to
assess. Bias can also have legal and ethical implications. For
these reasons, added attention should be given to fairness
and the potential for bias during the item writing process.
9 ARDMS Item Writing Handbook
Types of item review
The Item Development Process Writing a test item is an important step – but it is just that, a first step. Once you have completed
and submitted a test item, it undergoes a series of reviews. The goal of the review process is to
make certain that test content is developed according to best practice and that high-quality test
items are used to evaluate candidate competence.
Ed
ito
ria
l A review to determine whether the item is grammatically correct, meets style guidelines, and free of typographical and spelling errors
Inh
ou
se
SM
E A review by an Exam Clinical Specialist, an Inteleos staff member who holds ARDMS credentials; the review provides an initial check that the item is clinically relevant and reflective of current practice, and meets fairness criteria.
Co
mm
itte
e A review by the Assessment Committee specific to the examination; the review provides a check that the item is appropriate for the examination, the key is correctly identified, and the item is fair and unbiased.
Sta
tisti
ca
l Statistical properties (item difficulty and item discrimination) of all test items are evaluated at various times. Each item undergoes pretesting prior to being used as a scored item.
FAIRNESS GUIDELINES
✓ Unless required to address a job-specific or job-related content:
O References to gender, ethnicity, nationality, religious affiliation, etc. should
be avoided.
O Items should be gender-neutral.
O Controversial material that might invoke an emotional response should be
avoided.
✓ Any unnecessary referrals to words or content that might be less familiar to some
candidate groups (e.g., geographic, socioeconomic, ethnic) should be avoided.
✓ Opinion-based items should be avoided.
10 ARDMS Item Writing Handbook
Item Writing Guidelines
General submission guidelines
Guideline 1: Submit items that are your original work.
Do not plagiarize. Copying exact language from textbooks and/or journal articles constitutes
plagiarism.
Guideline 2: Provide at least one reference for each item submitted.
All items written for Inteleos are required to have at least one reference. Items without a
reference will not be accepted.
❖ Use sources that are commonly accepted in the subject area.
❖ Recently-published articles and pending research are generally not appropriate
references, since they are likely to contain material that is not yet commonly accepted or
known.
When submitting an item reference, please include the title, author, publication date, and page(s)
of the textbook or article used to develop the item.
Guideline 3: Use correct grammar, punctuation and spelling; do not use personal
pronouns.
Grammatical errors often help candidates identify the correct answer without comprehending the
subject matter.
Guideline 4: Use vocabulary and language appropriate for the subject matter and
the candidate population.
Abbreviations and acronyms are a particular issue of concern. Spell out the term or phrase the
first time it is used and place the abbreviation or acronym after it in parentheses. Thereafter, the
abbreviation may be used.
Guideline 5: Write items to assess important material.
Items should be based on the appropriate examination content outline and should not be focused
on trivial information, experimental techniques or unresolved industry debates. Items should
address well-recognized standards.
Write items asking candidates to recall specific measurements or numeric values only if there is a
published and professionally accepted standard and if the ability to remember the measurement
or value is important and relevant to professional practice.
AVOID GIVING THESE CLUES TO THE RIGHT ANSWER
11 ARDMS Item Writing Handbook
Poor Item:
What are the normal measurements of a uterus?
A. 6 cm long, 5 cm wide, and 4 cm thick
B. *8 cm long, 5 cm wide, and 4 cm thick
C. 8 cm long, 6 cm wide, and 2 cm thick
D. 10 cm long, 8 cm wide, and 5 cm thick
Better Item:
A uterine measurement of 8 cm long, 5 cm wide, and 4 cm thick, corresponds with which
patient profile?
A. *Nulliparous
B. Postpartum
C. Pre-pubertal
D. Postmenopausal
Guidelines for writing the stem
Guideline 6: Write the stem in the form of a question.
Inteleos uses only the “question and answer” format in its examinations. Do not submit items that
are formatted as incomplete or fill-in-the-blank sentences.
Appendix E provides a listing of potential ‘stem starters.’ Please note that these are not intended
to be a comprehensive list of questions that can be used within items. Rather, this Appendix
contains examples that can serve as inspiration for you as you begin to write.
Poor Item:
A small crater in a carotid artery plaque is called:
A. A thrombus
B. *An ulceration
C. A fibrous plaque
D. A calcific plaque
Better Item:
Which carotid plaque is associated with cerebral emboli?
A. Hyperechoic
B. *Ulcerated
C. Fibrous
D. Calcific
*indicates the correct response
*indicates the correct response
12 ARDMS Item Writing Handbook
Guideline 7: Focus on a single concept, fact, or problem in each item.
A well-written stem allows the candidate to formulate a tentative answer even before reading the
potential responses. This is because only a single concept is being tested.
Avoid adding unnecessary information to the stem, as it can unintentionally confuse candidates
and increase reading time.
Poor Item:
Heart attacks are the number one cause of death in the United States, and coronary artery
disease is preventable to some degree. Which of the following conditions is a risk factor for
premature artery disease?
A. Anemia
B. Cancer
C. Gall bladder disease
D. *High cholesterol
Better Item:
Which risk factor is associated with premature coronary artery disease?
A. Increased arterial elasticity
B. Decreased arterial pressure
C. Lower total cholesterol
D. *Elevated low-density lipoproteins
Guideline 8: Avoid using a negatively-worded stem.
Positively-worded stems are preferred to prevent confusion and misreading of the item.
Negatively-worded stems may be used in the event that the item contains important content, and
a positively-worded stem with three plausible distractors cannot be identified. However,
negatively-worded stems should very much be the exception.
In the event a negatively-worded stem must be used, the negative word in the stem, such as NOT
or EXCEPT, should be capitalized.
*indicates the correct response
13 ARDMS Item Writing Handbook
Poor Item:
Which Doppler finding is not associated with severe aortic insufficiency (AI)?
A. Shortened AI pressure half-time (PHT)
B. Increased maximum aortic velocity
C. Diastolic flow reversal ascending aorta
D. *Decreased left ventricular outflow tract (LVOT)/aortic velocity ratio
Better Item:
Which echocardiographic finding is decreased in the presence of severe aortic insufficiency?
A. Left ventricular outflow tract (LVOT)/aortic velocity ratio
B. Proximal isovelocity surface area (PISA) radius
C. *Pressure half-time (PHT)
D. Jet area
Guideline 9: Reference any images included in the stem within the text of the stem.
Briefly refer to the existence of an image somewhere in the stem when creating items that include
an image in the stem. Doing so will indicate to the candidate to expect to see an image.
Sample reference language:
Which condition does the sonographic finding in this image most likely represent?
Which finding is most consistent with the palpable mass identified in this image?
Guidelines for submitting images and digital video clips and using them within items are found in
Appendix C.
Guideline 10: Place the majority of the wording in the stem.
The central idea or theme of the item should be placed in the stem and not in the answer options.
Placing as much information as possible in the stem avoids unnecessary repetition of material in
the responses.
*indicates the correct response
14 ARDMS Item Writing Handbook
Poor Item:
What happens when a patient has severe proximal arterial occlusive disease?
A. The Doppler waveforms will not be distinguished from venous waveforms.
B. The Doppler waveforms will demonstrate slight flow reversal in the diastolic component.
C. *The Doppler waveforms will appear markedly dampened, making interpretation difficult
for distal segments.
D. The Doppler waveforms will eventually return to a relatively normal waveform pattern once
the disease process has been stabilized.
Better Item:
Which Doppler waveforms are consistent with severe proximal arterial occlusive disease?
A. Triphasic
B. Diastolic flow reversal
C. *Monophasic
D. High resistance
Guidelines for writing the answer options
Guideline 11: Use plausible distractors.
Distractors should be credible and representative of misconceptions found in a clinical setting.
One useful strategy in generating distractors is to brainstorm all of the potential misconceptions
an entry-level professional may have. Also think of common errors made in professional practice.
Poor Item: Which condition is characterized by shortness of breath when the patient is in the recumbent position?
A. Asthma B. Fainting C. *Orthopnea D. Snoring
Better Item: Which condition is characterized by shortness of breath when the patient is in the recumbent position?
A. Apnea
B. Dyspnea
C. Hyperpnea
D. *Orthopnea
*indicates the correct response
*indicates the correct response
15 ARDMS Item Writing Handbook
Guideline 12: Avoid use of the phrases “all of the above” or “none of the above” as
answer options.
“All of the above” and “None of the above” are convenient for item writers when you can’t think of
another plausible distractor. But they can be obvious giveaways for candidates, especially when
the phrases are used only occasionally.
“All of the above” facilitates guessing if a candidate thinks two answers are correct. “None of the
above” may mean that a candidate knew none of the answer options were correct, but it doesn’t
mean that he or she actually knew the correct answer.
Guideline 13: Keep all answer options homogeneous in format and content.
Answer options should be similar with regard to their length and the grammar used within them.
Poor Item:
What is a function of blood platelets?
A. Fighting disease
B. Transmitting oxygen to the body
C. *To aid in blood clotting
D. Carrying food to cells
Better Item:
Which component of blood aids in clotting?
A. Serum
B. *Platelets
C. Monocytes
D. Plasma
Poor Item:
Which condition is associated with anencephaly?
A. *Polyhydramnios
B. Abruptio placenta
C. Decreased maternal serum alpha-fetoprotein (MSAFP)
D. Two-vessel umbilical cord
Better Item:
Which condition is associated with anencephaly?
A. *Polyhydramnios
B. Abruptio placentae
C. Chorioangioma
D. Two-vessel cord
Grammar is not parallel. The three
distractors begin with an -ing verb
while the key does not.
Option length is not equal.
*indicates the correct response
16 ARDMS Item Writing Handbook
Items should also be homogenous in terms of content. Make sure that the item is testing a
single topic or concept and can be assigned to a single task on the content outline. Item stems
that ask candidates to select statements that are true or correct frequently do not contain
answer options which are homogeneous because these items do not contain a central idea in the
stem.
Poor Item:
Which statement is most accurate concerning a urinoma?
A. Irregular in shape
B. Orients superior and medial to the adjacent kidney
C. Caused by blunt trauma
D. *Located between Gerota fascia and the renal capsule
Better Item:
How does a urinoma typically present on ultrasound?
A. *A thin-walled anechoic collection
B. A mass filling the intrarenal space
C. Fluid collection superiorly and medially to the kidney
D. Encapsulation external to the kidney structures
Guideline 14: Avoid using specific determiners in the answer options.
Determiners such as “all”, “never”, “always”, or “often” encourage candidates to begin to
eliminate answer options without fully considering them. This improves the chances of guessing
the correct response.
There is not one single
theme to the item. Answer
options deal with shape,
location and etiology.
*indicates the correct response
1. GRAMMATICAL INCONSISTENCIES IN THE ANWER OPTIONS (Guideline 13).
2. SPECIFIC DETERMINERS INCLUDING ALWAYS AND NEVER IN THE ANSWER OPTIONS
(Guideline 14).
3. PAIRS OR TRIPLETS OF ANSWER OPTIONS (Guideline 15).
4. ANSWER OPTIONS THAT CONTAIN WORDING IDENTICAL OR SIMILAR TO WORDS IN
THE STEM.
5. CONSPICUOUS CORRECT OPTION.
6. BLATANTLY RIDICULOUS ANSWER OPTIONS.
AVOID GIVING THESE CLUES TO THE RIGHT ANSWER
17 ARDMS Item Writing Handbook
Poor Item:
Which description is most accurate regarding pancreatic pseudocysts?
A. Covered by a smooth internal wall
B. Are usually unilocular
C. *May mimic a solid mass
D. Have regular contours
Better Item:
Which description is most accurate regarding pancreatic pseudocysts?
A. Covered by a smooth internal wall
B. Contain anechoic contents
C. *May mimic a solid mass
D. Have regular contours
Guideline 15: Avoid a single pair or triplets in answer options.
Groupings of two or three of the answer options which are similar in wording or structure can
allow candidates to eliminate answer options and aid in guessing.
Poor Item:
What is the correct caliper placement when measuring an abdominal aortic aneurysm?
A. *Outer wall to outer wall
B. Patent lumen
C. Inner wall to outer wall
D. Inner wall to inner wall
Better Item:
What is the correct caliper placement when measuring an abdominal aortic aneurysm?
A. *Outer wall to outer wall
B. Inner wall to inner wall
C. Outer anterior lumen to inner posterior lumen
D. Inner anterior lumen to outer posterior lumen
Submitting Items Now that you understand the guidelines for writing items, you’re ready to start submitting items.
Writers are usually asked to submit 10 items per quarter.
All new writers will be assigned an item writing mentor. The writer will submit all items to the
mentor for approval. Mentors will review items and provide feedback.
*indicates the correct response
This item contains two
pairs which is acceptable.
*indicates the correct response
18 ARDMS Item Writing Handbook
Instructions on how to use ARDMS’s item writing portal to submit items are found in Appendix F.
19 ARDMS Item Writing Handbook
References Haladyna, T.M. (2004). Developing and validating multiple-choice test items (3rd ed.). New York:
Lawrence Erlbaum Associates, Inc.
Haladyna, T.M., Downing, S.M., & Rodriguez, M.C. (2002). A review of multiple-choice item-writing
guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-334.
Parshall, C. G. & Brunner, B. (2017). Content development and review. In Davis-Becker, S. &
Buckendahl, C.G. (Eds.), Testing in the Professions: Credentialing Policies and Practice (pp. 85-
104). New York: Routledge.
20 ARDMS Item Writing Handbook
Appendix A – Glossary of Terms Assessment
Committee A volunteer group responsible for developing an examination
Bias
A situation that occurs when an aspect of an examination means that
certain candidate groups are systematically advantaged or
disadvantaged
Candidate
An individual who is under consideration to receive a credential.
Passing an examination is typically a component of the process to
receive the credential.
Cognitive Demand An estimate of the likely mental behavior required of a typical test taker
in responding correctly to a test item (Haladyna & Rodriguez, 2013)
Content Outline
A document developed through a practice analysis that summarizes the
tasks that comprise competent practice and the related knowledge,
skills, and abilities that candidates are expected to master
Distractor An incorrect (but plausible) answer option in multiple-choice test items
Exam Clinical
Specialist
An Inteleos staff member who has one or more ARDMS credentials in
sonography and is responsible for providing inhouse subject matter
expertise in the development of ARDMS examinations
Exam Program
Manager
An Inteleos staff member who is responsible for overseeing the test
development process for one or more ARDMS examinations
Gap Analysis
An activity undertaken to provide a quantitative review of an
examination item bank to determine the content breakdown of items
within the bank in order to identify content areas that require more
items to be written. A gap analysis is performed to direct item writing
efforts.
Hotspot A type of image-containing item on a computer-based test. Test takers
respond to the item by clicking directly on the image.
Item
The basic unit of measurement on an examination, commonly referred
to as a question. They are referred to as items rather than questions
because they may be phrased as questions (in the case of multiple-
choice items) or tasks to perform (in the case of hotspots or other item
types).
Item Bank
A collection of items written for an examination. The collection is usually
maintained through the use of an electronic content management
system that stores not only the item itself, but also information such as
the area of the content outline the item is classified to, revision and
usage history, and statistical data on item performance.
Item Difficulty
The proportion of test takers who answered an item correctly. It is
usually measured by a statistic called the p value, which is determined
by dividing the number of test takers responding correctly by the total
number of test takers administered the item.
21 ARDMS Item Writing Handbook
Item Discrimination
A measure of how well an item is able to distinguish between higher
and lower ability test takers. It indicates the extent to which success on
a particular test item corresponds to success on the overall test. It is
frequently measured by a statistic called the point-biserial correlation
(ptbis).
Item Types The variety of formats in which test items may be written
Item Writer An expert in a particular field who composes an item for an ARDMS
examination
Key The correct response to an item
Practice Analysis
A research process for determining critical tasks performed within a job
role or practice. Also called Job Task Analysis (JTA), a practice analysis
is performed to determine the content outline for credentialing
examinations.
Pretesting The process of field testing a test item before it is used as an item that
counts toward test takers’ scores on an examination
Response Options All of the answer choices within a test item. These choices include the
distractors and the key.
Statement of Service
(SOS)
An agreement between Inteleos and an ARDMS volunteer that outlines
the terms and conditions by which voluntary work is performed on
behalf of Inteleos
Stem
The part of a test item that describes the task the test taker is to
perform to respond to the item. A stem can take a variety of formats,
depending on the item type, including a question or directions.
Subject Matter
Expert (SME)
An individual with experience and expertise in a particular field.
Individuals who hold ARDMS credentials and who provide input into the
test development process for ARDMS examinations are referred to
SMEs.
Test-wise
Ability of a test taker who is skilled in detecting inadvertent clues to the
right answer to an item, even when the test taker would otherwise not
know the answer
22 ARDMS Item Writing Handbook
Appendix B – Item Writing Checklist Does the submitted item:
☐ 1. Contain your original work?
☐ 2. Include a reference?
☐ 3. Use correct grammar, punctuation and spelling?
☐ 4. Use appropriate language and vocabulary?
☐ 5. Assess important material?
☐ 6. Meet fairness guidelines?
☐ 7. Have a stem that is in the question format?
☐ 8. Deal with a single concept, fact or problem?
☐ 9. Use positive wording?
☐ 10. Contain a reference to the image if the stem contains an image?
☐ 11. Have a majority of the wording in the stem?
☐ 12. Contain plausible distractors?
☐ 13. Not include “all of the above” or “none of the above” in the answer options?
☐ 14. Use answer options that are homogeneous in terms of format and content?
☐ 15. Not use specific determiners in the answer options?
☐ 16. Not contain answer options in a single pair or a triplet?
Refer to the body of this Handbook to view a detailed explanation of each guideline.
23 ARDMS Item Writing Handbook
Appendix C – Guidelines for Submitting
Images and Digital Video Clips All submitted material (items, images, digital video clips, etc.) become the property of ARDMS.
Please do not submit anything you have published, intend to publish or that is or will be used in
any other venue.
1. Remove all identifying information (patient name, hospital name, etc.) to comply with
relevant laws and regulations, including HIPAA. Please obtain appropriate permissions and
authorizations.
2. Please include only clips and frames that contain information relevant to the test item.
3. All images must be digital. No hardcopy images will be accepted, nor will images obtained
using cellphones.
a. JPG (compressed) file format is acceptable – use highest PPI (resolution) available.
Other acceptable file formats for images include PNG, BMP, and DICOM.
b. The image should be at least 1024 x 768 pixels (width x height).
c. Acceptable video formats include AVI, MP4, and WMV.
4. If the image is color, use RGB (Red, Green, Blue). If the image is black and white, use
grayscale.
5. To name the file, use the item ID (provided by the item authoring software) followed by the
name of the diagnosis. An exhibit classification is required. You must save the item to
generate the ID, for example, “52341 ventricular septal defect.”
24 ARDMS Item Writing Handbook
Appendix D – Taxonomy of Media-Enhanced
Items
Classification Name
A Single-select multiple choice with single still image
B Single-select multiple choice with single video
C Single-select multiple choice with side-by-side images
D Single-select multiple choice with multiple vertically-presented
media
E Single-select multiple choice with still images as answer
options
F Single-select multiple choice with videos as answer options
G Non-discrete single-select hotspot
25 ARDMS Item Writing Handbook
Item Type A: Single-select multiple choice (SSMC) with single still image
Description:
This is a four-option multiple choice item with a still image in the stem. Image may be
sonographic or another type of image, such as a chart or graph. All answer options are text-based.
Item template:
Sample stem: Which structure is demonstrated in this image?
A. Answer option A
B. Answer option B
C. Answer option C
D. Answer option D
Potential tasks to be assessed by the item
1. Identify
2. Recognize
3. Assess/evaluate
Technical specifications
1. The maximum recommended size of the image is 440x330 pixels to prevent candidates from
having to scroll to navigate through the item.
2. Arrow(s) may be used to mark a particular structure or location on the image. Arrows should be
white. Note: Arrows to illustrate calipers for fetal measurements on the Fetal Echocardiography
examination may be yellow.
3. Labeling on the image should not cue the correct response.
Item writing guidelines specific to the item type
1. The image should be referred to in the item stem.
2. The image must be the focal point of the item and necessary for responding to the item.
3. Any arrow(s) should be referred to in the stem. For example, “Which structure is indicated by
the arrow on this image?”
Static Image
26 ARDMS Item Writing Handbook
Item Type B: Single-select multiple choice (SSMC) with single video
Description:
This is a four-option multiple choice item with a video in the stem. All answer options are
text-based.
Item template:
Sample stem: Which abnormality is indicated by this video?
A. Answer option A
B. Answer option B
C. Answer option C
D. Answer option D
Potential tasks to be assessed by the item
1. Identify
2. Recognize
3. Assess/evaluate
Technical specifications
1. The maximum recommended size of the video is 440X330 pixels to prevent candidates
from having to scroll to navigate through the item.
2. The maximum recommended length of the video is 3 seconds.
3. Labeling on the video should not cue the correct response.
Item writing guidelines specific to the item type
1. The video should be referred to in the item stem.
2. The video must be the focal point of the item and necessary for responding to the item.
3. Any arrow(s) should be referred to in the stem. For example, “Which structure is
indicated by the arrow on this video?”
Video
27 ARDMS Item Writing Handbook
Item Type C: Single-select multiple choice (SSMC) with side-by-side images
Description:
This is a four-option multiple choice item with a set of two side-by-side images in the stem.
All answer options are text-based.
Item template:
Sample stem: Which finding is indicated by the arrows shown on the images?
A. Answer option A
B. Answer option B
C. Answer option C
D. Answer option D
Potential tasks to be assessed by the item
1. Identify
2. Recognize
3. Compare
4. Assess/evaluate
5. Integrate/synthesize
Technical specifications
1. The maximum size of each image is 440X330 pixels.
2. The images should be separated by a single space.
3. Arrow(s) may be used to mark a particular structure or location on the image(s). Arrows
should be white in color. Note: Arrows to illustrate calipers for fetal measurement on the
Fetal Echocardiography examination may be yellow.
4. Labeling on the image(s) or video(s) should not cue the correct response.
Item writing guidelines specific to the item type
1. The images should be referred to in the item stem.
2. The images should be the focal point of the item and necessary for responding to the
item.
Image Image
28 ARDMS Item Writing Handbook
3. Each image should contribute to the information needed by the candidate to respond to
the item. Their use should not be redundant.
29 ARDMS Item Writing Handbook
Item Type D: Single-select multiple choice (SSMC) with multiple vertically-presented media
Description:
This is a four-option multiple choice item. The stem includes vertically-presented media that
might include still images, videos, charts, or other graphics. Note that Item Type C is
preferred for stems with two still images to prevent scrolling.
Item template:
Sample stem: Based upon the image and video, which diagnosis is the most likely?
A. Answer option A
B. Answer option B
C. Answer option C
D. Answer option D
Potential tasks to be assessed by items
1. Assess/evaluate
2. Integrate/synthesize
Technical specifications
1. The maximum recommended size of each media is 440X330 pixels.
2. If a video is used, the maximum recommended length of the video is 3 seconds.
3. An item should include no more than one video.
4. Arrow(s) may be used to mark a particular structure or location on the image(s) or
video(s). Arrow(s) should be white in color. Note: Arrows to illustrate calipers for fetal
measurements on the Fetal Echocardiography examination may be yellow.
Image/Video/Graphic
Image/Video/Graphic
30 ARDMS Item Writing Handbook
5. Labeling on the image(s) or video(s) should not cue the correct response.
6. Scrolling should be kept to a minimum.
Item writing guidelines specific to the item type
1. The image(s) and/or video should be referred to in the item stem.
2. The image(s) and/or video must be the focal point of the item and necessary for
responding to the item.
3. Each image, graphic, and/or video should contribute to the information needed by the
candidate to respond to the item. Their use should not be redundant.
31 ARDMS Item Writing Handbook
Item Type E: Single-select multiple choice (SSMC) with still images as answer options
Description:
This is a four-option multiple choice item. Each answer option is a still image.
Item template:
Sample stem: Which image demonstrates a volvulus?
A.
B.
C.
D.
Potential tasks to be assessed by the item
1. Identify
2. Recognize
3. Assess/evaluate
Technical specifications
1. The maximum recommended size of each image is 440X330 pixels.
2. Arrow(s) may be used to mark a particular structure or location on the image(s) or
video(s). Arrows should be white in color. Note: Arrows to illustrate calipers for fetal
measurements on the Fetal Echocardiography examination may be yellow.
3. Labeling on the images should not cue the correct response.
4. Scrolling should be kept to a minimum.
Item writing guidelines specific to the item type
1. Use image options in place of text-based answer options to provide a more authentic
assessment.
2. Use of this item type may be appropriate when a text-based answer option would include
a ‘normal’ or ‘no abnormality’ as the key or a plausible distractor.
Image
Image
Image
Image
32 ARDMS Item Writing Handbook
Item Type F: Single-select multiple choice (SSMC) with videos as answer options
Description:
This is a four-option multiple choice item. Each answer option is a video.
Item template:
Sample stem: Which video clip demonstrates the inferior vena cava?
A.
B.
C.
D.
Potential tasks to be assessed by the item
1. Identify
2. Recognize
3. Assess/evaluate
Technical specifications
1. The maximum recommended length of the video is 3 seconds.
2. Arrow(s) may be used to mark a particular structure or location on the image(s) or
video(s). Arrows should be white in color. Note: Arrows to illustrate calipers for fetal
measurements on the Fetal Echocardiography examination may be yellow.
3. Labeling on the images should not cue the correct response.
4. Scrolling should be kept to a minimum.
Item writing guidelines specific to the item type
1. Use video options in place of text-based answer options to provide a more authentic
assessment.
2. Use of this item type may be appropriate when a text-based answer option would include
a ‘normal’ or ‘no abnormality’ as the key or a plausible distractor.
Video
Video
Video
Video
33 ARDMS Item Writing Handbook
Item Type G: Non-discrete single-select hotspot
Description:
This is an item which includes a single image in the stem. Candidates respond to the item by
clicking directly on the image. The item is referred to as a non-discrete hotspot because the
regions of the image available for selection are not visualized.
The stem includes standardized instructional language providing directions on how candidates
should respond to the item.
Item template:
Sample stem:
Identify the gallbladder.
Using your mouse, place the cursor on the appropriate region of the image and then left-click the
mouse button to indicate your selection.
Potential tasks to be assessed by the item
1. Identify
2. Recognize
3. Interpret
Technical specifications
1. The maximum recommended size of the image is 440X330 pixels.
2. Labelling on the images should not cue the correct response.
3. Candidates should not be required to scroll.
Item writing guidelines specific to the item type
1. Use hotspot images in place of text-based answer options to provide a more authentic
assessment and direct measurement of the content.
Still Image
34 ARDMS Item Writing Handbook
Appendix E – Sample Stem Starters Use the stem starters when you’re at a loss for words or don’t know how to verbalize the point
you’d like to make. Note that the fill-in-the-blanks indicate places in which words should be
inserted to complete the item. Fill-in-the-blanks should not be used within a completed item.
Which sonographic feature best
characterizes ______?
Which finding is (characteristic of, most
consistent, most often presents) with
______?
What is most likely represented by the finding
demonstrated in this image?
Which method is most (consistent, accurate,
appropriate) with ______?
Which (finding, abnormality) is most
consistent with _____?
Which diagnosis is most consistent with the
findings demonstrated on this image of a
______ in a patient with a history of ______?
Which sonographic finding is (most
commonly/most strongly) associated with
______?
Which description is associated with ______?
Which scanning technique would
(increase/reduce) the ______?
Which vessel would be (enlarged, occluded,
obliterated) in the presence of ______?
Which post-transplant complication
characterized by ______?
Which sonographic (view, plane, window)
best demonstrates ______?
Which finding requires further follow-up?
Which patient position is best suited for
evaluation of ______?
Which sonographic view is best for assessing
______?
Which technique is optimal for evaluation of
______?
Which adjustment is most likely to improve
visualization of ______?
Which sonographic finding is most likely to be
demonstrated in a ______ patient referred
for ______, with a history of ______?
Which associated abnormality presents with
______?
Which (region/area) would require focused
evaluation of ______?
What is the most likely diagnosis based on
this image?
Which measurement of the ______ should
prompt further evaluation of ______?
What is the characteristic feature of a
(normal/abnormal)______?
What is the most accurate sonographic
measurement for ______?
What is the optimum time frame to
obtain/perform ______?
What is a potential (advantage/
disadvantage) of ______?
Which condition may be (misdiagnosed/
mistaken for ______)?
Which symptom would be considered an
indication for ______?
Which symptoms are commonly associated
with _____?
What is the characteristic (shape,
appearance) of a ________?
Which region within the Doppler waveform
represents _________?
What is a major risk factor for ________in
patients with _________?
Which common sonographic finding is
associated with _________?
Visualization of which anatomy confirms
______?
35 ARDMS Item Writing Handbook
Appendix F – Instructions for Submitting
Items in the Remote Item Writing (RIW) Portal PRO! Is the Remoted Item Writing Portal that is used for volunteers to submit exam items. It is a
website that can be accessed from any internet-connected computer.
• Web address (URL): https://ardms-prod-riw.epic-exam.com/ (Bookmark this site for easy
access.)
• Username is your ARDMS #
• Your password will be assigned and emailed to you after you complete item writing training
and sign the Statement of Service agreement.
Item Writing
Username = ARDMS or
APCA ID#
36 ARDMS Item Writing Handbook
Navigating the Item Writing Portal
The front page features an overview of all projects assigned.
Navigating the Item Writing Portal
Assigned projects are shown. If you are writing for more than one project or exam, they will be shown underneath the Project Name header.
37 ARDMS Item Writing Handbook
Creating an Item
To access the item writing screen, select “Create Item”.
Creating an Item
Select the “Content” icon to select which area of the content outline this item will address.
38 ARDMS Item Writing Handbook
Creating an ItemIn the “Select Content” screen, use the scroll bar on the right to view all available content areas from which to choose the topic for this new item. Once a selection has been made, click the “Select” button. NOTE: Depending upon the project needs, all content areas may not display. Additionally, as content areas become full, they will disappear from this list.
Creating an ItemEnter the stem remembering to phrase it in the form of a question with a question mark (?) at the end.
39 ARDMS Item Writing Handbook
Inserting an Image from Item Writer’s LibraryTo upload an image from the writer’s library, select the image button.
Inserting an Image from Item Writer’s Library
Select “Browse” and find the image/video.
40 ARDMS Item Writing Handbook
Inserting an Image from Item Writer’s Library
The computer’s picture folder will open and a selection can be made.
Inserting an Image from Item Writer’s Library
Once the image is selected, the “General” tab will display along with the image. Toggle the description categories and “tag” the image by selecting no more than four tags.
41 ARDMS Item Writing Handbook
Inserting an Image from Item Writer’s Library
Note that exhibit tagging is required.
Inserting an Image from Item Writer’s Library
Select the “Details” tab and complete the information. If your item text refers to an arrow on the image, please let us know in the comments where the arrow needs to be placed by our in-house image editor. You may place other image-editing notes here.
42 ARDMS Item Writing Handbook
Inserting an Image from Item Writer’s Library
Select the “Reference” tab and choose the appropriate reference. See below if the reference is not listed. Use the scroll bar to view existing resources.
Inserting an Image from Item Writer’s Library
After selecting the reference, add the beginning page number for your exact topic. After you have completed this, click “Save Exhibit.” It will return to the create item screen and your image will be shown.
43 ARDMS Item Writing Handbook
Inserting an Image from Item Writer’s Library
If the reference is not listed, select “Reference Note.” Add the reference and include the title, editor/author, edition, publisher, publication date, and page number(s). Then click “Save Exhibit.”
Inserting an Image from Item Writer’s Library
A note about images…..
ALL PATIENT AND FACILTY INFORMATION SHOULD BE REMOVED BEFORE
UPLOADING AN IMAGE.
Please use the best image quality that you can. You may include more than one
image per question.
We ask that you use non-copywritten images that you have obtained from scanning
patients. (Not from a previously published work.) If you do not scan patients, you
may use images from your colleagues, provided you have their permission.
Individual file sizes should be kept to no more than 10 MB per image and 20 MB
per video.
44 ARDMS Item Writing Handbook
Inserting an Image from Inteleos Image Bank
To search and select an existing image, select the folder option.
Inserting an Image from Inteleos Image Bank
This window will open for you to choose from the “Exhibit Classifications” to select the image required. Use the scroll bar to view other classifications.
45 ARDMS Item Writing Handbook
Inserting an Image from Inteleos Image Bank
Select the appropriate image. Edit the criteria if needed, then click “Select Exhibit.”
Adding the Response Options to an Item
Next, add the responseoptions, keeping in mindto include the correct answer, or key. Then, indicate the key by clicking the small, round radio button to the left of the key.
NOTE: You may also add images to the answer choices.
46 ARDMS Item Writing Handbook
Referencing the Item
The final part ofsubmitting an item isadding the reference. Having valid and currentreferences for our itemsis one way we maintainthe credibility of ourexams.
Reference the over-arching concept of the item. For example, if your item addresses intracranial hemorrhages, include a reference that addresses them.
Referencing the Item
Use this scroll bar to view the existing resources in the library. Once youhave found your desiredreference, click on it.
You will then need to add the beginning page number for your exact topic for your item here.
Add the ending page number for your exact topic for your item here.
When completed, “Add Reference.”
47 ARDMS Item Writing Handbook
Referencing the ItemIf you do not see yourreference and edition inthe library, please addthe reference in theReference Notes section.
Add your referencewhere the arrow indicates. Include title ofwork, editor/author,edition, publisher,publication date, andpage(s). Then click “Add Reference Note.”
Submit your reference tothe Project Manager and it will be added to the reference library.
Save/Submit/Send to Mentor
Click Save to save your item in system for further editing (Status will be“Incomplete”)
If you have been paired with a Mentor, click Send to Mentor. The item will belocked for editing until the Mentor adds his/her feedback (within 5-7 days).
Click Submit – ONLY when all of the required fields have been completed. Once yousubmit an item, you cannot go back and edit it. (Status will be “Pending”)
The Save/Submit/Send to Mentor buttons are at the bottom ofthe page.
48 ARDMS Item Writing Handbook
FAQs
What about the other fields in PRO!—like Rationale andKeywords?We do not require you to use some of the additional features ofPRO!. You may skip them.
Where can I get some additional guidance on writing items?Ask your Mentor, email your project manager or ItemDevelopment Coordinator for additional help and guidance on writing items. The project manager information is shown on the home screen and an email can be initiated from there.
You can also view three short videos on our YouTube channel.https://www.youtube.com/playlist?list=PLydRf_PGVd9K66qkUxAeMZS1Jf7pLf1d0
I am credentialed in multiple specialties. Can I be a part ofmultiple writing projects?Yes! View all your projects on the Projects Overview page, the firstscreen you see when you log in.
49 ARDMS Item Writing Handbook
Do your exam items face the FACTs?
Fair
Test items should not advantage or disadvantage any candidate group. Make
sure that:
• The language used is able to be understood by all candidates.
• The content is not derogatory or prejudicial to any candidate groups.
Appropriate
Test items should be appropriate for the test purpose. Make sure that:
• The content of the item is within the scope of practice.
• The item can be classified to a specific task on the content outline.
Current
Test items should be reflective of current practice. Make sure that:
• The content of the item reflects current practice guidelines.
• The item is referenced to an up-to-date well-recognized source.
Technically accurate
Test items should be correct and valid. Make sure that:
• The content of the item is accurate for all appropriate practice situations.
• The key is correctly identified and all other answer options are plausible,
but incorrect.
©2019 American Registry for Diagnostic Medical Sonography, Inc.
All rights reserved.
2019 Version 6.1
WHO TO CONTACT
Questions concerning this Handbook, item writing deadlines, item writing and
mentoring assignments, the PRO! Remote Item Writing Portal (RIW), or the use of
references and/or media should be directed to:
Questions concerning potential conflicts of interest, or about other volunteer
opportunities should be directed to: