Developing a Validated Responsible Conduct of Research
Test Bank
Holly Phernetton, MPH, MSJames DuBois, PhD, DSc
Saint Louis University
Planned Collaboration with CITI
With ORI funding, DuBois, Mumford, Phernetton, Antes have developed a RCR test bank: >125 multiple-choice items aimed at assessing
basic knowledge of RCR 50 “pick 2” ethical decision-making items that
test ability to choose optimal responses to ethical problems described in vignettes
Planned Collaboration with CITI
All items will be available for CITI to use at no cost
This presentation describes how we have already validated the test items and how we propose to work with CITI for final construct validation using classical test theory
Background
Office of Research Integrity (ORI) identified 9 core areas that RCR courses should address (2000) However, there was no guidance or consensus
on what content should be taught within these core areas, nor what the overarching objectives of RCR training should be assessed
Further, there is no standard validated test that measures RCR education objectives
Process for Test Bank Development
3 Stages of Test Development1. Delphi Survey to validate objectives and
content areas2. Development of the Items
Multiple choice items Pick-2 items
3. Pilot-testing of items and calculation of psychometric properties
Validation Process
Content Validity Delphi survey: identify areas to assess Item development: evidence-based answers Expert review and revision
Construct Validity Pilot-test items Classical test theory
Content Validity
Level 1: Delphi survey Define objectives and determine important RCR
content to be taught and knowledge assessed by consensus panel
Level 2: Item development According to assessment recommendations from
Delphi panel Evidence-based answers from well-recognized RCR
curricula and text books Level 3: Expert review
Delphi Survey
Funded by ORI 2006-2007: four panels of experts convened
to develop a consensus: identify RCR overarching goals for instruction and assessment
Delphi process: anonymous process Minimizes groupthink and domineering group
members Maximizes benefits of group decision-making
Delphi Survey Formed 4 separate panels: each worked independently
and simultaneously Survey conducted online in 3 rounds Round 1: Open-ended items asking what should be the
objectives and content of RCR training Round 2: Presented lists of edited responses from
round 1—participants asked to rate the importance of teaching each topic
Round 3: Presented lists of edited responses from round 2—participants asked to rate importance of teaching and assessing each area
Delphi Survey
4 Panels 1. Objectives (n=18)
Overarching objectives and assessed objectives 2. Scientific Data (n=13)
Data management and research misconduct sections 3. Scientific Relationships (n=14)
Mentor-trainee responsibilities, collaborative science, conflicts of interest and commitment
4. Publication and Peer Review (n=13) Publication practices/responsible authorship and peer review
Delphi Survey Results
7 Core Areas of RCR instruction examined Panel Consensus: 2/3 of panelists supporting
a rating of “important” or “very important” Consensus on teaching 43 main topics Consensus on assessing learning in 21 of the 43
main topics
Content Validity Level 1: Delphi survey
Define objectives and determine important RCR content to be taught and knowledge assessed by consensus panel
Level 2: Item development According to assessment recommendations from Delphi panel Evidence-based answers from well-recognized RCR curricula and
text books Level 3: Expert review
Item Development Multiple Choice Questions (MCQ)
Developed by Saint Louis University(Dr. James DuBois, Holly Phernetton)
Stem and lead-in question: 1 correct answer and 3 distractors
Pick 2 Items (P2) Developed by University of Oklahoma
(Dr. Michael Mumford, Allison Antes) Stem: vignette presenting ethical problem Answer: pick 2 “optimal responses” from list of 8
options
MCQ Item Development
Follow National Board of Medical Examiner Guidelines
Rationale for NBME guidelines. Good test items: Discriminate between those who know and
those who do not know material Are constructed to reduce correct guessing by
“test savvy” participants Avoid being “tricky”
Rationale Behind the NBME Guidelines
Items contain only critically important content Must have 1 clearly best answer No negative questions such as
“each of the following is correct except” “which of the following is NOT correct”
No absolute terms such as “never” or “always” Avoid “all of the above” and “none of the
above answers” All responses are of roughly equal length and
form
Item Example: MCQA researcher commits research misconduct at a large
university which receives federal funding for research from the Public Health Services. Which of the following is correct regarding institutional obligations in such cases?
A. Academic institutions are legally obligated to report all allegations of misconduct to the Office of Research Integrity prior to investigation.
B. Institutions must have policies in place for the inquiry, investigation, and adjudication of allegations of misconduct.
C. The federal government has to be immediately contacted in every case of alleged misconduct.
D. Researchers found guilty of misconduct can never receive PHS funding again.
Item Example: P2Vignette:Dr. Davis has been culturing a strain of Escherichia coli in an
oxygenated broth. For 12 years he has been gradually reducing glucose concentration in the medium. As a mid-career evolutionary biologist, he is interested in the possible evolution of the strain over thousands of generations. He is targeting an extremely unlikely adaptation-the ability to take up and metabolize citrate, something that E. coli strains are notably unable to do. After about 30,000 generations, the adaptation appears. Indeed, the bacteria, which also shows some morphological changes, can thrive in a totally glucose-free medium, and phenotypes with the trait rapidly dominate the population.
Item Example: P2I
Vignette, cont’dAn investigation revealed a new set of genes in Davis’ E. coli
culture consistent with contamination by the common soil bacterium, Serratia marcascens. Yet, analysis could not identify a single living S. marcescens in the culture. S. marcescens readily metabolizes citrate, and the almost certain conclusion is that his E. coli acquired the adaptive trait through lateral gene transfer. This, itself, is an exciting finding, and Davis writes up and distributes a report for prepublication review by the research team. Two members object to Davis’ procedure section where he says that the E. coli culture was experimentally inoculated with S. marcescens. How should Davis respond? Choose two from the following:
Item Example: P2I A. Remind them that this is his laboratory and he alone makes the final
decisions about the content of publications from his laboratory. B. Agree that the students have a good and valid point, but that the machinery
of science is intolerant of accidental findings. C. Explain to his students that the intention behind a finding is irrelevant to the
fact being presented. D. Revise the paper’s method section to explain how S. marcescens found its
way into the culture. E. Explain that full disclosure would only taint the reputation of the laboratory
by admitting the breach in original procedure. F. Listen to the objections of his students, accept that he is outvoted, and change
the procedure section of the paper. G. Do not offer any explanation other than a reminder that solid publications are
what make a career and that if they are uncomfortable with this, they should consider leaving the lab.
H. Acknowledge to the students that they are probably correct and use the data as a basis for a follow-up study on lateral gene transfer under more tightly controlled conditions.
Content Validity Level 1: Delphi survey
Define objectives and determine important RCR content to be taught and knowledge assessed by consensus panel
Level 2: Item development According to assessment recommendations from Delphi panel Evidence-based answers from well-recognized RCR curricula and
text books Level 3: Expert review
Expert Review
3 Experts to review RCR test items; provide revision, comments MCQ (all three experts to review) P2 (one expert to review each area)
Biological Sciences Social Science Health Science
Construct Validity
Pilot-test Items CITI Program:
Implement test bank Each item will be completed by a minimum of 200
CITI participants A participant will complete only 20% of total battery of
items: 5 MCQ per 7 core areas, 2-3 P2 vignettes with 3 items each (41-44 items total)
Total of 1,000 CITI participants needed
CITI & Construct Validation
CITI program pilot-testing All items will be pilot-tested on the CITI training
program While participants can re-take test until they pass, only
their initial response will be considered in construct validation
Will use classical test theory to validate items: Will identify top 20% and bottom 20% of test performers Any items missed more frequently by the top 20% than the
bottom 20% will be treated as invalid. Such items will be discarded or revised
CITI Logistics CITI will upload test items using our Excel file Program test items to appear in random subsets After participants have completed all test items,
CITI will send de-identified data for statistical analysis
Test bank team will revise items and send CITI a finalized test bank for use by Jan 2010
CITI—Ethical Issues Participants can re-take the test until passing score
achieved Most items tested are covered by CITI educational
content A few test items will be piloted of which are not
directly covered in the CITI training program These will not be scored, but feedback will be
provided This will ensure fairness and will provide
educational value
CITI—Ethical Issues Will obtain IRB permission CITI will provide participants with
introductory statement and request for permission to use test data
Our approach distinguishes between: Consent to take test (not needed, because testing is
a required component of training and items are already content validated)
Consent to use data (will be requested) in any activities that extend beyond normal QI
Introductory Statement for Participants
“Most of the test items you will see address material directly covered in our training program. However, some items also test your ability to generalize to new ethical situations, and still others test your ethical problem-solving skills. In each case, you will be told the correct answer after you complete an item and will have the chance to re-take the test until you achieve a passing score.”
Upon Completion of the Test
“CITI is committed to quality improvement and sometimes analyzes de-identified test and survey data to identify ways to improve test items and course content. We may also describe our quality improvement efforts in publications or at conferences using analyses of de-identified test data. Do we have your permission to use your de-identified test data in this manner?
Yes No