1
CSR INTEGRATED REVIEW GROUP REVIEWS[peer review of peer review]
Don Schneider
National Institutes of HealthU.S. Department of Health and Human Services
November 3, 2008
CSR is committed to peer review of peer review
• Need to respond to changing science• Trace roots to Panel on Scientific
Boundaries for Review (PSBR)• Examine Integrated Review Group-specific
issues• Discuss general issues
2
The PSBR issued a report January 2000 outlining reorganization philosophy and expectations.
• URL - http://cms.csr.nih.gov/NewsandReports/ReorganizationActivitiesChannel/BackgroundUpdatesandTimeline/FinalPhase1Report.htm
• Be monitored continuously: assess peer review system at regular intervals, at the level of the IRG, and globally by the Advisory Committee
3
Working Groups Assess IRGs[Peer review of peer review]
• Peer reviewers report on study section operations
• Feedback from reviewers condensed by staff into a brief 5-10 page white paper report
• Subset of peer reviewers meet as working group to edit/confirm essence of report
• Report is shared with Advisory Committee and used as basis of taking action for improvements
4
IRG Working Groups, Round 1, provided baseline, 2000-2002
• Round 1 working groups – one-cycle visit of each study section, often by single pre-selected reviewer
• Pre-selection probably resulted in perturbations
• Nonetheless, useful information obtained that informed reorganization
• PSBR Reorganization occurred step-wise one IRG at a time from June 2003 to March 2005 5
IRG Working Groups, Round 2, were initiated in 2005 and are currently in progress
• Eliminate perturbation –peer reviewers of peer review selected post-study section meetings
• Solicit input from multiple reviewers each cycle
• Solicit input from reviewers over multiple, at least two, cycles
• Share white papers with Chairs prior to reporting to Advisory Committee
• Provide substantial basis for future actions 6
Schedule for IRG Reviews
July 2008• Brain Disorders & Clinical Neuroscience (BDCN)• Molecular, Cellular & Developmental Neuroscience (MDCN)
September 2008• Integrative, Functional & Cognitive Neuroscience (IFCN)• Emerging Technologies & Training in Neuroscience (ETTN)• Biology of Development & Aging (BDA)
January 2009• Biobehavioral & Biobehavioral Processes (BBBP)• Risk, Prevention & Health Behaviors (RPHB)
March 2009• Healthcare Delivery & Methodologies (HDM)• Epidemiology & Population Sciences (EPS)
7
Cont., Schedule for IRG Reviews
May 2009• Immunology (IMM)• Infectious Diseases & Microbiology (IDM)
July 2009• Oncology 1 – Basic Translational (OBT)• Oncology 2 – Translational Clinical (OTC)
September 2009• Bioengineering Sciences & Technologies (BST)• Surgical Sciences, Biomedical Imaging & Bioengineering
(SBIB)
November 2009• AIDS & Related Research (AARR)• Endocrinology, Metabolism, Nutrition & Reproductive
Sciences (EMNR)8
Cont., Schedule for IRG Reviews
January 2010• Vascular & Hematology (VH)• Cardiovascular & Respiratory Sciences (CVR)
March 2010• Musculoskeletal, Oral & Skin Sciences (MOSS)• Digestive, Kidney & Urological Systems (DKUS)
May 2010
Genes, Genomes & Genetics (GGG)
Biological Chemistry & Macromolecular Biophysics (BCMB)
July 2010• Interdisciplinary Molecular Sciences & Training (IMST)• Cell Biology (CB)
9
Format for IRG Reviews
Late afternoon• Present Peer Review Issues & Current Initiatives (CSR
Director)• Discuss; discuss further over dinner
Morning• Present Working Group White papers• Describe study sections and specific issues/metrics (Chiefs)• Discuss and improve study section guidelines (Chairs)• Forecast future issues & trends in science
Afternoon• Consider the critical role of the study section chair (CSR
Director, in closed session with Chairs)10
IRG specific issues and actions are notedMolecular, Cellular & Developmental Neuroscience
IRG• One study section split for second time• Initial attempt to charter a neurotechnology panel
led to PRAC requested crosscutting look and eventual formation of two study sections, as well as stimulating formation of crosscutting neuroscience IRG and contributing to momentum for new division
11
Cont. IRG-specific issues & actions
Brain Disorders & Clinical Neuroscience IRG• Split two study sections, each into two• Resolve referral issue so that brain tumor
applications focusing on neurological aspects are assigned to BDCN, with those focusing on cancer being assigned to Oncology IRG
12
Cont. IRG-specific issues & actions
Integrative, Functional & Cognitive Neuroscience IRG• No issues & no actions
Biology of Development & Aging IRG• Clustering of International & Cooperative Projects
in BDA, with great breadth of topics – perhaps pilot editorial board review
Emerging Technologies & Training in Neuroscience IRG
• First applications reviewed winter 2008, no revisions/actions yet
• One new study section growing rapidly, on verge of needing to split
13
General Review Issues & Actions
Reviewers• Seniority• Ad hoc/temporary• Telephone• Mail• Registry/College
Scoring• Streamlining/unscoring• Score compression• Scoring consistency throughout meeting
14
IRG Issues & Enhancing Peer Review
Reviewers• Flexible service• Systematic training• Shorter applications and critiques
Scoring• Smaller scale, 7 or perhaps 9 bins• Scores for all applications• Promote score spreading, ranking or order of
review
Emerging scientific trends• Selecting appropriate reviewers• Updating study section guidelines 15
The End