Overview

• Benefits of Serving as a Reviewer• Summary of Recent Changes• Review Time-Line• Review Criteria• Review Scoring• “Impact” vs. “Significance”• Ethics of Reviewing• Current Controversies• Reviewer Qualifications• How to Become a Reviewer

Benefits of Serving as a Reviewer

• Get a front row seat to the future• Become a more successful scientist• Learn more• Meet new colleagues• Become a better mentor• Give back• Shape the future

www.csr.nih.gov

Summary of Recent Changes

• Emphasis on “impact”• Bullet point format of reviews• “Tutorial” comments not encouraged• Discussion ordered by preliminary score• New investigator applications separated

Review Time Line

• Pre-assignment phase – 2 weeks• Review and submission phase – 4 to 6 weeks• Read phase – 5 days• Meeting phase – 2 days• Edit phase – 3 days

Scored Review Criteria

• Impact• Significance• Investigators• Innovation• Approach• Environment

Impact

• Summarize significant overall strengths and weaknesses

• Assess the likelihood that the project will exert a sustained powerful influence on the field

Significance

• Does the project address an important problem or a critical barrier to progress in the field?

• If the aims of the project are achieved, how will scientific knowledge, technical capability or clinical practice be improved?

• How will successful completion of the aim change the concepts, methods, technologies, treatments, services or preventive interventions that drive this field?

Investigator

• Are the researchers well suited to the project?

• If Early Stage or New Investigators (only R01), do they have appropriate experience and training?

• If established, have they demonstrated an ongoing record of accomplishments that have advanced their fields?

• If the project is collaborative or multi-PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?

Innovation

• Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions?

• Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field or novel in a broad sense?

• Is it a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation or interventions proposed?

Approach

• Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project?

• Are potential problems, alternative strategies, and benchmarks for success presented?

• If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?

Frequent Mistakes

• Description substituted for critique• Contradiction between strengths and weaknesses• Extreme brevity, especially if application is not discussed• Focus on the disease as opposed to the application• Mismatching between scores and narrative• Focus on minutiae, as in a manuscript review• Verbosity

Additional Review Criteria*

• Protection for human subjects• Inclusion of woman, minorities and children• Vertebrate animals• Biohazards• Resubmission• Renewal• Revision (competing supplement)• Multiple PI plan

*can affect overall score

Protection for Human Subjects*

• Evaluate risks as acceptable or unacceptable

• Evaluate protections as adequate or inadequate

• Items for evaluation– Risk to subjects– Adequacy of protection against risks– Potential benefits to subjects and others– Importance of the knowledge to be gained– Data and safety monitoring for clinical trials– Data and Safety Monitoring Plan– Justification for exempt status

*can affect overall score via “approach”

Inclusion of Women, Minorities and Children*

*can affect overall score via “approach”

A = Acceptable; U = Unacceptable

Vertebrate Animals*

• Evaluate as acceptable or unacceptable– Proposed use of animals– Species, strains, ages, sex and numbers– Justification for use of animals and for appropriateness

of the species and numbers proposed– Adequacy of veterinary care– Procedures to limit discomfort, distress, pain and injury– Methods of euthanasia

• Worksheet available from CSR website

*can affect overall score via approach

Additional Review Considerations*

• Application from foreign organizations• Select agents• Resource sharing plans• Data sharing• Model organisms• Genome-wide association studies• Budget and period of support

*do NOT affect overall score

“Impact” vs. “Significance”

• Three key phrases for “Impact”– Likelihood—derived from assessment of

investigator, approach and environment– Sustained powerful influence—derived from

significance and innovation– Research field—should be identified

• “Impact” is not – a 6th review criterion – the mean of scored criteria

“Cheever Types”

• Type 1– project is important – applicant is able to

do the work

“Cheever Types”

• Type 1– project is important – applicant is able to

do the work

• Type 2– project is important – applicant not able

to do the work

“Cheever Types”

• Type 1– project is important – applicant is able to

do the work

• Type 2– project is important – applicant not able

to do the work

• Type 3– project not important – applicant is able to

do the work

“Cheever Types”

• Type 1– project is important – applicant is able to

do the work

• Type 2– project is important – applicant not able

to do the work

• Type 3– project not important – applicant is able to

do the work

• Type 4– project not important – applicant not able to

do the work

Review Scoring

Categories of Weakness

• Minor– An easily addressable weakness that does not

substantially lessen impact

• Moderate– A weakness that lessens impact

• Major– A weakness that severely limits impact

Budget and Period of Support

• Is the budget realistic?

• Reductions can be recommended– Insufficient budget justification– Insufficient information about work in later years– Project can be completed in fewer years or with

smaller budget

• Panel recommendation summarized by SRO

• Budget does not affect overall priority score

• Overlap concerns noted in written comments (not discussed)

Additional Comments to Applicant

• Elaboration of ideas that are too complex for bullet points

• Further explain deficiencies in the application• Avoid “tutorial” suggestions for improvement

Before the Meeting

• Pre-assignment phase– Declare conflicts of interest

• Review and submission phase– Read the instructions!– Make travel arrangements– Review, complete and sign pre-review certification– Read, evaluate and write critique for assigned applications– Upload preliminary scores and critiques before deadline

• Read phase– Read other critiques for assigned applications– Prepare for discussion during meeting

Overview of the Meeting

• Welcome and call to order by chair

• Panel members introduce themselves

• Introductory remarks and instruction by SRO

• Order of review– New investigator R01 applications– Other R01 applications– R21 applications

• Order determined by average of preliminary overall scores from assigned reviewers

• ~ 50% of applications are discussed

How Reviews are Discussed

• Conflicted members leave the room

• Applicant and title announced by chair

• Preliminary overall scores from reviewers

• Discussion by primary reviewer

• Additional comments by other reviewers

• Human subjects and animal welfare concerns

• Panel discussion

• Summary by chair

• Revised overall scores announced

• Voting outside range

• Budget recommendations

• Conflicted members called back

How to be Effective During the Meeting

• Submit initial reviews before the deadline• Read comments from other reviewers before the

meeting

• Suggestions from experienced reviewers

Relax Talk about what matters Don’t rehash Focus on the application Don’t mince words Avoid mentoring

Paint the big picture Paint a balanced picture Give key background No unreasonable demands No favoritism by discipline Be flexible and have fun

After the meeting

• Edit phase– Revise criterion scores to reflect discussion– Revise critiques to reflect changes after discussion– Sign the electronic Conflict of Interest form

Ethics of Reviewing

• Conflicts acknowledged appropriately

• No use of non-public information for private purposes

• No consultation with anyone other than SRO

• Applications discussed only at the designated time (no outside discussion)

• No review-related communication with applicant

• Protection of confidentiality– No discussion with other individuals– All review materials password-protected– All review materials destroyed after meeting

• No allegations of research misconduct during discussion or in critique (discuss confidentially with SRO)

Current Controversies

• American Association of Immunologists– “Study sections must engage in an exceedingly

thoughtful process, with much discussion, the sharing of varied opinions, and the vocal expression of differing views. Providing this meaningful feedback is an essential part of the peer review process.”

• Francis Collins, Director NIH (reply)– “I sense that the central issue may be providing

feedback to applicants. Although we have made an effort to separate scientific assessment and advice to applicants because we do not want reviewers to re-write the applications, the review template does specifically contain a box where reviewers are invited to give feedback to applicants.”

Reviewer Qualifications

• Substantial independent research experience

• Prior peer-reviewed grants (R01 or equivalent)

• Understand the review process

• Dedicated to high quality, fair reviews

• Willing to consider service for a 4-year term

• Other considerations– Diversity of geographic representation– Diversity of gender, race and ethnicity

Characteristics of Great Reviewers

• Fairness and objectivity• Ability to articulate views succinctly• Ability to engage in productive exchanges• Ability to discuss non-assigned applications• Ability to work collegially in a group setting• Ability to facilitate or focus discussion• Effort to ensure fairness and consistency of

scoring throughout the meeting

How Much Time Does it Take?

• Reading a 25-page application ─ 2 to 4 hours• Writing a critique ─ 1 to 1.5 hours• Uploading critiques ─ 1 hour• Typical number of assigned applications ─ 8 to 10• Meeting ─ 3 days, including travel• Meetings per year ─ 3

How to Become a Reviewer

• Notify professional society or research dean– Nomination to CSR National Registry of Volunteer

Reviewers

• Express interest to a current reviewer– Scientific Review Officers appreciate suggestions

• Send CV to SRO who oversees a review group that could use your expertise– Descriptions of study section scope available at

http://cms.csr.nih.gov/PeerReviewMeetings/CSRIRGDescription/

Where to Get More Information

Go to www.csr.nih.gov