Date post: | 18-Dec-2015 |
Category: |
Documents |
Upload: | miles-thompson |
View: | 218 times |
Download: | 0 times |
Revising and Resubmitting: Practical Considerations Based
on the Psychology of Re-Reviews
Marc I. Rosen, M.D.
Most Successful Grants are Revise and Resubmits
Initial or Revised?
Number Application
s
Success Rate
Initial Submission
19,259 8.6%
Revise and Resubmit
5,373 37.2%
http://www.report.nih.gov/success_rates/index.aspx 2012 data for new R01s
Revising and Resubmitting: Lecture Structure
Deciding Whether to Revise and Resubmit
Suggestions for Revising and Resubmitting
ExampleMoral
Review Group Actions
Discussed and Scored with Impact Rating (score and percentile)
Triaged/Not Discussed with no Impact Rating but criteria scores (lower half)
Deferral; Not Recommended; Abstention
Deciding Whether to Resubmit: Consider the Score
Triaged/Not Discussed: How decided? Initial decision to triage at meeting rarely
overturned at meetingHard to completely convert a critic to a gung-ho
boosterCommittee is busy, focuses on close calls
2010-2012 R01s2.3% of unscored new R01s funded on resubmit8.7% of unscored continuation R01s fundedIncludes those not resubmittedhttp://report.nih.gov/FileLink.aspx?rid=880
Reading the Critiques• Read critiques carefully and calmly• Even if you are angry• Assume you got a good-faith, intelligent
review
• Let colleagues and mentors read the reviews for reality testing, support, and input
• Give more weight to comments that• Are in the “Summary”• Are made by more than one reviewer
Deciding Whether to Revise and Resubmit: Get More InformationContact Program Officer
-Ask about study section discussion-Ask about NIH Institute
interest in area?advice?
Talk through reviews with co-investigators and peers
• Reviewers assess your submitted material
• Reviewers are never totally wrong or right
• Extremely competitive process:
• Resubmission is common
• Avoid WYSIATI (what you see is all there is)---other talented people out there
Deciding Whether to Resubmit: Keep Perspective
Deciding Whether to Revise and Resubmit: Prospect TheorySunken Cost Fallacy
• Staying to the end of a boring movie hoping to recoup loss of spent money
• Using a fitness plan even when it’s painful
• It’s a fallacyLoss aversion: It’s not a rejection if you don’t give up
Thinking Fast and Slow (Daniel Kahneman)
Deciding Whether to Revise and Resubmit: Psychology
Overconfidence: Excessive Optimism• Only 5% of U. Chicago MBA students
predict they are in bottom 50%; most predict second decile
• 90% of drivers think they are above average
• Entrepreneurs say success rate for new business is 50% but predict personal success rate of 100%
• Few newlyweds expect to be among 50% who eventually divorce
Nudge, Thaler and Sunstein
Deciding Whether to Revise and Resubmit: Overconfidence?Test of Overconfidence: 90% Confidence Interval for
• Weight of earth in tons• 6.0 X 1024
• Percentage of world’s population who are Native English speakers (per CIA World Factbook 2009)• 4.83%
Reviews that Should NOT Make You Overconfident“This grant addresses an important topic”“Yale has superb facilities for this research”“The investigator is qualified”Only the first reviewer was critical of the
application and the grant was un-scored/poor score
Mild praise and the grant was un-scored/poor score
Meta-Critiques that May Not be Answerable
”There are already a lot of grants in this area”
“Not innovative”“Not significant”“Not exportable”
Consider Alternatives
A smaller grant (R21 instead of R01)
Another funding agencyA substantial change that you can submit as a new grant◦If you can answer critiques revise and
resubmit◦If you cannot answer critiques new grant
Deciding Whether to Revise and Resubmit• Do you have something better to work on for two-plus months?
Revising and Resubmitting: Lecture Structure
Deciding Whether to Revise and Resubmit
Suggestions for Revising and Resubmitting
ExamplesMoral
Consider the Person Behind the Review
◦ Look into research interests of people on the committee
◦ NIH REPORTER search of what committee has funded in the past
◦ Talk to people who have been on the committee
◦ Talk to your project officer
Consider the Person Behind the Review
◦ Reviewers want to avoid cognitive dissonance Cognitive dissonance
Inner drive to hold our attitudes and beliefs in harmony
Drive to avoid dissonance between themExamples
The Fox and the Sour GrapesThe review group that found fault with your grant
◦ So, don’t say the reviewer was wrong
Consider the Person Behind the Review
◦ Even Reviewers Who Change Their Minds Impacted by Anchoring to Prior Score Roulette Wheel Study of Anchoring 1: Volunteers shown rigged roulette wheel that stops at 10 or 65
What % of countries in the U.N. are in Africa?Wheel stops at 10---------25% averageWheel stops at 65---------45% average
http://en.wikipedia.org/wiki/Anchoring; Tversky and Kahneman, 1974
Consider the Reviewer’s Perspective
The reviewer who likes your application has to justify your response to a committee of 12+ very smart people
Make your response easy-to-follow
Use tables for complicated concepts (the reviewer can say to the committee—“He’s got a table laying that out.”)
Consider the Reviewer’s Perspective• If the reviewer likes the
application, he/she is more likely to agree with your justifications
• How juries decide:• They do not weigh the evidence • They do arrive at a narrative that
appears to fit the data• Answer meta-critiques
Revising and Resubmitting:Content Issues
Address any grant-killing meta-questions early in your response emphatically and clearly
Examples of Meta-Critiques
Critique: “They’ll never be able to pull this off – the project is not feasible.”
Answers: -Pilot data-Bring in collaborators who can pull it off
Examples of Meta-Critiques
Critique:“This was written by a slob who just does not know this topic well.”
Answers: -Emphasize how much the application has been cleaned up
-Consider adding expert who would have caught all your mistakes the first time.
Critique:“This was written by Kathy Carroll’s (Stephanie O’Malley’s, Rajita Sinha’s, Marc Potenza’s…) go-fer and is not really an independent application.”
Answer: -Spell out what is yours and what is not
Examples of Meta-Critiques
Examples of Meta-CritiquesCritique:“[zzzz’s] always make a hash out of [yyyy] research– it requires someone in my field of specialization. “
“The application would be strengthened by the involvement of a biostatistician.”
Answer: -Include someone with the recommended expertise
Examples of Meta-Critiques
Critique:”The study design is from hunger.”
Answer: -Table and/or figure justifying and explaining the study design
Revising and Resubmitting:Process IssuesRespond constructively and positively
◦The reviewer is always right (even if not).No more than 1-2 areas of
disagreement, but justify decision thoughtfully and respectfully
If not ready to submit at next deadline, DON’T◦Reviewers generally need a reason to
improve your score
Revising and Resubmitting:General Content Issues
“Thank you for the careful review of our proposal to […] We appreciate the careful, critiques”
Indicate method of highlighting changes (e.g., bold, italics in text)
Main criticism and responseList more minor criticisms (in italics) and
responses“Thank you for reconsidering our application”
Revising and Resubmitting:The Best Responses
Pilot DataRe-analysis of your own dataLiterature
Revising and Resubmitting:Weaker Responses
◦Logic◦Your opinion◦“In my clinical experience…”
Revising and Resubmitting:Don’t, Don’t, Don’t
(Usually) don’t answer questions that were not raised
Don’t malign the review process or the reviewer Don’t spend much effort pointing out that one
reviewer liked what another reviewer critiquedIf the reviewer asks for something that was already
in the application, be humble, e.g. “The information is presented more clearly this time in the methods as follows…”
Don’t get personal (no jokes, personal opinions, etc.)
Revising and Resubmitting:Don’t, Don’t, Don’t
• Don’t repeat every critical word from a review• Summarize criticisms (it was bad enough the first time)
• Don’t over-answer minor criticisms by writing a long essay that makes the criticism seem more major than it is
Revising and Resubmitting: Lecture Structure
Deciding Whether to Revise and Resubmit
Suggestions for Revising and Resubmitting
ExampleMoral
Example of Grant Review
4-year clinical trial to test computer-delivered counseling to improve engagement in work
Grant Review
• Lousy score of 270, 67th percentile• Program officer tells me they liked it, wanted to see it back
• Reviewer response:• Reviewer one liked• Reviewer two mixed• Reviewer three (statistician) gave it terrible score
Reviewer’s Potentially Grant-Killing Responses
“However, no data exists whether veterans would actually use the intervention.”
Summary Statement Recommends “Further conceptually develop and pilot test the internet-based intervention. Provide that data as a part of the proposal.”
Planned Response
• Agree with everything reviewers say and propose three-year, pilot-type, therapy development study to address it
Revising and Resubmitting: Lecture Structure
Deciding Whether to Revise and Resubmit
Suggestions for Revising and Resubmitting
ExampleMoral
Morals
It helps to enjoy the process◦Doing your best◦Advocating for something you believe in
◦Promoting yourselfYour CV lists grants◦No lasting harm from unfunded application
Moral“At the length, truth will out”
◦Shakespeare (Merchant of Venice)
“In the long run, we are all dead.”◦John Maynard Keynes
Thank you