Date post: | 30-Dec-2015 |
Category: |
Documents |
Upload: | finn-gardner |
View: | 36 times |
Download: | 1 times |
CSR Advisory Council Meeting
May 19, 2014
Editorial Board ReviewA Few Good Reviewers
Don Schneider, Ph.D.
• Format modeled on journal manuscript review
• First piloted in 2008 with 6 SBIR panels, just in time for TR01s, Challenge grants, DP1 etc.
• Two stages– First Stage – Mail reviewers– Second Stage – Editors
Past Experience
First Stage/Mail Reviewers
• Subject matter experts– Provide depth in review
• Focus on scientific and technical merit• 2-3 first stage mail reviewers per application• Submit full critiques• Give overall impact and criterion scores
– Overall impact scores not factored into final priority score
• Hold face-to-face meeting• Recruit broad experts
– Provide perspective in review (assign about 15 applications each)
• Focus on impact and significance• Assign 3 second stage reviewers per application• Consider first stage critiques in review• Write overall impact paragraph• Give overall impact score
– Final priority score based on second stage only
Second Stage/Editors
• Provides both depth and breadth in review
• Optimizes use of the best reviewers
• Scales well for large numbers of applications(second stage discusses a fraction of the applications)
Rationale
Perceived Advantages
• Involves no travel/teleconference for first stage reviewers
• Allows small, interactive face-to-face meetings
• Promotes better scoring and assessment of impact
• Lessens travel and lodging expenses and inconveniences
Review # of Applications Cost/application
Regular R01 F2F $518
DP1/Pioneer EB+I 244 $280
DP2/New Innov EB 593 $124
DP5/Early Indep EB+I 84 $875
Cost Considerations(Alicia Caffi)
• Recruitment of large numbers of reviewers
• Timeline– Tight, two sequential reviews – (in the 17 week cycle)
• More staff time required (SROs)
• Some sense of isolation by first stage reviewers
Challenges
• Each application examined by at least 5 reviewers
• Interactive, thoughtful discussions
• Overall scoring by second stage members
• Reviewers and staff like final review products
Review Outcomes
• Survey conducted by A Kopstein of reviewers participating in SBIR pilots 2008
• Outcomes were generally positive
– Majority willing to participate in either review stage in future
– Editorial Board Review:• Increases expert review
» 3/4ths of respondents• Preferred for their own applications
» 2/3rds of respondents
Survey