Mike Grusak and Mike StraussOffice of Scientific Quality Review
Surviving the Path to Peer Review Success
“1998 Farm Bill”* ARS research peer-
reviewed every 5 years Most review panelists
external to ARS Satisfactory review
before beginning research
Why OSQR Review?
*Technically the Agricultural Research, Extension, and Education Reform Act of 1998 and not considered a “Farm Bill.”
National Action Plan
OSQR Review
Retrospective evaluation
Stakeholder input
Input
Implement
PlanAssess
Input Objectives set (PDRAM)
Project Plan prepared.
Research initiated
Annual progress reviews
You are here
NPL Validation
COI Lists
“Life after the PDRAM…”FIRST: Review OSQR Handbook and Area/RL expectations
-Plan Drafted lead scientist and project team -Review by other colleagues-Review by RL-Revision
Revised plan to Area Officefor approval
(some require proof of outsider review)
If needed, plan revisedApproved Plan sent by Area to
Office of National Programs
Validation by National Program Leader
Due to OSQR
Validated plan returned to Area
Revision ifNeeded (through Area)
Some Advice• Set a time line
• Based on when the plan is due to the Area Office• What does your Area require?
• Some want proof of review outside your group.• If not you should still send the plan outside your
group for review.
• Schedule time for:• Each member of the team to write • Members to coordinate plans• Lead scientist to compile a cohesive document• Colleagues to review the plan• RL to review the plan• Revision of the plan following review
Who Oversees OSQR?Mike Grusak, ARS Scientific Quality Review Officer (SQRO)• Approves chairs and panelists• Certifies project plans
Mike Strauss, OSQR Coordinator• Schedules panels, trains chairs and panelists• Manages the review
Both Mikes:• Attend panel meetings • Read project plans• Read reviews • Read and evaluate responses
Who are the Reviewers?Panelists:are your colleagues
They read your peer-reviewed papers
are active scientistsMost are academics
often know your workWhich may or may not help!
take their task very seriouslyThey don’t want to give low scores!
76%
13%8%
AcademiaIndustryGovtRetiredFormer ARSOther
Who are the Reviewers?
How is a Panel Selected?
Suggestions/nominations from ONP, others
Potential chairs screened for conflicts
Candidates interviewed, SQRO approves
Chairs Coordinator develop proposed panelists SQRO reviews and approves panel
Reviewers NEED to know…
What is the problem?Why is it important?
Where are you going with it?How are you going to get there?
Don’t make them hunt for this!
Aggregated PlansSome plans have “independent” pieces. Be aware of…
- Clarity…why are all these pieces here?What links them?State, however, that they are independent
- Consistency…in level of detailSomeone needs to oversee the final product.
- Content (flow)…an “easy read?”The general format is not rigid.
- “Consensus” All parts should say the same thing!
Review Products
• Action Class Score
• Consensus review comments
Project Review CriteriaAdequacy of Approach and
Procedures
Probability of Successfully accomplishing the Project’s Objectives
Merit and Significance
Title and Investigators..………….page 1Signature Page……………...........page 2Table of Contents……….………….page 3Project summary (250 words)...page 4Objectives...…………..................page 5 Need for research (1-2 p)Scientific Background (5-7 p)Approach & Procedures (6-15 p)Prior Accomplishments (2 p)Literature CitedMilestone Table (1-3 p)Past Accomplishments of Project Team MembersIssues of Concern statementsAppendices (letters plus other material)
15 - 30 pages+
4 pages for tables/figures
Document Outline
Project Plan Components
Write this in active voice…And plain English!
What are you doing? Why is it important? Earlier work (1-2 sentences)?What will you do next?So what!?
Make it compelling!
Project Summary – 250 words
Project Plan Components
Express need scientifically AND in the context of NP Action Plan.
Be concise in statement of research purpose.Discuss potential benefits and anticipated
products.Briefly note the overall approach (e.g., …using
microarray technologies we will elucidate…”
DON’T repeat, the overview!
NEED FOR RESEARCH: 1-2 pagesWhere are you going?
Project Plan ComponentsWhy are all these pieces here? How do they
relate?Include a figure.
- Objectives and sub-objectives- Personnel- Outcomes- Related projects
“What is the use of a book,”thought Alice, “without
pictures…”
Objectives: 1-2 pages
Project Plan Components
Gap AnalysisDemonstrate understanding and gaps. Not an exhaustive. Rationale for the workHow will this fill knowledge gaps?
1/3 of project plan length
Note similar projects within and outside ARS and how your past work prepares for or leads to this (provide details in the Prior Accomplishments section…but say enough to convince reviewers you know the area).
Cite preliminary data from your projects, if available
Scientific Background: 5-7 pages
Project Plan ComponentsPrior Accomplishments: 1-2 pages
Prior project terminated within two yearsMajor objectives and accomplishments Prior project investigatorsImpact of prior work (science, technology, users)Pertinent publicationsA table or chart of past data can be very helpful.
Project Plan Components Approaches & Procedures: 6-12 pages
How are you going to get there?Experimental designApproaches and methods any why they are appropriate. Advantages and limitations (important if “risky”).Who will do what, how, whenCollaborations Letters in Appendix need to confirm what you say! For SCAs, a copy of the agreement is sufficient. Management, evaluation, and contingencies. What is your path to success? How will you monitor it?
MILESTONES AND OUTCOMES Project Plan Components
Summarizes the project
Dynamic over the project lifecycle
Goal or Hypothesis
Goal or Hypothesis
Project Title
Project No.
National Program (Number: Name)
Objective
NP Action Plan Component
NP Action Plan Problem Statement
Subobjective
Goal/Hypothesis
SYTeam Months Milestone Anticipated Product Progress/Changes
12
24
36
48
60
Goal/Hypothesis
SYTeam Months Milestone Anticipated Product
Progress/Changes
12
24
36
48
60
See the OSQR Handbook for this new format for Milestones Table
This column for
managementafter review.
• Readability and narrative flow• Connection between parts (a diagram)• Appropriate roles for all • Appropriate expertise on team or from
collaborators• Grammar/spelling/proofreading• Appropriate detail in Approach• Clear, proper, milestones• Real contingencies
Does the plan instill confidence in this team’s abilities?
Project Plan Checklist
Real Hypotheses—Are they testable? NOT REQUIRED but don’t use a general goal where a hypothesis is better!
Lack of connection--How/why do the parts of your plan relate? Or if part does not, why is it there?
Uneven presentation—Edit for consistency
Context of plan—How does this fit with other similar work within and outside ARS?
Statistically sound—Are replicates sufficient? How will you analyze…“We’ve always done it this way” is not sufficient.
Some Frequent Criticisms
How will it get done?—Who does what? What other resources are there? (postdocs, technicians, students…include in human and physical resources)
Vagueness that prevents real analysis—If the information is confidential say why you can’t tell them but say enough to allow some level of analysis.
Risk without justification--Risk can be good but ONLY if it’s apparent you are aware of the challenge and have justified it.
Data accumulation without analysis—It’s not enough to gather data, what will you do with it?
Vacancies—Guidance on Web
Publishing prior work--If you have the data already the panel will want to see HOW you got it!
…More
To keep in mind…The reviewers need to see the logical
“thread” through your work.
Don’t make readers “search” for what you are doing!
Be clear, accurate, and correct.
Don’t assume reviewers know you and your work…(a poor plan may not be saved even if they do!)
Some hints to success…Proofread Your Plan
Ask a nontechnical person to read your plan
Ask someone who hasn’t seen it to read and proofread your plan
Ask a highly critical colleague to read it thoroughly.
Are collaborations documented appropriately?
Check hypotheses…
Treat this the same care you would a competitive proposal. The reviewers will!
HypothesesDon’t avoid them if appropriate but don’t force
them if they are not.Must be falsifiable and testable.Not restatements of objectives. A GOAL may be better for some work like breeding
or germplasm characterization…but explain that!
EITHER hypothesis or a clear goal is essential.Seek review by a statistician.
What Happens After Review? No, Minor or Moderate Revision
Lead Scientist responds to comments. Scientific Quality Review Officer certifies compliance with recommendations.
Major Revision or Not Feasible Lead Scientist revises and responds to comments. Panel performs a second review assessing response to their comments and assigns a new Action Class Score. If still Major or Not Feasible, project is returned for administrative action. No further review.
Projects are reviewed no more than two times(There are no page limits for revised plans)
The Impact of Vertically Striped Voles (VSV) on Wheat, Rye, and Egg Production
R. U. Kidding 1321-38000-123-00D 1/5/2006
Frontiers of Vole Biology and Relativity Theory
What Happens After Review?
Can I disagree with the panel?This is a dialogueIf you really disagree…put it away for a few days!Then…Honestly consider panel opinions. Be polite but if you disagree say whyDON’T skip changes to planDON’T insult or impugn panelistsDO provide justification for your alternative
view
Panels are NOT perfect…they are colleagues
How not to disagreeQ: The panel does not see any [expertise] in this plan.
A: “I disagree.” [no explanation]Q: Can you provide some preliminary data to support this idea.
A: “Yes, we have preliminary data but can’t/won’t show it to you.”Q: The panel suggests you try this approach.
A: But that’s just too difficult.A: We’re not allowed to alter this project in any way. [not true!]A: We’ve done it our way for [x] years and see no reason to change.A: The panel 5 years ago approved this so you can’t change it.
Q: This is not a hypothesis. Fix it or change to a goal statement.A: I looked at Tom’s plan and Bill’s and their panel didn’t make them do this so I don’t think I should have to do it.
Q: Did you do a power analysis?A: No we did not, but we’ve always done it this way before.A. Yes. It said we needed more so we ignored it.
Good to know…• Reviewers may (on rare occasion) comment on Objectives…we discourage
redirecting them or tinkering with the wording and ask, if they do, to not consider that in their scoring.
• A final copy of your responses is sent to the panel (for their information) after it is certified.
• For plans scoring Moderate or higher, OSQR reviews the responses to assure they are thorough and appropriate; and may return them for additional work before certification if needed.
• The Officer can decline certification if, after several attempts, it is judged that the researchers have not or cannot adequately address reviewer comments (i.e., your plan does not “pass” until it is certified).
• For plans scoring Major Revision or below, while OSQR may briefly check to see if the responses are thorough and respectful…this is not a detailed review and does not assure panel re-review success.
Last Words
ProofreadSeek Review
thenproofread and seek more review
And lastlyProofread and Seek Review
However…
You can be grammatically correct and STILL unintelligible!
`Twas brillig, and the slithy toves Did gyre and gimble in the wabe;All mimsy were the borogoves, And the mome raths outgrabe.
Lewis Carroll
Be thoughtful, clear, and thorough
…and beware of overconfidence…
Questions?
The Message…
By page 7 (Need for Research)
Subject of your researchWhy it is importantWhat it will produceHow you are going to get there