Mike Grusak and Mike StraussOffice of Scientific Quality Review

Surviving the Path to Peer Review

Success

1998 Farm Bill ARS research peer-

reviewed every 5 years Most review panelists

external to ARS Satisfactory review

before beginning research

Why OSQR Review?

National Action Plan

OSQR Review and Certification.

Retrospective evaluation.

Stakeholder input

Input

Implement

PlanAssess

Input

Objectives set (PDRAM).

4. Project Plan prepared.

Research initiated

Annual progress reviews

Congressional Mandate

You are here

NPL Validation

“Life after the PDRAM…”FIRST: Review OSQR Handbook and Area/RL expectations

-Plan Drafted lead scientist and project team -Review by other colleagues-Review by RL-Revision

Revised plan to Area Officefor approval

(some require proof of outsider review)

If needed, plan revisedApproved Plan sent by Area to

Office of National Programs

Validation by National Program Leader

Due to OSQR

Validated plan returned to Area

Red denotes establishedDates for completion.

Revision ifNeeded (through Area)

Some Advice• Set a time line

• Based on when the plan is due to the Area Office• What does your Area require?

• Some want proof of review outside your group.• If not you should still send the plan outside your

group for review.

• Schedule time for:• Each member of the team to write • Members to coordinate plans• Lead scientist to compile a cohesive document• Colleagues to review the plan• RL to review the plan• Revision of the plan following review

Who Oversees OSQR?Mike Grusak, ARS Scientific Quality Review Officer (SQRO)• Approves each panel chair and panel member who

participates in OSQR• Responsible for certifying that project plans have completed

review

Mike Strauss, OSQR Coordinator• With two staff members, schedules all panels, contacts and

trains panel members, and coordinates the review and certification process.

Both Mikes:• Attend panel meetings • Read project plans• Read the OSQR reviews • Read responses to review

Who are the Reviewers?Panelists are your colleagues.

They read your peer-reviewed papers.

Panelists are active scientists.Most are academics (per the Farm Bill).

Panelist often know your work.And are often familiar with your excellence.

Panelists take their task very seriously.The devote many hours to each review.

They don’t want to give low scores!

76%

13%8%

0% 1% 2%

AcademiaIndustryGovtRetiredFormer ARSOther

Who are the Reviewers?

How is a Panel Selected?OSQR Receives suggestions/nominations from

ONP, Areas, others.

All potential chairs are screened for COIs.

Coordinator Interviews potential candidates and SQRO approves Chairs.

Chairs work with Coordinator to develop a balanced, proposed list of panel members. SQRO reviews and approves final list of panelists.

Reviewers NEED to know…What is the problem?Why is it important?

Where are you going with it?How are you going to get there?And how will you know you have

arrived?

Don’t make them hunt for this!

Aggregated PlansIn some cases plans may be comprised of several

independent pieces. In such cases be aware of…

- Clarity…why are all these pieces here?What links them?State, however, that they are independent

- Consistency…each piece should be equivalent in detailSomeone needs to oversee the final product.

- Content (flow)…is it an “easy read?”The general format is not rigid.Consider blending background and approach for each section.

- “Consensus” First parts (summary, need for research, objectives…need to present the whole picture.

The Message…

By page 7 (Need for Research) reviewers should know:

The subject of your researchWhy it is importantWhat it will produceHow you are going to get there

The rest of the document will “flesh out” this but if the reviewers don’t know it by this point, they probably won’t get it easily from the rest of the document.

Review Products

• Action Class Score

• Consensus review comments

Project Review CriteriaAdequacy of Approach and

Procedures

Probability of Successfully Accomplishing the Project’s Objectives

Merit and Significance

Title and Investigators..………….page 1Signature Page……………...........page 2Table of Contents……….………….page 3Project summary (250 words)...page 4Objectives...…………..................page 5 Need for research (1-2 p)Scientific Background (5-7 p)Approach & Procedures (6-15 p)Prior Accomplishments (2 p)Literature CitedMilestone Table (1-3 p)Past Accomplishments of Project Team MembersIssues of Concern statementsAppendices (letters plus other material)

15 - 30 pages

Document Outline

These are not boxes…they are guides to your narrative flow.

Project Plan Components

Write this in active voice.State the essential problem and why it is

important. What have you done to date (1-2 sentences)?How will you address the issue?Why is this important?

This is where you capture the interest of the reviewer. Make it compelling!

Project Summary – 250 words

Project Plan Components

Express need scientifically AND in the context of NP Action Plan.

Be concise in statement of research purpose.Discuss potential benefits and anticipated products.Identify relevant customers and stakeholders.Briefly note the principal methods you will utilize

(e.g., …using microarray technologies we will elucidate…”

Build upon, don’t repeat, the overview!

NEED FOR RESEARCH: 1-2 pagesWhere are you going?

Project Plan Components

Why are all these pieces here? How do they relate?Are there other similar projects?A figure can help! Should reflect your plan and be a

guide.- Objectives and sub-objectives- Personnel- Outcomes- Related projects

Plans can include 4 total pages of figures. Use them!

“What is the use of a book,” thought Alice, “without pictures…”

Objectives: 1-2 pagesHow does all this fit?

Project Plan Components

Highlight knowledge gaps.Literature demonstrates understanding and gaps. Not an exhaustive. Show a rationale for the objectives.How will this fill knowledge gaps?Limit to 1/3 of project plan lengthNote similar projects within and outside ARS and how your past work prepares for or leads to this (provide details in the Prior Accomplishments section…but say enough to convince reviewers you know the area).

Cite preliminary data from your projects, if available

Scientific Background: 5-7 pagesWhy are you going there?

Project Plan ComponentsPrior Accomplishments: 1-2 pages

What have you done before?

Highlighted briefly in the Background.

Name prior project terminated within two yearsMajor objectives and accomplishments Prior project investigatorsImpact of prior work (science, technology, users)Pertinent publicationsA table or chart of past data can be very helpful.

Project Plan Components Approaches & Procedures: 6-12 pages

How are you going to get there?Set out your Experimental design. Formulate REAL, testable, hypotheses!

Describe approaches and methods any why they are appropriate. Discuss advantages and limitations (important if methods are “risky”).Illustrate how objectives can be achieved. Who will do what, how, and when (including collaborators and SCAs!)Describe nature and extent of collaborations, including SCAs Letters in Appendix need to confirm what you say! For SCAs, a copy of the agreement is sufficient. Include management, evaluation and contingencies. What is your path to success? How will you monitor it?

MILESTONES AND OUTCOMES Project Plan Components

Summarizes the project

Dynamic over the project lifecycle

Goal or Hypothesis

Goal or Hypothesis

• Readability and narrative flow• Connection between parts (a diagram)• Appropriate roles for all • Appropriate expertise on team or from

collaborators• Grammar/spelling/proofreading• Appropriate detail in Approach• Clear, proper, milestones• Real contingencies

Does the plan instill confidence in this team’s abilities?

Project Plan Checklist

Real Hypotheses—Are they testable? NOT REQUIRED but don’t use a general goal where a hypothesis is better!

Lack of connection--How/why do the parts of your plan relate? Or if part does not, why is it there?

Uneven presentation—If different people wrote different parts, it will show!

Context of plan—How does this fit with other similar work within and outside ARS?

Statistically sound—Are replicates sufficient? How will you analyze…don’t just throw out jargon (“Data will be analyzed using ANOVA.”)…are replicates sufficient? When you’re done, will you be able to know if you arrived? “But we’ve always done it this way” is not sufficient.

Some Recent (frequent) Criticisms

How will it get done?—Who does what? What other resources are there? (postdocs, technicians, students…include in human and physical resources)

Vagueness that prevents real analysis—If the information is confidential say why you can’t tell them but say enough to allow some level of analysis.

Risk without justification--Risk can be good but ONLY if it’s apparent you are aware of the challenge and have justified it.

Data accumulation without analysis—It’s not enough to gather data, what will you do with it?

Vacancies—These can be a challenge. Guidance on web

Publishing prior work--If you have the data already the panel will want to see HOW you got it!

…More frequent Criticisms

To keep in mind…The reviewers need to see the logical

“thread” through your work.

Don’t make readers “search” for what you are doing!

Be clear, accurate, and correct.

Don’t assume reviewers know you and your work…(a poor plan may not be saved even if they do!)

Some hints to success…Proofread Your Plan

Ask a nontechnical person to read your plan

Ask someone who hasn’t seen it to read and proofread your plan

Ask a highly critical colleague to read it thoroughly.

Are collaborations documented appropriately?

Check hypotheses…

Treat this the same care you would a competitive proposal. The reviewers will!

Hypotheses1. Don’t avoid them if appropriate but don’t force

them if they are not.2. Must be falsifiable and testable.3. Not restatements of objectives. 4. May not be appropriate for work like breeding

or germplasm characterization…but explain that!

5. Seek review by a statistician.

Sometimes a GOAL is preferable to an Hypothesis. We recommend:

Nabel, Gary J., 2009The Coordinates of Truth

Science, 326 (5949): 53-54

What Happens After Review?

No, Minor or Moderate RevisionLead Scientist responds to comments. Scientific Quality Review Officer certifies compliance with recommendations.

Major Revision or Not Feasible Lead Scientist revises and responds to comments. Panel performs a second review assessing response to their comments and assigns a new Action Class Score. If still Major or Not Feasible, project is returned for administrative action. No further review.

Projects are reviewed no more than two times

(There are no page limits for revised plans)

The Impact of Vertically Striped Voles (VSV) on Wheat, Rye, and Egg Production

R. U. Kidding 1321-38000-123-00D 1/5/2006

Frontiers of Vole Biology and Relativity Theory

What Happens After Review?

Can I disagree with the panel?This is a dialogueIf you really disagree…put it away for a few days!Then…Honestly consider panel opinions. Be polite but if you disagree say whyDON’T skip changes to planDON’T insult or impugn panelistsDO provide justification for your alternative

view

Panels are NOT perfect…they are fellow scientists

How not to disagreeQ: The panel does not see any [expertise] in this plan.

A: “I disagree.” [no explanation]Q: Can you provide some preliminary data to support this idea.

A: “Yes, we have preliminary data but can’t/won’t show it to you.”Q: The panel suggests you try this approach.

A: But that’s just too difficult.A: We’re not allowed to alter this project in any way. [not true!]A: We’ve done it our way for [x] years and see no reason to change.

Q: This is not a hypothesis. Fix it or change to a goal statement.A: I looked at Tom’s plan and Bill’s and their panel didn’t make them do this so I don’t think I should have to do it.

Q: Did you do a power analysis?A: No we did not, but we’ve always done it this way before.A. Yes. It said we needed more so we ignored it.

Good to know…• A final copy of your responses is sent to the panel (for their information)

after it is certified.

• For plans scoring Moderate or higher, OSQR reviews the responses to assure they are thorough and appropriate; and may return them for additional work before certification if needed.

• The Officer can decline certification if, after several attempts, it is judged that the researchers have not or cannot adequately address reviewer comments (i.e., your plan does not “pass” until it is certified).

• For plans scoring Major Revision or below, while OSQR may briefly check to see if the responses are thorough and respectful…this is not a detailed review and does not assure panel re-review success.

Last Words

ProofreadSeek Review

thenproofread and seek more review

And lastlyProofread and Seek Review

However…

…Correct Grammar and Spelling are Important -- but not enough

`Twas brillig, and the slithy toves  Did gyre and gimble in the wabe;All mimsy were the borogoves,  And the mome raths outgrabe.

Lewis Carroll

Be thoughtful, clear, and thorough

…and beware of overconfidence…