Securing External Federal Funding
Janice F. Almasi, Ph.D.
Carol Lee Robertson Endowed Professor of Literacy
University of Kentucky
Institute of Education Sciences2 Grant Competitions per year
Janice F. Almasi, Ph.D., University of Kentucky
Current Funding Opportunities14 Long-term Programs of Research
Drag picture to placeholder or click icon to add
Be InformedSubscribe to Newsflash at ies.ed.gov/newsflash
Janice F. Almasi, Ph.D., University of Kentucky
IES Research Goals
Goal 1: Identification Identifying programs and practices associated with
better educational outcomes (secondary data analysis)
Goal 2: Development Projects Developing educational interventions
Goal 3: Efficacy and Replication Projects Determine if fully-developed interventions are effective
Goal 4: Scale-Up Goal 5: Measurement Projects
Janice F. Almasi, Ph.D., University of Kentucky
Prior to Peer Review Meeting
Triage identifies top 25 applications
Reviewers read, rate about 8 applications
Reviewers check for COIs
Each application assigned to at least 2 reviewers
Janice F. Almasi, Ph.D., University of Kentucky
Criteria
Significance
Research Plan
Personnel
Resources
Janice F. Almasi, Ph.D., University of Kentucky
Review Criterion Ratings(M
ore
Wea
knes
ses
than
St
reng
ths)
(Bal
ance
of
Stre
ngth
s an
d
Wea
knes
ses)
(Mor
e St
reng
ths
than
Wea
knes
ses)
1
Poor Excellent
765432
Janice F. Almasi, Ph.D., University of Kentucky
What Reviewers Look ForWhere Applications Tend to be Weak
Janice F. Almasi, Ph.D., University of Kentucky
Significance
Goal 1
• Theoretical and empirical rationale for study and practical importance of the intervention (e.g., program, practice) that will be examined
Goals 2 and 3
• Describe (a) the intervention (e.g., features, components) and the logic model for the intervention, (b) theoretical and empirical support for intervention, and (c) practical importance of the intervention
Janice F. Almasi, Ph.D., University of Kentucky
Significance in Goals 2 and 3
Context for Proposed Interventions Provide context for the proposed intervention by including
data on, or reviewing research describing, the attributes of typical existing practices.
Identify shortcomings of current practice and how they contribute to the rationale for the proposed intervention.
Provide context for understanding how much of a change the proposed intervention is intended to achieve.
Janice F. Almasi, Ph.D., University of Kentucky
Significance in Goals 2 and 3
Intervention, Theory of Change, Empirical/Theoretical Rationale Clearly describe the intervention Clearly describe the theory of change for the intervention
How do the features or components of the intervention relate to each other temporally (or operationally), pedagogically, and theoretically (e.g., why A leads to B)?
Provide a strong theoretical and empirical justification for the design and sequencing of the features or components of the intervention. Enables evaluation of: Relation between the intervention and its theoretical and
empirical foundation (e.g., is the proposed intervention a reasonable operationalization of the theory?)
Relation between the intervention and the outcome measures (e.g., do the proposed measures tap the constructs that the intervention is intended to address?)
Include a Logic Model
Janice F. Almasi, Ph.D., University of Kentucky
Significance in Goals 2 and 3
Practical Importance of Intervention When the proposed intervention is fully developed will it
have the potential to improve student outcomes in educationally meaningful increments, if it were implemented over the course of a semester or school year?
Would the proposed intervention be both affordable for and easily implemented by schools (e.g., not involve major adjustments to normal school schedules)?
Janice F. Almasi, Ph.D., University of Kentucky
Rese
arch
Pla
n:
Goal 2
Sample• Samples and settings used to
assess feasibility of intervention and for pilot data assessing promise of intervention
Iterative Development Process• Revision• Implementation• Observation• Revision
How do you define“operating as intended?”
What data will begathered to determine
how intervention isoperating?
How will the datagathered be used
to revise the intervention?
What criteria will be used to
determine if the intervention operates
as intended?
Janice F. Almasi, Ph.D., University of Kentucky
Rese
arch
Pla
n:
Goal 2
Feasibility of Implementation• Goal is a fully developed intervention• Data that addresses feasibility of
implementing in small sample of authentic education settings
• Promise of intervention in terms of outcomes
Pilot Study• Pilot data on outcome measures progressing
in right direction• Pilot data demonstrates implementation of
intervention is associated with behaviors consistent with theory of change
• No more than 30% of funds• Data should not be a test of efficacy
Janice F. Almasi, Ph.D., University of Kentucky
Rese
arch
Pla
n:
Goal 2
Measures• Clearly describe procedures for
gathering data to refine and revise the intervention and provide insight into feasibility and usability of proposed intervention• What needs to be observed?• How will observations be
gathered?• Clearly describe measures that will
be used (and reliability and validity if appropriate)
Janice F. Almasi, Ph.D., University of Kentucky
Reearch
Pla
n:
Goal 3
Research Questions• Pose clear, concise hypotheses
or research questions
Sample• Define sample to be selected• Define sampling procedures
(including justification for inclusion and exclusion)
• Strategies to be used to reduce attrition
Janice F. Almasi, Ph.D., University of Kentucky
Reearch
Pla
n:
Goal 3
Research Design• Provide detail!• How will threats to internal/external validity be
addressed?• Studies using random assignment are preferred
where feasible• What is unit of randomization and what
procedures will be used to make assignments to conditions?
Power• What power is needed to detect a reasonably
expected and minimally important effect?• How was effect size calculated?• If clusters are randomly assigned to treatment
conditions be sure to include intraclass correlation and anticipated effect size in power analysis
Janice F. Almasi, Ph.D., University of Kentucky
Rese
arch
Pla
n:
Goal 3
Measures• Justify appropriateness of measures• Are measures of practical interest to
educators and not overly aligned with intervention?
• Include reliability and validity informationFidelity of Implementation• How will implementation be documented and
measured?• How will factors associated with fidelity be
identified and assessed?• How will fidelity data be incorporated into
analyses of impact?• How do conditions in the school setting affect
fidelity of implentation?
Janice F. Almasi, Ph.D., University of Kentucky
Rese
arch
Pla
n:
Goal 3
Comparison Group• How does comparison group compare to intervention
on critical features of intervention?• Using a “business-as-usual” comparison is
acceptable but explain why using it is acceptable• How will contamination be avoided?
Mediating and Moderating Variables• Observational, survey, or qualitative methods are
encouraged to help identify factors that may explain the effect or lack of effect of intervention
Data Analysis• Quantitative: Specify statistical procedures and
include formulas where appropriate• Qualitative: Specific methods used to index,
summarize, and interpret data should be identified• Relation between hypotheses, measures, and
independent and dependent variables should be clear
Janice F. Almasi, Ph.D., University of Kentucky
Personnel
What role will each individual have in the project?
What qualifications, training, and experience do key personnel possess?
How will qualifications be used on the research?
Are key personnel dedicating sufficient time to competently implement proposed research?
Janice F. Almasi, Ph.D., University of Kentucky
Resources
Are the resources adequate to support the proposed activities in terms of: Facilities Equipment Supplies Institutional Support for Managing/Directing Grants and
Supporting Scholarship
Have partners shown support for implementation and support of the project?
Janice F. Almasi, Ph.D., University of Kentucky
Scientist Reviewer Critiques
A brief description of the overall application.
Identify each application’s key strengths and weaknesses in each of the evaluation areas and prepare critical, evaluative comments.
Integrated summary of the overall assessment of the application, including the main strengths and weaknesses of the application.
Janice F. Almasi, Ph.D., University of Kentucky
Overall Score
Overall Score Range Adjectival Equivalent
1.0 - 1.5 Outstanding
1.6 - 2.0 Excellent
2.1 - 2.5 Very Good
2.6 - 3.0 Good
3.1 - 4.0 Fair
4.1 - 5.0 Poor
Janice F. Almasi, Ph.D., University of Kentucky
Peer R
evie
w M
eetin
g P
roce
ss3. Notetaker
Summarizes the discussion orally and in writing
2. Full PanelDiscusses the application, asks questions, and offers additional
critiqueDiscusses the budget
1. Assigned Scientist Reviewers Share Overall ScoresReviewer 1 Summarizes the
application and its strengths and weaknesses in each
Reviewer 2 elaborates on areas of agreement or disagreement
Janice F. Almasi, Ph.D., University of Kentucky
Peer R
evie
w M
eetin
g P
roce
ss6. Assigned Scientist Reviewers
May edit/revise their original written critiques based on panel discussion
5. Full PanelPrivately assigns criteria scores and overall
scores
4. Assigned Scientist ReviewersAdjust initial recommended criteria scores and
overall scores