Date post: | 04-Jan-2016 |
Category: |
Documents |
Upload: | edwin-chase |
View: | 213 times |
Download: | 0 times |
CSREES Reporting Web Conference
April 14, 2008
E-mail questions to [email protected]
User Support
(202) 690-2910 or [email protected] Do not contact Texas A&M support FAQs and other information on the CSREES
Reporting Web Conference web page at www.csrees.usda.gov/rwc
E-mail questions to [email protected]
Format and Logistics
E-mail questions to [email protected] Also e-mail topic suggestions to
[email protected] Opportunity to vote for topics for next
conference Conferences are recorded and will be
available on the Reporting Web Conference web page at www.csrees.usda.gov/rwc
E-mail questions to [email protected]
Deb HamernikDeb is the CSREES National Program Leader for Animal Physiology and NRI Program Director for Bovine Genome Sequencing and Porcine Genome Sequencing. She has represented USDA on the NSTC-COS-Research Business Models Subcommittee to develop the Research Performance Progress Report (RPPR) since 2004.
(202) 401-4202
www.csrees.usda.gov/onesolution
Standard Progress Report
E-mail questions to [email protected]
Standard Reporting Across the Federal Government Implementation of the Federal Financial Assistance
management Improvement Act of 1999 (Public Law 106-107)
Facilitate information collection in lieu of numerous agency-specific forms
Does not change reporting requirements in OMB Circulars A-102 and A-110. Provides a standard format for collecting the information
Draft Research Performance Progress Report (RPPR) available at: www.nsf.gov/bfa/dias/policy/rppr/draftformat.pdf
E-mail questions to [email protected]
Standard Reporting Across the Federal Government
Agencies will use the standard categories & instructions developed for each category
Agencies may provide additional program-specific instructions to clarify program requirements
Agencies may develop additional agency- or program-specific reporting categories & instructions
E-mail questions to [email protected]
One Solution: CRIS Transition Standard Report
The revised CRIS AD 416 sections include: Goals/Objectives/Expected Outputs Methods Non-Technical Summary
The revised CRIS AD 421 sections include: Outputs Outcomes/Impacts Publications Participants Target Audiences Project Modifications
E-mail questions to [email protected]
Outputs
Activities, events, services, or products that reach people
Examples: conferences, field days, videos, curricula, patent applications, germplasm, genetic maps, students graduated, etc.
Do not report publications in this category
E-mail questions to [email protected]
Outputs--Dissemination
Dissemination refers to outreach activities to reach intended audiences to advance knowledge, encourage positive actions, or change conditions.
If educational materials and resources were distributed, describe the distribution method and the intended audience.
E-mail questions to [email protected]
Publications
Publications are outputs. For technical reasons, the CRIS system
collects publications in a separate box. Include paper or electronic publications Include status of publication (e.g., pending,
accepted, in press)
E-mail questions to [email protected]
Outcomes/Impacts
Changes in knowledge, actions, conditions Results of basic research projects should be
described as a change in knowledge (rather than experimental/technical details)
Results of extension activities should be described as a change in actions or conditions
E-mail questions to [email protected]
Participants Provide information about individuals who worked
on the project—their role and how they participated in the project
If applicable, describe partner organizations, collaborators, and contacts. Include collaborators outside the U.S.
Describe opportunities for training or professional development (trainees, K-12 teachers, producers, farmers, staff, volunteers, etc.
E-mail questions to [email protected]
Target Audiences
Provide information on target audiences for efforts designed to cause a change in knowledge, actions or conditions.
Include: individuals, groups, communities served by the project
Delivery of science-based knowledge to people through formal or informal educational programs
E-mail questions to [email protected]
Project Modifications
Describe major changes in approach and reason for change
Examples: changes in Assurance Statements (animals, humans, or biohazards); major problems or delays that have significant impact on rate of expenditures
E-mail questions to [email protected]
Tips
Do NOT re-enter the objectives and methods (already entered on the AD416)
Do NOT copy and paste abstracts for scientific meetings into the Standard Report
Use general terms for a lay audience More information is not necessarily better
information
E-mail questions to [email protected]
Standard Progress Report: For More Information
Deb Hamernik (202)401-4202 [email protected]
E-mail questions to [email protected]
Questions?
E-mail questions to [email protected]
For more information, visit the One Solution web page at www.csrees.usda.gov/onesolution
Djimé Adoum
Djimé assists the Director, Office of Planning and Accountability, in developing monitoring and evaluation systems to analyze program activities funded by CSREES and implemented by our Land Grant System partners, and provides leadership in strategic planning and the CSREES Portfolio Review process.
(202) 720-4564
www.csrees.usda.gov/opa
Practical, Realistic Approaches to
Measuring Impacts of Basic Research
Outline of the Presentation Reasons for Measuring Impact of Basic
Science Difficulties with Measuring Impact of Basic
Science Metrics and Efforts to Date R&D Criteria as Starting Point Experience from CSREES PREP Use of the Logic Model Summary and Conclusions
Why Measure the Impact of Basic Research?
Pressure due to limited resources Problems have become extremely complex
and require multi-disciplinary collaboration To secure public buy-in To demonstrate public value
Stating the Obvious
Measuring the impact of Basic Research is difficult
Attempts have been made to identify ways to measure
A few approaches have been determined to be of value
A Few Suggested Indicators
Inputs/Investments in S&T
Outputs Outcomes
Expenditures Comparison of expenditures Expenditures related to a product line Source of funding
Publications Patents Training of scientific and technical people Special Presentations and Honors Development of new testing methodologies Intellectual challenges
Measuring things of value to society, to the economy, to the nation and to society. Vaccines New plant variety Sustainable socio-economic and environmental development Less dependency on foreign oil
Metric Defined
Definition of a metric (Geisler 2000): It is a system of measurement that includes three elements:
the item being measured the unit of measurement the value of the unit
A Few Suggested Metrics
Bibliometric Analysis refers to measures of scientific and technical published outputs from science and its disciplines. It measures both quantity and quality (Geisler 2000).
A Few Suggested Metrics
Economic Analysis: a process that correlates financial measures for both investments/expenditures and outputs (source: Geisler 2000). It is extremely difficult to predict the outcomes of R&D
•.
A Few Suggested Metrics
Peer Review: A process by which a selective jury of experts in a given scientific field is asked to evaluate the undertaking of scientific activity or its outcomes (e.g., research, projects, or scientific publications.) Source: Geisler 2000
Efforts to date
A few models are selected to highlight evaluation of publically funded research:
United Kingdom’s Research Assessment Exercise (RAE)
The Japanese model The Australian (Australia’s Relative Funding Model)
- RFM The United States: GPRA and PART (NIH, NSF,
ARS and CSREES, NAS)
Evaluation based on the R&D Criteria
R&D Criteria Relevance
R&D investments must have clear plans, must be relevant to national priorities, agency missions,relevant fields, and “customer” needs, and must justify their claim on taxpayer resources.
QualityPrograms should maximize the quality of the R&D they fund through the use of a clearly stated,defensible method for awarding a significant majority of their funding.
PerformanceR&D programs should maintain a set of high priority, multi-year R&D objectives with annualperformance outputs and milestones that show how one or more outcomes will be reached.
To be used as broad guidelines applicable to all Federally funded R&D efforts.
CSREES Experience: Portfolio Review Expert Panel Process
R&D Criteria and Dimensions Relevance
ScopeFocusContemporary and/Emerging IssuesSolicitation and/or receptiveness for Stakeholders InputUtilization of Stakeholder Input
QualitySignificance of ResultsUsefulness and Utilization of ResultsIntegrationInterdisciplinary BalanceAlignment with Current State of Science
PerformanceProductivityComprehensiveness of Work ProducedAccountabilityManagement
To be used as broad guidelines applicable to all Federally funded R&D efforts.
The Logic Model
What is it? A roadmap, a conceptual framework, a program theory, program theory of action (Weiss, 1998; Patton, 1997, Bickman, 1987)
It is a concise way to show how a program is designed and will make a difference (Harvard Family
Research Project) It is the core of program planning, evaluation,
program management and communications (NAS, Kellogg Foundation, and UNW)
Use of logic model to
Set the context within which research takes place
Consider the concept of public value Provide a conceptual roadmap Ascertain the extent to which outputs led to
new knowledge, applications, solutions and reasonably consistent with expenditures
Provide framework for evaluation
Occur when a societal condition is improved due to a participant’s action taken in the previous column.
For example, specific contributions to:
- Increased market opportunities overseas and greater economic competitiveness
- Better and less expensive animal health
- Vibrant & competitive agricultural workforce
- Higher productivity in food provision
- Better quality-of-life for youth & adults in rural communities
- Safer food supply
- Reduced obesity and improved nutrition & health
- Higher water quality and a cleaner environment
Generic Logic Model for CSREES Reporting CSREES – Office of Planning & Accountability(This model is intended to be illustrative guide for reporting on CSREES-funded research, education and extension activities. It is not a comprehensive inventory of our programs.)
Outcomes
Actions
InputsSituation Activities
Knowledge
What we invest:
- Faculty
- Staff
- Students
- Infrastructure
- Federal, state and private funds
- Time
- Knowledge
- The collection of stakeholder opinions
Occurs when there is a change in knowledge or the participants actually learn:
- New fundamental or applied knowledge
- Improved skills
- How technology is applied
- About new plant & animal varieties
- Increased knowledge of decision-making, life skills, and positive life choices among youth & adults
- Policy knowledge - New improved methods
Description of challenge or opportunity
- Farmers face increasing challenges from globalization
- Opportunity to improve animal health through genetic engineering
- Insufficient # of trained & diverse professionals entering agricultural fields
- Youth at risk
- Invasive species is becoming an increasing problem
- Bioterrorism
- Obesity crisis
- Impaired water quality
EXTERNAL FACTORS - A brief discussion of what variables have an effect on the portfolio, program or project, but which cannot be changed by managers of the portfolio, program, or project. For example, a plant breeding program’s success may depend on the variability of the weather...etc.
Occur when there is a change in behavior or the participant’s act upon what they’ve learned and:
- Apply improved fundamental or applied knowledge
- Adopt new improved skills
- Directly apply information from publications
- Adopt and use new methods or improved technology
- Use new plant & animal varieties
- Increased skill by youth & adults in making informed life choices
- Actively apply practical policy and decision-making knowledge
Conditions
ASSUMPTIONS - These are the premises based on theory, research, evaluation knowledge etc. that support the relationships of the elements shown above, and upon which the success of the portfolio, program, or project rests. For example, finding animal gene markers for particular diseases will lead to better animal therapies.
What we do (Activities): - Design and conduct research- Publish scientific articles- Develop research methods and procedures- Teach students- Conduct non-formal education- Provide counseling- Develop products, curriculum & resources
Who we reach (Participation):
- Other scientists- Extension Faculty- Teaching Faculty- Students- Federal, state & private funders- Scientific journal, industry & popular magazine editors- Agencies- Policy and decision- makers- Agricultural, environmental, life & human science industries- Public
Outputs
Version 1.2
- New fundamental or applied knowledge
- Scientific publications
- Patents
- New methods & technology
- Plant & animal varieties
- Practical knowledge for policy and decision-makers
- Information, skills & technology for individuals, communities and programs
- Participants reached
- Students graduated in agricultural sciences
Logic model as a planning and an evaluation tool
Situation Inputs Activities Outputs Outcomes
Knowledge Actions Conditions
A problem of local and national interest
For example:
Reduce the nation’s dependency on foreign oil
$$$$
Expenditures or
Investments
Staff and partners
Genetically engineer a bacterium to convert cellulosic material into ethanol
One bacterium engineeredPatentsNew applications
Advanced knowledge
High efficiency conversation
Cost savingsImproved qualityReduced dependency on foreign oil
Assumptions External factors
Summary and Conclusions
It is fully recognized that measuring the impact of basic science is difficult (OMB and NAS)
Impact might not be realized long after studies are completed
Debate not over yet but limited public resources have led to scrutiny about return on investments and the need to document effectiveness and efficiency of investment
The use of logic model as a planning and evaluation tool has gained some ground
The CSREES PREP Process has been quite helpful
Questions?
E-mail questions to [email protected]
For more information, visit the Planning and Accountability web page at www.csrees.usda.gov/opa
E-mail questions to [email protected]
See you in June!!!
Next CSREES Reporting Web Conference will be on Thursday, June 12 from 2-4 pm (Eastern)
Visit the conference web site at www.csrees.usda.gov/rwc for: The recording of this conference The slides from this conference Recordings and slides from past conferences Announcements