A Credible Approach to Benefit-Cost Evaluation for Federal Research &Technology Programs: A U.S. Department of Energy Approach Presented at American Evaluation Association Conference November 2009 Gretchen Jordan, Sandia National Laboratories Rosalie Ruegg, TIA Consulting, Inc. Work presented here was completed for the U.S. Department of Energy (DOE) Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the authors. TIA Consulting, Inc. SAND Number: 2009-6686C
Transcript
A Credible Approach to Benefit-Cost Evaluation for Federal Research &Technology Programs:
A U.S. Department of Energy ApproachPresented at
American Evaluation Association ConferenceNovember 2009
Gretchen Jordan, Sandia National LaboratoriesRosalie Ruegg, TIA Consulting, Inc.
Work presented here was completed for the U.S. Department of Energy (DOE) Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the authors.
TIA Consulting, Inc. SAND Number: 2009-6686C
Jordan and Ruegg 2009 2
Outline
Overview of EERE Objectives for Benefit-Cost Study Guidelines Economic benefits and costs
Specifying next best Assessing additionality
Environmental, Security, and Knowledge benefits Challenges and Summary
Jordan and Ruegg 2009 3
Office of Energy Efficiency and Renewable Energy (EERE)
EERE accomplishes its mission through 10 Technology Development (TD) Programs and the Office of Technology Advancement and Outreach (TAO):
– Power Generation• Wind & Hydropower• Solar• Geothermal
– Energy Efficiency • Building Technologies• Industrial Technologies• Weatherization• Federal Energy Management
Jordan and Ruegg 2009 44
Purpose: to answer the following questions about EERE programs over their 30-year history
To what extent have the programs thus far:
• Produced economic benefits in terms of resource savings relative
to program costs?• Yielded environmental benefits in terms of Green House Gas
reductions and health effects from reduced air pollution? • Yielded energy security benefits in terms of reduced imported oil
and threats to the U.S. energy infrastructure?• Built a knowledge base within each respective field and
disseminated knowledge in and outside those fields?
• What has been the return on public investment in the EERE R&D energy programs thus far?
Jordan and Ruegg 2009 55
Approach: Key Features of the EERE R&D Benefit-Cost Studies
• Retrospective, following best-practice B-C methodology• Cluster approach to extend study usefulness• Comprehensive -- treatment of 4 types of benefits
- Economic- Environmental- Security
- Knowledge
• Consistent & uniform across studies (as appropriate)- Use of unifying framework (Mansfield Model)- Same set of Benefit-Cost conventions- Same set of economic performance measures- Uniform formatting of reports
Jordan and Ruegg 2009 6
“Cluster Analysis” Approach: benefits of elements of a research/technology cluster”
compared to entire cluster costs
Quantitative benefitsof selected elements
of a research/technology “cluster”
Investment costs of the selected elements for detailed study
Investment costs of entire clusterQualitative effects of other
elements in the cluster
Jordan and Ruegg 2009 7
What It Provides : Economic Benefits• Economic Performance Metrics
– Net benefits – Benefit-cost ratio – Internal rate of return
• Perspective is Return on Public Investment (Cost of EERE Program/Subprogram)
• All affected resources in the economy are included in Economic Metrics, such as– Investment costs– Energy costs– Labor costs
Jordan and Ruegg 2009 88
Areas of Special Focus in Estimating Economic Benefits
• Specifying the “Defender Technology”
What would have been “the next-best alternative” used in lieu of the subject technology?
• Accounting for “Additionality”
What was different about the subject technology and market as a result of the EERE Program/Subprogram?
Jordan and Ruegg 2009 99
Specifying the “Defender Technology”
• Merits of the subject technology are judged retrospectively against the “next best alternative” at the time the investment decision was made
• Counterfactual – what would otherwise have been used• Factors affecting the selection of the “defender”
- were investment decisions constrained or unconstrained?
- was the subject technology new to the world or an incremental improvement over an existing system?
- was the subject technology a total system or a component?
- was the subject technology a product or a process?
• Is static or dynamic modeling needed?
Jordan and Ruegg 2009 1010
Accounting for “Additionality”
Effects - Ways the program may have made a difference • accelerated technology entry into the marketplace • improved the performance characteristics of the
technology• changed the cost of a technology • increased market size
A Matrix for Assessing Attribution by Technology Stage
Categories of Information Needed for Additionality
Assessment
Technology Timeline (Stage of Research, Development, and Commercialization)
Preliminary & detailed
investigation
Develop components
Develop system
Validate/ demonstrate
Commer-cialize
Market Adoption
History of the technology
What DOE Did
What Others Did (Rival Explanations: Private Sector, Other Nations)
What Others Did (Rival Explanations –US & State Government)
The DOE Effect Type
Description of DOE Influence
And its strength
Basis of evidence of influence
Jordan and Ruegg 2009 1212
What It Provides: Environmental Benefits• Greenhouse Gas Effects
- physical units of GHG emissions avoided
- equivalent changes required to produce similar GHG effects (use of EPA’s GHG Equivalencies Calculator)
• Public health benefits from reduced air pollution (NOx, SO2, PM, etc.)– calculated using EPA’s COBRA Model
- mortality and morbidity measures
- health cost measures (which may be combined with economic estimates if data quality is sufficiently high)
• Any notable other effects -- water resource use, water discharges, land resource use, and solid waste generation (treated at a minimum qualitatively)
Jordan and Ruegg 2009 1313
What It Provides: Security Benefits
• Barrels of oil equivalent units avoided
• Monetary value will not be applied to barrels of oil equivalent units, as the methodology is considered at this time to require further development
• Notable effects on the security of infrastructure will be identified
• Future potential political and military security issues linked to GHG emissions will be acknowledged where the reduction of GHG emissions is notable
Jordan and Ruegg 2009 1414
What It Provides: Knowledge Benefits
• Outputs and dissemination paths of patents/papers/prototypes/models to commercial users within the target industry & to other industries
• “Comparison of citing of EERE-attributed patents/papers with other organizations
• Hot” patents/papers (with greater than expected citing intensity) traceable to EERE Program
• Knowledge creation and exchange through partnerships with companies and universities
• Licensing of intellectual property (limited)
• International knowledge flows
Jordan and Ruegg 2009 15Jordan and Ruegg 2009 15
Challenges
• Maintaining quality of measures across types of benefits
• Consistent and appropriate designation of each next-best alternative to use as a baseline for estimating economic benefits across studies
• Assessment of external influences that may constitute rival explanations of outcomes in estimating economic benefits
• Difficulties in measuring other types of benefits, e.g., security benefits, for which methodology is less well developed
• Inclusion of other important effects, e.g., international effects
• Interpretation of evaluation measures given in a combination of dollars and other units
Jordan and Ruegg 2009 16
SummaryEssential Study Content Characteristics
• Appropriate study design
• Clear account of technologies selected for detailed case study
• Appropriate designation of each next-best alternative
• Assessment of the context and rival explanations of outcome
• Systematic and appropriate data collection and analyses
• Findings are evidence-based, conservative, and study limitations are identified
Mansfield Model of Social Benefits from a Product Innovation that Reduces the Costs of the Industries Using It
Source: Edwin Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program, 1996.
Jordan and Ruegg 2009 20
Definitions
Source: TI A Consulting, I nc.
• Net Benefits: time-adjusted benefits minus costs
NB = ΣBPV – (ΣCPV + ΣIPV)
where ΣBPV = sum of present value benefits; ΣCPV = sum of present value non-investment cost; and ΣIPV = present value investment cost
• Benefit-to-Cost Ratio: time-adjusted benefits (net of time-adjusted non-investment costs) divided by time-adjusted investment cost
B/C = (ΣBPV - ΣCPV) / ΣIPV
• Internal Rate of Return (IRR): the solution interest rate (i) that equates the values of the streams of benefits and costs over time
ΣB(i) = (ΣC(i) + ΣI (i))
Jordan and Ruegg 2009 21
An example (preliminary data) PDC Drill Bit
-- one of a cluster of geothermal technologies
• The nature of the technology: PDC Drill bits have a harder and longer- lasting cutting surface which results in more feet drilled per hour and use of fewer drill bits per hour
• Cluster: Entire EERE Geothermal Program, given historical cost data constraints
• Next best alternative: the existing roller bit technology
• Economic Benefits. PDC drill bits have been adopted by the oil and gas sector (off shore drilling)
– $20+ billion total estimated benefits over the 1997-2007 period
• Attribution: Evidence suggests that DOE Role “very Important” - DOE worked closely with industry to develop and commercialize the technology
