Improving Survey Cost Estimates: Lessons Learned Applying GAO Best Practices at the US Census Bureau Josue De La Rosa U.S. Census Bureau International Field Directors and Technologies Conference May 17th-20th, 2015 Fort Lauderdale, Florida 1 Disclaimer: This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the Census Bureau.
Transcript
1
Improving Survey Cost Estimates: Lessons Learned Applying GAO Best Practices at the US
Census BureauJosue De La Rosa
U.S. Census Bureau
International Field Directors and Technologies Conference
May 17th-20th, 2015Fort Lauderdale, Florida
Disclaimer: This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the Census Bureau.
2
Objectives Introduce the GAO Best Practice Cost
Estimation Guide Twelve Steps of a High-Quality Cost Estimating
Process Lessons Learned Applied the Best Practices
3
Cost Estimation Is“Summation of individual cost elements, using
established methods and valid data, to estimate the future costs of a program, based on what is
known today.”
4
Why Is Cost Estimation Important
Formulating and explaining budgets Set priorities
Cost/ Benefit Analysis Evaluations
Mitigate the risk of: Budget variances Schedule slippage Scope and quality underperformances
5
“Suggested” Best Practices Guide is a suggestion, but used by GAO in
auditing/ evaluating programs
“quality estimating process resulted in unreliable cost estimates throughout each program’s life cycle.”
“Instead of using a cost model, it used an unsophisticated spreadsheet to extrapolate”
6
The Guide Authored by the
Government Accountability Office’s Applied Research and Methods
420 Pages, Published March 2009
7
Structure of Guide 12 Steps of a High-
Quality Cost Estimating Process Initiation & Research Assessment Analysis Presentation
20 Chapters 17 Best Practice Checklist 48 Case Studies
8
Define estimate’s purpose (1) Who needs the estimate and how will the
estimate be used? What level of detail is needed? When is the estimate due?
9
Develop estimating plan (2) Experience and trained
members Subject Matter Experts
IT Survey Methods FLD
Timeline and schedule
10
Define program characteristics (3)
Establish scope of work Technical Requirements Predecessors and successors Quantities and frequency Development, test, training and production
Personnel Risk
11
Determine estimating structure (4)
Thorough Work Break Down Structure = Good Best Practice
Results or product based WBS, not activity WBS matches scope of program WBS feeds schedule, Earned Value
Management metrics and cost estimate
12
Identify ground rules and assumptions (5)
In-scope/ out of scope Start and end date Identify any schedule or budget constraints,
inflation assumptions, and travel/ travel costs Determine technology refresh cycles Effects of new rules and regulations Risk associated with assumptiosn
13
Obtain data (6) Data collection plan
technical, schedule, cost, and risk data;
Adjust data for trends, inflation, learning, and quantity adjustments cost drivers;
Determine data reliability and accuracy;
Store data for future estimates
14
Develop point estimate (7) Estimate the cost of each WBS section
Analogy (A is like B) Engineering Build-Up Parametric (Cost = Quantity x Effort) Extrapolating Expert Opinion (Not recommend)
Sensitivity analysis = tested effect of changes to cost Key assumptions and factors on cost Schedule and quantities changes
16
Conduct risk and uncertainty analysis (9)
Determine for each risk: Level of cost, schedule & technical impacts Likelihood and impact of risk Min, max & likely range Monte Carlo simulation Risk adjusted cost (reserves for risk response) Risk management plan
17
Document the estimate (10) Purpose of estimate Review of previous steps Detail methodology
Team, date, approvals Schedule, technical baseline Data sources Estimating methods and justification Risk, uncertainty, sensitivity Compare to previous estimates
18
Present estimate to management for approval (11)
Develop a briefing that presents the documented life-cycle cost estimate;
Include an explanation of the baseline and any uncertainties;
Compare estimate to the budget; Focus on largest cost elements and cost drivers; Make the content clear and complete Act on and document feedback from management; Request acceptance of the estimate
19
Update the estimate to reflect actual costs and changes (12)
Update the estimate to reflect Replace estimates with EVM Report progress on meeting cost and schedule
estimates; Perform a post mortem and document lessons
learned for variance; Document all changes to the program and
how they affect the cost estimate
20
Lessons Learned Cost Estimation Professionals
Project management experience Business Analysis Survey methodology experience Organizational Maturity
Establishing a cost estimation plan Iterations (Agile)
21
Lessons Learned (continued) Importance of reliable data
Eg Census Planning Database http://www.census.gov/research/data/planning_database/
Treat IT systems as IT systems Relationship between Total Survey Error and cost drivers
