Post on 30-Sep-2015
description
transcript
Rick Holcomb AIR-4.2 Cost Department
(301) 342-2450 holcombra@navair.navy.mil
Dan GalorathGalorath Incorporated
(310) 414-3222 dgalorath@galorath.com
30 January, 2003 36th ADoDCAS1
NAVAIR SW Estimation Improvement Program
30 January, 2003 36th ADoDCAS 2
Agenda
Whats Necessary for Accurate Software Estimation?
NAVAIR Initiatives Galoraths Phase II Implementation of
SBIR N01-020
30 January, 2003 36th ADoDCAS 3
Whats Necessary for Accurate SW
Estimation?
30 January, 2003 36th ADoDCAS 4
What is Accuracy?
How close is the initial estimate to the final estimate?
How has functionality changed from the initial technical baseline to the final product? What impact has this had on the estimate?
30 January, 2003 36th ADoDCAS 5
Teamwork & Discipline are Required
Improvement in SW Estimation requires participation by both NAVAIR & its Industry Partners, not just the Cost Department.
Requires mature, disciplined software processes for all phases of the life-cycle for both the Acquisition and Development organizations to produce an accurate complete Technical & Programmatic Baseline.
the major factor in successful software development is disciplined execution. Report of the Defense Science Board Taskforce on Defense Software, November 2000.
30 January, 2003 36th ADoDCAS 6
A Mature Disciplined Development Environment is Important
Generally accurate software estimates can only be produced when high quality data about the development organization(s) is available.
High quality software data can only be generated by mature, disciplined Acquisition and Development organizations.
Even if data is available for an immature, undisciplined organization, attempting to estimate for this organization is problematic.
Having good data on an immature undisciplined organization is anoxymoron.Once data starts to become available on an organization, it indicates at least the start of an effort to increase the maturity and discipline of the software development process.
30 January, 2003 36th ADoDCAS 7
SW-CMM Summary
INITIAL
REPEATABLE
DEFINED
MANAGED
OPTIMIZING
Ad hoc; occasionally chaotic
Disciplined
Standard, consistent process
Predictable process
Continuouslyimproving process
1
2
3
4
5LOW
R I
S K
R I
S K
R I
S K
HIGHHIGH
Levels of Process Maturity
30 January, 2003 36th ADoDCAS 8
Probability of Success
(1) Capers Jones, Becoming Best In Class, Software Productivity Research, 1995 briefing(2) 1 Function Point = 50 SLOC ADA, 40 SLOC C++, 320 SLOC Assembly
0%
20%
40%
60%
80%
100%
120%
0 1 10 100 1,000 10,000 100,000
SW Size (Function Points)
Prod
uct S
ucce
ss
SEI 1 & 2 SuccessSEI 3,4, & 5 Success
30 January, 2003 36th ADoDCAS 9
With & Without Accurate Data
.
0 %
140%
-140%
...
.
.
..
..
..
..
. .
. . . .
.. . .
. .
.
.
. . . .. . . . . . . .... . . .. ..
. .. ..
.
.
.
. .. .. ...... . .. . ... . .. . .. ..
Without Historical Data With Historical DataVariance between + 20% to - 145% Variance between - 20% to + 20%
(Efforts = Labor Hours)
(Mostly Level 1 & 2) (Level 3)
Ove
r/Und
er P
erce
ntag
e
.
(Based on 120 projects in Boeing Information Systems)
.. . .
.
.
. .
..
.
. .
..
..
. .
..
.. .. . .. . . . . .. . . . . .. .
... . .. . .
. . . .. . . .. . . . .. . . . .
. . . . .. . . . . .. . . . .. . . . . .
. . . . . .. . . . . . . .
. . .. . . . .
. . . . .. . . .
. . . . . .. . . . . .
. . . . . .
John Vu, Boeing, keynote talk at SEPG 97, Software Process Improvement Journey (From Level 1 to Level 5)
30 January, 2003 36th ADoDCAS 10
NAVAIR Initiatives
30 January, 2003 36th ADoDCAS 11
Process Improvement
Developers of ACAT I, II, III, & IV Software Intensive Systems will achieve a SEI SW-CMM Level 3 or equivalent.
NAVAIRINST 5234.1 Policy on Software Evaluations for Naval Air Systems Command Programs
Achieve a corporate SW-CMM or CMMI Level III by 2005, and Level V by 2008.
NAVAIRINST 5234.2 Requirements for Process Improvement Actions for Naval Air Systems Command Software Acquisition, Development and Life-Cycle Support
Implement Corporate Requirements for Software Metrics & Measurement Collection.
NAVAIRINST 5234.5 Naval Air Systems Command Metrics for Software Intensive Systems
30 January, 2003 36th ADoDCAS 12
Software Resource Center
The NAVAIR Software Resource Center provides expert training & assistance to programs involved in the development of software intensive systems in areas such as:
SW-CMM & CMMITeam Software Process (TSP) / Personnel Software Process (PSP)Practical Software & Systems Measurement (PSM)Independent Expert Program Reviews (IEPR)Software EstimationSoftware Engineering Acquisition, Management & DevelopmentLessons Learned
30 January, 2003 36th ADoDCAS 13
Communications
The best results are achieved when all levels of the organization agree with the need for change rather then having it forced upon them.
A major aspect of NAVAIRs initiatives is the training and communications with the workforce.
30 January, 2003 36th ADoDCAS 14
Progress
F/A-18 Weapons Software Support Activity (WSSA), SW-CMM Level V.
Longest history of process improvement in NAVAIR. Prime contractor, Boeing, is also a Level IV. Recent developments have been very successful at meeting cost and schedule objectives.One of Top 5 DoD Software projects in 2001, http://www.stsc.hill.af.mil/top5projects/
AV8B WSSA, SW-SMM Level IV and TSP/PSP adopted.AV-8Bs Experience Using the TSP to Accelerate SW-CMM Adoption, Crosstalk Sep 2002, http://www.stsc.hill.af.mil/crosstalk/2002/09/hefley.html
1 SW-CMM Level III Group 3 SW-CMM Level II Groups 10 Groups adopting TSP/PSP 18 Groups started SW-CMM Process Improvement Efforts 8 Groups investigating CMMI
30 January, 2003 36th ADoDCAS 15
Improved Tools
SBIR N01-020, Software Cost & Schedule Estimation:
Demonstrate processes and tools that will estimate software cost and schedule with a standard deviation of 10% or less. Make best possible use of limited data sample sets for developing software cost and schedule estimates.Enhance update of estimates based on actual project data.Enhance collection and archival of project software metrics and measurements in order to support calibration and improve accuracy for both current and future projects
30 January, 2003 36th ADoDCAS 16
Galoraths Phase II Implementation of
SBIR N01-020
30 January, 2003 36th ADoDCAS 17
Project Plan
ID Task Name Finish
1 ProjectMiner Mon 4/19/042 ProjectMiner Release 1 Thu 7/3/03
106 ProjectMiner Release 1B Tue 9/23/03160 SEER-SEM Integration Tue 10/21/03161 ProjectMiner Release 2 Mon 1/26/04246 ProjectMiner Release 3 Wed 2/18/04273 ProjectMiner Release 4 Tue 3/23/04300 Transition - non-iterated tasks Mon 4/19/04
2/10 4/14 6/16 8/18 10/20 12/22 2/23 4/27 6/29 8/31 11/2 1/4 3/7anuary 21 June 1 October 11 February 21 July 1 Nov ember 11 March
30 January, 2003 36th ADoDCAS 18
Data Rights
Government has standard SBIR data rights for application.
Can view all requirements, design and source code of product.After 5 years, Government may go to another source to modify application, IF:
Galorath Incorporated has not copyrighted the application; however, it probably will.Government has not contracted with Galorath for further modifications of the application.
30 January, 2003 36th ADoDCAS 19
Plan to Meet NAVAIR Goals
SizingSizing made easier, and more possible earlier in the estimation process
EstimatingDecision makers will better understand the origins of an estimateEstimate will be more accurate, more reliable, and better integrated into the project life cycleImpact of new technologies and specific circumstances will be captured
PlanningEstimate will be integrated into earned value and other project management methodsAn integrated estimating methodology will be supported
Data WarehousingPast project acutals will be sorted and accessed for more accurate, more defensible estimates
30 January, 2003 36th ADoDCAS 20
Accurate Estimating RequiresAttention to Sizing
COTS entrySizing based on characteristics, a completely different approach to scope.
Size by ComparisonHighly accurate method of sizing new pieces of software against known past items.
AnalogiesSize new jobs against standardized benchmarks.
COTS Estimating By Characteristic
+ Characteristic Rating - Rigor of requirements Very High - Number of interfaces 70-100 - Complexity Very High - Application Domain Aero - Application Type Signal processing - >
COTS Estimating By Characteristic
COTS Estimating By Characteristic
Characteristic Rating
Specify COTS Characteristic
Rigor of requirements High
OK Cancel
Next >>
Help
30 January, 2003 36th ADoDCAS 21
Detailed Sizing Workups
Rather than one or two major entries, many separate sizing sub-entries can be made within any particular program being estimated. A wide range of sizing methods can be combined or used alongside each other.
Result Estimates can be top-down (general)OR bottom-up (detailed).
COTS Estimating By Characteristic+ SIZE Metric Quantity Legacy - Component A SLOC 1700 Pre-existing - Component B - part of A SLOC 120 Pre-existing - Component C ELOC 450 Pre-existing - Component D Classes 12... New - Component E SLOC 60 New - Component F - part of A Use Cases 2... Pre-existing - Component G COTS Sizing select for info COTS - Component H SLOC 15000 Pre-existing - Component I SLOC 300 Pre-existing - Component J - part of A Function Based select for info Pre-existing - Component K Detailed Objects 45... Pre-existing - Component L SLOC 1200 Pre-existing
Sizing View
30 January, 2003 36th ADoDCAS 22
SEER NexGens ProjectMinerPreliminary Proven Performance
Within 12%
Within10%
Within 20%
These are actual results on data that would be used in a production system. All data and tests are available for independent review.Please note that final results should be better, as the ProjectMiner methodology is still being prototyped.*
Using extremelylimited, relevant
past data.Results obtained using 5 aviation data points and
Least Absolute Error estimation.
Using extremely limited, relevant
past data, with good assumptions about
predicted project.Results obtained using
maximum information pair-wise comparisons, with accurate
conjecture on probable range.Using sufficientrelevant past data.
Results obtained using neural network.
With a modest amountof NAVAIR data.
Using a preliminary version of ProjectMiner, mixing Least Absolute Error, Weighted
Least Squares and Maximum Likelihood .
With NO relevant past data.Results regularly obtained using SEER-SEM,
in the future also using ProjectMinerconsensus estimates coupled with the
Development Factors model.
30 January, 2003 36th ADoDCAS 23
ProjectMiner Concept Many Approaches Make a Better
Estimate
Functional Forms
Estim
atin
g M
etho
ds
Data
Set
BestModels,Methods& Data
SettingsFor
CurrentEstimate
CombinedResults of
VariousStatisticalMethods
CombinedResults of
VariousStatisticalMethods
y=a*xb, etc
Multiplestatisticalmethods,
etc
NAVAIR,contractors,
industry,etc
30 January, 2003 36th ADoDCAS 24
ProjectMiner How It Works
Unlimited numberof core models
PastProject Data
DetermineRelevant
Data
DevelopPotentiallyRelevant
Equations
Apply Data andEquations To
EstimatingMethods
Adjust BestEstimate With
Empirically DerivedAdd-On Models
AssessPerformance
Of EachApproach
IntelligentlyCombine
BestApproaches
Unlimitednumber of add-
on models
30 January, 2003 36th ADoDCAS 25
Key Leveraging Technologies: Voting and Conformal Estimating
Voting finds the most accurate estimating
approaches.
Voting finds the most accurate estimating
approaches.
Conformal estimating keeps unique info about past projects and then uses that to improve
future estimates.
Conformal estimating keeps unique info about past projects and then uses that to improve
future estimates.oth
ercom
pariso
n
regres
sion
NAVAIR data
Contractor data
Other data
Combines the best methods, paying particular attention to:
The most relevant data sources.
Embedded intelligence about performance of specific methods and models against
specific data.
StandardInfo:
Size, Platform,Application, etc.
RESULT:A much more accurate estimate.
Invaluable additional info
UniqueInfo:
Specific Tools,Circumstances, etc.
30 January, 2003 36th ADoDCAS 26
Resolving Concerns Overa Data-Driven Approach
We dont know the sizeSize by comparison against 10-100s of similar projects an accurate, proven method.A large number of sizing methods offers many solutions to project scoping.
Theres not enough past dataNumerous estimating methods (least absolute error, etc) are specialized for small samples.
The data is very noisyModels & methods are carefully combined to produce a more robust estimate.Weighted least squares and other techniques help.
We dont have any dataBut the model does: several thousand, many from NAVAIRs own contractors.
Nothing may be relevantEstimating methods dont work that way they can find relevance even in different sorts of data.SEER parametric modeling is the backup for when nothing else works!
30 January, 2003 36th ADoDCAS 27Potential variation in duration
Design
Prototype
Spec
Code
Unit Test
______________________________________________________________________________________Month 3 6 9 12 15 18 21 24Midcourse Calibration
Quick EstimateCalibration Dialog
Activity Duration Chart
Midcourse Calibration
Phase duration estimates are
updated on the chart, with risk ranges (in red)
Dialog lets estimator input earned value
information for auto updating MS
Project
30 January, 2003 36th ADoDCAS 28
wInsight Earned Value Will Integrate With NexGen
30 January, 2003 36th ADoDCAS 29
Custom Lifecycles:Data-Driven or User-Defined
Lifecycle DesignerLifecycle Designer
Activities and labor allocations can be formulaic, fixed, or automatically determined by ProjectMiner.
Example - add FQT or systems integration.
30 January, 2003 36th ADoDCAS 30
Accurate Scheduling Feature
A set of estimated WBS elements
Determine which tasks should follow which, how critical the relations are
Determine which tasks should follow which, how critical the relations are
A set of properly scheduled elements
Automatic optimal scheduling
30 January, 2003 36th ADoDCAS 31
Software Estimation & Project Management
MS Project
S/W Estimation
MS Project w/ Software Estimation
30 January, 2003 36th ADoDCAS 32
Estimation and Transformation of Microsoft Project
Saves time by automatically laying out highly tailored plans in Microsoft Project
Automatically connects with estimation tools to obtain sophisticated software project resource & schedule estimates
Encourages best practices and uniform estimating by connecting project planners with enterprise information
Perfectly complements Project 2002s enterprise and usability focus
Arrow in the Quiverfor ProcessImprovement
30 January, 2003 36th ADoDCAS 33
Data-Driven Estimates Against Past Projects
..\..\..\business\01-016 NAVAIR SBIR N01-020 Software Estimating\Deliverables and Briefings\August 28th Briefing\scatterplot_run.avi
30 January, 2003 36th ADoDCAS 34
Data Collection Activities
NAVAIR is soliciting additional software data Will benefit the entire community To participate contact:
Rick Holcomb AIR-4.2 Cost Department
(301) 342-2450 holcombra@navair.navy.mil