ROI/JWS.PPT 04/21/23 1

Industry PerspectivesIndustry Perspectivesforfor

DoD SW Engineering S&TDoD SW Engineering S&T

Jim SturgesJim SturgesDirector, Engineering ProcessesDirector, Engineering Processes

Lockheed Martin CorporationLockheed Martin Corporation

ROI/JWS.PPT 04/21/23 2

Importance of AccuracyImportance of Accuracy

ROI/JWS.PPT 04/21/23 3

““The significant problems we The significant problems we face cannot be solved at the face cannot be solved at the same level of thinking where we same level of thinking where we were when we created them.”were when we created them.”

Albert EinsteinAlbert Einstein

ROI/JWS.PPT 04/21/23 4

Industry’s top needs from S&TIndustry’s top needs from S&T - Integrating Complex Systems - Integrating Complex Systems

• Management Development ResearchManagement Development Research

• Supported component librariesSupported component libraries

• Supported simulation frameworksSupported simulation frameworks

ROI/JWS.PPT 04/21/23 5

Management DevelopmentManagement Development

• DoD Program Managers must understand that Software Engineering is DoD Program Managers must understand that Software Engineering is one discipline among many that contribute to products, and one discipline among many that contribute to products, and does not does not stand alonestand alone

– ““Great Programmers will perfectly encode rotten requirements.” Great Programmers will perfectly encode rotten requirements.” – Bill Ballhaus, Chairman, The Aerospace Corporation– Bill Ballhaus, Chairman, The Aerospace Corporation

• Acquisition CyclesAcquisition Cycles– ““The problem with the acquisition process is that by the time the The problem with the acquisition process is that by the time the

people at the top are ready for the answer, the people at the bottom people at the top are ready for the answer, the people at the bottom have forgotten the question.” – Norm Augustine, have forgotten the question.” – Norm Augustine, Augustine’s LawsAugustine’s Laws

• Acquisition Maturity ModelAcquisition Maturity Model– ““Talking about quantitative management with this customer is laying Talking about quantitative management with this customer is laying

pearls before swine.” – Anonymous software engineerpearls before swine.” – Anonymous software engineer

ROI/JWS.PPT 04/21/23 6

All Programs - CPI

0.800

0.850

0.900

0.950

1.000

1.050

1.100

1.150

1.200

0 2 4 6 8 10 12 14

Period

Co

st P

erfo

rman

ce In

dex

One point of CPI is worth $270 MillionOne point of CPI is worth $270 Million

Level 4 (SW & SE) Company HighlightedLevel 4 (SW & SE) Company Highlighted

ROI/JWS.PPT 04/21/23 7

If you do what If you do what you always did, you always did, the way you the way you always did it….always did it….

ROI/JWS.PPT 04/21/23 8

Quality – Barry BoehmQuality – Barry Boehm

The cost to fix a defect found during operation phase may be as high The cost to fix a defect found during operation phase may be as high as 1000 times* the cost to fix it during requirements phaseas 1000 times* the cost to fix it during requirements phase

The cost to fix a defect found during operation phase may be as high The cost to fix a defect found during operation phase may be as high as 1000 times* the cost to fix it during requirements phaseas 1000 times* the cost to fix it during requirements phase

* Reference: Barry W. Boehm, Software Engineering Economics (Englewood Cliffs, NJ: Prentice-Hall)

0

100

200

300

400

500

600

Av

era

ge

Co

st

Ra

tio

Requirements Design Coding DevelopmentTesting

AcceptanceTesting

Operation

Phase in Which Found

Relative Cost to Fix an Error *

1 3-6 10 15-4030-70

40-1000

ROI/JWS.PPT 04/21/23 9

Quality -- DSMCC

um

ula

tive

Per

cen

tag

e L

ife

Cyc

le C

ost

Cu

mu

lati

ve P

erce

nta

ge

Lif

e C

ycle

Co

st

100%100%

90%90%

80%80%

70%70%

60%60%

50%50%

40%40%

30%30%

20%20%

10%10%

0%0%

ConceptConceptPhasePhase

DesignDesignPhasePhase

Develop-Develop-mentment

Prod/TestProd/TestPhasePhase

OperationsOperationsThroughThroughDisposalDisposal

8%8% 15%15% 20%20%

100%100%

Committed CostsCommitted Costs

70%70%

85%85%95%95%

3-6X3-6X

20-100X20-100X

500-1000X500-1000X

TimeTime

Full Program ExpendituresFull Program Expenditures Defense SystemsDefense SystemsManagement College - 9/1993Management College - 9/1993

50%50%

Cost to Extra

ct Defects

Cost to Extra

ct Defects

ROI/JWS.PPT 04/21/23 10

Productivity Variation - LMISProductivity Variation - LMIS

0

5

10

15

20

25

30

35

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8

Productivity Distribution

Per

cen

t

May-96

Mar-98

Aug-98

Jan-00

ROI/JWS.PPT 04/21/23 11

Supported Component LibrariesSupported Component Libraries

• Reuse offers dramatic productivity/quality benefitsReuse offers dramatic productivity/quality benefits• Design-for-reuse adds 25% to development costsDesign-for-reuse adds 25% to development costs

– Who will pay?Who will pay?– As long as PEOs control all the budgets, nobody As long as PEOs control all the budgets, nobody

will be incentivized to make this happenwill be incentivized to make this happen• NASA’s predecessor cataloged every possible airfoil NASA’s predecessor cataloged every possible airfoil

configuration, and airplane designers still use them configuration, and airplane designers still use them todaytoday– Why aren’t DoD-specific [or highly related] Why aren’t DoD-specific [or highly related]

software components deserving of the same software components deserving of the same attention?attention?

ROI/JWS.PPT 04/21/23 12

Supported Simulation Supported Simulation FrameworksFrameworks

• Simulation Based Acquisition Simulation Based Acquisition makes a lot of sensemakes a lot of sense– But each contractor has its own approachBut each contractor has its own approach– And each Service/PEO has its own approachAnd each Service/PEO has its own approach– No common approachNo common approach– No ongoing supportNo ongoing support

• Industry strongly supported a Joint SBA activity, but Industry strongly supported a Joint SBA activity, but DoD could never make it happenDoD could never make it happen

• Could S&T activity define and produce an SBA Could S&T activity define and produce an SBA framework that everyone can use?framework that everyone can use?

ROI/JWS.PPT 04/21/23 13

SummarySummary• It’s bad form to begin a presentation with an It’s bad form to begin a presentation with an

apology, so I’ll end with oneapology, so I’ll end with one– None of these three priorities involves rocket None of these three priorities involves rocket

science, and none will win a Nobel Prize. So I’m science, and none will win a Nobel Prize. So I’m sorry I didn’t show you the need for an agent-sorry I didn’t show you the need for an agent-based, web-enabled node-traverser.based, web-enabled node-traverser.

• But any of those three priorities, when But any of those three priorities, when successfully executed, would provide a successfully executed, would provide a significant major benefit to all of our stakeholderssignificant major benefit to all of our stakeholders

• Let’s invalidate Augustine’s 27Let’s invalidate Augustine’s 27thth Law: Law:– ““Software is like entropy. It is difficult to grasp, Software is like entropy. It is difficult to grasp,

weighs nothing, and obeys the Second Law of weighs nothing, and obeys the Second Law of Thermodynamics: it always increases.”Thermodynamics: it always increases.”