CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 1 Los Angeles,...

1

USC

C S E University of Southern CaliforniaCenter for Software Engineering

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling

18th International Forum on COCOMO and Software Cost ModelingLos Angeles, CAOctober 23, 2003

Ricardo Valerdi – USC Center for Software Engineering

Working Group Meeting

2

USC



COSYSMO Workshop AgendaMorning (8:30 AM – noon)

Brief Introduction on COSYSMO

Updates on action items

Updates on related work

Results from Delphi Round 2

Afternoon (1:00 PM – 5:00 PM)

SE Sizing discussion

Data collection form & NDAs

Synchronization with INCOSE activities

Open issues

Action items

3

USC



COSYSMO Introduction• Parametric model to estimate system

engineering costs• Includes 4 size & 14 cost drivers• Covers full system engineering lifecycle• Developed with USC-CSE Corporate

Affiliate and INCOSE participation

Conceptualize DevelopOper Test & Eval

Transition to Operation

Operate, Maintain, or Enhance

Replace orDismantle

4

USC



COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

+Volatility Factor

- Application factors-8 factors

- Team factors-6 factors

- Schedule driver WBS guided by ISO/IEC 15288

COSYSMO Operational Concept

5

USC



COCOMO II• Software• Development phases• 20+ years old• 200+ calibration points• 23 Drivers• Variable granularity• 3 anchor points• Size is driven by SLOC

COSYSMO• Systems Engineering• Entire Life Cycle• 2 years old• ~3 calibration points• 18 drivers• Fixed granularity• No anchor points• Size is driven by

requirements, I/F, etc

Model Differences

6

USC



Size Drivers vs. Effort Multipliers• Size Drivers: Additive, Incremental

- Impact of adding a new item inversely proportional to current size

10 11 rqts = 10% increase

100 101 rqts = 1% increase• Effort Multipliers: Multiplicative, system-wide

- Impact of adding a new item independent of current size

10 rqts + high security = 40% increase

100 rqts + high security = 40% increase

7

USC



4 Size Drivers1. Number of System Requirements*

2. Number of Major Interfaces

3. Number of Operational Scenarios

4. Number of Critical Algorithms

*Weighted by complexity, volatility, and degree of reuse

8

USC



Number of System RequirementsThis driver represents the number of requirements for the system-of-interest at a specific level of design. Requirements may be functional, performance, feature, or service-oriented in nature depending on the methodology used for specification. They may also be defined by the customer or contractor. System requirements can typically be quantified by counting the number of applicable “shall’s” or “will’s” in the system or marketing specification. Do not include a requirements expansion ratio – only provide a count for the requirements of the system-of-interest as defined by the system or marketing specification.

Easy Nominal Difficult

- Well specified - Loosely specified - Poorly specified

- Traceable to source - Can be traced to source with some effort

- Hard to trace to source

- Little requirements overlap

- Some overlap - High degree of requirements overlap

0.5 1.0 3.0From 2001

9

USC



Number of Major InterfacesThis driver represents the number of shared major physical and logical boundaries between system components or functions (internal interfaces) and those external to the system (external interfaces). These interfaces typically can be quantified by counting the number of external and internal system interfaces among ISO/ISE 15288-defined system elements.


- Well defined - Loosely defined - Ill defined

- Uncoupled - Loosely coupled - Highly coupled

- Strong consensus - Moderate consensus - Low consensus

- Well behaved - Predictable behavior - Poorly behaved

1.0 3.0 7.0From 2001

10

USC



Number of Operational ScenariosThis driver represents the number of operational scenarios that a system must satisfy. Such threads typically result in end-to-end test scenarios that are developed to validate the system and satisfy all of its requirements. The number of scenarios can typically be quantified by counting the number of unique end-to-end tests used to validate the system functionality and performance or by counting the number of use case sequence diagrams developed as part of the operational architecture.


- Well defined - Loosely defined - Ill defined

- Loosely coupled - Moderately coupled - Tightly coupled or many dependencies/conflicting requirements

- Timelines not an issue - Timelines a constraint - Tight timelines through scenario network

14.0 29.0 58.0From 2001

11

USC



Number of Critical AlgorithmsThis driver represents the number of newly defined or significantly altered functions that require unique mathematical algorithms to be derived in order to achieve the system performance requirements. As an example, this could include a complex aircraft tracking algorithm like a Kalman Filter being derived using existing experience as the basis for the all aspect search function. Another example could be a brand new discrimination algorithm being derived to identify friend or foe function in space-based applications. The number can be quantified by counting the number of unique algorithms needed to support each of the mathematical functions specified in the system specification or mode description document.


- Existing algorithms - Some new algorithms - Many new algorithms

- Basic math - Algebraic by nature - Difficult math (calculus)

- Straightforward structure - Nested structure with decision logic

- Recursive in structure with distributed control

- Simple data - Relational data - Persistent data

- Timing not an issue - Timing a constraint - Dynamic, with timing issues

- Library-based solution - Some modeling involved - Simulation and modeling involved

3.0 6.0 15.0From 2001

12

USC



14 Cost Drivers

1. Requirements understanding

2. Architecture understanding

3. Level of service requirements

4. Migration complexity

5. Technology Maturity

6. Documentation Match to Life Cycle Needs

7. # and Diversity of Installations/Platforms

8. # of Recursive Levels in the Design

Application Factors (8)

13

USC



Requirements understanding This cost driver rates the level of understanding of the system requirements by all stakeholders including the systems, software, hardware, customers, team members, users, etc.

Very low Low Nominal High Very High

Poor, unprecedented system

Minimal, many undefined areas

Reasonable, some undefined areas

Strong, few undefined areas

Full understanding of requirements, familiar system

1.4 0.81

EMR = 1.73

14

USC



Architecture understanding This cost driver rates the relative difficulty of determining and managing the system architecture in terms of platforms, standards, components (COTS/GOTS/NDI/new), connectors (protocols), and constraints. This includes tasks like systems analysis, tradeoff analysis, modeling, simulation, case studies, etc.


Full understanding of architecture, familiar system and COTS

Strong understanding of architecture and COTS, few undefined areas

Reasonable understanding of architecture and COTS, some weak areas

Minimal understanding of architecture and COTS, many undefined areas

Poor understanding of architecture and COTS, unprecedented system

2 level WBS 3-4 level WBS 5-6 level WBS >6 level WBS

1.28 0.77

EMR = 1.66

15

USC



Level of service requirementsThis cost driver rates the difficulty and criticality of satisfying the ensemble of level of service requirements, such as security, safety, response time, interoperability, maintainability, the “ilities”, etc.

Viewpoint Very low Low Nominal High Very High

Difficulty Simple Low difficulty, coupling

Moderately complex, coupled

Difficult, coupled KPPs

Very complex, tightly coupled

Criticality Slight inconvenience

Easily recoverable losses

Some loss High financial loss

Risk to human life

0.66 1.65

EMR = 2.50

16

USC



Migration complexity This cost driver rates the complexity of migrating the system from previous system components, databases, workflows, environments, etc., due to new technology introductions, planned upgrades, increased performance, business process reengineering, etc.


Introduction of requirements is transparent

Difficult to upgrade Very difficult to upgrade

1.0 1.50

EMR = 1.50

17

USC



Technology MaturityThe maturity, readiness, and obsolescence of the technology being implemented. Immature or obsolescent technology will require more System Engineering effort.

Viewpoint Very Low Low Nominal High Very High

Maturity Still in the laboratory

Ready for pilot use

Proven on pilot projects and ready to roll-out for production jobs

Proven through actual use and ready for widespread adoption

Technology proven and widely used throughout industry

Readiness Concept defined (TRL 3 & 4)

Proof of concept validated (TRL 5 & 6)

Concept has been demonstrated (TRL 7)

Concept qualified (TRL 8)

Mission proven (TRL 9)

Obsolescence

- Technology is outdated and use should be avoided in new systems- Spare parts supply is scarce

- Technology is stale- New and better technology is on the horizon in the near-term

- Technology is the state-of-the-practice- Emerging technology could compete in future

1.75 0.7

EMR = 2.50

18

USC



Documentation match to life cycle needs The breadth and depth of documentation required to be formally delivered based on the life cycle needs of the system.

Viewpoint Very low Low Nominal High Very High

Breadth General goals

Broad guidance, flexibility is allowed

Streamlined processes, some relaxation

Partially streamlined process, some conformity with occasional relaxation

Rigorous, follows strict customer requirements

Depth Minimal or no specified documentation and review requirements relative to life cycle needs

Relaxed documentation and review requirements relative to life cycle needs

Amount of documentation and reviews in sync and consistent with life cycle needs of the system

High amounts of documentation, more rigorous relative to life cycle needs, some revisions required

Extensive documentation and review requirements relative to life cycle needs, multiple revisions required

0.82 1.23

EMR = 1.50

19

USC



# and diversity of installations/platformsThe number of different platforms that the system will be hosted and installed on. The complexity in the operating environment (space, sea, land, fixed, mobile, portable, information assurance/security). For example, in a wireless network it could be the number of unique installation sites and the number of and types of fixed clients, mobile clients, and servers. Number of platforms being implemented should be added to the number being phased out (dual count).

Viewpoint Nominal High Very High

Sites/installations Small # of installations or many similar installations

Moderate # of installations or some amount of multiple types of installations

Large # of installations with many unique aspects

Operating environment

Not a driving factor Moderate environmental constraints

Multiple complexities/constraints caused by operating environment

Platforms Few types of platforms (< 5) being installed and/or being phased out/replaced

Modest # and types of platforms (5 < P <10) being installed and/or being phased out/replaced

Many types of platforms (> 10) being installed and/or being phased out/replaced

Homogeneous platforms Compatible platforms Heterogeneous, incompatible platforms

Typically networked using a single protocol

Typically networked using several consistent protocols

Typically networked using different protocols

1.0 1.50

EMR = 1.50

20

USC



# of recursive levels in the designThe number of levels of design related to the system-of-interest (as defined by ISO/IEC 15288) and the amount of required SE effort for each level.


Number of levels

1 2 3-5 6-7 >7

Required SE effort

Ad-hoc effort Maintaining system baseline with few planned upgrades

Sustaining SE for the product line, introducing some enhancements of product design features or optimizing performance and/or cost

Maintaining multiple configurations or enhancements with extensive pre-planned product improvements or new requirements, evolving

Maintaining many configurations or enhancements with extensive pre-planned product improvements, new requirements rapidly evolving

0.82 1.23

EMR = 1.50

21

USC



14 Cost Drivers (cont.)

1. Stakeholder team cohesion

2. Personnel/team capability

3. Personnel experience/continuity

4. Process capability

5. Multisite coordination

6. Tool support

Team Factors (6)

22

USC



Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team.


Culture Stakeholders with diverse expertise, task nature, language, culture, infrastructure Highly heterogeneous stakeholder communities

Heterogeneous stakeholder communitySome similarities in language and culture

Shared project culture

Strong team cohesion and project cultureMultiple similarities in language and expertise

Virtually homogeneous stakeholder communitiesInstitutionalized project culture

Communication Diverse organizational objectives

Converging organizational objectives

Common shared organizational objectives

Clear roles & responsibilities

High stakeholder trust level

1.23 0.82

EMR = 1.50

23

USC



Personnel/team capability Basic intellectual capability of a Systems Engineer to analyze complex problems and synthesize solutions.

Very Low Low Nominal High Very High

15th percentile 35th percentile 55th percentile 75th percentile 90th percentile

Personnel experience/continuity The applicability and consistency of the staff at the initial stage of the project with respect to the domain, customer, user, technology, tools, etc.


Experience Less than 2 months 1 year continuous experience, other technical experience in similar job

3 years of continuous experience



Annual Turnover

48% 24% 12% 6% 3%

1.46 0.68

1.42 0.71

EMR = 2.15

EMR = 2.0

24

USC



Process capability Maturity per CMMI or EIA 731.

Very low Low Nominal High Very High Extra High

CMMI Level 1 (lower half)

Level 1 (upper half)

Level 2 Level 3 Level 4 Level 5

EIA731 Performed SE process, activities driven only by immediate contractual or customer requirements, SE focus limited

Managed SE process, activities driven by customer and stakeholder needs in a suitable manner, SE focus is requirements through design

Defined SE process, activities driven by benefit to program, SE focus is through operation

Quantitatively Managed SE process, activities driven by SE benefit, SE focus on all phases of the life cycle

Optimizing SE process, continuous improvement, activities driven by system engineering and organizational benefit, SE focus is product life cycle & strategic applications

1.25 0.85

EMR = 1.60

25

USC



Multisite coordination Location of stakeholders, team members, resources, corporate collaboration barriers.

Viewpoint Very low Low Nominal High Very High Extra High

Collocation International, severe time zone impact

Multi-city and multi-national, considerable time zone impact

Multi-city or multi-company, some time zone effects

Same city or metro area

Same building or complex, some co-located stakeholders or onsite representation

Fully co-located stakeholders

Communications Some phone, mail

Individual phone, FAX

Narrowband e-mail

Wideband electronic communication

Wideband electronic communication, occasional video conference

Interactive multimedia

Corporate collaboration barriers

Severe export and security restrictions

Mild export and security restrictions

Some contractual & Intellectual property constraints

Some collaborative tools & processes in place to facilitate or overcome, mitigate barriers

Widely used and accepted collaborative tools & processes in place to facilitate or overcome, mitigate barriers

Virtual team environment fully supported by interactive, collaborative tools environment

1.34 0.86

EMR = 1.68

26

USC



Tool support Coverage, integration, and maturity of the tools in the Systems Engineering environment.


No SE tools Simple SE tools, little integration

Basic SE tools moderately integrated throughout the systems engineering process

Strong, mature SE tools, moderately integrated with other disciplines

Strong, mature proactive use of SE tools integrated with process, model-based SE and management systems

1.40 0.75

EMR = 1.87

27

USC



Action Items from Keystone (7/03)

1. Develop “short form” version of the Delphi instrument

2. Update application domain categories to include INCOSE and DoD

3. Include more commercially-oriented examples in Delphi form

4. Update Size drivers to reflect data collection effort vs. cost estimation effort

5. Incorporate new direction of IEEE 1220, ISO/IEC 15288,

and EIA/ANSI 632 life cycle stages

6. Follow up with BAE’s SE estimation efforts in New Hampshire

7. Continue work on “Technology Maturity” driver and include NASA/DARPA Technology Readiness Levels

8. Update “# of Critical Algorithms” driver

28

USC



Updates on Related Work

1. COSYSMO for Trade Studies

2. myCOSYSMO tool version 1.15

3. SE Sizing overview paper

29

USC



USC/Raytheon myCOSYSMO*

*Developed by Gary Thomas at Raytheon Garland

30

USC



Delphi Round 2 Participants (36)Paul Mohlman Aerospace

Abe Santiago* Aerospace

Anh Tu Aerospace

Maurie Hartman Boeing

Rob Flowe DAU

Denton Tarbet* Galorath

Gary Hafen LMCO

Rick Edison LMCO

Garry Roedler* LMCO

Carl Newman LMCO

Bob Beckley LMCO

Trish Persson LMCO

George Walther LMCO

Steven Wong Northrop Grumman

Steve Copoloff Northrop Grumman

Joe Kiernicki Northrop Grumman

Ed Tse Northrop Grumman

Bill Slobodian Northrop Grumman

Lori Vaughan Northrop Grumman

Gary Thomas* Raytheon

Randy Case Raytheon

Bob Vojtech Raytheon

John Rieff* Raytheon

Greg Cahill Raytheon

Larry S Kleckner Raytheon

Deke Dunlap* Raytheon

Ron Townsend Raytheon

Stan Parson Raytheon

Tony Jordano* SAIC

Charley Zumba SAIC

Jim Armstrong SPC

Sarah Sheard SPC

Chris Miller SPC

Barry Boehm* USC

Don Reifer* USC

George Friedman USC

*also participated in Round 1

31

USC



Delphi Survey Results

Suggested Rel. Effort

Delphi Respondents EMR

Easy – Nominal – Difficult Rel. Effort

Standard Deviation

# Requirements

0.5 – 1.0 – 2.0 0.6 – 1.5 – 7.3 0.3 – 1.7 – 13.1

# Interfaces 1.0 – 3.0 – 7.0 1.4 – 3.8 – 8.3 1.1 – 2.0 – 3.5

# of Ops Scen14.0 – 29.0 – 58.0

9.0 – 21.8 – 46.1

5.9 – 13.8 – 30.5

# Algorithms 3.0 – 6.0 – 15.0 2.6 – 5.3 – 14.9 1.72 – 2.4 – 6.7

Ratings for Size Drivers

32

USC



Delphi Round 2 Highlights (cont.)Range of sensitivity for nominal Size Drivers

# A

lgo

rith

ms

# R

eq

uir

emen

ts

# In

terf

aces

# S

cen

ario

s

5.3

Relative

Effort

1.5

3.8

21.8

12

8

4

Rel. Effort

0.6 – 1.5 – 7.3

1.4 – 3.8 – 8.3

9.0 – 21.8 – 46.1

2.6 – 5.3 – 14.9

16

20

# Requirements

# Interfaces

# of Ops Scen

# Algorithms

33

USC




Suggested EMR Delphi Respondents EMR

EMRStandard Deviation

Reqs Und 1.73 3.35 3.38

Arch Und 1.66 3.03 3.35

Level of Serv Req

2.50 6.61 17.83

Migration Complx

1.50 1.84 1.17

Tech Maturity 2.50 2.70 0.76

Documentation 1.50 2.26 3.37

# Install/Platforms

1.50 2.19 1.71

# levels in design

1.50 3.05 5.71

Ratings for Cost Drivers (Application Factors)

34

USC




Suggested Rel. Effort

Delphi Respondents EMR

EMRStandard Deviation

Stakeholder Coh

1.50 2.79 4.40

Pers/team Cap 2.15 3.09 3.47

Pers exp/cont 2.0 2.95 3.19

Process cap 1.6 1.79 0.33

Multisite Coord 1.68 1.99 1.51

Tool Support 1.87 1.87 0.47

Ratings for Cost Drivers (Team Factors)

35

USC



Delphi Round 2 Highlights (cont.)Range of sensitivity for Cost Drivers (Application Factors)

EMR 2.70

6.6

3.033.353.05

3

Req

uir

em

ents

un

d.

# o

f le

vels

in

des

ign

Lev

el o

f se

rvic

e re

qs.

Tec

hn

olo

gy

Mat

uri

ty

Arc

h.

un

der

sta

nd

ing

# o

f In

stal

l/p

latf

orm

s

2.262.19

6

Do

cum

enta

tio

n

Mig

rati

on

co

mp

lexi

ty

1.84

36

USC



Delphi Round 2 Highlights (cont.)Range of sensitivity for Cost Drivers (Team Factors)

EMR 1.87

3.09

1.992.95

2.75

Per

s ex

per

ien

ce/c

on

t

Sta

keh

old

er c

oh

esio

n

Per

s/te

am c

apab

ilit

y

To

ol

sup

po

rt

Mu

ltis

ite

coo

rdin

ati

on

1.79

3P

roce

ss c

apab

ilit

y

37

USC



# of Req

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

1

38

USC



# of IF

0.00

5.00

10.00

15.00

20.00

25.00

1

39

USC



# of Op

0.00

20.00

40.00

60.00

80.00

100.00

120.00

1

40

USC



# of Alg

0.00

5.00

10.00

15.00

20.00

25.00

30.00

1

41

USC



Systems Engineering Sizing# Requirements System Level (too high): The system shall provide notification of out-of- tolerance inputs and outputs to authorized parties. File Level (about right): Each system component has one or more files of parameters to monitor, report exceptions, and adjust tolerances in a familiar way. Parameter Level (too low): The system shall determine that the temperature T at point P, is between T11 and T12, and report exceptions to the safety monitor.

# Interfaces For hardware interfaces, use Number and Complexity of External Interface files. For user interfaces, use Number and Complexity of Input Files, Output Files, External Queries.

42

USC



Questions or Comments?Dr. Barry Boehm

[email protected]

Ricardo Valerdi

[email protected]

Website

http://www.valerdi.com/cosysmo

Date post:	04-Jan-2016
Category:	Documents
Upload:	lewis-james
View:	212 times
Download:	0 times

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 1 Los Angeles,...

Documents