Date post: | 31-Dec-2015 |
Category: |
Documents |
Upload: | joelle-lancaster |
View: | 39 times |
Download: | 0 times |
1
University of Southern California
Center for Systems and Software Engineering
November 2010Mauricio Peña
Dr. Ricardo Valerdi
CHARACTERIZING THE IMPACT OF
REQUIREMENTS VOLATILITY ON
SYSTEMS ENGINEERING EFFORT
2
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Agenda
• Overview
• Introduction and Background
• Implications to COSYSMO
• Method
• Causal Model
• Survey Results
• Conclusions
• Next Steps
3
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Motivation
“Requirements are the foundation of the project. They form the basis for design, manufacture, test and operations….changes in requirements later in the development cycle can have a significant cost impact, possibly resulting in cancellation”
INCOSE Systems Engineering Handbook (2006)
4
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Overview of Research Findings• Field research validated findings from prior studies:
– Requirements volatility is linked to an increase in rework and project size
– The impact of changing a requirement increases the later the change occurs in the system lifecycle
• The research provided additional insights:
1. Causal model linking volatility to a number of technical, organizational and contextual factors
2. The level of volatility is a function of lifecycle phase
3. Respondents from S/W intensive projects tend to expect more volatility than those who work on H/W intensive systems
4. There are spikes in volatility after the transitions between lifecycle phases
5. Requirements changes early in the lifecycle may not be considered “volatility”
5
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
0
100
200
300
400
500
F M A M J J A S O N
NewModified
Deleted
Time (Months)
# of
Req
uire
men
ts
Requirements Volatility Definitions
Requirements volatility is the % change in requirements (added, deleted, and modified) over a given time interval
Also known as:
Requirements creep: An increase in scope and/or number of system requirements
Requirements churn: Instability in the requirements set – requirements are frequently modified or reworked without necessarily resulting in an increase in the total number of requirements
Costello, R. and Liu, D. (1995). “Metrics for Requirements Engineering,” Journal of Systems and Software. Vol 29 (No. 1), pp. 39-63MIL-STD-498. 1994. Software Development and Documentation. U.S. DoD
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
J F M A M J J A S O N
Actual Volatility
Expected Volatility
Volatility Metrics (Monthly)
% o
f to
tal R
equi
rem
ents
Cha
ngin
g
6
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
SE Leading Indicators Guide
Leading Indicators are defined as “measures for evaluating the effectiveness of the systems engineering activities on a program in a manner that provides information about impacts that are likely to affect the system or program performance objectives.”
Rhodes, D., Valerdi, R., and Roedler, G. (2009). “Systems engineering leading indicators for assessing program and technical effectiveness.” Systems Engineering Vol. 12 (No. 1), pp 21-35.
7
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Requirements Trends as a Systems Engineering Leading Indicator
• Evaluates trends in the growth and change of the system requirements
• It helps to determine the stability and completeness of the system requirements which could potentially impact project performance
Systems Engineering Leading Indicators Guide, Version 2.0, 2010
8
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Implications to COSYSMO
• During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers
• However, there was insufficient data to incorporate volatility effects into the initial version of the model
• The primary objective of the research is to complete the requirements volatility extension to COSYSMO within the existing structure and scope of the model
Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.
9
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
COSYSMO Volatility Factor
COSYSMO
SizeDrivers
EffortMultipliers
Effort
Calibration
# Requirements# Interfaces# Scenarios# Algorithms
- Application factors-8 factors
- Team factors-6 factors
ReuseCategories
Volatility Factor
COSYSMO
SizeDrivers
EffortMultipliers
Effort
Calibration
# Requirements# Interfaces# Scenarios# Algorithms
- Application factors-8 factors
- Team factors-6 factors
ReuseCategories
Volatility Factor
COSYSMO
SizeDrivers
EffortMultipliers
Effort
Calibration
# Requirements# Interfaces# Scenarios# Algorithms
- Application factors-8 factors
- Team factors-6 factors
ReuseCategories
Volatility Factor
Fortune, J. (2009). Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO 2.0). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.
10
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Method
8USC-CSE Annual Research Review – 3/17/03
Analyze Existingliterature
1
2
3
4
5
6
7
PerformBehavioral Analysis
Identify RelativeSignificance
Perform Expert-Judgement, DelphiAssessment
Gather Project Data
Determine BayesianA-Posteriori Update
Gather more data;refine model
A-PRIORI MODEL+
SAMPLING DATA=
A-POSTERIORI MODEL
First Phase of the Study:• Review of relevant literature
• Data collected through field research: surveys and discussions conducted at industry/academic conferences and workshops
Boehm, B. (1981). Software Engineering Economics. Prentice Hall.
11
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Literature Background
• Most of the requirements volatility research to date has been focused on software systems
• Various research methods have been utilized to investigate the causes and effects of requirement volatility, including:
5
23
21
Simulation Model
Survey / Model
Survey
Project data analysis
Interviews
There is still a lack of empirical data on the quantitative impact of requirements volatility on for a broader base of engineering projects
12
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Cost Commitment on Projects
Detail Designand
Development
100
25
50
75
Conceptual-Preliminary
Design
Constructionand/or
Production
System Use, Phaseout,and Disposal
NEED
% Commitment to Technology,Configuration, Performance, Cost, etc.
Cost Incurred
System-Specific Knowledge
Ease of Change
Blanchard and Fabrycky (1998), Systems Engineering & Analysis, Prentice Hall, 1998
Changes to the System are more difficult to implement the later they occur in the lifecycle
13
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Causal Model (normative)
Based on the review of the literature, a causal model was developed that relates technical, organizational and contextual project factors to requirements volatility
Survey results were used to rank the level of subject-matter expert agreement with each of the postulated causes of requirements volatility.
Changes in org. structure /
leadership / process
Requirements volatility +/-
+
+/-
+/-Rework
+/-
-
SE Productivity
Number of Sys
Requirements
Project Schedule
SE Project Effort
SE CostProject Cost
Quality
Customer Satisfaction
Customer-Driven Scope Changes
SE Process Maturity
Technology Maturity
+/- (Valerdi 2005)
(Valerdi 2005)+
+
-
(Honour, 2004)(Ferreira, et al 2009)
(Valerdi 2005)(Honour, 2004)
(Houston, 2000)(Ferreira 2002)
(Charette et al,2003)
+/-
+
(Charette et al,2003)
(Houston, 2000)Zowghi et al. (2000)(Ferreira, et al 2009)
(Houston, 2000)(Ferreira 2002)(Charette et al,2003)
+/-(Valerdi 2005)
(Valerdi 2005)
+/-
Poor Understanding of
the System & Customer needs
Experienced staff
Contextual / Environmental
changes
1 2
Changes in co-dependent
Systems
3 5
76
Changes in COTS
Products
8 9
4
(Jones 1994)
+
(Jones 1994) (Kotonya and Sommerville 1998)
+
(2010 USC ARR Survey)
+
(Ferreira 2002)
-
(GAO, 2004)(Jones 1994)
-
(Curtis et al. 1988)Zowghi et al. (2000)(Kotonya and Sommerville 1998)
+-
(Ferreira, 2002)(GAO, 2004)
+
(Kotonya and Sommerville 1998)
+
(2010 USC ARR Survey)
Changes in org. structure /
leadership / process
Requirements volatility +/-
+
+/-
+/-Rework
+/-
-
SE Productivity
Number of Sys
Requirements
Project Schedule
SE Project Effort
SE CostProject Cost
Quality
Customer Satisfaction
Customer-Driven Scope Changes
SE Process Maturity
SE Process Maturity
Technology Maturity
Technology Maturity
+/- (Valerdi 2005)
(Valerdi 2005)+
+
-
(Honour, 2004)(Ferreira, et al 2009)
(Valerdi 2005)(Honour, 2004)
(Houston, 2000)(Ferreira 2002)
(Charette et al,2003)
+/-
+
(Charette et al,2003)
(Houston, 2000)Zowghi et al. (2000)(Ferreira, et al 2009)
(Houston, 2000)(Ferreira 2002)(Charette et al,2003)
+/-(Valerdi 2005)
(Valerdi 2005)
+/-
Poor Understanding of
the System & Customer needs
Poor Understanding of
the System & Customer needs
Experienced staff
Contextual / Environmental
changes
Contextual / Environmental
changes
1 2
Changes in co-dependent
Systems
3 5
76
Changes in COTS
Products
Changes in COTS
Products
8 9
4
(Jones 1994)
+
(Jones 1994) (Kotonya and Sommerville 1998)
+
(2010 USC ARR Survey)
+
(Ferreira 2002)
-
(GAO, 2004)(Jones 1994)
-
(Curtis et al. 1988)Zowghi et al. (2000)(Kotonya and Sommerville 1998)
+-
(Ferreira, 2002)(GAO, 2004)
+
(Kotonya and Sommerville 1998)
+
(2010 USC ARR Survey)
14
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Exploratory Survey
• Developed to gather the perspectives of subject-matter experts on the causes, impacts, and expected level of requirements volatility for a given system of interest
• Piloted at the 2010 USC-CSSE Annual Research Review
• Incorporated feedback and administered the survey at the 2010 Lean Advancement Initiative (LAI) knowledge exchange event in Dana Point, CA
• Organizations represented: – The Aerospace Corporation, Northrop Grumman Corporation
– The Boeing Company, Softstar Systems, Raytheon
– United Launch Alliance, Massachusetts Institute of Technology, University of Southern California, and
– United States Army
– United States Navy
15
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
>20%
>20%
>20% >20%
10-20%
10-20%
10-20%
10-20%
5-10%
5-10%
5-10%
5-10%
<5%<5%
<5%
<5%
don't knowdon't know
don't know don't know
0
5
10
15
20
Conceptualize Development Operational Test &Evaluation
Transition toOperation
Expected Requirements Volatility (%)
Num
ber
of R
espo
nden
ts
Expected Level of Volatility
Most respondents expect >20% volatility during the conceptualize phase of the project, decreasing to <5% in the transition to operation phase
16
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Impact of Hardware/Software Project Breakdown on Expected Volatility
Operational Test & Evaluation Lifecycle Phase Transition to Operation Lifecycle Phase
0%
20%
40%
60%
80%
100%
<5% 5-10% 10-20% >20%
Expected Requirements Volatility (%)
75% Hardware, 25% Software 50% Hardware, 50% Software
25% Hardware, 75% Software
% o
f R
espo
nden
ts p
er C
ateg
ory
0%
20%
40%
60%
80%
100%
<5% 5-10% 10-20% >20%
Expected Requirements Volatility (%)75% Hardware, 25% Software 50% Hardware, 50% Software
25% Hardware, 75% Software
% o
f R
espo
nden
ts p
er C
ateg
ory
Respondents from S/W intensive projects tend to expect more volatility later in the lifecycle
17
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
0%
20%
40%
60%
80%
100%
Re-work of work products Project size (Total # ofrequirements)
% o
f re
spon
dent
s
Impacts of Volatility
• In general, results of the survey support observations from the literature and causal model
• Most respondents stated that requirements volatility will cause a moderate to large increase in the number of system requirements and rework
Moderate decrease Moderate Increase
No impact Large Increase
18
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Survey Exercise
• Survey Exercise administered during the 2010 Practical Software and Systems Measurement Conference
• Participants were asked to:1. Draw a requirements volatility profile across the lifecycle
phases covered by COSYSMO
2. Draw an “ease of change” profile across the same life cycle phases to determine the volatility weighting factor
3. Discuss variation in 1 and 2 above for:
1. Large and Small Projects
2. Hardware and Software Projects
3. Development and Recurring Projects
19
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Expected Requirements Volatility Profile
Conceptualize Development Operational Test & Evaluation
Transition to Operation
% o
f R
equ
irem
ents
, A
dde
d, D
elet
ed o
r M
odi
fied
30%
15%
4 out of 9 participants indicated that requirements changes should not be considered volatility during the conceptualize phase
Localized peaks in volatility due to the transitions between lifecycle phases
Profile Representative of Participant Feedback
No significant differences between type of projects
20
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Ease of Change Profile
Conceptualize Development Operational Test & Evaluation
Transition to Operation
Eas
e of
Cha
nge
1
0.5
0.1
0
0.2
0.3
0.4
0.9
0.6
0.7
0.8
Representative of Participant Feedback
Conceptualize Development Operational Test & Evaluation
Transition to Operation
Eas
e of
Cha
nge
1
0.5
0.1
0
0.2
0.3
0.4
0.9
0.6
0.7
0.8
Representative of Participant Feedback
Cost Penalty defined as 1 / ease of change
Average Ease of Change Factor (Estimated)
Average Cost Penalty (Estimated)
0.8 0.4 0.1 0.05
1.25 2.5 10 20
21
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Conclusions
• Field research validated the literature findings that:
– Link volatility to an increase in rework and project size
– Predict a cost penalty due to late requirements changes
• Additional insights developed through the research:
1. Causal model linking volatility to a number of technical, organizational and contextual factors
2. The level of volatility is a function of lifecycle phase
3. The presence of localized peaks in requirements volatility after the transitions between lifecycle phases
4. Feedback that suggests requirements changes during the conceptualize phase should not be labeled as volatility
5. Respondents from S/W intensive projects tend to expect more volatility later in the lifecycle than those who work on H/W oriented systems
22
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Next Steps
• The findings from the field research will be used to further define the volatility extension to COSYSMO
– Additional work is required to understand the cost penalty of late requirements as it relates to systems engineering effort
– The point in the lifecycle during which volatility starts to be measured and accounted for also needs to be further defined
– Interviews with industry experts and mini-case studies will be conducted to validate the usefulness of the causal model
• Industry data will be collected to quantify the impact of requirements changes on systems engineering effort
23
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
References• Boehm, B. (1991). Software Risk Management: Principles and Practices. IEEE Software 8(1), pp 32-41.• Blanchard, B. and Fabrycky, W. (1998). Systems Engineering & Analysis, Prentice Hall, New York.• Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements
volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp 1568-1577.
• Fortune, J. (2009). Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO 2.0). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.
• General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software-intensive Weapon Acquisitions (GAO-04-393). Defense Acquisitions
• Honour, E. (2004). “Understanding the Value of Systems Engineering.” INCOSE 14th Annual International Symposium: Systems Engineering Managing Complexity and Change. Toulouse, France
• Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University
• INCOSE Systems Engineering Handbook, Version 3, INCOSE, June 2006• ISO/IEC (2008). ISO/IEC 15288:2008 (E) Systems Engineering - System Life Cycle Processes.• Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and
Sons, Ltd.• MIL-STD-498 (1994). Software Development and Documentation. U.S. Department of Defense.• Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts
Institute of Technology, INCOSE, and PSM.• Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation.
University of Southern California, Industrial and Systems Engineering Department.• Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project
Performance. Proceedings of the Ninth Asia-Pacific Software Engineering Conference
24
University of Southern California 25th Annual COCOMO Forum
Center for Systems and Software Engineering
Call for Participation
• In order to complete the requirements volatility extension of COSYSMO, we are seeking industry data for engineering projects in terms of:
– Systems engineering effort actuals (labor hours)
– Requirements volatility: the number of requirements, added, deleted, and modified added after the requirements baseline
• By providing these data your organization will benefit by:– Improving its ability to estimate the impact of requirements changes on
project cost
– Calibrating and tailoring the updated Model for your application domain
• USC-CSSE and LAI at MIT have proven processes in place to ensure the confidentiality and protection of the data with its Corporate Affiliates and Consortium Members
• Contact:Mauricio E. Pena [[email protected]]
Ricardo Valerdi [[email protected]]