SHRINKING THE CONE OF UNCERTAINTY WITH CONTINUOUS
ASSESSMENT FOR SOFTWARE TEAM DYNAMICS IN DESIGN AND
DEVELOPMENT
by
Pongtip Aroonvatanaporn
A Dissertation Presented to theFACULTY OF THE USC GRADUATE SCHOOLUNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of theRequirements for the Degree
DOCTOR OF PHILOSOPHY(COMPUTER SCIENCE)
August 2012
Copyright 2012 Pongtip Aroonvatanaporn
Acknowledgments
The journey through my research and the Ph.D. program would not have been
possible without the support from those around me.
First, I would like to thank my advisor, Dr. Barry Boehm, for everything that
he has done for me in guiding me through the research journey, and for always
being there when I needed help. He has been a great advisor and was the main
inspiration for me to pursue the Ph.D. degree.
Secondly, I would like to thank my dissertation committee members, Dr. Nenad
Medvidovic and Dr. Stan Settles, for the additional guidance and suggestions in
completing my dissertation. Also, I would like to extend my thanks to the professors
that were part of the Ph.D. qualifying exam committee, Dr. William GJ Halfond
and Dr. Aiichiro Nakano, for comments and feedback on my topic proposal.
My family - Papa, Mama, J’Na, P’Jo, Top, and Alysha - has given in a great
deal of support throughout my journey. All the moral and physical supports they
have given me were far more than I could ever ask for from anyone.
Endless thanks also go to my girlfriend, Praew, for her patience and understand-
ing throughout my entire journey. She had to put up with a lot during the long
ii
time it took for me to finish my degree. I can gladly say that the wait is now over
and it has certainly paid off!
I have had the great opportunity to work with many outstanding researchers at
the USC Center for Systems and Software Engineering, all of whom I am glad to call
my friends. My life as a Ph.D. student would not have been as enjoyable without
these people: Ivo Krka, Reza Safi, Jae Young Bang, Dr. Vu Nguyen, Nupul Kukreja,
Thomas Tan, Qi Li, and Julie Sanchez. Everyone has been supportive of me and I
will certainly miss all of them. I would like to give special mention to Dr. Supannika
Koolmanojwong and Dr. Jo Ann Lane for their numerous suggestions, comments,
and discussions about my research ideas. Their contributions have greatly impacted
my research direction and results, and I am very grateful.
Special thanks also go to Lizsl De Leon for her help with all the paperwork
regarding the Ph.D. degree. Because of her, many potential headaches were avoided.
Additional thanks also go to her and Geoffrey Spedding for hosting such fun “Ph.D.
Social” nights at their place. They had certainly made being a Ph.D. student more
enjoyable.
I would like to extend the credit of this dissertation to a few people that impacted
the research. Parts of the research idea and framework was inspired by Chatchai
Sinthop. Thanida Hongsongkiat helped significantly with the data analysis. Many
thanks go to her for going through thousands and thousands of records of data.
Sergio Romulo Salazar and Bhargav Rajagopalan contributed their effort in helping
with the tool development.
iii
My research idea and work could not have started without the work at IBM on
the Self-Check project, which contributed a big part to this dissertation. Dr. Peter
Haumer has been a great project manager, and he has always been supportive of
what I do. I would like also to thank Dr. Monvorath Phongphaibul, as my work at
IBM would not have begun without her recommendation to the Self-Check team.
Finally, my research and writing my dissertation would not have been possible
without the USC Software Engineering course. The past 4 years of being a teaching
assistant have taught me a lot and given me invaluable experience. I will certainly
miss the course, students, clients, instructors, and all who were involved. Thanks
to all of them for the data and the willingness to participate.
iv
Table of Contents
Acknowledgments ii
List of Tables viii
List of Figures ix
Abstract xii
Chapter 1: Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Terms and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 The Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.1 Imprecise Project Scoping with Inaccurate Estimations . . . 61.3.2 Lack of Effective Tracking and Assessment Tools . . . . . . . 71.3.3 Limitations in Software Cost Estimation Models . . . . . . . 91.3.4 The Optimism Phenomenon . . . . . . . . . . . . . . . . . . 9
1.4 Research Questions and Hypotheses . . . . . . . . . . . . . . . . . . 101.5 Research Contributions . . . . . . . . . . . . . . . . . . . . . . . . . 111.6 Organization of Dissertation . . . . . . . . . . . . . . . . . . . . . . 13
Chapter 2: Background and Related Work 142.1 The Incremental Commitment Model . . . . . . . . . . . . . . . . . 142.2 COCOMO II Model . . . . . . . . . . . . . . . . . . . . . . . . . . 162.3 USC Software Engineering Course . . . . . . . . . . . . . . . . . . . 182.4 IBM Self-Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.5 Continuous Assessment . . . . . . . . . . . . . . . . . . . . . . . . . 232.6 Early Detection of SE-Related Program Risks . . . . . . . . . . . . 242.7 Architecture Review Board . . . . . . . . . . . . . . . . . . . . . . . 252.8 Software Sizing and Estimation . . . . . . . . . . . . . . . . . . . . 272.9 Project Tracking and Assessment . . . . . . . . . . . . . . . . . . . 29
Chapter 3: Research Methodology 313.1 Overview of Research Design . . . . . . . . . . . . . . . . . . . . . . 31
v
3.2 Research Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.3 Obtaining the Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Chapter 4: The COTIPMO Framework 384.1 Project Progress Tracking Framework . . . . . . . . . . . . . . . . . 40
4.1.1 Tracking Based on Source Code . . . . . . . . . . . . . . . . 414.1.2 Tracking Based on Application Points . . . . . . . . . . . . . 47
4.2 Continuous Team Assessment Framework . . . . . . . . . . . . . . . 494.2.1 Questions from Team Assessment Data . . . . . . . . . . . . 514.2.2 SE Performance Risk and SE Competency Risk Frameworks 534.2.3 Refining the Questions . . . . . . . . . . . . . . . . . . . . . 54
4.3 Project Estimation Adjustment Framework . . . . . . . . . . . . . . 554.3.1 Correlation to Team’s Performance . . . . . . . . . . . . . . 574.3.2 Expert Judgment . . . . . . . . . . . . . . . . . . . . . . . . 614.3.3 COCOMO II Adjustment Calculation . . . . . . . . . . . . . 634.3.4 Assessment Frequency . . . . . . . . . . . . . . . . . . . . . 66
Chapter 5: The COTIPMO Tool 685.1 Powered by Jazz . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685.2 Support for Traditional Development Projects . . . . . . . . . . . . 695.3 Support for Rapid-Fielding Projects . . . . . . . . . . . . . . . . . . 755.4 Survey Answer Guidance . . . . . . . . . . . . . . . . . . . . . . . . 785.5 COTIPMO Tool Administration . . . . . . . . . . . . . . . . . . . . 795.6 Question Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . 805.7 Project Management . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Chapter 6: Evaluation and Analysis 836.1 Improved Software Estimation Accuracies . . . . . . . . . . . . . . 83
6.1.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 836.1.2 Hypothesis Support: H1, H2b . . . . . . . . . . . . . . . . . 85
6.2 Improved Project Issues and Defects Detection . . . . . . . . . . . . 866.2.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876.2.2 Hypothesis Support: H2a . . . . . . . . . . . . . . . . . . . 89
6.3 Improved Project Tracking . . . . . . . . . . . . . . . . . . . . . . . 906.3.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906.3.2 Hypothesis Support: H1, H3a, H3b . . . . . . . . . . . . . . 91
6.4 Improved Team Synchronization and Stabilization . . . . . . . . . . 926.4.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 926.4.2 Hypothesis Support: H2a, H3a . . . . . . . . . . . . . . . . 96
6.5 Reduced Project Effort . . . . . . . . . . . . . . . . . . . . . . . . . 966.5.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 976.5.2 Hypothesis Support: H3a . . . . . . . . . . . . . . . . . . . 102
vi
6.6 Improved Client Satisfaction . . . . . . . . . . . . . . . . . . . . . . 1036.6.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046.6.2 Hypothesis Support: H3b . . . . . . . . . . . . . . . . . . . 105
6.7 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066.8 Reducing the Threat to Validity . . . . . . . . . . . . . . . . . . . . 108
6.8.1 Average Project Effort Based on Experience Level . . . . . . 1106.8.2 Project Issues and Defects Based on Experience Level . . . . 1126.8.3 Experiment with Project from SDSU . . . . . . . . . . . . . 1136.8.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Chapter 7: Conclusions and Future Work 1177.1 General Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 1177.2 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . 1217.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Bibliography 125
Appendix A: Questions with COCOMO II Impact Factor 130
Appendix B: Question Set - Weekly ICSM Set 154
Appendix C: Question Set - Milestone ICSM Set 158
Appendix D: Question Set - General Set 164
Appendix E: Original Effort Categories 168
Appendix F: Re-categorized Effort Categories 172
Appendix G: Client Evaluation Form - One-Semester Projects 176
Appendix H: Client Evaluation Form - Two-Semester Projects 178
Appendix I: Project Status Survey 181
Appendix J: Multiple Linear Regression Analysis 183
Appendix K: Fit Y by X Test 184
Appendix L: Application Point Estimation 190
Appendix M: Student Background Questionnaire 191
vii
List of Tables
1.1 COCOMO II Productivity Range . . . . . . . . . . . . . . . . . . . 5
2.1 Goals, critical success factors, and questions of Early Detection ofSE-Related Program Risks framework . . . . . . . . . . . . . . . . . 25
3.1 Number of traditional development and rapid-fielding projects persemester . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.1 Team issues reported by project teams by semester . . . . . . . . . 524.2 Categorized questions . . . . . . . . . . . . . . . . . . . . . . . . . . 544.3 Coefficient of determination (R2) for COCOMO II parameters . . . 604.4 Weights applied to level of experience with COCOMO II . . . . . . 614.5 COCOMO II ratings and corresponding values . . . . . . . . . . . . 63
5.1 Application Point drivers vs. cost drivers . . . . . . . . . . . . . . . 78
6.1 Project team industry experience per semester . . . . . . . . . . . . 109
viii
List of Figures
1.1 The Standish Chaos Report . . . . . . . . . . . . . . . . . . . . . . 11.2 The Cone of Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1 Overview of the Incremental Commitment Spiral Model life cycleprocess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 COCOMO II and the Cone of Uncertainty . . . . . . . . . . . . . . 172.3 Self-Check Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.4 Architecture evidence shortfall penalties and resulting architecting
investment sweet spots . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1 Research Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.1 Workflow of the COTIPMO framework . . . . . . . . . . . . . . . . 394.2 Workflow for traditional development projects . . . . . . . . . . . . 414.3 Simulation results - overall progress . . . . . . . . . . . . . . . . . . 454.4 Simulation results - estimation error . . . . . . . . . . . . . . . . . . 464.5 Workflow for rapid-fielding projects . . . . . . . . . . . . . . . . . . 474.6 Process for Analyzing Team Assessment Data . . . . . . . . . . . . 514.7 Count of problems with team performance . . . . . . . . . . . . . . 584.8 Example of the expert survey . . . . . . . . . . . . . . . . . . . . . 62
5.1 The iteration list for traditional development projects . . . . . . . . 705.2 Survey ballot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725.3 Survey result - answers . . . . . . . . . . . . . . . . . . . . . . . . . 725.4 List of identified actions. . . . . . . . . . . . . . . . . . . . . . . . . 735.5 Survey Result - COCOMO II scale factors adjustment suggestions . 745.6 Survey Result - COCOMO II cost drivers adjustment suggestions . 745.7 The estimation list for rapid-fielding projects . . . . . . . . . . . . . 765.8 Survey Result - Developer’s capability and experience level and ICASE
maturity and experience level adjustment suggestions . . . . . . . . 775.9 Survey answer guidance . . . . . . . . . . . . . . . . . . . . . . . . 795.10 Question editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 805.11 Question set editor . . . . . . . . . . . . . . . . . . . . . . . . . . . 815.12 Project management page . . . . . . . . . . . . . . . . . . . . . . . 82
ix
6.1 Average number of mistakes in the COCOMO II ratings for scalefactors and cost drivers. . . . . . . . . . . . . . . . . . . . . . . . . 84
6.2 Average defects and issues filed by the team each year . . . . . . . . 876.3 Average defects and issues by phase - 2 semester projects . . . . . . 886.4 Average defects and issues by phase - 1 semester projects . . . . . . 896.5 The level of uncertainties within project . . . . . . . . . . . . . . . 936.6 The level of synchronization among team members . . . . . . . . . 936.7 Team’s strength . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 946.8 Difficulties in communication . . . . . . . . . . . . . . . . . . . . . 956.9 The average effort by week . . . . . . . . . . . . . . . . . . . . . . . 986.10 The average effort by activities . . . . . . . . . . . . . . . . . . . . 996.11 Effort reduction in risk identification and mitigation . . . . . . . . . 1006.12 Effort reduction in team synchronization . . . . . . . . . . . . . . . 1006.13 Effort reduction in project issue and problem resolution . . . . . . . 1016.14 Average effort per person . . . . . . . . . . . . . . . . . . . . . . . . 1026.15 Client evaluation results - Fall semester . . . . . . . . . . . . . . . . 1046.16 Client evaluation results - Spring semester . . . . . . . . . . . . . . 1056.17 Average weekly effort for projects with less than 1 year of industry
experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1106.18 Average weekly effort for projects with 1 to 3 years of industry expe-
rience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1116.19 Average weekly effort for projects with more than 3 years of industry
experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1116.20 Average project issues and defects by industry experience . . . . . . 1126.21 Survey results from SDSU . . . . . . . . . . . . . . . . . . . . . . . 1146.22 Reported effort reduction by SDSU students in risk mitigation, project
issue resolution, and team synchronization . . . . . . . . . . . . . . 115
7.1 A narrower Cone of Uncertainty . . . . . . . . . . . . . . . . . . . . 1207.2 The second Cone of Uncertainty . . . . . . . . . . . . . . . . . . . . 124
I.1 Project status survey - page 1 . . . . . . . . . . . . . . . . . . . . . 181I.2 Project status survey - page 2 . . . . . . . . . . . . . . . . . . . . . 182
J.1 Project status survey - page 2 . . . . . . . . . . . . . . . . . . . . . 183
K.1 Fit Y by X - PREC & FLEX . . . . . . . . . . . . . . . . . . . . . 184K.2 Fit Y by X - RESL & TEAM . . . . . . . . . . . . . . . . . . . . . 185K.3 Fit Y by X - PMAT & RELY . . . . . . . . . . . . . . . . . . . . . 185K.4 Fit Y by X - DATA & CPLX . . . . . . . . . . . . . . . . . . . . . 186K.5 Fit Y by X - RUSE & DOCU . . . . . . . . . . . . . . . . . . . . . 186K.6 Fit Y by X - TIME & STOR . . . . . . . . . . . . . . . . . . . . . 187K.7 Fit Y by X - PVOL & ACAP . . . . . . . . . . . . . . . . . . . . . 187
x
K.8 Fit Y by X - PCAP & PCON . . . . . . . . . . . . . . . . . . . . . 188K.9 Fit Y by X - APEX & PLEX . . . . . . . . . . . . . . . . . . . . . 188K.10 Fit Y by X - LTEX & TOOL . . . . . . . . . . . . . . . . . . . . . 189K.11 Fit Y by X - SITE & SCED . . . . . . . . . . . . . . . . . . . . . . 189
L.1 Application Point Estimations Procedure . . . . . . . . . . . . . . . 190
M.1 Student Background Questionnaire - Page 1 . . . . . . . . . . . . . 191M.2 Student Background Questionnaire - Page 2 . . . . . . . . . . . . . 192M.3 Student Background Questionnaire - Page 3 . . . . . . . . . . . . . 193M.4 Student Background Questionnaire - Page 4 . . . . . . . . . . . . . 194M.5 Student Background Questionnaire - Page 5 . . . . . . . . . . . . . 195
xi
Abstract
Team synchronization and stabilization are essential especially for large software
projects. Unfortunately, little is usually done to assess and reduce the uncertainties
and knowledge gaps that exist within the project. As the project progresses through
its life cycle, the team can gain more information about the project and the team’s
capabilities. These necessary data can be obtained through performing assessments
on the team and the project. However, these assessment procedures are often com-
plex, discouraging, and difficult to analyze; an effective framework and tool support
can greatly enhance the process. Hence, with improved assessment methods, soft-
ware project teams can quickly gather the necessary data, determine the actions
needed to improve performance, and achieve an improved project outcome in the
end.
Furthermore, uncertainties in projects can significantly impact the cost and
schedule estimations, which determine the scopes of the projects and the resources
required to complete them. However, developing accurate and realistic estimates
requires a high level of experience, expertise, and historical data. Oftentimes, once
the resources have been estimated, little is done to reduce the uncertainties in the
estimations as the project progresses through its life cycle. With improved software
xii
tracking and estimation mechanisms, teams can develop more accurate estimations,
allowing the goals of the project outcome to be monitored and achieved with the
available resources.
This research investigates the appropriateness of current project tracking and
team assessment methodologies, and develops a new assessment framework to effec-
tively improve team synchronization and stabilization as well as project effort
tracking and estimation. The COnstructive Team Improvement Process MOdel
(COTIPMO) is an assessment framework developed to effectively achieve this by
enabling software development teams to quickly track their project progress, con-
tinuously assess the team’s performance, and make adjustments to the project esti-
mates as necessary, thus shrinking the cone of uncertainty for software projects.
xiii
Chapter 1: Introduction
1.1 Motivation
Good team performance and accurate software cost and schedule estimations are
essential in determining the quality and timely delivery of the final product. The
2009 Standish Report reported that out of the 9000 projects surveyed, 32% were
delivered with full capability within budget and schedule, 24% were cancelled, and
44% were either over budget, over schedule, or under-delivered [Sta09]. These num-
bers are also consistent with the two previous reports in 2004 [Sta04] and 2006
[Sta06] as shown in Figure 1.1. This figure shows that nearly half the projects
were unsuccessful due to issues related to cost and schedule estimations, and that
software development projects have been consistent in replicating these shortfalls.
15%
19%
24%
51%
46%
44%
34%
35%
32%
2004
2006
2009
Standish Chaos Report Failed Challenged Succeeded
Figure 1.1: The Standish Chaos Report
1
The main motivation behind this research is derived from the well-known soft-
ware “Cone of Uncertainty” defined in [Boe81] and calibrated to completed projects
in [BAB+00]. As shown in Figure 1.2, the Cone of Uncertainty shows that until the
product is delivered, there exists a range of product sizes and costs that the project
can result in. Essentially, the wider the Cone of Uncertainty is for a project, the
more difficult it is for the project to determine a feasible product scope and staffing
level to ensure a timely delivery.
Size (SLOC)+ Cost ($)
+
+
++++
+
+
++
+++
4x
2x
1.5x
1.25x
x
0.25x
0.5x
Relative Size Range
CompletedPrograms
USAF/ESDProposals
Feasibility PlansandRqts.
ProductDesign
DetailDesign
Devel.andTest
Concept ofOperation
Rqts.Spec.
ProductDesignSpec.
DetailDesignSpec.
AcceptedSoftware
Phases and Milestones
Figure 1.2: The Cone of Uncertainty
With more accurate estimations and less uncertainty within the team, the num-
ber of failed or over budgeted projects could be reduced significantly. Yet produc-
ing accurate and realistic estimates requires high levels of expertise and experience
2
as well as good historical data. For highly precedented projects and experienced
teams, one can often use “yesterday’s weather” estimates of projects of comparable
sizes and historical productivity data to produce fairly accurate estimates of project
effort. More generally, though, the range of uncertainty in effort estimation is ini-
tially much larger, and decreases with accumulated problem and solution knowledge
within a Cone of Uncertainty. This is a challenge that unprecedented projects and
unfamiliar software development teams often face, as these data and resources are
not always readily available.
The focus of this research is to narrow the Cone of Uncertainty from the prod-
uct design stage to the end of the development life cycle. There are many other
uncontrollable factors that impact the level of uncertainties in a project prior to
the product design phase. The early stages of the project life cycle can be consid-
ered an “open trade space,” where the project can proceed in any direction due to
uncertainties in technologies and requirements.
1.2 Terms and Definitions
This section describes the terms and definitions used in this report and the scope
of the research.
Unfamiliar project. For the rest of this report, the definition of “unfamiliar”
projects, or teams, means they either have little knowledge or little experience
with the type of project being developed. This includes the following:
• Inexperienced in general
3
• Experienced, but in a new domain
• Experienced, but using new technologies
Continuous assessment. Continuous assessment is a type of assessment method-
ology that takes place over a period of time. In the software development con-
text, the process is done parallel to the development, instead of being done
at major milestones or reviews.
Traditional development project. A type of project in which the product must
be developed from scratch. The development team must write the majority
of the source code to implement the end user functionalities.
Rapid-fielding project. A type of project that aims at integrating and/or tai-
loring either one, or a set of, nondevelopmental items (NDI) or commercial
off-the-shelf (COTS) products. As defined in [KB10], this is when at least 30%
of the end user features and capabilities are provided by the NDI or COTS
products.
Team synchronization. The increase in the level of consistency among the team
members with respect to their awareness of each other’s understandings,
knowledge, experience, and capabilities. The focus is on how well the team
members work and coordinate with each other.
Team stabilization. The reduction in the magnitude of uncertainties that exist
within the team and project. The focus is on the number of unknowns that
could potentially prevent the team from performing effectively.
4
1.3 The Problems
When software development teams lack the proper data and experience, they cannot
accurately assess project size and team capabilities. These unknowns and uncer-
tainties can typically be reduced with proper assessments as the project progresses.
Unfortunately, team assessments are often overlooked, even though personnel uncer-
tainties often have significant influence on the Cone of Uncertainty. Table 1.1 shows
the productivity range of the COCOMO II parameters representing their magni-
tudes of impact on estimations and schedules. It is clear that human factors have
the most significant impact; therefore, synchronization and stabilization within the
development team is essential.
Table 1.1: COCOMO II Productivity RangeCategory Parameter Prod. Range Total
Product
RELY 1.54
10.36DATA 1.42CPLX 2.38RUSE 1.31DOCU 1.52
PlatformTIME 1.63
3.55STOR 1.46PVOL 1.49
Personnel
ACAP 2.00
16.07
PCAP 1.76PCON 1.51APEX 1.51PLEX 1.40LTEX 1.43
ProjectTOOL 1.50
3.28SITE 1.53SCED 1.43
5
In certain development paradigms where cost or schedule is fixed, the project
scopes must be adjusted to compensate for any changes in environment, resources, or
feature priorities in the project. These paradigms include the Schedule As An Inde-
pendent Variable (SAIV) and Cost As An Independent Variable (CAIV) [BPHB02].
Because uncertainties cannot be avoided, projects must be able to adapt to these
changing environments. Otherwise, schedules can slip, costs and budgets may over-
run, and product qualities may suffer significantly.
1.3.1 Imprecise Project Scoping with Inaccurate Estima-
tions
Without the proper data and experience, software development teams usually gen-
erate inaccurate estimates of the effort required for the product to be developed.
As a result, the teams are required to renegotiate with the clients to ensure that the
product to be developed is within the scope achievable by the development team.
Without the necessary data, it is nearly impossible for teams to make proper pre-
dictions with respect to project scope, complexity, and resources required. These
data include aspects and attributes that are specified in the COCOMO II model
in [BAB+00]. Typically, projects progress through their life cycles based on these
inaccurate estimates. This means that regardless of how well or poorly the projects
progress, the estimates remain constant.
When projects begin with the initial overestimation of resources or effort
required, the teams must negotiate with the clients to reduce the size of the projects.
6
This often results in clients needing to throw away some of the critical core capa-
bilities of the product, thus losing some of the expected benefits they had hoped
for from the completed project. The reality is that the team may actually have
enough resources to deliver all of the initial requirements prior to the re-scoping of
the project.
On the other hand, when projects underestimate the resources, the teams tend
to over promise the goals that the project can achieve. As the project progresses
towards the end of its life cycle, the team may start to realize that the remainder
of the project is more than they can manage to complete. When this happens, one
scenario is that they try to satisfy the client by attempting to complete the project
as quickly as possible, while the quality of the project may suffer greatly from this
attempt and result in higher long-term maintenance costs. Another scenario is that
they end up delivering a project that is not complete, thus leaving the clients with
unusable or unsustainable products.
1.3.2 Lack of Effective Tracking and Assessment Tools
To date, there have been no tools or data that effectively monitor the evolution
of the project’s progression in the Cone of Uncertainty. The tasks of manually
assessing the project’s progress are tedious and discouraging to the team because of
the amount of effort required and the complexity of the assessment process. In order
to collect enough information to have useful assessment data, the team often needs
to perform various surveys and reviews to determine how well the team performed
in the previous iterations [KKR08]. The data gathered from the assessment process
7
must be analyzed and evaluated by experts in order to develop meaningful results
and conclusions. The team may end up waiting for multiple project iterations before
the evaluation results can be used, thus making those results less effective.
In processes with high maturity level ratings such as CMMI levels 4 and 5,
development teams already must constantly go through various quantitative and
qualitative assessment tasks to ensure a high level of quality and performance. These
procedures can take a significant amount of time and effort to perform effectively.
The team may end up spending more effort on the assessment itself than on the
development of the project.
Furthermore, in traditional processes, to accurately report the progress of soft-
ware development projects, the teams are usually required to carefully count the
source lines of code (SLOC) developed, analyze the logical lines of code, and com-
pare them to the potentially inaccurate estimates discussed in Section 1.3.1. These
tasks require a significant amount of effort to perform manually; thus, the processes
are discouraging to the team. The more discouraging the processes are, the less
they get done effectively.
Without the proper tools or mechanisms to assess the team’s and the project’s
status and performance, the project would progress through its life cycle with a high
level of uncertainties. As the Cone of Uncertainty remains wide, the project esti-
mates also remain uncertain and inaccurate, while the project performance remains
unimproved.
8
1.3.3 Limitations in Software Cost Estimation Models
There is little that software cost estimation models can compensate for when soft-
ware projects lack the necessary information and knowledge at the time when cost
and schedule are estimated. Moreover, most estimation models require thorough
understanding of the model parameters as well as a certain level of expertise in
order to use them effectively. Without the proper understanding, teams may poten-
tially end up overstating the capability of the team’s personnel, or understating the
complexities of the project. These misrepresentations lead to inaccurate and non-
realistic estimations as discussed in Section 1.3.1.
Additionally, software projects are prone to changes in requirements, designs,
and infrastructure as they progress in the life cycle; therefore, the uncertainties in
resource estimations are constantly changing. This may be more apparent in agile
projects or when clients are constantly updating and changing the requirement
specifications. Software estimation models cannot automatically adapt to these
varying environments.
1.3.4 The Optimism Phenomenon
Finally, software estimators often run into the conspiracy of the “optimism” phe-
nomenon, where both clients and developers are overly optimistic about the project
and the team’s capabilities. This often results in estimates being at the bottom of
the curve in the Cone of Uncertainty, until the project progresses and discovers that
the levels of complexities and uncertainties are higher than expected. This often
results in major project overruns or failures, as indicated in the Standish reports.
9
Development teams may even discover that the actual size range lies on the top
curve of the cone, and end up needing to throw away core features or sacrificing
quality and time in order to finish the project.
Furthermore, from the business point of view, competitive bidders tend to be
overly optimistic in their estimations. Team capabilities are misrepresented in
project proposals, causing wide gaps between what the business customers want
versus what the team can deliver. This again introduces problems discussed in
Section 1.3.1.
1.4 Research Questions and Hypotheses
I have developed and evolved a framework for continuous monitoring and updating
of a project’s Cone of Uncertainty, guided by the following research questions (RQs)
and hypotheses:
RQ1. How can projects better initialize, reduce, and track their Cones of Uncer-
tainty (COUs)?
Hypothesis 1. Software development teams that perform continuous assess-
ments with this framework will (not) improve project scoping and esti-
mation.
RQ2. Do continuous team assessments aid in team synchronization and stabiliza-
tion of concurrent engineering?
10
Hypothesis 2a. Software development teams that perform continuous
assessments will (not) be better synchronized and stabilized in under-
standing and concurrent engineering.
Hypothesis 2b. The COCOMO II effort drivers and scale factors can (not)
be automatically adjusted to reflect real project situations through per-
forming survey-based assessments.
RQ3. How can collaboration technology support better COU initiation, reduction,
and tracking and how effective is this framework when used in the industry
level process?
Hypothesis 3a. The collaboration capabilities of the assessment framework
will (not) be cost-effective in improving COU initiation, reduction, and
tracking.
Hypothesis 3b. Software development teams that utilize this continuous
assessment framework would (not) outperform others using traditional
assessment methods.
The “(not)” forms of the hypotheses represent the null hypotheses that need to
be rejected to validate the results.
1.5 Research Contributions
The intended contributions provided by this research are the following:
11
• Identification of the strengths and shortfalls of currently available assessment
tools and frameworks, and improvements to the currently available assessment
capabilities.
• A framework and tool for tracking the Cones of Uncertainty in project estima-
tion and progress assessment. Evaluation and improvement of the framework
and tool through experimental project application.
• A survey-based assessment framework for synchronizing and stabilizing team
performance.
• A framework for adjusting and visualizing COCOMO II estimation accuracies
and associated Cones of Uncertainty.
• A tool support for the framework.
The frameworks and tool support have been deployed at the University of South-
ern California (USC) and experimented upon with 13 software engineering projects.
The results of the experiments have been compared with data from two previous
years, and the results validate the hypotheses that the teams utilizing the framework
showed improvement in project performance. The improvements observed were in
the following areas:
• Improved project estimation accuracy
• Improved project issue and defect detection
• Improved project tracking and scoping
12
• Higher level of team synchronization and stabilization
• Reduced project effort
1.6 Organization of Dissertation
The organization of this dissertation is as follows:
Chapter 2 presents the background of the research work and discusses the
strengths and shortfalls of the related work.
Chapter 3 discusses the details of the research methodology and the research
setup used in experimenting with the framework.
Chapter 4 presents the details on the COTIPMO framework developed to
address the problem with team synchronization, stabilization, and project estima-
tion.
Chapter 5 presents the tool developed to implement the COTIPMO framework,
including a detailed discussion on the usage of the tool itself.
Chapter 6 provides a detailed discussion on the results of the experiments.
Chapter 7 concludes the dissertation with a summary of the contributions, and
also discusses potential directions for future work.
13
Chapter 2: Background and
Related Work
To date, there are various techniques used for project tracking and assessment
as well as for software sizing and estimation. All of the methods and techniques
described in this chapter are widely known and used in the industry and have certain
levels of tool support.
2.1 The Incremental Commitment Model
The Incremental Commitment Model [BL07] [PM07], now called the Incremental
Commitment Spiral Model (ICSM) [Boe11], is a risk-driven spiral process model
with high support for concurrent engineering and emphasis on key aspects of dif-
ferent process models. The key principles of the ICSM include the following:
• stakeholder satisficing
• incremental and evolutionary growth of system definition and stakeholder
commitment
• iterative system development and definition
• concurrent system definition and development
• risk management
14
The model focuses on early verification and validation concepts of the V-model,
while promoting the multi-increment development process of the Rational Unified
Process (RUP).
Figure 2.1 shows the overview of the ICSM process, illustrating the full develop-
ment life cycle consisting of the Exploration, Valuation, Foundations, Devel-
opment, and Operation phases. At the end of each phase marked, by the anchor
points, the project’s feasibility evidences are assessed, reviewed, and stabilized
before continuing on to the next phases. The ICSM is a highly scalable process
that emphasizes the feasibility and stability of a project, while retaining the adapt-
ability to changes.
Figure 2.1: Overview of the Incremental Commitment Spiral Model life cycle process
15
2.2 COCOMO II Model
The COCOMO II Model (COnstructive COst MOdel) is a model to help software
estimators develop reasonable estimates of the cost and schedule required to com-
plete a software development project. The model supports the estimations during
the early design and post architecture stages. Details on the COCOMO II models
can be found in [BAB+00].
For this research, the Post-Architecture model is used for estimation, as the
focus of the research is from the product design stages onward. This is because
before this phase, the project can be considered as an open trade space consisting
of many other factors that could affect the direction of the development project.
Some of these factors include:
• requirements volatility
• unknown technologies
• unknown project personnel
From the product design stage onward, the project can assume that the high level
requirements and high level architecture have been specified and are generally stable.
The focus can be put on the software team dynamics and project details.
In this research, the two COCOMO II models used are:
1. COCOMO II Estimation model. The model consists of 5 scale factors
and 17 cost drivers for calculating the estimation. It takes size, in SLOC,
as the main input for effort estimation. This model is used for traditional
16
development projects where the majority of source code is written by the
development team.
2. COCOMO II Application Point model. The model is an application
composition model where it takes the number of screens, reports, and third
generation (3GL) language components to determine the system size, coupled
with assessments of personnel and tool capability to compute the effort. This
model is used by the rapid-fielding projects where the aim is to customize and
configure a set of COTS, or NDI, products.
Details on the use of the two estimation models are discussed later in Chapter 4.
Feasibility
Concept ofOperation
Rqts.Spec.
Plansand
Rqts.
ProductDesign
ProductDesignSpec.
DetailDesignSpec.
DetailDesign
Devel.and Test
AcceptedSoftware
Phases and Milestones
RelativeSize Range x
4x
2x
1.25x
1.5x
0.25x
0.5x ApplicationsComposition
(3 parameters)
Early Design(13 parameters)
Post-Architecture(23 parameters)0.67x
0.8x
Figure 2.2: COCOMO II and the Cone of Uncertainty
Figure 2.2 shows the relationship between the COCOMO II model and the Cone
of Uncertainty, which represents the number of parameters that can be estimated
17
during the different phases of the project life cycle. It is clear that towards the
beginning of the project, the number of uncertainties and unknowns are so high
that only a few parameters can be used for estimation. On the other hand, as
the cone narrows towards the later phases, more and more parameters can be used
because more information about the project is uncovered. However, even during
the “Post-Architecture” phase, the cone is still wide because there is usually high
number of uncertainties that affect the project. The scope of this research focuses
on the post-architecture phases.
Furthermore, as discussed in Section 1.3.3, one of the common problems in using
cost estimation models is that all the parameters of the model must be understood
correctly in order for the estimation to be accurate and meaningful. In unfamiliar
projects or inexperienced teams, the estimation model may not be utilized properly.
Specifically in the COCOMO II model, the 22 parameters (5 scale factors and 17 cost
drivers) are often rated incorrectly by the estimator, due to a misunderstanding of
the parameter definitions, which creates estimates that are inaccurate or incorrect.
One of the goals of this research is to provide a framework to help estimators improve
on rating these parameters.
2.3 USC Software Engineering Course
In the two-semester team-project based software engineering course sequence
CSCI577ab at USC [USC11], students learn to use best software engineering prac-
tices to develop software systems for real clients. They learn the ICSM process for
developing software-based systems and in order for the ICSM process to be usable
18
by the software engineering course, the process has been tailored to small-sized
software development projects, while also incorporating industrial-grade tools and
practices. Each team consists of five or six on-campus students with generally less
than 2 years of working experience, and one or two off-campus students who are
full-time professionals with at least 5 years of industry experience. Typically, the
on-campus students act as operational concept engineers, requirements engineers,
software architects, UML modelers, coders, life cycle planners, and feasibility ana-
lysts, while the off-campus students take on the roles of Integrated Independent
Verification and Validation (IIV&V) personnel, quality assurance personnel, and
testers. The course consists of both traditional development projects and rapid-
fielding projects, which are completed either within a 12-week (1 semester) or 24-
week (2 semesters) schedule depending on their scopes and complexities [KB10].
In addition, the teams utilize the COCOMO II model [BAB+00] to estimate
the resources required to complete the projects. Since the class schedule is fixed,
the requirements are prioritized and the project scopes are adjusted accordingly in
order to fit the timeframe and the available resources. Once the project estimation
has been performed to show that the software project can be completed within
the timeframe and available resources, it serves as a basis of commitment for the
stakeholders, and the team proceeds through the ICSM process life cycle to develop
the system. However, due to the lack of experience and necessary management
skills, the estimates calculated by the student teams often turn out to be inaccurate,
resulting in either over- or underestimations.
19
Not all the projects complete the entire life cycle due to shortages of registered
students during the second semester (Spring semester). Because students are not
required to go through both semesters of the course, the number of student enroll-
ments usually drops significantly after the end of the first semester (Fall semester).
Typically, 30% of the students continue into the second semester. For projects that
have been planned for a 2-semester timeframe, the projects must be dropped when
insufficient number of team members continue and register for the second semester.
These projects are not considered as failed projects since the reason for dropping
them is due to the uncontrollable factor of personnel continuity, not because of poor
project performance. On the other hand, 1-semester projects are always completed
and delivered within the Fall semester, and they usually make up 50-60% of the
total number of projects.
The experiments of this research were conducted with the USC software engi-
neering course, which will be discussed in more detail in Chapter 3.
2.4 IBM Self-Check
The IBM Self-Check [KK08] is a new generation, team evaluation framework that
blends together the benefits of typical assessment methods and retrospectives, while
eliminating their drawbacks. The development of the framework is based on the
realization that assessments done by simply selecting metrics are not enough to
effectively evaluate the teams because they often fall into the 5 common pitfalls of
standard assessments, which are the following:
1. bloated metrics
20
2. evil scorecards
3. lessons forgotten
4. forcing process
5. inconsistent sharing
These pitfalls are described in detail in [KKR08] and [KK08].
The goal of the IBM Self-Check is to avoid these pitfalls. The 4 main goals of
Self-Check are:
1. Learn - the subjective approach of Self-Check allows the teams to quickly
collect data, while reminding each team member about the key practices used.
2. Improve - teams often have hard times coming up with actions to take to
improve their performance, as typical assessment methodologies may produce
large amounts of data. The teams may not have time to identify all the actions
or there may simply be too many to implement. Self-Check helps narrow down
the scope of actions to take and implement.
3. Share - the goal of Self-Check is to produce a limited number of actions to take
so that they can be understood, shared, and implemented. The experience
reports are to be kept small. This can turn doubters into believers, as the
results appear to be implementable.
4. Respect team internals - the data gathered by Self-Check is done by the
team and for the team. This reinforces team ownership and encourages the
team members to implement the actions that have been identified.
21
Self-Check is done solely by the team. They either create or choose the questions
that they want to assess themselves on. These questions should reflect the core
practices that the team adopts. Then each team member answers the questions by
individually rating each question without knowing how the others answered. There
are no right or wrong answers; averages and deviations of the answers are used to
trigger discussions within the team. Figure 2.3 shows an example result of a Self-
Check session displaying the averages and deviations of the responses as well as the
areas of strengths and weaknesses [KK08].
(a) Big Picture (b) Iterative Detail
Figure 2.3: Self-Check Results. The charts show average scores and the standarddeviation of the answers provided by each team member. The scores range from0-10 with 10 being the best response.
Although the IBM Self-Check is able to overcome the common pitfalls of stan-
dard assessment methods, it does have some drawbacks of its own. The framework
requires that teams using Self-Check are able to effectively discuss the issues trig-
gered by Self-Check, and to develop implementable actions to resolve those prob-
lems. The teams must be experienced in order to utilize the framework to its full
potential. Otherwise, the Self-Check framework would not be as effective.
22
The assessment methodology used in this research is based heavily on the Self-
Check concept, which will be discussed later in detail in Chapter 4.
2.5 Continuous Assessment
The concept of continuous assessment has been widely used in education, where stu-
dents are continuously observed and assessed throughout their education to review
their performance [Wik12]. The frequent assessments throughout the school year
allow the teachers to know the strengths and weaknesses of their students, and to
help strengthen them as necessary. This also enables the teachers to plan the future
courses for the students based on the assessment results. The continuous assessment
concept is often used as an alternative to the traditional final examination system.
Continuous assessment has been applied to measuring and monitoring the capa-
bility of software process in Janne Jarvinen’s work on Measurement Based Contin-
uous Assessment (MCA) [Jar00]. The assessment method is intended to be applied
consistently at selected project intervals such as project milestones. This allows
software processes to be more visible to the development teams and enables early
detection of any deviation from the process. Another benefit of performing continu-
ous assessment is its cost effectiveness. Once implemented, the assessment is easy to
carry out and easy to manage throughout the life cycle. Furthermore, as issues and
potential problems are detected early when assessment is consistently performed,
they can be resolved and prevented before they become critical problems.
The MCA method is effective in providing support for management plans and
monitoring process capability in the development team. However, the method
23
only focuses on the aspects of the software process that the development team has
adopted, not the technical aspects and knowledge management within the team.
2.6 Early Detection of SE-Related Program
Risks
The main goal of the Early Detection of SE-Related Program Risks [BBI+09]
research is to transform the systems engineering (SE) process into an evidence-
based approach allowing for program risks to be detected and resolved early on in
the project. As projects grow in size, criticality, and volatility, their needs for vali-
dated feasibility evidence also increase significantly. Figure 2.4 shows the penalties
that projects face when they fail to address the risks and the investments required
to recover from these shortfalls. Problems that occur late in the project life cycle
are far more difficult to resolve and have much greater impact on the project itself.
Also, the penalties are significantly more severe for projects.
The Early Detection of SE-Related Program Risks research resulted in the SE
Performance Risk Framework and the SE Competency Risk Framework. Table
2.1 show the number of goals, critical success factors, and questions in the SE
Performance Risk and SE Competency Risk frameworks. These frameworks show
that the effectiveness of software engineering practices can be assessed both by
the performance of SE functions and the competency of the personnel performing
those practices. The goals, objectives, and questions in these frameworks contribute
24
Figure 2.4: Architecture evidence shortfall penalties and resulting architectinginvestment sweet spots
significantly to the development of the questions in the survey-based assessment
approach of my research framework as some questions are derived from them.
Framework Goals Critical Success Factors QuestionsSE Performance Risk 4 18 74SE Competency Risk 5 23 81
Table 2.1: Goals, critical success factors, and questions of Early Detection of SE-Related Program Risks framework
2.7 Architecture Review Board
The Architecture Review Board (ARB) [MRZ+05] concept was developed by AT&T
Bell Labs to review and validate the feasibility of the chosen architecture, and to
25
verify that it is a solution to the problem to be solved by the project. The review
process has proven to be an effective mechanism for increasing the likelihood of
successful project outcomes as their architectures become more stable and complete.
Some of the key valuable contributions of practicing architecture reviews include:
1. finding design problems early
2. leveraging experienced people
3. providing visibility to technical and management issues
4. rapidly identifying knowledge gaps
The ICSM process adopted by the student projects at the USC’s Software Engi-
neering course makes use of the ARB process to review the projects’ feasibility evi-
dence and risks at each major milestone. The major milestones that use the ARB
review process are 1) Foundations Commitment Review, 2) Development Commit-
ment Review, 3) Rebaselined Development Commitment Review, and 4) Transition
Readiness Review. During these reviews, every aspect of the projects is visited,
ranging from conceptual and architectural developments to feasibility analysis. If
the risks are not critical or are addressable, the teams continue on to the next
phase. Otherwise, the projects have a few days to readdress the identified risks
before their milestone deliverables are due. However, occasionally a project needs
to be drastically rescoped or dropped due to unaddressable risks and problems.
Furthermore, the ARB review process also points out any mistakes and inconsis-
tencies within the projects in order to synchronize and stabilize the project teams.
This is when knowledge gaps, personnel capability shortfalls, and management
26
issues become apparent and need to be reworked. However, this process only takes
place during the major milestones in the project where decisions are made about
the project status and future. Many problems can be detected early and resolved
before reaching the review process if assessments are continuously performed and
analyzed by the team before throughout the project life cycle. When compared to
the education system, the ARB’s can be seen as the mid-term and final examina-
tions. They are very effective for determining the performance of the development
teams during the major project milestones. However, it can also cause projects pre-
cious time for rework, and a project can even be dropped due to the late discovery
of problems and risks.
2.8 Software Sizing and Estimation
Story points [Coh06] are commonly used among agile development processes. The
method allows the team to analyze the team’s velocity, or productivity, based on the
story points completed, and use this data for planning future iterations. Planning
Poker [Gre02] is the tool often used for estimating the complexity of the story points.
However, the method requires expert opinions and analogies in order to estimate
accurately. Furthermore, in most cases, re-estimations only take place when story
sizes change, and are not based on the changes in the development progress.
The Program Evaluation and Review Technique (PERT) sizing method [PF79]
focuses on sizing the individual components based on the size distributions (opti-
mistic, most likely, and pessimistic) provided by the developers. Many tools such
as SEER-SEM [GE06], COCOMO 81 [Boe81], and PRICE-S [PRI98] utilize PERT
27
for sizing and estimation. The method reduces the bias towards overestimation
and underestimation, although people tend to choose the “most likely” estimates
towards the lower limit, while the actual product sizes cluster towards the upper
limit. Based on [Boe81], this underestimation bias is due to the following reasons:
• People are optimistic and tend to have a desire to please.
• People tend to not have a complete recall of past experiences.
• People are generally not familiar with the entire software job.
The Wideband Delphi predecessor to Planning Poker [Boe81] is an alterna-
tive method to the Delphi Technique [Far70] to broaden communication bandwidth
among team members to address any uncertainties. As with Planning Poker, the
estimations are presented to the participants for discussion on places where the
estimations vary.
Finally, there has been work done on improving the estimation accuracy of
COCOMO II by utilizing various mathematical models. First, the COCOMO-U
[YWT+06] extends the COCOMO II model to allow for creating estimations, even
when some of parameters are unknown or uncertain. It utilizes the Bayesian Belief
Network (BBN) to handle these uncertainties. However, in order for the tool to
compute correctly, the users must be knowledgeable and have expertise in specifying
the uncertainty of the unknown cost drivers.
In addition to using the BBN technique in COCOMO-U, the fuzzy logic model
has been applied to the COCOMO II model to treat uncertainties and unknowns in
the estimation. The FL-COCOMO II model [AO11] applies fuzzy logic to the cost
28
drivers, scale factors, and size parameters of the COCOMO II model. The exper-
iment result only shows about 8% improvement in the Mean Magnitude of Rela-
tive Error (MMRE). Furthermore, the neuro-fuzzy technique has been applied in
[HHRC07], combining the artificial intelligence and fuzzy-logic methods, to produce
improved and more accurate estimates. However, in order to utilize the neuro-fuzzy
method effectively, the model must be trained with relevant past project data. The
more relevant training data the model has, the more accurate the estimation would
be. As discussed in Section 1.1, unfamiliar projects and inexperienced teams may
not have these data readily available. Without the necessary training data, the
effectiveness of the model may not be any better than traditional human judgment.
2.9 Project Tracking and Assessment
Originally developed in the 1950s and 1960s, and summarized in [WL77], the PERT
network charts enable projects to effectively manage uncertainties by providing
mechanisms to identify critical paths and dependencies between tasks and activities.
The tasks and their corresponding paths are updated so the progress of the project
is visible to the developers and other stakeholders. However, even with proper tool
support, the number of tasks and dependencies can grow very large fairly quickly
especially when tasks and dependencies are not properly defined. As the charts
grow large, they require high overhead to maintain, and may be disregarded by the
development teams due to their complexity.
29
The Goal-Question-Metric (GQM) [Bas95] is another popular method for
progress tracking and measurement, with the ability to capture them from the con-
ceptual, operational, and quantitative levels. This allows the assessment process
to align with the organization environment as well as the project context. Various
tools have been developed to leverage the use of the GQM method, since the GQM
plans can grow very large and complex. The “GQM tool” [Lav00] automates the
GQM process to help manage the GQM plans, while the GQM-PLAN tool [AK99]
integrates the use of GQM into each stage and phase of the project life cycle. How-
ever, the GQM approach is only as good as the specification of the appropriate
goals, questions, and measurements to be monitored.
The Earned-Value Management (EVM) approach summarized in [Boe81] and
its corresponding agile Burn-up and Burn-down charts [Coc04] are good for cap-
turing the project progress based on the team’s velocity and completed features.
However, these approaches are less effective on large projects with major changes
during each iteration. There, EVM requires high overhead to accurately report the
progress, and to map them to the requirements for the earned-value analysis. When
projects constantly change, the development teams may end up spending more time
updating the earned-values, instead of spending time towards the actual develop-
ment activities. Similarly, in Burn-up and Burn-down charts, when major changes
occur, the charts may end up showing no progress, as the shift in requirement pri-
orities may prevent certain features from being completed. This prevents the team
from accurately determining the actual progress and the productivity rates of the
developers.
30
Chapter 3: Research Methodology
3.1 Overview of Research Design
There are two primary categories of effort estimation which are model-based and
expert-based approaches [MCHL06]. In the model-based method, historical data are
used to make predictions for future projects. On the other hand, the expert-based
method utilizes experience and expertise of the estimators to develop meaningful
and accurate estimates. The goal of the research effort is to develop a continu-
ous assessment framework for software development teams to effectively track soft-
ware project progress, assess team performance, and improve project estimations
based on those assessment data throughout the project life-cycle. This is done by
combining both the model-based and expert-based approaches in the assessment
framework.
In the model-based method, the goal is to develop a framework based on the
assessment data of past projects identifying the strengths, weaknesses, and issues
of those project teams and forming the assessment criteria to address them appro-
priately. Moreover, the framework utilizes the pre-calibrated COCOMO II model,
which further contributes to the model-based aspect of the framework, as the model
has been calibrated to 161 industry projects.
In the expert-based method, the research goal is to develop a framework to
improve software estimation accuracy for development teams based on the assess-
ment data. Two approaches were taken to determine the relationships between the
31
assessment questions and the COCOMO II parameters – by finding the correlation
with the team’s performance and by expert judgments. Experts and experienced
users of the COCOMO II model were surveyed to analyze all the assessment ques-
tions and to identify the corresponding level of impact that each question has on
the COCOMO II parameters.
The COTIPMO tool has been developed to implement the COTIPMO frame-
work in order to help projects reduce the Cone of Uncertainty, and to gather data
to observe and analyze the framework’s effectiveness when used by a software devel-
opment team throughout the project life cycle. Having a powerful tool allows for
the benefits of the framework to be realized effectively.
3.2 Research Timeline
Figure 3.1 shows the research timeline in developing the COTIPMO framework.
Before the actual development of the entire COTIPMO framework, a significant
amount of time was spent gathering team performance data from the USC Soft-
ware Engineering course in order to develop the assessment framework. This data
included the team’s strengths, weaknesses, and issues, and the assessment frame-
work has been developed to address them.
The development of the COTIPMO framework happened in four main phases. In
the first phase, the project progress tracking and estimation mechanism was devel-
oped. The focus was mainly on tracking the progress for traditional development
projects based on source code. In the second phase, the survey assessment frame-
work was developed using the gathered data previously mentioned and deriving the
32
Figure 3.1: Research Timeline
questions to address them. In parallel, COCOMO II experts and experienced users
were surveyed to identify the relationship between the assessment questions and the
COCOMO II parameters. In the third phase, the COTIPMO tool was developed
to implement the framework. More surveys were conducted with the COCOMO II
experts to refine the impact factor in the assessment questions. Finally, in the last
phase, the tool and framework were deployed for experiment with the USC Software
33
Engineering course in Fall 2011. Data was obtained and analyzed throughout the
Fall 2011 - Spring 2012 semesters.
3.3 Obtaining the Data
The application of the COTIPMO framework and tool was done in a classroom
setting using the data obtained from the real-client team project graduate software
engineering course sequence CSCI577ab at USC, discussed in Section 2.3. The
development teams used the tool weekly to report the development progress as part
of their progress report. For each week, each team member answered a general
project progress survey. For each milestone, a more in-depth survey was generated
in order to assess the milestone achievements and performance. As the projects
progressed during the semester, the teams continuously recalibrated the cost and
schedule estimations based on the improvements suggested by the COTIPMO tool
in order to reflect the teams’ status and performance.
Throughout the life cycle of the projects, data was collected from various aspects
of the project including the following:
• COCOMO II estimation data. As the projects used forms of the
COCOMO II model for resource estimation, the parameter ratings were col-
lected from every project. These data included the ratings for scale factors and
cost drivers for the Post-Architecture model, or the details for the Application
Point model.
34
• Project issues and defects. As the project progressed, the development
team, especially the IIV&V personnel, continuously filed issues and problems
that were found in the project. These included inconsistencies in project
understandings, unstable project concepts, and general defects in artifacts.
• Project risks. Each project went through a rigorous risk assessment and
management process as described in [Boe89]. The risks were constantly mon-
itored and prioritized to ensure that the top critical risks were addressed and
mitigated. The risk data were collected weekly from each development project
in order to show the risk resolution progression during the project life cycle.
• Individual and team effort. Weekly, each individual team member sub-
mitted an effort report on the number of hours he/she spent on certain project
activities during the week. The report covered all major aspects of the
project, ranging from operational concept development to project adminis-
tration and collaboration. The details on specific effort categories can be
found in Appendix E.
• Project performance. Throughout the entire life cycle, project teams sub-
mitted artifacts to be reviewed and graded by the teaching staff. Additionally,
during the major milestones, projects went through the ARB process as dis-
cussed in Section 2.7 where they received feedback from all the stakeholders.
The project performance were determined based on the project grades and
the stakeholder feedback.
35
• Client satisfactions. At the end of each semester, the clients evaluated their
development teams. The level of client satisfaction is another key factor in
determining the performance of the projects. Details on the client evaluation
form can be found in Appendix G and Appendix H.
• Student feedbacks. At the beginning and the end of each project life cycle,
students were surveyed to report on the status of their project. The survey
included asking team members on the following aspects:
– Level of project uncertainties
– Level of team synchronization
– Level of team strength
– Difficulties in mitigating risks, resolving project issues, and synchronizing
the team.
Details on the survey can be found in Appendix I.
Every semester since Fall 2009, the collection of the project data had been consistent
in the software engineering course. This allowed comparative analysis of project
performance between the year that the framework and tool were applied and the
years that they were not. Table 3.1 shows the number of traditional development
and rapid-fielding projects from the Fall 2009 to the Spring 2012 semesters. The
COTIPMO framework was deployed and used during the Fall 2011 - Spring 2012
year, and the data collected were compared with the Fall 2009 - Spring 2010 and
the Fall 2010 - Spring 2011 years.
36
Semester Traditional Rapid-Fielding TotalFall 2009 9 5 14Spring 2010 4 1 5Fall 2010 6 2 8Spring 2011 4 1 5Fall 2011 6 7 13Spring 2012 4 1 5
Table 3.1: Number of traditional development and rapid-fielding projects persemester
37
Chapter 4: The COTIPMO
Framework
The COnstructive Team Improvement Process MOdel (COTIPMO) [AKB12]
has been developed to address the problems stated earlier in Section 1.3 by improv-
ing team synchronization and team stabilization through effective progress tracking
and continuous assessment methods. The framework combines the model-based and
expert-based methodologies to create an effective team assessment mechanism and
improvements to the project estimates throughout the project’s life cycle. Instead
of requiring extensive historical data and a high level of expertise in the team,
the framework uses data gathered since the beginning of the project to assess and
improve the team as the project progresses through its life cycle. During each assess-
ment period, the data is analyzed to point out potential issues and weaknesses as
well as to suggest improvements to the project estimation for future iterations.
Although developing the framework required both historical data and expert
judgment in order to establish the assessment mechanisms to properly address issues
and performances in the development process, the framework does not require the
development teams to have this type of data and expertise in order to use the frame-
work effectively throughout the project life cycle. In the COTIPMO framework, the
project assessment is expected to be done consistently throughout the project life
cycle to help the development teams monitor their progress, while reducing any
38
uncertainties and knowledge gaps in the team. The duration between each assess-
ment period, or iteration, can be specified based on the team’s preferences or based
on necessity. The monitoring mechanism can help teams ensure the feasibility of the
project timeline and encourages the team to re-negotiate or re-scope the require-
ments, features, or budget if needed.
Identified
issues and
actions
Estimated
PM
Framework data Sub-method
Perform
survey
assessment
Traditional
development
project estimation
Rapid-fielding
project estimation
Traditional development project
Rapid-fielding project
ActivityLegend:
Adjust
COCOMO II
parameters
Figure 4.1: Workflow of the COTIPMO framework
Figure 4.1 shows the workflow of the COTIPMO framework. The traditional
development projects and rapid-fielding projects utilize different methods for track-
ing and computing the estimated effort, which are discussed later in this chapter.
However, both types of project follow the same methodology for team assessment
and team synchronization as well as the process for performance and estimation
improvements. The process is reiterated throughout the project life cycle.
39
4.1 Project Progress Tracking Framework
The first part of the framework has been developed to help unprecedented projects
and teams track their progression through the project life cycle. This helps the
project teams reduce the uncertainties of estimations, and achieve the eventual
convergence of the estimated and actual effort spent on the project. According to
Rob Thomsett in [Tho02], project managers and development teams often make
the common mistake of tracking project progress based on time sheets rather than
the tasks themselves. He stated, “Never confuse tracking projects with tracking
people.” The amount of time a person spends in the office or on the project has
little significance to the project progress itself.
Therefore, the project progress tracking framework of COTIPMO focuses on
tracking the developed components instead of the time spent on the project. This
allows the framework to report the actual progress on the project and update the
estimates accordingly. The framework utilizes the COCOMO II models to convert
the developed components into effort hours. This is viable because the COCOMO
II models take the size of the components along with other additional parameters to
calculate the effort required to develop the component. Instead of estimating the size
parameter, the actual sizes of the partially developed components are used, along
with the performer’s assessments of the component’s percentage of completion. This
allows the model to compute more accurate sizes and the amount of effort required
for the development.
The COTIPMO’s project tracking framework provides support for both tradi-
tional development projects and rapid-fielding projects. Depending on the type of
40
the project, the framework utilizes different COCOMO II models for estimation and
effort conversion. The two models used are:
1. COCOMO II Estimation Model. The model used when the size of the
components is in SLOC.
2. COCOMO II Application Point Model. The model used when the size
of the components is in number of application points (screens, reports, and
components) developed.
4.1.1 Tracking Based on Source Code
Unified CodeCount
Actual SLOC
EstimatedSLOC COCOMOII
EstimatedPM
Source code files
% REVL
% Developed
% Tested
% Integrated
Scale factors
Cost drivers
Developer’s inputFramework dataTool
Legend:
Figure 4.2: Workflow for traditional development projects
41
For traditional development projects, the framework integrates the Unified Code
Counter (UCC) tool [UCC] and the COCOMO II Estimation model to allow quick
progress tracking and estimation based on the amount of source code developed
[ASB10]. Figure 4.2 shows the process of computing the estimated effort, in person-
months (PM), based on the developed source code.
The assessment framework utilizes the COCOMO II Estimation model to esti-
mate the resources required to complete a software development project. It takes
the SLOC, adjusted with the requirements volatility factor (REVL) of each module,
along with the scale factors and effort multiplier parameters, and applies them to
the COCOMO II Estimation model to generate actual efforts in PM by using the
formula shown in equation 4.1. The resulting PM can then be converted to the
number of hours spent on the project. During the beginning of the project when
the source code development has not started, the Size of each module is estimated
by the team in the same way that the COCOMO II model is traditionally used.
The estimated sizes and the corresponding COCOMO II estimates can be updated
as the team gains more knowledge about the project. However, once the actual
implementation begins, the Size variable is based on the developed source code
reported by the UCC tool. The result of the calculation is the actual effort spent
42
towards development, which can be used as a basis to estimate the effort required
to complete the project.
PM = A× SizeE ×17∏i=1
EMi
E = B + 0.01 ×5∑
j=1
SFi
Hours = PM × 152
(4.1)
Where:
• A = 2.94 (a constant derived from historical project data)
• B = 0.91 (a constant derived from historical project data)
• PM is for Person-Months
• Size is in KSLOC
• EMi is the effort multiplier for ith cost driver
• SFj is the jth scale factor used to determine the economies or dis-
economies of scale
• Hours/PM is based on the COCOMO II nominal value of 152
hours/PM
Estimated SLOC =Adjusted SLOC
Estimated % Complete× 100
Estimated PM =Equivalent PM
Estimated % Complete× 100
(4.2)
Note that in Figure 4.2, there are two additional sets of input from the devel-
opers, which are the “% integrated” and “% tested”. These inputs had been added
43
to the framework to compute the earned value of the developed module. In the cal-
culation, the module is not considered to be “complete” until it is fully integrated
and tested. However, for the purpose of the estimation, the modules that are at
least 90% integrated and tested contribute towards the estimated SLOC as they are
considered to be significantly complete.
In experimenting with the software engineering projects at USC, the number of
modules were small due to the small sizes of the projects, thus making the use of
traditional earned value management less effective. Since the number of modules
were small, waiting for a module to be fully completed for the earned value to
be updated took too long for the project estimation to react to. For example, if
the number of modules was 4 and the project team took over half of the project
life cycle to completely develop, test, and integrate two modules in parallel, the
project progress remained at 0 for half the project, then jumped directly to 50%.
Reporting this type of progress would not be meaningful to the team as well as to
the estimation process itself.
Instead, the earned value aspect has been removed, so the calculation only takes
into consideration the “% developed” parameter. With this calculation, the estima-
tions are constantly updated weekly, so that the team can react to any unexpected
circumstances in the project resource allocations. However, the “% integrated” and
“% tested” inputs are still taken from the developers for future uses when project
sizes and complexities are significantly higher.
To initially validate the framework for assessing and tracking progress by source
code, a simulation was performed on two previously completed projects from the
44
USC software engineering course. The source code data were retrieved from the
Subversion repository to get weekly builds information. Additionally, effort data
and estimation data were gathered from the project to be used during the simula-
tion. The developers from the projects were involved in the simulation process to
provide additional data such as percentage completed and project statuses.
0
200
400
600
800
1000
1200
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Hou
rs
Weeks
Team A Progress Initial Estimate Actual Estimated
(a) Team A progress
0
200
400
600
800
1000
1200
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 H
ours
Weeks
Team B Progress Initial Estimate Actual Estimated
(b) Team B progress
Figure 4.3: Simulation results - overall progress
Figure 4.3 shows the results of the simulation, illustrating the comparison
between the overall estimated and actual efforts spent by both teams through-
out the 24 weeks of the project development life cycle [ASB10]. It is interesting
to observe the difference between the two teams in the behavior of the Cone of
Uncertainty.
Team A overestimated the effort required to complete the project by over 50%.
Based on the discussion with the team members, the main reason for their esti-
mation error was due to the fact that they were pessimistic about the developers
capabilities and assumed the project was more complicated than it actually was.
On the other hand, Team B underestimated their required effort by over 18% due
to the lack of experience in identifying the actual effort that would be required to
45
develop certain modules. Moreover, the developers were not experienced with the
development language, JSP, so they were not aware of the complexities that could
potentially occur during the project. It is also interesting to observe the result of
Team B that the upper part of the Cone of Uncertainty does exist. This means that
even if projects begin initially with an estimate on the upper curve of the cone, it
may not always mean an overestimation in the size or resource at the end.
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
% p
rogr
ess
Week
Percentage in Project Progress
Team A
Team B
(a) % project progress
0
10
20
30
40
50
60
70
80
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
% E
rror
Week
Percentage in Estimation Error
Team A
Team B
(b) % estimation error
Figure 4.4: Simulation results - estimation error
Furthermore, Figure 4.4 shows another view of the simulation results with the
projects’ actual progress and the percentage of estimation errors. The project
progress was calculated by using the effort converted from SLOC developed and
compared it against the adjusted estimated effort. Since the assessment process
allowed the estimations to become more and more accurate as the project moved
forward, the project progress became more realistic as well. At the same time, the
estimation error showed improvements week by week throughout the project life
cycle. The reductions in error rates were significant compared to the initial esti-
mates done by the developers, thus showing a much more accurate estimation when
utilizing the assessment framework.
46
The simulation results show that with just better tracking of actual progress
during the development life cycle, the estimated effort to complete can become
more and more accurate over time. The improved tracking mechanism allows for
better assessment of past team performance and allows for better prediction of
the future iterations. Thus, this contributes to the model-based approach of the
COTIPMO framework.
4.1.2 Tracking Based on Application Points
ActualNAP
Estimated NAP
COCOMOIIApplication
Point
EstimatedPM
# Screens
% Developed
% Tested
Developer’s inputFramework dataTool
# Reports
# 3GL Components
Productivity Rate
Developer’s experience and
capability
ICASE maturity and
capability
Legend:
Figure 4.5: Workflow for rapid-fielding projects
For rapid-fielding projects, the framework utilizes the Application Point model
of COCOMO II [BAB+00] for effort estimation and progress tracking. Instead
47
of tracking the number of lines of code written, the model tracks the number of
application points developed, which include the number of screens, reports, and
third-generation language (3GL) components. Figure 4.5 shows the process for
reporting the application points and using the Application Point model to compute
the estimated effort. The procedure for effort reporting and estimating with this
model is much less complex compared to that of the regular COCOMO II model as
there are fewer parameters to input and compute.
The initial estimates of the project are created based on the predicted number
of application points to be developed, customized, and configured. Similar to the
traditional development projects, the team can make updates to the estimates and
the COCOMO II Application Point model parameters during the beginning of the
project, while the actual implementation has not started. However, once the imple-
mentation begins, the team reports the number of application points completed up
to the time of each assessment.
NAP = (Screens+Reports+ Components) × reuse
Estimated NAP =NAP
Estimated % Complete
(4.3)
Where:
• NAP is the New Application Point
• Screens is the weighted number of screens
• Reports is the weighted number of reports
• Components is the weighted number of 3GL components
48
Note that the “% tested” input is not calculated with the estimation due to
project sizes and complexities as discussed in Section 4.1.
For details on estimating with the Application Point model and computing the
NAP and PM , refer to Appendix L. Similar to the previous section, converting
the person-months to hours is based on the COCOMO II nominal value of 152
hours/PM .
4.2 Continuous Team Assessment Framework
Team synchronization and stabilization are essential to successful project outcomes
because knowledge gaps and inconsistencies among the developers are common
problems in team projects. The second framework is an assessment technique to aid
in reducing those gaps in order to help stabilize the team and project understand-
ings. The methodology is based heavily on the IBM Self-Check concept in [KK08],
a survey-based approach effective for detecting and narrowing the knowledge gaps
among the team members.
With the concept of continuous assessment, the development team consistently
and continuously performs the assessment throughout the development life cycle.
The frequency of each assessment period can depend on the team situation. In a
more complex project and larger teams where team synchronization may be more
critical and difficult, the assessment periods can be more frequent. During the
major project milestones, a more in-depth assessment can be performed to ensure
that additional risks and issues are properly addressed.
A method has been developed to assess team performance in the following areas:
49
1. Requirements engineering
2. Business case analysis
3. Architecture and design development
4. Planning and control
5. Feasibility evidence
6. Personnel capabilities
7. Collaboration
Questions have been developed focusing on identifying the team’s strengths, weak-
nesses, and issues in order to help synchronize team understandings and stabilize
team performance. The survey questions do not contain right or wrong answers;
however, they ask each developer for his/her opinions and perspectives on the team’s
capabilities and performance. Similar to the IBM Self-Check approach, each team
member answers the survey questions individually without knowing the others’
answers. This is to prevent any biases that could exist in answering the ques-
tions. The deviation and mismatches in the answers are used to determine weak
and inconsistent areas, and the team must identify actions to resolve those issues
in order to prevent them from becoming potential problems in the future itera-
tions. The COTIPMO assessment framework currently consists of 34 questions.
Two approaches were utilized to develop the survey questions, which are:
1. Questions derived from team assessment data
50
2. Questions derived from SE Performance Risk and SE Competency Risk frame-
works [BBI+09]
4.2.1 Questions from Team Assessment Data
The first approach was to analyze the USC software engineering graduate student
projects by conducting team assessments for individual team members to evalu-
ate their team’s strengths, weaknesses, and issues focusing on the various aspects
mentioned earlier. The assessment was done with the software engineering projects
during the 2009 and 2010 academic years.
Team’s'assessment'data'
Strengths,'weaknesses,'issues'
Survey'ques7ons'
Figure 4.6: Process for Analyzing Team Assessment Data
Figure 4.6 shows the process for analyzing the team assessment data. First,
individual team members identified the strengths, weaknesses, and issues that they
discovered in their projects throughout the life cycle. These assessment data were
then analyzed, and the critical ones were picked out and categorized into their
respective categories. The questions were developed to address the knowledge gaps
and potential issues that commonly occurred in these assessment data. Since the
questions were derived from teams’ strengths, weaknesses, and issues, they are
focused mainly to help resolve team issues and stabilize team performance.
Table 4.1 shows the issues and weaknesses captured by the teams during each of
the semesters. These data were used to develop the set of questions to be used in the
51
survey assessment framework. The number of issues appeared significantly less in
Fall 2010 due to the decreased number of students and teams. The assessment data
resulted in 36 questions from Fall 2009 and 12 additional questions from Fall 2010.
These questions focus on reducing the knowledge gaps and resolving potential issues
that commonly occur within the team based on the observations from Fall 2009 and
2010 teams. Both semesters presented common aspects in issues and weaknesses.
Problem Fall 2009 Fall 2010Lack of understanding in domain/business process 37 12Exploring alternatives 14 8Lack of collaboration among team members 30 18Lack of collaboration with client 1 12Lack of review mechanisms 11 2Lack in sharing of information/understanding 27 13Stakeholders involvement 8 0Lack of help by team members 2 10Roles and responsibilities awareness 10 18Task assignments 31 7Planning + progress monitoring and control 47 17Inexperience with process 4 4Identifying risks 1 1No working prototypes 2 6Prototypes do not address critical issues 20 8Clarifying requirement issues/details 20 3Suggesting rather than listening 2 4Analyzing current system 4 3Evolutionary features 1 7Inexperience with technologies 14 20Delay in developing architecture 2 2
Table 4.1: Team issues reported by project teams by semester
52
4.2.2 SE Performance Risk and SE Competency Risk
Frameworks
In the second approach, the questions were adopted from the Early Identification
of SE-Related Program Risks framework [BBI+09]. Discussed in Section 2.6, the
research consists of 2 frameworks - SE Performance Risk and SE Competency Risk
- which show that the effectiveness of software engineering practices can be assessed
both by the performance of systems engineering (SE) functions and the competency
of the personnel performing those practices. Both frameworks are currently used
in the industry for assessing and analyzing the capabilities of the project personnel
and for identifying potential risks and weaknesses that should be addressed. The
questions adopted from these frameworks focus on evaluating the capabilities of
the team as a whole as well as the individual members. This is to establish that
the major concerns are identified and addressed by ensuring that the project team
has sufficient capabilities and experience. The COTIPMO assessment mechanism
adopted 20 questions from these frameworks.
Moreover, since the SE Performance Risk and SE Competency Risk frameworks
cover various aspects of systems and software engineering practices, they were used
to verify and validate the questions derived in the first approach. Because the first
approach focused specifically on team synchronization and stabilization, adopting
all the questions directly from the SE risk frameworks was not feasible as they
were geared towards analyzing and evaluating the performance and competency
of personnel. However, the various aspects of the performance and competency
risks were compared with the set of custom questions derived in the first approach
53
to make sure that the risks and concerns in software engineering practices were
properly addressed. This is to ensure that the all of the assessment questions used
are consistent with the industry practices and standards.
4.2.3 Refining the Questions
Both approaches in developing the assessment questions resulted in a total of 68
questions. The questions were then reevaluated, reworded, and refined to properly
address the 7 areas of software engineering practices mentioned at the beginning
of Section 4.2. Similarly addressed questions were combined together, while some
redundant questions were discarded. After the refinement, the number of questions
was reduced to 34 questions, with the different numbers and emphasis shown in
Table 4.2.
Category QuestionsRequirements Gathering 4Business Case Analysis 5Architecture Development 3Planning and Control 8Feasibility Analysis 6Personnel Capability 4Collaboration 4
Total 34
Table 4.2: Categorized questions
The questions were then classified into 3 sets as follows:
1. Weekly ICSM set consists of 19 questions. This set of questions is meant
to be used during each regular assessment period (i.e., weekly) throughout
the project. The questions have been customized for the ICSM process as
54
they cover specific areas such as the WinWin negotiation [BBHL95] and the
NDI evaluation [Koo10] processes. The Weekly set does not always need to
be used weekly, but it should be used for assessments in between the project
milestones.
2. Milestone ICSM set consists of 28 questions. Similar to the Weekly ICSM
set, this set of the questions is also geared towards the ICSM process. How-
ever, the milestone set covers the project details more in-depth, such as addi-
tional details on architecture designs, project verification mechanisms, and
evolutionary requirements.
3. General set consists of 20 questions. Unlike the other 2 sets, this question
set is a generic set that can be used by any project using any process. The
details are similar to that of the Weekly ICSM set, but the the generic set
does not cover ICSM specific practices.
The full set of assessment questions can be found in Appendix A.
4.3 Project Estimation Adjustment Framework
As mentioned in Section 1.3.1, using the COCOMO II estimation model often does
not reflect the actual project situations because planners do not have the neces-
sary understanding of the estimation model. The third framework aims to help
development teams adjust their COCOMO II inputs to reflect reality. This can be
done through using the data gathered from the survey-based assessment framework
described in the previous section and in [KKR08] because answering a series of
55
relevant questions is a more effective means of reflecting the actual project status
compared to simply providing ratings to the COCOMO II parameters.
The questions developed for the survey-based assessment method contain cor-
relations to the COCOMO II cost drivers and scale factors. Each question impacts
either one or multiple COCOMO II parameters, whereas some impact certain
parameters more than others. As team members answer the survey questions, the
framework analyzes the answers and provides suggestions on changes to be made
to the team’s COCOMO II estimates to reflect the way they answered the survey.
These adjustments are only provided as suggestions, because there may be other
factors that could impact the rating for each estimation parameter, so the estima-
tor must use some judgment and reasoning before actually applying those changes.
However, as the calculation is based on the team’s assessment data, the suggestions
are reflective of the team’s status representing a more “expert” advice than just the
estimator’s judgment alone.
Two approaches were used in computing the correlation between the questions
and the corresponding COCOMO II parameters, which were:
• Correlation to team’s performance
• Expert judgment
Based on these impact factors, the COTIPMO framework analyzes the assessment
data and computes the adjustments that should be made to the COCOMO II
parameter ratings.
56
4.3.1 Correlation to Team’s Performance
Since the majority of the assessment questions have been derived from analyzing
the performance of the development teams, the impact that each question has on
the corresponding COCOMO II parameters can be observed based on the team’s
performance and the team’s accuracy in estimating with the COCOMO II model.
This is done by tracing each assessment question back to the strengths, weaknesses,
and issues that triggered those questions as well as the specific teams that identified
them. Then the various aspects of the team were analyzed and compared with the
team performances as well as the quality and accuracy of COCOMO II estimation
developed by the team. However, due to insufficient assessment data, the corre-
lations could not be determined with statistical significance. The three analysis
methods are discussed here.
Performance vs. Project Issues Trend
The first method used to determine the correlation was to observe any direct rela-
tionship between team performance and the number of problems that were identified
by each team. Figure 4.7 shows the count of problems based on the various cate-
gories comparing them to the team grades from Fall 2009 and Fall 2010 semesters.
There existed no obvious trend that was presented with respect to how well the
team performed and the number of problems that existed within the team. Although
this may seem counterintuitive as weaker teams should tend to have more problems,
there were many factors that could have affected the result such as project complex-
ity, client cooperation, and weak problem identification. Furthermore, more active
57
0.82%
0.84%
0.86%
0.88%
0.9%
0.92%
0.94%
0.96%
0.98%
0%
10%
20%
30%
40%
50%
60%
F10T05% F10T01% F10T02% F10T06% F09T02% F10T04% F09T07% F09T09% F09T03% F10T08% F09T11% F09T01% F09T10% F09T08%
Team
%Grade
%Percentage%
Coun
t%of%p
roblem
s%%
Team%
%Count%of%Problems%by%Catagories%and%Team%Performance%
Architecture%Development% Business%case%analysis% CollaboraDon% Feasibility%Evidence%
Personnel%Capability% Planning%&%Control% Requirements%engineering% Grade%(%)%
Figure 4.7: Count of problems with team performance
teams could have been more motivated to file problems and issues than the less
active ones. Therefore, this trend analysis could not properly show the relationship
between the team performance and problems within the team.
Due to insufficient sample data, the relationship between the team performances
and their impact on the COCOMO II estimation could not be determined based on
either the correlation or trend analyses. Therefore, the impact factor between the
question and the COCOMO II parameters could not be effectively computed based
on this approach. However, in the future, once enough historical assessment data
is collected, the analysis should be performed again. This will be further discussed
in Section 7.3.
58
Multiple Linear Regression
A multiple linear regression was performed analyzing the relationship between the
team performance, based on grades, and COCOMO II parameter ratings estimated
by the team.
Y = Grade(%)
X = 5 Scale Factors, 17 Cost Drivers
Y = β0 + β1X1 + β2X2 + ...+ β22X22
R2 = Coefficient of Determination
(4.4)
After running the regression analysis, R2 showed that the test result was not accu-
rate due to insufficient sample size. Because the number of predictors was higher
than the sample size, the degree of freedom of residual became negative; therefore,
the R2 was invalid. Details on the multiple linear regression analysis results can be
found in Appendix J.
Fit Y by X Test
Since the relationship could not be determined based on the overall COCOMO
II estimation, in the second method, each individual COCOMO II parameter was
analyzed and compared with the team’s performance. This was done through a
59
”Fit Y by X” test by trying to fit the scale factors and cost drivers to the team’s
grades percentages.
Y = Grade(%)
X = 5 Scale Factors, 17 Cost Drivers
(4.5)
(a) Scale Factors
Scale Factors R2
PREC 0.204009FLEX 0.199592RESL 0.009128TEAM 0.0PMAT 0.069396
(b) Cost Drivers - Product
Product R2
RELY 0.091688DATA 0.006811CPLX 0.07863RUSE 0.142054DOCU 0.00623
(c) Cost Drivers - Personnel
Personnel R2
ACAP 0.024039PCAP 0.308707PCON 0.419576APEX 0.01846PLEX 0.049807LTEX 0.039306
(d) Cost Drivers - Platform
Platform R2
TIME 0.000249STOR 0.00245PVOL 0.045471
(e) Cost Drivers- Project
Project R2
TOOL 0.016051SITE 0.186363SCED 0.19182
Table 4.3: Coefficient of determination (R2) for COCOMO II parameters
Similarly, the resulting analysis showed very low R2 yield, thus it showed that
meaningful correlations could not be determined. Table 4.3 shows the R2 yields
for each scale factor and cost driver. The results show that the values were very
60
low, making the relationship insignificant. Details on the multiple linear regression
analysis results can be found in Appendix K.
4.3.2 Expert Judgment
The second approach to determine the relationship and impact factor between the
assessment questions and the COCOMO II parameters was to use experts’ advice
and judgment, which was done by surveying a group of COCOMO II experts
and experienced users. For each survey question, the expert would identify the
COCOMO II parameter that was impacted. Weights were applied to the level of
expertise and experience of the expert surveyed in order to justify valuable inputs
and reduce bias. Table 4.4 shows the weights that were applied to each of the
COCOMO II experts. The way the weights were determined was based on using
an even distribution among the experience spread.
Years of Experience Weight0 - 2 years 0.32 - 7 years 0.67+ years 1.0
Table 4.4: Weights applied to level of experience with COCOMO II
Figure 4.8 shows an example of the survey given to the COCOMO II experts
and experienced users. As shown in the figure, each expert went through the survey
assessment questions and identified the corresponding COCOMO II parameters that
were impacted by the question. For the parameters that were directly impacted,
the expert would mark the box with an “x”.
61
!! !! Scale!Factors! Cost!Drivers!Subject! Ques4on! PREC! FLEX! RESL! TEAM!PMAT! Product! Pla?orm! Personnel! Project!!! !! !! !! !! !! !! RELY! DATA!CPLX! RUSE! DOCU!TIME! STOR! PVOL! ACAP! PCAP! PCON!APEX! PLEX! LTEX! TOOL! SITE! SCED!
Business'Process/
Scoping''
Do'you'and'the'stakeholders'understand'the'exis9ng'and'proposed'scope,'boundary'and'
business'workflow'?' '' '' '' x' x' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' x' ''
Prototyping'
Are'the'prototypes'developed'early'enough'to'address'conceptual'issues'and'
knowledge?' '' '' '' '' '' x' '' x' x' '' '' '' '' '' '' '' '' '' '' '' '' ''
Review'
Do'you'consistently'review'each'other's'work'before'integra9ng'them'together?' '' '' '' '' '' '' '' '' '' '' '' '' '' x' '' '' x' x' '' '' '' ''
Mee9ngs'
Are'the'stakeholders'mee9ngs'held'weekly'with'clearly'defined'agendas'and'does'the'
project'progress'is'discussed'extensively.' '' '' '' x' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' ''
Figure 4.8: Example of the expert survey
Furthermore, in the USC software engineering course, the project teams were
surveyed at the end of the project to identify and determine the impact factor
for the COCOMO II parameters. At the time that the surveys were done, most
of the students had at least 4 months of experience with the COCOMO model.
First, in the same way, each team member individually identified the parameters
that were impacted by the survey question. Then the entire team discussed each
member’s decision as to why certain parameters were considered to be impacted by
the question. Finally, the team developed a consensus on the impact factor. This
is considered to be representative of a nominal level of experienced user.
A total of 10 COCOMO II experts and 19 project teams were surveyed. Both
types of data were then combined and averaged, resulting in the weight, or impact
factor, that each survey question has on the COCOMO II parameters.
Steps undertaken to compute the survey:
1. Each “x” represents a value of 1
2. Multiply the value by weights in Table 4.4
3. Sum up values for all surveys
62
4. Average the sum
5. Scale the averaged value to a maximum of 1
6. Repeat the steps for all COCOMO II parameters.
The resulting value represents the multiplier, or impact factor, used for the
COCOMO II adjustment calculation discussed in the next section.
4.3.3 COCOMO II Adjustment Calculation
The adjustments that are suggested to be made to the COCOMO II scale factors
and cost drivers are calculated based on the resulting survey assessment scores.
First, the survey answer scores are scaled to a range of 1 to 6 in order to be mapped
to the COCOMO II rating scales, which have been translated to integer values as
shown in Table 4.5. Normalizing the answer scores allow for direct computation
with the ratings of the COCOMO II parameters. Furthermore, this allows the
scales of the survey questions to be more flexible as the answer range can be set
with different scales such as an integer of 1-10 or a decimal of 1-5.
Rating ValueVery Low 1Low 2Nominal 3High 4Very High 5Extra High 6
Table 4.5: COCOMO II ratings and corresponding values
63
All the answers to the survey questions are averaged and used to compare to
the current rating of the SF or EAF. The adjustments to the scale factors (SF) and
cost drivers (EAF) are then computed based on the equations 4.6 and 4.7. This
helps ensure that the SF and EAF values are reflective of the team’s assessment
result.
SFadj =n∑
i=1
(SFimpact × (ANSavg − SFrating)) (4.6)
EAFadj =n∑
i=1
(EAFimpact × (ANSavg − EAFrating)) (4.7)
Where:
• n is the number of questions
• SFadj is the adjustment value for Scale Factor
• SFimpact is the impact factor for the SF
• SFrating is the current SF rating
• EAFadj is the adjustment value for Effort Adjustment Factor (cost
driver)
• EAFimpact is the impact factor of the EAF
• EAFrating is the current EAF rating
• ANSavg is the normalized average of the answers
The resulting adjustment value based on the calculation can be either a negative or
a positive value indicating the number of steps that that parameter rating should be
increased or decreased to. However, since automatic adjustments may not always
be feasible, due to other uncontrollable factors in the estimating environment, the
64
adjustment values are scaled down to the range of 1 to 3. This means that the final
adjustment is a value between 1 and 3, where 1 represent a minor adjustment to
the SF or EAF value, while a 3 represents a major one. These values are suggested
to the estimators in order for them to make educated judgments to adjust the
COCOMO II parameter rating accordingly.
Note that the mapping between the ratings and the values is linear in this case.
This is valid and applicable to the scenario in which the experiment of the frame-
work was conducted. As discussed in Sections 2.3 and 3.3, the software engineering
projects at USC are small e-services projects consisting of development teams with
minimal average industry experience. The majority of the data from these projects
show that they are basically linear. The data suggest that a rating of “High” repre-
sents twice the value of “Low”. This may be due to the fact that the development
teams, or the estimators, were either inexperienced, or because the projects were
so small, with low complexities, that efforts were not exponentially affected.
This, however, may not be applicable to industry projects. As team sizes grow
and project complexities increase, the cost and schedule of the project are expo-
nentially impacted because there are numerous factors that influence them, such
as:
• team communication
• development integration
• project administrator and coordination
• mission priorities
65
• requirements volatility
In order to make the model applicable to industry projects, the values used for
mapping the ratings should be calibrated to the specific industry and domain. The
calibration of these values is beyond the scope of this research and this dissertation.
4.3.4 Assessment Frequency
Because the COTIPMO framework adopts the continuous assessment concept, the
assessment is intended to be performed consistently throughout the project life
cycle. As mentioned earlier, the frequency of the assessment, or the length of each
iteration, can be defined by the development team as necessary, depending on the
situation of the project and team.
For larger teams, where communication and team synchronization may be more
difficult, the assessments should be performed more often in order to prevent any
potential knowledge gaps and risks that could occur among the team members.
On the other hand, for small sized teams, the assessment periods can be spread
apart more because communication may not be as much of a problem compared to
larger teams. However, this is again based on the necessity of the team and project
situation.
To determine the appropriate assessment frequency for the USC software engi-
neering projects, which are representative of small e-services projects consisting of
6-8 team members, 2 different frequencies were tested with the projects. The stu-
dent members were then surveyed at the end of Fall 2011 and Spring 2012 semesters
asking them how helpful the COTIPMO framework was to their project.
66
During the Fall 2011 semester, projects were required to perform the assessment
weekly. The feedback received from students was that the repetitiveness of the
assessment became mundane after a period of time, because they felt that the
weekly assessment was done too often. The assessment tasks became more of a
routine job, rather than ones that would add more value. In comparison, during
the Spring 2012 semester, students were asked to perform the assessment every
other week. The feedback from the Spring 2012 semester was much more positive;
the team members felt that the evaluations of the assessment became more realistic
because the team had more time to take action and make more progress in the
project in between the assessment periods.
This suggests that having less frequent assessments may be more effective for
smaller teams with smaller projects. However, the assessments should be performed
often enough in between project milestones for the framework to be effective.
67
Chapter 5: The COTIPMO Tool
Having an effective tool is essential in enabling the potentials of the framework.
The COTIPMO tool [AHB12] implements the COTIPMO framework described in
the previous chapter. The main focus of the tool, in addition to the accuracy of the
implementation, is to reduce the tediousness that is often involved in performing
assessments. The tool automates many of the processes including:
• source code counting and reporting
• effort reporting
• estimation calculations
• survey generation
• survey assessment result analysis
• COCOMO II adjustment computation
This provides an effective mechanism for development teams to perform project
assessments and review the results.
5.1 Powered by Jazz
The IBM Jazz platform [IBM] has been chosen as the supporting platform for the
COTIPMO tool for its scalability and strong support for high collaborative envi-
ronment. Jazz provides the following foundational capabilities that are currently
utilized by COTIPMO:
68
• User management
• Project and team management
• Resource and repository management
• RESTful web services architecture
This allows the development of the COTIPMO tool to focus on implementing the
framework features and capabilities instead of other minor, less critical features.
Moreover, Jazz has a highly extensible architecture that allows collaboration and
integration with other life cycle applications and tools working under a single logical
server. This means that the COTIPMO tool can be extended to be used as part of
existing project management tools such as the Rational Team Concert [Rat]. The
potential extensions to the tool will be discussed in the future work section in this
dissertation.
The COTIPMO tool provides support for both traditional development projects
and rapid-fielding projects. As discussed in the previous chapter, the mechanisms
for tracking the project progress and estimations differ between the two project
types, but the methodologies and processes are exactly the same.
5.2 Support for Traditional Development
Projects
The COTIPMO tool was developed to provide substantial benefits for traditional
development projects due to the integration of the Unified Code Counter (UCC)
69
tool. The automatic code counting capability takes away the complexity of size and
progress reporting process enabling the development teams to quickly assess their
progress and productivity.
Figure 5.1: The iteration list for traditional development projects
Figure 5.1 shows the main screen of the COTIPMO tool for a traditional devel-
opment project. The screen consists of 3 main sections: 1) the “Initial Project
Estimates”, 2) the “Iteration List”, and 3) the “Project Progress”. The “Initial
Project Estimates” section allows the team to enter the initial estimates for the
project. Based on the information known at the beginning of the project, the
70
development team specifies the modules planned for development and all the cor-
responding COCOMO II information for each module.
As the project progresses, the team periodically adds iterations to the tool, which
show up in the “Iteration List” section. For each iteration, if the development of
source code, or the components, has not started, the team can make updates to
their estimates based on the team’s status and knowledge. Otherwise, they can
upload the source code files, and the COTIPMO tool automatically counts the
number of logical lines of code for each file using the UCC tool on the backend.
The tool accumulates all the lines of code for all modules and, using the COCOMO
II model, computes the equivalent effort. The developers then enter the percentage
developed, tested, and integrated for each module. All of these data are used to
compute the new estimated effort required to complete the project. The progression
of the project with respect to the team’s accumulated effort spent and estimated
effort are shown in graphical format in the ”Project Progress” section, which allows
the teams to see their development progress as well as the improvements to their
estimations. The detailed implementation of the source code tracking framework
was developed and discussed in [ASB10].
For every iteration created, the COTIPMO tool automatically generates a cor-
responding survey for each team member to complete. Figure 5.2 shows the survey
ballot to be submitted by each member. As mentioned in Section 4.2, the concept of
the survey assessment is largely based on the IBM Self-Check methodology [KK08].
Each team member fills out and submits the survey individually, without knowing
each other’s answers, in order to reduce any bias in the answers.
71
Figure 5.2: Survey ballot
Figure 5.3: Survey result - answers
Figure 5.3 displays the result of the survey showing the answers given by each
team member. The standard deviation is computed to detect any inconsistencies
between the answers for each question. A high deviation in the answers means that
there are differences in opinions or understandings within the team; thus, a flag is
triggered raising an issue for the team to discuss that specific question.
72
Once the discussions have taken place, the team then identifies actions to take
to resolve or prevent those issues in the upcoming iterations. Figure 5.4 shows the
list of all the actions and the corresponding iteration that they were identified in.
The team is able to mark each action as ”resolved” once they have successfully
addressed the originating issue and completed that specific action. This allows the
team to effectively keep track of the tasks that they need to perform as well as any
outstanding problems that exist within the team and project.
Figure 5.4: List of identified actions.
Finally, based on the survey results, the COTIPMO tool analyzes the survey
data and automatically computes the adjustments that should be made to the
COCOMO II parameters. The computation algorithm is discussed in Section 4.3.3.
These adjustments are suggested to the development team as shown in Figures
5.5 and 5.6. The suggestions are reflective of the answers given by all the team
members.
73
Figure 5.5: Survey Result - COCOMO II scale factors adjustment suggestions
Figure 5.6: Survey Result - COCOMO II cost drivers adjustment suggestions
Since each survey question contains different levels of impact on each of the
COCOMO II parameters, these suggestions are calculated based on the model dis-
cussed in Section 4.3. The number of arrows (1, 2, or 3 arrows) represents the level
74
of adjustments that should be made to the corresponding COCOMO II parameter.
One arrow represents a minor increase or decrease in the rating, while three arrows
suggest that a major change is required. The developers must then use judgements
to make any necessary adjustments to the COCOMO II ratings based on these
recommendations.
5.3 Support for Rapid-Fielding Projects
The COTIPMO tool also provides strong support for rapid-fielding projects. As
these projects are often aimed at customizing and configuring nondevelopmental
items (NDI), the COTIPMO framework utilizes the COCOMO II Application Point
estimation model in [BAB+00] for effort estimation and tracking. The process of
reporting progress in the Application Point model is much less complex compared to
the regular COCOMO II model. Instead of reporting the number of SLOC written,
the development team reports the number of screens, reports, and third genera-
tion language (3GL) components developed, customized, and configured. These
are called application points. Additionally, the Application Point model uses the
developer’s capability and experience and the integrated computer-aided software
engineering (ICASE) toolset maturity and experience levels for calculating the pro-
ductivity rate as well as the capability level of the team. The effort spent on
development and estimated effort required to complete the project are computed
based on this information.
75
Figure 5.7: The estimation list for rapid-fielding projects
Similar to the traditional development project, the team continuously adds iter-
ations as the project progresses. Figure 5.7 shows the main screen listing the iter-
ations for rapid-fielding projects. They report the number of application points
completed up to each iteration as well as the percentage developed and tested for
each application point. The tool uses the COCOMO II Application Point model to
76
compute the New Application Points (NAP), converts them into equivalent effort,
and computes the new estimates based on these data.
For every iteration, the team members are also required to complete the survey
assessments individually. However, instead of suggesting adjustments to the scale
factors and cost drivers, the COTIPMO tool analyzes the assessment data and sug-
gests changes to the developer’s capability and experience and ICASE maturity and
experience levels, which are the two dynamic parameters that affect the productiv-
ity rate of the team. Figure 5.8 shows the suggestions computed by the COTIPMO
tool for rapid-fielding projects.
Figure 5.8: Survey Result - Developer’s capability and experience level and ICASEmaturity and experience level adjustment suggestions
Since the impact factors have only been surveyed and computed for the 22
COCOMO II scale factors and cost drivers, the impact factors for the Application
Point model drivers are mapped to some of the cost drivers as shown in Table 5.1.
The mapping is done based on the definitions of the cost drivers ensuring that they
match with the definitions of the Application Point drivers. This allows for the
tool to compute and suggest the adjustments to the two Application Point drivers
– 1) the developer’s experience and capability and 2) the ICASE maturity and
capability.
77
Application Point Driver COCOMOIICost Driver
Developer’s experience and capability
ACAPPCAPAPEXPLEXLTEX
ICASE maturity and capabilityLTEXTOOL
Table 5.1: Application Point drivers vs. cost drivers
5.4 Survey Answer Guidance
In order for the development team to utilize and answer the survey effectively, each
of the survey questions contains guidance to what various types of answers mean.
The guidance is as follows:
• Low. A guidance for what it means when the answer is low for the question.
• Medium. A guidance to what the average answer means.
• High. A guidance for a high answer given to answering the question.
In developing the guidance to the survey questions, the relationship between
the question and the COCOMO II parameter are taken into consideration. This
makes sense since the way the questions are answered impacts how the COCOMO
II estimation adjustments are calculated. To develop the guidance, first, all of the
corresponding COCOMO II parameters that are impacted by the question were
identified. Then referring to the definition of those parameters and the rating
guidance defined in [BAB+00], the guidance was combined from all the cost drivers
78
and scale factors, and then rewritten to enable the guidance to make sense and be
readable.
Figure 5.9 shows the survey view with the answer guidance displayed next to
the survey question. Having the answer guidance ensures that the survey takers
understand the questions correctly and provide the proper answers to the questions.
Figure 5.9: Survey answer guidance
5.5 COTIPMO Tool Administration
The COTIPMO tool also provides the administrative end for project managers
and administrators to make the necessary changes and preparation for assessments.
Although the COTIPMO framework comes with all the survey assessment questions
with the corresponding impact factor to the COCOMO II parameters, projects can
make specific adjustments to the surveys as necessary. However, any adjustments
made to the assessment framework should be done with expert judgments and
research data.
79
5.6 Question Preparation
The question editor allows for the administrator to manage the existing set of
questions as well as enter new questions. Figure 5.10 shows the editor view for
editing a question. The editor allows the administrator to enter the question details
along with the answer guidance. Details on the answer guidance developed for the
current set of assessment questions are given in Section 5.4. Furthermore, the
bottom section of the editor is the place to specify the impact factor between the
question and the COCOMO II parameters.
Figure 5.10: Question editor
Additionally, once the questions have been managed, a question set must be
created and assign to project teams to use for assessment. More details on the
80
assigning question sets to project teams will be covered in the next section (Section
5.7). Figure 5.11 shows the screen for editing the question set. It enables the
administrator to pick questions from all the available questions to add to the set.
Figure 5.11: Question set editor
5.7 Project Management
Although the IBM Jazz technology provides capabilities for team management,
there are few capabilities that needed to be extended in order to satisfy the needs
of the COTIPMO tool. Specifically, the information about the project type (tradi-
tional development or rapid-fielding) needs to be specified as well as the question
set that would be used on a regular basis and during the project milestones. Figure
5.12 shows the screen where all of the mentioned information can be assigned to
the projects.
81
Figure 5.12: Project management page
82
Chapter 6: Evaluation and
Analysis
6.1 Improved Software Estimation Accuracies
First, the correctness of the COCOMO II estimation parameters was observed by
analyzing the parameter ratings provided by each team keeping track of the mistakes
made by them. The ratings were considered to be incorrect when they were not
appropriate or not applicable to the team or project status. For example, if the
team reported that their programmers’ experiences (APEX, PLEX, and LTEX)
were high, but the team consisted of members with less than 2 years of industry
experience, these were considered as incorrect. The analysis was only done for the
Valuation and Foundations phases as the estimates computed during these phases
had the most significant impact on the project scoping.
6.1.1 Results
Figure 6.1 shows the average mistakes in COCOMO II scale factors and cost drivers
ratings of all the teams. Since each project had a different number of modules, the
averages of the cost driver mistakes were taken across all modules for each project.
The scale factors are entered once at the project level. Both Fall 2009 and Fall
2010 semesters showed a consistent number of errors in the ratings and showed
no improvements as the projects progressed. However, in Fall 2011, the projects
83
0.00#
1.00#
2.00#
3.00#
4.00#
5.00#
6.00#
Valua.on# Founda.ons# Valua.on# Founda.ons# Valua.on# Founda.ons#
Fall#2009# Fall#2010# Fall#2011#
Num
ber'o
f'mistakes'
Project'phase'
Average'COCOMO'II'Driver'Mistakes'Avg#EAF#/#Module#/#Team# Avg#SF#/#Team#
Figure 6.1: Average number of mistakes in the COCOMO II ratings for scale factorsand cost drivers.
showed fewer mistakes in their estimations after the COTIPMO tool was introduced.
More importantly, though, the projects showed improvements over time with better
accuracies towards the end of the semester during the Foundations phase. To ensure
the validity of the data, selected sets of sample data were taken from each year and
were evaluated and analyzed by an independent party. The results of the mistakes
identified were consistent with the initial analysis.
For all three years, the projects were carefully reviewed by the stakeholders
at every major milestone. The review process included analyzing the accuracy,
correctness, and appropriateness of the project estimates (i.e., the COCOMO II
parameter ratings) and had been consistent for all three years. Based on these
84
results, it showed that even though the projects’ estimates were periodically eval-
uated by the stakeholders to point out any errors, without proper guidance and
direction for corrections, the estimates and COCOMO II parameter ratings were
not effectively improved.
The COTIPMO framework’s model-based method of collecting project data
from the beginning of the project life cycle enables the COCOMO II estimation
to be improved over time. As the projects progressed, the assessment data and
the development progress is used by the framework to help the development teams
correct their COCOMO II parameter ratings. This makes the estimates to reflect
the project and team status at the time the assessment is done, thus, making the
estimates more accurate.
6.1.2 Hypothesis Support: H1, H2b
The results of improved software estimation accuracies support the following
hypotheses:
Hypothesis 1. Software development teams that perform continuous assessments
with this framework will (not) improve project scoping and estimation.
Hypothesis 2b. The COCOMO II effort drivers and scale factors can (not) be
automatically adjusted to reflect real project situations through performing
survey-based assessments.
With fewer estimation parameter errors with the COCOMO II model and the
improved cost estimation accuracies, it was clearly shown that the teams using
85
the COTIPMO framework and tool developed better project estimates. This was
achieved by analyzing the assessment data and adjusting the COCOMO II param-
eters based on the suggestions provided by the framework. Thus, the two null
hypotheses are refuted by the experiment results.
6.2 Improved Project Issues and Defects Detec-
tion
Throughout the project life cycle, the development team – especially the IIV&V
personnel – independently reviewed the projects and their artifacts to detect any
inconsistencies and defects in the documentation as well as in the team understand-
ing in general. The issues and defects that were filed and resolved by the team were
analyzed and were categorized into the following severities: 1) blocker, 2) critical,
3) major, 4) normal, 5) minor, 6) trivial, and 7) enhancement. Since these were
project related issues, the data were normalized and re-categorized into 1) minor,
2) normal and 3) critical severities in order to make the observations more visible.
The blocker, critical, and major issues were considered to be critical, which included
any inconsistencies, misunderstandings, or errors that impacted the project at the
conceptual level. The rest of the data were categorized as normal and minor respec-
tively, which included insignificant defects such as grammatical, typographical, and
formatting errors.
86
6.2.1 Results
Figure 6.2 shows that the average number of critical defects and issues had decreased
during the Fall 2011 - Spring 2012 semesters even though the total number
filed remained consistent with the previous years. As projects had varied sched-
ule (1 semester vs. 2 semesters), depending on complexity, they were grouped
into the Valuation-Foundations period and Development period. The Valuation-
Foundations period was when projects went through the process of requirements
engineering and designs. On the other hand, the Development period was when the
teams went through the actual implementation. The periods were grouped this way
because all projects had to go through these phases regardless of the timeline.
!"!!!!
!20.00!!
!40.00!!
!60.00!!
!80.00!!
!100.00!!
!120.00!!
!140.00!!
!160.00!!
!180.00!!
F09"S10! F10"S11! F11"S12! F09"S10! F10"S11! F11"S12!
Valua1on!"!Founda1ons! !Development!
#"Issues/D
efects"
Phase"
Average"Issues/Defects"Per"Team"Cri1cal! Normal! Minor!
Figure 6.2: Average defects and issues filed by the team each year
The behavior of the issues and defects were analyzed further by observing the
rates that they were filed during each project phase. Figures 6.3 and 6.4 show
87
the number of issues and defects filed by the development team during each phase
for both 1 semester and 2 semester projects. The data showed that in Fall 2011 -
Spring 2012, the average number of critical defects and issues remained fewer than
the previous years through all the phases.
0.00#
20.00#
40.00#
60.00#
80.00#
100.00#
120.00#
F09+S10# F10+S11# F11+S12# F09+S10# F10+S11# F11+S12# F09+S10# F10+S11# F11+S12# F09+S10# F10+S11# F11+S12# F09+S10# F10+S11# F11+S12#
Explora4on# Vaua4on# Founda4ons# Rebaselined#Founda4ons# Development#
Num
ber#o
f#Issue
s/De
fects#
Phase#
Average#Issues/Defects#by#Phase#+#2+Semester#Projects#Cri$cal( Normal( Minor(
Figure 6.3: Average defects and issues by phase - 2 semester projects
It was especially interesting to observe the significant reduction during the Val-
uation phase. Since the requirements were negotiated and gathered during this
phase, the number of uncertainties and number of inconsistencies were expected to
be fairly high. However, with the use of the COTIPMO tool, the number of issues
and defects recorded was significantly reduced. This was possibly due to the fact
that the assessment mechanisms of the tool helped detect any inconsistencies and
potential problems that existed in the team early before they turned into critical
issues.
88
0.00#
20.00#
40.00#
60.00#
80.00#
100.00#
120.00#
F09# F10# F11# F09# F10# F11# F09# F10# F11#
Explora2on# Vaua2on# Development#
Num
ber#o
f#Issue
s/De
fects#
Phase#
Average#Issues/Defects#by#Phase#G#1GSemester#Projects##Cri$cal( Normal( Minor(
Figure 6.4: Average defects and issues by phase - 1 semester projects
6.2.2 Hypothesis Support: H2a
The results of improved project issues and defect detection support the following
hypothesis:
Hypothesis 2a. Software development teams that perform continuous assessments
will (not) be better synchronized and stabilized in understanding and concur-
rent engineering.
The results clearly showed that the number of critical defects and issues had
been reduced after the COTIPMO framework and tool had been introduced. With
fewer inconsistencies in understanding, more stable project concepts, and reduced
defects in project artifacts, the project teams showed that they were more stabilized
and synchronized, thus refuting the null hypothesis 2a.
89
6.3 Improved Project Tracking
The project tracking mechanism allows for development teams to constantly moni-
tor the resources required for the teams to complete their project. These estimated
resources can be updated as necessary depending on the team’s productivity and
capability.
6.3.1 Results
With the better progress tracking mechanism of the COTIPMO tool, 2 projects were
able to deliver before the anticipated deadline during the Fall 2011 semester. The
first project was initially planned for a 24-week schedule, but based on the progress
tracking and re-estimations reported by the tool, they were able to determine that
the project only required half the resources and could be completed within a 12-week
schedule instead. The project immediately proceeded into the construction phase
and the product was delivered to the client with 100% of end user functionalities
implemented.
In addition, another project had to be re-scoped due to the client’s shift in goals
and needs. The project was initially planned to deliver a fully developed system
in a 24-week schedule; however, the client changed the project scope and asked for
a prototype of the system with only a subset of the end-user functionalities. The
team updated their project sizes and estimates in the COTIPMO tool. Based on
the project progress and prototypes developed at that time, the tool reported that
they had enough resources to complete the newly scoped project within a 12-week
90
timeframe. The project was completed with the prototyped functionalities delivered
to the client.
In the previous years, when projects had to switch from a 24-week to a 12-week
schedule, they required major re-scoping of features and capabilities in order to
meet the new deadlines.
6.3.2 Hypothesis Support: H1, H3a, H3b
The results of improved project tracking support the following hypotheses:
Hypothesis 1. Software development teams that perform continuous assessments
with this framework will (not) improve project scoping and estimation.
Hypothesis 3a. The collaboration capabilities of the assessment framework will
(not) be cost-effective in improving COU initiation, reduction, and tracking.
Hypothesis 3b. Software development teams that utilize this continuous assess-
ment framework would (not) outperform others using traditional assessment
methods.
Out of 13 projects, 10 were planned for a 2-semester development timeline.
Two of the projects were able to deliver within half of the timeframe through bet-
ter project and resource tracking provided by the COTIPMO framework without
compromising the capability requirements or product quality. None of the Fall
2009 - Spring 2010 and Fall 2010 - Spring 2011 projects were able to achieve this.
This refutes the 3 null hypotheses as the projects clearly had better tracking and
outperformed previous teams by delivering full capabilities within half the time.
91
6.4 Improved Team Synchronization and Stabi-
lization
To analyze the level of synchronization and stabilization within the project teams,
every team member was surveyed on the various aspects of team and project perfor-
mance. The survey, which was conducted on the students from all 3 years, focused
on the following aspects:
• Level of project uncertainties
• Level of team synchronization
• Team’s strength
• Level of difficulties in communication
Since not all the projects completed the entire life cycle as discussed in Section
2.3, the surveys were counted only for projects that were completed and delivered
to the clients. This was to ensure that the performances were observed for any
improvements over the entire project life cycle.
6.4.1 Results
Figures 6.5 and 6.6 show that the overall levels of team synchronization and stabi-
lization improved significantly during the Fall 2011 - Spring 2012 semesters. The
level of uncertainties are notably lower towards the end of the projects after the
COTIPMO framework had been introduced. Although it was expected that projects
would have high levels of uncertainties towards the beginning of the project life
92
cycle, the projects in Fall 2011 - Spring 2012 semesters also showed signs of lower
uncertainties compared to the previous years.
0%#5%#10%#15%#20%#25%#30%#35%#40%#45%#50%#
Beginning# End# Beginning# End# Beginning# End#
F092S10# F102S11# F112S12#
%"of"T
otal"Stude
nts"
Semester"
Level"of"Uncertain6es"Very#Low# Low# Average# High# Very#High#
Figure 6.5: The level of uncertainties within project
0%#
10%#
20%#
30%#
40%#
50%#
60%#
70%#
Beginning# End# Beginning# End# Beginning# End#
F094S10# F104S11# F114S12#
%"of"T
otal"Stude
nts"
Semester"
Level"of"Team"Synchroniza8on"Very#Low# Low# Average# High# Very#High#
Figure 6.6: The level of synchronization among team members
93
Furthermore, with lower levels of uncertainties in the projects, the levels of team
synchronization had increased significantly over time with the use of the COTIPMO
framework and tool. Figure 6.6 shows that for all 3 years, the levels of team syn-
chronization were very similar towards the beginning of the project; however, in the
Fall 2011 - Spring 2012 year, nearly all the teams were in a well synchronized state
towards the end of the project.
0%#
10%#
20%#
30%#
40%#
50%#
60%#
70%#
Beginning# End# Beginning# End# Beginning# End#
F094S10# F104S11# F114S12#
%"of"T
otal"Stude
nts"
Semester"
Team's"Strength"Very#Weak# Weak# Average# Strong# Very#Strong#
Figure 6.7: Team’s strength
In addition to the improved level of synchronization, each team member was
asked to analyze on how strongly they viewed their team’s performance throughout
the project. Figure 6.7 shows that the team strengths dramatically improved over
time with the introduction of the COTIPMO framework. Although it may be
true that team cohesion and strength generally improve over time as the team
members work together, the improvements in the Fall 2011 - Spring 2012 semesters
were significantly higher compared to the two previous years that were somewhat
94
consistent. Nearly 100% of the team members felt that the teams were strong at
the end of the project after the use of the COTIPMO framework.
0%#5%#10%#15%#20%#25%#30%#35%#40%#45%#50%#
F09+S10# F10+S11# F11+S12#
%"of"T
otal"Stude
nts"
Semester"
Difficul5es"in"Communica5on"Very#Easy# Easy# Average# Difficult# Very#Difficult#
Figure 6.8: Difficulties in communication
Finally, the COTIPMO framework is intended to help improve the effectiveness
of team communication because it helps point out certain areas that need to be dis-
cussed and doing so without specifically pinpointing the weak team member. This
allows the team to develop solutions to their potential problems as a team instead
of as an individual. Figure 6.8 shows the level of difficulties communication that the
teams experienced during the past 3 years. It is clear that the communication had
greatly improved in the Fall 2011 - Spring 2012 year. More team members felt that
they had fewer difficulties in communicating with each other effectively. With the
COTIPMO framework and tool, the teams were able to detect any knowledge gaps
and potential problems. As the tool suggestively trigger discussions for specific
areas of the project, the collaboration level among the team members increased,
95
while the level of difficulties decreased, since the problems can be pinpointed as a
team.
6.4.2 Hypothesis Support: H2a, H3a
The results of improved team synchronization and stabilization support the follow-
ing hypotheses:
Hypothesis 2a. Software development teams that perform continuous assessments
will (not) be better synchronized and stabilized in understanding and concur-
rent engineering.
Hypothesis 3a. The collaboration capabilities of the assessment framework will
(not) be cost-effective in improving COU initiation, reduction, and tracking.
The outcomes of conducting continuous assessments have shown improvements
in the project performances. The results of the surveys have shown significant
improvements in project and team synchronization with reduced uncertainties as
well as improved team strengths. Ultimately, the use of the COTIPMO framework
has had significant impact on the levels of communication within the team as it
greatly reduced the difficulties in communicating among the team members. This
refutes the null hypotheses 2a and 3a.
6.5 Reduced Project Effort
In addition to the positive feedback received from the Fall 2011 - Spring 2012
students, the use of the COTIPMO tool was compared with the previous years
96
(Fall 2009 - Spring 2010 and Fall 2010 - Spring 2011 semesters) when the tool and
framework were not yet introduced. For every week during the project life cycle,
each team member was required to report the effort, in hours, they spent on the
project activities in the following categories:
1. Operational concepts development
2. Requirements engineering
3. Design and architecture
4. Planning and control
5. Feasibility evidence analysis
6. Quality management
7. Testing
8. Communication and synchronization
9. Performance control
Detailed categories of the effort report submitted by each member can be found in
Appendix E and re-categorized in Appendix F.
6.5.1 Results
The average effort spent by each team member on the project during each week in
the semester is shown in Figure 6.9. The effort required for the projects in Fall 2011
97
- Spring 2012 was significantly reduced each week compared to that of the previous
years.
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
40.00
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Hou
rs
Week #
Average Effort Distribution F09-S10 F10-S11 F11-S12
Figure 6.9: The average effort by week
Moreover, further analysis was done on the average of the total effort spent
on each of the project activities as shown in Figure 6.10. In many of the activi-
ties such as operational concepts development, requirements engineering, planning
and control, and feasibility evidence analysis, the average effort spent was slightly
lower than the previous years. Since the amount of work required to perform these
activities remained the same across all three years, it was expected that the aver-
age efforts would only show some slight reduction. However, the most significant
reduction in effort was in the areas of communication and synchronization. The
level of effort required for the team members to synchronize with each other and
to stabilize the team’s knowledge and understanding had greatly decreased. The
COTIPMO tool and framework provided the teams with an effective mechanism
98
to detect inconsistencies within the team and helped reduce knowledge gaps that
existed. With this, the teams were able to quickly resolve those issues and focus
more attention on completing the project instead of having to waste effort in trying
to work in an unsynchronized team.
0.00
20.00
40.00
60.00
80.00
100.00
120.00
140.00
Operat
ional
Concep
ts Dev.
Requir
emen
ts Eng
ineeri
ng
Design
and A
rchite
cture
Dev.
Plannin
g and
Contro
l
Feasibi
lity Evid
ence
Analys
is
Develo
pmen
t/Imple
mentat
ion
Quality
Man
agem
ent
Testing
Comm. a
nd Syn
c.
Perform
ance
Contro
l
Hou
rs
Average Effort by Activities F09-S10 F10-S11 F11-S12
Figure 6.10: The average effort by activities
In the same survey used and discussed in Section 6.4, each of the team member
was asked to rate the level of reduction in effort required to do the following:
• Risk identification and mitigation
• Team synchronization
• Project issue and problem resolution
99
Figures 6.11, 6.12, and 6.13 show the results of the survey. The levels of effort
required to synchronize and stabilize the team were significantly reduced during the
Fall 2011 - Spring 2012 semester with generally higher numbers of team members
rating “more reduction” and “greatly reduced” in effort.
0%#5%#10%#15%#20%#25%#30%#35%#40%#45%#50%#
No#reduc0on# Li4le#reduc0on#
Average# More#reduc0on#
Greatly#reduced#
%"of"T
otal"Stude
nts"
Effort"in"Risk"Mi6ga6on"F09@S10# F10@S11# F11@S12#
Figure 6.11: Effort reduction in risk identification and mitigation
0%#
5%#
10%#
15%#
20%#
25%#
30%#
35%#
40%#
45%#
No#reduc0on# Li4le#reduc0on#
Average# More#reduc0on#
Greatly#reduced#
%"of"T
otal"Stude
nts"
Effort"in"Team"Synchroniza8on"F09@S10# F10@S11# F11@S12#
Figure 6.12: Effort reduction in team synchronization
100
0%#
5%#
10%#
15%#
20%#
25%#
30%#
35%#
40%#
45%#
No#reduc0on# Li4le#reduc0on#
Average# More#reduc0on#
Greatly#reduced#
%"of"T
otal"Stude
nts"
Effort"in"Project"Issue"Resolu8on"F09@S10# F10@S11# F11@S12#
Figure 6.13: Effort reduction in project issue and problem resolution
Furthermore, since the COTIPMO framework had been designed to have low
complexity in the assessment process, the team members were surveyed to see how
much time was required for them to perform the assessment during each period
in order to analyze its cost effectiveness. On average, each person spent about 10
minutes per assessment. This is equivalent to about 120 minutes (2 hours) of effort
per person for a 1-semester project and 240 (4 hours) minutes for a 2-semester
project. With a team consisting of 6-7 members, this adds roughly 60-70 minutes
of overhead cost per project per assessment period.
Figure 6.14 shows the average effort spent by each student during the Fall and
Spring semesters for all three academic years. Despite the additional overhead cost
of 2 hours person per semester for the assessment, experiment results showed that
the total overall effort of the project was decreased significantly (over 50 hours per
person in the Fall and 30-100 hours per person in the Spring). It is clear that using
the COTIPMO framework and tool is cost effective in projects.
101
0!
50!
100!
150!
200!
250!
300!
Fall Average! Spring Average!
Hou
rs!
Semester!
Average Effort Per Person!F09-S10! F10-S11! F11-S12!
Figure 6.14: Average effort per person
6.5.2 Hypothesis Support: H3a
The results of reduced project effort support the following hypotheses:
Hypothesis 3a. The collaboration capabilities of the assessment framework will
(not) be cost-effective in improving COU initiation, reduction, and tracking.
The level of effort required for the projects was significantly reduced with the
use of the COTIPMO framework and tool. Although there was a small amount of
overhead in performing the assessments, the results clearly showed that the over-
all effort had significantly decreased, while the level of effort required to mitigate
risks, synchronize teams, and resolve project issues had significantly improved. This
refutes the null hypothesis 3a as the use of the COTIPMO framework and tool have
proven to be very cost effective.
102
6.6 Improved Client Satisfaction
During each semester, the project clients evaluated their development teams stating
how satisfied they were with the product delivered as well as the performance of
the team throughout the project life cycle. In the evaluation process, the clients
answered a series of questions, qualitative and quantitative. The quantitative results
were then compiled, computed, and scaled to a maximum score of 20 points.
There were two sets of evaluation questions that were used to survey the clients
for their satisfaction. Both sets had the same core focus of evaluating the team’s
performance and the project’s success; however, there were also different aspects
that were analyzed depending on the time in the project life cycle that the evaluation
was performed. The first set was for the clients to evaluate the project teams during
the middle of the project life cycle. These were essentially the projects that are
planned for a 2-semester timeframe, and the evaluations were given out at the end
of the Fall semester. The evaluation focused how the project performed during the
Valuation and Foundation phases and to see if the project should continue based
on how the clients viewed the development teams.
On the other hand, the second set of evaluation questions were surveyed at the
end of the project life cycle. This included 1-semester projects, where the products
are delivered at the end of the Fall semester, and 2-semester projects with the Spring
semester timeline. Basically, the evaluation was performed after the products had
been delivered to the clients. In addition to the team’s performance and project
success, this set added focus to the delivery process as well as the overall satisfaction
with the working product.
103
6.6.1 Results
Figures 6.15 and 6.16 shows the results of the client evaluations from Fall 2009 to
Spring 2012. Out of all the evaluations over the 3 years, the lowest evaluation score
given to a team by the client was 15; therefore, the chart was created showing the
range of scores from 15 to 20. Comparing all the Fall semesters in Figure 6.15,
clients of Fall 2011 had the highest number of fully satisfied clients (38.5%) out of
all the teams. Additionally, the semester also had the lowest number of unsatisfied
clients when looking at the scores ranging from 15 to 17.
28.6%&
7.1%&
0.0%&
28.6%&
35.7%&
0.0%&
25.0%&
12.5%&
37.5%&
25.0%&
7.7%&
15.4%& 15.4%&
23.1%&
38.5%&
0.0%&
5.0%&
10.0%&
15.0%&
20.0%&
25.0%&
30.0%&
35.0%&
40.0%&
45.0%&
15&<&x&<=&16& 16&<&x&<=&17& 17&<&x&<=&18& 18&<&x&<=&19& 19&<&x&<=&20&
%"of"T
eams"
Score"
Client"Evalua5on"6"Fall"Semester"
Fall&2009& Fall&2010& Fall&2011&
Figure 6.15: Client evaluation results - Fall semester
Furthermore, the satisfaction levels during the Spring semesters showed an even
higher level of satisfaction from the clients. As mentioned earlier, the end of the
Spring semester marked the end of the life cycle for all the projects, and the products
104
0.0%$ 0.0%$
20.0%$ 20.0%$
60.0%$
20.0%$
40.0%$
0.0%$ 0.0%$
40.0%$
0.0%$ 0.0%$ 0.0%$
60.0%$
40.0%$
0.0%$
10.0%$
20.0%$
30.0%$
40.0%$
50.0%$
60.0%$
70.0%$
15$<$x$<=$16$ 16$<$x$<=$17$ 17$<$x$<=$18$ 18$<$x$<=$19$ 19$<$x$<=$20$
%"of"T
eams"
Score"
Client"Evalua5on"6"Spring"Semester"
Spring$2010$ Spring$2011$ Spring$2012$
Figure 6.16: Client evaluation results - Spring semester
were delivered and transitioned to the clients. During the Spring 2012 semester,
100% of the clients were satisfied with the projects, giving them all high scores
ranging from 18 to 20. In the previous years, scores were more spread out with a
higher number of unsatisfied clients.
6.6.2 Hypothesis Support: H3b
The results of improved client satisfaction support the following hypotheses:
Hypothesis 3b. Software development teams that utilize this continuous assess-
ment framework would (not) outperform others using traditional assessment
methods.
105
With higher client satisfaction, the results clearly showed that the project teams
that used the COTIPMO framework and tool outperformed the teams that did not.
This refutes the null hypothesis 3b.
6.7 Threats to Validity
This section discusses some of the potential threats to the validity of the research
data and the methods to reduce them.
Representativeness of projects. Most projects were small e-services projects,
which may not represent the industry at a larger scale. Nonetheless, the
projects were done for real clients with real fixed schedules and costs. Also, all
projects followed similar incremental development process and project activ-
ities to those used in the industry.
Representativeness of personnel. The student teams and their members may
not be representative of industry personnel since the majority of them had
less than 2 years of industry experience. However, even though the on-campus
students may be less experienced, the off-campus students and the clients were
working professionals.
Validity of assessment data. Since many of the survey questions were derived
from strengths, weaknesses, and issues observed from software engineering
students, the assessed data may not be valid in the industry. However, the
verifications of the survey questions were done using the SE Performance
106
Risk and SE Competency Risk frameworks, which are frameworks used in the
industry.
The unavoidable Hawthorne effect. The student teams of Fall 2011 were aware
that the experiments were being conducted with the COTIPMO tool. This
may have had some level of impact on the way the teams developed their
project estimates using the COCOMO II model built into COTIPMO, and
caused the classic Hawthorne effect on the experiment. However, for all three
years, the student teams were evaluated and graded based on their perfor-
mance. This means that all the teams were equally motivated to produce
correct and accurate estimation results and project outcomes. Additionally,
the set of students changed every year, which meant that they had no past
experiences or familiarity with the course and the process. It is arguable that
the Hawthorne effect could be minimal because the student teams simply fol-
lowed the process instead of focusing on the experiment itself. Furthermore,
more attention and interest were put into observing the improvements to the
COCOMO II parameter ratings that occurred over time during the Fall 2011
- Spring 2012 semesters, whereas the 2009 and 2010 years showed no signs of
improvements.
Changes in the life cycle process from Fall 2009 - Spring 2012. The life
cycle process remained largely the same from Fall 2009 to Spring 2012. A
minor change in the requirement negotiation process was introduced during
the Fall 2011 semester with the use of a new negotiation tool and the
Planning Poker concept in the requirement prioritization process. However,
107
the core process and practice were still based on the WinWin process in
[BBHL95], [Lee96], and [BEK+98]. Additionally, the team size was reduced
to 6 team members instead of 7 members from the previous years. This may
have slightly affected the level of synchronization required among the team
members; however, the focus was on observing the level of effort reduced in
communication and synchronization. Furthermore, it is arguable that the
decrease in 1 team member would not have had such significant impact. Also,
each year the amount of functionality provided by COTS, or NDI, products
and cloud services has increased, which was the main reason for reducing the
team size in Fall 2011 - Spring 2012 year.
6.8 Reducing the Threat to Validity
One of the biggest threats to validity for this research is the representativeness of the
personnel as the experiments were conducted on student projects, and the students
had minimal industry experience. The projects themselves are considered to be
representative because they were done for real clients with real cost and schedule
utilizing industry level process.
In order to reduce the potential threat to validity caused by the representative-
ness of the personnel, further analysis was performed on the data by evaluating the
data and project performances based on the level of industry experience within the
team. Each year, the students were required to report their background informa-
tion and level of experience. This information included their general background
108
information, years of industry experience, and various capabilities in software engi-
neering practices. Details on the student background questionnaire can be found in
Appendix M.
This information was gathered to determine the average level of industry expe-
rience of the team. Table 6.1 summarizes the results by category:
1. 0 to 1 year of industry experience
2. 1 to 3 years of industry experience
3. More than 3 years of industry experience
Years of Experience Fall2009
Spring2010
Fall2010
Spring2011
Fall2011
Spring2012
Less than 1 year 4 1 2 2 2 11 to 3 years 9 3 6 2 9 33+ years 1 1 1 1 2 1
Table 6.1: Project team industry experience per semester
This data was used to determine any significant differences in project perfor-
mance when the industry experience varies. The analysis was done on the following
data:
• Project effort
• Project issues and defects
Both of these data were evaluated and discussed in Sections 6.2 and 6.5; however,
they were chosen for further analysis because they have significant impact on the
project performance (level of synchronization and stabilization) and project esti-
mation (effort).
109
6.8.1 Average Project Effort Based on Experience Level
Figures 6.17, 6.18, and 6.19 show the average effort spent on the project each week
by the development teams of various industry experience levels. The results were
fairly consistent among all the experience levels, with the effort in Fall 2011 - Spring
2012 year being at least equal to or less than that of the previous years.
It is interesting to note the difference in effort reduction between the Fall and
Spring semesters. For those teams with little to no industry experience, the level
of effort was somewhat consistent with that of the year before, but less during the
development period in the Spring semester. On the other hand, teams with more
experience performed better and had less weekly effort during the Fall semester after
the COTIPMO framework was introduced. This may be due to the fact that teams
with more experience were able to utilize the framework more effectively than the
less experienced teams because of better understanding of the software engineering
0"
5"
10"
15"
20"
25"
30"
35"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10"11"12"13"14"15"16"17"18"19"20"21"22"23"24"25"
Hours&
Week&
Average&Effort&by&Week&Less&than&1&Year&of&Industry&Experience&
F09-S10" F10-S11" F11-S12"
Figure 6.17: Average weekly effort for projects with less than 1 year of industryexperience
110
0"
5"
10"
15"
20"
25"
30"
35"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10"11"12"13"14"15"16"17"18"19"20"21"22"23"24"25"
Hours&
Week&
Average&Effort&by&Week&1&to&3&Years&of&Industry&Experience&
F09-S10" F10-S11" F11-S12"
Figure 6.18: Average weekly effort for projects with 1 to 3 years of industry expe-rience
0"
10"
20"
30"
40"
50"
60"
70"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10"11"12"13"14"15"16"17"18"19"20"21"22"23"24"25"
Hours&
Week&
Average&Effort&by&Week&More&than&3&Years&of&Industry&Experience&
F09-S10" F10-S11" F11-S12"
Figure 6.19: Average weekly effort for projects with more than 3 years of industryexperience
and software development processes. The Fall semester was when all projects went
through the Valuation and Foundations phases.
111
Regardless of the observed effort reduction, some of the sample sizes may be
too small to be considered significant. Each year, out of all the projects, only 1
- 2 development teams had the average industry experience of more than 3 years.
However, the consistent results among the experience levels suggest similar levels
of benefits with the use of the COTIPMO framework.
6.8.2 Project Issues and Defects Based on Experience Level
In addition to analyzing the project effort, the project issues and defects were
analyzed to observe the level of team synchronization and stabilization among the
projects. Figure 6.20 shows the level of issues and defects that existed within the
project based on various experience levels.
4"team
s"(25"stud
ents)"
2"team
s"(15"stud
ents)"
2"team
s"(12"stud
ents)"
9"team
s"(45"stud
ents)"
5"team
s"(38"stud
ents)"
9"team
s"(56"stud
ents)"
1"team
s"(7"stud
ents)"
1"team
s"(7"stud
ents)"
2"team
s"(14"stud
ents)"
1"team
s"(5"stud
ents)"
2"team
s"(9"stud
ents)"
1"team
s"(4"stud
ents)"
3"team
s"(15"stud
ents)"
3"team
s"(15"stud
ents)"
3"team
s"(15"stud
ents)"
1"team
s"(5"stud
ents)"
1"team
s"(3"stud
ents)"
1"team
s"(6"stud
ents)"
"5""""
"50.00""
"100.00""
"150.00""
"200.00""
"250.00""
"300.00""
"350.00""
"400.00""
F09" F10" F11" F09" F10" F11" F09" F10" F11" S10" S11" S12" S10" S11" S12" S10" S11" S12"
0"to"1"yrs" 1"to"3"yrs" 3+"yrs" 0"to"1"yrs" 1"to"3"yrs" 3+"yrs"
Fall" Spring"
#"Issues/D
efects"
Years"of"Experience"
Average"Issues/Defects"Per"Team"based"on"Industry"Experience"Minor"
Normal"
CriEcal"
Figure 6.20: Average project issues and defects by industry experience
112
The results clearly show consistent benefits from the use of the COTIPMO
framework across all experience levels. The number of critical defects and issues
was significantly reduced during the Fall 2011 - Spring 2012 semesters. Although
there may be inconsistent peaks in the data, such as the highly experienced team in
Fall 2009 with a significantly higher number of defects, the overall trend appears to
be consistent across the entire data set. The data suggest that after the introduction
of the COTIPMO framework, the teams had improved levels of team synchroniza-
tion and stabilization with reduced numbers of issues and defects regardless of the
average level of industry experience of the development team.
6.8.3 Experiment with Project from SDSU
To validate the COTIPMO framework and tool outside of the USC setting, the
experiment was conducted with 1 project from San Diego State University (SDSU).
The project was a small-sized e-service project consisting of 3 developers. The
project followed a similar ICSM development process and used the COTIPMO tool
every other week.
Similar with the USC projects, the SDSU student team was surveyed at the
beginning and at the end of the project. Each team member was asked to analyze
the level of team synchronization and the level of project uncertainties as well as
how strongly they viewed the team. As shown in Figure 6.21, the survey results
clearly show that at the end of the project, the level of team synchronization and
team’s strength had increased significantly, while the level of project uncertainties
had decreased.
113
0%#
20%#
40%#
60%#
80%#
Beginning# End#
%"of"T
otal"Stude
nts"
Level"of"Team"Synchroniza8on"Very#Low# Low# Average# High# Very#High#
(a) Level of Team Synchronization
0%#
20%#
40%#
60%#
80%#
Beginning# End#
%"of"T
otal"Stude
nts"
Level"of"Uncertain5es"Very#Low# Low# Average# High# Very#High#
(b) Level of Uncertainties
0%#
20%#
40%#
60%#
80%#
Beginning# End#
%"of"T
otal"Stude
nts"
Team's"Strength"Very#Low# Low# Average# High# Very#High#
(c) Team’s Strength
Figure 6.21: Survey results from SDSU
Furthermore, each team member was asked to report on the levels of effort
required to mitigate risks, to resolve project issues, and to synchronize the team.
Figure 6.22 shows that the majority of the students reported significant reduction
in effort required for risk mitigation and team synchronization, while the reduction
for resolving project issues was only average.
Although, there were no past project data to compare the performances of the
projects that utilized the COTIPMO framework with those that did not, the behav-
iors of project performance improvements of the SDSU project are consistent with
the data gathered and observed from the USC projects. This suggests that the use
114
0%#
10%#
20%#
30%#
40%#
50%#
60%#
70%#
Risk#mi0ga0on# Project#issues#resolu0on# Team#synchroniza0on#
%"of"T
otal"Stude
nts"
Student"Reported"Effort"Reduc5on"
No#reduc0on# LiDle#reduc0on# Average# More#reduc0on# Greatly#reduced#
Figure 6.22: Reported effort reduction by SDSU students in risk mitigation, projectissue resolution, and team synchronization
of the COTIPMO framework and tool are also applicable in other project environ-
ments as well.
6.8.4 Discussion
Dividing the data into different experience levels allows a more in-depth analysis of
the benefits generated by the COTIPMO framework. The further analysis of the
data based on various levels of industry experience suggests that experience levels
have insignificant influence on the results achieved from utilizing the COTIPMO
framework and tool. In addition to being consistent among various experience
levels, the results were also consistent with the overall results discussed earlier in
this chapter.
115
The results of the analysis contributes to reducing the potential threat to validity
of the representativeness of the personnel. The consistent outcomes in the analysis
suggest that non-experienced students were as equally representative as the students
with higher industry experience, who were already representative of the industry.
Furthermore, since the clients, the projects, and the process are also representative
of the industry, they help contribute to minimizing the personnel threat as well.
Finally, the COTIPMO framework and tool had been experimented with 1 soft-
ware engineering project from San Diego State University (SDSU). The results of
the experiment were consistent with that of the USC software engineering projects.
Although the one project from SDSU was a very small and insignificant sample size,
the fact that it had consistent results outside of the USC setting and environment
suggests the applicability and suitability of the framework.
116
Chapter 7: Conclusions and
Future Work
7.1 General Conclusions
A team improvement framework for continuous assessment has been developed to
aid in the team synchronization and stabilization process by reducing the levels of
uncertainties that exist in a project and among the team members. The unknowns
and uncertainties in a project can be greatly reduced with the use of proper assess-
ment mechanisms. However, since team assessments can be tedious, labor intensive,
and require a high level of expertise and time for data analysis, they are often over-
looked as effective means for team improvements.
The COnstructive Team Improvement Process MOdel, or COTIPMO, has
been introduced to help software development teams reach a higher level of syn-
chronization and stabilization without having to go through complex processes. It
consists of three main parts:
1. Project progress tracking. For tracking the project progress based on
developed components.
2. Continuous team assessment. For the development team to perform
survey-based assessments in order to detect any knowledge gaps and potential
issues within the team.
117
3. COCOMO II estimation adjustment. For analyzing the assessment data
and computing suggestions to the COCOMO II estimation as necessary in
order to make the estimates more accurate and realistic.
The process framework provides mechanisms for software development teams to
quickly track their project progress based on the amount of development completed
and to detect issues and knowledge gaps within the team through its quick assess-
ment method. As the team continuously performs the assessment throughout the
project life cycle, uncertainties are reduced, while team and project understandings
increase. Additionally, the framework provides suggestions to the adjustments that
should be made to the COCOMO II estimates created by the team. This allows the
development team to continuously monitor the accuracy of their project estimates
and make rational adjustments to them as necessary.
The framework provides strong support for both traditional development
projects and rapid-fielding projects. For traditional development projects, the
COTIPMO framework relies on the Unified Code Counter (UCC) tool to report
the project progress based on the SLOC developed and uses the COCOMO II esti-
mation model for effort conversions. For rapid-fielding projects, on the other hand,
the framework uses the COCOMO II Application Point model to track the num-
ber of screens, reports, and 3GL components completed by the developers. The
assessment framework of COTIPMO analyzes the team’s survey assessment data
and provides improvement suggestions to the parameters used for estimation cal-
culation in both COCOMO II models.
118
The COTIPMO framework and tool were introduced and deployed in the USC’s
graduate software engineering course consisting of 79 graduate students and 13
projects. As shown in the analysis, the utilization of the COTIPMO framework
and tool provided the teams with the mechanism to effectively synchronize and
stabilize the teams in various areas such as communications, understanding, and
performance. The teams showed significant improvements in project estimates and
performance. With simple and effective assessments, the teams’ performance had
greatly improved with reduced uncertainties, while the effort required for the project
had substantially decreased. With better estimates and effective project tracking
mechanisms, the teams were able to constantly monitor the progress and the fea-
sibility of their projects, ensuring that the scopes could be delivered within the
available resources. Their estimates became more accurate and realistic over time,
reflecting the actual teams’ performances, while the team members were better
synchronized and stabilized because potential problems could be detected early on
before they became critical issues or defects.
Ultimately, the teams were able to spend less effort on the projects in order to
achieve an equivalent or a higher level of quality of work compared to the previous
years. Furthermore, some projects were able to deliver earlier than planned as they
were able to effectively track their development progress, recalibrate their resource
estimations, and realize the ability to complete within a shorter timeframe.
In conclusion, with improved project tracking, estimation, and assessments, the
use of the COTIPMO framework has shown fewer uncertainties and higher levels
of stabilization within the project. This implies a narrower Cone of Uncertainty as
119
shown in Figure 7.1. The cone has not yet been calibrated to the projects to get
the exact values for the corresponding relative size and cost ranges. The narrower
cone is implied from the anecdotal evidence obtained from the experiment results.
Figure 7.1: A narrower Cone of Uncertainty
This dissertation focused on presenting the framework for improving team per-
formance, team synchronization, and project progress tracking with a model devel-
oped to apply to specific software development scenarios. The results achieved
from the experiments are applicable to the settings and scenarios that they were
conducted in. They are valid for small sized e-services software engineering projects
with development teams consisting of members with little industry experience. The
results, however, are suggestive to being applicable to larger industry projects of var-
ious domains, but with necessary adjustments to the model framework and project
processes.
120
7.2 Summary of Contributions
The main contributions of this research are as follows:
• Investigation and analysis of current project tracking and assessment methods.
• The process model framework for improving team performance.
• The tool implementing the framework to help realize the potential benefits of
the framework itself.
• The empirical results showing the resulting improvements over previous years.
This dissertation describes the COTIPMO framework developed to aid software
development teams in tracking the project progress, assessing the team status and
performance, and improving their cost and effort estimations in the process. In
the process of developing the framework, various project tracking and assessment
frameworks as well as software sizing and estimation mechanisms were investigated
and analyzed. The framework has been developed using the COCOMO II Esti-
mation and Application Point models for project progress tracking and estimation
and the IBM Self-Check concepts for the continuous assessment. The COTIPMO
tool, an implementation of the framework, was developed with the integration of
the UCC tool and powered by the IBM Jazz technology on the backend. To vali-
date the hypotheses, data have been collected from 3 years of Software Engineering
courses comparing the various performance aspects among development teams that
utilized the framework and teams that did not.
121
7.3 Future Work
This section discusses some of the future work and potential research directions.
Re-compute COCOMO II impact factor based on statistical analysis.
As discussed in Section 4.3.1, there was not enough data to observe any correla-
tion significance between the assessment questions and the team performance. Once
enough data is gathered, the analysis should be performed again in order to identify
the actual correlation with the team performance. This will allow direct relation-
ships, and perhaps, more accurate impact factors between the assessment questions
and the COCOMO II parameters.
Experiment with industry projects of various sizes and domains.
Although the projects done in the USC Software Engineering class were compa-
rable to similar small e-service projects in the industry, the experiment should be
repeated on industry projects with various sizes and domains in order to verify and
validate the effectiveness of the COTIPMO framework and tool when used at a
similar and on a larger scale.
Extend the COTIPMO tool to use with project life cycle applications.
The COTIPMO tool can be integrated with some configuration management tools
such as Subversion, CVS, or Rational Team Concert. This would enhance the
software development progress tracking process to be more automated as the tool
can constantly monitor the source code checked in to the version control system.
The tool can also utilize the differencing function of the UCC allowing it to count
the number of SLOC that were added, modified, and deleted from the previous
122
version in addition to the total logical SLOC. This would make the tracking of
progress even more accurate and more realistic.
Explore and integrate the use of software measurement techniques.
Currently, the COTIPMO tool and framework reports the progress made to the
project during each assessment period. However, it does not deal with the actual
productivity levels of the developers. Some of the potential methods that can be
used include the Personal Software Process [Hum95] and the Team Software Process
[Hum99]. Both of these processes provide practical ways to measure the time spent
on each activity and their effectiveness as well as ways to determine the capabilities
of the team and its personnel.
Utilize the Earned-Value Management approach. Due to the small size
of the projects and the small number of modules per project, the earned-value man-
agement concept was not used in the computation of the estimation, as discussed
in Section 4.1. When the framework is used on industry projects, the earned-value
management approach can be applied to the estimation calculations, which can
provide more realistic values for estimation.
Deal with the second Cone of Uncertainty. Although the initial estimates
for cost and schedule are important in determining the time, resources, and budget
required for project completion, the ability to adapt the estimations to changing
environments, requirements, and performances allows teams to better control qual-
ity and help ensure timely deliveries of the products. Figure 7.2 shows that a second
Cone of Uncertainty is introduced to projects during the design and implementation
phases due to uncertainties with technologies, organizations, personnel, and project
123
priority shifts. This effect is relatively small for 24-week projects, but much more
significant for large projects. Effectively dealing with issues and problems related to
this cone early on in the project can help development teams easily accommodate
changes in future increments with less overhead and costs.
Feasibility
Concept of Operation
Rqts. Spec.
Plans and
Rqts.
Product Design
Product Design Spec.
Detail Design Spec.
Detail Design
Devel. and Test
Accepted Software
Phases and Milestones
RelativeCost Range x
4x
2x
1.25x
1.5x
0.25x
0.5x
0.67x
0.8x
Uncertainties in competition, technology, organizations,
mission priorities
Figure 7.2: The second Cone of Uncertainty
124
Bibliography
[AK99] Janaina C. Abib and Tereza G. Kirner. A GQM-based tool to supportthe development of software quality measurement plans. SIGSOFTSoftw. Eng. Notes, 24:75–80, July 1999.
[AHB12] Pongtip Aroonvatanaporn, Thanida Hongsongkiat, and Barry Boehm.Improving software development tracking and estimation inside the coneof uncertainty. Technical Report USC-CSSE-2012-504, University ofSouthern California, 2012.
[AKB12] Pongtip Aroonvatanaporn, Supannika Koolmanojwong, and BarryBoehm. COTIPMO: A COnstructive Team Improvement ProcessMOdel. In Proc. of the 2012 Int. Conf. on Software and Systems Process(ICSSP’12), Zurich, Switzerland, 2012.
[ASB10] Pongtip Aroonvatanaporn, Chatchai Sinthop, and Barry Boehm.Reducing estimation uncertainty with continuous assessment: track-ing the “cone of uncertainty”. In Proc. of the IEEE/ACM Int. Conf. onAutomated Software Engineering (ASE’10), pages 337–340, Antwerp,Belgium, 2010.
[AO11] I. Attarzadeh and S.H. Ow. Improving estimation accuracy of theCOCOMO II using an adaptive fuzzy logic model. In Fuzzy Systems(FUZZ), 2011 IEEE International Conference on, pages 2458 –2464,june 2011.
[BAB+00] Barry Boehm, Chris Abts, A. Winsor Brown, Sunita Chulani, Brad-ford K. Clark, Ellis Horowitz, Ray Madachy, Donald Reifer, and BertSteece. Software Cost Estimation with COCOMO II. Prentice-Hall,2000.
[Bas95] Victor R. Basili. Applying the Goal/Question/Metric paradigm in theexperience factory. In N. Fenton, R. Whitty, and Y. Lizuka, editors,Software Quality Assurance and Measurement: Worldwide Perspective,pages 21–44. International Thomson Computer Press, 1995.
125
[BBHL95] Barry Boehm, Prasanta Bose, Ellis Horowitz, and Ming June Lee. Soft-ware requirements negotiation and renegotiation aids. In Proc. of the17th Int. Conf. on Software Engineering (ICSE ’95), pages 243–253,Seattle, Washington, United States, 1995.
[BBI+09] Barry Boehm, Winsor Brown, Dan Ingold, Jo Ann Lane, George Fried-man, Kathleen Dangle, Linda Esker, Forrest Shull, Rich Turner, JonWade, Mark Weitekamp, Paul Componation, Sue O’Brien, Dawn Saba-dos, and Julie Fortune. Early identification of SE-related program risks.Technical Report SERC-2009-TR-001, Systems Engineering ResearchCenter, September 2009.
[BEK+98] Barry Boehm, Alexander Egyed, Julie Kwan, Dan Port, Archita Shah,and Ray Madachy. Using the WinWin Spiral Model: A case study.Computer, 31:33–44, July 1998.
[BL07] Barry Boehm and Jo Ann Lane. Using the Incremental CommitmentModel to integrate system acquisition, systems engineering, and soft-ware engineering. CrossTalk, October 2007.
[Boe81] Barry Boehm. Software Engineering Economics. Prentice-Hall, 1981.
[Boe89] Barry Boehm. Software Risk Management. IEEE Computer SocietyPress, 1989.
[Boe11] Barry Boehm. Some future software engineering opportunities and chal-lenges. In Sebastian Nanz, editor, The Future of Software Engineering,pages 1–32. Springer, Berlin, Heidelberg, 2011.
[BPHB02] Barry Boehm, Dan Port, Li-Guo Huang, and Winsor Brown. Using TheSpiral Model and MBASE to generate new acquisition process models:SAIV, CAIV, and SCQAIV. CrossTalk, pages 20–25, January 2002.
[Coc04] Alistair Cockburn. Earned-value and burn charts (burn up and burndown). In Crystal Clear. Addison-Wesley, 2004.
[Coh06] Mike Cohn. Agile Estimating and Planning. Prentice-Hall, 2006.
[Far70] John A. Farquhar. A preliminary inquiry into the software estimationprocess. Technical Report RM-7271-PR, The Rand Corporation, SantaMonica, CA, 1970.
[GE06] Daniel D. Galorath and Michael W. Evans. Software Sizing, Estimation,and Risk Management. Auerbach Publications, 1 edition, March 2006.
126
[Gre02] James Grenning. Planning poker. http://renaissancesoftware.net/files/articles/PlanningPoker-v1.1.pdf, April 2002.
[HHRC07] Xishi Huang, Danny Ho, Jing Ren, and Luiz F. Capretz. Improvingthe cocomo model using a neuro-fuzzy approach. Appl. Soft Comput.,7(1):29–40, January 2007.
[Hum95] Watts S. Humphrey. A Discipline for Software Engineering. Addison-Wesley, 1995.
[Hum99] Watts S. Humphrey. Introduction to the Team Software Process.Addison-Wesley, 1999.
[IBM] IBM. Jazz foundation. https://jazz.net/projects/
jazz-foundation/. [Online; accessed 3-May-2012].
[Jar00] Janne Jarvinen. Measurement based continuous assessment of softwareengineering processes. PhD thesis, Technical Research Centre of Fin-land, 2000.
[KB10] Supannika Koolmanojwong and Barry Boehm. The Incremental Com-mitment Model process patterns for rapid-fielding projects. In Proc.of the 2010 Int. Conf. on New Modeling Concepts for Today’s SoftwareProcesses: Software Process (ICSP’10), pages 150–162, Paderborn, Ger-many, 2010.
[KK08] Per Kroll and William Krebs. Introducing IBM Rational Self Checkfor software teams. http://www.ibm.com/developerworks/rational/library/edge/08/may08/kroll_krebs, May 3 2008. [Online; accessed20-January-2012].
[KKR08] William Krebs, Per Kroll, and Edward Richard. Un-assessments reflec-tions by the team, for the team. In Proc. of the Agile 2008, pages 384–389, Washington, DC, USA, aug 2008.
[Koo10] Supannika Koolmanojwong. The Incremental Commitment SpiralModel Process Patterns for Rapid-Fielding Projects. PhD thesis, Uni-versity of Southern California, 2010.
[Lav00] Luigi Lavazza. Providing automated support for the GQM measurementprocess. Software, IEEE, 17(3):56 –62, may/jun 2000.
[Lee96] Ming June Lee. Foundations of the WinWin Requirements NegotiationSystem. PhD thesis, University of Southern California, 1996.
127
[MCHL06] Tim Menzies, Zhihao Chen, Jairus Hihn, and Karen Lum. Selectingbest practices for effort estimation. Software Engineering, IEEE Trans-actions on, 32(11):883 –895, nov. 2006.
[MRZ+05] Joseph F. Maranzano, Sandra A. Rozsypal, Gus H. Zimmerman,Guy W. Warnken, Patricia E. Wirth, and David M. Weiss. Archi-tecture reviews: practice and experience. Software, IEEE, 22(2):34 –43, march-april 2005.
[PF79] L. Putnam and A. Fitzsimmons. Estimating software costs. Datama-tion, pages 189–198, September 1979.
[PM07] R. W. Pew and A. S. Mavor. Human-System Integration in the SystemDevelopment Process: A New Look. National Academy Press, 2007.
[PRI98] PRICE Systems. Your guide to PRICE-S: Estimating cost and scheduleof software development and support, 1998.
[Rat] Rational Team Concert. https://jazz.net/projects/
rational-team-concert/.
[Sta04] Standish Group. Chaos summary 2004. http://standishgroup.com,2004.
[Sta06] Standish Group. Chaos summary 2006. http://standishgroup.com,2006.
[Sta09] Standish Group. Chaos summary 2009. http://standishgroup.com,2009.
[Tho02] Rob Thomsett. Radical Project Management. Prentice Hall, April 2002.
[UCC] Unified Code Counter. http://sunset.usc.edu/research/
CODECOUNT/. [Online; accessed 22-January-2012].
[USC11] CSCI577 USC Software Engineering I Class Website. http://
greenbay.usc.edu/csci577/fall2011/site/index.html, 2011.
[Wik12] Wikipedia. Continuous Assessment. http://en.wikipedia.org/wiki/Continuous_assessment, 2012. [Online; accessed 13-June-2012].
128
[WL77] Jerome D. Wiest and Ferdinand K. Levy. A Management Guide toPERT/CPM. Prentice-Hall, Englewood Press, 1977.
[YWT+06] Da Yang, Yuxiang Wan, Zinan Tang, Shujian Wu, Mei He, and MingshuLi. COCOMO-U: An extension of COCOMO II for cost estimation withuncertainty. In Qing Wang, Dietmar Pfahl, David Raffo, and Paul Wer-nick, editors, Software Process Change, volume 3966 of Lecture Notes inComputer Science, pages 132–141. Springer Berlin / Heidelberg, 2006.
129
Appendix A: Questions with
COCOMO II Impact Factor
1. [Architecture Development] Do you constantly validate the architecture with
the requirements, prototypes, and concepts?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.48 0 0.5
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0.17 0 0.03 0 0.52
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.55 0 0 0 0 0.3
Cost Driver - Project
TOOL SITE SCED
0.2 0 0
2. [Architecture Development] Have you explored the alternatives for technolo-
gies to be used or jump to a familiar one?
130
Scale Factors
PREC FLEX RESL TEAM PMAT
0.26 0.35 0 0 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.23 0.23 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.3 0 0 0.3 0.2 0.5
Cost Driver - Project
TOOL SITE SCED
0.25 0 0
3. [Architecture Development] Has the system been architected for ease of adding
or dropping borderline features? Were these evolutionary requirements visu-
alized properly by the team?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0.42 0 0 0.27
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.38 0.33 0.33
131
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.3 0 0 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
4. [Business Case] Is there a valid business case for the system, relating the life
cycle system benefits to the system total cost of ownership?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0 0.23
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.18 0.4 0.38
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.3 0 0 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
132
5. [Business Case] Do prototypes developed for verifying the validation of
requirements addresses the critical issues(risks,use cases and GUI) and even-
tually evolve as requirement changes amongst stakeholders
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0.28 0.4 0 0.29
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.32 0 0.34
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.32 0.43 0 0 0 0.4
Cost Driver - Project
TOOL SITE SCED
0.2 0 0
6. [Business Case] Do you and the stakeholders understand the existing and
proposed scope, boundary and business workflow ?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0.37 0 0.3 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.1 0 0
133
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.5 0 0 0.1 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
7. [Business Case] Have questions about the fit of the system into the stakehold-
ers’ context - acquirers, end users, administrators, interoperators, maintainers,
etc. - been adequately explored?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0.23 0.46 0 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0.23 0.35 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0.3
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.15 0 0 0 0 0
134
Cost Driver - Project
TOOL SITE SCED
0 0 0
8. [Business Case] Are the prototypes developed early enough to address con-
ceptual issues and knowledge?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.58 0 0.2
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.07 0.2 0 0 0.27 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
9. [Collaboration] Do you consistently review each other’s work before integrat-
ing them together?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.68 0.72
135
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.15 0 0.37
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0 0 0 0.18 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
10. [Collaboration] Are the stakeholders meetings held weekly with clearly defined
agendas and does the project progress is discussed extensively.
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.62 0.5
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0.25
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.23 0 0.2 0 0 0
136
Cost Driver - Project
TOOL SITE SCED
0 0.43 0
11. [Collaboration] Do you have the proper mechanisms to ensure high level of
collaboration and keeping all stakeholders in the loop (i.e. use of Google
Groups, MSN meetings, etc.)?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.74 0.28
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0 0 0 0 0 0
Cost Driver - Project
TOOL SITE SCED
0.43 0.24 0
12. [Collaboration] Are the team members co-operative in sharing the knowledge
of their expertise and helping each other in development?
137
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.88 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.1 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0 0.1 0.36 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
13. [Feasibility Evidence] Have at least two alternative approaches(considering
COTS/NDI/NCS) been explored and evaluated?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0.32 0.25 0 0.27
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.17 0.1 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
138
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.33 0 0 0.3 0.4 0
Cost Driver - Project
TOOL SITE SCED
0.1 0 0
14. [Feasibility Evidence] Have the claimed quality of service guarantees been
validated?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.43 0 0.37
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0.33 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0.17 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0 0 0 0.23 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
15. [Feasibility Evidence] Have COTS/NDI/Services scalability, compatibility,
quality of service, and life cycle support risks been thoroughly addressed?
139
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.6 0 0.5
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0.37 0 0 0 0.3
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.42 0 0 0.12 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
16. [Feasibility Evidence] Did the team identify the riskiest modules and whether
the riskiest modules development strategy has been considered.
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.68 0.35 0.2
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0.22
Cost Driver - Platform
TIME STOR PVOL
0 0 0
140
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0 0 0 0.03 0.27 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
17. [Feasibility Evidence] Has the project identified a full set of representative
operational scenarios across which to evaluate feasibility?
Scale Factors
PREC FLEX RESL TEAM PMAT
0.22 0 0.38 0 0.47
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0.4
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.32 0 0 0 0.3 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
18. [Feasibility Evidence] Does the project have adequate processes in place to
define the verification, test, and validation, and acceptance of systems and
system elements at all phases of definition and development?
141
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.5 0 0.85
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0.2 0.13 0 0.4
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0 0.23 0 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
19. [Personnel Capability] Does the team have required Domain knowledge and
experience to develop a stable architecture for the system
Scale Factors
PREC FLEX RESL TEAM PMAT
0.5 0 0.65 0 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.02 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
142
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.4 0.63 0 0.46 0.67 0.4
Cost Driver - Project
TOOL SITE SCED
0 0 0
20. [Personnel Capability] Have sufficiently talented and experienced program and
systems engineering managers been identified? Have they been empowered to
tailor process and to enforce development stability?
Scale Factors
PREC FLEX RESL TEAM PMAT
0.5 0 0 0.47 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.58 0.48 0 0.31 0.43 0.4
Cost Driver - Project
TOOL SITE SCED
0 0 0
21. [Personnel Capability] Are the stakeholders who have been identified as crit-
ical to the success of the project represented by highly qualified personnel
143
- those who are collaborative, representative, empowered, committed, and
knowledgeable?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.34 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.37 0.5 0.2 0.26 0.48 0.55
Cost Driver - Project
TOOL SITE SCED
0 0 0
22. [Personnel Capability] Did the team take any effort for improving the techno-
logical skills and experience?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.02 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
144
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.27 0.48 0.02 0.38 0.3 0.52
Cost Driver - Project
TOOL SITE SCED
0 0 0
23. [Planning and Control] Do the team members communicate to the project
manager their areas of strengths and weaknesses so the plan and work distri-
bution can be adjusted accordingly?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.56 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.28 0.2 0 0.07 0 0
145
Cost Driver - Project
TOOL SITE SCED
0 0.13 0
24. [Planning and Control] Does the team have a realistic scheduled project plan
according to the current situation and can the plan sustain any unprecedented
changes in the future?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.48 0.37
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.25 0.17 0.2 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
25. [Planning and Control] Do you constantly monitor and review the progress
of the project giving yourself enough buffer time to respond to changes and
unexpected situations?
146
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.42 0 0.67
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.1 0 0 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
26. [Planning and Control] Is the quantity of developer systems engineering per-
sonnel assigned, their skill and seniority mix, and the time phasing of their
application throughout the program life cycle, appropriate?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.43 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
147
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.12 0.37 0 0.4 0.2 0.37
Cost Driver - Project
TOOL SITE SCED
0.13 0 0
27. [Planning and Control] Is there a top-to-bottom plan for how the total system
will be integrated and tested? Does it adequately consider integration facilities
development and earlier integration testing?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.32 0 0.67
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0.38
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.2 0 0 0.08 0.23 0
148
Cost Driver - Project
TOOL SITE SCED
0 0 0
28. [Planning and Control] Is the project adequately identifying and managing its
risks, while applying corrective action where necessary?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.49 0 0.55
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0.26 0 0 0 0.23
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.3 0 0 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
29. [Planning and Control] Are all the modules being tested and validated by
testers and the stakeholders?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0.4 0 0.6
149
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0.23 0 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.1 0 0 0.08 0.17 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
30. [Planning and Control] Is the team and stakeholders aware of each and every
team members roles and responsibilities and is the work distribution fair and
according to the expertise?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.54 0
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0
Cost Driver - Platform
TIME STOR PVOL
0 0 0
150
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.3 0.27 0.13 0.33 0.25 0.25
Cost Driver - Project
TOOL SITE SCED
0 0 0
31. [Requirements Gathering] Are the win conditions constantly maintained
throughout the project life cycle?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0 0.53
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0.48
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0 0 0 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
32. [Requirements Gathering] Was the WinWin negotiation successful in terms of
capturing, agreeing, and prioritizing the requirements?
151
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0 0 0.38 0.5
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0.43
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.3 0 0 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
33. [Requirements Gathering] Do the requirements and scope of the project take
into consideration the evolutionary changes?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0.5 0 0 0.2
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0 0 0.22
Cost Driver - Platform
TIME STOR PVOL
0 0 0
152
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.17 0 0 0 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
34. [Requirements Gathering] Project details, requirements, boundaries and scope
are thoroughly researched and understood by the team?
Scale Factors
PREC FLEX RESL TEAM PMAT
0 0.35 0.42 0.4 0.63
Cost Driver - Product
RELY DATA CPLX RUSE DOCU
0 0 0.18 0 0.42
Cost Driver - Platform
TIME STOR PVOL
0 0 0
Cost Driver - Personnel
ACAP PCAP PCON APEX PLEX LTEX
0.45 0.2 0 0.17 0 0
Cost Driver - Project
TOOL SITE SCED
0 0 0
153
Appendix B: Question Set -
Weekly ICSM Set
ID Category Subject Question
1 Architecture
Development
Validation Do you constantly validate the architec-
ture with the requirements, prototypes,
and concepts?
2 Architecture
Development
Alternatives Have you explored the alternatives for
technologies to be used or jump to a famil-
iar one?
3 Business Case Business Pro-
cess/Scoping
Do you and the stakeholders understand
the existing and proposed scope, bound-
ary and business workflow?
4 Business Case System Con-
text
Have questions about the fit of the system
into the stakeholders’ context - acquir-
ers, end users, administrators, interoper-
ators, maintainers, etc. - been adequately
explored?
5 Business Case Prototyping Are the prototypes developed early
enough to address conceptual issues and
knowledge?
154
ID Category Subject Question
6 Collaboration Review Do you consistently review each other’s
work before integrating them together?
7 Collaboration Meetings Are the stakeholders meetings held weekly
with clearly defined agendas and does the
project progress is discussed extensively.
8 Collaboration Co-
ordination
Are the team members co-operative in
sharing the knowledge of their expertise
and helping each other in development?
9 Feasibility
Evidence
Alternatives Have at least two alterna-
tive approaches(considering
COTS/NDI/NCS) been explored and
evaluated?
10 Feasibility
Evidence
NDI Feasibil-
ity
Have COTS/NDI/Services scalability,
compatibility, quality of service, and
life cycle support risks been thoroughly
addressed?
11 Feasibility
Evidence
Operational
scenarios
Has the project identified a full set of rep-
resentative operational scenarios across
which to evaluate feasibility?
155
ID Category Subject Question
12 Personnel
Capability
Personnel
capabilities
Have sufficiently talented and experienced
program and systems engineering man-
agers been identified? Have they been
empowered to tailor process and to enforce
development stability?
13 Personnel
Capability
CRACK Are the stakeholders who have been iden-
tified as critical to the success of the
project represented by highly qualified
personnel - those who are collaborative,
representative, empowered, committed,
and knowledgeable?
14 Personnel
Capability
Team Spirit Did the team take any effort for improving
the technological skills and experience?
15 Planning and
Control
Strengths and
Weaknesses
Do the team members communicate to the
project manager their areas of strengths
and weaknesses so the plan and work dis-
tribution can be adjusted accordingly?
16 Planning and
control
Project Moni-
toring
Do you constantly monitor and review
the progress of the project giving yourself
enough buffer time to respond to changes
and unexpected situations?
156
ID Category Subject Question
17 Planning and
Control
Risk Manage-
ment
Is the project adequately identifying and
managing its risks, while applying correc-
tive action where necessary?
18 Planning and
Control
Roles and
Responsibili-
ties
Is the team and stakeholders aware of each
and every team members roles and respon-
sibilities and is the work distribution fair
and according to the expertise?
19 Requirements
Gathering
WinWin Are the win conditions constantly main-
tained by the shaper throughout the
project life cycle?
20 Requirements
Gathering
Clarity Project details, requirements, boundaries
and scope are thoroughly researched and
understood by the team?
157
Appendix C: Question Set -
Milestone ICSM Set
ID Category Subject Question
1 Architecture
Development
Validation Do you constantly validate the architec-
ture with the requirements, prototypes,
and concepts?
2 Architecture
Development
Alternatives Have you explored the alternatives for
technologies to be used or jump to a famil-
iar one?
3 Architecture
Development
Evolutionary Has the system been architected for ease
of adding or dropping borderline fea-
tures? Were these evolutionary require-
ments visualized properly by the team?
4 Business Case Business case Is there a valid business case for the sys-
tem, relating the life cycle system benefits
to the system total cost of ownership?
5 Business Case System Con-
text
Have questions about the fit of the system
into the stakeholders’ context - acquir-
ers, end users, administrators, interoper-
ators, maintainers, etc. - been adequately
explored?
158
ID Category Subject Question
6 Business Case Prototyping Are the prototypes developed early
enough to address conceptual issues and
knowledge?
7 Collaboration Meetings Are the stakeholders meetings held weekly
with clearly defined agendas and does the
project progress is discussed extensively.
8 Collaboration Technology
support
Do you have the proper mechanisms to
ensure high level of collaboration and
keeping all stakeholders in the loop (i.e.
use of Google Groups, MSN meetings,
etc.)?
9 Collaboration Co-
ordination
Are the team members co-operative in
sharing the knowledge of their expertise
and helping each other in development?
10 Feasibility
Evidence
Alternatives Have at least two alterna-
tive approaches(considering
COTS/NDI/NCS) been explored and
evaluated?
11 Feasibility
Evidence
Level of Ser-
vice
Have the claimed quality of service guar-
antees been validated?
159
ID Category Subject Question
12 Feasibility
Evidence
NDI Feasibil-
ity
Have COTS/NDI/Services scalability,
compatibility, quality of service, and
life cycle support risks been thoroughly
addressed?
13 Feasibility
Evidence
Risk Manage-
ment
Did the team identify the riskiest modules
and whether the riskiest modules develop-
ment strategy has been considered.
14 Feasibility
Evidence
Operational
scenarios
Has the project identified a full set of rep-
resentative operational scenarios across
which to evaluate feasibility?
15 Feasibility
Evidence
Validation
and Verifica-
tion
Does the project have adequate processes
in place to define the verification, test, and
validation, and acceptance of systems and
system elements at all phases of definition
and development?
16 Personnel
Capability
Personnel
capabilities
Have sufficiently talented and experienced
program and systems engineering man-
agers been identified? Have they been
empowered to tailor process and to enforce
development stability?
160
ID Category Subject Question
17 Personnel
Capability
CRACK Are the stakeholders who have been iden-
tified as critical to the success of the
project represented by highly qualified
personnel - those who are collaborative,
representative, empowered, committed,
and knowledgeable?
18 Personnel
Capability
Team Spirit Did the team take any effort for improving
the technological skills and experience?
19 Planning and
Control
Strengths and
Weaknesses
Do the team members communicate to the
project manager their areas of strengths
and weaknesses so the plan and work dis-
tribution can be adjusted accordingly?
20 Planning and
control
Detail Plans Does the team have a realistic sched-
uled project plan according to the cur-
rent situation and can the plan sustain any
unprecedented changes in the future?
21 Planning and
control
Project Moni-
toring
Do you constantly monitor and review
the progress of the project giving yourself
enough buffer time to respond to changes
and unexpected situations?
161
ID Category Subject Question
22 Planning and
Control
Personnel
capabilities
Is the quantity of developer systems engi-
neering personnel assigned, their skill and
seniority mix, and the time phasing of
their application throughout the program
life cycle, appropriate?
23 Planning and
Control
Overall plan Is there a top-to-bottom plan for how the
total system will be integrated and tested?
Does it adequately consider integration
facilities development and earlier integra-
tion testing?
24 Planning and
Control
Validation Are all the modules being tested and val-
idated by testers and the stakeholders.?
25 Requirements
Gathering
WinWin Are the win conditions constantly main-
tained by the shaper throughout the
project life cycle?
26 Requirements
Gathering
WinWin Was the WinWin negotiation successful in
terms of capturing, agreeing, and priori-
tizing the requirements?
27 Requirements
Gathering
Evolutionary Do the shaped requirements and scope
of the project take into consideration the
evolutionary changes?
162
ID Category Subject Question
28 Requirements
Gathering
Clarity Project details, requirements, boundaries
and scope are thoroughly researched and
understood by the team?
163
Appendix D: Question Set -
General Set
ID Category Subject Question
1 Business Case Business Pro-
cess/Scoping
Do you and the stakeholders understand
the existing and proposed scope, bound-
ary and business workflow?
2 Business Case Prototyping Are the prototypes developed early
enough to address conceptual issues and
knowledge?
3 Collaboration Review Do you consistently review each other’s
work before integrating them together?
4 Collaboration Meetings Are the stakeholders meetings held weekly
with clearly defined agendas and does the
project progress is discussed extensively.
5 Collaboration Technology
support
Do you have the proper mechanisms to
ensure high level of collaboration and
keeping all stakeholders in the loop (i.e.
use of Google Groups, MSN meetings,
etc.)?
164
ID Category Subject Question
6 Collaboration Co-
ordination
Are the team members co-operative in
sharing the knowledge of their expertise
and helping each other in development?
7 Feasibility
Evidence
Level of Ser-
vice
Have the claimed quality of service guar-
antees been validated?
8 Feasibility
Evidence
Risk Manage-
ment
Did the team identify the riskiest modules
and whether the riskiest modules develop-
ment strategy has been considered.
9 Personnel
Capability
Domain
Knowledge
and Exper-
tise/Experience
Does the team have required domain
knowledge and experience to develop a
stable architecture for the system
10 Personnel
Capability
Personnel
capabilities
Have sufficiently talented and experienced
program and systems engineering man-
agers been identified? Have they been
empowered to tailor process and to enforce
development stability?
11 Personnel
Capability
Team Spirit Did the team take any effort for improving
the technological skills and experience?
165
ID Category Subject Question
12 Planning and
Control
Strengths and
Weaknesses
Do the team members communicate to the
project manager their areas of strengths
and weaknesses so the plan and work dis-
tribution can be adjusted accordingly?
13 Planning and
control
Detail Plans Does the team have a realistic sched-
uled project plan according to the cur-
rent situation and can the plan sustain any
unprecedented changes in the future?
14 Planning and
control
Project Moni-
toring
Do you constantly monitor and review the
progress of the project and giving yourself
enough buffer time to respond to changes
and unexpected situations?
15 Planning and
Control
Personnel
capabilities
Is the quantity of developer systems engi-
neering personnel assigned, their skill and
seniority mix, and the time phasing of
their application throughout the program
life cycle, appropriate?
16 Planning and
Control
Risk Manage-
ment
Is the project adequately identifying and
managing its risks, while applying correc-
tive action where necessary?
17 Planning and
Control
Validation Are all the modules being tested and val-
idated by testers and the stakeholders.?
166
ID Category Subject Question
18 Planning and
Control
Roles and
Responsibili-
ties
Is the team and stakeholders aware of each
and every team members roles and respon-
sibilities and is the work distribution fair
and according to the expertise?
19 Requirements
Gathering
Evolutionary Do the requirements and scope of the
project take into consideration the evolu-
tionary changes?
20 Requirements
Gathering
Clarity Project details,requirements,boundaries
and scope are thoroughly researched and
understood by the team?
167
Appendix E: Original Effort
Categories
Operational Concept Develop-
ment
Implementation
Analyze current system Explore and evaluate alternatives
Identify shared vision Analyze and prioritize capabilities to
prototype
Establish new operational concept Acquire NDI/NCS
Identify system transformation Prepare development / operational
environment
Identify organizational and opera-
tional transformation
Develop prototype
Identify objectives, constraints, and
priorities
Develop component
Assess operational concept Assess prototype / component
Documenting of OCD Tailor NDI/NCS
System and Software Require-
ments Development
Integrate components
Set up WinWin negotiation context Transition the system
Negotiation (during meeting) Develop transition plan
168
Negotiation using WikiWinWin tool
(after meeting)
Develop support plan
Identify win conditions Provide training
Identify issues Develop user manual
Negotiate options Documenting of prototype
Develop requirements definition Testing
Assess requirements definition Identify test cases
Documenting of SSRD Identify test plan
Documenting of WinWin negotiation
report
Identify test procedures
System and Software Architec-
ture Development
Record test results
Analyze the proposed system Identify regression test package
Define technology-independent archi-
tecture
Documenting test-related documents
Define technology-specific architecture Perform custom component test
Specify architecture styles, patterns,
and frameworks
Perform NDI/NCS component test
Assess system architecture Perform integration test
Documenting of SSAD Perform acceptance test
Life Cycle Planning Perform regression test
Identify milestones and products Perform unit test
169
Identify responsibilities and skills Project Administration
Identify life cycle management
approach
Create and maintain project website
Estimate project effort and schedule
using COCOMO II
Interact with clients
Estimate project effort and schedule
using COCOTS
Interact between team members
Assess life cycle content Learn about Incremental Commit-
ment Model
Detail project plan Learn about application domain
Record project progress Prepare transition site
Record project individual effort Manage code configuration
Identify development iteration Attend ARB
Assess development iteration Planning and control
Perform core capabilities drive-
through
Control project performance
Documenting of LCP Quality Management
Documenting of iteration-related doc-
uments
Gather definitions
Track problem and report closure Construct traceability matrix
Feasibility Evidence Description Identify configuration management
strategy
170
Analyze business case Identify quality management strategy
Identify, assess, and manage skills Verify and validate work products
Provide architecture feasibility evi-
dence
Assess quality management strategy
Provide process feasibility evidence Documenting of QMP
Assess feasibility evidence Documenting of SID
Assess/Evaluate NDI/NCS candidates Documenting of review-related docu-
ment
Check components interoperability
Documenting of FED
171
Appendix F: Re-categorized Effort
Categories
Operational Concept Develop-
ment
Feasibility Evidence Develop-
ment
Analyze current system Analyze business case
Identify shared vision Identify, assess, and manage skills
Establish new operational concept Provide architecture feasibility evi-
dence
Identify system transformation Provide process feasibility evidence
Identify organizational and opera-
tional transformation
Assess feasibility evidence
Identify objectives, constraints, and
priorities
Assess/Evaluate NDI/NCS candidates
Assess operational concept Check components interoperability
Documenting of OCD Documenting of FED
Requirements Engineering Implementation
Set up WinWin negotiation context Explore and evaluate alternatives
Negotiation (during meeting) Analyze and prioritize capabilities to
prototype
Negotiation using WinWin tool (after
meeting)
Acquire NDI/NCS
172
Identify win conditions Prepare development / operational
environment
Identify issues Develop prototype
Negotiate options Develop component
Develop requirements definition Assess prototype / component
Assess requirements definition Tailor NDI/NCS
Documenting of SSRD Integrate components
Documenting of WinWin negotiation
report
Transition the system
Design and Architecture Devel-
opment
Provide training
Analyze the proposed system Develop user manual
Define technology-independent archi-
tecture
Documenting of prototype
Define technology-specific architecture Testing
Specify architecture styles, patterns,
and frameworks
Identify test cases
Assess system architecture Identify test plan
Documenting of SSAD Identify test procedures
Planning and Control Record test results
Identify milestones and products Identify regression test package
Identify responsibilities and skills Documenting test-related documents
173
Identify life cycle management
approach
Perform custom component test
Estimate project effort and schedule
using COCOMO II
Perform NDI/NCS component test
Estimate project effort and schedule
using COCOTS
Perform integration test
Assess life cycle content Perform acceptance test
Detail project plan Perform regression test
Record project progress Perform unit test
Record project individual effort Performance Control
Identify development iteration Manage code configuration
Assess development iteration Attend ARB
Perform core capabilities drive-
through
Planning and control
Documenting of LCP Control project performance
Documenting of iteration-related doc-
uments
Quality Management
Track problem and report closure Gather definitions
Develop transition plan Construct traceability matrix
Develop support plan Identify configuration management
strategy
174
Communication and Synchro-
nization
Identify quality management strategy
Create and maintain project website Verify and validate work products
Interact with clients Assess quality management strategy
Interact between team members Documenting of QMP
Learn about Incremental Commit-
ment Model
Documenting of SID
Learn about application domain Documenting of review-related docu-
ment
Prepare transition site
175
Appendix G: Client Evaluation
Form - One-Semester Projects
1
577A Client Evaluation Project Name: __________________________________________ Team # _______ Please provide a ranking where indicated. Use a 1 - 5 scale where 1 is low and 5 is high. Where comments are requested, please include any descriptive statements you wish to add. FOR THE FIRST TWO ITEMS, consult your team's Development Commitment Package documentation by clicking on the project name in the table shown on the course webpage http://greenbay.usc.edu/csci577/fall2008/site/projects/index.html 1. Operational Concept Description (especially Sections: 2. Shared Vision; 3. System Transformation). How well did the team capture your ideas of what the new system should do? Ranking: (1-5) Comments: 2. Team helpfulness Did the team suggest new idea about your project? Did the team support you in any project complexity? Ranking : (1-5) Comments: FOR THE NEXT ITEMS, base your responses on your interactions with the project team and their product. 3. Team Responsiveness: Was the team responsive to you and to your requirements? Did the team answer any questions you might have had? How successful was the requirements negotiation between you and your team? Ranking : (1-5) Comment: 4. Project Results: How satisfied are you with the prototype? How well does the proposed system meet the need identified by your project? Ranking: (1-5) Comment: 5. Project Future: Do you think this project should be carried forward for development in CS577b? Has the team adequately identified and managed the potential risks and complications associated with the project? Do you foresee difficulties in transitioning this project into the development and implementation phase? Ranking : (1-5) Comment:
176
2
6. Team Communication: How effective was the team in communicating with you? Do you feel you had enough meetings with them? Did the team make effective use of your time and expertise? What means did you use to communicate? Ranking : (1-5) Comment: 7. Tools: Regarding software tools students used in the class 7a. Did the team share WinWin negotiation results with you? If so, how? Comment 7b. Did the team mention or describe any other software engineering tools? Comment: 8. Your Learning: Did you gain a better understanding of software engineering and information technology by participating in this project? Please provide specifics where possible. Ranking : (1-5) Comment: 9. Suggestions about the course processes from your (a client's) perspective. Comment: 10. Overall Value: Did you feel your participation was worthwhile? Would you participate again? Did you find participation valuable as either a teaching or research activity? Ranking : (1-5) 3 Comment: If you have any other comments that you would like to offer, please feel free. We may get back to you in the future on these for clarification.
177
Appendix H: Client Evaluation
Form - Two-Semester Projects
1/3
577B Client Evaluation Team # _____________________________ Project _____________________________ The items listed below will ask for either a ranking or a yes/no response. For rankings, use a 1-5 scale where 1 is low and 5 is high. For yes/no items, please indicate your choice. Additional comments will be very helpful for evaluation and planning purposes. Please use the questions in parenthesis as starting points for your comments. Documentation 1. User Manual: Rank your level of satisfaction with the User Manual. (Is it well written? Will it reasonably answer both user and administrator questions? Did your team ask you to evaluate the User Manual? Did they incorporate your comments?) Ranking: (1 to 5) Comments: 2. System Documents: Rank your level of satisfaction with the As Built system documents. Ranking: (1 to 5) Comments: Team Interaction 3. Team Responsiveness: How responsive was the team to your needs as a client? Did they address your project objectives and concerns? Ranking: (1 to 5) Comments: 4. Team Communication: How satisfied were you with the team’s communication with you? (Did they tell you what you needed to know when you needed to know it? Did they explain their work/the issues in terms you could understand?) Ranking: (1 to 5) Comments: System Preparation and Testing 5. How effective were the students at installing the software and providing any necessary set-up and configuration support to enable you to get started? (Had they anticipated and prepared for any set-up issues or problems?) Were the students responsive and effective in handling any configuration adjustments that might have been needed once you began using the system? Ranking: (1 to 5) Comments: 6. Did the students help to adequately prepare you to handle ongoing support and maintenance of the system? Yes/No: Comments: 7. Training: Did the team provide appropriate training on how to use the system? (How was this done?)
178
2/3
Yes/No: Comments: 8. Training Quality: (Answer only if you received training for the system) – How adequate was your system training? Ranking: (1 to 5) Comments: 9A. Software Testing: Did the team ask you to test the final product? Yes/No: 9B. Software Testing: If yes, did you test all of the system’s features? Yes/No: 9C. Software Testing: How much time did you spend doing system testing? 10. Software Test Results: (Answer only if you participated in software testing) – Please rate how close the functions you used came to meeting your expectations. Did they work as you expected? Ranking: (1 to 5) Comments: Implementation 11. Is the system up and running in full release? or in a beta release? Is any additional programming required to achieve sufficient basic functionality for public release? Yes/No: Comments: 12. Hardware & Software: Do you need additional hardware and/or software to implement the system? Yes/No: Comments: 13. Other implementation issues: Are there any other issues which impact whether or not the system can be implemented? Yes/No: Comments: Overall Value 14. Rate how successful the product is, as delivered, at achieving your objectives and desired functionality. Has the team made the right choices and trade-offs? Ranking: (1 to 5) Comments: 15. Rate the value / anticipated benefits of the product you’ve received. Is it a worth the time you’ve spent working with the team over the past 2 semesters? Will the product provide sufficient utility? Does it have long term potential or applicability beyond the need outlined in your original proposal? Ranking: (1 to 5) Comments:
179
3/3
Summary 16. Your Learning: Did you learn anything new this semester? Yes/No: Comments: 17. Teaching: Did you feel you made a contribution to the education of graduate CSCI students at USC? Yes/No: Comments: 18. Research: Did you feel you made a research contribution by participating in this project? Yes/No: Comments: 19. Recommendation for Others: Would you recommend participation in future projects to your colleagues? (Are there specific projects, either your own or projects of your colleagues, which you would recommend for future courses?) Comments: 20. Feel free to provide any other comments or suggestions for future course improvement.
180
Appendix I: Project Status Survey
Figure I.1: Project status survey - page 1
181
Figure I.2: Project status survey - page 2
182
Appendix J: Multiple Linear
Regression Analysis
Figure J.1: Project status survey - page 2
183
Appendix K: Fit Y by X Test
Figure K.1: Fit Y by X - PREC & FLEX
184
Figure K.2: Fit Y by X - RESL & TEAM
Figure K.3: Fit Y by X - PMAT & RELY
185
Figure K.4: Fit Y by X - DATA & CPLX
Figure K.5: Fit Y by X - RUSE & DOCU
186
Figure K.6: Fit Y by X - TIME & STOR
Figure K.7: Fit Y by X - PVOL & ACAP
187
Figure K.8: Fit Y by X - PCAP & PCON
Figure K.9: Fit Y by X - APEX & PLEX
188
Figure K.10: Fit Y by X - LTEX & TOOL
Figure K.11: Fit Y by X - SITE & SCED
189
Appendix L: Application Point
Estimation
1. Step 1: Assess Application Counts: estimate the number of screens, reports, and 3GL components that will comprise this application. Assume the standard definitions of these elements in your ICASE environment.
Step 2: Classify each element instance into simple, medium and difficult complexity levels depending on
values of characteristic dimensions. Use the following scheme:
For Screens For Reports Numbe
r of Views contai
ned
# and source of data tables Number of Sections
contained
#and source of data tables Total<4 (<2srvr<
3clnt)
Total<8 (2/3 srvr 3-5 clnt)
Total 8+ (>3 srvr >5 clnt)
Total<4 (<2srvr<
3clnt)
Total<8 (2/3 srvr 3-5 clnt)
Total 8+ (>3 srvr >5 clnt)
<3 simple simple medium 0-1 simple simple medium 3-7 simple medium difficult 2 or 3 simple medium difficult >8 medium difficult difficult 4+ medium difficult difficult
Step 3: Weigh the number in each cell using the following scheme. The weights reflect the relative effort
required to implement an instance of that complexity level:
Element Type Complexity-Weight Simple Medium Difficult
Screen 1 2 3 Report 2 5 8
3GL Component 10 2. Step 4: Determine application Points: add all the weighted element instances to get one number, the
Application Point count. Step 5: Estimate percentage of reuse you expect to be achieved in this project. Compute the New
Application Points to be developed, NAP=(Application Points) (100-%reuse)/100. Step 6: Determine a productivity rate, PROD=NAP/person-month, from the following scheme.
Developers’ experience and capability Very Low Low Nominal High Very High ICASE maturity and capability Very Low Low Nominal High Very High
PROD 4 7 13 25 50 Step 7: Compute the estimated person-months: PM=NAP/PROD.
Figure L.1: Application Point Estimations Procedure
190
Appendix M: Student Background
Questionnaire
Figure M.1: Student Background Questionnaire - Page 1
191
Figure M.2: Student Background Questionnaire - Page 2
192
Figure M.3: Student Background Questionnaire - Page 3
193
Figure M.4: Student Background Questionnaire - Page 4
194
Figure M.5: Student Background Questionnaire - Page 5
195
