Tailoring a Large Organization’s Systems Engineering ...

_Tailoring a Large Organization’s Systems Engineering Process to Meet Project-
Specific Needs
M.S. in Mechanical Engineering, May 2005, Texas A&M University
A Dissertation submitted to
of The George Washington University
in partial satisfaction of the requirements
for the degree of Doctor of Philosophy
August 31, 2016
Dissertation directed by
Thomas Mazzuchi
Professor of Engineering Management and Systems Engineering & of Decision Sciences
ii
The School of Engineering and Applied Science of The George Washington University
certifies that Matthew S. Graviss has passed the Final Examination for the degree of Doctor
of Philosophy as of July 27, 2016. This is the final and approved form of the dissertation.
Tailoring a Large Organization’s Systems Engineering Process to
Meet Project-Specific Needs
Matthew S. Graviss
Dissertation Research Committee:
Engineering, Dissertation Co-Director
Engineering & of Decision Sciences, Dissertation Co-Director
Steven Stuban, Professorial Lecturer in Engineering Management and
System Engineering, Committee Member
Systems Engineering, Committee Member
Systems Engineering, Committee Member
All rights reserved
Tailoring a Large Organization’s Systems Engineering Process to Meet Project-Specific
Needs
Systems engineers are faced with the challenge of adhering to broad systems
engineering policies, while simultaneously tailoring systems engineering processes to meet
the unique challenges facing their projects. Tailoring is often performed ad hoc, and
determining which stages, steps, and artifacts of the process are necessary can be time-
consuming and challenging. Systems engineering guidebooks across industry and
government organizations often stress the importance of tailoring, yet offer little practical
guidance on how to perform the function. This dissertation proposes a model for
automating the systems engineering tailoring process through defining a set of
organizational rules and a minimal set of project-specific inputs. To evaluate the proposed
approach, the model is analyzed through several case studies within the Department of
Homeland Security.
1.3. Intended Contributions ................................................................................................ 3
2.1. Introduction .................................................................................................................. 6
2.2. Process Tailoring Guidance in Systems Engineering Standards and Handbooks ... 6
2.2.1. Industry Standards and Guidelines .................................................................... 6
2.2.2. Government Standards and Guidelines........................................................... 10
2.2.3. Capability Models ............................................................................................ 12
2.3. Process Reuse ............................................................................................................. 14
2.4.2. Pre-Formed Method for Process Tailoring ..................................................... 18
2.4.3. Rule-Based Method for Process Tailoring ..................................................... 20
2.5. Process Tailoring Case Studies ................................................................................. 21
vi
2.6. Summary ..................................................................................................................... 27
3.1. Introduction ................................................................................................................ 29
3.2. Identify the Initial Set of Tailoring Considerations ................................................. 30
3.3. Refine the Set of Tailoring Considerations for an Organization ............................ 33
3.4. Definitions .................................................................................................................. 34
3.5.1. Base Rules ......................................................................................................... 35
3.5.2. Prioritization Rules ........................................................................................... 37
3.6. Apply Model to a New Project and Validate ........................................................... 38
Chapter 4: Case Studies ............................................................................................................ 39
4.1. Definition of Hypothesis ........................................................................................... 39
4.2. Pilot Project Selections .............................................................................................. 41
4.2.1. Selection of the Organization .......................................................................... 41
4.2.2. Selection of the Projects ................................................................................... 43
4.3. Method of Comparison .............................................................................................. 44
4.4. Effects of Confounding Factors ................................................................................ 45
4.5. Case Study Planning .................................................................................................. 45
4.6. Case Study Conduct ................................................................................................... 46
4.6.1. Organizational Tailoring Consideration Selection ......................................... 46
vii
4.6.2.2. Rules ........................................................................................................ 54
4.6.3. Model-Based Project Tailoring Plans ............................................................. 62
4.7. Analysis and Results .................................................................................................. 64
4.7.1. Analysis of Projects .......................................................................................... 65
4.7.2. Analysis of Tailoring Considerations ............................................................. 68
4.7.3. Analysis of Process Elements .......................................................................... 74
Chapter 5: Conclusions and Future Work ............................................................................... 75
References ................................................................................................................................... 78
Appendix A: DHS Systems Engineering Lifecycle Stages, Reviews, and Artifacts ............ 84
Appendix B: Case Study Rule Set ............................................................................................ 89
Appendix C: Project Cases ........................................................................................................ 98
Appendix D: Analysis of Process Elements ........................................................................... 187
viii
Figure 2-1: Inputs-process-outputs diagram for project planning (INCOSE, 2015)............... 7
Figure 2-2: Adaptation sequence (ISO/IEC, 2008, 38) ............................................................. 9
Figure 2-3: Systems engineering reuse framework (Fortune and Valerdi, 2013) ................. 15
Figure 2-4: Overall procedure of process customization (Ahn et al., 2003, 296) ................. 18
Figure 2-5: Tailoring rule structure (Kang et al., 2008) .......................................................... 20
Figure 2-6: Example of a tailoring rule (Kang et al., 2008) .................................................... 21
Figure 2-7: Steps for process tailoring (Xu and Ramesh, 2008) ............................................ 22
Figure 2-8: Method tailoring elicitation framework (de Cesare et al., 2008, 161)................ 23
Figure 2-9: Process slicing with case-based reasoning (Park and Bae, 2013)....................... 24
Figure 2-10: Overview of case retrieval method (Kang et al., 2008) ..................................... 26
Figure 3-1: Overview of the systems engineering process tailoring model (SEPTM) ......... 29
Figure 3-2: Establishing organizational tailoring considerations ........................................... 33
Figure 3-3: Establishing the rule set ......................................................................................... 36
Figure 4-1: DHS systems engineering lifecycle process (DHS, 2010) .................................. 42
Figure 4-2: Example of a model input and output for a project ............................................. 63
Figure 4-3: Case Study Results by Project ............................................................................... 67
ix
Table 4-2: Relevant process elements for security impact ...................................................... 49
Table 4-3: Relevant process elements for privacy impact ...................................................... 50
Table 4-4: Relevant process elements for intelligence impact ............................................... 50
Table 4-5: Relevant process elements for interoperability impact ......................................... 50
Table 4-6: Relevant process elements for accessibility ........................................................... 50
Table 4-7: Relevant process elements for technology demonstration .................................... 51
Table 4-8: Relevant process elements for environmental tailoring consideration ................ 51
Table 4-9: Relevant process elements for development methodology impact ...................... 51
Table 4-10: Relevant process elements for development type impact ................................... 52
Table 4-11: Relevant process elements for project size .......................................................... 52
Table 4-12: Establishing the rule set for the DHS case study ................................................. 62
Table 4-13: Definition of match: correct and incorrect results ............................................... 64
Table 4-14: Case study results by project................................................................................. 66
Table 4-16: Default model attributes ........................................................................................ 70
Table 4-17: Tailoring consideration analysis example ............................................................ 72
Table 4-18: Case study analysis results .................................................................................... 73
Table 4-19: Process elements with the highest failure rate ..................................................... 75
1
Most organizations develop their own systems engineering (SE) guidelines to specify
the stages, reviews, and artifacts that should be executed on projects. Standard processes
are important for today’s projects, as they employ standardized terminology and may
prevent project teams from reinventing the wheel; however, formal and informal processes
must be balanced to yield the efficiencies gained through standardization and the
effectiveness gained by adaptability (Laufer, 2001). Since no two projects are identical, no
two processes are either (Humphrey, 1989). Organizations must develop tailorable
guidelines that are broad enough to span the range of projects within their portfolio (Hwang
& Park, 2006). This places the burden of tailoring the organization’s SE guidelines on
project systems engineers.
Process tailoring refers to the modification of a standardized process to meet the unique
needs of a project (Xu & Ramesh, 2007). Tailoring can include determining which stages
and reviews are necessary and should be executed multiple times, which artifacts should be
produced, and even what content within an artifact is necessary. Successful tailoring results
in modified processes that achieve the goals of the standard process model (ISO/IEC
15288, 2015). Tailoring is often performed ad hoc, without guidelines or rules (Pedreira et
al., 2007), and is usually completed based on the tacit knowledge of the project team (Xu
and Ramesh, 2009). Project budget and schedule, as well as product quality, are directly
2
affected by process implementation. Choosing and adapting the right design method is
critical to successfully executing SE concepts and principles (Verma and Fabrycky, 1997).
Significant research has been done on tailoring software processes for projects, but the
proposed approaches have not been applied to SE. Existing SE guidelines and handbooks
stress the importance of tailoring SE processes to meet the unique needs of a project, but
provide little guidance on process tailoring (Browning et al., 2006; Pereira et al., 2007).
Therefore, project teams perform process tailoring ad hoc, making a decision on each
process element as to whether or not the element is required and, if so, whether or not it
should be tailored.
1.2. Research Goal and Approach
This dissertation aims to answer the following question: Can a model be developed to
customize a SE process based on the unique needs of a project, thereby reducing the
amount of process tailoring decisions required on a given project?
To investigate this question, previous research was explored to understand existing SE
process tailoring guidance factors that influence SE process tailoring, and process tailoring
approaches. The systems engineering process tailoring model (SEPTM) was developed as
a prototype. Its feasibility was evaluated by applying it to several case studies, and
comparing the actual and modeled SE processes for each case.
3
1.3. Intended Contributions
The introduction of a model that reduces the amount of effort required to tailor an
organization’s SE process to meet the unique needs of a project is useful to SE academics
and practitioners. Applying software engineering solutions to SE is an ongoing trend
(Boehm, 2006). From an academic standpoint, this research extends that trend by using
process tailoring approaches in software engineering and applying them to systems
engineering.
For SE practitioners, this research and associated model offer a method for SE process
tailoring that is rooted in SE best practices and has been evaluated against several real-
world case studies. This dissertation proposes a rule-based approach to SE process
tailoring, offering a significant number of custom SE process variations compared to a pre-
formed approach and without relying on a deep repository required of a case-retrieval
approach. With this proposed approach, the number of tailoring decisions could be reduced
from the total number of elements in a governing SE process to the number of rules
established within an SE organization and applied to the project.
1.4. Dissertation Structure
This dissertation is organized into five chapters. Chapter 1 introduces the research
topic. Chapter 2 reviews the literature relevant to this dissertation topic. It includes research
into guidance for process tailoring offered by SE standards and guidebooks within industry
and the federal government. Previous research into the factors that influence process
4
tailoring is explored. Additionally, research of practical approaches to process tailoring is
reviewed.
Chapter 3 introduces the SEPTM, which is proposed for reducing the effort required to
tailor SE processes based on the unique attributes of a project. In this chapter, tailoring
considerations that influence process tailoring are explained. The process of customizing
the tailoring considerations for a particular organization and definitions for tailoring actions
(i.e., keep standard, tailor, or delete) are explained. Additionally, Chapter 3 describes the
process by which process tailoring rules are formed based on the organizational tailoring
considerations and an organization’s standard SE process. Chapter 3 concludes with a
description of how the model is applied to a particular project to generate a project-specific
SE process based on the project’s unique characteristics and environment.
Chapter 4 defines the hypothesis and the method of using multiple case studies to
evaluate the proposed approach to SE process tailoring. This chapter also introduces an
analysis of 24 case studies within the Department of Homeland Security (DHS), which is
used to evaluate the SEPTM against real-world SE process tailoring instances. A SEPTM-
generated tailored SE process is produced for each of the 24 projects. Three analyses were
performed to investigate the validity of the SEPTM. First, each pair of actual and modeled
tailored SE processes was compared to answer the following question: How well does the
model match the actual tailored SE process of each project? Second, to identify
improvements that should be explored, the actual and modeled results relevant to each
tailoring consideration and associated process elements were evaluated by answering the
following question: Did the model fail to match a majority of projects for any particular
5
tailoring consideration? Third, an analysis of each process element was performed to
understand which elements differed significantly between the actual and modeled tailored
SE processes for the 24 projects.
Finally, Chapter 5 highlights the conclusions of this research and recommends
additional research in SE process tailoring.
6
The research strands pertinent to this dissertation are the following:
process tailoring guidance in systems engineering (SE) standards and handbooks;
process reuse;
2.2. Process Tailoring Guidance in Systems Engineering Standards and Handbooks
The following subsections highlight the tailoring guidance from industry and
government sources.
The International Council on Systems Engineering (INCOSE) developed the Systems
Engineering Handbook (SEH) (INCOSE, 2015) as a guide to SE professionals and other
engineers who perform SE activities The handbook defines SE and lifecycle stages, and
then proceeds to describe a variety of processes, methods, and activities that systems
engineers should consider executing during projects.
This handbook also stresses the importance of tailoring the SE process: “Thoughtful
tailoring and intelligent application of the SE processes described in this handbook are
7
essential to achieve the proper balance between the risk of missing project technical and
business objectives on the one hand and process paralysis on the other hand.”
In the project planning phase, the SEH highlights the project tailoring strategy as one of
the key process inputs (Figure 2-1).
Figure 2-1: Inputs-process-outputs diagram for project planning (INCOSE, 2015)
The SEH offers an overarching process to tailoring. This process begins with
identifying the circumstances that affect the tailoring strategy and ends with the selection of
subprocesses that require tailoring. In addition, the SEH identifies tailoring considerations,
which include factors such as system complexity and organizational policies.
The International Organization for Standardization (ISO) and the International
Electrotechnical Commission (IEC) developed standard 15288 (ISO/IEC, 2015) to provide
8
a common process framework for managing the lifecycle activities of man-made systems.
The standard recommends tailoring for a project’s particular requirements and needs, and
in Annex A, it offers guidance for tailoring of the process contained within. Suggested
considerations for process tailoring include
“stability of, and variety in, operational environments;
risks, commercial or performance, to the concern of interested parties;
novelty, size and complexity;
integrity issues such as safety, security, privacy, usability, availability.”
Similar to the research described in this dissertation, ISO/IEC 15288 focuses tailoring
activities on deleting or adding selected tasks and activities (ISO/IEC, 2015).
ISO/IEC Technical Report 24748 (ISO/IEC, 2010) is a companion document to the
ISO/IEC 15288 standard and offers further guidance on the use of lifecycle models for
systems. The report focuses on the adaptation of the ISO/IEC 15288 standard; Figure 2-2
(which was extracted from the report) depicts the recommended process for adapting the
standards to meet the unique needs of a project.
9
Project environment and characteristics can include the particular lifecycle model,
project scope, and several specialty considerations: human, health, safety, security,
interoperability, dependability, and environmental impacts. For the lifecycle model, the
aforementioned report identifies examples of lifecycle models for particular domains such
as hardware, software, and services. Per the report, tailoring decisions can include both
adding and deleting outcomes, activities, and tasks.
The Institute of Electrical and Electronics Engineers (IEEE) published standard 1220
(IEEE, 2005) for the application and management of the SE process. Similar to the
10
INCOSE SEH, IEEE 1220 describes the tasks that systems engineers should complete to
deliver solutions tailored to customer needs. It recommends that systems engineers
establish an engineering plan for the technical effort, and the tailored application of the
standard. Per IEEE 1220, a project should “tailor the activities of each task by adding or
deleting activities, or tailor the subprocess tasks by adding or deleting tasks, according to
the scope of the project and in accordance with enterprise policies and procedures” (IEEE,
2005, 37).
The National Aeronautics and Space Administration (NASA) Systems Engineering
Handbook (NASA, 2007) was developed to provide NASA engineers with an overarching
description of SE. In each lifecycle phase described in the handbook, tailoring of the
standard SE process to meet project needs is stressed.
The NASA Systems Engineering Processes and Requirements document, NPR
7123.1B (NASA, 2013), is a companion document to the NASA SE handbook, and goes
into greater detail on the NASA SE process and what SE practitioners must execute.
Regarding process tailoring, NPR 7123.1B identifies environmental factors such as project
scope, system complexity, number of interfaces, serviceability, and safety as important
tailoring considerations. The NPR uses a SE management plan for defining a project’s SE
strategy and seeking relief (tailoring) from the SE requirements imposed in the NASA SE
policy. A NASA SEMP also includes a compliance matrix that identifies SE policy
requirements from which a project seeks relief.
11
The Defense Acquisition University’s Systems Engineering Fundamentals Guide
(DAU, 2001) provides a basic explanation of the technical aspects of systems development
and acquisition. Consistent with other standards and guides, this guide stresses the
importance of tailoring the SE process for the different needs of each type of project. Needs
can vary due to “system size and complexity, level of system definition detail, scenarios
and missions, constraints and requirements, technology base, major risk factors, and
organizational best practices and strengths” (DAU, 2001, 9).
The Naval Systems Engineering Guide (Naval Systems Engineering Group, 2004) was
developed to assist program teams in implementing SE across the program’s acquisition
lifecycle. The process implementation strategy described in the guide includes a specific
task for program teams to develop a tailored version of the guide. Each subprocess detailed
in the guide includes a caveat that it should be tailored to for the system’s application,
complexity, or other reason.
Army Acquisition Procedures (Army, 2014) for research, development, and acquisition
highlight encouragement from Department of Defense and Army leadership to tailor
acquisition processes. From requirements development to production engineering and
sustainment, from risk management to data requirements, the Army Acquisition Procedures
authorize tailoring to meet the needs of a program:
This tailoring of required oversight and review documentation to
the needs of each specific program is a key element of the
acquisition streamlining process. Continued emphasis will be
placed on this effort to reduce the amount of documentation that
must be prepared to support program reviews.” (Army, 2014, 179)
12
The Army Systems Engineering Policy (Army, 2005) encourages tailoring the contents
of the systems engineering plan, and includes tailoring considerations such as program size,
commercial off the shelf (COTS) based programs, and for programs in sustainment.
The Department of Homeland Security (DHS) established the systems engineering
lifecycle (SELC) as a framework that captures a common description of SE processes and
activities for the DHS enterprise (DHS, 2010, 3). The SELC was designed to support
tailoring based on project characteristics such as size, scope, complexity, risk, and security
categorization. The SELC requires each project to create an SELC tailoring plan that
details the approved adapted SE process a project intends to execute. For additional
emphasis, the DHS SELC notes that “tailoring is the cornerstone of any lifecycle process”
(DHS 2010, 4).
2.2.3. Capability Models
The Electronic Industries Alliance (EIA) EIA/IS-731.1 is a systems engineering
capability model that can be used to “develop, improve, and assess systems engineering
capability” (EIA, 2002, 2). It offers focus areas, themes, specific practices, generic
practices, and generic attributes that span the SE discipline. In particular, generic practices
are used to increase the ability to perform the specific practices. Six levels of capability are
defined to assess an organization’s level of performance in each particular generic and
specific practice:
Capability Level 2 – Managed
Capability Level 3 – Defined
Capability Level 4 – Measured
Capability Level 5 – Optimizing
A fundamental progression from levels zero through two to levels three through five is
an organization’s ability to have tailoring guidance that accompanies a standard SE process
and tailored applications of the standard process for each program.
An expressed limitation of the systems engineering capability model is that it is “not
intended to specify the details of ‘how to’ implement process activities.” As such, it does
not provide methods or tools to apply to the process.
The Capability Maturity Model Integration for Development (CMMI-DEV) assists
development organization managers and engineers regarding the application of CMMI best
practices (CMMI-DEV, 2010). It defines the levels of capability for process improvement:
Capability Level 0 – Incomplete
Capability Level 1 – Performed
Capability Level 2 – Managed
Capability Level 3 – Defined
Capability Level 3 is achieved primarily when processes are appropriately tailored:
A critical distinction between capability levels 2 and 3 is the scope
of standards, process descriptions, and procedures. At capability
level 2, the standards, process descriptions, and procedures can be
quite different in each specific instance of the process (e.g., on a
particular project). At capability level 3, the standards, process
descriptions, and procedures for a project are tailored from the
organization’s set of standard processes to suit a particular project
or organizational unit and therefore are more consistent, except for
14
2010, 25)
2.3. Process Reuse
Process tailoring can be considered a form of standard process reuse (Yoon et al.,
2001); however, tailoring involves process application to a unique project environment, as
opposed to the flexibility of the standard process itself. Particularly in software
engineering, patterns have been explored as a viable option for process reuse; in fact, the
concept has been extended to a method for developing pattern forms in systems
architecture (Cloutier and Verma, 2007).
In addition, Hagge and Lappe (2005) emphasize the need for capturing knowledge that
can be reused in future projects. They propose four introductory patterns for requirements
engineering and discuss a method for sharing observations across projects (Hagge and
Lappe, 2005). Rising (1999) also highlights the use of patterns as a way to transfer
knowledge from senior engineers to novices.
Fortune and Valerdi (2013) propose a framework for reusing SE products within an
organization (Figure 2-3) and highlight key considerations for process reuse. Similar to
process tailoring, process reuse begins with understanding the factors that influence the
decision to reuse an existing subprocess or element.
15
Figure 2-3: Systems engineering reuse framework (Fortune and Valerdi, 2013)
Among the key considerations for process reuse, Fortune and Valerdi (2013) argue that
a robust knowledge management system is critical to the success of process reuse. The
authors also note the reuse of a product at too high a level may not apply to the specific
environment of the new project. Conversely, reusing products at too low a level can cause
tailoring of enough significance that the effort saved by process reuse may be lost (Fortune
and Valerdi, 2013). This argues for additional guidance on process tailoring.
16
2.4. Process Tailoring Methods in Software Literature
Extending knowledge from the software process domain to the SE process domain has
been ongoing for quite some time (Boehm, 2006). Research in software process tailoring is
well-documented and can be extended to SE. Henderson-Sellers et al. (2014) provide an
extensive literature review of the application of method engineering to specific situations in
software development. Their review spans the various approaches to method tailoring and
case studies in the application of method tailoring.
Kuhrmann et al. (2014) also conducted a mapping study on the feasibility of method
engineering, which pertains to the process of constructing methods for software
development. The authors highlight 83 articles contributing to the research of software
process tailoring, and they categorize previous research into the following categories
(research type facets initially introduced by Wieringa et al., 2005):
Experience
Validation
Evaluation
Philosophical
Opinion
Solution proposal
The variety of research and tailoring approach proposals highlights the lack of a
common lexicon in method engineering (Kuhrmann et al., 2014). Of the 83 papers, less
than 20% are in the experience, validation, or evaluation categories. Other papers propose
17
solutions or remain at the theoretical level. This in part led to the conclusion that “strong
empirical evidence of the feasibility of ME” is still lacking (Kuhrmann et al., 2014, 1066).
In the few cases where process tailoring support at the project level has been researched
beyond the framework level, various methods have been applied. This dissertation
separates the previous methods for process tailoring into three types: case retrieval, pre-
formed, and rule-based.
2.4.1. Case Retrieval Method for Process Tailoring
The case retrieval-based method formulates a tailored software process based on
retrieving a tailored process from a library of historical projects (Ahn et al., 2003; Park and
Bae, 2013). Figure 2-4 depicts the process that Ahn et al. (2003) used for case retrieval.
They performed retrieval by comparing the values of domain factors of a new project to
those of past projects in the library. Manual tailoring follows the case-based reasoning
technique. Ahn et al. (2003) performed case-based reasoning by calculating a similarity
function, the degree to which a case in the process library exhibits project characteristics
(domain factors) that are comparable to those of the new project.
18
Figure 2-4: Overall procedure of process customization (Ahn et al., 2003, 296)
The case retrieval approach has two primary challenges. First, its accuracy is limited to
the extensiveness of the library of previous approaches, which requires a mature
organization. Second, the organization must have an established knowledge management
system that not only archives previous projects, but also tags the projects with the values
and domain factors required for retrieval.
2.4.2. Pre-Formed Method for Process Tailoring
In the pre-formed approach to process tailoring, an organization develops a standard set
of tailoring forms that apply to particular categories of projects (Plogert, 1996). The
approach of Plogert (1996) for applying tailoring forms consists of four steps:
19
1. Determination of the project class: Three classes of projects are proposed, namely
administrative information technology, technical or scientific information
technology, and commercial off the shelf (COTS).
2. Determination of the project size: Three project sizes are proposed.
3. Selection of the corresponding form, based on the first two steps.
4. Checking implementing conditions: Reassesses form selection based on additional
considerations for criticality, complexity, or other predefined conditions.
The DHS also supports the pre-formed approach to tailoring its SELC. The
organization has identified seven pre-formed “SELC tailored paths”:
1. System or product development
2. Information technology infrastructure
6. Information technology services
7. Facilities or construction (Tuttle, 2014)
The challenge with this approach is that it only includes the first level of tailoring, such
as hardware or software, and does not address the significant variability that can exist from
project to project within an organization. In addition, this approach does not easily allow
for growth; to modify the approach based on lessons learned would require revisiting the
pre-tailored processes altogether.
2.4.3. Rule-Based Method for Process Tailoring
Hausen (1998) employed a rule-based approach to software process tailoring. This
approach identifies the values of factors that address the rationale for tailoring based on
specific project characteristics. Each rule is created such that for a specific value of a
domain factor that exists for a new project, certain process elements are tailored (Kang et
al., 2008). Figure 2-5 depicts the tailoring rule structure used by Kang et al. (2008).
Tailoring actions include adding, removing, and replacing a process element. Figure 2-6
provides an example of a tailoring rule.
Figure 2-5: Tailoring rule structure (Kang et al., 2008)
21
Figure 2-6: Example of a tailoring rule (Kang et al., 2008)
2.5. Process Tailoring Case Studies
2.5.1. Framework Application Case Studies
Xu and Ramesh (2008) used two case studies to investigate how two project teams
performed process tailoring and what was the rationale for the decisions made. Their
research showed that the teams performed process tailoring based on the challenges they
faced, on project goals, and on environmental factors. Based on their case study analysis,
Xu and Ramesh (2008) proposed the model below (Figure 2-7).
22
Figure 2-7: Steps for process tailoring (Xu and Ramesh, 2008)
De Cesare et al. (2008) conducted a method tailoring case study at S-Service, a
medium-sized software development company. They define method tailoring as follows:
“methodologies are normally adapted to meet specific contextual characteristics” (de
Cesare et al., 2008, 157). The authors developed a method tailoring framework to elicit
tailoring methods from the company’s engineering team (Figure 2-8). S-Service tailored
the rational unified process for the studied case. Because the company’s objective was to
adapt the rational unified process for most of its projects, the tailoring activities focused
on what process elements were not necessary and therefore could be deleted. For project-
level tailoring, de Cesare et al. (2008) found that the methods (i.e., process elements)
were selected from a repository of previously used processes.
23
Figure 2-8: Method tailoring elicitation framework (de Cesare et al., 2008, 161)
The framework presented was rooted in previous research and applied to document a
real-world case. As the research of de Cesare et al. (2008) is limited to a single case study,
the authors recommend additional research be conducted within organizations with
“different contextual characteristics” (de Cesare et al., 2008, 166).
2.5.2. Model Application Case Studies
Park and Bae (2013) propose an approach to software process tailoring using process
slicing and case-based reasoning. With process slicing, the authors identify
interdependencies within the process and across subprocesses. This allows for an analysis
of which process elements should be tailored. Using case-based reasoning, the most similar
24
past tailoring case is retrieved to reduce false positive dependencies generated by process
slicing. Retrieval of the most similar case is performed using structural similarity in the
same way Ahn et al. (2003) did. Figure 2-9 depicts this process.
Figure 2-9: Process slicing with case-based reasoning (Park and Bae, 2013)
Their case study was conducted in support of a government organization, which has
several software development projects. The case consisted of a single organization and 23
projects within the organization, which should adhere to the organization’s standard
process or be tailored from the standard process.
The standard process contained 20 subprocesses, each with several artifacts and
activities. Park and Bae (2013) used the following tailoring considerations in their case
study:
Lifecycle
25
Schedule risk
Cost risk
A preliminary process slicing was generated using the organization’s standard process
interdependencies. Each project’s characteristics were used to retrieve the most similar case
from the organization’s repository. The intersection of the similar case and the preliminary
process slice generated the final tailored process. For each project, the authors compared
actual and modeled results (tailoring cases). The results showed that the use of case-based
reasoning increases the accuracy of the model when compared to results with just the
process slicing method.
Kang et al. (2008) performed a multiple case study to evaluate their proposed case
retrieval approach to software process tailoring: determining the applicable tailoring rules,
performing a similarity calculation, and retrieving the most similar case from the repository
of historical processes (Figure 2-10).
26
Figure 2-10: Overview of case retrieval method (Kang et al., 2008)
For their case studies, Kang et al. (2008) designed 30 projects for the military domain
and used the standard process of the Korean armed forces. After modeling the standard
27
process, Kang et al. (2008) tailored the standard process for the 30 cases. To validate their
model, they followed these steps:
1. Choose a case from the repository
2. Retrieve the most similar case using the model
3. Retrieve the most similar case by comparing the case to the other 29 cases
4. Compare the retrieved processes
5. Repeat the first four steps for all other cases
The results of their research showed that their approach retrieves the most similar case
in 70% of the cases. Kang et al. (2008) note that their research is limited by using artificial
cases, citing data availability as the primary inhibitor.
2.6. Summary
The common theme spanning the aforementioned industry and government standards
and guidelines is that tailoring the standard process is critical to effective SE planning and
execution. Moreover, each standard and guideline identifies various organizational and
environmental factors that influence the adaptation of a standard SE process to meet the
individual requirements of a project.
Another common theme is that none of the standards and guidelines offer a practical
method for SE process tailoring. At most, some guidelines offer a general framework that
starts with identifying what is important to process tailoring and ends with a custom SE
process for a project, with little detail in between.
28
Significant research has been done on tailoring software processes for projects, but the
proposed approaches have not been applied to SE. Software process tailoring case studies
that propose frameworks serve as a starting point for process tailoring methods; the
limitation of this research is the lack of decision support assistance when an SE practitioner
must tailor organizational processes at the project level. Additional case studies that offer
both a high-level adaptation process and decision support are limited to the software
research field. Even in this field, researchers identify case study and experiential research
as a shortfall in the literature.
This dissertation presents a multi-staged approach to SE process tailoring of adapting
industry-identified tailoring considerations to an organization and providing a decision
support tool to assist project systems engineers with customizing an SE process for their
projects. Therefore, this dissertation seeks to bridge the gap between the recognition of the
importance of SE process tailoring and project-level SE process tailoring by leveraging
process tailoring research in the software field.
29
3.1. Introduction
This chapter describes the systems engineering process tailoring model (SEPTM) that
was developed as a tool for tailoring systems engineering (SE) processes to meet the
unique needs of a project. Figure 3-1 presents an overview of the proposed SEPTM.
At a high level, the model was constructed based on an analysis of the generic SE
process described in the INCOSE Systems Engineering Handbook (INCOSE, 2015) that
influences process tailoring. This analysis forms the basis of an initial set of tailoring
considerations, from which a subset that is applicable to a particular organization was
selected. A rule set is then developed that links the tailoring considerations and the
organizational SE process. A systems engineer can then apply the rule set to the unique
attributes of a project to generate a customized SE process based on the conditions of a
particular project. Each of these activities is further described in the next subsections.
Figure 3-1: Overview of the systems engineering process tailoring model (SEPTM)
30
3.2. Identify the Initial Set of Tailoring Considerations
An important aspect of the model is determining the tailoring considerations used to
identify the critical aspects of a project. The tailoring considerations will broadly inform
what SE activities are needed for a given project.
The development of the initial list of tailoring considerations is based on an analysis of
the general SE process described in the INCOSE System Engineering Handbook
(INCOSE, 2015) (henceforth called INCOSE handbook) and its discussion of
considerations and enablers for process tailoring. As discussed in the Literature Review,
several handbooks and guidebooks stress the importance of SE process tailoring and
identify the considerations for process tailoring. The INCOSE handbook was selected as
one industry standard from which to determine the model’s initial tailoring considerations.
In Chapter 5, an investigation of tailoring considerations and their impact on SE process
tailoring decisions is identified as an area of future research.
The INCOSE handbook states the appropriate SE approach will vary based on the
unique needs of a project, and it also describes various lifecycle approaches: waterfall,
incremental, and agile (INCOSE, 2015). In the waterfall approach, SE stages are performed
serially; in the incremental approach, SE stages are performed serially multiple times; and
in the agile approach, the project goes through short development cycles with frequent
product releases (Aoyama, 1998). As part of the agile development cycles, the SE stages of
requirements definition, design, development, and testing are also performed iteratively.
Moreover, the associated reviews and documentation are performed and updated with each
release (Cao and Ramesh, 2008). Each of these approaches varies in terms of stage and
31
review frequency, as well as the amount of documentation required. As a result, a project’s
lifecycle approach must be a critical tailoring consideration.
The INCOSE handbook also describes crosscutting technical methods and includes a
section on specialty engineering analyses that must be considered by a project’s systems
engineer; however, not all of the analyses will apply to every project (INCOSE, 2015). For
example, interoperability should be considered for complex projects and those that
interface with other projects. An organization may have specific artifacts that are required
to address interoperability for certain projects or allow for deleting interoperability artifacts
based on the project type.
The INCOSE handbook’s tailoring section includes a discussion of SE process tailoring
and identifies considerations and enablers for tailoring: project scope, risk tolerance,
complexity and precedence of the system, and organizational/enterprise policies and
infrastructure (INCOSE, 2015). While the handbook identifies risk tolerance as a factor in
process tailoring, it does advise of the importance of placing tailoring emphasis on the
system and the project objectives, or project scope.
An analysis of the general SE process and tailoring discussion in the aforementioned
handbook resulted in the following list of tailoring considerations:
1. Lifecycle approach
2. Project scope
4. Organizational/enterprise policies and infrastructure
5. Modeling, simulation, and prototyping
32
8. Cost effectiveness analysis
9. Electromagnetic compatibility analysis
10. Environmental impact analysis
15. Safety and health hazard analysis
16. Sustainment engineering analysis
17. Training needs analysis
19. Value engineering
Of the 19 tailoring considerations, the first 4 (lifecycle approach, project scope,
complexity and precedence of the system, and organizational/enterprise policies and
infrastructure) apply to all organizations (INCOSE, 2015). The remaining 15 vary in
applicability, depending on the types of projects an organization executes. This requires an
organization to determine the appropriate subset of tailoring considerations for its project
portfolio.
33
3.3. Refine the Set of Tailoring Considerations for an Organization
In the second analytical step, the overarching list of tailoring considerations is reduced
to one that includes only the tailoring considerations relevant for that organization. This
step is performed by comparing the list of tailoring considerations to an organization’s
policies and standard SE process (Figure 3-2). An organization’s SE process is one of
many organizational processes, which CMMI defines as
a collection of definitions of the processes that guide activities in
an organization. These process descriptions cover the fundamental
process elements (and their relationships to each other such as
ordering and interfaces) that should be incorporated into the
defined processes that are implemented in projects, work groups,
and work across the organization. A standard process enables
consistent development and maintenance activities across the
organization and is essential for long-term stability and
improvement. (CMMI, 2010)
34
The organizational SE process must be analyzed to confirm which of the enterprise
tailoring considerations are incorporated into the standard process. For example, is mass
properties engineering analysis discussed in the SE process, and are there process stages,
reviews, or artifacts tied to mass properties engineering analysis in the standard SE
process? Additionally, other organizational policies relating to systems development or
acquisition must be examined; for example, does the organization have security
requirements for technology?
Each organizational tailoring consideration is then converted to a question with
possible “values” as answers. The tailoring considerations can have values that are either
yes or no (e.g., environmental impacts need to be considered or not) or multiple (e.g.,
lifecycle approach may be waterfall, incremental, or agile). Forming the questions and
values becomes a critical input to establishing the rule set for the SEPTM.
3.4. Definitions
Before a tailoring rule set can be developed, some tailoring operations must be defined.
For the purposes of this research, the following definitions apply:
“Standard” is used when a stage, review, or artifact is to be developed consistent
with the scope of the activity defined in the organizational SE policy.
“Tailored” is used when a stage, review, or artifact is modified from the scope in
the organizational SE policy. This could include combining two or more stages,
reviews, or artifacts into one, and also repeating stages or reviews multiple times
within a project. For example, the process tailoring model recommends that the
35
integration readiness review occur with each release in a project that uses agile
development methodology.
“Deleted” is used when the stage, review, or artifact is not needed and is removed
from the project’s tailored SE process altogether. An example of this is the removal
of a system design document for a commercial off the shelf (COTS) acquisition.
3.5. Establishing the Rule Set
3.5.1. Base Rules
Once the tailoring considerations have been selected for an organization, and tailoring
conditions have been defined, a rule set must be generated to link the tailoring
considerations and associated values to the elements of the organization’s standard SE
process and its process elements. This research applies the CMMI-DEV definition of
process element, which is
the fundamental unit of a process. A process can be defined in
terms of subprocesses or process elements. A subprocess is a
process element when it is not further decomposed into
subprocesses or process elements. Each process element covers a
closely related set of activities (e.g., estimating element, peer
review element). Process elements can be portrayed using
templates to be completed, abstractions to be refined, or
descriptions to be modified or used. A process element can be an
activity or task. (CMMI, 2010)
Figure 3-3 depicts the matrix used to establish the rule set.
36
Figure 3-3: Establishing the rule set
In area 1 of the table, all elements (1 to M) of the standard SE process are listed
vertically. For example, the DHS SE process has 107 process elements (“M”). In area 2,
each tailoring consideration (1 to N) and associated values (1 to O) are listed horizontally.
“N” represents the number of tailoring considerations identified for the organization, as
described in section 2.2, and the values represent potential attributes of a project within the
organization.
As an example, if tailoring consideration 1 is interoperability analysis, the potential
values would be “yes” and “no” given that a system may be standalone or connected. For
each intersection of a process element and a tailoring consideration-value combination in
area 3, a tailoring rule is established. This requires two levels of decisions. First, a
determination is made as to whether a tailoring consideration is relevant or not for a
particular process element; and second, if so, a determination is made for each value of the
tailoring consideration whether the process element should be kept standard, tailored, or
Tailoring Considerations → TC 1 TC 2 … TC N
Values →
2
37
deleted. Thus, each relevant intersection in area 3 will contain either “S” for standard, “T”
for tailored, or “D” for deleted.
In summary, the resulting rule set is a list of tailoring considerations for which each
value drives a tailoring decision (standard, tailored, or deleted) for every relevant process
element within the organizational SE process.
3.5.2. Prioritization Rules
Because the SEPTM executes multiple rules at the same time, it is essential to resolve
any conflicts in situations where two or more rules are linked to the same process element.
For example, suppose the development type and development methodology tailoring
considerations trace to the same process element, and the associated questions for those
tailoring considerations are the following:
Does the project intend to develop a solution or procure a COTS solution to meet
project requirements?
Will the project be using the waterfall, incremental, or agile development
methodology?
In this example, a prioritization rule that de-conflicts the recommended tailoring
decision is necessary, such as,
if the project is a COTS procurement, the process element should not be executed;
else, if the project is a development effort and employing a waterfall or an
incremental methodology, the process element should be executed (standard or
tailored);
38
else, if the project employs agile development, the process element should be
tailored.
3.6. Apply Model to a New Project and Validate
Once the model has been developed for an organization, project systems engineers can
use it for tailoring SE processes to meet the needs of a specific project. To do so, a systems
engineer must have a solid understanding of the acquisition strategy and the project
environment. This is true whether applying a rule-based methodology, as is the case here,
or tailoring an SE process ad hoc. To exercise the model, a systems engineer must align the
characteristics of the project to a value associated with each tailoring consideration. Once
this is complete, the model outputs the tailored SE process, identifying which of the full set
of the organization’s SE process elements should be executed as standard, which should be
tailored, and which can be deleted.
After the model has been applied to a real-world situation, lessons learned must be
incorporated into the model by reassessing the organization’s list of tailoring considerations
and the organization’s rule set. Moreover, as SE research is constantly refining best
practices and industry standards are updated, the organization must also revisit the initial
tailoring consideration set to assess changes from industry.
39
Chapter 4: Case Studies
A multiple case study approach was selected to investigate whether the SEPTM can
reduce the amount of manual process tailoring of the SE lifecycle on actual projects. The
case study approach was adapted from Kitchenham’s (1995) seven steps for performing
case studies:
3. Identify the method of comparison
4. Consider the effects of confounding factors
5. Plan the case study
6. Conduct the case study
7. Analyze and report the results
Each of the above steps is described in detail in the following subsections.
4.1. Definition of Hypothesis
Consistent with the goal of this dissertation, the hypothesis for the case study is as
follows:
Using the SEPTM to tailor a project’s systems engineering process can reduce the
amount of manual tailoring required.
This hypothesis will be considered met if the conditions below are achieved.
40
1. The number of decisions made to tailor a project’s process are reduced. Henderson-
Sellers et al. (2014) note that determining the best project methodology requires
specific tailoring to a project’s unique situation. Measuring the benefit of such an
activity can be based on the amount of effort required to tailor standard processes
(Xu and Ramesh, 2007). For the purposes of this analysis, this research extends that
definition to the number of manual decisions required to produce a project-specific
systems engineering (SE) process. By design, the SEPTM fixes the number of
decisions on the number of tailoring considerations established for an organization.
If an organization applies all of the initial 19 tailoring considerations, the number of
manual decisions will be 19. This of course makes the second condition all the
more important.
2. The SEPTM approach matches the ad hoc approach to an acceptable level.
Consistent with validation studies of software engineering practices, a 75%
acceptable matching level is used (Daneva and Ahituv, 2010; Krishnan et al., 1999;
Ramasubbu et al., 2005). Thus, for the model to be considered valid for a given
project, at least 75% of the process elements must match. For the model to be
considered valid overall, at least 75% of the projects must meet that threshold.
Two additional analyses were performed: First, an analysis of each tailoring
consideration was conducted to understand if the model failed to match a majority of
projects for any particular tailoring consideration. Second, an analysis of each process
element was also performed to understand which elements had the most significant
differences between the projects’ plans and the model’s outputs.
41
4.2. Pilot Project Selections
For this case study, both an organization with an acquisition portfolio and projects
within that portfolio were selected.
First, an organization was selected with
a diverse enough acquisition portfolio, to make tailoring of the standard process an
essential task in the SE planning of a project;
a standard SE process, required as a starting point for projects within the
organization’s portfolio, but that promoted process tailoring;
a standard way for projects to report planned tailoring approaches to the standard
SE process; this allows for a comparison of the model against tailoring approaches
used on real-world projects.
Second, projects were selected within the organization’s portfolio based on data availability
and variability across the tailoring considerations.
4.2.1. Selection of the Organization
The organization selected for the case study is the Department of Homeland Security
(DHS), an organization with a large acquisition portfolio, given its five core missions:
1. “Prevent terrorism and enhance security,
2. Secure and manage our borders,
3. Enforce and administer our immigration laws,
4. Safeguard and secure cyberspace, and
5. Ensure resilience to disasters.” (DHS Mission, 2016)
42
Each of these missions requires the acquisition of technology and infrastructure to
support the organization’s frontline officers and agents. Therefore, the DHS acquisition
portfolio is diverse, ranging from border security surveillance technologies and
infrastructure to information technology systems for immigration and cybersecurity.
The DHS published a systems engineering lifecycle (SELC), defined as “a guiding
framework that provides a vocabulary, order, and description of the activities enabling
efficient and effective delivery of capability to users” (DHS, 2010). Figure 4-1 depicts the
DHS SELC.
Figure 4-1: DHS systems engineering lifecycle process (DHS, 2010)
The DHS standard SELC process identifies 9 SE stages, 11 technical reviews, and 87
artifacts to be executed through the life of a project, for a total of 107 SE process elements.
Figure 4-1 also provides a note that mandates an SELC tailoring plan be developed for
each project within the DHS acquisition portfolio. This offered a standard format of data
with which to compare the model’s results to the actual SE process tailoring approaches.
43
4.2.2. Selection of the Projects
The second key element required for the case study is a set of tailored SE processes
implemented on projects within the organization. Twenty-four DHS projects were selected
for analysis based on data availability and variability across the tailoring considerations
(Table 4-1). For example, some projects executed full research and development, while
others were COTS acquisitions; some used a waterfall development approach, while others
used an incremental or agile. Quantitative data regarding tailoring outcomes were gathered
and analyzed from each of the 24 projects and compared the data to the model, as described
in Section 3.3.
Case Study
Technology Demonstration Planned Planned/Not Planned 7/17
Environmental Impact Impact/No Impact 7/17
Development Methodology Waterfall/Incremental/Agile 11/11/2
Development Type Development/COTS-NDI 19/6
4.3. Method of Comparison
Given that the projects vary due to their unique characteristics, the projects were not
compared to each other or to a single output of the SEPTM. Rather, the 24 projects’
characteristics were each input into the model, generating 24 SEPTM-based tailored SE
processes. As a result, each project has a tailored SE process, developed by the project
team, and a tailored SE process, developed by the SEPTM. Each model-data combination
is then analyzed.
4.4. Effects of Confounding Factors
Internal validity of the case study can be affected by confounding factors. Kitchenham
(1995) identifies the most significant confounding factors as “likely to be:
Learning how to use a method or tool as you try to assess its benefits.
Using staff who are either very enthusiastic or very skeptical about the method
or tool.
Comparing different application types.”
The SEPTM, once developed for an organization, offers a set of questions whose
answers are inputs to the model based on a project’s characteristics. Unlike using a
complicated tool to automate software development, for example, using the SEPTM is as
simple as completing a questionnaire regarding a project. Therefore, the effect of learning
how to use the SEPTM is avoided, because the inputs to the model are the characteristics of
the project and, thus, available to the project team. This also eliminates the concern of
enthusiasm or skepticism in employing the tool. To reduce the effect of comparing
different application types, multiple projects are used, as Kitchenham (1995) suggests.
4.5. Case Study Planning
1. Define the hypothesis and the associated metrics.
2. Collect case study data.
3. Analyze and select organizational tailoring considerations.
4. Generate rule set for the SEPTM.
5. Generate tailored SE approaches for each project using SEPTM.
46
4.6. Case Study Conduct
As detailed in Section 3.2, initial tailoring considerations were identified in Step 1 of
the SEPTM. Based on an analysis of the INCOSE handbook (INCOSE, 2015), 19 initial
tailoring considerations were identified to serve as the starting point for determining the
tailoring considerations suitable for a particular organization.
In the conduct of this case study, analysis began with selecting organizational tailoring
considerations for the DHS. Based on the DHS tailoring considerations, a rule set was
established that links the DHS tailoring considerations to the DHS SELC process. After
generating a SEPTM-based tailored SE process for each of the 24 DHS projects, each pair
of model-based and original SE processes was compared to evaluate whether the model
could reduce the number of tailoring decisions while retaining the original project SE
processes at an acceptable level. The subsections below describe the case study in detail.
4.6.1. Organizational Tailoring Consideration Selection
As described in Section 3.3, the first step in applying the tailoring model to an
organization is determining the subset of the 19 overarching tailoring considerations that
are applicable to the organization. As stated previously, the first four tailoring
considerations apply to all organizations and were adapted for the DHS tailoring
consideration list:
4. Organizational/enterprise policies and infrastructure
Analysis of the DHS organizational/enterprise policies yielded additional
considerations pertaining to security, privacy, and intelligence (DHS, 2015).
Additional analysis of the DHS SELC process was performed to determine which of
the remaining 15 tailoring considerations are applicable. Of these, five were applicable to
all projects per the DHS SE process, six were not discussed in the DHS SE process guide,
and five may apply depending on the needs of the specific project. In support of the focus
of this case study on process tailoring, the four tailoring considerations that may be
applicable were included as DHS tailoring considerations in the model. As a result, the 10
organizational tailoring considerations identified and applied for this research are as
follows:
3. Intelligence impact
4. Interoperability impact
7. Environmental impact
9. Development type (complexity and precedence of the system)
48
4.6.2. Rule Set
4.6.2.1. Linking Tailoring Considerations to the Standard Process
With the 10 DHS tailoring considerations in place, the rule set for the DHS was
established by linking the tailoring considerations to the standard DHS SE process. An
initial determination of relevance was made for each combination of tailoring consideration
and process element. The process elements that do not link to any tailoring consideration
were considered to be part of the default process.
Table 4-2 through Table 4-11 provide the relevant process elements for each tailoring
consideration.
49
DHS Standard SE Process Elements
FIPS 199 Security Categorization
Security Requirements Traceability Matrix
System Security Plan
Disaster Recovery Plan
Security Risk Assessment
Contingency Plan
Security Incident Reports
Certification and Accreditation Updates (every 3 years or when major change is made)
50
Privacy Threshold Analysis
Privacy Impact Assessment
Table 4-4: Relevant process elements for intelligence impact
Intelligence Support Plan
HLS EA Program Alignment Decision Request
Service Reuse Plan
Data Management Plan
Service Level Agreements
Data Architecture Document
Technology Insertion Package
Data Insertion Package
Service Insertion Package
51
Section 508 National Security Exception Determination
Section 508 Electronic and Information Technology Accessibility Plan
Table 4-7: Relevant process elements for technology demonstration
Technology Demonstrator Results Report
Environmental Impact Assessment
Requirements Definition Stage
Functional Requirements Document
Requirements Traceability Matrix
System Requirements Document
Logical Design Document
System Design Document
Table 4-10: Relevant process elements for development type impact
Design Stage
Development Stage
Table 4-11: Relevant process elements for project size
53
Concept of Operations
Operational Requirements Document
Configuration Management Plan
Risk Management Plan
Quality Assurance Plan
Functional Requirements Document
Requirements Traceability Matrix
Logical Design Document
System Design Document
Site Prep Plan
4.6.2.2. Rules
Each tailoring consideration was then developed into a question that, when answered,
drives tailoring for the project. The questions and associated rules used in this research are
below.
Tailoring consideration: security impact
Question: Will the solution acquired contain any information or data requiring special
protections?
Rules:
If yes, process elements linked to this tailoring consideration should be executed
(standard or tailored). Note that standard or tailored is acceptable to promote
additional tailoring beyond the model.
If no, process elements linked to this tailoring consideration should not be
executed (deleted).
Tailoring consideration: privacy impact
Question: Will the solution acquired relate in any way to an individual (for example,
social security number, communication traffic, image or video, radioactivity level)?
Possible values: yes, no
Rules:
55
executed (deleted).
Question: Will the solution acquired contain any intelligence data or information?
Rules:
executed (deleted).
Question: Will the solution acquired contain any equipment or interconnected system
or subsystem of equipment or software?
Rules:
56
executed (deleted).
Tailoring consideration: accessibility impact
Questions: (1) Will the solution acquired contain any information or data that is
accessible to a user of the solution? (2) Will the solution qualify for a Section 508
exception?
Rules:
If yes to both questions, the accessibility plan should be deleted and the
exception determination request should be executed (standard or tailored). Note
that standard or tailored is acceptable to promote additional tailoring beyond the
model.
If yes to the first question and no to the second, the accessibility plan should be
executed (standard or tailored) and the exception determination request should
not be executed (deleted).
executed (deleted).
Tailoring consideration: technology demonstration
Question: Does the project strategy include any plan for pilots or demonstrations?
57
Rules:
executed (deleted).
Tailoring consideration: environmental impact
Question: Could the solution acquired have any potential impact on the environment
(for example, radiating equipment, construction)?
Rules:
executed (deleted).
Tailoring consideration: development methodology
Question: Will the project be using the waterfall, incremental, or agile development
methodology?
Rules:
58
If waterfall or incremental, the process elements linked to this tailoring
consideration should be executed (standard or tailored). Note that standard or
tailored is acceptable to promote additional tailoring beyond the model.
Additionally, an assumption was made that the tailoring approach for a project
would be updated with each increment of a project. As a result,incremental is
treated the same as waterfall for this analysis.
If agile, process elements linked to this tailoring consideration should be
tailored.
Tailoring consideration: development type
Question: Does the project intend to develop a solution or procure a COTS solution to
meet project requirements?
Rules:
If yes, process elements linked to this tailoring consideration should not be
executed (deleted).
If no, process elements linked to this tailoring consideration should be executed
Tailoring consideration: project size
Question: Is the project’s lifecycle cost greater than $300M?
Rules:
59
If yes, the process elements linked to this tailoring consideration should be
executed (standard or tailored). Note that standard or tailored is acceptable to
promote additional tailoring beyond the model.
If no, the process elements linked to this tailoring consideration should be
tailored.
4.6.2.3. Rule Prioritization
In certain cases, the same process element will map to multiple tailoring considerations.
For those cases, a prioritization schema was developed to yield only one model output for
each process element.
Associated questions:
methodology?
Is the project’s lifecycle cost greater than $300M?
Rule: If either the project is not greater than $300M or agile development is employed,
the process element should be tailored; else, it should be executed as standard or tailored.
Rule priority: project size, development methodology, and interoperability impact
methodology?
60
Will the solution acquired contain any equipment or interconnected system or
subsystem of equipment or software?
Rule: If the project does not have an interoperability impact, the process element should
not be executed. If the project does not have an interoperability impact and either the
project is not greater than $300M or agile development is employed, the process element
should be tailored; else, the process element should be executed as standard or tailored.
Rule priority: development type and development methodology
Does the project intend to develop a solution or procure a COTS solution to meet
project requirements?
methodology?
Rule: If the project is a COTS procurement, the process element should not be
executed; else, if the project is a development effort and employing a waterfall or an
incremental methodology, the process element should be executed (standard or tailored);
else, if the project employs agile development, the process element should be tailored.
Rule priority: project size and security impact
Is the project’s lifecycle cost greater than $300M?
Will the solution acquired contain any information or data requiring special
protections?
61
Rule: If the project has no security impact, the project should not execute the process
element (should be deleted). If the project has a security impact and the project is not
greater than $300M, the process element should be tailored; else, the process element
should be executed as standard or tailored.
4.6.2.4. Rule Set Summary
By design, each of the process elements that do not trace to a tailoring consideration
should be executed (standard or tailored). Therefore, a full modeled tailoring plan can be
developed and compared to a fully executed tailoring plan.
The result of establishing the rule set is a full intersection of the tailoring considerations
and DHS SELC process elements (Table 4-12). The complete case study rule set is
contained in Appendix B: Case Study Rule Set. Note again that for any given tailoring
consideration, only a subset of process elements will be relevant; that is, empty table
entries reflect that there is no relevance between that tailoring consideration and the
intersecting standard SE process element. For example, in Table 4-12, tailoring
consideration 2 and tailoring consideration 10 are relevant to process element 1.
62
Table 4-12: Establishing the rule set for the DHS case study
Tailoring Considerations
→
TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 TC 7 TC 8 TC 9 TC 10
S e
o lo
g y
e n ta
p m
p m
iz e
Values → Y N Y N Y N Y N Y N Y N Y N W I A D C Y N
Process Element #1
Process Element #107
ST D ST D T D
Values: Y=Yes, N=No, W=Waterfall, I=Incremental, A=Agile, D=Developmental, C=COTS
Entries: S=Standard, T=Tailored, D=Deleted
4.6.3. Model-Based Project Tailoring Plans
With the rule set established for DHS as described above, the next step is inputting the
attributes of each of the 24 projects into the model to generate a custom SE process for
each project. Figure 4-2 depicts an example of a project’s input and output, highlighting
63
that the input is addressing the 10 tailoring considerations based on project characteristics,
and the output is a tailoring decision for each of the 107 elements within the DHS SE
process. Appendix C: Project Cases includes the following:
Project attributes
Tailored SE approach for each project
Modeled SE approach for each project
A determination of whether or not, for each process element, the model matches the
project’s plan
Figure 4-2: Example of a model input and output for a project
64
4.7. Analysis and Results
Tailoring approaches from the projects and those generated using the model for each
project are now available for analysis. Three analyses were performed to investigate the
SEPTM approach to SE process tailoring.
First, the hypothesis was evaluated based on an analysis of how well the SEPTM
approach matches the ad hoc approach for each project and overall. Second, an analysis of
each tailoring consideration was conducted to understand if the model failed to match a
majority of projects for any particular tailoring consideration, thereby highlighting any
potential improvements to the model. Third, an analysis of each process element was
performed to understand which elements had the most significant differences between the
projects’ plans and the model’s outputs. The following subsections describe the three
analyses and associated results.
For the purposes of this research, at the process element level, a match occurs when the
actual result (AR) is the same as the modeled result (MR). Possible actual tailoring
decisions are standard (S), tailored (T), and deleted (D); and possible modeled tailoring
decisions are standard or tailored (ST), tailored (T), and deleted (D).ST is used because
.
65
4.7.1. Analysis of Projects
Recall the hypothesis from Section 4.1:
Using the SEPTM to tailor a project’s systems engineering process can reduce the
amount of manual tailoring required.
The success criteria for confirming this hypothesis are the following:
Criterion 1: The number of decisions required to generate a custom SE process for
a project is reduced.
Criterion 2: The SEPTM approach matches the ad hoc approach to an acceptable
level; that is, for the model to be considered valid for a given project, at least 75%
of the process elements must match; for the model to be considered valid overall, at
least 75% of the projects must meet that threshold.
Before presenting the specific results of the analysis, a key point must be noted. The
analysis of the 11 projects using the incremental development methodology showed that
the tailoring plan for each project was limited to a single increment of the project, with a
corresponding note that the tailoring plan would be updated prior to the initiation of each
66
new increment. As such, the incremental development projects actually reflect a waterfall-
based tailoring strategy and were evaluated accordingly.
Table 4-14 displays the results of comparing the actual and modeled case study data.
Figure 4-3 depicts these results in order of highest match percentage.
Table 4-14: Case study results by project
Project
SST TST TT DD SD TD DST DT ST
Project 1 34 44 0 3 0 0 26 0 0 81 76%
Project 2 48 37 0 2 0 0 20 0 0 87 81%
Project 3 64 18 0 12 8 0 5 0 0 94 88%
Project 4 14 18 9 27 11 2 20 1 5 68 64%
Project 5 21 21 17 11 11 0 24 1 1 70 65%
Project 6 39 9 0 24 5 2 28 0 0 72 67%
Project 7 48 8 0 40 0 1 10 0 0 96 90%
Project 8 82 20 0 1 0 0 4 0 0 103 96%
Project 9 81 2 0 7 1 0 16 0 0 90 84%
Project 10 69 6 3 3 0 0 9 0 17 81 76%
Project 11 49 17 0 16 0 1 24 0 0 82 77%
Project 12 42 7 0 24 8 0 26 0 0 73 68%
Project 13 40 11 0 32 0 0 24 0 0 83 78%
Project 14 92 7 0 3 0 0 5 0 0 102 95%
Project 15 92 8 0 4 0 0 3 0 0 104 97%
Project 16 76 19 0 5 1 0 6 0 0 100 93%
Project 17 53 20 0 8 1 1 24 0 0 81 76%
Project 18 78 10 0 3 12 0 4 0 0 91 85%
Project 19 41 11 0 31 1 0 23 0 0 83 78%
Project 20 43 16 6 18 3 0 10 1 10 83 78%
Project 21 9 17 15 17 3 6 38 2 0 58 54%
Project 22 33 21 7 6 2 2 24 0 12 67 63%
Project 23 61 17 17 4 0 0 6 0 2 99 93%
Project 24 45 18 10 9 0 3 14 0 8 82 77%
% of Projects that Match 75%
67
Figure 4-3: Case Study Results by Project
Per Table 4-14, 75% of projects had at least 75% of actual and modeled results that
match. This, along with a reduction of the number of process tailoring decisions from 107
to 10, demonstrates the hypothesis that using the SEPTM to tailor a project’s SE process
can reduce the amount of manual tailoring required.
That said, it is important to understand the results where the model did not match the
actual data. There were three projects (4, 5, and 18) that had more than ten incorrect
matches due to the AR being S and MR being D. This occurs when the model recommends
deleting a process element that was actually included in a project’s tailored process. If the
68
model were to be deemed the “correct” approach, this scenario would be considered a false
positive. If the actual approach were to be deemed as the “correct” approach, then these
instances would point to the model being incorrect. Also, there were several projects that
did not perform process elements that were recommended by the model (DST column in
Table 4-14). If the model were to be deemed as the “correct” approach, this scenario would
be considered a false negative. If the actual approach were to be deemed as the “correct”
approach, then these instances would point to the model being incorrect. While these are
likely cases of non-compliance with the DHS standard SE process, there may be valid
reasons for not performing those steps that are not addressed with the established ten
tailoring considerations of the model. As will be discussed in Section 5, future research
should include an evaluation of the “correct approach”, which will firmly determine true
and false positives and negatives.
Sections 4.7.2 and 4.7.3 include additional investigations that will clarify reasons why
the model may not have reproduced the actual approach. Section 4.7.2 investigates issues
with the tailoring considerations, and 4.7.3 investigates issues with individual process
elements, respectively.
4.7.2. Analysis of Tailoring Considerations
There are two questions forming the analysis of each tailoring consideration. Table
4-15 lists those questions and provides a corresponding example using the security tailoring
consideration. Each of the 10 tailoring considerations was assessed by comparing the
model’s output for each project to the data from each project’s SE process tailoring plan.
Table 4-15: Tailoring consideration analysis questions
69
To analyze each tailoring consideration, a default process was established based on
default project attributes within the rule set (Table 4-16). Projects that had an attribute
different from the default were compared to the model for that particular attribute. For
example, the default project attribute for the security tailoring consideration is “No;”
therefore, the 14 projects that do have security impacts were compared to the model to
assess whether the projects performed the process elements relevant to the security
tailoring consideration.
Analysis Question Security Tailoring Co

Date post:	28-Mar-2022
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

Tailoring a Large Organization’s Systems Engineering ...

Documents