transcript
_Tailoring a Large Organization’s Systems Engineering Process to
Meet Project-
Specific Needs
M.S. in Mechanical Engineering, May 2005, Texas A&M
University
A Dissertation submitted to
of The George Washington University
in partial satisfaction of the requirements
for the degree of Doctor of Philosophy
August 31, 2016
Dissertation directed by
Thomas Mazzuchi
Professor of Engineering Management and Systems Engineering &
of Decision Sciences
ii
The School of Engineering and Applied Science of The George
Washington University
certifies that Matthew S. Graviss has passed the Final Examination
for the degree of Doctor
of Philosophy as of July 27, 2016. This is the final and approved
form of the dissertation.
Tailoring a Large Organization’s Systems Engineering Process
to
Meet Project-Specific Needs
Matthew S. Graviss
Dissertation Research Committee:
Engineering, Dissertation Co-Director
Engineering & of Decision Sciences, Dissertation
Co-Director
Steven Stuban, Professorial Lecturer in Engineering Management
and
System Engineering, Committee Member
Systems Engineering, Committee Member
Systems Engineering, Committee Member
All rights reserved
Tailoring a Large Organization’s Systems Engineering Process to
Meet Project-Specific
Needs
Systems engineers are faced with the challenge of adhering to broad
systems
engineering policies, while simultaneously tailoring systems
engineering processes to meet
the unique challenges facing their projects. Tailoring is often
performed ad hoc, and
determining which stages, steps, and artifacts of the process are
necessary can be time-
consuming and challenging. Systems engineering guidebooks across
industry and
government organizations often stress the importance of tailoring,
yet offer little practical
guidance on how to perform the function. This dissertation proposes
a model for
automating the systems engineering tailoring process through
defining a set of
organizational rules and a minimal set of project-specific inputs.
To evaluate the proposed
approach, the model is analyzed through several case studies within
the Department of
Homeland Security.
1.3. Intended Contributions
................................................................................................
3
2.1. Introduction
..................................................................................................................
6
2.2. Process Tailoring Guidance in Systems Engineering Standards
and Handbooks ... 6
2.2.1. Industry Standards and Guidelines
....................................................................
6
2.2.2. Government Standards and
Guidelines...........................................................
10
2.2.3. Capability Models
............................................................................................
12
2.3. Process Reuse
.............................................................................................................
14
2.4.2. Pre-Formed Method for Process Tailoring
..................................................... 18
2.4.3. Rule-Based Method for Process Tailoring
..................................................... 20
2.5. Process Tailoring Case Studies
.................................................................................
21
vi
2.6. Summary
.....................................................................................................................
27
3.1. Introduction
................................................................................................................
29
3.2. Identify the Initial Set of Tailoring Considerations
................................................. 30
3.3. Refine the Set of Tailoring Considerations for an Organization
............................ 33
3.4. Definitions
..................................................................................................................
34
3.5.1. Base Rules
.........................................................................................................
35
3.5.2. Prioritization Rules
...........................................................................................
37
3.6. Apply Model to a New Project and Validate
...........................................................
38
Chapter 4: Case Studies
............................................................................................................
39
4.1. Definition of Hypothesis
...........................................................................................
39
4.2. Pilot Project Selections
..............................................................................................
41
4.2.1. Selection of the Organization
..........................................................................
41
4.2.2. Selection of the Projects
...................................................................................
43
4.3. Method of Comparison
..............................................................................................
44
4.4. Effects of Confounding Factors
................................................................................
45
4.5. Case Study Planning
..................................................................................................
45
4.6. Case Study Conduct
...................................................................................................
46
4.6.1. Organizational Tailoring Consideration Selection
......................................... 46
vii
4.6.2.2. Rules
........................................................................................................
54
4.6.3. Model-Based Project Tailoring Plans
.............................................................
62
4.7. Analysis and Results
..................................................................................................
64
4.7.1. Analysis of Projects
..........................................................................................
65
4.7.2. Analysis of Tailoring Considerations
.............................................................
68
4.7.3. Analysis of Process Elements
..........................................................................
74
Chapter 5: Conclusions and Future Work
...............................................................................
75
References
...................................................................................................................................
78
Appendix A: DHS Systems Engineering Lifecycle Stages, Reviews, and
Artifacts ............ 84
Appendix B: Case Study Rule Set
............................................................................................
89
Appendix C: Project Cases
........................................................................................................
98
Appendix D: Analysis of Process Elements
...........................................................................
187
viii
Figure 2-1: Inputs-process-outputs diagram for project planning
(INCOSE, 2015)............... 7
Figure 2-2: Adaptation sequence (ISO/IEC, 2008, 38)
.............................................................
9
Figure 2-3: Systems engineering reuse framework (Fortune and
Valerdi, 2013) ................. 15
Figure 2-4: Overall procedure of process customization (Ahn et al.,
2003, 296) ................. 18
Figure 2-5: Tailoring rule structure (Kang et al., 2008)
.......................................................... 20
Figure 2-6: Example of a tailoring rule (Kang et al., 2008)
.................................................... 21
Figure 2-7: Steps for process tailoring (Xu and Ramesh, 2008)
............................................ 22
Figure 2-8: Method tailoring elicitation framework (de Cesare et
al., 2008, 161)................ 23
Figure 2-9: Process slicing with case-based reasoning (Park and
Bae, 2013)....................... 24
Figure 2-10: Overview of case retrieval method (Kang et al., 2008)
..................................... 26
Figure 3-1: Overview of the systems engineering process tailoring
model (SEPTM) ......... 29
Figure 3-2: Establishing organizational tailoring considerations
........................................... 33
Figure 3-3: Establishing the rule set
.........................................................................................
36
Figure 4-1: DHS systems engineering lifecycle process (DHS, 2010)
.................................. 42
Figure 4-2: Example of a model input and output for a project
............................................. 63
Figure 4-3: Case Study Results by Project
...............................................................................
67
ix
Table 4-2: Relevant process elements for security impact
...................................................... 49
Table 4-3: Relevant process elements for privacy impact
...................................................... 50
Table 4-4: Relevant process elements for intelligence impact
............................................... 50
Table 4-5: Relevant process elements for interoperability impact
......................................... 50
Table 4-6: Relevant process elements for accessibility
...........................................................
50
Table 4-7: Relevant process elements for technology demonstration
.................................... 51
Table 4-8: Relevant process elements for environmental tailoring
consideration ................ 51
Table 4-9: Relevant process elements for development methodology
impact ...................... 51
Table 4-10: Relevant process elements for development type impact
................................... 52
Table 4-11: Relevant process elements for project size
.......................................................... 52
Table 4-12: Establishing the rule set for the DHS case study
................................................. 62
Table 4-13: Definition of match: correct and incorrect results
............................................... 64
Table 4-14: Case study results by
project.................................................................................
66
Table 4-16: Default model attributes
........................................................................................
70
Table 4-17: Tailoring consideration analysis example
............................................................
72
Table 4-18: Case study analysis results
....................................................................................
73
Table 4-19: Process elements with the highest failure rate
..................................................... 75
1
Most organizations develop their own systems engineering (SE)
guidelines to specify
the stages, reviews, and artifacts that should be executed on
projects. Standard processes
are important for today’s projects, as they employ standardized
terminology and may
prevent project teams from reinventing the wheel; however, formal
and informal processes
must be balanced to yield the efficiencies gained through
standardization and the
effectiveness gained by adaptability (Laufer, 2001). Since no two
projects are identical, no
two processes are either (Humphrey, 1989). Organizations must
develop tailorable
guidelines that are broad enough to span the range of projects
within their portfolio (Hwang
& Park, 2006). This places the burden of tailoring the
organization’s SE guidelines on
project systems engineers.
Process tailoring refers to the modification of a standardized
process to meet the unique
needs of a project (Xu & Ramesh, 2007). Tailoring can include
determining which stages
and reviews are necessary and should be executed multiple times,
which artifacts should be
produced, and even what content within an artifact is necessary.
Successful tailoring results
in modified processes that achieve the goals of the standard
process model (ISO/IEC
15288, 2015). Tailoring is often performed ad hoc, without
guidelines or rules (Pedreira et
al., 2007), and is usually completed based on the tacit knowledge
of the project team (Xu
and Ramesh, 2009). Project budget and schedule, as well as product
quality, are directly
2
affected by process implementation. Choosing and adapting the right
design method is
critical to successfully executing SE concepts and principles
(Verma and Fabrycky, 1997).
Significant research has been done on tailoring software processes
for projects, but the
proposed approaches have not been applied to SE. Existing SE
guidelines and handbooks
stress the importance of tailoring SE processes to meet the unique
needs of a project, but
provide little guidance on process tailoring (Browning et al.,
2006; Pereira et al., 2007).
Therefore, project teams perform process tailoring ad hoc, making a
decision on each
process element as to whether or not the element is required and,
if so, whether or not it
should be tailored.
1.2. Research Goal and Approach
This dissertation aims to answer the following question: Can a
model be developed to
customize a SE process based on the unique needs of a project,
thereby reducing the
amount of process tailoring decisions required on a given
project?
To investigate this question, previous research was explored to
understand existing SE
process tailoring guidance factors that influence SE process
tailoring, and process tailoring
approaches. The systems engineering process tailoring model (SEPTM)
was developed as
a prototype. Its feasibility was evaluated by applying it to
several case studies, and
comparing the actual and modeled SE processes for each case.
3
1.3. Intended Contributions
The introduction of a model that reduces the amount of effort
required to tailor an
organization’s SE process to meet the unique needs of a project is
useful to SE academics
and practitioners. Applying software engineering solutions to SE is
an ongoing trend
(Boehm, 2006). From an academic standpoint, this research extends
that trend by using
process tailoring approaches in software engineering and applying
them to systems
engineering.
For SE practitioners, this research and associated model offer a
method for SE process
tailoring that is rooted in SE best practices and has been
evaluated against several real-
world case studies. This dissertation proposes a rule-based
approach to SE process
tailoring, offering a significant number of custom SE process
variations compared to a pre-
formed approach and without relying on a deep repository required
of a case-retrieval
approach. With this proposed approach, the number of tailoring
decisions could be reduced
from the total number of elements in a governing SE process to the
number of rules
established within an SE organization and applied to the
project.
1.4. Dissertation Structure
This dissertation is organized into five chapters. Chapter 1
introduces the research
topic. Chapter 2 reviews the literature relevant to this
dissertation topic. It includes research
into guidance for process tailoring offered by SE standards and
guidebooks within industry
and the federal government. Previous research into the factors that
influence process
4
tailoring is explored. Additionally, research of practical
approaches to process tailoring is
reviewed.
Chapter 3 introduces the SEPTM, which is proposed for reducing the
effort required to
tailor SE processes based on the unique attributes of a project. In
this chapter, tailoring
considerations that influence process tailoring are explained. The
process of customizing
the tailoring considerations for a particular organization and
definitions for tailoring actions
(i.e., keep standard, tailor, or delete) are explained.
Additionally, Chapter 3 describes the
process by which process tailoring rules are formed based on the
organizational tailoring
considerations and an organization’s standard SE process. Chapter 3
concludes with a
description of how the model is applied to a particular project to
generate a project-specific
SE process based on the project’s unique characteristics and
environment.
Chapter 4 defines the hypothesis and the method of using multiple
case studies to
evaluate the proposed approach to SE process tailoring. This
chapter also introduces an
analysis of 24 case studies within the Department of Homeland
Security (DHS), which is
used to evaluate the SEPTM against real-world SE process tailoring
instances. A SEPTM-
generated tailored SE process is produced for each of the 24
projects. Three analyses were
performed to investigate the validity of the SEPTM. First, each
pair of actual and modeled
tailored SE processes was compared to answer the following
question: How well does the
model match the actual tailored SE process of each project? Second,
to identify
improvements that should be explored, the actual and modeled
results relevant to each
tailoring consideration and associated process elements were
evaluated by answering the
following question: Did the model fail to match a majority of
projects for any particular
5
tailoring consideration? Third, an analysis of each process element
was performed to
understand which elements differed significantly between the actual
and modeled tailored
SE processes for the 24 projects.
Finally, Chapter 5 highlights the conclusions of this research and
recommends
additional research in SE process tailoring.
6
The research strands pertinent to this dissertation are the
following:
process tailoring guidance in systems engineering (SE) standards
and handbooks;
process reuse;
2.2. Process Tailoring Guidance in Systems Engineering Standards
and Handbooks
The following subsections highlight the tailoring guidance from
industry and
government sources.
The International Council on Systems Engineering (INCOSE) developed
the Systems
Engineering Handbook (SEH) (INCOSE, 2015) as a guide to SE
professionals and other
engineers who perform SE activities The handbook defines SE and
lifecycle stages, and
then proceeds to describe a variety of processes, methods, and
activities that systems
engineers should consider executing during projects.
This handbook also stresses the importance of tailoring the SE
process: “Thoughtful
tailoring and intelligent application of the SE processes described
in this handbook are
7
essential to achieve the proper balance between the risk of missing
project technical and
business objectives on the one hand and process paralysis on the
other hand.”
In the project planning phase, the SEH highlights the project
tailoring strategy as one of
the key process inputs (Figure 2-1).
Figure 2-1: Inputs-process-outputs diagram for project planning
(INCOSE, 2015)
The SEH offers an overarching process to tailoring. This process
begins with
identifying the circumstances that affect the tailoring strategy
and ends with the selection of
subprocesses that require tailoring. In addition, the SEH
identifies tailoring considerations,
which include factors such as system complexity and organizational
policies.
The International Organization for Standardization (ISO) and the
International
Electrotechnical Commission (IEC) developed standard 15288
(ISO/IEC, 2015) to provide
8
a common process framework for managing the lifecycle activities of
man-made systems.
The standard recommends tailoring for a project’s particular
requirements and needs, and
in Annex A, it offers guidance for tailoring of the process
contained within. Suggested
considerations for process tailoring include
“stability of, and variety in, operational environments;
risks, commercial or performance, to the concern of interested
parties;
novelty, size and complexity;
integrity issues such as safety, security, privacy, usability,
availability.”
Similar to the research described in this dissertation, ISO/IEC
15288 focuses tailoring
activities on deleting or adding selected tasks and activities
(ISO/IEC, 2015).
ISO/IEC Technical Report 24748 (ISO/IEC, 2010) is a companion
document to the
ISO/IEC 15288 standard and offers further guidance on the use of
lifecycle models for
systems. The report focuses on the adaptation of the ISO/IEC 15288
standard; Figure 2-2
(which was extracted from the report) depicts the recommended
process for adapting the
standards to meet the unique needs of a project.
9
Project environment and characteristics can include the particular
lifecycle model,
project scope, and several specialty considerations: human, health,
safety, security,
interoperability, dependability, and environmental impacts. For the
lifecycle model, the
aforementioned report identifies examples of lifecycle models for
particular domains such
as hardware, software, and services. Per the report, tailoring
decisions can include both
adding and deleting outcomes, activities, and tasks.
The Institute of Electrical and Electronics Engineers (IEEE)
published standard 1220
(IEEE, 2005) for the application and management of the SE process.
Similar to the
10
INCOSE SEH, IEEE 1220 describes the tasks that systems engineers
should complete to
deliver solutions tailored to customer needs. It recommends that
systems engineers
establish an engineering plan for the technical effort, and the
tailored application of the
standard. Per IEEE 1220, a project should “tailor the activities of
each task by adding or
deleting activities, or tailor the subprocess tasks by adding or
deleting tasks, according to
the scope of the project and in accordance with enterprise policies
and procedures” (IEEE,
2005, 37).
The National Aeronautics and Space Administration (NASA) Systems
Engineering
Handbook (NASA, 2007) was developed to provide NASA engineers with
an overarching
description of SE. In each lifecycle phase described in the
handbook, tailoring of the
standard SE process to meet project needs is stressed.
The NASA Systems Engineering Processes and Requirements document,
NPR
7123.1B (NASA, 2013), is a companion document to the NASA SE
handbook, and goes
into greater detail on the NASA SE process and what SE
practitioners must execute.
Regarding process tailoring, NPR 7123.1B identifies environmental
factors such as project
scope, system complexity, number of interfaces, serviceability, and
safety as important
tailoring considerations. The NPR uses a SE management plan for
defining a project’s SE
strategy and seeking relief (tailoring) from the SE requirements
imposed in the NASA SE
policy. A NASA SEMP also includes a compliance matrix that
identifies SE policy
requirements from which a project seeks relief.
11
The Defense Acquisition University’s Systems Engineering
Fundamentals Guide
(DAU, 2001) provides a basic explanation of the technical aspects
of systems development
and acquisition. Consistent with other standards and guides, this
guide stresses the
importance of tailoring the SE process for the different needs of
each type of project. Needs
can vary due to “system size and complexity, level of system
definition detail, scenarios
and missions, constraints and requirements, technology base, major
risk factors, and
organizational best practices and strengths” (DAU, 2001, 9).
The Naval Systems Engineering Guide (Naval Systems Engineering
Group, 2004) was
developed to assist program teams in implementing SE across the
program’s acquisition
lifecycle. The process implementation strategy described in the
guide includes a specific
task for program teams to develop a tailored version of the guide.
Each subprocess detailed
in the guide includes a caveat that it should be tailored to for
the system’s application,
complexity, or other reason.
Army Acquisition Procedures (Army, 2014) for research, development,
and acquisition
highlight encouragement from Department of Defense and Army
leadership to tailor
acquisition processes. From requirements development to production
engineering and
sustainment, from risk management to data requirements, the Army
Acquisition Procedures
authorize tailoring to meet the needs of a program:
This tailoring of required oversight and review documentation
to
the needs of each specific program is a key element of the
acquisition streamlining process. Continued emphasis will be
placed on this effort to reduce the amount of documentation
that
must be prepared to support program reviews.” (Army, 2014,
179)
12
The Army Systems Engineering Policy (Army, 2005) encourages
tailoring the contents
of the systems engineering plan, and includes tailoring
considerations such as program size,
commercial off the shelf (COTS) based programs, and for programs in
sustainment.
The Department of Homeland Security (DHS) established the systems
engineering
lifecycle (SELC) as a framework that captures a common description
of SE processes and
activities for the DHS enterprise (DHS, 2010, 3). The SELC was
designed to support
tailoring based on project characteristics such as size, scope,
complexity, risk, and security
categorization. The SELC requires each project to create an SELC
tailoring plan that
details the approved adapted SE process a project intends to
execute. For additional
emphasis, the DHS SELC notes that “tailoring is the cornerstone of
any lifecycle process”
(DHS 2010, 4).
2.2.3. Capability Models
The Electronic Industries Alliance (EIA) EIA/IS-731.1 is a systems
engineering
capability model that can be used to “develop, improve, and assess
systems engineering
capability” (EIA, 2002, 2). It offers focus areas, themes, specific
practices, generic
practices, and generic attributes that span the SE discipline. In
particular, generic practices
are used to increase the ability to perform the specific practices.
Six levels of capability are
defined to assess an organization’s level of performance in each
particular generic and
specific practice:
Capability Level 2 – Managed
Capability Level 3 – Defined
Capability Level 4 – Measured
Capability Level 5 – Optimizing
A fundamental progression from levels zero through two to levels
three through five is
an organization’s ability to have tailoring guidance that
accompanies a standard SE process
and tailored applications of the standard process for each
program.
An expressed limitation of the systems engineering capability model
is that it is “not
intended to specify the details of ‘how to’ implement process
activities.” As such, it does
not provide methods or tools to apply to the process.
The Capability Maturity Model Integration for Development
(CMMI-DEV) assists
development organization managers and engineers regarding the
application of CMMI best
practices (CMMI-DEV, 2010). It defines the levels of capability for
process improvement:
Capability Level 0 – Incomplete
Capability Level 1 – Performed
Capability Level 2 – Managed
Capability Level 3 – Defined
Capability Level 3 is achieved primarily when processes are
appropriately tailored:
A critical distinction between capability levels 2 and 3 is the
scope
of standards, process descriptions, and procedures. At
capability
level 2, the standards, process descriptions, and procedures can
be
quite different in each specific instance of the process (e.g., on
a
particular project). At capability level 3, the standards,
process
descriptions, and procedures for a project are tailored from
the
organization’s set of standard processes to suit a particular
project
or organizational unit and therefore are more consistent, except
for
14
2010, 25)
2.3. Process Reuse
Process tailoring can be considered a form of standard process
reuse (Yoon et al.,
2001); however, tailoring involves process application to a unique
project environment, as
opposed to the flexibility of the standard process itself.
Particularly in software
engineering, patterns have been explored as a viable option for
process reuse; in fact, the
concept has been extended to a method for developing pattern forms
in systems
architecture (Cloutier and Verma, 2007).
In addition, Hagge and Lappe (2005) emphasize the need for
capturing knowledge that
can be reused in future projects. They propose four introductory
patterns for requirements
engineering and discuss a method for sharing observations across
projects (Hagge and
Lappe, 2005). Rising (1999) also highlights the use of patterns as
a way to transfer
knowledge from senior engineers to novices.
Fortune and Valerdi (2013) propose a framework for reusing SE
products within an
organization (Figure 2-3) and highlight key considerations for
process reuse. Similar to
process tailoring, process reuse begins with understanding the
factors that influence the
decision to reuse an existing subprocess or element.
15
Figure 2-3: Systems engineering reuse framework (Fortune and
Valerdi, 2013)
Among the key considerations for process reuse, Fortune and Valerdi
(2013) argue that
a robust knowledge management system is critical to the success of
process reuse. The
authors also note the reuse of a product at too high a level may
not apply to the specific
environment of the new project. Conversely, reusing products at too
low a level can cause
tailoring of enough significance that the effort saved by process
reuse may be lost (Fortune
and Valerdi, 2013). This argues for additional guidance on process
tailoring.
16
2.4. Process Tailoring Methods in Software Literature
Extending knowledge from the software process domain to the SE
process domain has
been ongoing for quite some time (Boehm, 2006). Research in
software process tailoring is
well-documented and can be extended to SE. Henderson-Sellers et al.
(2014) provide an
extensive literature review of the application of method
engineering to specific situations in
software development. Their review spans the various approaches to
method tailoring and
case studies in the application of method tailoring.
Kuhrmann et al. (2014) also conducted a mapping study on the
feasibility of method
engineering, which pertains to the process of constructing methods
for software
development. The authors highlight 83 articles contributing to the
research of software
process tailoring, and they categorize previous research into the
following categories
(research type facets initially introduced by Wieringa et al.,
2005):
Experience
Validation
Evaluation
Philosophical
Opinion
Solution proposal
The variety of research and tailoring approach proposals highlights
the lack of a
common lexicon in method engineering (Kuhrmann et al., 2014). Of
the 83 papers, less
than 20% are in the experience, validation, or evaluation
categories. Other papers propose
17
solutions or remain at the theoretical level. This in part led to
the conclusion that “strong
empirical evidence of the feasibility of ME” is still lacking
(Kuhrmann et al., 2014, 1066).
In the few cases where process tailoring support at the project
level has been researched
beyond the framework level, various methods have been applied. This
dissertation
separates the previous methods for process tailoring into three
types: case retrieval, pre-
formed, and rule-based.
2.4.1. Case Retrieval Method for Process Tailoring
The case retrieval-based method formulates a tailored software
process based on
retrieving a tailored process from a library of historical projects
(Ahn et al., 2003; Park and
Bae, 2013). Figure 2-4 depicts the process that Ahn et al. (2003)
used for case retrieval.
They performed retrieval by comparing the values of domain factors
of a new project to
those of past projects in the library. Manual tailoring follows the
case-based reasoning
technique. Ahn et al. (2003) performed case-based reasoning by
calculating a similarity
function, the degree to which a case in the process library
exhibits project characteristics
(domain factors) that are comparable to those of the new
project.
18
Figure 2-4: Overall procedure of process customization (Ahn et al.,
2003, 296)
The case retrieval approach has two primary challenges. First, its
accuracy is limited to
the extensiveness of the library of previous approaches, which
requires a mature
organization. Second, the organization must have an established
knowledge management
system that not only archives previous projects, but also tags the
projects with the values
and domain factors required for retrieval.
2.4.2. Pre-Formed Method for Process Tailoring
In the pre-formed approach to process tailoring, an organization
develops a standard set
of tailoring forms that apply to particular categories of projects
(Plogert, 1996). The
approach of Plogert (1996) for applying tailoring forms consists of
four steps:
19
1. Determination of the project class: Three classes of projects
are proposed, namely
administrative information technology, technical or scientific
information
technology, and commercial off the shelf (COTS).
2. Determination of the project size: Three project sizes are
proposed.
3. Selection of the corresponding form, based on the first two
steps.
4. Checking implementing conditions: Reassesses form selection
based on additional
considerations for criticality, complexity, or other predefined
conditions.
The DHS also supports the pre-formed approach to tailoring its
SELC. The
organization has identified seven pre-formed “SELC tailored
paths”:
1. System or product development
2. Information technology infrastructure
6. Information technology services
7. Facilities or construction (Tuttle, 2014)
The challenge with this approach is that it only includes the first
level of tailoring, such
as hardware or software, and does not address the significant
variability that can exist from
project to project within an organization. In addition, this
approach does not easily allow
for growth; to modify the approach based on lessons learned would
require revisiting the
pre-tailored processes altogether.
2.4.3. Rule-Based Method for Process Tailoring
Hausen (1998) employed a rule-based approach to software process
tailoring. This
approach identifies the values of factors that address the
rationale for tailoring based on
specific project characteristics. Each rule is created such that
for a specific value of a
domain factor that exists for a new project, certain process
elements are tailored (Kang et
al., 2008). Figure 2-5 depicts the tailoring rule structure used by
Kang et al. (2008).
Tailoring actions include adding, removing, and replacing a process
element. Figure 2-6
provides an example of a tailoring rule.
Figure 2-5: Tailoring rule structure (Kang et al., 2008)
21
Figure 2-6: Example of a tailoring rule (Kang et al., 2008)
2.5. Process Tailoring Case Studies
2.5.1. Framework Application Case Studies
Xu and Ramesh (2008) used two case studies to investigate how two
project teams
performed process tailoring and what was the rationale for the
decisions made. Their
research showed that the teams performed process tailoring based on
the challenges they
faced, on project goals, and on environmental factors. Based on
their case study analysis,
Xu and Ramesh (2008) proposed the model below (Figure 2-7).
22
Figure 2-7: Steps for process tailoring (Xu and Ramesh, 2008)
De Cesare et al. (2008) conducted a method tailoring case study at
S-Service, a
medium-sized software development company. They define method
tailoring as follows:
“methodologies are normally adapted to meet specific contextual
characteristics” (de
Cesare et al., 2008, 157). The authors developed a method tailoring
framework to elicit
tailoring methods from the company’s engineering team (Figure 2-8).
S-Service tailored
the rational unified process for the studied case. Because the
company’s objective was to
adapt the rational unified process for most of its projects, the
tailoring activities focused
on what process elements were not necessary and therefore could be
deleted. For project-
level tailoring, de Cesare et al. (2008) found that the methods
(i.e., process elements)
were selected from a repository of previously used processes.
23
Figure 2-8: Method tailoring elicitation framework (de Cesare et
al., 2008, 161)
The framework presented was rooted in previous research and applied
to document a
real-world case. As the research of de Cesare et al. (2008) is
limited to a single case study,
the authors recommend additional research be conducted within
organizations with
“different contextual characteristics” (de Cesare et al., 2008,
166).
2.5.2. Model Application Case Studies
Park and Bae (2013) propose an approach to software process
tailoring using process
slicing and case-based reasoning. With process slicing, the authors
identify
interdependencies within the process and across subprocesses. This
allows for an analysis
of which process elements should be tailored. Using case-based
reasoning, the most similar
24
past tailoring case is retrieved to reduce false positive
dependencies generated by process
slicing. Retrieval of the most similar case is performed using
structural similarity in the
same way Ahn et al. (2003) did. Figure 2-9 depicts this
process.
Figure 2-9: Process slicing with case-based reasoning (Park and
Bae, 2013)
Their case study was conducted in support of a government
organization, which has
several software development projects. The case consisted of a
single organization and 23
projects within the organization, which should adhere to the
organization’s standard
process or be tailored from the standard process.
The standard process contained 20 subprocesses, each with several
artifacts and
activities. Park and Bae (2013) used the following tailoring
considerations in their case
study:
Lifecycle
25
Schedule risk
Cost risk
A preliminary process slicing was generated using the
organization’s standard process
interdependencies. Each project’s characteristics were used to
retrieve the most similar case
from the organization’s repository. The intersection of the similar
case and the preliminary
process slice generated the final tailored process. For each
project, the authors compared
actual and modeled results (tailoring cases). The results showed
that the use of case-based
reasoning increases the accuracy of the model when compared to
results with just the
process slicing method.
Kang et al. (2008) performed a multiple case study to evaluate
their proposed case
retrieval approach to software process tailoring: determining the
applicable tailoring rules,
performing a similarity calculation, and retrieving the most
similar case from the repository
of historical processes (Figure 2-10).
26
Figure 2-10: Overview of case retrieval method (Kang et al.,
2008)
For their case studies, Kang et al. (2008) designed 30 projects for
the military domain
and used the standard process of the Korean armed forces. After
modeling the standard
27
process, Kang et al. (2008) tailored the standard process for the
30 cases. To validate their
model, they followed these steps:
1. Choose a case from the repository
2. Retrieve the most similar case using the model
3. Retrieve the most similar case by comparing the case to the
other 29 cases
4. Compare the retrieved processes
5. Repeat the first four steps for all other cases
The results of their research showed that their approach retrieves
the most similar case
in 70% of the cases. Kang et al. (2008) note that their research is
limited by using artificial
cases, citing data availability as the primary inhibitor.
2.6. Summary
The common theme spanning the aforementioned industry and
government standards
and guidelines is that tailoring the standard process is critical
to effective SE planning and
execution. Moreover, each standard and guideline identifies various
organizational and
environmental factors that influence the adaptation of a standard
SE process to meet the
individual requirements of a project.
Another common theme is that none of the standards and guidelines
offer a practical
method for SE process tailoring. At most, some guidelines offer a
general framework that
starts with identifying what is important to process tailoring and
ends with a custom SE
process for a project, with little detail in between.
28
Significant research has been done on tailoring software processes
for projects, but the
proposed approaches have not been applied to SE. Software process
tailoring case studies
that propose frameworks serve as a starting point for process
tailoring methods; the
limitation of this research is the lack of decision support
assistance when an SE practitioner
must tailor organizational processes at the project level.
Additional case studies that offer
both a high-level adaptation process and decision support are
limited to the software
research field. Even in this field, researchers identify case study
and experiential research
as a shortfall in the literature.
This dissertation presents a multi-staged approach to SE process
tailoring of adapting
industry-identified tailoring considerations to an organization and
providing a decision
support tool to assist project systems engineers with customizing
an SE process for their
projects. Therefore, this dissertation seeks to bridge the gap
between the recognition of the
importance of SE process tailoring and project-level SE process
tailoring by leveraging
process tailoring research in the software field.
29
3.1. Introduction
This chapter describes the systems engineering process tailoring
model (SEPTM) that
was developed as a tool for tailoring systems engineering (SE)
processes to meet the
unique needs of a project. Figure 3-1 presents an overview of the
proposed SEPTM.
At a high level, the model was constructed based on an analysis of
the generic SE
process described in the INCOSE Systems Engineering Handbook
(INCOSE, 2015) that
influences process tailoring. This analysis forms the basis of an
initial set of tailoring
considerations, from which a subset that is applicable to a
particular organization was
selected. A rule set is then developed that links the tailoring
considerations and the
organizational SE process. A systems engineer can then apply the
rule set to the unique
attributes of a project to generate a customized SE process based
on the conditions of a
particular project. Each of these activities is further described
in the next subsections.
Figure 3-1: Overview of the systems engineering process tailoring
model (SEPTM)
30
3.2. Identify the Initial Set of Tailoring Considerations
An important aspect of the model is determining the tailoring
considerations used to
identify the critical aspects of a project. The tailoring
considerations will broadly inform
what SE activities are needed for a given project.
The development of the initial list of tailoring considerations is
based on an analysis of
the general SE process described in the INCOSE System Engineering
Handbook
(INCOSE, 2015) (henceforth called INCOSE handbook) and its
discussion of
considerations and enablers for process tailoring. As discussed in
the Literature Review,
several handbooks and guidebooks stress the importance of SE
process tailoring and
identify the considerations for process tailoring. The INCOSE
handbook was selected as
one industry standard from which to determine the model’s initial
tailoring considerations.
In Chapter 5, an investigation of tailoring considerations and
their impact on SE process
tailoring decisions is identified as an area of future
research.
The INCOSE handbook states the appropriate SE approach will vary
based on the
unique needs of a project, and it also describes various lifecycle
approaches: waterfall,
incremental, and agile (INCOSE, 2015). In the waterfall approach,
SE stages are performed
serially; in the incremental approach, SE stages are performed
serially multiple times; and
in the agile approach, the project goes through short development
cycles with frequent
product releases (Aoyama, 1998). As part of the agile development
cycles, the SE stages of
requirements definition, design, development, and testing are also
performed iteratively.
Moreover, the associated reviews and documentation are performed
and updated with each
release (Cao and Ramesh, 2008). Each of these approaches varies in
terms of stage and
31
review frequency, as well as the amount of documentation required.
As a result, a project’s
lifecycle approach must be a critical tailoring
consideration.
The INCOSE handbook also describes crosscutting technical methods
and includes a
section on specialty engineering analyses that must be considered
by a project’s systems
engineer; however, not all of the analyses will apply to every
project (INCOSE, 2015). For
example, interoperability should be considered for complex projects
and those that
interface with other projects. An organization may have specific
artifacts that are required
to address interoperability for certain projects or allow for
deleting interoperability artifacts
based on the project type.
The INCOSE handbook’s tailoring section includes a discussion of SE
process tailoring
and identifies considerations and enablers for tailoring: project
scope, risk tolerance,
complexity and precedence of the system, and
organizational/enterprise policies and
infrastructure (INCOSE, 2015). While the handbook identifies risk
tolerance as a factor in
process tailoring, it does advise of the importance of placing
tailoring emphasis on the
system and the project objectives, or project scope.
An analysis of the general SE process and tailoring discussion in
the aforementioned
handbook resulted in the following list of tailoring
considerations:
1. Lifecycle approach
2. Project scope
4. Organizational/enterprise policies and infrastructure
5. Modeling, simulation, and prototyping
32
8. Cost effectiveness analysis
9. Electromagnetic compatibility analysis
10. Environmental impact analysis
15. Safety and health hazard analysis
16. Sustainment engineering analysis
17. Training needs analysis
19. Value engineering
Of the 19 tailoring considerations, the first 4 (lifecycle
approach, project scope,
complexity and precedence of the system, and
organizational/enterprise policies and
infrastructure) apply to all organizations (INCOSE, 2015). The
remaining 15 vary in
applicability, depending on the types of projects an organization
executes. This requires an
organization to determine the appropriate subset of tailoring
considerations for its project
portfolio.
33
3.3. Refine the Set of Tailoring Considerations for an
Organization
In the second analytical step, the overarching list of tailoring
considerations is reduced
to one that includes only the tailoring considerations relevant for
that organization. This
step is performed by comparing the list of tailoring considerations
to an organization’s
policies and standard SE process (Figure 3-2). An organization’s SE
process is one of
many organizational processes, which CMMI defines as
a collection of definitions of the processes that guide activities
in
an organization. These process descriptions cover the
fundamental
process elements (and their relationships to each other such
as
ordering and interfaces) that should be incorporated into the
defined processes that are implemented in projects, work
groups,
and work across the organization. A standard process enables
consistent development and maintenance activities across the
organization and is essential for long-term stability and
improvement. (CMMI, 2010)
34
The organizational SE process must be analyzed to confirm which of
the enterprise
tailoring considerations are incorporated into the standard
process. For example, is mass
properties engineering analysis discussed in the SE process, and
are there process stages,
reviews, or artifacts tied to mass properties engineering analysis
in the standard SE
process? Additionally, other organizational policies relating to
systems development or
acquisition must be examined; for example, does the organization
have security
requirements for technology?
Each organizational tailoring consideration is then converted to a
question with
possible “values” as answers. The tailoring considerations can have
values that are either
yes or no (e.g., environmental impacts need to be considered or
not) or multiple (e.g.,
lifecycle approach may be waterfall, incremental, or agile).
Forming the questions and
values becomes a critical input to establishing the rule set for
the SEPTM.
3.4. Definitions
Before a tailoring rule set can be developed, some tailoring
operations must be defined.
For the purposes of this research, the following definitions
apply:
“Standard” is used when a stage, review, or artifact is to be
developed consistent
with the scope of the activity defined in the organizational SE
policy.
“Tailored” is used when a stage, review, or artifact is modified
from the scope in
the organizational SE policy. This could include combining two or
more stages,
reviews, or artifacts into one, and also repeating stages or
reviews multiple times
within a project. For example, the process tailoring model
recommends that the
35
integration readiness review occur with each release in a project
that uses agile
development methodology.
“Deleted” is used when the stage, review, or artifact is not needed
and is removed
from the project’s tailored SE process altogether. An example of
this is the removal
of a system design document for a commercial off the shelf (COTS)
acquisition.
3.5. Establishing the Rule Set
3.5.1. Base Rules
Once the tailoring considerations have been selected for an
organization, and tailoring
conditions have been defined, a rule set must be generated to link
the tailoring
considerations and associated values to the elements of the
organization’s standard SE
process and its process elements. This research applies the
CMMI-DEV definition of
process element, which is
the fundamental unit of a process. A process can be defined
in
terms of subprocesses or process elements. A subprocess is a
process element when it is not further decomposed into
subprocesses or process elements. Each process element covers
a
closely related set of activities (e.g., estimating element,
peer
review element). Process elements can be portrayed using
templates to be completed, abstractions to be refined, or
descriptions to be modified or used. A process element can be
an
activity or task. (CMMI, 2010)
Figure 3-3 depicts the matrix used to establish the rule set.
36
Figure 3-3: Establishing the rule set
In area 1 of the table, all elements (1 to M) of the standard SE
process are listed
vertically. For example, the DHS SE process has 107 process
elements (“M”). In area 2,
each tailoring consideration (1 to N) and associated values (1 to
O) are listed horizontally.
“N” represents the number of tailoring considerations identified
for the organization, as
described in section 2.2, and the values represent potential
attributes of a project within the
organization.
As an example, if tailoring consideration 1 is interoperability
analysis, the potential
values would be “yes” and “no” given that a system may be
standalone or connected. For
each intersection of a process element and a tailoring
consideration-value combination in
area 3, a tailoring rule is established. This requires two levels
of decisions. First, a
determination is made as to whether a tailoring consideration is
relevant or not for a
particular process element; and second, if so, a determination is
made for each value of the
tailoring consideration whether the process element should be kept
standard, tailored, or
Tailoring Considerations → TC 1 TC 2 … TC N
Values →
2
37
deleted. Thus, each relevant intersection in area 3 will contain
either “S” for standard, “T”
for tailored, or “D” for deleted.
In summary, the resulting rule set is a list of tailoring
considerations for which each
value drives a tailoring decision (standard, tailored, or deleted)
for every relevant process
element within the organizational SE process.
3.5.2. Prioritization Rules
Because the SEPTM executes multiple rules at the same time, it is
essential to resolve
any conflicts in situations where two or more rules are linked to
the same process element.
For example, suppose the development type and development
methodology tailoring
considerations trace to the same process element, and the
associated questions for those
tailoring considerations are the following:
Does the project intend to develop a solution or procure a COTS
solution to meet
project requirements?
Will the project be using the waterfall, incremental, or agile
development
methodology?
In this example, a prioritization rule that de-conflicts the
recommended tailoring
decision is necessary, such as,
if the project is a COTS procurement, the process element should
not be executed;
else, if the project is a development effort and employing a
waterfall or an
incremental methodology, the process element should be executed
(standard or
tailored);
38
else, if the project employs agile development, the process element
should be
tailored.
3.6. Apply Model to a New Project and Validate
Once the model has been developed for an organization, project
systems engineers can
use it for tailoring SE processes to meet the needs of a specific
project. To do so, a systems
engineer must have a solid understanding of the acquisition
strategy and the project
environment. This is true whether applying a rule-based
methodology, as is the case here,
or tailoring an SE process ad hoc. To exercise the model, a systems
engineer must align the
characteristics of the project to a value associated with each
tailoring consideration. Once
this is complete, the model outputs the tailored SE process,
identifying which of the full set
of the organization’s SE process elements should be executed as
standard, which should be
tailored, and which can be deleted.
After the model has been applied to a real-world situation, lessons
learned must be
incorporated into the model by reassessing the organization’s list
of tailoring considerations
and the organization’s rule set. Moreover, as SE research is
constantly refining best
practices and industry standards are updated, the organization must
also revisit the initial
tailoring consideration set to assess changes from industry.
39
Chapter 4: Case Studies
A multiple case study approach was selected to investigate whether
the SEPTM can
reduce the amount of manual process tailoring of the SE lifecycle
on actual projects. The
case study approach was adapted from Kitchenham’s (1995) seven
steps for performing
case studies:
3. Identify the method of comparison
4. Consider the effects of confounding factors
5. Plan the case study
6. Conduct the case study
7. Analyze and report the results
Each of the above steps is described in detail in the following
subsections.
4.1. Definition of Hypothesis
Consistent with the goal of this dissertation, the hypothesis for
the case study is as
follows:
Using the SEPTM to tailor a project’s systems engineering process
can reduce the
amount of manual tailoring required.
This hypothesis will be considered met if the conditions below are
achieved.
40
1. The number of decisions made to tailor a project’s process are
reduced. Henderson-
Sellers et al. (2014) note that determining the best project
methodology requires
specific tailoring to a project’s unique situation. Measuring the
benefit of such an
activity can be based on the amount of effort required to tailor
standard processes
(Xu and Ramesh, 2007). For the purposes of this analysis, this
research extends that
definition to the number of manual decisions required to produce a
project-specific
systems engineering (SE) process. By design, the SEPTM fixes the
number of
decisions on the number of tailoring considerations established for
an organization.
If an organization applies all of the initial 19 tailoring
considerations, the number of
manual decisions will be 19. This of course makes the second
condition all the
more important.
2. The SEPTM approach matches the ad hoc approach to an acceptable
level.
Consistent with validation studies of software engineering
practices, a 75%
acceptable matching level is used (Daneva and Ahituv, 2010;
Krishnan et al., 1999;
Ramasubbu et al., 2005). Thus, for the model to be considered valid
for a given
project, at least 75% of the process elements must match. For the
model to be
considered valid overall, at least 75% of the projects must meet
that threshold.
Two additional analyses were performed: First, an analysis of each
tailoring
consideration was conducted to understand if the model failed to
match a majority of
projects for any particular tailoring consideration. Second, an
analysis of each process
element was also performed to understand which elements had the
most significant
differences between the projects’ plans and the model’s
outputs.
41
4.2. Pilot Project Selections
For this case study, both an organization with an acquisition
portfolio and projects
within that portfolio were selected.
First, an organization was selected with
a diverse enough acquisition portfolio, to make tailoring of the
standard process an
essential task in the SE planning of a project;
a standard SE process, required as a starting point for projects
within the
organization’s portfolio, but that promoted process
tailoring;
a standard way for projects to report planned tailoring approaches
to the standard
SE process; this allows for a comparison of the model against
tailoring approaches
used on real-world projects.
Second, projects were selected within the organization’s portfolio
based on data availability
and variability across the tailoring considerations.
4.2.1. Selection of the Organization
The organization selected for the case study is the Department of
Homeland Security
(DHS), an organization with a large acquisition portfolio, given
its five core missions:
1. “Prevent terrorism and enhance security,
2. Secure and manage our borders,
3. Enforce and administer our immigration laws,
4. Safeguard and secure cyberspace, and
5. Ensure resilience to disasters.” (DHS Mission, 2016)
42
Each of these missions requires the acquisition of technology and
infrastructure to
support the organization’s frontline officers and agents.
Therefore, the DHS acquisition
portfolio is diverse, ranging from border security surveillance
technologies and
infrastructure to information technology systems for immigration
and cybersecurity.
The DHS published a systems engineering lifecycle (SELC), defined
as “a guiding
framework that provides a vocabulary, order, and description of the
activities enabling
efficient and effective delivery of capability to users” (DHS,
2010). Figure 4-1 depicts the
DHS SELC.
Figure 4-1: DHS systems engineering lifecycle process (DHS,
2010)
The DHS standard SELC process identifies 9 SE stages, 11 technical
reviews, and 87
artifacts to be executed through the life of a project, for a total
of 107 SE process elements.
Figure 4-1 also provides a note that mandates an SELC tailoring
plan be developed for
each project within the DHS acquisition portfolio. This offered a
standard format of data
with which to compare the model’s results to the actual SE process
tailoring approaches.
43
4.2.2. Selection of the Projects
The second key element required for the case study is a set of
tailored SE processes
implemented on projects within the organization. Twenty-four DHS
projects were selected
for analysis based on data availability and variability across the
tailoring considerations
(Table 4-1). For example, some projects executed full research and
development, while
others were COTS acquisitions; some used a waterfall development
approach, while others
used an incremental or agile. Quantitative data regarding tailoring
outcomes were gathered
and analyzed from each of the 24 projects and compared the data to
the model, as described
in Section 3.3.
Case Study
Technology Demonstration Planned Planned/Not Planned 7/17
Environmental Impact Impact/No Impact 7/17
Development Methodology Waterfall/Incremental/Agile 11/11/2
Development Type Development/COTS-NDI 19/6
4.3. Method of Comparison
Given that the projects vary due to their unique characteristics,
the projects were not
compared to each other or to a single output of the SEPTM. Rather,
the 24 projects’
characteristics were each input into the model, generating 24
SEPTM-based tailored SE
processes. As a result, each project has a tailored SE process,
developed by the project
team, and a tailored SE process, developed by the SEPTM. Each
model-data combination
is then analyzed.
4.4. Effects of Confounding Factors
Internal validity of the case study can be affected by confounding
factors. Kitchenham
(1995) identifies the most significant confounding factors as
“likely to be:
Learning how to use a method or tool as you try to assess its
benefits.
Using staff who are either very enthusiastic or very skeptical
about the method
or tool.
Comparing different application types.”
The SEPTM, once developed for an organization, offers a set of
questions whose
answers are inputs to the model based on a project’s
characteristics. Unlike using a
complicated tool to automate software development, for example,
using the SEPTM is as
simple as completing a questionnaire regarding a project.
Therefore, the effect of learning
how to use the SEPTM is avoided, because the inputs to the model
are the characteristics of
the project and, thus, available to the project team. This also
eliminates the concern of
enthusiasm or skepticism in employing the tool. To reduce the
effect of comparing
different application types, multiple projects are used, as
Kitchenham (1995) suggests.
4.5. Case Study Planning
1. Define the hypothesis and the associated metrics.
2. Collect case study data.
3. Analyze and select organizational tailoring
considerations.
4. Generate rule set for the SEPTM.
5. Generate tailored SE approaches for each project using
SEPTM.
46
4.6. Case Study Conduct
As detailed in Section 3.2, initial tailoring considerations were
identified in Step 1 of
the SEPTM. Based on an analysis of the INCOSE handbook (INCOSE,
2015), 19 initial
tailoring considerations were identified to serve as the starting
point for determining the
tailoring considerations suitable for a particular
organization.
In the conduct of this case study, analysis began with selecting
organizational tailoring
considerations for the DHS. Based on the DHS tailoring
considerations, a rule set was
established that links the DHS tailoring considerations to the DHS
SELC process. After
generating a SEPTM-based tailored SE process for each of the 24 DHS
projects, each pair
of model-based and original SE processes was compared to evaluate
whether the model
could reduce the number of tailoring decisions while retaining the
original project SE
processes at an acceptable level. The subsections below describe
the case study in detail.
4.6.1. Organizational Tailoring Consideration Selection
As described in Section 3.3, the first step in applying the
tailoring model to an
organization is determining the subset of the 19 overarching
tailoring considerations that
are applicable to the organization. As stated previously, the first
four tailoring
considerations apply to all organizations and were adapted for the
DHS tailoring
consideration list:
4. Organizational/enterprise policies and infrastructure
Analysis of the DHS organizational/enterprise policies yielded
additional
considerations pertaining to security, privacy, and intelligence
(DHS, 2015).
Additional analysis of the DHS SELC process was performed to
determine which of
the remaining 15 tailoring considerations are applicable. Of these,
five were applicable to
all projects per the DHS SE process, six were not discussed in the
DHS SE process guide,
and five may apply depending on the needs of the specific project.
In support of the focus
of this case study on process tailoring, the four tailoring
considerations that may be
applicable were included as DHS tailoring considerations in the
model. As a result, the 10
organizational tailoring considerations identified and applied for
this research are as
follows:
3. Intelligence impact
4. Interoperability impact
7. Environmental impact
9. Development type (complexity and precedence of the system)
48
4.6.2. Rule Set
4.6.2.1. Linking Tailoring Considerations to the Standard
Process
With the 10 DHS tailoring considerations in place, the rule set for
the DHS was
established by linking the tailoring considerations to the standard
DHS SE process. An
initial determination of relevance was made for each combination of
tailoring consideration
and process element. The process elements that do not link to any
tailoring consideration
were considered to be part of the default process.
Table 4-2 through Table 4-11 provide the relevant process elements
for each tailoring
consideration.
49
DHS Standard SE Process Elements
FIPS 199 Security Categorization
Security Requirements Traceability Matrix
System Security Plan
Disaster Recovery Plan
Security Risk Assessment
Contingency Plan
Security Incident Reports
Certification and Accreditation Updates (every 3 years or when
major change is made)
50
DHS Standard SE Process Elements
Privacy Threshold Analysis
Privacy Impact Assessment
Table 4-4: Relevant process elements for intelligence impact
DHS Standard SE Process Elements
Intelligence Support Plan
DHS Standard SE Process Elements
HLS EA Program Alignment Decision Request
Service Reuse Plan
Data Management Plan
Service Level Agreements
Data Architecture Document
Technology Insertion Package
Data Insertion Package
Service Insertion Package
51
Section 508 National Security Exception Determination
Section 508 Electronic and Information Technology Accessibility
Plan
Table 4-7: Relevant process elements for technology
demonstration
DHS Standard SE Process Elements
Technology Demonstrator Results Report
DHS Standard SE Process Elements
Environmental Impact Assessment
DHS Standard SE Process Elements
Requirements Definition Stage
Functional Requirements Document
Requirements Traceability Matrix
System Requirements Document
Logical Design Document
Data Architecture Document
System Design Document
Table 4-10: Relevant process elements for development type
impact
DHS Standard SE Process Elements
Design Stage
Development Stage
Table 4-11: Relevant process elements for project size
DHS Standard SE Process Elements
53
Concept of Operations
Operational Requirements Document
Configuration Management Plan
Risk Management Plan
Quality Assurance Plan
Functional Requirements Document
Requirements Traceability Matrix
Logical Design Document
Data Architecture Document
System Design Document
Site Prep Plan
4.6.2.2. Rules
Each tailoring consideration was then developed into a question
that, when answered,
drives tailoring for the project. The questions and associated
rules used in this research are
below.
Tailoring consideration: security impact
Question: Will the solution acquired contain any information or
data requiring special
protections?
Rules:
If yes, process elements linked to this tailoring consideration
should be executed
(standard or tailored). Note that standard or tailored is
acceptable to promote
additional tailoring beyond the model.
If no, process elements linked to this tailoring consideration
should not be
executed (deleted).
Tailoring consideration: privacy impact
Question: Will the solution acquired relate in any way to an
individual (for example,
social security number, communication traffic, image or video,
radioactivity level)?
Possible values: yes, no
Rules:
55
If yes, process elements linked to this tailoring consideration
should be executed
(standard or tailored). Note that standard or tailored is
acceptable to promote
additional tailoring beyond the model.
If no, process elements linked to this tailoring consideration
should not be
executed (deleted).
Question: Will the solution acquired contain any intelligence data
or information?
Possible values: yes, no
Rules:
If yes, process elements linked to this tailoring consideration
should be executed
(standard or tailored). Note that standard or tailored is
acceptable to promote
additional tailoring beyond the model.
If no, process elements linked to this tailoring consideration
should not be
executed (deleted).
Question: Will the solution acquired contain any equipment or
interconnected system
or subsystem of equipment or software?
Possible values: yes, no
Rules:
56
If yes, process elements linked to this tailoring consideration
should be executed
(standard or tailored). Note that standard or tailored is
acceptable to promote
additional tailoring beyond the model.
If no, process elements linked to this tailoring consideration
should not be
executed (deleted).
Tailoring consideration: accessibility impact
Questions: (1) Will the solution acquired contain any information
or data that is
accessible to a user of the solution? (2) Will the solution qualify
for a Section 508
exception?
Rules:
If yes to both questions, the accessibility plan should be deleted
and the
exception determination request should be executed (standard or
tailored). Note
that standard or tailored is acceptable to promote additional
tailoring beyond the
model.
If yes to the first question and no to the second, the
accessibility plan should be
executed (standard or tailored) and the exception determination
request should
not be executed (deleted).
If no, process elements linked to this tailoring consideration
should not be
executed (deleted).
Tailoring consideration: technology demonstration
Question: Does the project strategy include any plan for pilots or
demonstrations?
57
Rules:
If yes, process elements linked to this tailoring consideration
should be executed
(standard or tailored). Note that standard or tailored is
acceptable to promote
additional tailoring beyond the model.
If no, process elements linked to this tailoring consideration
should not be
executed (deleted).
Tailoring consideration: environmental impact
Question: Could the solution acquired have any potential impact on
the environment
(for example, radiating equipment, construction)?
Possible values: yes, no
Rules:
If yes, process elements linked to this tailoring consideration
should be executed
(standard or tailored). Note that standard or tailored is
acceptable to promote
additional tailoring beyond the model.
If no, process elements linked to this tailoring consideration
should not be
executed (deleted).
Tailoring consideration: development methodology
Question: Will the project be using the waterfall, incremental, or
agile development
methodology?
Rules:
58
If waterfall or incremental, the process elements linked to this
tailoring
consideration should be executed (standard or tailored). Note that
standard or
tailored is acceptable to promote additional tailoring beyond the
model.
Additionally, an assumption was made that the tailoring approach
for a project
would be updated with each increment of a project. As a
result,incremental is
treated the same as waterfall for this analysis.
If agile, process elements linked to this tailoring consideration
should be
tailored.
Tailoring consideration: development type
Question: Does the project intend to develop a solution or procure
a COTS solution to
meet project requirements?
Rules:
If yes, process elements linked to this tailoring consideration
should not be
executed (deleted).
If no, process elements linked to this tailoring consideration
should be executed
(standard or tailored). Note that standard or tailored is
acceptable to promote
additional tailoring beyond the model.
Tailoring consideration: project size
Question: Is the project’s lifecycle cost greater than $300M?
Possible values: yes, no
Rules:
59
If yes, the process elements linked to this tailoring consideration
should be
executed (standard or tailored). Note that standard or tailored is
acceptable to
promote additional tailoring beyond the model.
If no, the process elements linked to this tailoring consideration
should be
tailored.
4.6.2.3. Rule Prioritization
In certain cases, the same process element will map to multiple
tailoring considerations.
For those cases, a prioritization schema was developed to yield
only one model output for
each process element.
Associated questions:
Will the project be using the waterfall, incremental, or agile
development
methodology?
Is the project’s lifecycle cost greater than $300M?
Rule: If either the project is not greater than $300M or agile
development is employed,
the process element should be tailored; else, it should be executed
as standard or tailored.
Rule priority: project size, development methodology, and
interoperability impact
Associated questions:
Will the project be using the waterfall, incremental, or agile
development
methodology?
60
Will the solution acquired contain any equipment or interconnected
system or
subsystem of equipment or software?
Rule: If the project does not have an interoperability impact, the
process element should
not be executed. If the project does not have an interoperability
impact and either the
project is not greater than $300M or agile development is employed,
the process element
should be tailored; else, the process element should be executed as
standard or tailored.
Rule priority: development type and development methodology
Associated questions:
Does the project intend to develop a solution or procure a COTS
solution to meet
project requirements?
Will the project be using the waterfall, incremental, or agile
development
methodology?
Rule: If the project is a COTS procurement, the process element
should not be
executed; else, if the project is a development effort and
employing a waterfall or an
incremental methodology, the process element should be executed
(standard or tailored);
else, if the project employs agile development, the process element
should be tailored.
Rule priority: project size and security impact
Associated questions:
Is the project’s lifecycle cost greater than $300M?
Will the solution acquired contain any information or data
requiring special
protections?
61
Rule: If the project has no security impact, the project should not
execute the process
element (should be deleted). If the project has a security impact
and the project is not
greater than $300M, the process element should be tailored; else,
the process element
should be executed as standard or tailored.
4.6.2.4. Rule Set Summary
By design, each of the process elements that do not trace to a
tailoring consideration
should be executed (standard or tailored). Therefore, a full
modeled tailoring plan can be
developed and compared to a fully executed tailoring plan.
The result of establishing the rule set is a full intersection of
the tailoring considerations
and DHS SELC process elements (Table 4-12). The complete case study
rule set is
contained in Appendix B: Case Study Rule Set. Note again that for
any given tailoring
consideration, only a subset of process elements will be relevant;
that is, empty table
entries reflect that there is no relevance between that tailoring
consideration and the
intersecting standard SE process element. For example, in Table
4-12, tailoring
consideration 2 and tailoring consideration 10 are relevant to
process element 1.
62
Table 4-12: Establishing the rule set for the DHS case study
Tailoring Considerations
→
TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 TC 7 TC 8 TC 9 TC 10
S e
o lo
g y
e n ta
p m
p m
iz e
Values → Y N Y N Y N Y N Y N Y N Y N W I A D C Y N
DHS Standard SE Process Elements
Process Element #1
Process Element #107
ST D ST D T D
Values: Y=Yes, N=No, W=Waterfall, I=Incremental, A=Agile,
D=Developmental, C=COTS
Entries: S=Standard, T=Tailored, D=Deleted
4.6.3. Model-Based Project Tailoring Plans
With the rule set established for DHS as described above, the next
step is inputting the
attributes of each of the 24 projects into the model to generate a
custom SE process for
each project. Figure 4-2 depicts an example of a project’s input
and output, highlighting
63
that the input is addressing the 10 tailoring considerations based
on project characteristics,
and the output is a tailoring decision for each of the 107 elements
within the DHS SE
process. Appendix C: Project Cases includes the following:
Project attributes
Tailored SE approach for each project
Modeled SE approach for each project
A determination of whether or not, for each process element, the
model matches the
project’s plan
Figure 4-2: Example of a model input and output for a project
64
4.7. Analysis and Results
Tailoring approaches from the projects and those generated using
the model for each
project are now available for analysis. Three analyses were
performed to investigate the
SEPTM approach to SE process tailoring.
First, the hypothesis was evaluated based on an analysis of how
well the SEPTM
approach matches the ad hoc approach for each project and overall.
Second, an analysis of
each tailoring consideration was conducted to understand if the
model failed to match a
majority of projects for any particular tailoring consideration,
thereby highlighting any
potential improvements to the model. Third, an analysis of each
process element was
performed to understand which elements had the most significant
differences between the
projects’ plans and the model’s outputs. The following subsections
describe the three
analyses and associated results.
For the purposes of this research, at the process element level, a
match occurs when the
actual result (AR) is the same as the modeled result (MR). Possible
actual tailoring
decisions are standard (S), tailored (T), and deleted (D); and
possible modeled tailoring
decisions are standard or tailored (ST), tailored (T), and deleted
(D).ST is used because
.
65
4.7.1. Analysis of Projects
Recall the hypothesis from Section 4.1:
Using the SEPTM to tailor a project’s systems engineering process
can reduce the
amount of manual tailoring required.
The success criteria for confirming this hypothesis are the
following:
Criterion 1: The number of decisions required to generate a custom
SE process for
a project is reduced.
Criterion 2: The SEPTM approach matches the ad hoc approach to an
acceptable
level; that is, for the model to be considered valid for a given
project, at least 75%
of the process elements must match; for the model to be considered
valid overall, at
least 75% of the projects must meet that threshold.
Before presenting the specific results of the analysis, a key point
must be noted. The
analysis of the 11 projects using the incremental development
methodology showed that
the tailoring plan for each project was limited to a single
increment of the project, with a
corresponding note that the tailoring plan would be updated prior
to the initiation of each
66
new increment. As such, the incremental development projects
actually reflect a waterfall-
based tailoring strategy and were evaluated accordingly.
Table 4-14 displays the results of comparing the actual and modeled
case study data.
Figure 4-3 depicts these results in order of highest match
percentage.
Table 4-14: Case study results by project
Project
SST TST TT DD SD TD DST DT ST
Project 1 34 44 0 3 0 0 26 0 0 81 76%
Project 2 48 37 0 2 0 0 20 0 0 87 81%
Project 3 64 18 0 12 8 0 5 0 0 94 88%
Project 4 14 18 9 27 11 2 20 1 5 68 64%
Project 5 21 21 17 11 11 0 24 1 1 70 65%
Project 6 39 9 0 24 5 2 28 0 0 72 67%
Project 7 48 8 0 40 0 1 10 0 0 96 90%
Project 8 82 20 0 1 0 0 4 0 0 103 96%
Project 9 81 2 0 7 1 0 16 0 0 90 84%
Project 10 69 6 3 3 0 0 9 0 17 81 76%
Project 11 49 17 0 16 0 1 24 0 0 82 77%
Project 12 42 7 0 24 8 0 26 0 0 73 68%
Project 13 40 11 0 32 0 0 24 0 0 83 78%
Project 14 92 7 0 3 0 0 5 0 0 102 95%
Project 15 92 8 0 4 0 0 3 0 0 104 97%
Project 16 76 19 0 5 1 0 6 0 0 100 93%
Project 17 53 20 0 8 1 1 24 0 0 81 76%
Project 18 78 10 0 3 12 0 4 0 0 91 85%
Project 19 41 11 0 31 1 0 23 0 0 83 78%
Project 20 43 16 6 18 3 0 10 1 10 83 78%
Project 21 9 17 15 17 3 6 38 2 0 58 54%
Project 22 33 21 7 6 2 2 24 0 12 67 63%
Project 23 61 17 17 4 0 0 6 0 2 99 93%
Project 24 45 18 10 9 0 3 14 0 8 82 77%
% of Projects that Match 75%
67
Figure 4-3: Case Study Results by Project
Per Table 4-14, 75% of projects had at least 75% of actual and
modeled results that
match. This, along with a reduction of the number of process
tailoring decisions from 107
to 10, demonstrates the hypothesis that using the SEPTM to tailor a
project’s SE process
can reduce the amount of manual tailoring required.
That said, it is important to understand the results where the
model did not match the
actual data. There were three projects (4, 5, and 18) that had more
than ten incorrect
matches due to the AR being S and MR being D. This occurs when the
model recommends
deleting a process element that was actually included in a
project’s tailored process. If the
68
model were to be deemed the “correct” approach, this scenario would
be considered a false
positive. If the actual approach were to be deemed as the “correct”
approach, then these
instances would point to the model being incorrect. Also, there
were several projects that
did not perform process elements that were recommended by the model
(DST column in
Table 4-14). If the model were to be deemed as the “correct”
approach, this scenario would
be considered a false negative. If the actual approach were to be
deemed as the “correct”
approach, then these instances would point to the model being
incorrect. While these are
likely cases of non-compliance with the DHS standard SE process,
there may be valid
reasons for not performing those steps that are not addressed with
the established ten
tailoring considerations of the model. As will be discussed in
Section 5, future research
should include an evaluation of the “correct approach”, which will
firmly determine true
and false positives and negatives.
Sections 4.7.2 and 4.7.3 include additional investigations that
will clarify reasons why
the model may not have reproduced the actual approach. Section
4.7.2 investigates issues
with the tailoring considerations, and 4.7.3 investigates issues
with individual process
elements, respectively.
4.7.2. Analysis of Tailoring Considerations
There are two questions forming the analysis of each tailoring
consideration. Table
4-15 lists those questions and provides a corresponding example
using the security tailoring
consideration. Each of the 10 tailoring considerations was assessed
by comparing the
model’s output for each project to the data from each project’s SE
process tailoring plan.
Table 4-15: Tailoring consideration analysis questions
69
To analyze each tailoring consideration, a default process was
established based on
default project attributes within the rule set (Table 4-16).
Projects that had an attribute
different from the default were compared to the model for that
particular attribute. For
example, the default project attribute for the security tailoring
consideration is “No;”
therefore, the 14 projects that do have security impacts were
compared to the model to
assess whether the projects performed the process elements relevant
to the security
tailoring consideration.
Analysis Question Security Tailoring Co