Technology Readiness Levels - ANSER · information about Technology Readiness Levels and the AFRL...

HOMELAND SECURITY STUDIES AND

ANALYSIS INSTITUTE

The Homeland Security Act of 2002 (Section 305 of PL 107-296, as codified

in 6 U.S.C. 185) , herein referred to as the “Act,” authorizes the Secretary of

the Department of Homeland Security (DHS), acting through the Under

Secretary for Science and Technology, to establish one or more federally

funded research and development centers or FFRDCs to provide independent

analysis of homeland security issues. Analytic Services Inc. operates the

Homeland Security Studies and Analysis Institute (HSsaI) as a FFRDC for

DHS under contract HSHQDC-09-D-00003.

HSsaI provides the government with the necessary expertise to conduct: cross-

cutting mission analysis, strategic studies and assessments, development of

models that baseline current capabilities, development of simulations and

technical evaluations to evaluate mission trade-offs, creation and evolution of

high-level operational and system concepts, development of top-level system

and operational requirements and performance metrics, operational analysis

across the homeland security enterprise, and analytic support for operational

testing evaluation in tandem with the government’s acquisition process. HSsaI

also works with supports other federal, state, local, tribal, public and private

sector organizations that make up the homeland security enterprise.

HSsaI’s research is undertaken by mutual consent with DHS and is organized

by Tasks in the annual HSsaI Research Plan. This report presents the results of

research and analysis conducted under

Task 09-01.03.02.13, System Readiness Assessment

of HSsaI’s Fiscal Year 2009 Research Plan. The purpose of the task is to (i)

update and document a technology readiness level calculator; (ii) document a

technology readiness assessment methodology, and (iii) explore how HSSAI

can apply system readiness assessment approaches to support DHS in the

future. This report presents the results of research and analysis conducted in

fulfillment of (iii).

The results presented in this report do not necessarily reflect official DHS

opinion or policy.

HOMELAND

SECURITY

STUDIES AND

ANALYSIS

INSTITUTE

DEPARTMENT OF HOMELAND

SECURITY SCIENCE AND

TECHNOLOGY READINESS

LEVEL CALCULATOR (VER

1.1)

Final Report and User‟s Manual

September 30, 2009

Prepared for

Department of Homeland Security Science and

Technology Directorate

Dr. David McGarvey

Mr. Jim Olson

Dr. Scott Savitz

Task lead

Dr. Gerald Diaz

Division Manager

Mr. George Thompson

Deputy Director

Department of Homeland Security Science and Technology Readiness Level Calculator

ii

ACKNOWLEDGMENTS

This is a minor revision to a report published in 2008 by the Homeland Security Institute,

the predecessor organization to HSSAI. We wish to acknowledge, with thanks, the very

considerable contributions of Tavis Steenbeke, the co-leader of the task that lead to that

report and of Megan McHugh and Laura Parker who were co-authors of that report.

We would like to extend our gratitude to former HSSAI member Ric Blacksten and to

HSSAI member Melanie Cummings for their talent, support, and dedication to various

aspects of this project.

We would also like to thank William Nolte and James Bilbro for sharing their insight and

information about Technology Readiness Levels and the AFRL TRL Calculator (ver 2.2).

It was an invaluable education for us, and critical to the development of the DHS S&T

RL Calculator (Ver1.1).

Finally, we would like to thank Randy Long, Doug Drabkowski, Angela Ervin and

Samuel Francis at the Department of Homeland Security, and Christopher Smith and his

staff at the Transportation Security Laboratory, for working with us to develop user

requirements for the modified calculator and providing feedback on the content of the

DHS S&T RL Calculator.

For information about this publication or other HSSAI research, contact

Homeland Security Studies and Analysis Institute

Analytic Services Incorporated

2900 S. Quincy Street

Arlington, VA 22206

Tel (703) 416-3550 • Fax (703) 416-3530

http://www.homelandsecurity.org

HSSAI Publication Number: 09-01.03.02.13-02

This publication updates and supersedes

HSI Publication Number 08-02.01.04-01

http://www.homelandsecurity.org/


iii

TABLE OF CONTENTS Executive Summary ..................................................................................................................................................... v

Introduction ................................................................................................................................................................. 1

Objective ................................................................................................................................................................ 2

Methodology and Scope ......................................................................................................................................... 2

Background ............................................................................................................................................................ 3 Technology Readiness Levels (TRLs) .............................................................................................................. 3 Modification of TRL scale ............................................................................................................................... 4

AFRL TRL Calculator (ver 2.2) ................................................................................................................................. 7

Overview of the DHS S&T RL Calculator (Ver1.1) ................................................................................................. 9

Summary and Recommendations ............................................................................................................................ 13

Appendix A: User’s Manual for the DHS S&T RL Calculator (Ver1.1) .............................................................. 15

Starting Up ........................................................................................................................................................... 15 For users with Excel 2003 and compatible versions of Excel ....................................................................... 15 For users with Excel 2007 ............................................................................................................................. 15

Quick Start Instructions ........................................................................................................................................ 16

Detailed Operating Instructions ............................................................................................................................ 16

DHS S&T RL Calculator Worksheets .................................................................................................................. 16 START HERE Worksheet ............................................................................................................................... 16 RL Calculator Worksheet .............................................................................................................................. 18 Summary Report Worksheet .......................................................................................................................... 21

Appendix B: TRL, MRL, and PRL Definitions and Descriptions used

in the DHS S&T RL Calculator (Ver1.1)................................................................................................................. 23

Appendix C: Glossary of Terms for the DHS S&T RL Calculator (Ver1.1) and Acronyms .............................. 27

Acronyms ............................................................................................................................................................. 55

Glossary References ............................................................................................................................................. 57

Appendix D: Decision algorithms, Change Log for the DHS S&T RL Calculator (Ver1.1),

and Explanation of Administrative Functions ........................................................................................................ 59

Decision Algorithms ............................................................................................................................................. 59

Log of modifications ............................................................................................................................................ 59

Administrative Functions ..................................................................................................................................... 60

Appendix E: Readiness Level Questions by Category ............................................................................................ 63


iv

LIST OF TABLES TABLE 1: NASA TRL SCALE .......................................................................................................................................... 3

TABLE 2: DHS S&T RL CALCULATOR (VER1.1) HIGH-LEVEL TRL DEFINITIONS ........................................................ 10

TABLE 3: DHS S&T RL CALCULATOR (VER1.1) HIGH-LEVEL MRL DEFINITIONS ....................................................... 10

TABLE 4: DHS S&T RL CALCULATOR (VER1.1) HIGH-LEVEL PRL DEFINITIONS......................................................... 10

TABLE 5: TECHNOLOGY READINESS LEVEL DEFINITIONS AND DESCRIPTIONS ............................................................. 23

TABLE 6: MANUFACTURING READINESS LEVEL DEFINITIONS AND DESCRIPTIONS ...................................................... 24

TABLE 7: PROGRAM READINESS LEVELS DEFINITIONS AND DESCRIPTIONS* ............................................................... 26

TABLE 8: DHS S&T TECHNOLOGY READINESS LEVEL QUESTIONS FOR THE MODIFIED CALCULATOR ....................... 63

TABLE 9: DHS S&T MANUFACTURING READINESS LEVEL QUESTIONS FOR THE MODIFIED CALCULATOR ................. 66

TABLE 10: DHS S&T PROGRAMMATIC READINESS LEVEL QUESTIONS FOR THE MODIFIED CALCULATOR ................. 68

LIST OF FIGURES FIGURE 1: DHS S&T RL CALCULATOR START HERE WORKSHEET. ......................................................................... 17

FIGURE 2: COLOR-CODE KEY FOR RL SCALES IN RL CALCULATOR AND SUMMARY REPORT WORKSHEETS. ................ 18

FIGURE 3: GREEN AND YELLOW SET POINTS. ................................................................................................................ 19

FIGURE 4: EXAMPLE OF TRL QUESTIONS FOR RL 1. ..................................................................................................... 19

FIGURE 5: EXAMPLE OF RL LAYOUT. ........................................................................................................................... 20

FIGURE 6: EXAMPLE OF REAL-TIME COLOR CODED READINESS LEVELS. ....................................................................... 21


v

EXECUTIVE SUMMARY

Technology Readiness Assessments (TRAs) are important procedures for organizations managing

resource allocation for technology development programs. A metric commonly employed in TRAs for

approximating the degree of maturity of a technology is the Technology Readiness Level (TRL) scale

first developed by the National Aeronautics and Space Administration (NASA). This scale has been

implemented and modified since the early 1990‘s in government programs. Ultimately, this work resulted

in a calculator, the Air Force Research Laboratory (AFRL) TRL Calculator, that helps a user assess the

TRL, Programmatic Readiness Level (PRL), and Manufacturing Readiness Level (MRL) of a given

technology or system. This calculator is oriented in its terminology and structure to the Department of

Defense (DoD) research, development, and acquisition process.

In 2008 the Homeland Security Institute (HSI), the predecessor to the HSSAI modified the existing AFRL

TRL Calculator for use within the Department of Homeland Security Science and Technology

Directorate. That new calculator, the Department of Homeland Security (DHS) Science and Technology

(S&T) Readiness Level (RL) Calculator Version 1.0, allowed users to assess the RL of technology, as

well as the RLs of manufacturing, and programmatics, independently of one another.

This document presents Version 1.1, an updated version of the calculator that is compatible with either

Excel 2003 or Excel 2007. (Version 1.0 was developed using Excel 2003.) It revises and supersedes the

previous Final Report and Users Manual. Although we believe the calculator is a useful tool for S&T

Program Managers, its full potential is yet to be realized. Based on our research and analysis to date,

HSSAI recommends that DHS S&T take appropriate steps to

Develop entrance and exit criteria for each RL category.

Develop guidelines for specific applications of the calculator.

Continue refining concepts and terms specific to DHS S&T.

Develop a DHS S&T-specific TRA methodology.

Validate DHS S&T RL Calculator results.

Apply lessons learned by other organizations to modification of RL methodologies.

As described in the body of the report, these efforts will entail a combination of research and analysis

(e.g., identifying new questions and methods) and management actions (e.g., developing a management

directive to govern the use of TRAs in S&T processes).


vi


1

INTRODUCTION

Technology Readiness Assessments (TRAs) are established tools used to qualify technology development

and help make investment decisions within those programs in order to deploy systems or elements of

technology to an end user in a timely fashion. The National Aeronautics and Space Administration

(NASA) and the Department of Defense (DoD) have taken the lead among government agencies in

incorporating TRAs into their technology development programs and refining the TRA process

specifically to their organizations in order to produce operational systems on schedule and within budget.

TRAs are vital to the process of maturing technologies to the point where they can be operationally

produced and deployed. They support that process by

Providing metrics for technology maturity. TRAs help guide evaluation and tracking of

technology maturity levels and program milestones.1, 2

Identifying risk associated with technologies and investment requirements. TRAs help inform

decisions associated with allocating resources and funds for a given technology development.

Identifying potential problems early in a development process when solutions are less expensive

and easier to execute. TRAs provide a systematic method for ensuring the success of a project by

tracking completion of various steps as a project develops.

Identifying gaps in testing, demonstration, and knowledge of a technology‘s current readiness

level and the information and steps necessary to reach the required technology readiness level.2

In the DoD, TRAs are a requirement of all acquisition programs. The DoD Technology Readiness

Assessment Deskbook defines TRAs as ―a systematic, metric-based process and accompanying report

that assesses the maturity of certain technologies.‖1 Although the Department of Homeland Security

(DHS) Science and Technology Directorate (S&T) does not have acquisition programs, per se, it does

have the requirement to identify and develop technologies for use in homeland security.

A metric commonly employed in TRAs for approximating and summarizing the degree of maturity of a

technology is the Technology Readiness Level (TRL) scale first developed by the National Aeronautics

and Space Administration (NASA) and since adapted and adopted by the Department of Defense (DoD)

and other agencies. The TRL scale begins at 1 (―basic principles observed and reported‖) and goes

through 9 (―actual system flight proven through successful mission operations‖). While a TRA does much

more than assign a TRL, the TRL provides a useful summary index.

Technology Readiness Levels (TRLs) can facilitate a cost-effective, systematic process of transitioning

technology from development to an operational environment by identifying a program‘s technical risk in

areas such as design, architecture, cost, schedule, and manufacturing. As a result, TRLs can be a useful

tool for DHS Program Managers (PMs) and other decision makers in assessing technology development

programs.

1 Deputy Under Secretary of Defense for Science and Technology (DUSD(S&T)). (May 2005). Technology Readiness Assessment (TRA)

Deskbook.

2 U.S. Department of Energy, Office of Environmental Management. (March 2008). Technology Readiness Assessment (TRA)/Technology

Maturation Plan (TMP) Process Guide.


2

In 2001, William Nolte and coworkers at the Air Force Research Laboratory (AFRL) developed a TRL

Calculator tool to aid in assigning and documenting TRLs in technology development programs. This

calculator was used to help create a standard repeatable method for determining the TRLs. Initially

released in January 2002, it is a Microsoft Excel workbook that presents a user with a series of questions

to assess the maturity or readiness level (RL) of the technology. Because TRLs by themselves do not

provide a full picture of risk associated with a program/project or the difficultly required to advance a

program/project, Nolte further refined his calculator by May 2004 to include Manufacturing Readiness

Levels (MRLs) and Programmatic Readiness Levels (PRLs) (AFRL TRL Calculator (ver 2.2)).

Objective

In 2008, the Homeland Security Institute (HSI) adapted the AFRL TRL Calculator for use within the

DHS S&T Directorate by PMs or others interested in tracking the progress of technology development.

The objective was to improve the transition of technologies into and out of development within S&T,

provide more uniformity in program management, improve the documentation of technology readiness

decisions, and facilitate development of TRA methodologies, including entrance and exit criteria for a

given maturity level throughout the development life cycle of a technology within a DHS program.

Version 1.0 of the DHS S&T Readiness Level (RL) Calculator was released in December 2008. 3 It was

developed using Excel 2003.

In 2009, HSI‘s successor—the HSSAI–revised the calculator. Version 1.1 is now compatible with Excel

2007. The revised User‘s Manual provides instructions for use with either version of Excel. In addition,

the revised Final Report updates the Institute‘s recommendations for continued development and use of

the calculator.

Methodology and Scope

The DHS S&T RL calculator was originally developed as follows. Using open-source information, HSI

analysts gathered information specific to TRL scales and historical modifications of those scales. They

met with various DHS S&T stakeholders to facilitate development of end user requirements, and then

attempted to clarify ambiguous RL questions, define all terms used in a DHS-specific orientation, and

apply the lessons learned from historical modifications. As a result, the modified calculator agreed with

DHS S&T terminology and documented management procedures. In addition, modifications included

advanced user functionality, such as the ability to update the calculator‘s questions as new DHS lexicon

and/or management processes are adopted and generate RL-specific reports for DHS PM analysis. How

RL categories impact one another, however, or the importance of various questions in each of the RLs

was not addressed in the modified calculator. Also outside the scope of this task of calculator

development were RL entrance and exit criteria.

Potential users were assumed to reside within the Chemical/Biological Division (CBD) of DHS S&T,

although it was anticipated that the product would be useful in other divisions tasked with technology

development.

3 McGarvey, D. et.al. (31 December 2008). Department of Homeland Security, Science and Technology Directorate, Readiness Level Calculator

(HSI Publication 08-02.01.04-01). Version 1.0 of the calculator was developed under contract W81XWH-04-D011, ―Homeland Security

Institute.‖ The Homeland Security Institute was established in 2004 as DHS‘s Federally Funded Research and Development Center (FFRDC)

for homeland security studies and analyses. In March 2009, DHS established a successor organization, the Homeland Security Studies and

Analysis Institute (HSSAI).


3

Background

Technology Readiness Levels (TRLs)

TRLs were originally developed by NASA. In the early 1980s, NASA observed delays and costs overruns

of its programs. Analysis by Werner Gruel, NASA comptroller, concluded that immature technologies

resulted in unpredictable development costs and schedules, and the resultant cost growth and schedule

slip.4 To address this cause-and-effect, the NASA instituted TRLs, a methodical system that provides a

consistent framework for assessing technology maturity and maintaining cost and schedule within a

program.5 TRLs are relevant to both simple and complex technological systems, as well as to their

component subsystems. They are applicable to software and/or hardware or to systems encompassing

both software and hardware elements. TRLs can also be used as exit criteria for program life cycle

phases. In terms of program management, knowing a program‘s TRL can prescribe an action plan of

activities that still need to be accomplished in order to facilitate transition for a particular technology to an

operational end user.

TRLs provide measures that can indicate a program‘s risk and potential for success of transitioning a

technology to an end user, in other words, the value of investment. For example, a low TRL, e.g. TRL 1,

represents high programmatic risk because the technology requires investment in more developmental

milestones; while a high TRL, e.g., TRL 7, indicates the technology has matured further in development,

achieved more milestones, and is associated with less programmatic risk.

By the 1990s, TRLs had been modified to address nine readiness levels used across NASA today. The

NASA TRL levels and definitions are included below in Table 1: NASA TRL scale.

Table 1: NASA TRL scale5

NASA

TRL

Definition

1 Basic principles observed and reported

2 Technology concept and/or application formulated

3 Analytical and experimental critical function and/or characteristic proof-of-concept

4 Component and/or breadboard validation in laboratory environment

5 Component and/or breadboard validation in relevant environment

6 System/subsystem model or prototype demonstration in a relevant environment (ground or space)

7 System prototype demonstration in a space environment

8 Actual system completed and ―flight qualified‖ through test and demonstration (ground or space)

9 Actual system ―flight proven‖ through successful mission operations

In 1999, the DoD adopted the use of TRLs during its acquisition phase to aid in the decisions made

during technology development.6,7

4 Bilbro, J. JB Consulting International-Technology Readiness Levels. Last accessed on 12/19/08 at

http://jbconsultinginternational.com/TechnologyReadinessLevel.aspx,

5 Mankins, J. (April 6, 1995). Technology Readiness Levels—A White Paper. Advanced Concepts Office, Office of Space Access and

Technology, NASA.

6 DoD 5000.1. (October 23, 2000). The Defense Acquisition System.

http://jbconsultinginternational.com/TechnologyReadinessLevel.aspx


4

Modification of TRL scale

Because the TRL scale is applicable to many different technologies and can be interpreted or modified for

specific types of technologies, several groups have adapted TRL definitions for their own use. The Army

has developed a mapping of the TRLs to software and the Army Medical Research and Materiel

Command has defined corollaries for biomedical TRLs.8 In addition, TRL scales have since been adopted

internationally in Canada, the United Kingdom, and Japan.

The TRL scale, however, only provides a snapshot of the maturity of a technology or system at a given

point in time. 9 In fact, in an ongoing development program, the technology will invariably be in constant

flux. And depending on the type of technology, the TRL may decrease or increase as a result of

availability of components, changes in funding, or mission directives. Other shortcomings of the TRL

scale, pointed out by Jim Smith,10 include blurring contributions to readiness (e.g., how programmatics

influence TRLs), product criticality (e.g., an increased importance for developing a technology may push

necessity for skipping TRLs or assuming completion of steps to reach self-imposed or mandated

deadlines), software aging (e.g., passé or critical software components of systems may require updating,

impacting the overall systems TRL), and readiness in context (e.g., who is really looking at the RL and

why?).

In general, readiness is a measure of the suitability of a technology or product for use within a larger

system in a particular context. It is also a measure of the risks associated with developing or investing in a

program associated with developing the technology or system. But TRLs do not provide a full assessment

of the difficulty of integrating technology into an operational system, provide no guidance of the potential

uncertainty in moving through the maturation of the technology, and offer no comparative analysis

techniques for alternative TRLs.

Alternative methodologies for assessing these RLs are being developed. Some combine the desirable

aspects of TRLs with additional readiness attributes, such as PRLs and MRLs, to better assess program

risk. These new methodologies include developing evaluation criterion, or milestones, tailored to the

context of the RL assessment. In short, improvements in existing methodologies and development of

alternative methodologies to assess RLs is an ongoing field of study. In an effort to address readiness

attributes not captured when using the NASA or DoD TRL scale, numerous types of readiness levels have

been created since the inception of TRLs, including MRLs, PRLs, Integration Readiness Levels (IRLs),

and System Readiness Levels (SRLs). Further, some of these new methodologies suggest assessing a

combination of scales in order to generate a more accurate picture of a technology‘s maturity.

Integration Readiness Levels (IRLs)

The existing TRL scale does not accurately address integration of a component technology into a

complete system. In other words, component technologies may advance at different speeds along the TRL

7 Sauser, B., et.al. (April 7-8, 2006). From TRL to SRL: The Concept of System Readiness Levels. Conference on Systems Engineering

Research, Los Angeles, CA.

8 Graettinger, C. et.al. (August 2002). Using the ―Technology Readiness Levels‖ Scale to Support Technology Management in the DoD‘s

ATD/STO Environments: A Findings and Recommendations Report by the Software Engineering Institute Conducted for Army CECOM.

Carnegie Mellon Software Engineering Institute, Pittsburgh, PA.

9 Nolte, W. Technology Readiness Level Calculator (ver 2.2).

10 Smith, J. (2005). An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software. Proceedings of the 38th Hawaii

International Conference on Systems Sciences.


5

scale, however, as a system, an adequate TRL is difficult to assess. Developers of IRLs contend most

development of complex systems fail at these ―integration points.‖ IRLs were developed for an accurate

assessment of interface maturity between developing technologies.7

At this time IRLs are not formally a part of the DHS acquisition process, and hence are not included in

the DHS S&T RL Calculator.

System Readiness Levels (SRLs)

TRLs most accurately apply to individual technologies or system components. It becomes more complex

to apply the existing TRL scale to a system. SRLs indicate the level of maturity applied at the system-

level. SRLs are determined by using the current concept of TRLs combined with IRLs. The SRL of a

given system is a function of individual component TRL maturities and the links between them, as

indicated by the IRL.7 SRLs are useful when going from the individual technology to a system context

which may involve multiple technologies, as is the case with most technologies in the operational

environment.

At this time SRLs are not formally a part of the DHS acquisition process, and hence are not included in

the DHS S&T RL Calculator.

Manufacturing Readiness Levels (MRLs)

In 2003, the Government Accountability Office (GAO) recommended in GAO Report 03-476,

establishing cost, schedule, and quality targets for product manufacturing early on in technology

development in order to obtain process maturity.11 The report suggests that design and manufacturing

knowledge should be obtained early in product development for a product to be successful. In response,

the Joint Defense Manufacturing Technology Panel developed MRL definitions as well as Manufacturing

Readiness Assessments (MRAs)12. This MRL scale helps program managers assess manufacturing risks,

which will facilitate identification of areas that require additional management attention or investment.

―Manufacturing readiness and producibility are as important to the successful development of a system as

are readiness and the capabilities for the system.‖1

Though MRLs were created from the manufacturing perspective to evaluate ―manufacturing readiness‖ of

a product and supplement existing TRLs, they, too, have limitations. A limitation of the MRLs is that the

lower MRL levels can be difficult to correlate to corresponding TRL numbers due to the technology

immaturity (i.e., it is difficult to know what types of manufacturing steps are required when a technology

concept hasn‘t yet been proven). As a result of these limitations, MRL levels 1 and 2 are not used in the

DHS Readiness Level Calculator.

Programmatic Readiness Levels (PRLs)

PRLs were developed to address program management concerns, such as documentation of programmatic

milestones seen as vital to successful technology product development.13 A PRL scale was developed by

HSI, in 2008, to align with the TRLs. The PRL scale follows basic systems engineering steps and is

discussed further in the following sections.

11 GAO Report 03-476. (May 2003). Defense Acquisitions, Assessments of Major Weapons Programs.

12 Joint Defense Manufacturing Technology Panel Manufacturing Readiness Level Working Group. (February 2007). MRL Guide.

13 Nolte, W. (October 20, 2003 ). Technology Readiness Level Calculator. NDIA Systems Engineering Conference.


6


7

AFRL TRL CALCULATOR (VER 2.2)

As described in the Introduction, William Nolte of the AFRL developed a TRL Calculator in 2001 to

create a standard, repeatable method for determining TRLs. This original TRL calculator (ver 1.0),

released in January 2002, is a Microsoft Excel spreadsheet application that presents a user with a series of

questions by TRL about the technology. The methodology for the TRL assessment used in the calculator

was refined by May 2004, to include questions relating to TRLs, MRLs, and PRLs (TRL Calculator (ver

2.2). The ―user‖ has the option of assessing the ―overall TRL‖ based on three combinations of questions:

1) only TRL questions,

2) TRL questions and PRL or MRL questions, or

3) all three categories of questions together.

In any case, TRL questions are always required.14

The AFRL TRL Calculator (ver 2.2) calculates the overall TRL of the technology or system in question

by averaging the responses to all selected categories in a RL. In other words, PRLs and MRLs are

ultimately additional TRL questions, if included. If the user cannot answer those questions associated

with manufacturing, for example, the overall TRL of the technology or system for that RL will be low,

even if the TRL questions alone can be adequately addressed.

The user always has the option of looking at the results if only the technology readiness questions are

considered, but not the option of looking at results for only manufacturing or programmatic readiness.

14 Nolte, W. (2004). TRL Calculator Version 2.2 Release Notes.


8


9

OVERVIEW OF THE DHS S&T RL CALCULATOR

(VER1.1)

The DHS S&T RL Calculator Version 1.0 was developed by HSI, in 2008, for the DHS S&T Chem/Bio

Division PMs and others interested in assessing technology and associated programs. HSI facilitated

developing end user requirements with DHS S&T potential users.

Like its predecessor, Version 1.1 of the calculator provides the user options to assess Technological,

Programmatic, and/or Manufacturing Readiness Levels (herein, referred to as categories) for a given

technology or system. The calculator allows a user to assess any or all of the RLs required for successful

technology development and transition: TRLs, MRL, and PRLs. The user has the ability to do these

assessments separately, and at the same time. Each of these assessments is independent of the other. In

other words, it is essentially three calculators in one workbook. All of these categories have been defined

along a scale of 1-9. (However, the lowest MRL a user can achieve or begin to address readiness is 3,

since it is assumed that a technology or system at a TRL 1 or 2 does not allow for a meaningful ability to

address manufacturability.)

In general, the levels can be grouped into three higher-level activities:

1. RLs 1-3: Research and Development (R&D): these activities most likely occur in a basic

laboratory setting, prior to identification of a sponsoring organization.

2. RLs 4-6: Technology Demonstration: these activities occur as a result of funding provided by

a sponsoring organization, such as DHS S&T.

3. RLs 7-9: Production and Deployment: these activities occur once the technology has been

transferred from the sponsoring organization to the customer or end user.

Deviations from this grouping occur. For instance in some cases systems are taken into development

when some of the required technologies are only at TRL 2 and in some cases the sponsoring organization

supports development through TRL 7 or higher before transition to the customer or end user.

These basic groups begin to show how all three RL categories (technology, manufacturing,

programmatic) are intimately linked, generally requiring information and interaction with one another.

How RL categories impact one another or the importance of various questions in each of the RLs are not

addressed in the DHS S&T RL Calculator. Nor does the calculator address the question of what TRL,

MRL, or PRL levels are required for different entrance and exit criteria. This aspect is left for future

modifications based on DHS S&T programmatic directives.

As with the AFRL TRL Calculator (ver 2.2), the DHS S&T RL Calculator (Ver1.1) is an Excel

workbook.15 The user specifies what categories of RLs are to be calculated (TRL, MRL, or PRL

separately, or any combination of these). In addition to a visual scale for each category that indicates the

readiness levels achieved and not achieved, this calculator also has the capability to generate category-

specific summary reports. Each summary report details each RL and the responses provided by the user.

These reports are intended to facilitate discussions about the results, in addition to preparing a PM for

next steps in order to progress to the next readiness level.

15 Two variants are provided, one for use with Excel 2003 and compatible versions of Excel and one for use with

Excel 2007.


10

The following tables provide the top-level definition for each level in the RL categories (as modified by

HSI) (Tables 2-4). Definitions of specific terms can be found in Appendix C: Glossary of Terms. Full

explanations of the RLs are provided in Appendix B: TRL, MRL, and PRL Definitions and Descriptions

used in the DHS S&T RL Calculator (Ver1.1).

Table 2: DHS S&T RL Calculator (Ver1.1) high-level TRL definitions

TRL TRL Definition

1 Basic principles observed and reported.

2 Technology concept and/or application formulated.

3 Analytical and experimental critical function and/or characteristic proof-of-concept.

4 Component and/or breadboard validation in laboratory environment.

5 Component and/or breadboard validation in relevant environment.

6 System/subsystem model or prototype demonstration in a relevant environment.

7 System prototype demonstration in an operational environment.

8 Actual system completed and qualified through test and demonstration.

9 Actual system proven through successful mission operations.

Table 3: DHS S&T RL Calculator (Ver1.1) high-level MRL definitions

MRL MRL Definition

3 Manufacturing process development.

4 Critical manufacturing processes prototyped.

5 Prototype manufacturing system.

6 Manufacturing process maturity demonstration.

7 Manufacturing processes proven.

8 Manufacturing concepts identified.

9 Laboratory manufacturing process demonstration.

Table 4: DHS S&T RL Calculator (Ver1.1) high-level PRL definitions

PRL PRL Definition

1 Identification of basic scientific concepts and Performers.

2 Establishment of program with identified customer and technology.

3 Program risk, requirements, and performance characteristics and measures are determined.

4 Integrated Product Teams and working groups for developing and transitioning technology are

established.

5 Systems engineering methodology, system architecture and end user involvement are established.

6 Formal requirement documents, final Test and Evaluation Master Plan, and Systems Engineering Plan

are complete.

7 Finalized Verification, Validation and Accreditation of system.

8 Training and Test and Evaluation Documentation are complete.

9 Safety and Training is complete.


11

Appendix A provides operating instructions for the calculator. While there are several worksheets within

the workbook, only two require a user to physically enter information in order to generate TRL, MRL,

and PRL-specific summary reports: the Start Here and RL Calculator workbooks.


12


13

SUMMARY AND RECOMMENDATIONS

HSSAIhas modified the existing AFRL TRL Calculator for use within the DHS S&T. This new calculator,

the DHS S&T RL Calculator (Ver 1.1), allows users to assess the RLs of technology, manufacturing, and

programmatic, independently of one another. This calculator is a step towards developing a standardized

TRA methodology within DHS for assessing and tracking progress in technology development programs.

While this calculator should be useful as a guide for Program Managers (PMs) and for conducting

program- and technology-related budget allocations, there is still work to be done. HSSAImakes the

following recommendations based on its efforts associated with the research into TRAs, RLs, and the

resultant DHS S&T RL Calculator (Ver1.1).

1. Develop entrance and exit criteria for each RL category. Currently, the DHS S&T RL

Calculator has generic types of milestones presented as questions that have no more weight than

a previous or following question. Further, a user can achieve any RL without achieving lower

RLs because there is no requirement that the level below, or specific steps need to be achieved

first. HSSAI recommends that certain questions be identified as mandatory for passing to the

next RL. In this way, PMs would be assured that all projects are addressing the same criterion

for a given level and when a technology has been deemed RL 5, for example, have an

understanding of exactly what has been accomplished with that classification.

2. Develop guidelines for specific applications of the calculator. While this task focused on

DHS S&T Chem/Bio Division PMs, the results of an assessment may vary by user. A laboratory

PM, for example, may require different information than a sponsoring organization (e.g.,

manufacturing questions may not impact a laboratory‘s information collection in RLs 1-3, but

may impact a sponsoring organization‘s willingness to assume responsibility for a technology

development).

3. Continue refining concepts and terms specific to DHS S&T. Having common understanding

on the meaning of terms is important for ensuring minimal confusion among those assessing

RLs and those making policy decisions based on them.

4. Develop a DHS S&T-specific TRA methodology. TRAs historically have been modified to

apply to specific organizations. To facilitate understanding between DHS, and other

organizations (i.e., laboratories, manufacturers), a DHS TRA methodology should be vetted and

made into a management directive. This would also allow a user to add questions specific to

documented DHS programmatic milestones.

5. Validate DHS S&T RL Calculator results. Consider having an outside organization assess a

given project‘s RL using the calculator and compare the results to the same assessment

performed by DHS S&T PMs. Differences between the assessments would be valuable starting

points for discussions and setting expectations.

6. Apply lessons learned by other organizations to modification of RL methodologies. TRLs

have shortfalls, For instance, they do not tell us what efforts have preceded or what efforts are

expected to follow the achievement of the TRL. In addition, they do not account for system

integration issues. There is precedence, however, for modifying the RL methodology in such a

way that accounts for ―other factors‖ that should be considered, in order to more accurately

understand what is required to move from one RL to the next or address integration issues. For

example, James Bilbro developed the Advancement Degree of Difficultly (AD2) as another


14

assessment scale for technology maturity. AD2 is the description of what is required to move a

system, subsystem, or a component from one TRL to another taking into account the cost,

schedule, risk, people and tools available, and organizational aspects, such as the ability of an

organization to reproduce existing technology.


15

APPENDIX A: USER‟S MANUAL FOR THE DHS S&T RL

CALCULATOR (VER1.1)

Starting Up

There is no ―installation‖ of the DHS RL Calculator. The calculator is a Microsoft (MS) Excel workbook

template. Two variants are provided. One, DHS RL Calculator for Excel 2003 1.1.xls is for use with

Excel 2003 and some compatible versions of Excel. The other, DHS RL Calculator for Excel 2007

1.1.xlsm, is for use with Excel 2007.

For users with Excel 2003 and compatible versions of Excel

Use the version of the calculator titled DHS RL Calculator for Excel 2003 1.1.xls.

Before attempting to use the calculator, make sure that Excel is set to Medium security so that the macros

in the workbook will run. To do this, open Microsoft Excel. Select "Tools", then "Macros" and

"Security." Pick the "Medium" radio button.

Open the calculator workbook - DHS RL Calculator for Excel 2003 1.1.xls. A dialog box will pop up

asking whether or not you wish to allow macros to run. The default selection is "Disable Macros."

Change this to "Enable Macros" or the calculator won't work.

The calculator will be open to the START HERE worksheet.

For users with Excel 2007

Use the version of the calculator titled DHS RL Calculator for Excel 2007 1.1.xlsm.

When using Excel 2007 it is necessary that macros be enabled each time the calculator is opened.

To do this click on the ―Options‖ box next to the ―Security Warning Some active content has been

disabled‖ warning that appears when the calculator is first opened. Choose ―Enable this content‖ from the

options that appear.

The calculator will open to the START HERE worksheet.

http://www.homelandsecurity.org/hsireports/DHS RL Calculator for Excel 2003 1.1.xls

http://www.homelandsecurity.org/hsireports/DHS RL Calculator for Excel 2007 1.1.xlsm

http://www.homelandsecurity.org/hsireports/DHS RL Calculator for Excel 2007 1.1.xlsm


16

Quick Start Instructions 1 Enter identification information (Start Here)

2 Select RL categories to be used (Start Here)

3 If desired, select level to begin RL questions for each RL from drop

down scales (Start Here)

4 Click ―Continue to Calculator‖

5 Save workbook with unique identifier

6 Set green and yellow set points (RL Calculator)

7 Begin answering questions until ready to generate summary report(s)

8 Save workbook with different name

9 Click ―Generate Report‖

10 Save workbook with yet another name

11 Print summary reports

Detailed Operating Instructions

The DHS S&T RL Calculator (Ver1.1) is an Excel workbook containing six worksheets visible to a user:

1. Start Here: This is the default opening worksheet. The user should enter identification

information, select RL categories and, if desired, the level for each category to begin assessing

specific RLs on the next worksheet.

2. RL Calculator: The user should set threshold values for green and yellow set points and answer

questions until desired stopping point (see RL Calculator worksheet explanation below). Also,

the user can generate reports from this worksheet.

3. Three (3) summary report worksheets: A RL summary report worksheet will only appear if

that RL category (TRL, MRL, and PRL) has been selected on the Start Here worksheet.

4. Glossary: Definitions for terms used throughout the calculator. This can be accessed from each

worksheet or by directly clicking on the tab. See Appendix C: Glossary of Terms.

DHS S&T RL Calculator Worksheets

START HERE Worksheet

It is essential that the user begin by selecting categories desired on the START HERE worksheet.

Deviation from this will cause the calculator to function incorrectly. Figure 1 shows the Start Here

worksheet.

On the START HERE worksheet the user should enter the Project Name, Project Manager, and Date the

RL is being calculated in the spaces provided at the top of the worksheet.


17

Figure 1: DHS S&T RL Calculator START HERE worksheet.

Glossary: This button opens the glossary of terms worksheet. These terms are used in the calculator

and/or related to concepts addressed in the questions. The glossary may be consulted at any point in the

operation of the calculator.

Select technology types to be included: The user selects whether the project to be evaluated consists of

hardware only, software only, or a combination of hardware and software.


18

Select types of RLs to evaluate: The user selects which types of RLs are to be included in the evaluation

– Technology, Manufacturing, and/or Programmatic. For each RL selected, the user has the option of

choosing a specific level to begin the questions. In addition, full definitions for each scale are provided.

Assuming a given RL has already been achieved: The user also has the option of assuming that a given

RL has been achieved for each of the RLs to be included in the evaluation. This allows the user to avoid

answering the questions for that and lower RLs. Note: If the user decides to begin assessing RLs at a

given level, all levels below the one selected will be assumed 100% completed for that category. The user

will not be able to change this once ―Continue to Calculator‖ is clicked.

“Continue to Calculator”: Once the user is satisfied with the answers supplied on the START HERE

work sheet, but not before, the user must select the ―Continue to Calculator‖ button. This is an

irreversible step that prompts the user to now save the workbook with a suitable name in a suitable

location and takes the user to the TRL Calculator worksheet, which is the calculator proper.

The newly named workbook can now be saved and opened at will while the questions on the TRL

Calculator worksheet are being answered. Note: Once the file is saved after ―continue to calculator‖ is

selected, the user cannot go back to the START HERE worksheet and change or enter new information to

generate new reports without causing malfunction. The only way to change information on the START

HERE page, at this point, is to completely start over by closing the document and reopening the template.

RL Calculator Worksheet

Upper portion of the worksheet

Program Name, Program Manager, and Date will have been copied from the START HERE worksheet.

There are two buttons on the top of the worksheet that will be of interest only if parts of the bottom

portion of the worksheet have been completed.

1. ―Start Over on this Page‖ button unassumes all Levels, unchecks all boxes, but does not show

any RL selected as omitted on the START HERE page.

2. ―Create Report‖ button generates RL summary reports of questions answered. See RL

Summary Reports below.

RL scales appear for each of the RL categories selected on the START HERE worksheet. Figure 2 shows

the key for those scales based on the answers to the questions in the lower part of the worksheet (see

detailed explanation in the following section, Lower portion of the worksheet).

Figure 2: Color-code key for RL scales in RL Calculator and Summary Report worksheets.

Colors are calculated based on two ―set points.‖ Set points are percentages of questions required by the

user to be answered in order to achieve a ―green‖ or ―yellow‖ grade (Figure 3 shows default values of

Most, if not all, tasks required for this TRL have been achieved.

This TRL has not been achieved.

Many of the tasks required for this TRL are justifiably achieved.


19

100% and 75%, respectively). If a set of questions has not been answered to a combined total percentage

of the yellow set point, the level will be red.

Figure 3: Green and yellow set points. In this example, the user must answer 100% of the questions for a

given level to achieve “green.” If the user answers between 99.9% and 75%, the level will be yellow. Any

percentage below 75 will show red.

Lower portion of the worksheet

For each RL category selected on the START HERE worksheet, the calculator then poses a set of

questions for each level on its related scale of 1-9. See Appendix E for list of questions by category and

level. Figure 4 provides an example of how the questions appear in the calculator.

Figure 4: Example of TRL questions for RL 1.

Questions are preceded by several data fields. Figure 5 shows these fields and explanations are provided

below.

Apply? (Y/N): There is a drop-down menu here for ‗Y‘ or ‗N‘. The default setting is that all

questions apply. The user can decide that certain questions do not apply to the project/technology

being evaluated. Entering an ‗N‘ in this field will remove a question from the evaluation (green

‗Y‘ turns to red ‗N‘). The question will still appear in the list, but any answer supplied will be

ignored in the calculations. If ‗N‘ is selected for a question, it will also appear on the appropriate

RL Summary Report worksheets in a field named ―Questions marked Not Applicable.‖.

H/SW Both: In this field, ‗H‘ indicates hardware, ‗S‘ indicates software, and ‗B‘ indicates both

hardware and software. The user should not try to change these.

Ques Catgry: In the Question Category field, ‗T‘ indicates a technology question, ‗M‘ indicates

manufacturing, and ‗P‘ indicates programmatic. The user should not try to change these.

% Complete: The user ―answers‖ each question according to a percentage of 100 completed. For

example, if the question is successfully answered as ‗yes‘, the user will check the box that

indicates 100%. If the answer is a partial ‗yes‘, but not 100%, a percentage of 100 can also be

inserted manually by changing the green 100 or by using the sliding bar associated with the

question. Default for all questions is ‗no‘ or 0%. Answers provided will appear in the Summary

Report. The green cell displays 100 by default and is technically counted as “0%” or not

completed unless the checkbox is manually clicked or a different value is added.

Do you want to assume completion checkbox: Above the questions for each level there is a red

check box that allows the user to assume completion of this RL. Note: This box applies to all the

RLs being considered at that level (technology, manufacturing, and/or programmatics). The user

Green Set Point: 100% Yellow Set Point: 75%

Both Catgry

B T 0

B T 0

S T 0

S T 0

B T 0

S T 0

S T 0

Have mathematical formulations of concepts that might be realizable in software been developed?

Have the basic principles of a possible algorithm been formulated?

Are physical laws/assumptions for new technology defined?

Do rough calculations support the concept?

% Complete

Do paper studies confirm basic scientific principles of new technology?

Know what software needs to do in general terms?

Level 1 (Check all that apply or use slider for % complete)

Does it appear the concept can be supported by software?


20

cannot assume completion of individual RLs on this page, only on the START HERE worksheet

using the drop-down scales.

Figure 5: Example of RL layout.

The background of the questions is color-coded to indicate whether the question refers to technology,

manufacturing, or programmatics, information that is also included in the Ques Catgry field (Blue:

Technology, Green: Programmatic, and Pink: Manufacturing).

RL Questions

Questions should be answered beginning with those at the lowest RL and proceeding down the page. As

the questions are being answered the calculator continually recomputes the RLs achieved and displays the

results in the scales in the upper portion of the worksheet.

At its core, the calculator sums responses to questions for a given category and RL from the list of

questions and follows a few pre-defined rules: summation of all questions answered ―yes,‖ some

percentage of ―yes,‖ ―not applicable,‖ and ―not answered.‖ As the questions posed are answered, the

calculator displays a red, green, yellow color-coded scale associated with each category of question (this

will appear in the upper section of the worksheet).

The calculator contains two user-defined thresholds or ―set points‖: green and yellow. As questions are

answered, the calculator displays a color-coded scale for that RL according to the set points.

The calculator displays a ―green‖ status for a given RL if the percentage of questions answered equals or

exceeds the green set point. The default setting for the green set point is 100% (i.e., each question for a

given category must be answered 100% to achieve green), but this can be changed by the user.

The calculator displays a ―yellow‖ status for a given RL if the percentage of questions answered equals or

exceeds the yellow set point, but is below the green set point. The default setting for the yellow set point

is 75% (i.e., the combined totals of all the questions in that category for that RL must equal 75% or

greater to achieve yellow). NOTE: The calculator has been modified to calculate the portion of 100%

each question contributes. No questions within a category carry more value than another. For example, if

there are 10 technology questions for RL 1, each question has a value of 10% of the total. If a percentage

of 100 is inserted for a given question, that percentage is multiplied by the value of that question which is

linked to the total number of questions for that category and RL. Answered questions are automatically

summed for that RL in real time. If the total percentage of combined answers meets or exceeds a given

Ques

Apply? H/SW Ques

(Y/N) Both Catgry

Y H P 100

Y H M 100

Y B P 29

Y B P 100

Y H M 100

Y B P 100

Y B P 100

Y H M 100

Y H M 50

Y B P 100

Y B P 100

Y B P 100

Y B P 100

Is customer participating in requirements generation?

Has customer representative to work with R&D team been identified?

Have programmatic risk mitigation strategies been documented?

Have design techniques been identified and/or developed?

Has preliminary value analysis been performed?

Have programmatic risks been identified?

Can key components needed for breadboard be produced?

% Complete

Does basic laboratory research equipment verify physical principles?

Readiness Level 3 (Check all that apply or use slider for % complete)

Have scaling studies been started?

Have system performance characteristics and measures been documented?

Have current manufacturability concepts been assessed?

Has analysis of alternatives been completed?

Has Technology Transition Agreement (TTA) including possible TRL for transition been drafted?

Do you want to assume completion of Level 3?


21

threshold, it will display the appropriate color. If the total does not meet either green or yellow ranges, the

scale will display red for that RL.

Green indicates a satisfactory completion of the given level. Yellow indicates a percentage lower than

green, and red indicates the user has not completed the level to a satisfactory degree. Questions marked

―not applicable‖ do not count against this calculation.

NOTE: If the user wanted to start at TRL 3 on the START HERE worksheet, then all questions pertaining

to Technology at levels 2 and below would automatically be answered ―yes‖ (and would be depicted as

green on the scale related to that RL as shown below for TRL SCALE). In addition, if more than one

category was selected on the START HERE page, also shown in the example below, TRL questions for

levels 1 and 2 will be automatically checked 100%.

Figure 6: Example of real-time color coded readiness levels. In the example provided, there is a TRL of 2, an

MRL of 3, and a PRL of 2.

„Comments‟ text box: The user has the option to enter information concerning the RL under

consideration in this text box. These comments will automatically show up on the Summary Report

worksheets.

“Generate Report”: At any time during the process of answering the questions the user may press the

Create Report button on the upper portion of the worksheet to generate reports of the status of each RL

under consideration. Pressing this button will open a spreadsheet showing the status of one of the RLs

under consideration. Responses in the RL Calculator worksheet can be changed, however, at a later date

by saving the file once before generating a report and once after with different names. If the user desires

to change or add responses, he/she should open the file saved before generating reports and go to RL

Calculator worksheet after opening the file (worksheet tab at the bottom of the document).

Summary Report Worksheet

At the top of a report there is a summary of the RL achieved or partially achieved, A list of any question

marked as Not Applicable will be in the next section of the report. These question are listed regardless of

the level achieved or the RL of the report.

Lists for each readiness level of all the questions for this level are sorted in descending order by the

degree of completion provided appear next. Those answered as completed to meet the green standard are

presented with a green background; those not meeting the green standard but satisfying the yellow

standard are presented with a yellow background, and those not meeting the yellow standard are presented

with a red background. If comments have been added on the RL Calculator worksheet, they will show up

under each associated RL summary. The user has the option to edit or add to this text. Finally, each RL

1 2 3 4 5 6 7 8 9

G G

1 2 3 4 5 6 7 8 9

G Y

1 2 3 4 5 6 7 8 9

G G Y

PRL SCALE both Hardware and Software

TRL SCALE both Hardware and Software

MRL SCALE both Hardware and Software


22

will show the percentage of questions answered. The user may print copies of these reports. (Use Print

Preview to adjust the printout).

See Appendix D for details on editing the DHS S&T RL Calculator (Ver1.1).


23

APPENDIX B: TRL, MRL, AND PRL DEFINITIONS AND DESCRIPTIONS USED IN THE DHS S&T RL CALCULATOR (VER1.1)

Table 5: Technology Readiness Level Definitions and Descriptions

Level TRL Definitions DoD TRL Level Descriptions* DHS TRL Descriptions

Res

earc

h a

nd

Dev

elo

pm

ent

1

Basic principles observed and

reported.

Lowest level of technology readiness. Scientific research begins to be translated into applied research

and development. Examples might include paper studies of a technology's basic properties.

Scientific research begins the first steps toward applied research and development. Examples include

paper studies of a technology‘s basic properties, exploration of a technical phenomenon, and definition of

a technical concept. This level represents the origin of technology readiness.

2

Technology concept and/or

application formulated.

Invention begins. Once basic principles are observed, practical applications can be invented.

Applications are speculative and there may be no proof or detailed analysis to support the assumptions.

Examples are limited to analytic studies.

Once basic principles are observed and proven repeatable, practical applications can be formulated.

Applications are speculative and there may be no proof or detailed analysis to support the assumptions.

Examples are limited to analytic studies, device phenomenology, and experimentation.

3

Analytical and experimental critical

function and/or characteristic proof of

concept.

Active research and development is initiated. This includes analytical studies and laboratory studies to

physically validate analytical predictions of separate elements of the technology. Examples include

components that are not yet integrated or representative.

Active research and development is initiated. This includes analytical and laboratory studies to physically

validate analytical predications of separate elements of the technology. Examples include components

that are not yet integrated.

Tes

tin

g a

nd

Dem

on

stra

tio

n 4

Component and/or breadboard

validation in laboratory environment.

Basic technological components are integrated to establish that they will work together. This is relatively

"low fidelity" compared to the eventual system. Examples include integration of "ad hoc" hardware in

the laboratory.

Basic technological components are integrated to establish that they will work together. Examples

include integration of modules and components in the laboratory.

5

Component and/or breadboard

validation in relevant environment.

Fidelity of breadboard technology increases significantly. The basic technological components are

integrated with reasonably realistic supporting elements so it can be tested in a simulated environment.

Examples include "high fidelity" laboratory integration of components.

The basic technological components are integrated with reasonably realistic supporting elements so it can

be tested in simulated environment. Examples include ―high-fidelity‖ laboratory integration of

components and software.

6

System/subsystem model or prototype

demonstration in a relevant

environment.

Representative model or prototype system, which is well beyond that of TRL 5, is tested in a relevant

environment. Represents a major step up in a technology's demonstrated readiness. Examples include

testing a prototype in a high-fidelity laboratory environment or in simulated operational environment.

Representative model or prototype system is tested in a relevant environment. Represents a major step up

in a technology‘s demonstrated readiness. Examples include testing a prototype in a high-fidelity

laboratory environment or in a simulated operational environment.

7

System prototype demonstration in an

operational environment.

Prototype near, or at, planned operational system. Represents a major step up from TRL 6, requiring

demonstration of an actual system prototype in an operational environment such as an aircraft, vehicle,

or space. Examples include testing the prototype in a test bed aircraft.

Prototype near, or at, planned operational system level. Represents a major step up from TRL 6,

requiring demonstration of an actual system prototype in an operational environment.

Pro

du

ctio

n a

nd

Dep

loy

men

t

8

Actual system completed and

qualified through test and

demonstration.

Technology has been proven to work in its final form and under expected conditions. In almost all cases,

this TRL represents the end of true system development. Examples include developmental test and

evaluation of the system in its intended weapon system to determine if it meets design specifications.

Technology had been proven to work in its final form and under expected operational deployment

conditions. In almost all cases, this TRL represents completion of system development. Examples

include test and evaluation of the system in its intended system configuration and operational

requirement.

9

Actual system proven through

successful mission operations.

Actual application of the technology in its final form and under mission conditions, such as those

encountered in operational test and evaluation. Examples include using the system under operational

mission conditions.

Actual application of the technology in its final form and under mission conditions, in accordance with

the user‘s Concept of Operations.

* From DoD 5000.2-R, Appendix 6, April 5, 2002.


24

Table 6: Manufacturing Readiness Level Definitions and Descriptions

Level MRL Definitions Original DoD MRL Descriptions* AFRL Calculator MRL Descriptions

¥ DHS MRL Descriptions

Res

earc

h a

nd

Dev

elo

pm

ent

1*

Manufacturing

feasibility assessed.

This is the lowest level of manufacturing readiness. The focus is on a top level assessment of feasibility and

manufacturing shortfalls. Basic manufacturing principles are defined and observed. Begin basic research in the

form of studies (i.e. 6.1 funds) to identify producibility and material solutions.

n/a n/a

2*

Manufacturing

concepts defined.

This level is characterized by developing new manufacturing approaches or capabilities. Applied Research

translates basic research into solutions for broadly defined military needs. Begin demonstrating the feasibility

of producing a prototype product/ component with very little support/data available. Typically this level of

readiness is associated with Applied Research (i.e. 6.2 funds) in the S&T environment and includes

identification and study of material and process approaches, including modeling and simulation.

n/a n/a

3

Manufacturing

concepts identified.

This begins the first real demonstration of the manufacturing concepts. This level of readiness is typical of

technologies in the S&T funding categories of 6.2 and 6.3. Within these levels, identification of current

manufacturing concepts or producibility has occurred and is based on laboratory studies. Materials have been

characterized for manufacturability and availability but further evaluation and demonstration is required.

Models have been developed in a lab environment that may possess limited functionality.

Assessment of current manufacturability

concepts or producibility needs for key

breadboard components.

This begins the first real demonstrations of the manufacturing

concepts. Identification of current manufacturing concepts or

producibility has occurred and is based on laboratory studies.

Key components needed for breadboards can be produced.

Tes

tin

g a

nd

Dem

on

stra

tio

n

4

Laboratory

manufacturing process

demonstration.

This level of readiness is typical for S&T Programs in the 6.2 and 6.3 Advanced Development categories, and

acts as an exit criteria for the Concept Refinement (CR) phase approaching a Milestone (MS) A decision. This

level indicates that the technologies are ready for the Technology Development phase of acquisition.

Technologies should have matured to at least TRL 4. At this point, required investments, such as

manufacturing technology development have been identified; processes to ensure manufacturability,

producibility and quality are in place; and manufacturing risks have been identified for prototype build.

Manufacturing cost drivers have also been identified. Producibility assessments of design concepts have been

completed. Design Key Performance Parameters (KPPs) have been identified as well as any special needs for

tooling, facilities, material handling and skills.

Key processes identified and assessed in lab.

Mitigation strategies identified to address

manufacturing/producibility shortfalls. Cost as

an independent variable (CAIV) targets set and

initial cost drivers identified.

Technologies have matured sufficiently to determine required

manufacturing technology development investments;

processes to ensure manufacturability, producibility and

quality are in place; and manufacturing risks have been

identified for prototype build. Key manufacturing processes

have been identified, assessed in the laboratory and potential

manufacturing problems have been documents.

5

Manufacturing process

development.

This level of maturity is typical of the mid-point in the Technology Development phase of acquisition, or in the

case of key technologies, near the mid-point of an ATD program. Technologies should have matured to at least

TRL 5. The Industrial Base has been assessed to identify potential manufacturing sources. A manufacturing

strategy has been refined and integrated with the Risk Management Plan. Identification of enabling/critical

technologies and components is complete. Prototype materials, tooling and test equipment, as well as personnel

skills have been demonstrated on components in a production relevant environment, but many manufacturing

processes and procedures are still in development. Manufacturing technology development efforts have been

initiated or are ongoing. Producibility assessments of key technologies and components are ongoing. A cost

model has been constructed which is based upon a detailed end-to-end value stream map.

Trade studies and lab experiments define key

manufacturing processes and sigma levels

needed to satisfy CAIV targets. Initial

assessment of assembly needs conducted.

Process, tooling, inspection, and test equipment

in development. Significant engineering and

design changes. Quality and reliability levels not

yet established. Tooling and machines

demonstrated in lab. Physical and functional

interfaces have not been completely defined.

The Industrial Base has been assessed to identify potential

manufacturing sources and yields. Identification of

enabling/critical technologies and components is complete and

prototypes have been created. Producibility assessments of

key technologies and components are ongoing and tooling and

machines have been demonstrated in the lab. A

manufacturing strategy has been refined and a risk

management plan has been incorporated. A cost model has

been constructed.

6

Critical manufacturing

processes prototyped.

This MRL is associated with readiness for a MS B decision to initiate an acquisition program by entering into

the System Development and Demonstration (SDD) phase of acquisition. It is normally seen as the level of

manufacturing readiness that denotes completion of S&T development and acceptance into a baseline system

design. Technologies should have matured to at least TRL 6. An initial manufacturing approach has been

developed. The majority of manufacturing processes have been defined and characterized, but there are still

significant engineering and/or design changes. However, preliminary design of critical components has been

completed so that producibility assessments of key technologies can be complete. Prototype materials, tooling

and test equipment, as well as personnel skills have been demonstrated on systems and/or subsystems in a

production relevant environment. Detailed cost analysis includes design trades, and all cost targets have been

allocated. Producibility considerations shape system development plans. Industrial Capabilities Assessment

(ICA) for MS B has been completed and long lead and key supply chain elements are identified.

Critical manufacturing processes prototyped,

targets for improved yield established. Process

and tooling mature. Frequent design changes

still occur. Investments in machining and tooling

identified. Quality and reliability levels

identified. Design to cost goals identified.

An initial manufacturing approach has been developed. The

majority of manufacturing processes have been defined,

characterized and are available, but there are still significant

engineering and/or design changes. Producibility assessments

of key technologies are complete. Production demonstrations

on systems/subsystems in a relevant environment are

complete to include prototype materials, tooling and test

equipment, and personnel skills.

7

Prototype

manufacturing system.

This level of manufacturing readiness is typical for the mid-point of the System Development and

Demonstration (SDD) Phase leading to the Design Readiness Review. Technologies should be maturing to at

least TRL 7. System detailed design activity is underway. Material specifications have been approved and

materials are available to meet the planned pilot line build schedule. Manufacturing processes and procedures

have been demonstrated in a production representative environment. Detailed producibility trade studies and

risk assessments are underway. Cost model have been updated with detailed designs, rolled up to system level

Prototype system built on soft tooling, initial

sigma levels established. Ready for low rate

initial production (LRIP). Design changes

decrease significantly. Process tooling and

inspection and test equipment demonstrated in

production environment. Manufacturing

processes generally well understood. Machines

System detailed design activity is underway. Material

specifications have been approved and materials are available.

Few design changes are occurring. Production planning is

complete and prototypes represent actual form, fit and

function. Production tooling and test equipment design and

development have been initiated. Low-rate initial production


25

Level MRL Definitions Original DoD MRL Descriptions* AFRL Calculator MRL Descriptions

¥ DHS MRL Descriptions

and tracked against allocated targets. Unit cost reduction efforts have been prioritized and are underway.

Supply chain and supplier QA elements have been assessed and long lead procurement plans are in place.

Production tooling and test equipment design and development have been initiated.

and tooling proven. Materials initially

demonstrated in production and manufacturing

process and procedures initially demonstrated.

Design to cost goals validated.

is ready. The cost model has been updated.

Pro

du

ctio

n a

nd

Dep

loy

men

t

8

Manufacturing process

maturity

demonstration.

This level is associated with readiness for a MS C decision, and entry into Low Rate Initial Production (LRIP).

Technologies should have matured to at least TRL 7. At this point, detailed system design is essentially

complete and sufficiently stable to enter low rate production. All materials are available to meet the planned

low rate production schedule. Manufacturing and quality processes and procedures have been proven in a pilot

line environment. Processes are under control such that any known producibility risks pose no significant risk

for low rate production. There is an Engineering Cost Model driven by the stable de-tailed design and this cost

model has been validated. The Industrial Capability Assessment for MS C has been completed and shows that

the supply chain is established and stable.

Manufacturing processes demonstrate

acceptable yield and producibility levels for

pilot line, LRIP, or similar item production. All

design requirements satisfied. Manufacturing

process well understood and controlled to 4-

sigma or appropriate quality level. Minimal

investment in machine and tooling - machines

and tooling should have completed

demonstration in production environment. All

materials are in production and readily available.

Cost estimates <125% cost goals (e.g., design to

cost goals met for LRIP).

Detailed system design is essentially complete and sufficiently

stable to enter low-rate initial production. All materials are

available to meet this planned production schedule.

Manufacturing and quality processes and procedures have

been proven in a pilot line environment. Manufacturing

processes demonstrate acceptable yield and producibility

levels. The Cost Model is stable and is validated.

9

Manufacturing

processes proven.

The system, component or item has been previously produced, is in production, or has successfully achieved

low rate initial production. This level of readiness is normally associated with readiness for entry into Full Rate

Production (FRP). During LRIP all systems engineering/design requirements should have been met such that

there are minimal system changes. Technologies should have matured to at least TRL 8. Major system design

features are stable and have proven in test and evaluation. Materials are available to meet planned rate

production schedules. Manufacturing processes and procedures are established and con-trolled in a low rate

production environment to three-sigma or some other appropriate quality level to meet design key

characteristic tolerances. Production risk monitoring is ongoing. LRIP cost targets met, with learning curves

validated. Actual cost model developed for FRP environment, with impact of continuous improvement.

Manufacturing line operating at desired initial

sigma level. Stable production. Design stable,

few or no design changes. All manufacturing

processes controlled to six-sigma or appropriate

quality level. Affordability issues built into

initial production and evolutionary acquisition

milestones. Cost estimates <110% cost goals or

meet cost goals (e.g., design to cost goals met).

The system, component or item has been previously produced,

is in production, or has successfully achieved low rate initial

production. There are minimal design changes and design

goals have been met. Materials are available to meet planned

rate production schedules. Manufacturing processes and

procedures are established and controlled in the production

environment to meet appropriate quality level. Production risk

monitoring is ongoing. Actual cost model has been developed

with possible evolutionary acquisition.

10*

Full Rate Production

demonstrated and lean

production practices in

place.

The system, component or item is in full rate production. This level of manufacturing is normally associated

with the Production or Sustainment phases of the acquisition life cycle. Technologies should have matured to

at least TRL 9. Engineering/design changes are few and generally limited to quality and cost improvements.

System, components or items are in rate production and meet all engineering, performance, quality and

reliability requirements. All materials, manufacturing processes and procedures, inspection and test equipment

are in production and controlled to six-sigma or some other appropriate quality level. Rate production unit

costs meet goals, and funding is sufficient for production at required rates. Lean practices are well established

and continuous process improvements are ongoing.

n/a n/a

*From DoD Manufacturing Readiness Assessment (MRA) Deskbook, Appendix A, 29 May 2008. ¥

From the AFRL TRL Calculator (ver 2.2) MRL Definitions.


26

Table 7: Program Readiness Levels Definitions and Descriptions*

PRL Definition DHS PRL Description

Res

earc

h a

nd

Dev

elo

pm

ent

1 Identification of basic scientific concepts and Performers. Lowest level of program readiness. Scientific observations have been reported in peer reviewed reports. Sponsor or funding source has been identified, and it is known who will perform the

research and the schedule.

2

Establishment of program with identified customer and

technology.

Program that will support technology development has been identified. Customer has expressed interest in application of technology. Functional requirements documented. Requirements

tracking system initiated. Analytical studies have been reported to scientific journals. Investment strategy is prepared. Capabilities and limitations of researchers and research facilities,

preliminary qualitative risk analysis, and preliminary market analysis are documented.

3

Program risk, requirements, and performance characteristics

and measures are determined.

System performance characteristics and measures are documented (hardware only). Customer representative to work with development team identified. Customer is participating in

requirements generation. Transition agreement including possible TRL for transition has been drafted. Scaling studies have been started. Programmatic risks have been identified.

Programmatic risk mitigation strategies have been documented. Preliminary value analysis has been preformed.

Tes

tin

g a

nd

Dem

on

stra

tio

n

4

Integrated Product Teams and working groups for

developing and transitioning technology are established.

Draft requirements document exists. System performance metrics and S&T exit criteria have been established. Development team requirements have been derived from system requirements.

Software program size estimated (software only). Draft conceptual designs are documented. Initial cost drivers are identified. CAIV targets are set. Scaling documents and diagrams of

technology and Systems Engineering Plan (SEP) are complete. Customer has reviewed draft transition agreement. Integrated Product Team (IPT) is formally established and customer

representative is a member of the IPT. A formal risk management program has been initiated. Project risk management is integrated with project management. Availability dates for any

missing technologies are established.

5

Systems engineering and architecture and end user

involvement are established.

Systems engineering has begun and system architecture document is finalized. End users have been involved in developing physical, functional, and performance characteristics (hardware

only). Engineering drawings, wiring diagrams, and design drawings are complete and finalized (H). Draft SEP addresses integration, test and evaluation, mechanical and electrical interfaces

and expected final performance. Programmatic risk management plan is documented. A configuration management plan has been implemented and documented. Formal inspection of all

modules/components is complete. Draft Test and Evaluation Master Plan (TEMP) is documented. Draft SEP is complete (hardware only). Customer has committed to a transition agreement.

A Business Case for the product is developed. Failure Mode and Effects Analysis (FMEA) is complete (hardware only). Value analysis and life-cycle cost analysis have been performed.

6

Formal requirement documents, final Test and Evaluation

Master Plan, and Systems Engineering Plan are complete.

Draft design drawings are complete (hardware only). Started collection of maintainability, reliability, and supportability data. Design to cost goals have been identified. Final Test and

Evaluation Master Plan (TEMP) and final Systems Engineering Plan (SEP) have been accepted. Technology Transition Agreement is up to date and has been approved by all parties. All

scaling issues are identified and analysis support procurement. A system interface control process has been initiated. End user and developer have reviewed requirements for production

(hardware only). Systems requirements specification is complete. All changes are controlled and documented using configuration management. Final technical report is complete.

Verification, Validation and Accreditation (VV&A) is initiated and the alpha version software has been released (software only). Acquisition program milestones are established. Customer

agrees it can manufacture, deploy, and maintain the system. Formal requirements document is complete.

7 Finalized Verification, Validation and Accreditation of

system.

Verification step of VV&A is complete and beta version software has been released (software only). Draft design drawings are complete and design to cost goals have been validated

(hardware only). Scaling is complete.

Pro

du

ctio

n a

nd

Dep

loy

men

t 8 Training and Test and Evaluation Documentation are

complete.

Software user documentation is complete and under configuration control (software only). Training and maintenance documentation are complete and under configuration control. VV&A

validation step and accreditation step are complete; software works in real world and is authorized for use in intended system (software only). Technical DT&E documentation is complete.

9

Safety and Training is complete. Safety and adverse effects issues have been identified and mitigated. Training Plan, Supportability Plan, and Program Protection Plan are implemented. All programmatic documentation and

OT&E documentation is complete.

*From the DHS S&T RL Calculator.


27

APPENDIX C: GLOSSARY OF TERMS FOR THE DHS S&T

RL CALCULATOR (VER1.1) AND ACRONYMS

For completeness, many terms not used in the DHS S&T RL Calculator (Ver1.1) are included in the

glossary. These include terms used in the AFRL TRL Calculator (ver 2.2), on which the DHS S&T RL

Calculator (Ver1.1) is based, as well as other terms frequently used in discussions of readiness level

assessments.

-A-

Acceptance Document: An acceptance document captures the concurrence of the customer, sponsor, and

other stakeholders that the project has been completed and meets its objectives. The most common form

of formal acceptance document is the customer acceptance document, acknowledging that the project has

been developed as the customer originally requested.

Acceptance Test: Enables the customer to determine whether or not to accept the system or if system is

usable; it involves formally testing the system to determine if its acceptance criteria are satisfied. May

apply to both manufacturing and technology development.

Accreditation: Formal certification that a model or simulation is acceptable to be used for a specific

purpose. One step in Verification, Validation and Accreditation (VV&A).

Acquisition: The conceptualization, initiation, design, development, test, contracting, production,

deployment, logistic support, modification, and disposal of systems/technologies, supplies, or services

(including construction).

Acquisition Plan: Written document that reflects specific actions necessary to execute the approach

established in the acquisition strategy and guiding contractual implementation. Shows how all hardware,

software, and capabilities, along with resources, are to be obtained during the life of the program.

Acquisition Management: Management of all or any of the activities within the broad spectrum of

―acquisition.‖ (also known as Program Management).

Ad Hoc: Formed or used for specific or immediate problems or needs.

Administrative Scalability: The ability for an increasing number of organizations to easily share a single

distributed system.

Advanced Technology Demonstration (ATD): ATD is a demonstration of the maturity and potential of

advanced technologies for enhanced operational capability or cost effectiveness. The results are proof of

technological feasibility and assessment of component operability and producibility.

Advancement Degree of Difficulty (AD2): Description of what is required to advance the level of

maturity of a system or component from one TRL to another. This term in not used in the calculator, but a

useful concept for future discussion.

Algorithm: A step-by-step procedure for solving a problem or accomplishing some end especially by a

computer.

Alpha Release Software: (see Software Release Life-cycle)


28

Analysis of Alternatives (AoA): Evaluation of the operational effectiveness and cost estimates of

alternative material systems to meet a mission need. Assesses the advantages and disadvantages of

alternatives being considered to satisfy requirements, includes sensitivity of each alternative to possible

changes in key assumptions or variables. Assists decision makers in selecting the most cost-effective

material alternative to satisfy a mission need.

Analytical Studies: Separating a whole into its elemental parts or basic principles.

Architecture Diagrams: Descriptions (often graphical) of the tasks and activities, operational elements,

and information flows required to accomplish or support a functional system. (see System Architecture

Diagrams)

Architecture: The structure of a system‘s components, their relationships, and the principles and

guidelines governing their design and evolution over time.

Assessment: The evaluation and interpretation of measurements and other information to provide a basis

for decision making.

Availability: The ratio of operating time (reliability) to downtime (maintainability/ supportability).

-B-

Back-of-the-Envelope: More than a guess, but less than a proof. The phrase is generally used in

mathematics, physics, and engineering. It refers to the practice of quickly jotting down calculations on the

nearest available piece of paper, such as the back of an envelope. Currently, this concept is presented as

the first technology readiness question, ―Do rough calculations support the concept?‖

Balanced Scorecard (BSC): Structured methodology for using performance measurement to set

performance metrics or goals, allocate and prioritize resources, confirm or change current policy or

program directions to meet those goals, and report on the success in meeting those goals. Through the

balanced scorecard, an organization monitors both its current performance (finance, customer satisfaction,

and business process results) and its efforts to improve processes, motivate and educate employees, and

enhance information systems—its ability to learn and improve.

Basic Principles: The foundations of the physical sciences rest upon key concepts and theories, each of

which explain and/or model a particular aspect of the behavior of nature.

Beta Version Software: A beta version is the first version released outside the organization or

community that develops the software, for the purpose of evaluation or real-world testing. The process of

delivering a beta version to the users is called beta release. Beta software generally includes all features,

but may also include known issues and bugs of a less serious variety. (see Software Release Life-cycle)

Brassboard: An experimental device (or group of devices) used to determine feasibility and to develop

technical and operational data. It normally will be a model sufficiently hardened for use outside of

laboratory environments to demonstrate the technical and operational principles of immediate interest. It

may resemble the end item, but is not intended for use as the end item.

Breadboard: A representation of a system/component that can be used to determine concept feasibility

and to develop technical data. Typically configured for laboratory use to demonstrate the technical

principles of immediate interest. May resemble final system/component in function only. Probably has its

http://en.wikipedia.org/wiki/Mathematical_proof


29

origins in the practice in electronics of using a breadboard or other convenient surface on which to attach

the components of a device under development. In hardware, the breadboard is used to determine

feasibility and to develop technical data. In software, it is an iteratively designed piece of software

designed to evolve into a full-fledged system (in contrast to a throwaway prototype, often associated with

a proof-of-concept that is intended to be discarded).

Bug: A fault in a program which causes the program to perform in an unintended or unanticipated

manner.

Business Case: The rationale for an investment as well as certain details of the acquisition (such as

mission need, alternatives analysis, cost, schedule, and performance).

-C-

Capability Gap: A deficiency in current capabilities or an opportunity to provide new capabilities (or

enhance existing capabilities) through the use of new technologies. Identified through Integrated Product

Teams (IPTs). (also known as Capability Shortfalls or Gaps)

Coding: The generation of software source code.

Compatibility: A measure of the ability of two or more systems (or system components) to exchange

information and use the information that has been exchanged. (see Interoperability)

Complete: (1) Having all necessary parts, elements, or steps. (2) Fully carried out.

Component: (1) A single element of technology. The lowest sub-system that provides sufficient

granularity to identify technical risks and opportunities. (2) A constituent part. (see Module)

Concept of Operations (CONOPS): A formal document that describes the user's environment and

processes relative to a new or modified system; defines the users, assumptions, or intent in regard to an

operation or series of operations. It is designed to give an overall picture of the operation. (also known as

Operational Concept)

Conceptual Design: (1) The first phase of design, in which drawings are the dominant tool and product.

Usually, drawings in this phase are simple. (2) An early stage in the life of a project that should result in

the preparation of a document containing a functional program, sketches and outline drawings, a concept

estimate and a set of design standards. (also known as Preliminary Design or Functional Design)

Configuration Control: (see Configuration Management and Interface Control Management)

Configuration Management: Permits the orderly development of a system, component, or configuration

item. The discipline of identifying the configuration of a hardware/software system at each life cycle

phase for the purpose of controlling changes to the configuration and maintaining the integrity and

traceability of the configuration through the entire life cycle. Ensures that designs are traceable to

requirements, that change is controlled and documented, that interfaces are defined and understood, and

that there is consistency between the product and its supporting documentation. Configuration

management provides documentation that describes what is supposed to be produced, what is being

produced, what has been produced, and what modifications have been made to what was produced.


30

Configuration Management Plan: A formal document that establishes formal configuration

management practices in a system development/maintenance program. Describes the technical and

administrative actions taken to identify and document the functional and physical characteristics of a

Configuration Item (CI), to control changes to a CI and its characteristics, and to record and report change

processing and implementation status. It provides a complete audit trail of decisions and design

modifications.

Control Engineering: An engineering discipline dealing with the design and implementation of control

systems. Used extensively in nearly every industry, control engineering is a large sub-field of Systems

Engineering. Control systems theory is an active field of applied mathematics involving the investigation

of solution spaces and the development of new methods for the analysis of the control process. (see

Interface Control)

Cost as an Independent Variable (CAIV): Methodology used to acquire and operate affordable systems

by setting aggressive, achievable Life Cycle Cost (LCC) objectives and managing achievement of these

objectives by trading off performance and schedule, as necessary.

Cost-Benefit Analysis: An analytic technique that compares the costs and benefits of investments,

programs, or policy actions in order to determine which alternative or alternatives maximize net profits.

Net benefits of an alternative are determined by subtracting the present value of costs from the present

value of benefits.

Cost Driver: In activity based costing (which states that products consume activities and activities

consume resources) any factor which causes a change in the cost of an activity. An activity can have more

than one cost driver attached to it. For example, a production activity may have the following associated

cost-drivers: a machine, machine operator(s), floor space occupied, power consumed, and the quantity of

waste and/or rejected output.

Critical Decisions (CD): Major milestones that ensure the project meets applicable mission, design,

security, and safety requirements. Critical Decisions may include the following: there is a need that

cannot be met through other than material; the selected alternative and approach is the optimum solution;

definitive scope, schedule and cost baselines have been developed; the project is ready for

implementation; and the project is ready for turnover or transition to operations. These should be

developed along with technology readiness level entrance and exit criteria.

Critical Design Review (CDR): A multi-disciplined technical review to ensure that a system can proceed

into fabrication, demonstration, and test, and can meet stated performance metrics within cost, schedule,

risk, and other system constraints. Generally this review assesses the system final design as captured in

product specifications for each configuration item in the system‘s product baseline, and ensures that each

configuration item in the product baseline has been captured in the detailed design documentation. This

concept is addressed as ―Have designs been verified through formal inspection process?‖

Critical Technology Elements (CTEs): New or novel technology that a platform or system depends on

to achieve successful development or production or to successfully meet a system operational threshold

requirement. This term in not used in the calculator.

Cross-Technology Effects: Effects of new or developing technologies that could increase/decrease

technological performance and reduce/increase risk to project.

http://www.businessdictionary.com/definition/activity.html

http://www.businessdictionary.com/definition/costing.html

http://www.businessdictionary.com/definition/state.html

http://www.businessdictionary.com/definition/product.html

http://www.businessdictionary.com/definition/activity.html

http://www.businessdictionary.com/definition/resource.html

http://www.businessdictionary.com/definition/factor.html

http://www.businessdictionary.com/definition/cause.html

http://www.businessdictionary.com/definition/change.html

http://www.businessdictionary.com/definition/cost.html

http://www.businessdictionary.com/definition/production.html

http://www.businessdictionary.com/definition/machine.html

http://www.businessdictionary.com/definition/floor.html

http://www.businessdictionary.com/definition/power.html

http://www.businessdictionary.com/definition/quantity.html

http://www.businessdictionary.com/definition/waste.html

http://www.businessdictionary.com/definition/output.html


31

Customer: Individual, project, or organization responsible for accepting the product or for specifying the

requirements for and formally accepts delivery of a new or modified system; and authorizes payment. The

customer is external to the project, but not necessarily external to the organization. The customer may be

a higher level project. Customers are a subset of stakeholders. The customer may or may not be the user.

Customer Requirements: Statements of fact and assumptions that define the expectations of the system

in terms of mission objectives, environment, constraints, and measures of effectiveness and suitability

(MOE/MOS).

-D-

Debugging: Methodical process of finding and reducing defects in a computer program or a piece of

electronic hardware, thus making it behave as expected.

Deployment (Fielding): Activities necessary to deliver, transport, receive, process, assemble, install,

checkout, train, operate, house, store, or field the system to achieve full operational capability.

Design: The activity of going from the product concept to a finished physical item. (see Design Process

and Engineering Design and Manufacturing Design)

Design Acceptance Tests: (see Factory Acceptance Tests)

Design Drawing: (see Engineering Drawing)

Design Freeze: This is a program management term referring to the completion of design cycles. Design

is an iterative process and often occurs in cycles. In addition, designs are often modified due to

manufacturing constraints or conflicting requirements. When a design freeze is declared, there should be

low or no flexibility in changes to the system design for that cycle.

Design Process: Four stages of design: 1) Pre-production design, 2) Design during production, 3) Post-

production Design Feedback for future designs, and 4) Redesign. Specifically, these stages include the

following steps, respectively 1) a statement of design goals; analysis of current design goals; investigating

similar design solutions in the field or related topics; specifying requirements of a design solution for a

product (product design specification) or service; conceptualizing and documenting design solutions; and

presenting design solutions, 2) continuation and improvement of a designed solution; testing a designed

solution, and 3) introducing the designed solution into the environment; summary of process and results,

and 4) any or all stages in the design process repeated (with corrections made) at any time before, during,

or after production.

Design Requirements: The ―build to,‖ ―code to,‖ and ―buy to‖ conditions and/or specifications for

products and ―how to execute‖ specifications for processes expressed in technical data packages and

technical manuals.

Design Solution: (see Design Process)

Design Specification: A specification that establishes precise measurement, tolerances, materials, in

processes and finished product tests, quality control, inspection requirements, and other specific details of

the deliverable.


32

Design To Cost (DTC): Management concept that emphasizes cost effective design (minimizing cost

while achieving performance). DTC activities are those actions which are undertaken to meet cost

objectives through explicit design activities.

Desktop Environment: Term adopted as a way to describe a particular style of user interface provided

by software. Desktop environment describes the style of a user interface, in that it is like a desktop.

However, a program, or set of programs, which simulates a desktop environment, may sometimes

themselves be referred to as a desktop environment, with a desktop environment being considered either a

window manager, or a suite of programs which includes a window manager. In relation to computer

science, desktop environment refers to a type of user interface and to a lesser extent a particular class of

window manager.

Development Specification: States all necessary design requirements of a configuration/development

item in terms of performance. Essential physical constraints are included. Development specifications

state requirements for the development of items below the system level. They specify all of the required

item functional characteristics and the tests required to demonstrate achievement of those characteristics.

Developmental Test & Evaluation (DT&E): Any testing used to assist in the development and

maturation of products, product elements, manufacturing or support processes. Used to verify status of

technical progress, verify that design risks are minimized, substantiate achievement of contract technical

performance, and certify readiness for initial Operational Testing & Evaluation (OT&E). DT&E is meant

to ensure that technical requirements are met (as specified in its specifications). DT&E is normally

accomplished by the developing organization (in contrast to OT&E, which should be accomplished by an

independent agency). In short, the purpose of DT&E is the verification of the system design (whereas the

purpose of OT&E is the validation of the design).

-E-

End User: An individual or organization who operates or interacts directly with the system; one who uses

the services of the system. The end user may or may not be the customer.

Engineering Design: An activity in the production process where a system/technology product is

configured and specific form is decided. The activity is sometimes seen as a late step in the R&D process

and sometimes as an early step in the manufacturing process. (see Design)

Engineering Design Freeze: (see Design Freeze)

Engineering Drawing: Technical drawing used to define requirements for engineered items, and is

usually created in accordance with standardized conventions for layout, nomenclature, interpretation,

appearance (such as typefaces and line styles), size, etc. Its purpose is to accurately and unambiguously

capture all the geometric features of a product or a component. The end goal of an engineering drawing is

to convey all the required information that will allow a manufacturer to produce that component. The

process of producing engineering drawings, and the skill of producing them, is often referred to as

technical drawing. Technical drawing includes the various fields and technologies underpinning

electronics.

Engineering Feasibility: Determination of whether the technology needed for the system exists, how

difficult it will be to build, and the level of experience using that technology. The assessment is based on

an outline design of system requirements in terms of Input, Processes, Output, Fields, Programs, and


33

Procedures. This can be quantified in terms of volumes of data, trends, frequency of updating, etc, in

order to estimate if the new system will perform adequately or not. (also known as Technology and

System Feasibility)

Entry Criteria Checklist: List describing minimum accomplishments required to be completed by each

program prior to entry into the next phase or effort.

Evolutionary Acquisition: Strategy for rapid acquisition of mature technology for the user. An

evolutionary approach delivers capability in increments, recognizing, up front, the need for future

capability improvements. The objective is to balance needs and available capability with resources, and to

put capability into the hands of the user quickly. The success of the strategy depends on consistent and

continuous definition of requirements, and the maturation of technologies that lead to disciplined

development and production of systems that provide increasing capability towards a materiel concept.

Exit Criteria: Program-specific accomplishments that must be demonstrated before a program can

progress further in the current acquisition phase, or transition to the next acquisition phase. Exit criteria

are normally selected to track progress in important technical, schedule, or management risk areas. They

serve as gates that, when successfully passed or exited, demonstrate that the program is on track to

achieve its final program goals and should be allowed to continue with additional activities within an

acquisition phase or be considered for continuation into the next acquisition phase. Exit criteria are some

level of demonstrated performance outcome (e.g., level of engine thrust), the accomplishment of some

process at some level of efficiency (e.g., manufacturing yield), or successful accomplishment of some

event (e.g., first flight), or some other criterion (e.g., establishment of a training program or inclusion of a

particular clause in the follow-on contract) that indicates that aspect of the program is progressing

satisfactorily.

-F-

Factory Acceptance Testing: A test on an initial production model or engineering design model to prove

that the system design meets its requirements. Used to qualify the design and/or the manufacturing

processes. These tests may include performance tests, life tests, environmental tests, and reliability tests.

They are also referred to as Design Acceptance Tests, Functional Qualification Tests, and First Article

Tests.

Failure Mode and Effects Analysis (FMEA): Procedure by which each potential failure mode is

analyzed to determine its effects on the system and then classified according to its severity of the effect of

failures on the system. It is widely used in manufacturing industries in various phases of the product life

cycle. Failure causes are any errors or defects in process, design, or item, especially those that affect the

customer, and can be potential or actual. Effects analysis refers to studying the consequences of those

failures. A FMEA also documents current knowledge and actions about the risks of failures for use in

continuous improvement. FMEA is used during the design stage with an aim to avoid future failures.

Later it is used for process control, before and during ongoing operation of the process. Ideally, FMEA

begins during the earliest conceptual stages of design and continues throughout the life of the product or

service. The purpose of the FMEA is to take actions to eliminate or reduce failures, starting with the

highest-priority ones. It may be used to evaluate risk management priorities for mitigating known threat

vulnerabilities. FMEA helps select remedial actions that reduce cumulative impacts of life cycle risks

from a systems failure (fault). Used to 1) Develop system requirements that minimize the likelihood of

failures. 2) Develop of methods to design and test systems to ensure that the failures have been


34

eliminated. 3) Evaluate of the requirements of the customer to ensure that those do not give rise to

potential failures. 4) Identify certain design characteristics that contribute to failures, and minimize or

eliminate those effects. 5) Track and manage potential risks in the design. This helps avoid the same

failures in future projects. 6) Ensure that any failure that could occur will not injure the customer or

seriously impact a system.

Feasibility Study: (1) Study to determine and document a project's viability or the discipline of planning,

organizing, and managing resources to bring about the successful completion of specific project goals and

objectives. The term is also used to describe the preliminary analysis of an existing system to see if it is

worth upgrading all or a part (also known as feasibility analysis). The term is also used to refer to the

resulting document. The results of this study are used to make a decision whether or not to proceed with

the project. It is an analysis of possible alternative solutions to a problem and a recommendation on the

best alternative. (2) A study of the applicability or desirability of any management or procedural system

from the standpoint of advantages versus disadvantages in any given case.

Feasible: Capable of being done or carried out.

Finalize: (1) To put into a final or finished form. (2) To give final approval.

First Article Tests: (see Factory Acceptance Tests)

Form, Fit and Function: (1) Physical, functional, and performance characteristics or specifications that

uniquely identify a component or device and determine its interchangeability in a system. (2) Refers to

system packaging and function and how close the system is to its final configuration.

Formal: Following set or prescribed rules.

Full Rate Production (FRP): Contracting for economic production quantities following stabilization of

the system design and validation of the production process.

Function: (1) A set of related activities that are part of a process; often known as a sub-process within a

process. (2) The action for which a thing is specially fitted or used or for which a thing exists.

Functional Analysis: The examination of a function to identify all sub functions necessary to the

accomplishment of that function, and the identification of functional relationships and interfaces and the

capturing of those relationships in a functional architecture. The subsequent flow-down of upper-level

performance requirements to lower-level sub functions.

Functional Design: (see Conceptual Design)

Functional Qualification Tests: (see Factory Acceptance Tests)

Functional Requirements: The necessary tasks, actions or activities that the system or component must

accomplish. Functional requirements identified in requirements analysis will be used as the top-level

functions for functional analysis.

Functional Requirements Document: A formal document of the functional requirements of a system;

the baseline for system validation.


35

Functionality: (1) Describe the capabilities of a technology/system or how it manages transactions based

on various settings or parameters. (2) A set of attributes that bear on the existence of a set of functions

and their specified properties. The functions are those that satisfy stated or implied needs.

-G-

Geographic Scalability: The ability to maintain performance, usefulness, or usability regardless of

expansion from concentration in a local area to a more distributed geographic pattern. (see Scalability)

-H-

Hardware: (1) Physical components of a system or technology. (2) Products made of material and their

components (mechanical, electrical, electronic, hydraulic, pneumatic). Computer software and technical

documentation are excluded.

High-fidelity Laboratory Environment: Testing with equipment that can simulate and validate all

system specifications within a laboratory setting. (see Laboratory Environment)

High-fidelity: Characterized by minimal distortion. Addresses form, fit, and function.

-I-

Inspection: Examination of the item and associated descriptive documentation which compares

appropriate characteristics with predetermined standards to determine conformance to requirements

without the use of special laboratory equipment or procedures.

Integrated Product Team (IPT): Team composed of representatives from appropriate functional

disciplines working together to identify capability gaps, build successful products, identify and resolve

issues, and make sound and timely recommendations to facilitate decision making.

Integration Readiness Level (IRL): A systematic measurement of the interfacing of compatible

interactions for various technologies and the consistent comparison of the maturity between integration

points (TRLs). This term has been included for future discussion.

Integration: The systematic, structured and progressive activity of testing, validating, and verifying the

interactions between components up to the overall system.

Interface: (1) Functional and physical characteristics required to exist at a common boundary or

connection between persons, between systems, or between persons and systems. (2) A system external to

the system being analyzed that provides a common boundary or service that is necessary for the other

system to perform its mission in an undegraded mode (e.g., a system that supplies power, cooling,

heating, air services, or input signals).

Interface Control Process or Management: (1) Process used to ensure compatibility among

components being designed and fabricated. (2) All interfaces between co-functioning items need to be

identified and documented so that their integrity may be maintained through a disciplined configuration

control process. In some cases a formal interface management process must be employed in order to

define and document the interface. (3) The purpose of interface management is to document that the

detailed design of each of the co-functioning items contains the necessary information to assure that the

items, when individually designed and produced will work together, and document if items needs to be


36

changed for any reason, its performance, functional or physical attributes, that are involved in the

interface, act as constraints on the design change. During development, part of the contractor‘s design

effort is to arrive at and document external interface agreements, as well as to identify, define, control and

integrate all lower-level (i.e., detailed design) interfaces. Possible interfaces include: between systems and

within a system; external interfaces with other systems; internal interfaces between components of the

system; personnel, non-developmental items, facilities; and interfaces between acquiring activities and

supplying activities.

Interface Design: Interface design and its specification are concerned with assuring that the pieces of a

system connect and inter-operate with other parts of the system and with external systems as necessary.

Interface design also includes assuring that system interfaces be able to accept new features, including

mechanical, electrical, and logical interfaces, including reserved wires, plug-space, command codes and

bits in communication protocols. This is known as extensibility. Human-Computer Interaction (HCI) or

Human-Machine Interface (HMI) is another aspect of interface design, and is a critical aspect of modern

Systems Engineering. (see Interface Control Process)

Interoperability: (1) The ability of systems, units, or forces to provide services to and accept services

from other systems, units, or forces and to make use the services, units, or forces and to use the services

so exchanged to enable them to operate effectively together. (2) The condition achieved among

communications-electronics systems or items of communications-electronics equipment when

information or services can be exchanged directly and satisfactorily between them and/or their users. The

degree of interoperability should be defined when referring to specific cases. (3) A measure of the ability

of two or more systems (or system components) to exchange information and use the information that has

been exchanged. (see Compatibility)

Investment Strategy: A set of rules, behaviors or procedures, designed to guide the selection of an

investment. Usually the strategy will be designed around the investor's risk-return tradeoff: some

investors will prefer to maximize expected returns by investing in risky assets, others will prefer to

minimize risk, but most will select a strategy somewhere in between.


37

-J, K-

Key Decision Point (KDP): Major decision points that separate the phases of development. These are

points when decision authorities determine the readiness of a program/project to progress to the next

phase of the life cycle (or to the next TRL). This term is not included in the calculator; however, it is

included here for future discussion. (see Milestone)

-L-

Laboratory Environment: Testing with equipment that can simulate system specifications within a

laboratory setting.

Life Cycle: The period that starts with the initial product specification and ends with the withdrawal of

the product from the marketplace. A product life cycle is characterized by certain defined stages,

including research, development, introduction, maturity, decline, and abandonment.

Life Cycle Cost (LCC): The total cost of acquisition and ownership of a system over its life. It includes

the cost of development, acquisition, operations, and support (to include manpower), and where

applicable, disposal.

Live Simulations: Live simulations are simulated operations with real operators and real equipment.

Examples are operational tests and initial production run with soft tooling. (see Operational Tests and

Soft Tooling)

Load Scalability: The ability for a distributed system to easily expand and contract its resource pool to

accommodate heavier or lighter loads. Alternatively, the ease with which a system or component can be

modified, added, or removed, to accommodate changing load.

Low-Fidelity: A representative of the component or system that has limited ability to provide anything

but first order information about the end product. Low-fidelity assessments are used to provide trend

analysis.

Low Rate Initial Production (LRIP): LRIP ensures efficient manufacturing capability. The purpose of

this effort is to establish an initial production base for the system, to permit an orderly ramp-up sufficient

to lead to a smooth transition to Full Rate Production (FRP), and to provide production representative

articles for Initial Operational Test and Evaluation (OT&E).

-M-

Maintainability: The ability of an item to be retained in, or restored to, a specified condition when

maintenance is performed by personnel having specified skill levels, using prescribed procedures and

resources, at each prescribed level of maintenance and repair. The ease with which a software system (or

system component) can be modified to correct faults, improve performance, or other attributes, or adapt to

a changed environment.

Manufacturability: (1) The characteristics considered in the design cycle that focus on process

capabilities, machine or facility flexibility, and the overall ability to consistently produce at the required

quality level instead of product elegance. (2) The extent to which a new product concept or prototype is

figured to be capable of effective and efficient manufacturing by available resources. Asked at time of


38

pre-R&D screening, and again prior to authorizing production. Is answered today by computer-aided-

manufacturing software. (also called Producibility)

Manufacturing Design: The activity of determining the manufacturing process that will be used to make

a new product. (see Manufacturing Process)

Manufacturing Process: (1) Manufacturing is the application of tools and a processing medium to the

transformation of raw materials into finished goods for sale. This effort includes all intermediate

processes required for the production and integration of a product's components. (2) The producing of

goods or wares by manual labor or machinery, often on a large scale and with division of labor.

Manufacturing Questions: These questions apply to the manufacturing process of the technology in

question.

Market Analysis: A documented investigation of a market that is used to inform planning activities

particularly around decision of inventory, purchase, work force expansion/contraction, facility expansion,

and purchases of capital equipment.

Measures of Effectiveness (MOE): A measure of operational success. MOEs are used in the analysis of

alternatives, requirements document and test and evaluation. A meaningful MOE must be quantifiable

and a measure to what degree the real objective is achieved.

Metric: (1) A standard of measurement. (2) Application and system designers use metrics to facilitate

performance monitoring and troubleshooting. (3) Software metrics are the statistics describing the

structure or content of a program. A metric should be a real objective measurement (e.g., number of bugs

per lines of code).

Milestone: (1) The point at which a recommendation is made and approval sought regarding starting or

continuing an acquisition program, i.e., proceeding to the next phase. (2) In the context of scheduling, a

specific definable accomplishment in the contract network that is recognizable at a particular point in

time. Milestones have zero duration, do not consume resources, and have defined entry and exit criteria.

A milestone may mark the start and/or finish of an interim step, event, or program phase.

Mission: The objective or task, together with the purpose, which clearly indicates the action to be taken.

Mission Need: A statement of operational capability required to perform an assigned mission or to

correct a deficiency in existing capability to perform the mission.

Mitigation Strategies: (1) Appropriate approach to prevent or handle risk causes that could result in

significant harm or loss. (2) Planning to avoid or overcome adverse impacts to project.

Model: (1) A functional form of a system generally reduced in scale, near or at operational specification.

(2) A simplified representation or abstraction (i.e., of a process, activity, or system) intended to explain its

behavior.

Modeling & Simulation (M&S): Provides analytical tools with applications in physics and engineering

for components, components, and systems; one-on-one system performance, few-on-few system, and

force-on-force combat utility and effectiveness. Simulation capabilities include live, virtual, and

constructive. Ownership and operation of M&S depends on use application and include organizations

responsible for requirements determination, technology research, system development, system and unit


39

training, independent analysis as well as test and evaluation. The system evaluator will determine

availability of and the need for M&S analyses during development of the SEP. Evaluators must ensure the

M&S used are accredited before use.

Module: (1) A separate and distinct unit of hardware or software that may be used as a component in a

system. (2) An independently compliable software component made up of one or more procedures or

routines or a combination of procedures and routines.

-N, O-

Operational Concept: Identifies operational need definition, operational environments, constraints on

system, performance requirements, user and maintainer roles, structure of the organizations that will

operate, support and maintain the system, and interfaces with other systems. (see Concept of Operations).

Operational Environment: Environment that addresses all of the operational requirements and

specifications required of the final system or technology, including platform and packaging.

Operational Requirements: (1) A system capability or characteristic required to accomplish approved

mission needs. Operational requirements are typically performance parameters, but they may also be

derived from cost and schedule. For each parameter, an objective and threshold value must also be

established. (2) Define the basic need and, at a minimum, answer the following questions: Operational

distribution or deployment: Where will the system be used? Mission profile or scenario: How will the

system accomplish its mission objective? Performance and related parameters: What are the critical

system parameters to accomplish the mission? Functional: How are the various system components to be

used? Effectiveness requirements: How effective or efficient must the system be in performing its

mission? Operational life cycle: How long will the system be in use by the user? Environment: What

environments will the system be expected to operate in an effective manner?

Operational Requirements Analysis: Operational requirements are User- or user representative-

generated validated needs developed to address mission area deficiencies, evolving threats, emerging

technologies or weapon system cost improvements.

Operational Requirements Document (ORD): A statement containing performance and related

operational parameters for the proposed concept or system. As with all program-related documents, the

ORD will evolve in its completeness and accuracy as program progresses.

Operational Test & Evaluation (OT&E): (1) Field test, under realistic conditions, of any item (or key

component) for the purpose of determining the effectiveness and suitability for use by end users; and the

evaluation of the results of such tests. (2) The test and evaluation of a system prototype to assure that it

meets its operational requirements (as specified in the ORD) when operated by typical users. OT&E

should be accomplished by an agency which is independent of the development organization which

developed the system. In short, the purpose of OT&E is the validation of the system design (whereas the

purpose of DT&E is the verification of the design by the development organization).

Operational Test Plan: Documents specific operational test scenarios, objectives, Measures of

Effectiveness (MOE), threat simulation, detailed resources, known test limitations and the methods for

gathering, reducing, and analyzing data.


40

Operations Manual: A formal document that provides a detailed operational description of the system

and its interfaces.

-P-

Paper Studies: Scientific publications, white papers, etc., that provide basis for scientific exploration of

new technology applications.

Performance Characteristics/Levels/Attributes: (1) Those operational and support characteristics of

the system that allow it to effectively and efficiently perform its assigned mission over time. The support

characteristics of the system include both supportability aspects of the design and the support elements

necessary for system operation. (2) Comparison between the outputs obtained and the set goals for

outputs of a process. (3) A quantitative measure characterizing a physical or functional attribute relating

to the execution of an operation or function. Performance attributes include quantity (how many or how

much), quality (how well), coverage (how much area, how far), timeliness (how responsive, how

frequent), and readiness (availability, mission/operational readiness). Performance is an attribute for all

systems, people, products, and processes including those for development, production, verification,

deployment, operations, support, training, and disposal. Thus, supportability parameters, manufacturing

process variability, reliability, and so forth, are all performance measures.

Performance Predictions: Estimations of the performance characteristics.

Performance Requirements: The extent to which a function must be executed; generally measured in

terms of quantity, quality, coverage, timeliness or readiness. During requirements analysis, performance

(how well does it have to be done) requirements will be interactively developed across all identified

functions based on system life cycle factors, and characterized in terms of the degree of certainty in their

estimate, the degree of criticality to system success, and their relationship to other requirements.

Physical Work Breakdown Structure: Work breakdown structure for development of a technology.

This is a technology-related aspect, slightly different than the Programmatic Work Breakdown Structure.

(see Work Breakdown Structure)

Pilot Production Line: Production line to test new manufacturing methods and procedures.

Preliminary Design: (see Conceptual Design)

Pre-production Form (or Preproduction Prototype): An article in final form employing standard parts,

representative of articles to be produced subsequently in a production line.

Procurement: Acquisition of property, goods or services through purchase, hire, lease, rental or

exchange

Producibility: The relative ease of manufacturing an item or system. This relative ease is governed by

the characteristics and features of a design that enables economical fabrication, assembly, inspection, and

testing using available manufacturing techniques. Includes some or all of the activities in the six sigma

methodology.

Product Breakdown Structure (PBS): A hierarchical breakdown of the hardware and software products

of the program/project.


41

Product Design Specification: Statement of what a not-yet-designed product is intended to do. Its aim is

to ensure that the subsequent design and development of a product meets the needs of the user. This term

is not included in the calculator, but it is included here as an example of documentation for future

discussion.

Production Environment: Environment where production process occurs.

Production Planning: The broad range of activities initiated early in the acquisition process, and

continued through a production decision, to ensure an orderly transition from development to cost-

effective rate production or construction.

Production Process: The process of converting raw materials by fabrication into required material. It

includes the functions of production-scheduling, inspection, Quality Control (QC), and related processes.

Program Cost Analysis: Program cost is the total of all expenditures, in any appropriation and fund,

directly related to definition, design, development, and deployment. Program cost does not include

Operations and Support (O&S) costs incurred at an individual site after operational cutover of any

increment at that site, even though other sites may exist that have not yet completed deployment.

Program Management: Process of allocating responsibility for planning, organizing, staffing,

controlling, and leading the combined efforts of participating/assigned personnel and organizations, for

the management of a specific program or project, throughout the system life cycle.

Program Protection Plan (PPP): Program protection is the safeguarding of information anywhere in the

acquisition process, to include the technologies being developed, the support systems (e.g., test and

simulation equipment), and research data. The PPP document is used to coordinate and integrate all

protection efforts designed to deny access to anyone not authorized or not having a need-to-know and

prevent inadvertent disclosure of leading edge technology to foreign interests.

Programmatic: Pertains to the cost, schedule and performance characteristics of an acquisition program.

Programmatic Questions: For the purposes of the TRL calculator, this refers to the project management

aspects of developing a technology. As much as possible, these questions should be answered with

documentation. These questions generally follow a systems engineering approach to project management.

Programmatic Work Breakdown Structure: (see Work Breakdown Structure)

Proof-of-Concept: (1) Analytical and experimental demonstration of hardware/software concepts that

may or may not be incorporated into subsequent development and/or operational units. (2) A program that

defines what will be proven and determines both the criteria and methods for the proof-of-concept. Once

the proof-of-concept is demonstrated, a prototype program may be initiated.

Prototype: (1) A physical or virtual model used to evaluate the technical or manufacturing feasibility or

utility of a particular technology or process, concept, end item, or system. (2) An ensemble that is suitable

for the evaluation of design, performance, and production potential and is not required to exhibit all the

properties of the final system. Prototypes are not installed in the field, nor are they available for

operational use (except perhaps as a stopgap pending the availability of production units). They are

generally maintained only long enough to establish technical feasibility.


42

Pseudocode: Program code unrelated to the hardware of a particular computer and requiring conversion

to the code used by the computer before the program can be used.

-Q-

Qualitative: Referring only to the characteristics of something being described, rather than exact

numerical measurement.

Quality: The composite of materiel attributes including performance features and characteristics of a

production or service to satisfy a customer's given need.

Quality Assurance: (1) A planned and systematic pattern of all actions necessary to provide confidence

that adequate technical requirements are established, that products and services conform to established

technical requirements, and that satisfactory performance is achieved. (2) A discipline used by program

management to objectively monitor, control, and gain visibility into the development or maintenance

process. This term is not explicitly utilized in the calculator, but is included for future discussion.

Quality Assurance Plan: (1) A planned and systematic means for assuring management that defined

standards, practices, procedures, and methods of the process are applied. (2) A formal plan to ensure that

delivered products satisfy contractual agreements, meet or exceed quality standards, and comply with

approved systems development or maintenance processes. This term is not explicitly utilized in the

calculator, but is an example of a document that would support the programmatic aspects of the

technology development.

Quality Control (QC): The system or procedure used to check product quality throughout the acquisition

process.

-R-

Readiness: Measure of the suitability of a product for use within a larger system (e.g., development of a

management information system or sustainment of a deployed tactical information processing system).

Reflects some measure of the risks of using it in the larger system: higher readiness denotes lower risk;

lower readiness, higher risk.

Realistic: Relating to the representation of objects, actions or conditions as they actually are.

Relevant Environment: (1) Not the actual operational environment, but an environment that mostly

addresses operational requirements and specifications required of the final system or technology. (2) A

testing environment that simulates the key aspects of the operational environment.

Reliability: (1) Ability of a system and its parts to perform its mission without failure, degradation, or

demand on the support system. (2) The measure of the degree to which a system ensures mission success

by functioning properly over its intended life. It has a low and acceptable probability of failure, achieved

through simplicity, proper design, and proper application of reliable parts and materials. In addition to

long life, a reliable system is robust and fault tolerant.

Requirements: (1) An essential attribute or characteristic of a system. It is a condition or capability that

must be met or passed by a system to satisfy a contract, standard, specification, or other formally imposed

document or need. (2) A singular documented need of what a particular product or service should be or

do. It is a statement that identifies a necessary attribute, capability, characteristic, or quality of a system in


43

order for it to have value and utility to a user. Sets of requirements are used as inputs into the design

stages of product development. Requirements show what elements and functions are necessary for the

particular project. Requirements can be categorized in several ways: Customer, Functional, Operational,

Performance, and Design requirements. (3) The need of an operational user. Progressively evolves to

system-specific performance requirements.

Requirements Analysis: Once the general requirements are gleaned, an analysis of the scope of the

development should be determined and clearly stated. This is often called a scope document. Certain

functionality may be out of scope of the project as a function of cost or as a result of unclear requirements

at the start of development.

Requirements Document: (1) A singular documented need of what a particular product or service should

be or do. (2) A statement that identifies necessary attributes, capabilities, characteristics, or qualities of a

system or technology in order for it to have value and utility to a user. (3) Sets of requirements are used as

inputs into the design stages of product development.

Requirements Management: (see Requirements Tracking System)

Requirements Traceability: (see Requirements Tracking System)

Requirements Traceability Matrix: (1) Provides a method for tracking the functional requirements and

their implementation through the development process. (2) Cross matrix that traces the requirements

through each stage of the requirements gathering process. High level concepts will be matched to scope

items which will map to individual requirements which will map to corresponding functions. This matrix

should also take into account any changes in scope during the life of the project. At the end of a project,

this matrix should show each function built into a system, its source and the reason that any stated

requirements may not have been delivered.

Requirements Tracking System: Includes two facets: requirements management and requirements

traceability. 1) Requirements management: management of all requirements received by or generated by

the project, including both technical and nontechnical requirements as well as those requirements levied

on the project by the organization. 2) Requirements traceability: evidence of an association between a

requirement and its source requirement, its implementation, and its verification. (see Requirements

Traceability Matrix)

Research: Includes all scientific study and experimentation directed toward increasing knowledge and

understanding in those fields of the physical, engineering, environmental, and life sciences related to long

term national security needs.

Risk: (1) An uncertain event that may impact on the project in a positive or negative way. The gap

between the maturity of the technology and the product‘s requirements represents the risks or unknowns

about the technology. (2) A measure of the exposure to which an organization may be subject. This is a

combination of the likelihood of a business disruption occurring and the possible loss that may result

from such business disruption. (3) A measure of the inability to achieve program objectives within

defined cost and schedule constraints. Risk is associated with all aspects of the program, e.g., technology,

design processes, Work Breakdown Structure (WBS) elements, etc. It has two components: (1) The

probability of failing to achieve a particular outcome; and (2) The consequences of failing to achieve that

outcome.

http://en.wikipedia.org/wiki/Requirements_analysis


44

Risk Analysis: (1) The process of identifying areas of risk; the probability of the risk occurring, and the

seriousness of its occurrence. (2) A detailed examination of each identified program risk which refines the

description of the risk, isolates the cause, and determines the impact of the program risk in terms of its

probability of occurrence, its consequences, and its relationship to other risk areas or processes.

Risk Assessment: (1) A study of vulnerabilities, threats, likelihood, loss or impact, and theoretical

effectiveness of security measures. (2) The determination of quantitative or qualitative value of risk

related to a concrete situation and a recognized threat. Quantitative risk assessment requires calculations

of two components of risk R, the magnitude of the potential loss L, and the probability p that the loss will

occur. (3) The process of identifying program risks within risk areas and critical technical processes,

analyzing them for their consequences and probabilities of occurrence, and prioritizing them for handling.

Risk Management: (1) Management of the risks to assets through the identification, selection and use of

countermeasures, justified by the identified risks in terms of their potential impact upon services if failure

occurs and the reduction of those risks to an acceptable level. (2) All plans and actions taken to identify,

assess, mitigate, and continuously track, control, and document program risks. (see Risk Waterfall

Analysis)

Risk Management Plan: A document which records the results of the risk planning process. A formal

document that identifies program risks and specifies the plans to reduce these risks.

Risk Waterfall Analysis or Diagrams: (1) Defined as taking snapshots of forecasts or plans at different

points in time and comparing them in order to gain visibility into changes over time - what changed, how

much, and when? (2) Visualizations that link existing efforts to both known risk resolutions and

technology maturity advancement, so that the overall uncertainty reduction rate of the plan to reduce the

risk and uncertainty can be predicted and shown. These risk waterfall diagrams serve as both a check on

the system as plans are executed to see that uncertainty is being reduced as expected, and as a feedback

mechanism to assure the technical and project managers that the plan has been developed in a way to

reduce uncertainty as fast as possible. (3) Waterfall diagrams are used to incorporate risk mitigation

activities in the standard project management procedures. They differ in this way from a risk register,

which tracks and monitors risks separately from other project activities. Extracted from the context of a

project network diagram, with the project risks qualitatively tracked over time and divided into three

color-coded zones of severity. The red zone corresponds to high or unacceptable risks; the yellow zone

corresponds to moderate but unacceptable risks; and the green zone corresponds to low, acceptable risks.

Rough Calculations: (see Back-of-the-Envelope)

-S-

Safety: The application of engineering and management principles, criteria, and techniques to optimize

safety within the constraints of operational effectiveness, time, and cost throughout all phases of the

system life cycle.

Safety Release: A formal document issued to a user test organization before any hands-on use or

maintenance by personnel. The safety release indicates the system is safe for use and maintenance by

typical user personnel and describes the system safety analyses. Operational limits and precautions are

included. The test agency uses the data to integrate safety into test controls and procedures and to

determine if the test objectives can be met within these limits.


45

Safety/Health Verification: The development of data used to evaluate the safety and health features of a

system to determine its acceptability.

Scalability: (1) Proportion or measurement. (2) As a property of systems, requires defining the specific

requirements for scalability on those dimensions which are deemed important. For example, a system

whose performance improves after adding hardware, proportionally to the capacity added, is said to be a

scalable system. An algorithm, design, networking protocol, program, or other system is said to scale if it

is suitably efficient and practical when applied to large situations (e.g., a large input data set or large

number of participating nodes in the case of a distributed system). If the design fails when the quantity

increases then it does not scale. (3) In software engineering, scalability is a desirable property of a system,

a network, or a process, which indicates its ability to either handle growing amounts of work in a graceful

manner, or to be readily enlarged. For example, it can refer to the capability of a system to increase total

throughput under an increased load when resources (typically hardware) are added. (4) Relationship of an

object to another object. (5) Relationship of the size of a drawing to the size of the actual object. When an

engineering drawing is drawn to scale, all parts of the drawing relate to this ratio. (6) Scalability can be

measured in various dimensions, such as: Load scalability, Geographic scalability, and Administrative

scalability.

Scientific Methodology: Bodies of techniques for investigating phenomena, acquiring new knowledge,

or correcting and integrating previous knowledge. To be termed scientific, a method of inquiry must be

based on gathering observable, empirical, and measurable evidence subject to specific principles of

reasoning. A scientific method consists of the collection of data through observation and experimentation,

and the formulation and testing of hypotheses. Although procedures vary from one field of inquiry to

another, identifiable features distinguish scientific inquiry from other methodologies of knowledge.

Scientific researchers propose hypotheses as explanations of phenomena, and design experimental studies

to test these hypotheses. These steps must be repeatable in order to dependably predict any future results.

Theories that encompass wider domains of inquiry may bind many hypotheses together in a coherent

structure. This in turn may help form new hypotheses or place groups of hypotheses into context. Among

other facets shared by the various fields of inquiry is the conviction that the process be objective to reduce

a biased interpretation of the results. Another basic expectation is to document, archive, and share all data

and methodology so they are available for careful scrutiny by other scientists, thereby allowing other

researchers the opportunity to verify results by attempting to reproduce them. This practice, called full

disclosure, also allows statistical measures of the reliability of these data to be established. Steps include:

Define the question, Gather information and resources (observe), Form hypothesis, Perform experiment

and collect data, Analyze data, Interpret data and draw conclusions that serve as a starting point for new

hypothesis, Publish results, Retest (frequently done by other scientists).

Scientific Report: Document that describes the process, progress, or results of technical or scientific

research, or the state of a technical or scientific research problem.

Sigma Levels: (1) Measures of the quality control of a process. (2) In the quality control of

manufacturing processes the standard deviation of a performance parameter of production items. To say

that a manufacturing process is ―controlled to the 4-sigma level,‖ for instance, is to say that the statistical

distributions of key performance parameters measured on production units are clustered about a mean

value at least 4 sigma units away from the threshold for rejection. It is common practice to assume that

performance parameters are normally distributed. In this case, ―controlled to the 4-sigma level‖ would

mean that the probability of rejection would be less than 32 parts per million (if only one-sided deviations

in performance are considered). To hedge against the possibility that performance of a process will drift

http://en.wikipedia.org/wiki/Technique

http://en.wikipedia.org/wiki/Phenomenon

http://en.wikipedia.org/wiki/Knowledge

http://en.wikipedia.org/wiki/Observable

http://en.wikipedia.org/wiki/Empirical

http://en.wikipedia.org/wiki/Measure

http://en.wikipedia.org/wiki/Evidence

http://en.wikipedia.org/wiki/Reasoning

http://en.wikipedia.org/wiki/Observation

http://en.wikipedia.org/wiki/Hypotheses

http://en.wikipedia.org/wiki/Fields_of_science

http://en.wikipedia.org/wiki/Hypothesis

http://en.wikipedia.org/wiki/Experiment

http://en.wikipedia.org/wiki/Research

http://en.wikipedia.org/wiki/Theory#Science

http://en.wikipedia.org/wiki/Objectivity_(science)

http://en.wikipedia.org/wiki/Bias

http://en.wikipedia.org/wiki/Scientific_data_archiving

http://en.wikipedia.org/wiki/Data_sharing_(Science)

http://en.wikipedia.org/wiki/Methodology

http://en.wikipedia.org/wiki/Reproducibility

http://en.wikipedia.org/wiki/Reliability_(statistics)


46

over time after performance measures are made, a degradation of, typically, 1.5 sigma units is frequently

subtracted from the measured value of sigma, so that performance in tests at the 4-sigma level would lead

only to claims that the failure rate is less than that corresponding to 4 – 1.5 = 2.5 sigma‘s, or a claimed

rejection rate of less than 6210 per million. Similarly, a claim that the manufacturing process is

―controlled to the 6-sigma level‖ is frequently interpreted to mean that the failure rate is less than that

corresponding to a one-sided normal distribution at 6 – 1.4 = 4.5 sigma units from the mean, or less than

3.4 per million.

Simplified Environment: Reduced to basic essentials of operational environment; diminished in scope

or complexity.

Simulated Operational Environment: (1) A real environment that can simulate all the operational

requirements and specifications required of the final system but not necessarily concurrently. (2) A

simulated environment that allows for testing of a virtual prototype. Used in either case to determine

whether a developmental system meets the operational requirements and specifications of the final

system. (see Relevant Environment)

Simulation: Method for implementing a model. It is the process of conducting experiments with a model

for the purpose of understanding the behavior of the system modeled under selected conditions or of

evaluating various strategies for the operation of the system within the limits imposed by developmental

or operational criteria. Simulation may include the use of analog or digital devices, laboratory models, or

"test bed" sites.

Soft Tooling: Method for low volume production when the justification cannot be made for investment in

high volume production.

Software: (1) General term used to describe a collection of computer programs, procedures and

documentation that perform some tasks on a computer system. There are several types of software,

including: application software (word processors), system software (operating systems that interface with

hardware to provide the services for application software), and middleware which controls and

coordinates distributed systems. Software includes websites, programs, video games etc., which are coded

by programming languages like C, C++, etc. (2) Software is sometimes used in a broader context to mean

anything which is not hardware but which is used with hardware, such as film, tapes and records.

Software Architecture: The architecture of a software system refers to an abstract representation of that

system. Architecture is concerned with making sure the software system will meet the requirements of the

product, as well as ensuring that future requirements can be addressed. The architecture also addresses

interfaces between the software system and other software products, as well as the underlying hardware or

the host operating system.

Software Construction or Computer Programming: The process of writing, testing,

debugging/troubleshooting, and maintaining the source code of computer programs. This source code is

written in a programming language. The code may be a modification of an existing source or something

completely new. The purpose of programming is to create a program that exhibits a certain desired

behavior (customization). The process of writing source code requires expertise in many different

subjects, including knowledge of the application domain, specialized algorithms and formal logic. (also

known as Programming or Coding)

http://en.wikipedia.org/wiki/Software_architecture


47

Software Deployment: Starts after the code is appropriately tested, is approved for release and is sold or

otherwise distributed into a production environment. Deployment should include training and support for

software use.

Software Design: Process of problem-solving and planning for a software solution. After the purpose and

specifications of software are determined, software developers will design or employ designers to develop

a plan for a solution. It includes low-level component and algorithm implementation issues as well as the

architectural view.

Software Development Process: A structure imposed on the development of a software product. There

are several models for such processes, each describing approaches to a variety of tasks or activities that

take place during the process, including software requirements analysis, software architecture, software

design, implementation, and testing, and software deployment and maintenance. (also known as Software

Life Cycle and Software Process)

Software Domain Analysis: Generally the first step in attempting to design a new piece of software,

whether it be an addition to an existing software, a new application, a new component or a whole new

system. It is an important prelude to extracting and gathering the requirements. The more knowledgeable

developers are about the domain, the less work required. Domain analysis will facilitate a better

understanding of what is being said by experts.

Software Engineering: The application of a systematic, disciplined, quantifiable approach to the

development, operation, and maintenance of software. It encompasses techniques and procedures, often

regulated by a software development process, with the purpose of improving the reliability and

maintainability of software systems. The effort is necessitated by the potential complexity of those

systems, which may contain millions of lines of code. The discipline of software engineering includes

knowledge, tools, and methods for software requirements analysis, software design, software

construction, software testing, and software maintenance.

Software Implementation: Part of the software development process where software engineers actually

program the code for the project.

Software Maintenance: (1) Maintaining and enhancing software to cope with newly discovered

problems or new requirements can take far more time than the initial development of the software. It may

be necessary to add code that does not fit the original design to correct an unforeseen problem or it may

be that a customer is requesting more functionality and code can be added to accommodate their requests.

(2) In software engineering, it is the modification of a software product after delivery to correct faults, to

improve performance or other attributes, or to adapt the product to a modified environment.

Software Release Life Cycle: Composed of stages that describe the stability of a piece of software and

the amount of development it requires before final release. Each major version of a product usually goes

through a stage when new features are added, or the alpha stage; a stage when it is being actively

debugged, or the beta stage; and finally a stage when all important bugs have been removed, or the stable

stage.

Software Requirements Analysis: The most important task in creating a software product is extracting

the requirements. Customers typically have an abstract idea of what they want as an end result, but not

what software should do. Incomplete, ambiguous, or even contradictory requirements are recognized by

skilled and experienced software engineers at this point. Frequently demonstrating live code may help

http://en.wikipedia.org/wiki/Software_maintenance

http://en.wikipedia.org/wiki/Software

http://en.wikipedia.org/wiki/Software_engineering

http://en.wikipedia.org/wiki/Software_Elements_Analysis


48

reduce the risk that the requirements are incorrect. Software requirements analysis includes software

domain analysis and specification.

Software Specification: The task of precisely describing the software to be written. In practice, most

successful specifications are written to understand and fine-tune applications that were already well-

developed, although safety-critical software systems are often carefully specified prior to application

development. Specifications are most important for external interfaces that must remain stable.

Software Testing: An empirical investigation conducted to provide stakeholders with information about

the quality of the product or service under test, with respect to the context in which it is intended to

operate. This part of the software development process includes, but is not limited to, the process of

executing a program or application with the intent of finding software bugs.

Spiral Development: In this process, a desired capability is identified, but the end-state requirements are

not known at program initiation. Those requirements are refined through demonstration and risk

management; there is continuous user feedback; and each increment provides the user the best possible

capability. The requirements for future increments depend on feedback from users and technology

maturation.

Sponsor: Represents the operational needs of the organizational element and ultimately the end-users of

the required system. The sponsor identifies capability gaps, conducts requirements analysis, and

participates in long range planning process and the prioritization of needs. The sponsor participates in all

phases of the acquisition to ensure that the item or system being acquired meets operational requirements.

In many contexts, the word "sponsor" refers to the sponsoring organization, and the term "sponsor's

representative" is the executive empowered to represent the Sponsor for a given investment.

Stable: Firmly established; not changing or fluctuating.

Supportability Data: A key component of availability. It includes design, technical support data, and

maintenance procedures to facilitate detection, isolation, and timely repair and/or replacement of system

anomalies. This includes factors such as diagnostics, prognostics, real time maintenance data collection,

and human system integration considerations.

Supportability Plan: (1) Describes how design and total system will be sustained. (2) Ensures a fielded

systems/technology will maintain a state of readiness and safety, trained operators and maintainers, with

the smallest logistical requirements.

System: Integrated set of constituent pieces that are combined in an operational or support environment

to accomplish or deliver a defined objective or capability.

System Architecture Diagrams: A description, including graphics, of systems and interconnections

providing for or supporting functions.

System Architecture: System Architecture describes the entire system. It includes the physical

architecture produced through design synthesis and adds the enabling products and services required for

life cycle employment, support, and management.

System Readiness Level (SRL): An index of maturity applied at the system-level concept with the

objective of correlating this indexing to appropriate systems engineering management principals. It is

designed to be a function of the individual TRLs in a system and their subsequent integration points with


49

other technologies, IRL. This term is not used in the calculator, but is important to understand for future

discussion.

System Requirements: Required system behavior

System Requirements Document (SRD): The SRD provides a description of the required system

behavior without prescribing a technical solution.

System Requirements Specification (SRS): Document listing the requirements of a system that is

planned to be developed. This includes analysis of business needs of customer and end users to help

identify business problems and propose solutions. Business Requirements constitute a specification of

simply what the business wants. This is usually expressed in terms of broad outcomes the business

requires, rather than specific functions the system may perform. Specific design elements are usually

outside the scope of this document, although design standards may be referenced.

Systems Engineering: (1) Interdisciplinary field of engineering that focuses on how complex engineering

projects should be designed and managed. Issues such as logistics, the coordination of different teams,

and automatic control of machinery become more difficult when dealing with large, complex projects.

Systems engineering deals with work-processes and tools to handle such projects, and it overlaps with

both technical and human-centered disciplines such as control engineering and project management. (2) A

discipline that concentrates on the design and application of the whole system as distinct from the parts. It

requires examining a problem in its entirety, taking into account all the facets and variables and relating

the social to the technical aspect. The translation of operational requirements into design, development,

implementation concepts, and requirements in the life cycle of a system.

Systems Engineering Plan (SEP): A description of the program‘s overall technical approach including

processes, resources, metrics, applicable performance incentives, and the timing, conduct, and success

criteria of technical reviews.

System Feasibility: (see Engineering Feasibility)

-T-

Technical Drawing: (see Engineering Drawing)

Technical Report: (see Scientific Report)

Technical Performance Measurement (TPM): Describes activities undertaken to obtain design status

beyond schedule and cost. A TPM is an assessment that estimates the values of essential performance

parameters of the current design of Work Breakdown Structure (WBS) product elements. It forecasts the

values to be achieved through the planned technical program effort, measures differences between

achieved values and those allocated to the product element, and determines the impact of these

differences on system effectiveness. This term is not used in the calculator, but may be a type of

programmatic document that can be included in the development of a given technology.

Technological Forecasting: (see Trend Analysis)

Technology: (1) Human innovation in action that involves the generation of knowledge and processes to

develop systems that solve problems and extend human capabilities. (2) The innovation, change, or

modification of the natural environment to satisfy perceived human needs and wants. (3) A piece of


50

equipment or a technique for performing a particular activity. (4) A capability given by the practical

application of knowledge.

Technology Feasibility: (see Engineering Feasibility)

Technology Assessment (TA): A systematic process that ascertains the need to develop or infuse

technological advances into a system. The technology assessment process makes use of basic systems

engineering principles and processes within the framework of the PBS (Product Breakdown Structure). It

is a two-step process comprised of 1) the determination of the current technological maturity in terms of

technology readiness levels and 2) the determination of the difficulty associated with moving a

technology from one TRL to the next one.

Technology Development Plan (TDP): A document that identifies technologies to be developed,

heritage systems to be modified, alternative paths to be pursued, fallback positions and corresponding

performance de-scopes, milestones, metrics, and key decision points. This term is not used in the

calculator, but may be a type of programmatic document that can be included in the development of a

given technology.

Technology Insertion: Process of updating the performance or extending the life of aging systems and

platforms through integration of new, proven technologies that meet a set of requirements for lower life

cycle costs, while improving performance, reliability, and ensuring future technology growth potential.

This term is not used in the calculator, but may be incorporated into a programmatic document that can be

included in the development of a given technology.

Technology Maturation Plan (TMP): A planning document that lays out the activities required to bring

immature technologies up to the desired TRL. This term is not used in the calculator, but may be a type of

programmatic document that can be included in the development of a given technology.

Technology Readiness Assessment (TRA): A systematic, metrics-based process and resulting report

that assesses the maturity of technologies. TRAs are may be conducted by Program Managers (PMs) or

by independent groups.

Technology Refresh: A strategy to provide cost-effective support and upgrade strategies, to keep a

program ahead of the obsolescence curve. This strategy should result in regular upgrades instead of major

end-of- life modifications or follow-on systems. This term is not used in the calculator, but may be

incorporated into a programmatic document that can be included in the development of a given

technology.

Technology Transition Agreement (TTA): An agreement between the developing organization and the

sponsoring organization of a technology that specifies the product to be delivered by a project to an

acquisition program. It is developed and signed at the project level.

Technology Transition: The process of advancing scientific, operational, technological, or process

development efforts, such as devices, subassemblies, algorithms, systems, and processes, from one state

of product realization to the next stage in the development life cycle.

Test & Evaluation Master Plan (TEMP): 1) Documents the overall structure and objectives of the Test

and Evaluation (T&E) program. 2) Provides a framework within which to generate detailed T&E plans

and it documents schedule and resource implications associated with the T&E program. The TEMP

identifies the necessary Developmental Test and Evaluation (DT&E), Operational Test and Evaluation


51

(OT&E), and Live Simulations. It relates to program schedule, test management strategy and structure,

and required resources. The operational test organization reviews it and provides the operational test

planning for inclusion. The TEMP is then negotiated between the program office and operational test

organization. The TEMP includes a reaffirmation of the user requirements, and to an extent, an

interpretation of what those requirements mean in various operational scenarios. 3) Identifies the tasks

and activities required so the entire system can be adequately tested to assure a successful

implementation.

Test and Evaluation (T&E): Occurs during all major phases of the development life cycle, beginning

with system planning and continuing through the operations and maintenance phase. Ensures standardized

identification, refinement, and traceability of the requirements as such requirements are allocated to the

system components.

Threshold: (1) A minimum acceptable operational value below which the utility of the system becomes

questions. (2) The minimum acceptable value necessary to satisfy the need. If threshold values are not

achieved, performance is degraded, may be too costly to achieve, or may no longer be timely when

achievable.

Timing Constraints: Interacting hardware and software components that must deliver a specific

functionality under constraints on relative timing of their actions.

Tooling and Machining: (1) Tooling is a powered mechanical device, typically used to fabricate metal

components of machines by machining, which is the selective removal of metal. (2) The tools and support

devices required to setup and position equipment for a production run, which may be consumed during

that run or remain available for additional jobs.

Training Plan: A formal document that outlines the objectives, needs, strategy, and curriculum to be

addressed for training users of the new or enhanced system.

Training: The formal process of depicting, simulating, or portraying the operational characteristics of a

system or system component in order to make someone proficient in its use.

Transition: The transfer of responsibility for a product or system from an R&D organization to the end

user and the eventual support organization. Transition requires careful, up-front planning so that 1) the

user/support organization is capable, after transition, of procuring, deploying, operating, supporting, and

ultimately disposing of the system, and 2) the manufacturing organization is capable of manufacturing the

product or system.

Trend Analysis: Primarily, a trend analysis deals with the characteristics of technology, such as levels of

technical performance, like speed of a military aircraft, the power in watts of a particular future engine,

the accuracy or precision of a measuring instrument, the number of transistors in a chip in the year 2015,

etc. The forecast does not have to state how these characteristics will be achieved. Secondly,

technological forecasting usually deals with only useful machines, procedures or techniques. This is to

exclude from the domain of technological forecasting those commodities, services or techniques intended

for luxury or amusement. (also known as Technological Forecasting)

-U-

User Interface: (see Desktop Environment)


52

User Representative: Person formally designated to represent single or multiple users in the

requirements and acquisition process.


53

-V-

Validation: (1) The review of documentation by an operational authority other than the user to confirm

the need or operational requirement. (2) The process of determining the degree to which a model,

simulation, or federation of models and simulations, and their associated data are accurate representations

of the real world from the perspective of the intended use(s).

Value: The ratio of function to cost. Value can be increased by improving the function or reducing the

cost.

Value Analysis: (see Value Engineering)

Value Engineering: (1) Evaluating whether or not a technology fulfills a capability gap. (2) A systematic

method to improve the "value" of goods and services by examination of function (i.e., what something

"does" not what it is). In value engineering "functions" are always described in a two word abridgment of

an active verb and measurable noun (what is being done - the verb - and what it is being done to - the

noun) and to do so in the most non-prescriptive way possible. This is the basis of what value engineering

refers to as "function analysis." Value engineering uses rational logic (a unique "how" - "why"

questioning technique) and the analysis of function to identify relationships that increase value. It is

considered a quantitative method similar to the scientific method, which focuses on hypothesis -

conclusion to test relationships, and operations research, which uses model building to identify predictive

relationships. (3) In the United States, value engineering is specifically spelled out in Public Law 104-

106, which states ―Each executive agency shall establish and maintain cost-effective value engineering

procedures and processes." Value engineering is sometimes taught within the project management or

industrial engineering body of knowledge as a technique in which the value of a system‘s outputs is

optimized by crafting a mix of performance (function) and costs. In most cases this practice identifies and

removes unnecessary expenditures, thereby increasing the value for the manufacturer and/or their

customers. (also known as Value Management or Value Methodology)

Verification: The process of determining whether the system meets the requirements specified in the

SRD; the process of determining that a computer model, simulation, or federation of models and

simulations implementations and their associated data accurately represents the developer's conceptual

description and specifications.

Verification and Validation Plan: A formal document that describes the process to verify and validate

the requirements. This plan is usually created during the ―planning phase‖ and updated thereafter.

Verification, Validation and Accreditation (VV&A): Three step process of checking that a software

system meets specifications and that it fulfills its intended purpose. (see Verification and Validation and

Accreditation).

Vulnerability: (1) The characteristics of a system that cause it to suffer a definite degradation (loss or

reduction of capability to perform the designated mission) as a result of having been subjected to a certain

(defined) level of effects in an unnatural (man-made) hostile environment. Vulnerability is considered a

subset of survivability. (2) A weakness in an infrastructure, population or other target that may be

exploited to inflict damage or injury. May be the result of characteristics inherent in the infrastructure,

population or target (e.g., the susceptibility of living organisms to infection when exposed to viruses or

bacteria for which they do not have immunity). May also be the result of protective measures that are


54

inadequate to thwart a specific form of attack (e.g., anti-theft physical security measures faced with an

armed assault). May also result from response capabilities that are inadequate to prevent cascade effects,

thus allowing the damage or injury to continue growing beyond that inflicted in the initial attack.

-W-

Wiring Diagram: A simplified conventional pictorial representation of an electrical circuit. It shows the

components of the circuit as simplified standard symbols, and the power and signal connections between

the devices. Arrangement of the components interconnections on the diagram does not correspond to their

physical locations in the finished device (also known as a circuit diagram, electrical diagram, elementary

diagram, or electronic schematic).

Work Breakdown Structure (WBS): (1) The WBS provides the framework for program and technical

planning, cost estimating, resource allocation, performance measurement, technical assessment, and status

reporting. The WBS should include a WBS dictionary. The WBS should define the system to be

developed or produced. It should also display the system as a product-oriented family tree composed of

hardware, software, services, data, and facilities. (2) A product-oriented, hierarchical division tree of

deliverable items and associated services that related the elements of work to each other and to the end

item. The WBS is developed by starting with the end objective required and successively subdividing it

into manageable components in terms of size and complexity.

-X, Y, Z-

Yield: Amount or quantity produced or returned.


55

Acronyms

AD2—Advancement Degree of Difficultly

ADM—Acquisition Decision Memorandum

ATD—Advanced Technology Development

AoA—Analysis of Alternatives

BSC—Balanced Scorecard

CAIV—Cost as an Independent Variable

CD—Critical Decisions

CDR—Critical Design Review

CI—Configuration Item

CONOPS—Concept of Operations

CTE—Critical Technology Element

DTC—Design to Cost

DT&E—Developmental Test & Evaluation

FMEA—Failure Mode and Effects Analysis

FRP—Full Rate Production

IRL—Integration Readiness Level

IPT—Integrated Product Team

KDP—Key Decision Point

LCC—Life Cycle Cost

LRIP—Low Rate Initial Production

M&S—Modeling and Simulation

MOE—Measures of Effectiveness

MOS—Measures of Suitability

ORD—Operational Requirements Document

OT&E—Operational Test and Evaluation

PBS—Product Breakdown Structure

PPP—Program Protection Plan


56

QC—Quality Control

R&D—Research and Development

SEP—Systems Engineering Plan

SRD—System Requirements Document

SRL—System Readiness Level

SRS—System Requirements Specification

T&E—Test and Evaluation

TA—Technology Assessment

TDP—Technology Development Plan

TEMP—Test & Evaluation Master Plan

TMP—Technology Maturation Plan

TPM—Technical Performance Measurement

TRA—Technology Readiness Assessment

TRL—Technology Readiness Level

TTA—Technology Transition Agreement

VV&A—Verification, Validation and Accreditation

WBS—Work Breakdown Structure


57

Glossary References

Army Regulation 73–1: Test and Evaluation Policy. (November 10, 2004).

Bilbro, J. (February 2007). Mitigating the Adverse Impact of Technology Maturity. Project Management

Challenge 2007: Fourth Annual NASA Project Management Conference. February 6-7, 2007.

Bilbro, J. (2002). Technology Readiness Level Calculator Ver 1.0 Beta.

Bilbro, J. Introduction to Technology Assessment. (PowerPoint).

Bridgefield Group ERP/Supply Chain Glossary. Last accessed 12/19/08 at http://www.bridgefieldgroup.com/bridgefieldgroup/glos6.htm

Carnegie Mellon Software Engineering Institute. (March 2002). Capability Maturity Model® Integration

(CMMISM), Version 1.1.

Defense Acquisition University. (December 16, 2004). Defense Acquisition Guidebook. Last accessed on

12/19/08 at https://akss.dau.mil/dag/TOC_GuideBook.asp?sNode=R8-4-0&Exp=Y

Defense Acquisition University. (January 2005). Test and Evaluation Management Guide. Fifth Edition.

Defense Acquisition University. DAU Glossary of Defense Acquisition Acronyms and Terms, 12th

Edition. Last accessed 12/09/08 at https://akss.dau.mil/pv/glossary.aspx

Department of Defense. (December 8, 2008). DoDI 5000.02. Last accessed on 12/19/08 at https://akss.dau.mil/dag/DoD5002/DoD5002-3.3.asp

Department of Defense. (May 2005). Technology Readiness Assessment (TRA) Deskbook. Prepared by

the Deputy Under Secretary of Defense for Science and Technology (DUSD(S&T)).

Department of Defense Systems Management College. (January 2001). System Engineering

Fundamentals. Last accessed on 12/19/08 at http://www.dau.mil/pubs/pdf/SEFGuide%2001-01.pdf

DHS S&T RDT&E (Research, Development, Testing and Evaluation) Process Glossary webpage.

Obtained from Sam Francis.

Dictionary.com Unabridged (v 1.1). Random House, Inc. December, 12 2008. Last accessed on 12/19/08

at http://dictionary.reference.com

Federal Aviation Administration. (February 2000). Configuration Management Guide/Part IV.

<http://fast.faa.gov/NAS_CM_Guide/html/sec4AC_DEF.htm>

Federal Emergency Management Agency (FEMA). (2008). National Response Framework Resource

Center. Last accessed 12/19/08 at http://www.fema.gov/emergency/nrf/glossary.htm#U

Gjerde, A. Waterfall Analysis -- What is it? How does it work? Steelwedge Software. Last accessed on

12/19/08 at http://www.steelwedge.com/newsletter/sept05/newsletter_Gjerde_Sept05.html

Glossary of Terms in New Products Management. (2007). Last accessed on 12/19/08 at http://faculty.stonehill.edu/glantos/Lantos1/PDF_Folder/BA347_PDF/Glossary%20F%2007.htm

http://www.bridgefieldgroup.com/bridgefieldgroup/glos6.htm

https://akss.dau.mil/dag/TOC_GuideBook.asp?sNode=R8-4-0&Exp=Y

https://akss.dau.mil/pv/glossary.aspx

https://akss.dau.mil/dag/DoD5002/DoD5002-3.3.asp

http://www.dau.mil/pubs/pdf/SEFGuide%2001-01.pdf

http://dictionary.reference.com/

http://fast.faa.gov/NAS_CM_Guide/html/sec4AC_DEF.htm

http://www.fema.gov/emergency/nrf/glossary.htm#U

http://www.steelwedge.com/newsletter/sept05/newsletter_Gjerde_Sept05.html

http://faculty.stonehill.edu/glantos/Lantos1/PDF_Folder/BA347_PDF/Glossary%20F%2007.htm


58

Graettinger, C. et al. (September 2002). Using the Technology Readiness Levels Scale to Support

Technology Management in the DoD‘s ATD/STO Environments: A Findings and Recommendations

Report Conducted for Army CECOM. Carnegie Mellon Software Engineering Institute. Pittsburgh, PA.

Idaho National Laboratory. Risk Mitigation and Management. Last accessed on 12/19/08 at https://inlportal.inl.gov/portal/server.pt?open=512&objID=334&PageID=1862&cached=true&mode=2&userID=488

Joint Defense Manufacturing Technology Panel (JDMTP). (May 29, 2008). Manufacturing Readiness

Assessment (MRA) Deskbook.

Manager‘s Guide to Technology Transition in an Evolutionary Acquisition Environment. Version 1.0.

Defense Procurement and Acquisition Policy. January 31, 2003. Office of the Under Secretary of Defense

(Acquisition, Technology and Logistics).

Merriam-Webster Online. Last accessed on 12/19/08 at http://www.merriam-webster.com/

Missile Defense Agency Technology Applications Program. (March 2003). MDA/DV Hardware Maturity

Checklists for Technology Readiness Levels (TRLs). Last accessed on 12/19/08 at http://mdatechnology.net/pdf/trl_checklist.pdf

NASA Systems Engineering Handbook, NASA/SP-2007-6105 Rev 1.

Navy Commercial Services Management. Reference Library: Competitive Sourcing Terminology. Last

accessed on 12/19/08 at http://competitivesourcing.navy.mil/reference_documents/defs.cfm?ltr=F

Nolte, W. et al. Technology Readiness Calculator (ver 2.2).

Nowak, D. CBRN Defense Initiative to Improve Technology Transition.

Procurement Executives Association. (1998). Guide to a Balanced Scorecard Performance Management

Methodology. Last accessed on 12/19/08 at http://www.management.energy.gov/documents/BalancedScorecardPerfAndMeth.pdf

Requirements Generation System. (April 15, 2001). Chairman of the Joint Chiefs of Staff Instruction. J-8.

CJCSI 3170.01 B.

Six Sigma. Last accessed on 12/19/08 at http://www.isixsigma.com/dictionary

Smith, J. (April 2004). An Alternative to Technology Readiness Levels for Non-Developmental Item

(NDI) Software. Carnegie Mellon Software Engineering Institute. Pittsburgh, PA.

Suaser, B et al. (2006). From TRL to SRL: The Concept of System Readiness Levels.

The Defense Systems Engineering Handbook. (2002). The Defense Engineering Group. Department of

Mechanical Engineering. University College London

The Owner's Role in Project Risk Management (2005). Committee for Oversight and Assessment of U.S.

Department of Energy Project Management, National Research Council. Board on Infrastructure and the

Constructed Environment. Last accessed on 12/19/08 at http://www.nap.edu/catalog.php?record_id=11183

U.S. Department of Energy Office of Environmental Management. (March 2008). Technology Readiness

Assessment (TRA)/Technology Maturation Plan (TMP) Process Guide.

https://inlportal.inl.gov/portal/server.pt?open=512&objID=334&PageID=1862&cached=true&mode=2&userID=488

http://www.merriam-webster.com/

http://mdatechnology.net/pdf/trl_checklist.pdf

http://competitivesourcing.navy.mil/reference_documents/defs.cfm?ltr=F

http://www.management.energy.gov/documents/BalancedScorecardPerfAndMeth.pdf

http://www.isixsigma.com/dictionary

http://www.nap.edu/catalog.php?record_id=11183


59

APPENDIX D: DECISION ALGORITHMS, CHANGE LOG

FOR THE DHS S&T RL CALCULATOR (VER1.1), AND

EXPLANATION OF ADMINISTRATIVE FUNCTIONS

Decision Algorithms

There are three Readiness Levels (RL) addressed in the HSSAI modified RL Calculator: Technical,

Manufacturing, and Programmatic. Each of these is further delineated by questions at sub-levels. TRL 1

through 9, MRL MRL 3 through 9, and PRL 1 through 9 . Each RL is independently calculated and

reported upon The RLs have been defined so that in a ―normal‖ development a program would have about

the same technology, manufacturing, and programmatic readiness levels. There are no MRLs of 1 and 2

since it is believed that programs with TRLs of 1 and 2 are not mature enough to have MRLs

determined...

On the initial entry page of the HSSAI version, START HERE, there is a listing of a single requirement

that must be demonstrated to reach the RL sub-level. The listing is hidden if a User chooses to ―Omit‖ an

RL. The actual calculation of an RL sub-level is accomplished by dividing the total number of questions

answered into the RL score obtained for each individual RL at each individual sub-level. The score

obtained for any RL is calculated by summing the applicable sub-level questions individual percentages,

e.g., 100% is 1, 45% is 0.45. The percentages are originally shown as 100% on the RL Calculator page.

This will not be the percentage used in the calculation unless the check box is marked or the percentage

set manually by entering a number where there is presently a 100. If the User wants to have the

calculation include the question, either the check box must be marked or a percentage entered.

Only questions that are Applicable are considered in the calculation. The scale color-code for the RL

within a sub-level, whether Green or Yellow, is changed from the default color Red by comparing the

resulting division to the Set Point selected by the User. If the division result is equal to or greater than the

Set Point, that RL sub-level is changed to the appropriate color in the scale score. Each sub-level of each

RL is independent of the results of other sub-levels or other RLs. A lower level can be at a lower value

regardless of what sub-level was reached at the upper level. This means that a MRL 4 could be Yellow

while a MRL 5 could be Green.

Log of modifications

Many cosmetic changes the AFRL TRL Calculator (ver 2.2) were incorporated into the HSSAIversion.

The Summary page of the AFRL Calculator was used as a basis for the START HERE page of the

HSSAIversion. Many of the items on the Summary page were retained but relocated within the Calculator.

Added in the HSSAIversion, though, was the drop-down listing for each RL, and a button (Continue to

Calculator) to assist the User in determining the next step in the process. A link was provided at the

extreme right of each question to provide the User with a full description of that level, if needed.

The only other page within the AFRL TRL Calculator (ver 2.2) used is the TRL Calculator page that

formed the basis for the RL Calculator page of the HSSAIversion. The TOP LEVEL VIEW section of the

AFRL TRL Calculator (ver 2.2) was moved to the START HERE page of the HSSAIversion for the TRL

section and was used as the basis for the MRL and PRL level questions that were added on the START


60

HERE page. The selections at the top of the TRL Calculator page were all moved and/or changed in

format to the START HERE page.

The AFRL TRL Calculator (ver 2.2) calculates a number of combined scores in addition to the individual

RL scores on the TRL Calculator page. These combined scores were not used in the HSSAIversion. The

AFRL TRL Calculator (ver 2.2) did not seem to use the percentage complete in the calculation of a RL.

The HSSAIversion moved a step further in the direction of calculating the Yellow score by determining

the percentage complete for each RL at each sub-level.

The AFRL TRL Calculator (ver 2.2) TRL Calculator page contained all of the results. The HSSAIversion

enhanced this page by breaking up the Hardware and Software section of the AFRL TRL Calculator (ver

2.2) into different sections and pages. In the HSSAIversion, each RL has its own horizontal thermometer

score (labeled as Scale) indicator which indicates the level of readiness with color coding. The spinner of

the ―% complete is now set at:‖ of the AFRL TRL Calculator (ver 2.2) was changed in the HSSAIversion

to two spinners, one for the Green Set Point and one for the Yellow Set Point that can be changed

individually. The requirement of the Yellow being no nearer the Green than by 15% in the AFRL TRL

Calculator (ver 2.2) was kept in the HSSAIversion.

In the AFRL TRL Calculator (ver 2.2 )the Summary button returned the User to the Summary page so the

User could view the Summary of the Technology‘s Readiness to Transition section in the lower half of

the page. The HSSAIversion has a Create Report button instead. Clicking this button will create the

applicable report for each RL. Those RLs that were selected as Omit on the START HERE page will not

have a report created for them.

Each report of the HSSAIversion has three sections. The first section of the report repeats the

project/program information entered by the User on the START HERE page, has a horizontal

thermometer score indicator for that RL, and displays a Summary level for the Green and Yellow if

applicable. The report then lists all questions, and their sub-level, that were marked as Not Applicable for

that RL. Questions marked Not Applicable for the other RLs will not be shown.

The final section of the report is a full listing of all applicable questions for that RL. The data includes the

sub-level, a code to indicate whether Hardware or Software or Both, the percentage complete, and the

actual text of the question. At the end of each sub-level, the report shows the calculated value of the

overall completion percentage for that sub-level with any Comments the User entered for that sub-level.

Administrative Functions

The following section provides instruction on how to edit the DHS S&T RL Calculator (Ver1.1) MS

Excel workbook successfully.

Before modifying the DHS S&T RL Calculator (Ver1.1)

Unhide any hidden sheets (select Format, Sheet, Unhide, and then the sheet to be unhidden. Repeat until

all sheets are unhidden).

Then on each sheet individually (unless you know how to do all sheets at once):

Gridlines and Row and column headers should be enabled (select Options from the Tools menu and check

the appropriate boxes).


61

All rows and columns should be unhidden (click the extreme upper left header marker, then select

Format, Row, Unhide, and Format, Column, Unhide)

Editing DHS S&T RL Calculator (Ver1.1) text

All text within the Calculator can be modified. If the cell does not seem to accept changes, check whether

the sheet containing the text is Protected. If it is, on the Tools menu, point to Protection, and then click

Unprotect Sheet. If prompted, enter the protection password for the worksheet. If a range is to be edited

then on the Tools menu, point to Protection, and then click Allow Users to Edit Ranges.

The Report Default sheet will need to be edited if text is to be changed on the reports other than the text

on the RL Calculator sheet. If the text to be edited is not on the Report Default sheet or from the RL

Calculator sheet, the macros will need to be accessed to determine where the text is located. The

appropriate line of code will need to be edited to change that text.

Editing DHS S&T RL Calculator Questions

The questions in the Calculator can be changed but doing so may have unintended consequences. Before

changing any text, view all the properties of any shape (Option Button, Check Box, or Slider) associated

with that question. Once the text of the question is changed, check any shape for its associated properties

and adjust them accordingly.

Editing DHS S&T RL Calculator Categories

The text for the RLs on the START HERE sheet may be modified within the constraints of the merged

cell width. Exceeding the word limit will cause the text to either wrap, causing the row height to adjust, or

hide beneath the Full Description cell.

There is space allotted for only 10 TRLs, 9 PRLs, and 7 MRLs. If more are required, inserting rows will

cause the macros to crash unless the location of cells is corrected in the macros.

If any RLs are to be deleted, it would be best to hide the row, rather than try to completely adjust the

macros. In any case, the macros might need adjustment if the level is used by any macro. Only a full

testing of the Calculator with the changes incorporated will determine what has to be done.

Adding/removing questions

If questions are to be eliminated from either the START HERE or the RL Calculator sheet in the

Calculator, all shapes on that sheet will need to be checked for proper associations and properties. When a

shape is removed from Excel, sometimes the program realigns the names of shapes and sometimes a

shape is actually placed over another current shape.

Extra lines are provided on the RL Calculator page if questions are to be added to the Calculator.

After adding all the text (Apply (Y/N)?, H/SW Both, 100 for complete, and the question), the

slider and check box must be added. While adding them, note their Excel given name. These two

names must be added to the columns to the right of the questions as well as copying the formulas

from existing rows and adding any additional information. Once the lines are added, the macros

may need adjustment, particularly the ones that move the shapes off of and on to the screen.


62

If additional lines are needed for questions, inserting rows might require changing the addressing

of cells within the macros. The range names will also need to be checked for complete coverage

of the slider and check box names within a sub-level.


63

APPENDIX E: READINESS LEVEL QUESTIONS BY

CATEGORY

Table 8: DHS S&T Technology Readiness Level Questions for the Modified Calculator

Technology Readiness Level Questions

TRL 1

B Do rough calculations support the concept?

B Do basic principles (physical, chemical, mathematical) support the concept?

S Does it appear the concept can be supported by software?

S Are the software requirements known in general terms?

B Do paper studies confirm basic scientific principles of new technology?

S Have mathematical formulations of concepts been developed?

S Have the basic principles of a possible algorithm been formulated?

B Has a scientific methodology or approach been developed?

TRL 2

B Has potential system or component applications been identified?

B Have paper studies confirmed system or component application feasibility?

B Has an apparent design solution been identified?

H Have the basic components of the technology been identified?

B Has the user interface been defined?

H Have technology or system components been at least partially characterized?

H Have performance predictions been documented for each component?

S Has preliminary software coding that confirms basic principles been documented?

B Has a functional requirements generation process been initiated?

H Does preliminary analysis confirm basic scientific principles?

S Have experiments validating the concept been performed with synthetic data?

B Are basic scientific principles confirmed with analytical studies?

B Do all individual parts of the technology work separately? (No real attempt at integration)

S Is the hardware that the software will be hosted on available?

B Are output devices available?

TRL 3

H Have predictions of components of technology capability been validated?

S Have analytical studies verified performance predictions and produced algorithms?

H Can all science applicable to the technology be modeled or simulated?

S Is outline of software algorithms documented?

H Do experiments or modeling and simulation (M&S) validate performance predictions of technology

capability?

S Does preliminary coding verify that software can satisfy an operational requirement?

B Do experiments verify feasibility of application of technology?


64


H Do experiments or modeling and simulation (M&S) validate performance predictions of components

of technology capability?

B Have cross-technology effects (if any) been identified?

B Do paper studies indicate that technology or system components can be integrated?

B Are the technology or system performance metrics established?

S Have technology or system performance characteristics been confirmed with representative data

sets?

S Do algorithms run successfully in a laboratory environment, possibly on a surrogate processor?

S Has inventory of available software that does similar tasks been completed?

S Has existing software been examined for possible reuse?

B Has scientific feasibility of proposed technology been fully demonstrated?

B Does analysis of present technologies show that proposed technology or system fills a capability

gap?

TRL 4

B Have cross-technology effects (if any) been fully identified and documented?

H Has acceptance testing of individual components been performed?

H Has performance of components and interfaces between components been demonstrated?

B Does draft system architecture plan exist?

B Have end user technology/system requirements been documented?

H Does breadboard demonstrate functionality of all components?

S Have algorithms been converted to pseudocode?

S Has analysis of data requirements and formats been completed?

S Do stand-alone modules align with preliminary system architecture plan?

H Has component compatibility been demonstrated?

B Does technology demonstrate basic functionality in simplified environment?

B Have performance characteristics been demonstrated in a laboratory environment?

S Does prototype solve synthetic full-scale problems or process fully representative data sets?

S Have all functions or modules been demonstrated in a laboratory environment?

S Have ad hoc integration of functions or modules been demonstrated?

B Have low-fidelity assessments of system integration and engineering been completed?

TRL 5

B Have cross-technology effects (if any) been fully identified, analyzed, and documented?

B Have internal system interface requirements been documented?

B Have external interfaces been documented?

B Has analysis of internal interface requirements been completed?

B Does the breadboard have realistic interfaces?

S Is coding of individual functions/modules completed?

B Can all system specifications be simulated and validated within a laboratory environment?

H Has a brassboard been developed?

B Is the laboratory environment high-fidelity?


65


S Have functions been integrated into modules?

B Have individual component functions been verified through testing?

S Have system modules been debugged?

S Has integration of modules/functions been demonstrated in a laboratory environment?

S Have algorithms been run on a processor that can be fielded in an operational environment?

B Have objective and threshold operational requirements been developed?

B Has a Product Breakdown Structure been developed?

TRL 6

B Have system integration issues been addressed?

B Is the operational environment fully known?

B Have performance characteristics been verified in a simulated operational environment?

B Has prototype been tested in a simulated operational environment?

B Has system been tested in realistic environment outside the laboratory?

B Has an inventory of external interfaces been completed?

S Has analysis of timing constraints been completed with satisfactory results?

S Has analysis of database structures and interfaces been completed?

S Does the prototype functionally handle realistic problems?

S Have software algorithms been integrated with existing systems?

S Has functionality of integrated modules been tested?

S Is software documentation available?

B Has engineering feasibility been fully demonstrated?

TRL 7

H Can unavailable system components be simulated using modeling and simulation (M&S)?

B Have all interfaces been tested individually under stressed and anomalous conditions?

S Do algorithms run on processor(s) in an operational environment?

B Has technology or system been tested in a relevant environment?

H Are available components representative of production components?

B Has operational testing of technology/system in relevant environment been completed?

B Has fully integrated prototype been demonstrated in actual or simulated operational environment?

TRL 8

B Are all technology/system components form, fit, and function compatible?

B Is technology/system form, fit, and function compatible with operational environment?

B Has technology/system form, fit, and function been demonstrated in operational environment?

S Has software been thoroughly debugged?

B Is technical Developmental Test and Evaluation (DT&E) successfully completed?

TRL 9

B Does technology/system function as defined in Operational Concept document?

B Has technology/system has been deployed in intended operational environment?

B Has technology/system been fully demonstrated?


66


B Has Operational Test and Evaluation (OT&E) been successfully completed?

Table 9: DHS S&T Manufacturing Readiness Level Questions for the Modified Calculator

Manufacturing Readiness Level Questions

MRL 3

H Does basic laboratory research equipment verify physical principles?

H Have design techniques been identified and/or developed?

H Have current manufacturability concepts been assessed?

H Can key components needed for breadboard be produced?

MRL 4

H Are most system components available (laboratory surrogates in some cases)?

H Do parts and components exist in pre-production form?

H Has breadboard been assembled?

S Have designs been verified through formal inspection process?

H Have scalable technology prototypes been produced?

H Have design techniques been identified so small applications may be analyzed/simulated?

B Have integration studies been started?

H Have key manufacturing processes been identified?

H Have key manufacturing processes been assessed in the laboratory?

H Are potential manufacturing problems documented?

MRL 5

H Is all pre-production hardware available?

H Have manufacturing targets for improved yield, if needed, been established?

H Have key manufacturing processes been validated and documented?

H Has engineering design freeze been implemented?

H Have prototypes been created?

H Have tooling and machining been demonstrated in the laboratory?

H Have mitigation strategies been identified to address possible manufacturing shortfalls?

B Have quality and reliability issues been documented (target levels may not yet be set)?

H Has an assessment of component assembly needs been performed?

H Have sigma levels needed to satisfy Cost as an Independent Variable (CAIV) targets been defined?

H Have production processes been reviewed?

MRL 6

H Have quality and reliability levels been established?

B Has manufacturing design freeze been reached?

H Have investment needs for process and tooling been determined?

B Has factory acceptance testing been performed?


67

Manufacturing Readiness Level Questions

H Have critical manufacturing processes been prototyped?

H Is most manufacturing hardware available?

H Has a manufacturing process been selected?

H Are components functionally compatible with operational system?

B Have all integration demonstrations been completed?

H Have major production issues been resolved?

H Are manufacturing process and tooling mature?

H Are production demonstrations complete?

MRL 7

H Have materials, processes, methods, and design techniques been identified?

H Have materials and manufacturing process and procedures been demonstrated?

H Has process tooling and inspection been demonstrated in production environment?

H Have tooling and machining for manufacturing been proven?

H Are there no or few manufacturing process design changes?

B Is data for Reliability, Maintainability, and Supportability analysis available?

H Are materials, processes, methods, and design techniques at least moderately developed & verified?

H Is pre-production hardware available?

H Are quality levels for manufacturing processes (e.g., "sigma levels") established?

S Have most software bugs been removed?

H Is production planning complete?

H Do prototypes represent actual form, fit, and function?

H Ready for Low Rate Initial Production (LRIP)?

MRL 8

H Have design to cost goals (DTC) been met for Low Rate Initial Production (LRIP)?

H Have tooling and machining been demonstrated in production environment?

H

Have manufacturing processes been demonstrated by pilot production line or Low Rate Initial

Production (LRIP)?

H Have manufacturing processes demonstrated acceptable yield and producibility levels?

H Is manufacturing process controlled to an appropriate quality level?

H Are all materials in production and readily available?

B Has maintainability, reliability, and supportability data collection been completed?

H Ready for Full Rate Production (FRP)?

MRL 9

H Have design to cost (DTC) goals been met?

H Are affordability issues built into initial production and possible evolutionary acquisition?

H Is manufacturing design stable (few or no design changes)?

H Are all manufacturing processes controlled to appropriate quality level?

H Is production stable?


68

Table 10: DHS S&T Programmatic Readiness Level Questions for the Modified Calculator

Programmatic Readiness Level Questions

PRL 1

B Have scientific observations been reported in peer reviewed reports?

B Has a sponsor or funding source been identified?

B Is it known who will perform the research and the schedule?

PRL 2

B Has an R&D program that will support technology development been identified?

B Is the end user of the technology known?

B Has a customer expressed interest in application of technology?

B Have draft functional requirements been documented?

B Has a requirements tracking system been initiated?

B Have results of analytical studies been reported to scientific journals, etc.?

B Has a R&D investment strategy been prepared?

B

Are capabilities and limitations of potential researchers and research facilities documented or

previously validated?

B Is the research approach documented?

B Has preliminary qualitative risk analysis been documented?

B Has preliminary market analysis been documented?

PRL 3

H Have system performance characteristics and measures been documented?

B Has customer representative to work with R&D team been identified?

B Is customer participating in requirements generation?

B Has Technology Transition Agreement (TTA) including possible TRL for transition been drafted?

B Have scaling studies been started?

B Has analysis of alternatives been completed?

B Have programmatic risks been identified?

B Have programmatic risk mitigation strategies been documented?

B Has preliminary value analysis been performed?

PRL 4

B Does draft requirements document exist?

B Have system performance metrics been established?

B Have design requirements been derived from system requirements?

B Have program exit criteria and milestones been established?

S Are software specifications known?

B Have draft conceptual designs been documented?

B Have initial cost drivers been identified?

B Have Cost as an Independent Variable (CAIV) targets been set?


69


B Are scaling studies and architecture diagrams completed?

B Has draft Systems Engineering Plan (SEP) been completed?

B Has customer reviewed draft Technology Transition Agreement?

B Have work breakdown structures (physical and programmatic) been developed?

B Has Integrated Product Team (IPT) formally been established?

B Is customer representative a member of IPT?

B Has a formal risk management program been initiated?

B Has project risk management been integrated with project management?

B Have availability dates for missing technologies (if any) been established?

B Have functional requirements been finalized?

PRL 5

B Has systems engineering begun?

B Has system architecture been finalized?

H Have end users been involved in developing physical, functional, and performance characteristics?

B Have engineering drawings and wiring diagrams been completed?

B Does draft Systems Engineering Plan (SEP) address integration?

B Does draft Systems Engineering Plan (SEP) address test and evaluation?

B Does draft Systems Engineering Plan (SEP) address mechanical and electrical interfaces?

B Does draft Systems Engineering Plan (SEP) address expected final performance?

B Is the programmatic risk management plan documented?

B Has a configuration management plan been implemented?

B Has formal inspection of all components been completed?

B Is the configuration management plan documented?

B Is the draft Test and Evaluation Master Plan (TEMP) documented?

H Is the draft Systems Engineering Plan (SEP) completed?

B Has customer committed to a Technology Transition Agreement?

B Has a Business Case for the product been developed?

H Has Failure Mode and Effects Analysis (FMEA) been completed?

B Has value analysis been performed?

B Has a life-cycle cost analysis has been performed?

PRL 6

B Have engineering drawings and wiring diagrams been finalized?

B Has collection of maintainability, reliability, and supportability data started?

B Have design to cost (DTC) goals been identified?

B Has the Test & Evaluation Master Plan (TEMP) been finalized?

B Has the Systems Engineering Plan (SEP) been finalized?

B Is the Technology Transition Agreement current?

B Does analysis of scalability support procurement?

B Has an interface control process been initiated?


70


H Have requirements for production been reviewed by manufacturer and end user?

B Has the Technology Transition Agreement been approved by all parties?

B Has system requirements specification document been completed?

B Are all changes controlled/documented using configuration management?

B Has the final technical report been completed?

S Has Verification, Validation and Accreditation (VV&A) been initiated?

S Has alpha version software been released?

B Has final Technology Transition Agreement been signed by appropriate parties?

B Have acquisition program milestones been established?

B Does customer agree it can manufacture, deploy, and maintain the system?

B Is formal requirements document completed?

PRL 7

S Has Verification, Validation and Accreditation (VV&A) verification step been completed?

B Is scaling completed?

H Have design to cost (DTC) goals been validated?

S Has beta version software been released?

PRL 8

B Has Interface Control Process been completed?

S Is software user documentation completed and under configuration control?

B Is training documentation completed and under configuration control?

B Is maintenance documentation completed and under configuration control?

B Have final architecture diagrams been completed?

S Has Verification, Validation and Accreditation (VV&A) validation step been completed?

B Has technical Developmental Test and Evaluation (DT&E) documentation been completed?

S

Has Verification, Validation and Accreditation (VV&A) accreditation step been completed

(authorized for use in intended system)?

PRL 9

B Have safety/adverse effects issues been identified and mitigated?

B Has Training Plan been implemented?

B Has Supportability Plan been implemented?

B Has Program Protection Plan been implemented?

B Has all programmatic documentation been completed?

B Has Operational Test and Evaluation (OT&E) documentation been completed?

Date post:	02-May-2018
Category:	Documents
Upload:	donga
View:	230 times
Download:	5 times