T&E Best Practices Final Report

Best PracticesApplicable to DoD DevelopmentalTest and Evaluation

June 1, 1999

A study performed by theScience Applications International Corporation for the

Directorate of Test, Systems Engineering and EvaluationOffice of the Secretary of DefenseWashington, D.C.

Foreword

The work documented in this report was conducted by the Science

Applications International Corporation (SAIC) under a contract with the

Directorate of Test, Systems Engineering and Evaluation, Office of the

Secretary of Defense. The publication of this SAIC report does not

indicate endorsement of its contents by the Department of Defense, nor

should it be construed as reflecting an official position of the Department.

ii

Contents

FOREWORD iii

ACKNOWLEDGMENTS vii

EXECUTIVE SUMMARY ES-1

I. STUDY FRAMEWORK I-1Utility of Best Practices I-1Study Objective I-1Study Methodology I-2Study Focus I-2Key Questions I-3Challenges I-5

II. COMMERCIAL BEST PRACTICES II-1Commercial T&E Compared with DoD T&E II-1What Constitutes a Best Practice? II-1T&E Best Practices in Commercial Enterprises II-2

III. DISCUSSION III-1Common Characteristics Across Industries III-1Focus on Adding Product Value III-1T&E Function Integrated into Product Development Process III-1Technical Personnel Maintain Customer Awareness III-2Product Development Testing With Little or No Customer Involvement III-2Commercial Enterprises Are Flexible III-2Technology Is Applied When Practical III-3“Product” Is More Important Than “Process” III-3Suitability Is At Least as Important as Effectiveness III-4Commercial Testers Have Significant Responsibility III-4Competition Continues Throughout Life Cycle III-4Schedules Have Priority III-5Software Testing III-6T&E Best Practices in the DoD III-7

IV. CONCLUSIONS AND RECOMMENDATIONS IV-1Conclusions

Best Practices Can Be Applied to T&E IV-1DoD Has Best Practices

But Needs to Communicate Them More Effectively IV-1Commercial Best Practices Can Be Adapted for DoD Use IV-1

iii

Emphasis Could Be Increased on Reducing Time Required for DoD Test Programs IV-3

More Focused Management Oversight May Result in Better Programs IV-4

Process Not Geared to Evolutionary Upgrades IV-4Opportune Time for ODTSE&E Best Practice Initiatives IV-4

Recommendations Primary Initiatives IV-4Other Potential Actions IV-5

APPENDICESA. Organizational Participants and ContactsB. Representative Discussion Topics for Industry VisitsC. Bibliography and References

iv

Acknowledgments

The sponsor and authors of this report express their sincere gratitude to the following companies for their generous contributions of time, information, and insight. Their achievements in product development validate the best practices in test and evaluation which each of them employs.

Boeing Commercial Airplane Group

Dura Pharmaceuticals, Inc.

Hewlett-Packard Company

Lockheed Martin Corporation

Microsoft Corporation

National Steel and Shipbuilding Company

Pathology Associates International Corporation

Qualcomm, Inc.

Stac, Inc.

Teledyne Ryan Aeronautical

Wyle Laboratories

v

Executive Summary

Study Objective

Today, more and more companies are engaged in “benchmarking” to compare their products, services, and processes against those of other relevant organizations and to determine how their results measure up against the “best practices” of others. Although management structures vary, many organizations share common business practices and can learn from one another.

In April 1998, the Deputy Director for Systems Assessment in the Office of the Director for Test, Systems Engineering and Evaluation (ODTSE&E) within the Office of the Secretary of Defense (OSD), commissioned this study to identify and examine the “best practices” used by industry and government organizations in the conduct of their developmental test and evaluation activities. The purpose of the study, conducted by the Science Applications International Corporation, was to assess the potential applicability and utility of these best practices to the business processes of Department of Defense (DoD) test and evaluation.

From the outset of this study, it was acknowledged that some of the identified commercial best practices might already be in use (to some degree) at various locations within the DoD. For this reason, it was recognized that some of the study’s recommendations might be more useful at some DoD organizations than others.

Study Methodology

The study was conducted in four phases: Literature Survey, Analysis, Synthesis, and Reporting.

(1) During Phase I, Literature Survey, a bibliography of over 150 citations was created and a study library assembled.(2) During Phase II, Analysis, documents were reviewed, information requirements defined, and data gathered from selected industry participants.(3) During Phase III, Synthesis of Best Practices, the best practices in test and evaluation were defined based on industry inputs, document research, and study team experience.(4) During Phase IV, Reporting, a report and briefing were prepared to consolidate the study results.

ES-1

Study Focus

The focus of this study was on best practices in developmental test and evaluation (DT&E). DT&E can be defined as T&E conducted throughout the acquisition process to assist in the engineering design and development process and to verify the attainment of technical performance specifications and supportability objectives.

This study sought to answer the fundamental question: “What are the best practices in test and evaluation that are currently employed by successful enterprises to support the maturation of product design; measure the performance of the production-ready version; and verify product acceptability for the end-user application?”

Questions asked of the industry participants were designed to address the following general areas:• Why do you test?• How do you test?• When do you stop?

– How much is enough?– Who says how much is enough?

• What is the value added by T&E?• What do you consider your T&E best practices? Why?

The overall list of potential discussion topics with industry is shown in Appendix B; however, not all of the participants chose to discuss all of the topics. For one reason, the more a company’s best practices might be perceived as contributing to its competitive advantage, the more they might be viewed as proprietary intellectual property and less appropriate for open discussion.

Challenge

The test and evaluation process in the Department of Defense is highly structured and well documented, starting with the DoD 5000-series acquisition directives and continuing down through implementing Service and Agency regulations to detailed guidelines and handbooks for planning and conducting tests. Although there are some variations in how T&E is accomplished among DoD Components, a high degree of similarity exists, despite the differences in Service cultures, missions, and platforms.

In sharp contrast, there are few analogous, industry-wide T&E standards for commercial enterprises; hence, there are enormous variations in T&E practices from industry to industry, from company to company within an industry, and even from product to product within a company. In fact, a single company may have different approaches to testing from department

ES-2

to department or team to team. Therefore, an initial concern was that the research might reveal few, if any, common T&E practices, let alone best T&E practices.

During the course of the visits to the companies listed in Appendix A, however, it became clear that certain practices were followed by many, if not all, of the participating commercial enterprises. These were assessed to be best practices if, in the judgment of the study staff and the companies employing them, they: (1) added significant value to the process by which a product was created; (2) helped create a better product in a cheaper, faster manner; or (3) contributed in a traceable way to the success of the company.

Study Results

While the study team found no “silver bullets,” it did identify some twenty practices used by commercial enterprises that are relevant to ODTSE&E business practices. These practices have been grouped under the categories “Policy,” “Planning,” “Test Conduct,” and “Test Analysis.” Because of the differences in environments, policies, and objectives between DoD and commercial enterprises, the practices were examined for their applicability to DoD and judged feasible for implementation within the Department. The depth and breadth of the study staff’s experience in both DoD and industry formed part of the basis for this judgment.

T&E Best Practices in Commercial Enterprises

The following best practices found in commercial enterprises are potentially applicable for use by DoD organizations involved in T&E oversight, conduct, and resource allocation (in situations where these practices are not already in use).

Policy-Related

(1) Establish T&E as a fully integrated part of the development and deployment team, not a separate entity.

(2) Use permanent T&E groups rather than ad hoc organizations.

(3) Demonstrate concern for and support continuous improvements in the T&E process.

(4) Maintain a sharp focus on the product and the competition.

(5) Remember the ultimate objective is customer satisfaction.

(6) Develop and test products in incremental stages.

ES-3

(7) Use simple, practical metrics to measure progress and support program decisions.

Planning-Related

(8) Use templates to shorten T&E programs.

(9) Adapt test plans to reflect changing needs.

(10) Involve the T&E function early in the development cycle.

(11) Design testability into products early.

(12) Ensure that requirements are consistent with the available technology.

(13) View investments in T&E as necessary product development expenses, and budget accordingly.

(14) Follow a logical sequence of activities: i.e., develop the evaluation plan before the test plan, and prepare the test plan before conducting the test.

Test Conduct-Related

(15) Develop “fast-track” procedures to minimize product development cycle-time.

(16) Use information technology to facilitate T&E.

(17) Aim for test adequacy, not perfection.

(18) Minimize test specifications and documentation.

Test Analysis-Related

(19) Make constructive use of test failures.

(20) Ensure that the test organization reports high enough in the management hierarchy to be effective.

DoD has recognized the need for best T&E practices, although prior initiatives have not necessarily sought to borrow from commercial practice.

Discussion

The study team found a number of common attitudes toward T&E among the commercial enterprises interviewed. They included the following:

(1) Commercial enterprises test to add value to a product, specifically for reasons directly related to product marketability (not a DoD function) and safety.

ES-4

(2) The T&E function in commercial enterprises is closely integrated into the overall product development process, as is the case with most successful DoD programs.

(3) All program levels maintain a customer awareness; this includes technical staff members, not just management and marketing personnel.

(4) In most commercial enterprises, the manufacturer executes product development testing with little or no customer involvement.

(5) Commercial enterprises exhibit considerable flexibility and reflect an ability to “turn on a dime” in response to changing demands.

(6) Commercial enterprises use technology for practical reasons, to enhance the “bottom line.”

(7) The “product” is more important than the “process.”

(8) The suitability (e.g., reliability, availability, and maintainability) of a system is at least as important as its effectiveness.

(9) Commercial testers are generally accorded significant responsibility for the test process.

(10) Competition continues throughout a product’s life; it does not end with the award of a contract for development, as is usually the case in the DoD.

(11) Compared with the DoD, commercial enterprises are more willing to identify additional funds for a program than to permit a schedule delay.

T&E Best Practices in the DoD

During the course of the study, it was noted that a number of best practices in the conduct and management of T&E are already in use within DoD. Observations based on those best practices include the following:

(1) The T&E programs designed for prototype demonstrations and competitive fly-offs are both effective and efficient in their use of time and resources.

(2) The streamlined T&E programs often employed in “black” programs are usually quite successful.

(3) The application of new technology has improved test data acquisition and analysis.

ES-5

(4) Past emphasis by the DTSE&E office on the development and use of T&E tools (e.g., modeling and simulation; automated test planning and review; and the simulation, test, and evaluation process) is paying dividends.

(5) T&E is a valuable tool for managing program acquisition risks.

Conclusions

As a result of the study efforts, the team concluded:

(1) Some commercial best practices can be applied to DoD test and evaluation. The T&E best practices presented in this report are all assessed as applicable for employment within the DoD.

(2) Studies supported by various DoD offices have already identified some best practices, but they may need to be communicated more effectively.

(3) Emphasis could be increased on reducing the time required for DoD test programs (“test cycle-time”).

(4) More focused management oversight may result in better programs.

(5) The DoD acquisition and test cycle is still structured for major new weapons system developments rather than evolutionary upgrades, which are becoming an increasingly significant workload for the DoD test community.

(6) This is an opportune time for the DTSE&E office to support best practice initiatives.

Recommendations

The DTSE&E office is in a unique position to continue to be the DoD catalyst for the development and promotion of developmental T&E best practices, tools, and facilities. It is recommended that the DTSE&E consider actions leading to the adoption and implementation of the best practices in developmental T&E based on commercial and DoD experience.

Two areas merit particular attention in the near term: (1) test cycle-time reduction through the use of streamlining and appropriate “fast-track” or accelerated procedures (as, for example, the pharmaceuticals industry seeks to do in obtaining FDA approval); and (2) T&E process improvement.

While the above two areas deserve high priority, the DTSE&E office might also consider the following areas when examining ways in which

ES-6

the DoD developmental T&E process could be improved. (The commercial best practice most relevant to the recommendation is listed in parentheses at the end of each paragraph.) It is recommended that the DTSE&E office:

(1) Use its influence to promote commercial attitudes toward excellence within the DoD. These include product (system) quality, awareness of the competition (threat) and a commitment to customer (warfighter) satisfaction. (#4: Maintain a sharp focus on the product and the competition; and #5: Remember the ultimate objective is customer satisfaction.)

(2) Promote the use of standardized test templates and protocols for similar systems undergoing the same general kinds of testing, as for example the communications equipment industry does for new generations of cellular phones. (#8: Use templates to shorten T&E programs.)

(3) Examine methods by which the approval processes for Test and Evaluation Master Plans and detailed test plans might be shortened and accelerated. (#9: Adapt test plans to reflect changing needs.)

(4) Continue to emphasize the value of early involvement by the T&E community. (#10: Involve the T&E function early in the development cycle. #11: Design testability into products early.)

(5) Continue to insist that test planning consider the needs of the system evaluator and that testing follow an approved plan. (#14: Follow a logical sequence of activities, i.e., develop the evaluation plan before the test plan, and prepare the test plan before conducting the test.)

(6) Explore additional ways in which rapidly emerging information technology can be used to make T&E better, faster, and cheaper. (#16: Use information technology to facilitate T&E.)

(7) Continue to scrutinize Test and Evaluation Master Plans and detailed test plans to ensure that testing will generate sufficient information to address the critical issues while at the same time avoiding the expenditure of time and resources on nonessential data. (#17: Aim for adequacy, not perfection; and #18: Minimize specifications and documentation.)

Although not every identified best practice contributed explicitly to the above recommendations, all are considered applicable to DoD developmental T&E and worthy of consideration by the DTSE&E office. In particular, it is the opinion of the study team that there are no

ES-7

provisions in the DoD 5000-series acquisition directives or other T&E policy documents that prevent adoption of these practices.

ES-8

Section IStudy Framework

Utility of Best Practices

Commercial enterprises have long recognized the importance of identifying “best practices,” i.e., methods and techniques that successful companies find effective in cutting their costs and increasing their profits. Today, more and more companies are engaged in “benchmarking” to compare their products, services, and processes against those of other relevant organizations, and to determine how their results measure up against the “best practices” of others. Although organizational structures vary throughout diverse industries, many organizations perform similar business processes. Large and small corporations, government agencies, and even non-profit associations, share common business practices and can learn from one another.

Government organizations, in particular the Department of Defense (DoD), have sometimes been criticized for producing systems that reach the customer (user) late, cost more than originally anticipated, do not function as expected, and employ outmoded technology. Test and evaluation (T&E)1 can play an important role in minimizing such situations by assisting the Department in developing, acquiring, and deploying its systems in a manner that makes such criticism invalid.

Study Objective

In April 1998, the Deputy Director for Systems Assessment in the Office of the Director for Test, Systems Engineering and Evaluation (ODTSE&E) within the Office of the Secretary of Defense (OSD), commissioned this study to identify and examine the “best practices” used by industry and government organizations in the conduct of their developmental test and evaluation activities. The purpose of the study, conducted by the Science Applications International Corporation, was to assess the potential applicability and utility of these “best practices” to the business processes of Department of Defense test and evaluation.

From the outset of this study, it was acknowledged that some of the identified commercial best practices might already be in use (to some degree) at various locations within the DoD. For this reason, it was

1There is no one term used by commercial enterprises to describe the function that the Department of Defense calls “test and evaluation.” The function is known variously by such names as “quality assurance,” “quality control,” or “verification and validation.” For clarity within this study, the nomenclature “T&E” is used to apply to both government and industry, thus the reader will see the terminology “commercial T&E.”

I-1

recognized that some of the study’s recommendations might be more useful at some DoD organizations than others.

Study Methodology

The study was conducted in four phases: Literature Survey, Analysis, Synthesis, and Reporting.

(1) During Phase I, Literature Survey, a bibliography of over 150 citations was created and a study library assembled.(2) During Phase II, Analysis, documents were reviewed, information requirements defined, and data gathered from selected industry participants. The commercial enterprises represented a spectrum of industry niches: high-technology air vehicles and electronic systems, large hardware manufacturers (aircraft and ships), large and small software companies, telecommunications, pharmaceuticals, consumer electronics, clinical trials, and commercial testing. The specific companies that participated in the study are listed in Appendix A.(3) During Phase III, Synthesis of Best Practices, the best practices in test and evaluation were defined based on industry inputs, document research and study team experience.(4) During Phase IV, Reporting, a report and briefing were prepared to consolidate the study results.

Study Focus

The focus of this study was on the best practices in developmental test and evaluation (DT&E). DT&E can be defined as T&E conducted throughout the acquisition process to assist in the engineering design and development process and to verify the attainment of technical performance specifications and supportability objectives.

The topic of this study falls in the intermediate area of practices. It addresses neither broad T&E policies nor the details of “how to conduct a test,” at the upper end and lower ends of the hierarchy shown in Figure I-1.

I-2

Figure I-1. This study focuses on the intermediate area of practices in a hierarchy of T&E constructs.

If the study focus were to be condensed into a single basic question, it would be:

“What are the best practices in test and evaluation that are currently employed by successful enterprises to:(1) Support the maturation of product design?(2) Measure the performance of the production-ready version?(3) Verify product acceptability for the end-user application?”

The word “successful” is emphasized in the above question, since “best practices” conclusions are usually based on winners, even though “lessons learned” studies from failed ventures can often be very helpful. The commercial enterprises selected to participate in this study are generally regarded as successful in their niches, and their best practices are, therefore, well worth examination.

Key Questions

Questions asked of the industry participants were designed to address the following general areas:• Why do you test?• How do you test?• When do you stop?

– How much is enough?– Who says how much is enough?

• What is the value added by T&E?• What do you consider your T&E best practices? Why?

I-3

From these initial, generic queries the study team developed a detailed set of questions that were used to guide discussions during the interview sessions with industry representatives. The complete list of questions is provided in Appendix B. Essentially, they addressed the following areas:

• Why does your organization conduct test and evaluation? – To support design qualification? – Quality assurance? – Verification and validation? – Technology assessment? – Response to government requirements?– Liability limitation? – Assist development? – Manage risk? – Define limitations? – Support marketing decisions? – Provide information to create advertising materials and address

customer questions?• Typically, what percentage of a product’s development cost is spent

on testing and evaluation? – How does that break down as to test articles, test facilities, and

instrumentation, and labor? – Can you quantify the value added by T&E?

• Do you have a permanent T&E organization, or is an ad hoc team created for each product? – What is the reporting structure for the T&E activity, and what

degree of independence does it have? – To whom does the “chief tester” report?

• How do you decide how much testing is enough? – What are the criteria used?– Who makes the decision?

• What are the most significant technology challenges to your T&E programs? – What technologies have you exploited to enable and assist your

T&E programs? – In particular, what use do you make of modeling and simulation to

support your T&E programs?• What do you consider to be the best T&E practices of your

organization? – Can you relate them to the overall success of your products?

Because of wide variations in size, style, product lines, and culture, not all questions were appropriate for every commercial enterprise. The questions

I-4

were not used to script a formal, “question-and-answer” session, but rather to help define the types of information sought. When they were invited to participate in this study, companies were assured that their responses would be handled on a nonattribution basis; thus, individual answers are not reported. In addition, “company privacy” was respected, and no proprietary information was discussed.

Challenges

In the effort to compile best commercial practices in T&E, the study team faced a number of challenges. Foremost among them was the absence of uniformity in the ways in which commercial enterprises conduct test and evaluation. The DoD T&E paradigm is highly structured and well documented, starting with the DoD 5000-series acquisition directives and continuing down through implementing Service and Agency regulations to detailed guidelines and handbooks for planning and conducting tests. Although there are some variations in how T&E is accomplished among DoD Components, a high degree of similarity exists, despite the differences in Service cultures, missions, and platforms.

In sharp contrast, there are few analogous, industry-wide T&E standards for commercial enterprises; hence, there are enormous variations in T&E practices from industry to industry, from company to company within an industry, and even from product to product within a company. In fact, a single company may have different approaches to testing, from department to department or team to team. This makes isolating common commercial practices very difficult, let alone identifying best practices.

Many companies regard their T&E practices as an important part of their overall product development process and hence proprietary in nature. Since the success of their products depends, in part, on the effectiveness and efficiency of their T&E practices, especially in the crucial area of “time-to-market,” some commercial enterprises are reluctant to share information on testing processes they regard as contributing to their competitive position. This is especially true when a company has invested significant resources to develop a formal, documented process.

In some highly competitive industries (e.g., pharmaceuticals), better, faster, and cheaper T&E methods are also very proprietary, and hence not amenable to disclosure. In addition, sources tend to be less candid concerning areas in which they view their performance as less than satisfactory. The better defined their T&E processes, the better able the

I-5

sources were to discuss T&E and answer the study questions, since they had obviously done more prior thinking.2

While the study team found no “silver bullets,” it did identify twenty best practices used by commercial enterprises that are relevant to ODTSE&E business practices. Because of the differences in environments, policies, and objectives between DoD and commercial enterprise, the practices were examined for their applicability and judged feasible for implementation within DoD. The depth and breadth of the study staff’s experience in both DoD and industry formed part of the basis for this judgment.

The twenty practices have been grouped under the categories “Policy,” “Planning,” “Test Conduct,” and “Test Analysis” and are discussed in Section II.

2 It is important to note that the study team encountered no difficulties attributable to a lack of cooperation by any of the participants.

I-6

Section IICommercial Best Practices

Commercial T&E Compared with DoD T&E

In some cases, the context in which commercial test and evaluation is conducted may be quite different from that of the Department of Defense. In other cases, the environment, pressures, and scenarios are quite analogous to those of the DoD, as for example in the pharmaceuticals industry. The test and evaluation of a new pharmaceutical product must be adequate to ensure approval by the Food and Drug Administration; in addition, the company’s own concerns regarding product efficacy and safety must be addressed. These factors would suggest a long, careful T&E program. On the other hand, the interests of shareholders and venture capitalists lie with the earliest possible return on their investments. Pressures from competitors developing similar products also reinforce the need to achieve a minimal time-to-market so that market share can be maximized. These factors lead to streamlined testing, “fast-track” T&E, and test cycle-time reduction efforts. Therefore, a balance has to be struck on the issue of “How much testing is enough?”

One objective of this study was to examine commercial T&E practices when they are unconstrained by DoD T&E policies to see how industry performs when left alone. DoD contractors are obliged to abide by the Department’s 5000-series directives, regardless of their applicability to every particular situation. Questions of interest concerned organizational structure, methodologies, and technology. It was anticipated that the answers to these and other similar questions might suggest areas where DoD practices could be enhanced by borrowing from commercial experience.

What Constitutes a “Best Practice?”

The scope of this study was limited to the sample of commercial enterprises listed in Appendix A, augmented by the literature references in Appendix C, and the experience of the study staff. Since T&E practices vary from one type of industry to another, from company to company within an industry, and even between organizations within a single company, there was concern initially that the research might reveal few, if any, common T&E practices, let alone best T&E practices.

During the course of the industry visits, however, it became clear that certain practices were followed by many, if not all, of the participating commercial enterprises. These were assessed to be best practices if, in the

II-1

judgment of the study staff and the companies employing them, they accomplished the following:

• Added significant value to the process by which a product was created;

• Helped create a better product in a cheaper, faster manner; or

• Contributed in a traceable way to the success of the company.

Of course, not every commercial enterprise followed every one of the best practices, and the list which follows is far from exhaustive. Also, the list is based on positive examples from successful commercial enterprises, not lessons learned from failures.3

T&E Best Practices in Commercial Enterprises

The following best practices found in commercial enterprises are potential candidates for adoption by DoD organizations involved in developmental T&E oversight, conduct, and resource allocation (in situations where they are not already in use). Each best practice is described in subsequent paragraphs.

Policy-Related








Planning-Related





3 A company’s success was determined based on ordinary economic measures and reputation.

II-2







(17) Aim for adequacy, not perfection.




(20) Ensure that the test organization reports high enough in the management hierarchy.

POLICY-RELATED BEST PRACTICES


Commercial enterprises focus on the successful development and deployment (creation, marketing, and sales) of products that satisfy customer needs, with T&E playing a supporting role toward this goal, as appropriate and necessary. The T&E function, with its concomitant structure, people, and facilities is never permitted to enjoy a status as a dominant or separate entity. Organizational relationships are structured to maintain an equilibrium where T&E can contribute, but not control. Without subverting the independence of the testers and their ability to expose defects, the goal is to foster cooperation and trust, not an adversarial relationship.

The T&E function is fully integrated with the other specialties in the team. This practice is encouraged by the “concurrent engineering” approach to system development and is somewhat similar to the DoD Integrated Product Team (IPT) and Integrated Product and Process Development (IPPD) concepts described in DoD Regulation 5000.2-R. T&E

II-3

involvement is expected to be substantive and early in the product development cycle. This approach promotes an emphasis on the high quality of testing early. “Constructive tension” between developers and testers is not viewed as productive or affordable, and is suppressed or held at low levels; cooperation is promoted.


All of the commercial enterprises have permanent standing organizations with responsibilities for testing and evaluation. This reflects two fundamental concepts: (1) Although the T&E function must be integrated into the overall development process, the T&E discipline has its own distinctive skill sets, techniques, tools, and attitudes. (2) It helps to have an active T&E organization that can participate at all times throughout the product development cycle. The size, composition, training, and demographics of testing and evaluation groups varies widely, however. For example, one commercial enterprise claimed that its best developers were assigned T&E duties, while others used testing as a way of training developers. Several commercial enterprises have no Quality Assurance or Quality Control organizations as such. Although they expressed a commitment to quality, they regard it as a product attribute achieved by the overall development process, including T&E, and do not view testing as a compartmented, organizational function.


All of the commercial enterprises had defined processes for product development, including the role of test and evaluation, although there was wide variation in the degree of formality. These processes tended to be “homegrown” based on the experience and needs of the particular business, rather than a textbook model.4 Several commercial enterprises spoke of the difficulty in establishing their T&E process, sometimes over the resistance of skilled individual contributors,5 but all believed some sort of formal process was important for successful product development and more than worth the investment required.

All expressed strong interest in, and commitment to, process improvement. Without a mechanism for change, even a “perfect” process will become obsolete and counterproductive as real-world conditions evolve. Process improvement efforts sometimes include “benchmarking,”

4 In particular, the various “Maturity Matrix” methodologies promulgated by the Carnegie-Mellon Software Engineering Institute were not widely used.5 This was especially true in software development.

II-4

a type of best practices program that examines how competitors or others in similar businesses carry out the same activities. During gaps between major development activities, a cadre of experienced test and information technology personnel is often assigned to improving process tools. Not surprisingly, process establishment and improvement were accomplished most easily in those commercial enterprises that were attempting to overcome serious problems in product development (cost, schedule, or performance). Conversely, process improvement was more difficult to institute in successful organizations, since the need was not always as obvious to the staff.


Not unexpectedly, every commercial enterprise demonstrates a very strong awareness of its competitors and a very strong commitment to the success of its products. These factors play directly into product development decisions and actions in a practical way. The competition serves as a kind of analog to the “threat” in the DoD paradigm, in that each commercial enterprise is trying to achieve superior performance more rapidly than its competitors.


Regardless of their product niche, organizational structure, or test and evaluation policies, every commercial enterprise’s T&E group demonstrates a keen awareness of the customer’s needs and an appreciation that these needs must be met. This sensitivity is present at all levels in the organization, not just in those nearest to management or marketing. T&E activities are ultimately, and often very directly, traceable to ensuring customer satisfaction with the product. One test manager commented, “If we [don’t do things right], we’re out of business.” Adopting a “customer mentality” is encouraged, and testers think about how a customer will, can, or might use or misuse the product. Sometimes customer feedback is formalized, as in customer beta testing of software products. With customer satisfaction driving the structure of the T&E program, any tendencies to conduct “testing for testing’s sake” or other non-value-added activities are minimized.


Many commercial enterprises regard as a crucial risk management strategy the evolutionary development of a product in small but useful increments,

II-5

with T&E conducted on each increment before proceeding to the next.6 In this way, the product never diverges too far from the proposed design, or acquires too many defects. Costly and time-consuming fixes and retrofits are thereby minimized.


Most commercial enterprises employ simple metrics to manage their development and test programs, even those with fairly elaborate and fully articulated processes. These metrics are usually derived from the commercial enterprise’s own experience and culture, and relate in a direct and practical way to product performance and cost. Very complicated “textbook” process measures are rarely used. Decisions regarding “How much testing is enough?” or “When should we stop testing and start shipping?” are not often made solely on the basis of statistical analysis of confidence, except in certain highly controlled “laboratory” scenarios. Usually such decisions are based on a combination of “engineering judgment” and “business case analysis,” with one eye on the market and the other on the competition.

PLANNING-RELATED BEST PRACTICES


Commercial enterprises that create products in a series of models or versions which are different, but of the same general type, often use templates to shorten the planning, execution, and analysis phases of T&E programs. Speed of testing is an absolute competitive advantage, provided quality of testing can be sustained at the same time. Once created, standard T&E templates for a product type are archived and reused. Use of standard templates also permits the valuable comparison of T&E results from one product generation to the next, since standard processes, environments, and scenarios are employed. One testing manager, however, warned that slavish use of templates can lead to a “cookbook” mentality for T&E and suggested that they not be used without careful analysis of the specific system or product and its peculiar operating environment. One commercial enterprise template-user even prepares a formal “Concept of Operations” document modeled on the DoD format.

6 This practice is particularly suited to software-intensive products, where some commercial enterprises employ a “daily build” or “nightly build” approach.

II-6


Several commercial enterprises boasted of their ability to “turn on a dime.” By this they meant adaptability, or flexibility, or agility—the ability to change directions quickly as unforeseen circumstances dictate. This is different from the simple ability to conduct T&E rapidly. Adaptability permits the deletion of planned testing if events show it to be unnecessary, or the addition of testing to meet unanticipated needs. It also permits T&E to focus on areas of higher risk. Adaptive test planning requires a relatively low level of approval for changes and hence decentralized authority for T&E changes.

The commercial practice of adaptive test planning contrasts sharply with the DoD norm, where T&E Master Plan and detailed test plan preparation and approval involve a cumbersome, protracted process. The lengthy approval cycle, which typically entails sign-off by numerous organizations within one or more Services prior to final OSD approval, results in an extreme reluctance to change these documents unless absolutely necessary. The exceptions are instances where major program perturbations have occurred requiring urgent action. When compared with industry, DoD rarely demonstrates the ability “to turn on a dime.”


Most commercial enterprises believe that testing and evaluation considerations need to be addressed very early in the product development cycle, as early as the design phase, or even the conceptual phase. They believe that early insights provided by T&E results can contribute to evolving product design, improve product performance, reduce risks, eliminate surprises, and minimize costly budget and schedule overruns. They want to know as much as possible about the product as soon as possible.


When testers are involved early in the development cycle, the opportunity exists to design into the product features that can facilitate subsequent T&E activities. Several commercial enterprises believe that designing testability into the product, where possible, is a valuable practice, which increases the types and amounts of test data available and decreases the length and cost of the T&E program. Ultimately, this can improve product performance, profitability, and time-to-market.

II-7


In the DoD system acquisition paradigm, requirements are expressed for a system to meet a set of mission needs; technology is then sought as an “answer” to the requirements “question.” This technology is then “systemized” by the acquisition process outlined in the 5000-series regulations.

In the commercial product development paradigm, however, the technology is usually invented first; requirements and specifications for a product exploiting that technology are then created based on a market analysis, and the product development cycle begins. The commercial model thus resembles the DoD Advanced Concept Technology Development (ACTD) process. Figure II-1 juxtaposes the DoD and commercial paradigms.

Figure II-1. Commercial enterprises and DoD view technology andrequirements from different perspectives.

The difference between the DoD and commercial paradigms affects their respective conduct of T&E in a subtle way. In testing a DoD system against its requirements, two basic failure modes can be revealed: (1) a failure of the technology, i.e., an inappropriate selection of the wrong technology, and (2) a failure of the system, i.e., an incompetent systemization of the right technology. But in the commercial paradigm, characteristics of the technology (including limitations) have already been incorporated into the requirements, so that commercial testing mainly has to address how well the product meets the needs of the market.

In the DoD paradigm, the key questions to be addressed by testing are:

• Was an appropriate technology selected to meet the system requirements?

• Was the technology successfully incorporated into a system which satisfies the user?

II-8

By contrast, the key question in the commercial paradigm is only the second one:

• Was the technology successfully incorporated into a product which satisfies the customer?

The DoD approach thus tends to involve more risk because it may ask an inappropriate or immature technology to do more than it has been shown capable of, and then may compound the problem by attempting to “systemize” that technology in an ineffective way.


Most of the commercial enterprises in this study acknowledged the importance of investing in test resources, including facilities, laboratories, instrumentation, articles for test, and test data processing assets. The return on investment in test resources is viewed as an element of the overall value added by T&E and the cost of testing as a necessary business expense of product development. One commercial enterprise, in fact, cited its extensive and expensive investment in hardware and software test laboratories and field instrumentation as its “Number 1 Best Practice” and a real discriminator versus its competitors.7

The DoD has historically lagged industry significantly in terms of test facility improvement and modernization and in replacement of physical plant assets. When commercial enterprises were asked what percentage of product development cost would typically be attributable to T&E (including labor), the responses ranged from approximately 15% to approximately 50%.8 Understandably, the lower percentages came from hardware-intensive and the higher from software-intensive products. The 50% figure was offered by a software producer that employs the practice of pairing one tester with each programmer.

(14) Follow a logical sequence of activities: develop the evaluation plan before the test plan, and prepare the test plan before conducting the test.

Most commercial enterprises are keenly aware of the costs of testing (facilities, test articles, labor, etc.) and the impact T&E can have on increasing product time-to-market. They are definitely not in the business of unwarranted “testing for testing’s sake,” which would ultimately have to be paid for in product pricing. They are generally not driven by

7 This commercial enterprise also used its facilities to test all its competitors’ products against industry standards.8 Remarkably, the shipbuilders and the airplane builders quoted precisely the same number.

II-9

independent, internal testing organizations to conduct unneeded T&E without regard for the return on investment. They also tend to be concerned that testing provide “answers, not numbers,” i.e., practical information directly useful in product improvement, fault correction, and risk reduction. For this reason, an evaluation plan is usually created to define what (minimal) data is needed from testing and how the data will be used. The evaluation plan is often based on a template. After the data requirements are thus defined, a test plan for obtaining the data is generated. The evaluation and test plans may be combined in the same physical document, but their distinct purposes are acknowledged. Ideally, the test plan is coordinated and approved before testing begins, so that no effort is wasted.

TEST CONDUCT-RELATED BEST PRACTICES


One company representative spoke of spending millions of dollars to develop processes that would expedite the review and approval of products by a federal agency with oversight and regulatory authority for the safety and efficacy of such products. Such “fast-track” procedures are tailored to the specific nature of the product in development and the federal agency criteria that must be met. There is a degree of risk involved in developing and proposing these procedures, inasmuch as they depart from “textbook” procedures, which have already been approved for use on previous products. The potential payoff can outweigh the risk, however, because of the premium placed on reducing time-to-market, especially in an industry such as pharmaceuticals.9 The greater the delay in bringing a new product to market, the longer stockholders, venture capitalists, and other investors must wait to realize a return on their money. Delays also offer a competitor the opportunity to place its product on the shelves first and obtain an initial market share advantage.

The issue of first-to-patent is also a crucial matter in industry. DoD Regulation 5000.2-R makes this sort of streamlining available for DoD systems, at least in principle, under its “tailoring” provisions. A strong “business case” can be made for investing in streamlined T&E procedures where urgency of readiness issues are present. Emphasis on reducing development/test cycle-time also expedites the identification and correction of problems, which are to be expected in any high-technology development program.

9 “Speed is everything in this industry,” said Pfizer CEO William Steere in a May 1998 interview with FORTUNE magazine.

II-10


DoD testers are used to viewing advances in technology (e.g., stealth) as challenges, since the weapon systems that exploit them are subjected to T&E to demonstrate their effectiveness and suitability. Technology often plays an equally important role as an enabler of testing— information technology is perhaps the best example. In fact, T&E could be considered a branch of information science, since T&E involves the creation, processing, and interpretation of data. All of the commercial enterprises participating in this study have aggressive programs to take advantage of advances in information technology, including automated test tools, sophisticated simulations, and networking. One company has a large test group on the other side of the world that is continuously networked with developers and testers at headquarters. This allows everyone access to 100% common information in real time and permits the sharing of ideas regarding problems and solutions. Another company requires developers and testers to use the software product under development in their daily work, so that they can gain familiarity and discover problems.10 For test programs on products that entail larger volumes of data or extensive data processing and analysis, emphasis is placed on improving data processing tools and comparing actual with predicted results.


Related to the question of “How much testing is enough?” is the practice adopted by several commercial enterprises of consciously not aiming for 100% detection of faults.11 By marketing or deploying products that are “good enough, but not perfect,” a commercial enterprise can place a usable product in customers’ hands early, preempt competitors and win market share, and initiate and maintain a revenue stream. This also avoids wasting resources pursuing exotic and irrelevant deficiencies not germane to customer satisfaction, and running afoul of Pareto’s Law by spending 80% of the available assets trying to fix 20% of the problem set.12

The risk assumed, however, is that a product with deficiencies or missing promised features will create customer disappointment and regret, with consequent loss to the commercial enterprise’s reputation and future sales; in the extreme case, warranty, safety, and liability problems could arise. As is the case with DoD, commercial enterprises that are regulated by

10 The company jokingly referred to this as “eating your own dog food.”11 This practice is particularly applicable to software-intensive systems, where successive builds or versions are normally planned as part of the product development strategy, with defects in a current version correctable in subsequent releases. This practice also provides a solution to the problem of “requirements slip,” since features planned for a given build, but not achieved, can be easily deferred to a later build.12 Vilfredo Pareto (1848–1923), an Italian economist, postulated that almost every situation can be viewed as an 80/20 ratio.

II-11

government agencies, e.g., the Federal Aviation Administration or the Food and Drug Administration, may not have the luxury of employing this practice. When used, the commercial enterprise typically prioritizes defects (or missing features) by their potential impact on intended customers (for example, by multiplying the probability of occurrence times a measure of the harm that might be caused). The deficiencies that are considered unacceptable by the user are remedied before the product is shipped.


Commercial enterprises performing both DoD and commercial work of a comparable nature commented on the “over-specification” and “over-documentation” required for the DoD. They often feel they are being told not only what to build, but how to build it in a degree of detail that is unnecessary, sometimes inappropriate or even contradictory, and stifling of innovation or attempts to do T&E “better, faster, and cheaper.” The documentation workload is often viewed as non- or counterproductive. Left to their own devices, commercial enterprises claim to be able to do superior work with less resources. One company, which provides similar products to both DoD and commercial customers, believes the quality and performance are equal for both clients, but with significantly less cost and time on the commercial side; hence, the commercial client enjoys a lower price and the commercial enterprise a higher profit margin. Commercial organizations see the risk of product failure and the need to make a profit as at least as strong, if not more powerful, motivations to succeed.

TEST ANALYSIS-RELATED BEST PRACTICES


In the commercial world, test events are regarded as information-generation experiments paid for in terms of labor, facilities, and test articles, with the emphasis on “What did we learn?” As is the case within the DoD, care is taken to collect and preserve test results in a database to which all members of the test team have access. Testers and evaluators are expected to contribute to the “analyze” and “fix” activities of the “test-analyze-fix” process, not just the “test” part. Test failures drive redesign and retest. Although highly sensitive to schedule pressures, T&E programs are not usually “schedule-driven” in the sense that test plans are rigidly followed without addressing the consequences of test failures. On the other hand, test failures rarely completely terminate product development, in light of the sunk costs to that point.

II-12

(20) Ensure that the test organization reports high enough in the management hierarchy to be effective.

For the most part, commercial test organizations tend to be structured relatively informally. Their focus is directed toward the product and the competition, and far less toward ownership of the process. One result is that information regarding test results is communicated more informally (and more rapidly). Nevertheless, most commercial enterprises acknowledged that a formal T&E reporting channel has to exist at a level high enough to prevent the suppression of bad news and to permit action to be taken based on test results, if necessary (including the expenditure of resources). The inclusion of the T&E organization as part of the development team facilitates the flow of information vertically as well as horizontally, but it does not fully replace the discipline and responsibility promoted by the formal pipeline.

II-13

II-14

Section IIIDiscussion

Common Characteristics Across Industries

Because of the diversity of commercial test and evaluation practices, it is impossible to formulate strict generalizations about how industry performs the developmental T&E function. The study team did, however, observe some common characteristics across the commercial enterprises it interviewed. When compared with the DoD developmental T&E process, many of these characteristics suggest areas in which the latter process might be improved.

Focus on Adding Product Value

The first observation is that commercial enterprises test and evaluate for eminently practical reasons: to place safe, effective products in the hands of their customers and to make a profit in so doing. They avoid any need to create documentation, establish facilities, or engage in testing that does not add value to the product or increase its profitability. This perspective imposes obvious constraints and limitations on the type and amount of testing.

T&E Function Integrated into Product Development Process

The testing function is organized in many different ways in commercial industry. Sometimes “Test” and “T&E” are absent from the organizational title, the function being subsumed under “Quality Assurance,” “Quality Control,” “Verification and Validation,” or some other name.13 By and large, the basic components of the classical test planning process14 are carried out, but tailored to the economics of the specific product. Test conduct and reporting tend to be accomplished by the same personnel responsible for test planning. In particular, “T” and “E” tend not to be organizationally separated. The T&E function is usually closely integrated into the overall product development process, as is the case with most successful DoD programs; there is very little of a “we versus they,” adversarial mentality, since that is not considered affordable in commercial enterprise. As a consequence, testers tend not to be stigmatized as expensive outsiders who delay or disrupt the otherwise orderly progress of a product to market.15 Most organizations

13 The phrase “T&E” is rare outside companies with DoD contracts or experience, although the content of their work is similar or even identical. Some companies had to be informed that what they were doing would be recognized as “T&E” by the DoD.14 The classical test planning process goes from Strategy to Concept, to Master Plan, to Design, and finally to Detailed Plan.15 One software organization even claimed that its career path involved promotion from developer to tester.

III-1

acknowledged that the “chief tester” assigned to a product has to report high enough in the hierarchy to prevent the unwarranted suppression of bad news about that product.

Technical Personnel Maintain Customer Awareness

An interesting common characteristic among the companies participating in this study was a high degree of “market awareness” among the technical personnel, not just the marketers and managers. All levels seemed to be very well informed about potential customers (DoD analog: “system users/operators”) and competitors (DoD analog: “enemies”), as well as competing products (DoD analog: “threat systems”). This awareness is a major benefit in maintaining the focus of the product development staff.16 It also conditions attitudes about product and process security and confidentiality, since it is important that sensitive information not make its way into competitors’ hands or be disclosed prematurely to customers. Occasionally, a company will use its T&E program as the basis for advertising, usually with the theme that the robustness of the test program should create confidence in the reliability, safety, and performance of the product.17 (In a similar way, the DoD makes public the “good news” from successful test programs.) Presumably, T&E results would also be made available if needed to counter a challenge to product claims or to support litigation in liability or other legal cases.

Product Development Testing With Little or No Customer Involvement

In most commercial enterprises, the T&E group exhibits a keen awareness of the customer’s needs and fully appreciates the importance of meeting those needs. At the same time, however, the manufacturer usually executes product development testing with little or no customer involvement.

Commercial Enterprises Are Flexible

Commercial enterprises understand the differences between an experiment to obtain information, a demonstration to show or prove something, and a test to establish success or failure; and they use each appropriately as driven by economics. These distinctions are sometimes blurred in the DoD when T&E events may be encumbered by political or management issues. Commercially, the price of failure is well understood, since a major failure in a key test can mean expensive redesign or termination of

16 Continual awareness of the “customer” (system user/operator) and “competitor” (threat) is not always evident at all levels in the DoD system acquisition environment, to the detriment of bottom-line “profit” (readiness).17 For an excellent example of this practice, see the two-page automotive ad just inside the front cover of the August 17, 1998, issue of FORTUNE magazine.

III-2

product development, with consequent career impacts or even loss of the company. Thus, as in the DoD, commercial enterprises are highly motivated to do testing right. Compared with the DoD, commercial enterprises usually make less of a distinction between evaluation supported by actual test results and assessment based on other sources. They are usually flexible enough to take the approach dictated by judgment and economics, especially early in the product development process. This contrasts with the difficulties that attended the introduction into the DoD in the late 1980s of the concept of Early Operational Assessment, when testers responded reluctantly when asked to make assessments in the absence of complete test data.

Technology Is Applied When Practical

For the most part, commercial enterprises seem quick to exploit such high-technology T&E enablers as modeling and simulation, high-performance computing, distributed networks, and artificial intelligence, when and where a business case can be made for doing so. Industry invests in technology solutions when they can be taken to the bottom line and is unlikely to be driven by technology advocates seeking to promote particular technologies for their own sake. Industry uses information technologies widely to facilitate T&E, although there are few quantitative metrics to prove that increases in productivity accrue. One software developer considers it a best practice that employees are required to use their own software as it is being developed to gain practical familiarity with it and aid in the debugging process.

“Product” Is More Important than “Process”

Commercial enterprises focus on the product, not the process for developing it. Process exists only to the extent that it facilitates the “better, faster, cheaper” creation of product. DoD Regulation 5000.2-R promotes the adoption of a strong systems engineering process underlying systems acquisition. Textbook systems engineering practices are used to varying extents in industry, showing greater prevalence as the complexity of the system or product under development increases, along with the greater need to “keep track of everything.” Many companies develop their own “homegrown,” ”common-sense” approaches to the systems engineering function, perhaps unaware that they have reinvented the discipline for their purposes. These tailored systems engineering processes, however, have the advantage that their owners understand them; they have no superfluous, wasteful, extra features; and they can be modified easily and cheaply to accommodate change. In commercial practice, systems engineering is viewed more as providing “engineering

III-3

insight,” than “acquisition oversight,” the DoD perspective. Commercial use of concepts that are similar to the DoD’s “Integrated Product Teams” and “Integrated Product and Process Development” is natural and widespread, although different terminology may be employed.

Suitability Is At Least as Important as Effectiveness

In the DoD acquisition process, the so-called “ilities” (e.g., reliability, availability, and maintainability), or system characteristics related to how “suitable” the system is for placement into service, are often, in practice, considered less important than those performance characteristics related to how “effective” the system is in carrying out its mission. Part of the rationale for this is that the “ilities” are viewed as correctable through component or subsystem improvement as the system life-cycle proceeds, whereas performance capabilities must be demonstrable at or before approval for production and deployment.

In the commercial world, however, the “ilities” are given equal weight with performance characteristics, since both affect customer acceptance of the product. Adequate T&E of all product elements—hardware, software, and human factors or user interfaces—is considered crucial, especially for consumer products. This same concern with customer acceptance is reflected in such practices as beta testing of software by potential purchasers, product trials, and test marketing.

Commercial Testers Have Significant Responsibility

Testers in commercial industry are generally accorded significant responsibility and “ownership” of the test process, given the importance to business decision-makers of accurate test results. The motivation of, and training for, testers tends to be commensurate with that responsibility, and general management spends less time and money overseeing and monitoring their conduct of testing.

Competition Continues Throughout Life Cycle

A comparison of DoD T&E with commercial T&E is affected by their fundamentally different business models: DoD is essentially its own customer. The commander of an Army armored unit does not shop for the make and model of tank that best suits his needs. The commander is issued the tank the Army itself has developed, tested, and procured, the prime contractor competition having already taken place much earlier in the acquisition process.

III-4

The consumer of commercial products, however, enjoys the benefits of industry competition right up to the point of sale, where he or she has the opportunity to make an independent assessment of comparative product value, including that added by T&E. If a consumer were to acquire a new automobile the way the Army acquires a new tank, he or she would have to develop a set of requirements for a new car; allow Ford, Chrysler, and General Motors to bid their designs; select a winner (based, perhaps on the evaluation of prototypes); and then conduct T&E on the (single) finished product to ensure its effectiveness and suitability with respect to the original requirements. In the case of the tank, the use of T&E as part of the competitive process would end after the winning design is selected; in the commercial world, competition continues through the point of sale.

Schedules Have Priority

An interesting asymmetry between DoD and industry practices exists with regard to T&E cost and schedule considerations. In commercial enterprises, the product development schedule is extremely important, because of its implications for time-to-market and hence market share. Therefore, funding for test facilities, test articles, instrumentation, labor, and other test support categories is made available so that testing can be completed on time, and the schedule can be met.

In DoD programs, on the other hand, funding for T&E support is often inadequate in light of the goals of the program. However, in the absence of the market pressures experienced on the commercial side, the schedule is adjustable, so the program is delayed to accommodate available funding. Thus, the DoD tends to cope with T&E by permitting schedule variance under a fixed resource regime, whereas commercial enterprise prefers to add resources so that the schedule remains fixed.

In addition, because of an annual federal budget process that is affected by continual political and bureaucratic machinations, funding for DoD programs tends to be somewhat volatile. Requirements and specifications are less stable in DoD programs also, with industry better disciplined about freezing and maintaining baselines. When product capabilities are late in meeting requirements, commercial enterprises are likely to move the unmet requirements into a subsequent block or build or version, in order to preserve the sanctity of the schedule and the ability to ship on time. This is particularly true in the case of software-intensive products.

III-5

Software Testing

The need to follow a rigid software development process is often at odds with the “hacker” or “cowboy” culture to which many of the most talented programmers belong. Many software developers regard themselves as practitioners of an art, rather than technicians whose efforts can be made to conform to a process. The challenge for the enterprise, then, becomes one of preserving the creative hacker culture while controlling and coordinating software feature development to meet cost and schedule objectives.

In the May 1998 issue of Software magazine, John H. Mayer discusses “The Right Way to View Testing.” Based in part on comments from some commercial software developers (not those participating in this study), he offers the following observations:

• Software quality is rarely a selling point; it’s usually viewed as an expense.• In fact, testing is often considered an impediment to time-to-market goals.• Sixty-two percent of all organizations don’t have a formal quality assurance or test organization.• A large majority place a much higher priority on meeting schedule deadlines than producing high quality software.• But the enlightened understand that extensive testing up front leads to fewer bugs when it comes time for product release.• The key is to recognize quality assurance and test as integral parts of the overall development process.• The issue is not improving developer productivity or finding bugs.• The issue is being able to deploy an application successfully on time with minimal risk and maximum confidence.• To be avoided: a functionally oriented development environment where all the code development is done up front and a low-quality product is then “thrown over the wall” to a test organization.• To be encouraged: building in quality during development via the use of rules, standards, and metrics to minimize faults downstream and shorten/simplify testing.• Moving test up to the front of the development cycle more than pays for itself.• The best way to force early testing is to schedule incremental or evolutionary releases.• The test group needs to be tightly integrated with the development and business functions and have significant input into the development schedule.• The best way to force integration is to organize by project teams with all functions, including testing, represented [Integrated Product Teams, in DoD parlance]• The use of automated testing tools can significantly reduce the “delay” introduced by the need to test and help to focus on the highest priority areas.

III-6

The “best of breed” companies which participated in this study generally agree with these observations and have adopted practices consistent with them.

T&E Best Practices in the DoD

During the course of the study, it was noted that a number of best practices in the conduct and management of T&E are already in current use within DoD, specifically:

(1) The T&E programs designed for prototype demonstrations and competitive fly-offs are both effective and efficient in their use of time and resources.

(2) The streamlined T&E programs often employed in “black” programs are usually quite successful.

(3) The application of new technology has improved test data acquisition and analysis.

(4) Past emphasis by the DTSE&E office on the development and use of T&E tools (e.g., modeling and simulation; automated test planning and review; the simulation, test, and evaluation process) is paying dividends.

(5) T&E is a valuable tool for managing program acquisition risks.

DoD has long recognized the need for best T&E practices, although prior initiatives have not necessarily sought to borrow from commercial practice.

III-7

III-8

Section IVConclusions and Recommendations

Conclusions

Best Practices Can Be Applied to T&E

What can be learned from successful commercial programs to improve the DoD test process, beyond paying closer attention to the basics? While commercial practices are not all directly applicable in a DoD environment, some deserve attention. The study team concluded that there are best practices that can be applied to the “business” of developmental T&E. The primary feature of all successful test programs is not the discovery of one “magical,” new best practice; instead, it is that these programs have identified and incorporated well-known, existing best practices into their planning and execution processes. The commercial T&E best practices described in this report are all considered applicable for DoD use.

DoD Has Best Practices But Needs to Communicate Them More Effectively

From its lengthy experience in successfully developing complex weapon and support systems, DoD has generated numerous best practices in T&E. Unlike companies that have mechanisms for transferring best practice knowledge from one program to another, DoD lacks an effective method for communicating information about best practices to the individuals and organizations who can most benefit from them. Traditionally, the DTSE&E office has met the challenge and played a strong leadership role in the development and dissemination of best practices, in addition to discharging its T&E oversight responsibilities. Downsizing initiatives within DoD will make the challenge even greater in the future.

Commercial Best Practices Can Be Adapted for DoD Use

Operating under a different set of motivations and constraints from those of DoD, commercial enterprises have developed best practices in test and evaluation. Some of these best practices are remarkably consistent across widely diverse industry, organization, and product types. The best practices found in commercial enterprises include the following:

Policy-Related



IV-1






Planning-Related





(12) Ensure that requirements are consistent with available technology.










(20) Ensure that the test organization reports high enough in the management hierarchy.

If consistently applied, commercial best practices in T&E have the potential to improve the quality, productivity, and value of DoD T&E programs. Some commercial best practices (e.g., early involvement of T&E in the product development process, integration of the T&E function with other development functions, and the use of modeling and simulation) are already incorporated into the DoD T&E paradigm.

IV-2

Obstacles to the use of some commercial best practices in DoD T&E programs are largely attributable to attitudes rather than legal or regulatory barriers. Although not every identified best practice contributed explicitly to the above recommendations, all are considered applicable to DoD developmental T&E and worthy of consideration by the DTSE&E office. In particular, it is the opinion of the study team that there are no provisions in the DoD 5000-series acquisition directives or other T&E policy documents that prevent adoption of these practices.

Emphasis Could Be Increased on Reducing Time Required for DoD Test Programs

DoD test programs undergo extensive technical and safety reviews, but more attention could be paid to test cycle-time reduction; by contrast, in most commercial test programs, “schedule is king,” and significant effort is invested in minimizing the time required to complete adequate testing. There is a direct benefit to be realized in completing testing earlier or compressing the test schedule. Several basic mechanisms can be used to reduce test cycle-time:

(1) Reduce test program content.

(2) Accomplish testing more effectively/efficiently.

(3) Use test facilities and resources more intensively (e.g., multiple shifts, seven-day weeks).

(4) Eliminate duplicative testing.

(5) Budget and fund testing and test planning earlier in the program.

In most DoD test programs, the content is already at or near a minimum. In fact, it could be argued that, for some programs, test content has been reduced below an acceptable minimum. While a review of program content should not be overlooked, more opportunities for cycle-time reduction probably exist in the area of test process effectiveness/efficiency. Efficiencies can be realized by exploiting advances in information processing technology. So-called “fast-track” T&E programs are common in commercial practice, for example, in the pharmaceutical industry where lengthy “time-to-market” can have enormous costs, up to and including survival of the company.

For many major weapon system development programs, the fixed overhead cost of the test support personnel and associated infrastructure can be a greater cost driver than the variable per test or per mission cost. This latter cost includes the direct cost associated with such items as range

IV-3

support and data processing.18 Adding test support personnel to accelerate a test program can be less expensive over the duration of a development program than conducting a program stretched out over several years. More importantly, decreasing the time needed for the test program, particularly during early development, may allow for the identification and correction of deficiencies earlier in the development process, and consequently lower overall development costs.

More Focused Management Oversight May Result in Better Programs

In highly classified government programs, where access is restricted and there is a small, highly integrated test team, the number of oversight personnel is also limited. The relative success and short test cycle of these programs suggests that the levels of management oversight, which are the norm on major acquisition programs, could be reduced if greater management focus could be achieved. Lean and focused T&E oversight was the rule in the commercial enterprises interviewed for this study.

Process Not Geared to Evolutionary Upgrades

The DoD acquisition and test cycle is still structured for major new weapons system developments rather than evolutionary upgrades, which are becoming an increasingly significant workload for the DoD test community. A more flexible tailored approach deserves greater attention and study.

Opportune Time for ODTSE&E Best Practice Initiatives

The ODTSE&E could stimulate the use of T&E best practices in DoD by promoting their adoption and use. Because of ongoing acquisition initiatives related to cost, quality, and productivity in the current DoD political, organizational, and funding environment, the present time would be nearly ideal for the issuance of T&E best practice guidance.

Recommendations

Primary Initiatives

Over the past several years, the Office of the Director, Test, Systems Engineering and Evaluation has sponsored a wide variety of initiatives to improve developmental T&E tools, practices, processes, and facilities.19 This office is in a unique position to continue to be the DoD catalyst for the development and promotion of developmental T&E best practices,

18 One exception is a test program where a high-cost test asset is destroyed in every full-up test, e.g., most missile flight tests.19 Examples include automated test planning, risk management, modeling and simulation, and investments in instrumentation improvements.

IV-4

tools, and facilities. It is recommended that the Director, Test, Systems Engineering and Evaluation consider actions leading to adoption and implementation of the best practices in T&E based on commercial and DoD experience.

In particular, two areas merit attention in the near term: (1) test cycle-time reduction through the use of fast-track or accelerated procedures and (2) T&E process improvement.

TEST CYCLE-TIME REDUCTION

Commercial enterprises place a high priority on cycle-time reduction and the use of “fast-track” procedures. With few exceptions, the DoD places little emphasis on the importance of planning and providing resources for individual test programs in a manner that minimizes the test cycle-time. For each development program and its associated test and evaluation effort, it is recommended that special attention be directed early in the planning cycle (and periodically throughout the program development) toward compressing the test schedule where possible without compromising test objectives. Further, it is recommended that cycle-time reduction be addressed explicitly in any submissions to the DTSE&E relative to the Central Test Equipment Improvement Program (CTEIP).

T&E PROCESS IMPROVEMENT

The utility of any process, test and evaluation included, is largely determined by how up to date it is. Commercial organizations systematically review their processes and tools in an ongoing effort to improve their test infrastructure. They employ senior personnel in their process improvement activities and make major capital investments in more effective test support tools. They also make extensive use of metrics to evaluate expected improvements. Thus, concern and support for continuous T&E process improvement is recommended for DoD.

Other Potential Actions

While test cycle-time reduction and process improvement are extremely important and deserve immediate attention, the commercial best practices identified in this study suggest other areas the DTSE&E office might consider as it seeks to improve the T&E process. A few of them are discussed below, with the commercial best practice from which each recommendation stems indicated in italics at the end of each paragraph. It is recommended that the DTSE&E:

(1) Use its influence to promote commercial attitudes toward excellence within the DoD. These include product (system) quality, awareness

IV-5

of the competition (threat) and commitment to customer (warfighter) satisfaction. (#4: Maintain a sharp focus on the product and the competition; and #5: Remember the ultimate objective is customer satisfaction.)

(2) Promote the use of standardized test templates and protocols for similar systems undergoing the same general kinds of testing, for example as the communications equipment industry does for new generations of cellular phones. (#8: Use templates to shorten T&E programs.)

(3) Examine methods by which the approval processes for Test and Evaluation Master Plans and detailed test plans might be shortened and accelerated. (#9: Adapt test plans to reflect changing needs.)

(4) Continue to emphasize the value of early involvement by the T&E community. (#10: Involve the T&E function early in the development cycle. #11: Design testability into products early.)

(5) Continue to insist that test planning consider the needs of the system evaluator and that testing follow an approved plan. (#14: Follow a logical sequence of activities, i.e., develop the evaluation plan before the test plan, and the test plan before conducting the test.)

(6) Explore additional ways in which rapidly emerging information technology can be used to make T&E better, faster, and cheaper. (#16: Use information technology to facilitate T&E.)

(7) Continue to scrutinize Test and Evaluation Master Plans and detailed test plans to ensure that testing will generate sufficient information to address the critical issues while at the same time avoiding the expenditure of time and resources on nonessential data. (#17: Aim for adequacy, not perfection; and #18: Minimize specifications and documentation.)

Although not every identified best practice contributed explicitly to the above recommendations, all are considered applicable to DoD developmental T&E and worthy of consideration by the DTSE&E office. In particular, it is the opinion of the study team that there are no provisions in the DoD 5000-series acquisition directives or other T&E policy documents that prevent adoption of these practices.

IV-6

Appendix AOrganizational Participants and Contacts

The commercial enterprises listed below participated in this study and provided valuable inputs for use in addressing the study questions. (Not all organizations originally contacted consented to be involved in the study.) The individual listed served as the principal point of contact for the organization; typically, one or more others in the organization were also key providers of information about its T&E practices. Within the limits imposed on the study imposed by schedule and resources, enterprises were selected to represent the following important industry niches:

• High Technology Air Vehicles & Electronic Systems (ACTD-Structured Program)

• High Technology Air Vehicles & Electronic Systems(Streamlined Prototypes)

• Large Hardware (Aircraft)• Large Hardware (Ships)• Large Software• Small Software• Telecommunications• Pharmaceuticals and Medical Devices• Consumer Products (Electronic)• Clinical Trials• Commercial Testing

The participating commercial enterprises and their primary points of contact were as follows:

Teledyne Ryan Aeronautical2701 North Harbor DriveSan Diego, CA 92186-5311Mr. Norman S. SakamotoVice President, Engineering

Stac, Inc.12636 High Bluff DriveSan Diego, CA 92130-2093Mr. Jim NicolVice President, Product Development

Boeing Commercial Airplane GroupP.O. Box 3707Seattle, WA 98124-2207Mr. Dennis FloydChief Engineer - Test, Validation & CertificationAirplane Creation Process Strategy Team

A-1

National Steel and Shipbuilding Company2798 Harbor DriveSan Diego, CAMr. Robert RichardsonSupervisor, Test Engineering

Microsoft CorporationOne Microsoft WayRedmond, WA 98052-6399Mr. S. SomasegarGeneral Manager, Windows NT5 Release

Pathology Associates International CorporationSuite 1, 15 Worman’s Mill CourtFrederick, MDMs. Carol HoffmanQuality Assurance Director

Qualcomm, Inc.6455 Lusk BoulevardSan Diego, CA 92121Mr. Stephan PalmerProgram Manager, System Test & Interoperability

Dura Pharmaceuticals, Inc.5880 Pacific Center BoulevardSan Diego, CA 92121Dr. Lloyd FlandersSenior Vice PresidentProgram Management and R&D Planning

Wyle Laboratories128 Maryland StreetEl Segundo, CA 90245Mr. Drexel L. SmithVice President

Hewlett-Packard Company16399 West Bernardo DriveSan Diego, CA 92127Mr. Scott HockR&D Project Manager

Lockheed Martin Corporation1011 Lockheed WayPalmdale, CA 93599-3507Mr. George CusimanoDirector of Flight TestLockheed Skunk Works

A-2

Appendix BRepresentative Discussion Topics for Industry Visits

• Why does your organization conduct test and evaluation? To support design qualification? Quality assurance? Verification and validation? Technology assessment? Response to government requirements? Liability limitation? Assist development? Manage risk? Define limitations? Support marketing decisions? Provide information to create advertising materials and address customer questions?

• Do you make a distinction between “developmental” testing (performed to support product development and maturation) and “operational” or customer testing (performed to determine the acceptability of the product in its intended market).

• Typically, what percentage of a product’s development cost is spent on testing and evaluation? How does that break down as to test articles, test facilities and instrumentation, and labor? Can you quantify the value added by T&E?

• Do you have a formal, documented T&E process applicable throughout the organization to all products, or is a specific approach created for each product? Is there a common approach for test strategy, design, planning, conduct, and reporting? How are these activities accomplished?

• Do you have a permanent T&E organization or is an ad hoc team created for each product? What is the reporting structure for the T&E activity and what degree of independence does it have? To whom does the “Chief Tester” report?

• What functions are carried out by the T&E organization? What control does it have over budget? Schedule?

• How are individuals assigned to the T&E organization? How would you describe its demographics and culture? What training is provided to those assigned? How are they motivated?

• How do you decide how much testing is enough? What are the criteria used, and who makes the decision?

• Do you conduct periodic reviews during the test execution phase to determine which tests, if any, can be eliminated based on results to date?

• Is cycle-time an important issue in executing your test process? What steps do you take to minimize cycle-time? Test data processing turnaround time? Test data analysis time?

B-1

• What do you consider to be the major risks and rewards of a robust T&E program? What are the principal constraints and limitations to conducting the “ideal” test program?

• To what extent do your markets and customers influence the type and amount of T&E conducted, as opposed to purely internal considerations?

• How do you handle the security of T&E data when proprietary and company private information are involved?

• What use of T&E results do you make in your advertising and publicity efforts? To what extent would you respond to customer queries by referring to T&E results?

• What proportion of your T&E effort has the objective of “experimentation” (learning about the product) versus “demonstration” (proving that the product has required or desired capabilities and characteristics)? Do you distinguish among testing versus evaluation versus assessment?

• What are the most significant technology challenges to your T&E programs? What technologies have you exploited to enable and assist your T&E programs? In particular, what use do you make of modeling and simulation to support your T&E programs?

• How do you trade off cost and schedule considerations with the need for a robust test program? To what extent are your T&E programs “designed to cost”? Do you attempt to apply test cost and test cycle-time reduction strategies? How do you adapt your T&E program to managed changes in the overall product development program? To unanticipated surprises?

• In a typical product development, do you start with a set of requirements based on customer and market surveys and then develop the technological “answer” to meet them, or do you start with a technology and attempt to develop a product which will exploit it in a commercially successful way?

• Do you follow a formal system engineering process for product development, of which T&E is an element? Do you attempt to “system engineer” the T&E activity itself?

• How do you arrange for and guarantee the availability of test articles and test facilities and instrumentation? Is this a serious challenge?

• How do you determine the parameters to be measured by your test instrumentation?

B-2

• Do you test the “ilities” (reliability, availability, maintainability, interoperability, etc.) not directly related to product performance? If not, how do you estimate them?

• Do you distinguish between the T&E of prime systems versus support systems, e.g., as to type or amount?

• In your T&E programs, how do you apportion the relative importance of hardware, software and people viewed as elements of the system under test? How do you address user/operator interactions with the hardware and software elements?

• To what extent do your T&E programs have formal documentation? Baselines and requirements? T&E plans? T&E reports? What level of approval is required? How are documentation changes effected?

• How are metrics and criteria for success and failure established? What are typical metrics and criteria? Do you use pass/fail or goals & thresholds or both? Do they address technical parameters? Performance measures? Operational considerations?

• How are the results of T&E used by decision-makers in your organization?

• Does your organization employ integrated product teams or concurrent engineering? If so, how are the needs of the T&E program coordinated?

• What industry, government, or company standards do you follow in conducting your T&E programs?

• How are monitoring and oversight of your T&E programs conducted?

• Do your products have built-in test capabilities?

• What use of product prototypes is made in your T&E programs? Production-representative articles?

• What is your tradeoff between the use of statistical analysis and engineering judgment in establishing confidence in your T&E results?

• What do you regard as the value added by T&E accomplished early in the product development process?

• What is your approach to beta testing by potential customers versus in-house operational testing by user surrogates?

• In your organization, who has ultimate “ownership” of the T&E process and the responsibility for it being carried out adequately?

• To what extent does your organization practice evolutionary or incremental product development, i.e., create a mature product via a

B-3

series of pre-planned enhancements? How does your T&E methodology accommodate this process? How do you deal with “requirements slip” or the unwanted migration of requirements from one increment or build to the next?

• What is your approach to processing test data to generate information and knowledge, i.e., “getting answers from numbers”?

• How do you employ T&E results to address concerns about product liability and safety?

• What degree of stability do you encounter in T&E requirements and funding? Is this a significant challenge?

• What do you consider to be the best T&E practices of your organization? Can you relate them to the overall success of your products?

B-4

Appendix CBibliography and References

Adolph, Charles E., et al., “Independent Review of the Development Risks Associated With the Comanche Armed Reconnaissance Helicopter,” Institute for Defense Analyses Paper P-3016, 1994, FOUO.

Adolph, Charles E., “Test and Evaluation Reflections and Recommendations,” American Institute of Aeronautics and Astronautics Paper, 1994.

Adolph, Charles E. & Duncan, Robert C., “Test and Evaluation Process Improvements,” ITEA Journal of Test and Evaluation, Vol. XI, No. 4, 1990.

Anderson, Brian K.; Calcutt, Jr., Harry M. & Hannen, Paul, “The Role of Modeling & Simulation in the B1-B Defensive System Upgrade Test Program,” ITEA Journal of Test and Evaluation, Vol. 18, No. 3, 1997.

Armed Forces Communications and Electronics Association, “Evolutionary Acquisition Study,” 1993.Atkinson, James H., “Test and Evaluation of Command, Control, Communications and Intelligence

Systems,” ITEA Journal of Test and Evaluation, Vol. XI, No. 1, 1990.Backs, Richard T. & Parker, Mary H., “Logistics Test: A New Dimension for Test and Evaluation,” ITEA

Journal of Test and Evaluation, Vol. XVI, No. 3, 1995.Ballard, Dennis; Camp, Alan; Loper, Janelle & Montagne, Ernest, “Operational Test and Evaluation

Tailored for Streamlined Acquisition,” ITEA Journal of Test and Evaluation, Vol. 18, No. 2, 1997.Banister, Grady H., “Importance of the Human Variable in Testing,” ITEA Journal of Test and

Evaluation, Vol. XV, No. 2, 1994.Barlow, Barry R., “The GBU-28,” Office of History, Air Force Development Test Center, 1994.Battershell, A. Lee, “Technology Approach: DoD Versus Boeing (A Comparative Study),” Acquisition

Review Quarterly, Summer, 1995.Bayer, Charlene W., “Coping with Perceptions of Risk,” ITEA Journal of Test and Evaluation, Vol. XIII,

No. 4, 1992.Beers, Suzanne M. & Vachtsevanos, George J., “High-Level Decision Making from Low-Level Test

Data,” ITEA Journal of Test and Evaluation, Vol. 18, No. 2, 1997.Bishop, Glade M., “From Software Usability to System Usability: An Approach to the Human Factors

Aspects of Operational Evaluations,” ITEA Journal of Test and Evaluation, Vol. XV, No. 2, 1994.Blankenship, Samuel M., “Advances in Test and Evaluation Technology,” ITEA Journal of Test and

Evaluation, Vol. 18, No. 1, 1997.Blue Ribbon Defense Panel, Report to the President and the Secretary of Defense on the Department of

Defense (Appendix F, “Staff Report on Operational Test and Evaluation”), 1970.Bolles, Robert; Zielinski, Wayne; Marshall, Steven & Perkins, Richard, “V-22 Avionics Software

Integrated Test Team: A Case Study on Developing a Contractor-Sponsor Partnership in Software Testing,” ITEA Journal of Test and Evaluation, Vol. 18, No. 4, 1997.

Box, George E.P.; Hunter, William G. & Hunter, J. Stuart, Statistics for Experimenters, John Wiley & Sons, Inc., 1978.

Brown, C. David, “Ground Truth Data: The Anchor for Modeling and Simulation,” ITEA Journal of Test and Evaluation, Vol. XVI, No. 2, 1995.

Brown, C. David, “A Tester Prepares for Synthetic Acquisition,” ITEA Journal of Test and Evaluation, Vol. 17, No. 2, 1996.

Brown, C. David; Ayala, Rudy & Berlin, Brett, “High Performance Computing Modernization Program Enhances Test and Evaluation Capabilities,” ITEA Journal of Test and Evaluation, Vol. 19, No. 1, 1998.

Burge, Albert R., “Test and Evaluation Based on Metrics, Measures, Thresholds and Indicators,” ITEA Journal of Test and Evaluation, Vol. 17, No. 3, 1996.

Camp, Alan & Thompson, Mark, “Defense Information System Testing,” ITEA Journal of Test and Evaluation, Vol. 18, No. 1, 1997.

Clarke, Lloyd W. & Gardner, Lee S., “How Much Testing Is Enough?: An Iterative, Mathematical Programming Approach,” ITEA Journal of Test and Evaluation, Vol. XVI, No. 3, 1995.

Cohen, Michael L.; Rolph, John E. & Steffey, Duane L., Statistics, Testing and Defense Acquisition, National Academy Press, 1998.

Conlon, John C.; Lilius, Walter A. & Tubbesing, Frank H., Test and Evaluation of System Reliability, Availability and Maintainability, Office of the Director Defense Test and Evaluation, 1981.

C-1

Coyle III, Philip E., “New Directions in Operational Testing,” ITEA Journal of Test and Evaluation, Vol. 17, No. 2, 1996.

Crabtree, Jim W., “Overhauling Test and Evaluation Standards,” ITEA Journal of Test and Evaluation, Vol. XV, No. 3, 1994.

Cropley, D. H., “Optimising Measurement Information Acquisition in Test and Evaluation,” ITEA Journal of Test and Evaluation, Vol. 17, No. 3, 1996.

Defense Science Board, “Report of Task Force on Test and Evaluation,” 1974.Defense Science Board, “Report of Task Force on Test & Evaluation Policies,” 1977.Defense Science Board, “Report of the Acquisition Cycle Task Force,” 1978.Defense Science Board, “Simulation, Readiness and Prototyping,” DTIC #ADA 266125, 1993.Defense Science Board, “Defense Acquisition Reform, Phase I,” DTIC #ADA 268734, 1993.Defense Science Board, “Defense Acquisition Reform, Phase II,” DTIC #ADA 286410, 1994.Defense Science Board, “Acquiring Defense Software Commercially,” DTIC #ADA 286411, 1994.Defense Science Board, “Defense Acquisition Reform, Phase II (Appendix D, Jet Engine Commercial

Practices Panel),” DTIC #ADA 286413, 1994.Defense Science Board, “Defense Acquisition Reform, Phase III,” DTIC #ADA 316651, 1995.Defense Science Board, “Outsourcing and Privatization,” DTIC #ADA 316936, 1995.Defense Science Board, “Achieving an Innovative Support Structure for 21st Century Military

Superiority,” DTIC #ADA 320394, 1996.Defense Science Board, “Tactics and Technology for 21st Century Military Superiority,” 1996.Defense Systems Management College, Test and Evaluation Management Guide, 1988.DeMillo, R.A. & Gagliano, R.A., “Policy Recommendations for Software Test and Evaluation: System

Level Issues,” ITEA Journal of Test and Evaluation, Vol. V, No. 1, 1984.DeMillo, Richard A.; McCracken, W. Michael; Martin, R. J. & Passafiume, John F., Software Testing

and Evaluation, [textbook based on the Director, Defense Test and Evaluation, Software Test and Evaluation Project], The Benjamin/Cummings Publishing Co., Inc., 1987.

Department of Defense Directive 5000.1, “Defense Acquisition,” 1996.Department of Defense Regulation 5000.2-R, “Mandatory Procedures for Major Defense Acquisition

Programs (MDAPs) and Major Automated Information System (MAIS) Acquisition Programs,” 1996.

Department of the Navy (Reliability, Maintainability and Quality Assurance Directorate), “Best Practices: How to Avoid Surprises in the World’s Most Complicated Technical Process,” NAVSO P-6071, 1986.

Deputy Director (Test and Evaluation), Office of the Director of Defense Research and Engineering, “T&E Guidelines” (Nine volumes—Missile Weapon Systems, Aircraft Systems, Ship Systems, Ground Vehicle Systems, ASW Systems, Airborne ECM Systems, Airborne General Surveillance Radar Systems, Command and Control Systems, Common Test Gear), 1974.

Dicker, Michael P., “Technology Forecasting: An Engineering Case Study,” ITEA Journal of Test and Evaluation, Vol. XV, No. 2, 1994.

Director, Assessment and Evaluation, Office of the Assistant Secretary of the Army (Research, Development and Acquisition), “Simulation Support Plan Guidelines,” 1997.

Director, Operational Test & Evaluation, “FY 97 Report,” 1988.Director, Test & Evaluation, Office of the Secretary of Defense, “Integrating Test and Evaluation with

Acquisition Management Risk,” 1994.Director, Test, Systems Engineering and Evaluation & Director, Operational Test and Evaluation, Office

of the Secretary of Defense, “Simulation, Test and Evaluation Process (STEP) Guidelines,” 1997.Dix, Heidi, “Test and Evaluation Dilemma Requires Calculated Approach,” National Defense, February

1997.Drezner, J.A., “The Nature and Role of Prototyping in Weapon System Development,” RAND R-4161-

ACQ, 1992.Duesterhaus, David A., “Government Test Facility Supports Commercial Aircraft Engine Testing,” ITEA

Journal of Test and Evaluation, Vol. 17, No. 4, 1996.Eichblatt, Emil J., “Use of Simulation to Evaluate Tactical Missile Performance,” ITEA Journal of Test

and Evaluation, Vol. XV, No. 3, 1994.Farley, Dixie, “Benefit Versus Risk: How the FDA Approves New Drugs,” ITEA Journal of Test and

Evaluation, Vol. XII, No. 1, 1991.Ferguson, Paul C., “From Advanced Tactical Fighter (ATF) to F-22,” Office of History, Aeronautical

Systems Center, Air Force Material Command, 1996.

C-2

Finch, James R., “Essential Suitability Training for Test Team Personnel,” ITEA Journal of Test and Evaluation, Vol. X, No. 2, 1989.

Freind, Harry, “What To Do About Experimental Design,” Total Product Development Symposium, American Supplier Institute, 1997.

Fries, Arthur, “Design of Experiments in Operational Test and Evaluation,” ITEA Journal of Test and Evaluation, Vol. XIV, No. 4.

Gable, Deborah L., “Acquisition of the F-16 Fighting Falcon (1972 - 1980),” Air Command and Staff College, Maxwell AFB, AL, 1987.

Gansler, Jacques S., Defense Conversion: Transforming the Arsenal of Democracy, MIT Press, 1995.Gansler, Jacques S., “National Security and T&E at the Beginning of the 21st Century,” ITEA Journal of

Test and Evaluation, Vol. XIV, No. 1, 1993.Gansler, Jacques S., “Needed Changes in Weapons Testing,” ITEA Journal of Test and Evaluation, Vol.

X, No. 1, 1989.Gehrig, John F.; Brown, C. David & Finfera, James P., “ITEA and MORS Symposium Considers How

Much Testing Is Enough?,” ITEA Journal of Test and Evaluation, Vol. XV, No. 2, 1994.Gehrig, John F. & Mion, Peggy A., “Case Study: Cost of Testing vs. Program Cost,” ITEA Journal of

Test and Evaluation, Vol. XIV, No. 1, 1993.Geiger, Clarence J., “Small Wonder: Development of the F-16 Fighting Falcon, 1975 - 1980,” History

Office, Aeronautical Systems Command, Air Force Systems Command, 1986, CONFIDENTIAL.Geiger, Clarence J., “History of the F-16: Prototype to Air Combat Fighter, 1971 - 1975,” History Office,

Aeronautical Systems Division, Air Force Systems Command, 1977.Geismar, Donald D., “Use of Modeling and Simulation in Navy Operational Testing,” ITEA Journal of

Test and Evaluation, Vol. X, No. 3, 1989.General Accounting Office, “Best Practices—Successful Application to Weapon Acquisitions Requires

Changes in DOD’s Environment,” GAO/NSIAD 98-56, 1998.General Accounting Office, “Best Practices—DOD Can Help Suppliers Contribute More to Weapon

System Programs,” GAO/NSIAD-98-87, 1998.General Accounting Office, “Test and Evaluation—DOD Has Been Slow in Improving Testing of

Software-Intensive Systems,” NSIAD-93-198, 1993.Giadrosich, Donald L., “Modeling and Simulation in Operational Test and Evaluation,” ITEA Journal of

Test and Evaluation, Vol. XI, No. 1, 1990.Giadrosich, Donald L. & Curtis, Hugh G., “Software Evaluation: The Importance of a Total Systems

Approach,” ITEA Journal of Test and Evaluation, Vol. XIII, No. 1, 1992.Glennan, T.K.; Bodilly, S.J.; Camm, F.A.; Mayer, K.R. & Webb, T., “Barriers to Managing Risk in

Large Scale Weapons System Development Programs,” RAND MR-248-AF, 1993.Greenlee, Donald R. (Editor), Defense Systems Management Review, Vol. I, No. 5, 1977 (Special Issue

on DoD T&E).Greenlee, Donald R., “Evaluation and the Evolutionary Acquisition Strategy,” ITEA Journal of Test and

Evaluation, Vol. XIII, No. 1, 1992.Greenlee, Donald R., “How Much Testing Is Enough?,” ITEA Journal of Test and Evaluation, Vol. XIV,

No. 4, 1993.Greenlee, Donald R., Test and Evaluation in the United States Department of Defense, Director Defense

Test and Evaluation, 1983.Hall, Janet A. & Graffam, James H., “Test Methodology Transfer: Commercial Application of Today’s

Government T&E Standards,” ITEA Journal of Test and Evaluation, Vol. XV, No. 4, 1994.Harrison, George B., “Using Modeling & Simulation to Support Operational Testing,” ITEA Journal of

Test and Evaluation, Vol. 17, No. 2, 1996.Hartka, Joseph E., “A Unified Approach to T&E as a Model for the Operation of Integrated Product

Teams,” ITEA Journal of Test and Evaluation, Vol. 17, No. 1, 1996.Haslem, William J., “Twelve Testing Mistakes,” ITEA Journal of Test and Evaluation, Vol. VII, No. 2,

1986.Hawley, John K., “Testing Human-Machine Systems in the Information Age,” ITEA Journal of Test and

Evaluation, Vol. 17, No. 2, 1996.Hilliard, Don, “Modeling and Simulation of Airborne Weapon Systems,” ITEA Journal of Test and

Evaluation, Vol. XV, No. 3, 1994.Hines, Dennis O. & Norton, William J., “Simulation, Test and Evaluation Process,” ITEA Journal of Test

and Evaluation, Vol. 18, No. 3, 1997.

C-3

Holiday, Gerald R., “Human Error: Planning for the Dreaded Unknown,” ITEA Journal of Test and Evaluation, Vol. XV, No. 2, 1994.

International Test and Evaluation Association, “Test and Evaluation of Command and Control Systems Developed by Evolutionary Acquisition,” Study conducted for the Office of the Secretary of Defense, 1987.

Johnson, R. & Birkler, J.L., “Three Programs and Ten Criteria: Evaluating and Improving Acquisition Program Management and Oversight Processes Within the Department of Defense,” RAND MR-758-OSD, 1996.

Kaminski, Paul G., “Simulation, Test and Evaluation: An Integral Process,” ITEA Journal of Test and Evaluation, Vol. XVI, No. 4, 1995.

Kass, Richard A., “Calibrating Questionnaires and Evaluators,” ITEA Journal of Test and Evaluation, Vol. V, No. 3, 1984.

Kass, Richard A., “Design of Valid Operational Tests,” ITEA Journal of Test and Evaluation, Vol. 18, No. 2, 1997.

Kass, Richard A., “Writing Measures of Performance to Get the Right Data,” ITEA Journal of Test and Evaluation, Vol. XVI, No. 2, 1995.

Keck, Eric, “The Utility of Advanced Distributed Simulation to Test and Evaluation,” ITEA Journal of Test and Evaluation, Vol. XV, No. 3, 1994.

Keeton, Brian A.; Floyd, Timothy S. & McDougal, George F., “Harnessing Parallel Computing for Test and Evaluation,” ITEA Journal of Test and Evaluation, Vol. XV, No. 3, 1994.

Kelf, Michael A.; Marciniak, Robert D. & Walter, Jeremy, “Government-University Partnership Evaluates Commercial Technology for Undersea Warfare Applications,” ITEA Journal of Test and Evaluation, Vol. 18, No. 4, 1997.

Kent, G.A., “Exploiting Technology,” RAND P-7403, 1988.Kohn, Allen, “F-22 Program Makes Test and Evaluation Part of Integrated Product Development,” ITEA

Journal of Test and Evaluation, Vol. 17, No. 1, 1996.Logan, Glenn T. & Silver, Michael O. “Implications of an Open Systems Approach for Test and

Evaluation,” ITEA Journal of Test and Evaluation, Vol. 18, No. 4, 1997.Marks, David M., Testing Very Big Systems, McGraw-Hill, Inc., 1992.Martin, James N., Systems Engineering Guidebook: A Process for Developing Systems and Products,

CRC Press, 1997.Mayer, John H., “The Right Way to View Testing,” Software Magazine, May 1998.Mayer, K.R., “The Development of the Advanced Medium-Range Air-to-Air Missile, a Case Study of

Risk and Reward in Weapon System Acquisition,” RAND N-3620-AF, 1993.McDonald, Janet E. & McDonnell, John, “Joint Data Base Elements for Modeling and Simulation,” ITEA

Journal of Test and Evaluation, Vol. XVI, No. 2, 1995.McIninch, Thomas P. (Pseudonym), “The Oxcart Story,” Air Force Magazine, November, 1994.McKinney, Dana B., “Modeling and Simulation for Weapons Systems T&E,” ITEA Journal of Test and

Evaluation, Vol. XV, No. 3, 1994.McKinzie, Gordon A., “Working Together: Team Approach Used to Develop and Test Boeing 777, ITEA

Journal of Test and Evaluation, Vol. 17, No. 1, 1996.Meth, Martin A., “Practical Rules for Reliability Test Programs,” ITEA Journal of Test and Evaluation,

Vol. XIV, No. 4, 1993.Modjeski, Richard B. & Ward, Allan C., “Testing the Limits: The Application of Artificial Intelligence

to Test and Evaluation,” ITEA Journal of Test and Evaluation, Vol. XI, No. 3, 1990.Montagne, Ernest & Thompson, Mark, “Using Risk Assessment to Determine the Scope of Operational

Tests for Software-Intensive System Increments,” ITEA Journal of Test and Evaluation, Vol. 19, No. 1, 1998.

Moyer, Janette M., “Guidelines for the Use of Commercial Charge Policy at Major Range and Test Facility Base,” ITEA Journal of Test and Evaluation, Vol. 17, No. 2, 1996.

National Aeronautics and Space Administration, Office of Safety and Mission Assurance, “Spacecraft Test Effectiveness Program Management Plan,” 1995.

National Research Council, “Statistical Methods for Testing and Evaluating Defense Systems: Interim Report,” National Academy Press, 1995.

Neufeld, Jacob, “The F-15 Eagle: Origins and Development, 1964 - 1972,” Office of Air Force History, 1974, SECRET.

Norton, William J., “Striking a Balance: Balancing Modeling and Simulation with Flight Testing,” ITEA Journal of Test and Evaluation, Vol. 17, No. 1, 1996.

C-4

O’Connor, Michael J., “New Testing Techniques Improve Data Analysis,” ITEA Journal of Test and Evaluation, Vol. XV, No. 3, 1994.

Office of Federal Procurement Policy, Office of Management and Budget, “A Guide to Best Practices for Past Performance,” 1995.

Okagaki, Karen & Helmuth, Richard, “An Expert Systems Approach to Test Planning,” ITEA Journal of Test and Evaluation, Vol. XIV, No. 2, 1993.

Parth, Frank R., “Systems Engineering Drivers in Defense and in Commercial Practice,” Systems Engineering, Vol. 1, No. 1, 1998.

Patenaude, Anne, “Challenges to the Use of Modeling and Simulation in Weapon System Acquisition,” ITEA Journal of Test and Evaluation, Vol. 18, No. 3, 1997.

Patenaude, Anne, “Modeling and Simulation Effectiveness in Weapon System Acquisition,” ITEA Journal of Test and Evaluation, Vol. 18, No. 1, 1997.

Patenaude, Anne; Helmuth, Richard; Potter, Patrick & Borowski, Michael, “Final Report—Study on the Effectiveness of Modeling and Simulation in the Weapon System Acquisition Process,” Office of the Deputy Director, Test, Systems Engineering and Evaluation (Test Facilities and Resources), 1996.

Paul, Raymond, “Development and Test Concepts for Commercial Digital Electronic Components,” ITEA Journal of Test and Evaluation, Vol. 17, No. 4, 1996.

Pollard, Ray & Haug, John, “Principles of Development Testing,” ITEA Journal of Test and Evaluation, Vol. XIV, No. 4, 1993.

Portmann, Helmut H., “Simulation-Based Acquisition Catalyst for Sensitive Weapons Development,” National Defense, November 1996.

Reece, Rick, “Combined Developmental and Operational Testing: A Framework,” ITEA Journal of Test and Evaluation, Vol. XIII, No. 3, 1992.

Reig, Raymond W.; Gailey III, Charles K.; Lt Col Weber, USAF, Wayd & George, Lisa R., “A Study of the Relationship between Initial Production Test Articles Used in a System Development Program and the Success of That Program,” Defense Systems Management College, DSMC Press Technical Report TR 2-95, 1995.

Reynolds, Matthew T., Test and Evaluation of Complex Systems, John Wiley & Sons, Ltd., 1996.Rich, M.D. & Dews, E., “Thoughts on Reforming the Military Acquisition Process,” RAND P-7352,

1987.Rich, Michael & Dews, Edmund, “Improving the Military Acquisition Process,” RAND R-3373-AF/RC,

1986.Rodrigues, Louis J., Director, Defense Acquisition Issues, National Security and International Affairs

Division, General Accounting Office, “Defense Acquisition—Improved Program Outcomes Are Possible,” Testimony before the Subcommittee on Acquisition and Technology, Committee on Armed Services, U.S. Senate, GAO/T-NSIAD-98-123, 1998.

Rolphe, John E. & Steffey, Duane L., “Statistical Issues in Defense Analysis and Testing: Summary of a Workshop,” National Research Council, 1994.

Roth, M. Scott, “Automated Test Planning System,” ITEA Journal of Test and Evaluation, Vol. XVI, No. 1, 1995.

Sanders, Patricia A., “Test and Evaluation to Meet the Modernization Challenge,” ITEA Journal of Test and Evaluation, Vol. 18, No. 1, 1997.

Secretary of Defense, “Actions to Accelerate the Movement to the New Workforce Vision,”Report to the Congress, 1997.Sizemore, N. L., “Test Techniques for Knowledge-Based Systems,” ITEA Journal of Test and

Evaluation, Vol. XI, No. 2, 1990.Smith, Drexel, “A Test and Evaluation Paradox,” ITEA Journal of Test and Evaluation, Vol. 19, No. 3,

1998.Smith, G.K.; Shulman, H.L. & Leonard, H.S., “Application of F-117 Acquisition Strategy to Other

Programs in the New Acquisition Environment,” RAND MR-749-AF, 1996.Sommer, G.; Smith, G.K.; Birkler, J.L. & Chiesa, J.R., “The Global Hawk Unmanned Aerial Vehicle

Acquisition Process: A Summary of Phase I Experience,” RAND MR-809-DARPA, 1997.Stanley, William L. & Birkler, John L., “Improving Operational Suitability Through Better Requirements

and Testing,” RAND R-3333-AF, 1986.Starr, Stuart H., “Technology Assessment: Models, Simulations and Tools Emerging to Support the

Acquisition Process,” ITEA Journal of Test and Evaluation, Vol. XV, No. 3, 1994.Steffey, Duane L., “Toward a Decision-Theoretic Approach to the Design of Operational Tests,” Draft.

C-5

Stevens, Roger T., Operational Test and Evaluation: A Systems Engineering Process, Robert E. Krieger Publishing Co., Inc., 1986.

Stipp, David, “Why Pfizer Is So Hot,” Fortune, May 11, 1998.Thirtle, M.R.; Johnson, R.V. & Birkler, J.L., “The Predator ACTD: A Case Study for Transition Planning

to the Formal Acquisition Process,” RAND MR-899-OSD, 1997.Under Secretary of Defense (Acquisition and Technology); Service Acquisition Executives & Director,

DARPA, Testimony before the Senate Armed Services Subcommittee on Acquisition and Technology, 12 March 1998.

Under Secretary of Defense (Acquisition and Technology) and DoD Inspector General, Testimony before the Senate Armed Services Subcommittee on Acquisition and Technology, 19 March 1998.

Watson, George M., “The A-10 Close Air Support Aircraft 1970 - 1976,” Office of History, Air Force Systems Command, 1993.

Wilson, Jeff & Barlow, Mel, “Eliminating Redesign: First Time Test Success,” ITEA Journal of Test and Evaluation, Vol. IX, No. 3, 1988.

Wyant, Kerry W.; Patel, Alka B. & Hanes, Carol E., “Identifying and Establishing Data Standards for Test and Evaluation,” ITEA Journal of Test and Evaluation, Vol. 17, No. 2, 1996.

Wyant, Kerry W.; Patel, Alka B.; Wilcott, James L. & Prostick, Joel, “Data Standards for the Future,” ITEA Journal of Test and Evaluation, Vol. XVI, No. 2, 1995.

Zerr, John J., “T&E from a Business Perspective,” ITEA Journal of Test and Evaluation, Vol. 19, No. 3, 1998.

C-6

Date post:	14-Oct-2014
Category:	Documents
Upload:	james
View:	894 times
Download:	4 times

T&E Best Practices Final Report

Documents