Redesigning the T&E Process to Support the Air Force’s Develop and Sustain Warfighting Systems (D&SWS)

Donald Paul Waters1 and Elmer Standridge.2 Air Force Flight Test Center, Edwards AFB, CA, 93524

The United States Air Force (USAF) is faced with serious challenges in rising costs, an aging fleet and a constrained budget that is forcing a re-evaluation of the way in which it Develops and Sustains Warfighting Systems (D&SWS). Following rigorous processes to re-design the processes associated with D&SWS, the USAF has focused on meeting three objectives: Deliver agreed upon capability on-time, on-cost; Increase the availability of systems at a reduced cost; and finally, reduce cycle time from funded need to delivery. The Test and Evaluation (T&E) community within the USAF has also been tasked with re-designing its processes to meet these objectives. Following a structured process, Enterprise Value Stream Mapping and Analysis, the USAF T&E Design team mapped the “As-Is” process and developed the proposed “To-Be” state. This “To-Be” state recognized a change in the expectations of support that a program should receive from T&E, to the role of “trusted advisor” with defined processes and analysis behind the advice provided. It also recognized that T&E must be able to support the program early on, pre milestone B to gain the credibility with the program during critical early programmatic decisions. In order to achieve this “To-Be” state, the USAF T&E community has initiated three objectives for 2008 that will start the journey to the envisioned end-state. These objectives are: Pre MS-B Support; Workforce Management; and Knowledge Management.

Nomenclature AAC Air Armament Center AEDC Arnold Engineering Development Center AF/TE Headquarters Air Force Test and Evaluation AFFTC Air Force Flight Test Center AFMC Air Force Material Command AFMC/A3 Air Force Material Command, Directorate Air, Space and Information Operations AFOTEC Air Force Operational Test and Evaluation Center AFSO21 Air Force Smart Operations for 21st Century APB Acquisition Program Baseline CDD Capability Development Document CPD Capability Production Document CTF Consolidated Test Force D&SWS Air Force’s Develop and Sustain Warfighting Systems DT Developmental Test HPTs High Performance Teams ICD Initial Capability Development ITT Integrated Test Team LCM Life Cycle Management MS-B Milestone B OODA Observe, Orient, Decide and Act 1 Division Chief, Capabilities Integration, AFFTC/XP, 195 East Popson Ave, Edwards AFB, CA 93524, and AIAA Member. 2 Special Assistant to the Executive Director, Air Force Flight Test Center, 100 Kindel Dr, Suite A214, Arnold AFB, TN, 37389,.

1

U.S. Air Force T&E Days 5 - 7 February 2008, Los Angeles, California

AIAA 2008-1683

This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.

OT Operational Test SAF/AQ Secretary of the Air Force, Acquisition Office SIPOC Supplier, Input, Process, Output, and Customer T&E Test and Evaluation TEMPs Test and Evaluation Master Plans USAF United States Air Force

I. Introduction The United State Air Force (USAF) is facing a variety of challenges that are causing senior leadership to look for

dramatic changes in the way we do our business to increase the efficiency of our processes and eliminate waste. Specifically some of the key driving forces are:

• the increasing pressure from Congress on the budget, • the USAF having been at war since the 1980’s with the current war requiring highly dynamic responses

to meet an asymmetrical threat, • the increasing age of the fleet and the associated increase in maintenance costs, • increased personnel costs due to rising healthcare and support costs, • and finally a 31% increase in the cost of JP8 fuel in the last year alone.

In order to address these issues and more, the USAF has undergone a major re-evaluation of its’ processes, looking for ways to streamline, eliminate waste, and gain efficiencies that can then be used to fund recapitalization of the USAF fleet. Figure 1 captures the top level processes for the USAF.

This defining of processes is based on the approach espoused by Hammer in his book Beyond Reengineering. All of the processes captured in Figure 1 will have some kind of influence on the Test and Evaluation (T&E) conducted throughout the Air Force, but the process that is the major driver for T&E is the Develop and Sustain Warfighting Systems (D&SWS) process being led by Gen Carlson and LtGen Hoffman. This process is responsible for streamlining how the USAF goes about the business of providing material solutions to the warfighter.

The rest of this paper will go into more detail about the developments ongoing in the D&SWS and their impact on USAF T&E, and the redesign of the T&E processes to meet the needs of the evolving D&SWS processes. Section 2 of this paper provides an overview of the D&SWS processes and the ongoing changes that will impact T&E. Section 3 of this paper will describe the process design work conducted on the T&E processes under the direction of BrigGen Joe Lanni. Section 4 will describe some of the findings of the T&E design team and finally, section 5 will describe some of the initiatives that are ongoing to better meet the D&SWS requirements.

Figure 1 USAF Level Processes

II. Develop and Sustain Warfighting Systems Overview As seen in Figure 1, the Develop and Sustain Warfighting Systems is one of the 4 core processes of the USAF.

It goes beyond Air Force Material Command to include all aspects of the way the USAF gets weapon systems into the hands of the operators and makes sure that they are sustained. Figure 2 (next page) illustrates the sub-process map for D&SWS.

2

The 3 core processes in D&SWS are the Continuous Capability Planning, the Technology Development and the overall Lifecycle Management of the weapon system. It is interesting to note that there are no separate acquisition and sustainment processes. This was done intentionally to highlight that acquisition and sustainment are all part of the Lifecycle Management process and can not be effectively considered independently of each other. The other core processes also operate in an integrated fashion with Lifecycle Management throughout the cradle to grave management of a weapon system. Figure 3 illustrates how these processes interact throughout the life a system.

The enabling processes of Test and Evaluation, Supply Chain Operations and Sourcing operate on an as-needed basis to support the core processes. Governing the core and enabling processes are two governance processes: Institutionalize Standard Work, and Oversight/Command and Control.

Each of the 8 sub-processes within the D&SWS process are undergoing process re-design to stream-line, standardize and make more effective the individual sub-processes. This includes conducting a value-stream analysis and process map for each of the sub-processes and creating a “To-Be” state for each of the sub-processes. In order to keep all of the sub-processes aligned and feeding a more efficient and effective D&SWS process, the entire

development is coordinated by a Core Design Team. Additionally, based on the communication with the rest of the Air Force and Department of Defense, three customer oriented objectives were identified that became the driving objectives for all of the D&SWS sub-processes. These objectives are:

D&SWS Sub-Processes

Technology Development

Figure 2: D&SWS Sub-Process Map

Governing Sub-Processes

Life Cycle Management of DataEnabling Sub-Processes

Program Execution(Sustainment)

S&T Exploration

Capability Requirements Development

Program Execution(Acquisition)

Program Definition& Risk Reduction

All 5 Activities Being Executed Simultaneously All the Time

High-level Integrated D&SWS Process

Figure 3 Integration of D&SWS Processes

1) Agreed-to capability on-time / on-cost 2) Improved availability at reduced cost 3) Reduced cycle time from funded need to delivery

The first objective highlights the need for the D&SWS community to work with the operators to agree upon the

cost, schedule and capability performance early on and then work to meet those agreed to expectations. The second drives home the requirement to design in reliability and maintainability and to reduce the time and cost of sustaining the system by making the right decisions in technology and development so that more systems are available to the operations. Finally, the third overall objective is recognition that the Air Force processes for developing new weapons systems needs to become more agile. Taking 20+ years to develop a system too often means that the requirements have changed, the D&SWS process needs to be able to operate a quicker Observe, Orient, Decide and Act (OODA) loop than potential adversaries.

3

One of the key lessons learned early on in the evaluation of the D&SWS process was that it would not be possible to make the sub-processes efficient enough to provide the billions of dollars required for recapitalization of the fleet. What needed to be addressed was making the process more effective. Figure 4 highlights the impact of cost overruns in comparison to the total cost of a sub-process.

With the 3 D&SWS objectives and the emphasis on the need to increase the effectiveness of delivering systems to the operator, each of the sub-processes came up with issues and opportunities, 27 in all, to address inefficiencies and increase the overall effectiveness of the D&SWS process. These issues and opportunities were categorized into 4 focus area for improving the D&SWS processes.

1) Create Efficiency with Strategic

Enterprise Approach 2) Reduce Large $ Overruns by

Launching and Maintaining High Confidence Programs

Figure 4: Emphasis on Effectiveness

3) Life Cycle Management (LCM) Focused Planning and Decision Making

4) Steamline Meetings/Reports and Drive Toward Standard Processes

The idea behind the first Focus Area was that much of the D&SWS process was not being conducted with an Enterprise focus, instead each process was conducted in a stove-pipe manner with little understanding of other ongoing activities. The initiatives in this focus area are designed to gain efficiencies through increased sharing of information and a broader view of individual activities. The second Focus Area was focused largely on improving the requirements definition and better defining the start of a program with a detailed understanding of the costs of the entire life-cycle of a system. The LCM Focus Area was focused on better understanding the trade-offs on life-cycle costs throughout the decision making process. The final Focus Area captured the increasing need to have standard processes across the D&SWS process. This standardization of work and results reporting would help eliminate non-value added reporting.

More details on the D&SWS processes are better left to a different forum. This high-level background is provided to set the context for the Test and Evaluation Sub-Process efforts and the impacts on the T&E community.

III. Test and Evaluation Sub-Process Re-design As discussed in Section II of this paper, the sub-processes all went through developing a process map for the

sub-process, a value stream map and the development of a “To-Be” state that was then integrated into the overall D&SWS process. To accomplish this, a T&E Design Team was created under the leadership of the Process Owner BGen Joe Lanni, then Air Armament Center (AAC) Vice Commander, and Co-Leads Mr. Dave Bond, Executive Director of the Air Force Flight Test Center (AFFTC), and Mr. Jerry Kitchen, Technical Director of the Air Force Operational Test and Evaluation Center (AFOTEC). The team consisted of senior representatives from Operational Test (Air Combat Command 53rd Wing and AFOTEC – Det 2), Space Test (Air Force Space Command Operational Test), Command and Control Test (505th Command and Control Wing), Air Force Material Command (AFMC) Developmental

Brig Gen (sel) Joseph LanniProcess Owner

Manage Test Competencies

Manage Test Business

T&E Governance

Develop Test Solutions

Operate & Maintain Test Capabilities

Develop Test Capabilities

Manage Test Knowledge

T&E high-level process model

Conduct Tests

Customer Processes

D&SWSContinuous

Capability PlanningLife Cycle Mgt

Core

Governing

4

Enabling

D&SWS / T&E Conduct Tests &

Test Competencies

AF / Develop Warfighters

DoD / Other Svc

Evaluate Tests

Figure 5: T&E Process Map

Test (AFFTC, Arnold Engineering Development Center, and AAC) and Headquarters Air Force (AF/TE). The team met for one week a month for 7 months with periodic telecons in between meetings to hammer out the rigorous value-stream process. Figure 5 captures the T&E process map that formed the basis for future analysis and synthesis. Process Map and Process Decomposition

Again following the Hammer methodology for process reengineering, the T&E processes were mapped into governing processes, core processes and enabling processes. Each of the processes within the T&E sub-process were broken down to increasing levels of detail so that the processes could be analyzed and evaluated for areas of improvement. Figure 6 shows the next level of process decomposition.

Process Decomposition

CORE

Figure 6: T&E Process Decomposition

The T&E Governance covered the leadership, assignment of work, prioritization and other decision making aspects of the T&E process. The core T&E processes of Develop Test Solutions, Conduct Test and Evaluate Tests covered the conduct of the actual T&E of the systems. The Evaluate Tests was specifically identified to emphasize the importance of evaluation in the T&E process and manage the activities required to implement this process. The other core process of the Operate and Maintain Test Capabilities captured the stewardship role that accompanies the Major Range and Test Facility Base designation of the DoD test infrastructure.

The enabling T&E processes were required to support the test and evaluation activities. The Develop Test Capabilities was the sub-process where the advancement of T&E capabilities to meet new requirements or take advantage of new technologies was captured. The Manage Test Competencies sub-process captured the hiring, training and managing of the T&E workforce. The Manage Test Knowledge was specifically called out so that the T&E community could better manage it’s product of testing – the knowledge gained about the system under test and about the test processes. Finally, the Manage Test Business gathered together all of the financial and business processes required to support T&E activities.

5

Capturing the “As-Is” Processes Defining the T&E processes and associated sub-processes, as described above, provided a framework for the

definition of the “As-Is” state of T&E. To ensure a systematic approach to capturing the “As-Is”, the T&E Design Team relied on a rigorous Enterprise Value Stream Mapping and Analysis approach developed by the Lean Advancement Initiative and Massachusetts Institute of Technology. Figure 7 captures the high-level systems engineering approach that was used to capture the “As-Is” state.

The T&E Design Team defined the following categories of stakeholders. • Customers

o Folks that pay for our services, Program Offices

• Leadership o AF and DoD

• Users o Warfighters

• Suppliers o TPS, MajComs, Industry

• Partners o NASA, Support Contractors,

Industry For each of the classes of stakeholder,

the T&E Design Team contacted representatives of that class to gain a better understanding of what the stakeholder valued from Air Force T&E activities and how well these valued activities were being performed. For the most part, the stakeholders felt that Air Force T&E performed important activities that they, the stakeholder valued. In answer to the question of how Air Force T&E was performing on those valued activities, the responses were mixed. n many cases, the performance was judged to be very good and in other cases the performance was judged to need improvement.

Based on these inputs, the T&E Design Team created a SIPOC (Supplier, Input, Process, Output, and Customer) for each of the decomposed processes captured in Figure 6. The SIPOC further decomposed the processes by looking at who provided what to the process and then what the outputs were and who the customer was for the process. he T&E Design Team also looked at all the interlinkages of the sub-processes. This further definition of the sub-processes enabled the T&E Design Team to get to a point where enablers of the process and issues with the process could be identified.

Customer / Stakeholder Needs – for each customer, build the value stream

Product Characteristics

Product Characteristics CategorizeCategorizeRelationship

CharacteristicsRelationship

CharacteristicsProduct

Enablers/IssuesProduct

Enablers/Issues

PEOPLEPEOPLE

INFRASTRUCTUREINFRASTRUCTURE

PROCESSESPROCESSES

LEADERSHIPLEADERSHIP

How do you connect?

How do you connect?

What does customer think

of you

What does customer think

of you

Why or why not?Why or why not?

Are you trusted?Are you trusted?

How often do you interact?

How often do you interact?

How stable are players?

How stable are players?

What drives relationship?What drives

relationship?

When needed?When needed?

For What?For What?

Who produces?Who produces?

What makes it good?

What makes it good?

How is it made?How is it made?

How delivered?How delivered?

#1#1

#3#3

#6#6

#5#5

#7#7

#2#2

#4#4

What is it?What is it?

Figure 7: As-Is Process Identification

Developing the “To-Be” Process

Figure 8 captures the steps the T&E Design Team went through to develop the envisioned end-state for the T&E processes.

Integrate Enablers Across Customers-- Understand / integrate value streams

Integrate Enablers/Issues

Integrate Enablers/Issues

Determine Linkages / Interfaces

Determine Linkages / Interfaces

Ties / Needs from other Processes

Ties / Needs from other Processes

Dependencies between Initiatives &

tasks

Dependencies between Initiatives &

tasks

Info Provided to other ProcessesInfo Provided to other Processes

Timing of InitiativesTiming of Initiatives

Resource EstimationResource Estimation

Implementation Planning and

Execution

Implementation Planning and

Execution

Synthesize “Big Rock” Issues

Synthesize “Big Rock” Issues

Determine Integrated

Tasks

Determine Integrated

Tasks

People #1People #1

Process #1Process #1

Leadership #2Leadership #2

Leadership #1Leadership #1

Infrastructure #1Infrastructure #1

People #2People #2

Process #2Process #2

PEOPLEPEOPLE

INFRASTRUCTUREINFRASTRUCTURE

PROCESSESPROCESSES

LEADERSHIPLEADERSHIP

PEOPLEPEOPLE

INFRASTRUCTUREINFRASTRUCTURE

PROCESSESPROCESSES

LEADERSHIPLEADERSHIP

PEOPLEPEOPLE

INFRASTRUCTUREINFRASTRUCTURE

PROCESSESPROCESSES

LEADERSHIPLEADERSHIP

PEOPLEPEOPLE

INFRASTRUCTUREINFRASTRUCTURE

PROCESSESPROCESSES

LEADERSHIPLEADERSHIP

Evaluate ASAP Initiatives

Develop the To-Be Process

Figure 8: To-Be Process Identification

6

By looking at the “As-Is” results, the issues were then synthesized into four categories of People, Process, Infrastructure and Leadership. The “Big Rock Issues” are listed in Table 1 below.

Table 1: Big Rock Issues

People Processes Infrastructure Leadership Get the right people for the job – some permanent, some temporary/short term

Standard enterprise processes for: • Products & pieces of

test work • Business of test • Communications

Develop infrastructure to provide representative, realistic, & standard test environments • Sufficient test assets • Instrumentation

innovation – cheap, reconfigurable, non-intrusive, common

• Robust live, virtual, constructive capacity and capability nodes and entities

Interface w/ Acquisition • Enhance directives /

policies • Help develop

consensus among key leaders / decision-makers about what’s good enough

• Present / defend objective test results and evaluations

• Help hold line on requirements

Groom people for leadership in T&E

Manpower • Allocation of slots • Accounting of

positions • Accounting of work

Backbone and policies, tools, processes for virtual T&E enterprise • Remote C2 • Virtual training support

Workforce Management • Take care of people we

assign to “unconventional” jobs

• Develop, mandate, resource test tech school / training

Put qualified testers in Key positions

Early involvement – users, OT, DT, ITT • Consistent CTF

strategy • Stability of participants

Common Business of Test tools • Common cost

accounting systems • Master index of T&E

capabilities • Common cost policy

for use of T&E capabilities

• Common scheduling of and access to T&E capabilities

• Common capacity models (incl Industry)

Develop and Maintain Relationships • Industry, • Other Services, • Key Leaders

Prepare people properly System Under Test Requirements • Understanding of • Stability of • Flexibility in meeting

Get Rid of the Seams • Business, • DT / OT, • Mission Areas • Intraservice

Develop a professional test culture

Develop T&E Inc. Culture

7

In addition to identifying the “Big Rock” issues, the T&E Design Team also proposed initiatives to close the gap between the “As-Is” state and the “To-Be” states. As part of this development of initiatives, the T&E Design Team evaluated 5 initiatives that the Air Force had started under the auspices of Air Force Smart Operations for 21st Century (AFSO21). These AFSO21 initiatives were kicked off to solve many of the same issues as the T&E Design Team and were absorbed and allowed a jump start in key areas.

The final step that was part of the “To-Be” process determination was to integrate the T&E sub-process initiatives with the rest of the D&SWS processes. It was during these discussions that the fine-tuning of the T&E “To-Be” was captured. Because T&E is an enabling process under D&SWS, the T&E Design Team recognized that the rest of D&SWS and in particular the acquisition programs should expect a new culture and behavior. These expectations were codified in a recent briefing to the SAF/AQ (Ms Sue Payton) and the AFMC/CC (Gen Bruce Carlson). This new culture and behavior is captured in Figure 9.

This is part of the change in culture of T&E described in the Table 1 “Big Rocks”. The ideal interface between T&E and the Lifecycle Management Processes is that T&E is integrated and engaged early in the program to support the program to achieve program goals. This support consists of providing effective plans that are integrated into the program. T&E also needs to ensure that the results and reports meet the requirements of the programs to support LCM decisions. By achieving this change in mind-set of the T&E community, they will be able to earn the desired status of “trusted advisor” to the programs and leverage the experience gained across multiple programs to support the LCM processes.

What the PM Should Expect

That T&E is vested in PROGRAM successThat T&E plans effectively to support PROGRAMSound/professional working relationship with PROGRAM office, prime, and each otherT&E integrated to minimize time, risk, cost to PROGRAMResults/reports are objective, timely, supported by facts and relevant to PROGRAM technical and operational objectivesT&E experience (having tested multiple programs) will help identify potential PROGRAM issues early

Figure 9: T&E Support to Programs

IV. Test and Evaluation Initiatives and Status While a number of initiatives were identified as part of the “To-Be” design, only 3 were approved for initial

implementation: Pre-MS B Support, Knowledge Management, and Workforce Management. These were approved due to the immediate impact on D&SWS and due to the long-term changes required to implement. These initiatives were proposed by the new T&E Process Owner, Mr. Dave Bond and were approved by the D&SWS Process Council in August of 2007. A brief explanation of each of these initiatives is provided, however, the details are likely to change as these initiatives gain traction. Pre-MS B Support

The quickest payoff for D&SWS is in Launching High Confidence Programs and T&E is engaged in supporting this with the Pre-MS B initiative. The objectives of this initiative is to participate in the requirements definition phase of the program by providing analysis of the technical and test feasibility of the requirements and then using that early knowledge of the program requirements to provide inputs into the test programmatic decisions. Figure 10 (next page) is a cartoon that captures how T&E is working to support the programs early on. Specifically, the Pre-MS B initiative team, led by Dr. Dave Jerome (AFMC/A3 Deputy Director), is developing standard processes for providing inputs to requirements documents (Initial Capability Document – ICD, Capability Development Document – CDD, and Capability Production Document – CPD), participating in High Performance Teams (HPTs) in scoping the requirements and drafting the requirements documents, and supporting the Acquisition Program Baseline (APB) development by providing higher fidelity and rigorous Test and Evaluation Master Plans (TEMPs). This team is leveraging the extensive process definition already existing in AFOTEC and looking to implement a parallel process in both the Space and AFMC/Developmental Test communities. Current plans call for prototype implementation in Feb/Mar 2008 with full implementation by Summer 2008.

8

Task: Process for “right” early involvement

ICDDevelop

HPT

CDDDevelop

HPT

CPDDevelop

HPT

TES

Generic CONOPS

ACQ Strategy

Contract

T&E Early Involvement

Products

ACQ Early Involvement

Products

Reqts Early Involvement

Products

RTO Selection

TEMP

Specific CONOPS

SE PlanTech Plan

MSA

MSB

MSC

AOA

APB LRIP

Focus of T&E 1-16Pre MS-B

Figure 10: Pre-MS B Support

Knowledge Management The Knowledge Management initiative has begun scoping the effort to meet both the short term requirements

and the long term needs of the T&E community. The scoping is critical as this initiative captures everything from managing all data and information associated with test programs and the business of test to transferring implicit knowledge in the test community into explicit knowledge that can be shared among the community. The initiative team is being led by Dr. Ed Kraft (AEDC Technical Advisor) and is currently in the process of developing an implementation plan and charter for the initiative.

Workforce Management

The Workforce Management initiative has just installed an initiative leader, Mr. Dan Roth (AFFTC Technical Advisor) and is in the process of building an initiative team and developing an implementation plan and charter. The objective of this initiative is to identify the requirements for T&E experience throughout the Air Force and then based on those requirements develop the processes, policies, and training to meet the requirements.

V. Conclusion As indicated in this paper, the Air Force has severe challenges that it is trying to address by significantly

changing the processes that it uses to conduct its missions. The key driving forces for the T&E community are the changes occurring in the D&SWS process where key changes include Launching High Confidence programs. The AF T&E Design Team went through a rigorous process to capture the “As-Is” process and propose a “To-Be” process that better meets the needs of the D&SWS core processes. The approved initiatives for the T&E community are underway and should begin bearing fruit in the spring of 2008 as the Pre-MS B processes complete prototyping and become part of the new way of doing business.

Acknowledgments The authors would like to thank Ms. Leslie Underwood and Mr. Mark McAndrew for their support in preparing

this paper.

9