THE EFFECTS OF DOD ACQUISITION REFORM STREAMLINING INITIATIVES ON THESOFTWARE DEVELOPMENT PROCESS: A CASE STUDY
Wyatt E. Shankle, PEProduct Assurance Directorate,
U.S. Army Aviation and Missile CommandHuntsville, Alabama
Abstract
This paper describes the impact of certain DODacquisition streamlining initiatives on the softwaredevelopment process for an Army weapon system, withemphasis on the Software Quality Assurance (SQA)program. Elimination of mandatory military standardsallowed our project to develop internal standards andprocedures, which provided greater flexibility for thesoftware development methodology, documentation,configuration management, testing, and softwarequality assurance. Relieved from the adversarial role oftraditional military procurements, SQA was able tofunction in a cooperative manner. Lack of traditionalmilitary standards, however, forced the project todevise additional documentation to fill in unexpectedgaps in the development process.
Introduction.
Purpose.This is a brief case study of an ongoing softwaredevelopment project. Its purpose is to provide insightinto the experiences of an SQA program for a softwaredevelopment project after the implementation of DODacquisition streamlining initiatives. While severalstreamlining initiatives were applied to this project, thispaper focuses on two critical acquisition streamlininginitiatives: 1) the effect on this project of eliminatingmilitary standards; and 2) the effect on SQA in anintegrated software development environment. Thereader is cautioned that all projects are unique, and thatthe experiences of this project with acquisitionstreamlining initiatives may not be representative ofother projects.
Background.Until a few short years ago, military standards weretaken for granted by government engineers supportingweapon system development. In most cases, SoftwareQuality Assurance (SQA) engineers used DOD-STD-2167, Defense System Software Development, andDOD-STD-2168, Defense System Software QualityProgram, as the basic standards for softwaredevelopment. These two standards, along with theirrelated data item descriptions, comprised the entiresoftware life cycle. Various policies and regulationswere promulgated concerning these standards, with theintention of ensuring that the military standard way ofdeveloping software was firmly entrenched. Thesestandards were incorporated into almost every contractfor weapon system procurement. The documentationrequired on a typical procurement was comprehensive,to say the least. Typical documentation included, butwas not limited to, software development plans,software configuration management plans, softwarequality program plans, software test plans, software testdescriptions, software development folders, softwarerequirements specifications, interface requirementsspecifications, software design documents, softwareproduct specifications, and software version descriptiondocuments. All the plans were expected to referenceadditional, detailed procedures, such as software qualityevaluation procedures, configuration managementprocedures, coding standards and conventions, etc.DOD-STD-2167 and DOD-STD-2168 providedevaluation criteria against which to judge contractordocumentation, and SQA increasingly focused onreviewing this documentation. It was not uncommonfor SQA to reject a document, and then go throughseveral iterations of resubmittals until the documentwas finally approved. In the meantime, of course,development proceeded without approved plans. Thisformal, arms length approach of dealing with thedeveloper was encouraged by the governmentprocurement system.
1American Institute of Aeronautics and Astronautics
Military Standards Challenged.Army project offices fund SQA to make sure that theyreceive a quality software product. Under the formalprocurement system, however, the main avenueavailable to SQA to accomplish this was to require asubstantial amount of documentation from thecontractor. Consequently, SQA applied all thenecessary standards, included all the necessary reviews,and required all the necessary contract deliverables inorder to be able to provide assurance to the projectoffice that it was, in fact, getting a quality softwareproduct. There was simply little other choice. Thingsbegan to change, however, as more and more people inboth the private sector and government began tochallenge the procurement process and the verynecessity of military standards. Eventually, themomentum became unstoppable, and most militarystandards were eliminated.1'2 The quality community,including SQA, was shocked and disappointed.Because the use of military standards was so universal,it was difficult to conceive how to carry on withoutthem. The use of commercial standards wasencouraged in place of the defunct military standards,so the government quality community educated itselfabout the ISO 9000 series standards and began usingthem in new contracts. The general feeling, however,was that ensuring quality would be much more difficult.
Integrated Development Encouraged.During roughly the same period, the concept ofdeveloping systems in a serial, functionally separated,throw-it-over-the-wall approach was also beingchallenged. In most cases, various functional offices,directorates, and centers provide support to projects.Typically, a software quality engineer might beassigned to support one or two projects. The main goalof the engineer was to represent SQA and advocate theSQA "position" at all times. Similarly, personnel fromother engineering groups, logistics, procurement,safety, etc., were responsible to represent theirfunctional organization. Each organization added itsrequirements to the project, which increased thecontract cost and schedule. Sometimes, functionalorganizations were not included soon enough in the lifecycle, and preventable problems were not caught. Adifferent way of development was proffered throughacquisition streamlining. This method went by severaldifferent names, such as concurrent engineering, andintegrated process and product development (IPPD).Briefly stated, the IPPD approach emphasizes teams ofpeople from the various functional disciplines workingjointly throughout the life cycle. This approach waspromulgated throughout the DOD.3
Starting Fresh.
Technical Approach.Such was the situation in DOD when, in 1996, aftermuch preparation, development began on this project.Several acquisition streamlining initiatives wereincorporated into this effort. The technical approachfor this project was to use commercial off-the-shelfhardware components and operating system software asmuch as possible. Emphasis was placed on thesoftware development rather than hardware. Thesystem would be developed jointly by government andcontractors working as a team. All functionaldisciplines supporting the project were to be collocated.In January 1997, a full-time software quality engineerwas assigned to support this project.
A Different Approach to SQA.Traditionally, SQA served in an independent,adversarial role. In this new environment, however, theworld had been turned upside down. Policies andregulations concerning the independent authority of thequality organization had been weakened or eliminatedover the past several years. Project management let itbe known that bureaucratic enforcement of detailedstandards would no longer be acceptable. Everyonewas expected to contribute to the project. Governmentengineers would write software code, as opposed tosimply managing their contractor counterparts. Early inthe life of the project, SQA made the consciousdecision that the developers must be trusted to do theright thing, and that the mission of SQA was to help theproject to develop and deliver a quality weapon systemto the customer. SQA would explicitly not mandatehow things had to be done.
Project Planning.The first major activity of any project is planning. Inthe past, military standards were used extensively toguide the planning process. Even after eliminatingmilitary standards, these standards were still useful forreference and educational purposes. Also, theelimination of military standards does not meanelimination of all standards. As part of the SoftwareEngineering Institute (SEI) Level 3 rating, a completeset of standards and procedures for softwaredevelopment had been developed. However, while aprevious data item description might run up to a dozenpages, the typical internal standard for the samedocument might run two pages. The project was givenmuch more leeway in determining what goes into a
American Institute of Aeronautics and Astronautics
plan. For the traditional SQA engineer, this was likeletting the fox guard the hen house. SQA usuallyreviews all the project plans, compares them against theappropriate standards, and ensures compliance. Withoutdetailed standards, this was impossible. So, instead oftelling the project what it had to do, SQA kept askingthe project what it planned to do. As the project planstook shape, SQA reviewed early drafts and questionedspecific areas. This forced the developers to think aboutwhat they really wanted to do. As the weeks passed andinitial suspicion faded, a working relationshipdeveloped between the project and SQA. Gradually,SQA became a participant in the project planningprocess. Not only did SQA write its own plan, butSQA made significant contributions to nearly everyother plan too. Not all suggestions from SQA wereadopted, but a surprising number of items wereexpanded and revised due to input from SQA.
This approach to SQA and project planning has severaladvantages over the traditional "beat them over thehead" approach.
First, the developers feel ownership for the projectplans and software development process. Developersbecame more creative in thinking of better ways to dobusiness. Plans were not cut-and-pasted from somebodyelse's way of doing things. The project devised severalimprovements over the normal development process.Process improvements included modifying the standardwaterfall model for software development; creating abetter way of doing peer inspections; and establishing aflexible software version control process that does notcreate undue red tape. None of these items, bythemselves, are earth-shattering events; but they typifythe improvements that can be made when engineers aregiven appropriate flexibility. Moreover, this flexibilitycreates intrinsic motivation within the developers.4 Bywriting their own plans the way they really feel itshould be done, the developers are motivated to followthe rules instead of ignore them.
Second, a greater level of understanding is created forthe entire software development process. SQA adoptedan attitude of "do it any way you think best, but write itdown in the plan". When engineers were forced towrite the process down in black and white, many gapsbecame evident. While the flexibility to eliminateunnecessary documentation requirements is generallybeneficial, in some cases it became evident that some"unnecessary" standards are actually necessary. Byasking the developers to create a diagram or write adescription of how the process should work, they gain
understanding about how things fit together and whythey are necessary. A good example is softwareconfiguration management (CM). The originalsoftware development plan and configurationmanagement plans were very limited in this area. Afterseveral discussions and pointed questions about howconfiguration control would actually work on a day-to-day basis, it was discovered that several key processeshad been completely overlooked. This forced thedevelopers to think about how the CM process couldhelp their project if done properly. The final result wasa succinct set of detailed CM procedures writtenparticularly for this project. Another example is thesoftware safety plan. Rather than depending on aboiler-plated go-by, the project wrote its own safetyplan. This seemingly simple task turned into quite acomplex exercise, as it was soon discovered that thesoftware safety requirements created a ripple effectwhich impacted the software requirementsspecifications, requirements traceability, configurationmanagement practices, peer inspection process, codingstandards, and test planning. The project developed ahealthy appreciation for assuring software safety.
Third, SQA was able to focus on the more importantprocess issues. Sometimes one cannot see the softwarequality forest because of the standards compliancetrees. SQA was neither funded nor staffed to performextensive independent verification and validation of thecorrectness of the software. This meant that there wasno way for SQA to guarantee the software workedbased on SQA evaluation alone. Ensuring complianceto standards is important, but still will not guaranteegood software. The answer lies in the Quality System.The "Quality System" for any project is supposed tocomprise everyone; not just the SQA group.Consequently, it is the job of SQA to ensure that theproject establishes a good software developmentprocess that will deliver a quality software product atthe end. The process includes proper generation ofrequirements, adequate documentation andspecifications, configuration management, peerinspections and internal reviews by the developers oftheir own products, software testing, and otheractivities. The process is defined in the various projectplans. SQA participation in project planning, as notedearlier, focused on ensuring the software developmentprocess would produce a quality product.
The SOA Plan.The job of making sure that all the plans and proceduresare actually followed falls to the SQA group. Likeeveryone else, SQA needed its own plan. In this case,
American Institute of Aeronautics and Astronautics
the SQA plan was based entirely on what the projectitself said it would do, as documented in its own plansand procedures. This was a real eye-opener to theproject, as they saw listed everything that SQA wasplanning to check. Since everything was based on theproject plans, and since the project wrote its own plans,there was little disagreement about what SQA wouldexpect. Nearly all the standard SQA checklists andprocedures were tailored by SQA to comply with theproject plans. In some cases, the procedures werewritten directly by the developers and used by SQA toensure compliance. One example is the softwaredevelopment folders (SDFs). The concept of SDFs hasbeen around for a long time. During the planningstages, the developers completely eliminated thestandard SDF format and created their own. This newSDF format is entirely electronic and is integrated withthe project's software version control tool. An SDFdirectory structure and description of contents waswritten by the developers, reviewed by SQA, approvedby the system lead, and then distributed throughout theproject. SQA uses the SDF guidance as-is whenconducting monthly SDF audits.
Software Documentation.One of the most important advantages to eliminatingmilitary standards is the flexibility gained when writingdocumentation. The developers reviewed thedocumentation standards, then tailored those standardsto define the level and extent of softwaredocumentation for the various computer softwareconfiguration items (CSCIs). The softwaredevelopment plan documents which CSCIs must have arequirements specification (all of them), which needdesign documents (tactical software only), and whatgets included in the software product specifications.The format for the documentation was also determined,based on the needs of the project. For example, thetraceability section of the software requirementsspecification, normally a required section using '2167-style documentation, was instead distributed throughoutthe requirements. In this manner, the developers andtest group had to think about the test method and testlevel of each individual requirement as it was written.The traceability information was entered into adatabase, which produces reports that accomplish thesame purpose as the '2167-style traceability tables,except that the database provides much more capability.This function is performed by SQA, and is seen as verybeneficial by the rest of the project. This promotes apositive image of SQA to the rest of the project, andhelps maintain good working relationships.
There are some problems with being so flexible withstandards. One of the problems with eliminatingmandatory standards is that a hole is left which mustsomehow be filled. Military standards, if nothing else,provided a level of comfort by giving explicit guidanceon how to do almost everything. But if the standardsare eliminated, what then? The original expectation wasthat existing guidance would be sufficient, but theproject soon learned otherwise. For a time, no oneknew what to do next. It took a while for thefloundering to stop, and for the project to work throughvarious documentation issues and develop its ownguidance to replace the standards. This is timeconsuming, additional, unplanned work. Anotherproblem is overzealous elimination of softwaredocumentation. For example, it was determined by theproject that a hardware requirements specification wasnot needed, since the hardware was composed ofcommercial off-the-shelf items. It became evident lateron that some system requirements needed a "home" fortraceability purposes; and that the hardware was notquite as simple as supposed. A hardware requirementsspecification was born out of necessity.
Functioning in an Integrated Development.
A Systems Approach.An IPPD effort was not established for this project inthe formal sense, but a team effort was requirednevertheless. Although good in theory, functioning asan integrated team is not easy in practice. The tendencyis to continue working under the old rules out of habit.Under the old rules, the developers would writedocuments, then toss them over the wall to SQA forreview. As an integrated team, however, SQA neededto participate in the development of the project plans,requirements specifications, design, coding, testing,peer inspections, and other activities. This meantseeing early draft products from the developers in orderto provide input while the product was still beingdeveloped. This goes against the grain for most people,however, since they expect SQA to look for and report"problems" with their products. Eventually thisresistance faded and it became standard procedure tocirculate everything to SQA. One prerequisite forintegrated development is to take a systems approach.A systems approach to project management has manyadvantages.5 This is especially important duringintegrated development. Although this project is not anideal example for using the systems approach, it didaccomplish several key systems engineering objectives.To this end, the project made several decisions that
American Institute of Aeronautics and Astronautics
promote a systems approach, although these decisionswere not formally categorized as such. The decisionthat had the most impact was requiring everyonesupporting the project to be collocated in the samefacility. This included developers, test, SQA, CM,technical writers, training, and others. Only a few part-time support personnel were not collocated. Beingcollocated has tremendous advantages for the project.Serendipitous encounters in the hallway are anexcellent means of informal communication. Peoplestay focused on the project, and do not get caught up inturf battles concerning their functional organizations.Another decision that promotes a systems approach isthe participatory nature of many tasks. Input frommany groups is sought during the early stages of work.For example, developing the requirementsspecifications involved nearly everyone, including thedevelopers, test group, SQA, hardware support,technical writers, and others. There are severaladvantages to participating early in the development ofthe various products. The greatest advantage is beingable to influence products early enough so that changesdo not cause excessive rework. For example, SQAdiscovered that the software requirements specificationsneeded better traceability to the system requirements.In the past, this would have caused extensive rework toa specification, because SQA would normally not see ituntil it was essentially already complete. In this case,the traceability problem was detected in the first draftsection, before the rest of the document was evenwritten. This prevented the need for later, extensiverework. Another advantage is that the "formal" reviewcycle proceeds more quickly, since SQA and othershave already reviewed the product in several draftstages. Mistakes caught at this point can more easily becorrected, and the earlier the better. In the past, SQAoften did not see a document until the final draft justprior to formal approval. It is much more difficult toconvince someone to make a change at that point.Because most of the review takes place early, the finalreview is just a matter of reviewing the latest changes.One positive result is that configuration control boardmeetings can be scheduled with only two or three daysnotice. This is a considerable time saver. Theadvantages of integrated development come at a cost,however. Integrated development allows SQA toperform most of its reviews in parallel withdevelopment, which shortens the project schedule, butrequires more total review time by SQA due to multipledraft iterations. Multiplied by the number of plans,procedures, specifications, peer reviews, and otheractivities, this adds up to a lot of work for SQA.
SQA is NOT "Inspector 12".The word "audit" conjures images of diligent inspectorscombing through the records, looking for mistakes anderrors that can be documented and blamed on someone.After some serious soul-searching, SQA implementedan informal "never fail an audit" policy. Of course, thisdoes not mean an audit is always passed on the first try.The way this policy works is analogous to getting theoil changed in your automobile. You want to changeyour oil because you know that it will help preventfuture problems. When you take your car in, a checklistis used to inspect several items, such as fluid levels,brake lights, and horn. If anything is not right, it isfixed on the spot. When you leave, you know that allthe items on the checklist are now correct. That is howan SQA audit works for this project. Everyone wants todo things the way the plans say they should be done,but some things slip by. During an audit, SQA useschecklists based on the project plans. When problemsare found, SQA helps get them fixed. By the end of theaudit, everything on the checklist is correct. WhileSQA maintains internal records of problems found, the"official" audit report shows that everything on thechecklist is now correct. There is no blame. SQAnotes systemic problems, then educates developers onhow to avoid them in the future. SQA never conductssurprise audits.
When Problems Are Found.No SQA program would be complete without acorrective action system. Although no one "fails" anaudit, it is still required that SQA document and trackall problems until closed. This is done via the internalSQA defect log. The SQA defect log is not provided toanyone outside the SQA group. Sometimes problemsare found with products that are under version control.To correct such problems, the project has an officialproblem reporting and tracking system. SQA uses thissystem as needed, but so does everyone else on theproject. It is not an SQA problem reporting system - itis everyone's problem reporting system. Thiscombined system tracks problems reported by SQA,test, safety, and the developers themselves. In this way,SQA is just one more pair of eyes trying to help findand correct problems. There is no blame associatedwith this system. When systemic problems are noted,SQA resolves them through informal discussion withthe group leads. Informal communication works betterthan formal communication to get results.
Evolving Role of SQA.Over the course of this project, the role of SQA hasevolved considerably. In the beginning, SQA was
American Institute of Aeronautics and Astronautics
perceived in the traditional role of independent monitorover the project. This quickly changed as SQAconsciously shifted from trying to mandate how thingsshould be done; to helping the project deliver a qualityproduct based on how the developers wanted to do thejob. As a relationship of trust was established, SQAbecame a participant in the development of thesoftware, instead of just reviewing products after-the-fact. The project requested SQA to take on additionalresponsibilities, such as developing and maintaining atraceability database, writing portions of the projectplans, interfacing with the Safety Office, performingreliability estimates, and developing procedures forpeer inspections. SQA still reviews products forcompliance to standards, but the developers themselvesdetermine these standards. People actively requestSQA to review their products in early, draft stagesbecause they see SQA feedback as constructive andbeneficial. SQA has evolved into a software systemsengineering resource for the project.
Conclusions.
Acquisition streamlining initiatives have severaladvantages and disadvantages (Table 1). Theseinitiatives are not a panacea for all project ills. Severalunexpected problems occurred during this project as aresult of acquisition streamlining initiatives.Nevertheless, the overall experience of this project isthat acquisition streamlining, if properly implemented,
can be beneficial to a software development effort andthe SQA program. A systems approach to projectmanagement, while useful for almost any project, iscritical for acquisition streamlining initiatives tosucceed. Finally, for this project, the government andcontractors are jointly developing the system, asopposed to the government simply managing a contract.It would not possible for SQA to serve in the same roleif the government were managing a contractor fromafar.
References:
[1] Fowler, Charles A., (1994), "Defense acquisition:grab the ax", IEEE Spectrum, October, pp. 55-59.
[2] Secretary of Defense Memorandum, "Specifications& Standards - A New Way of Doing Business",William J. Perry, 29 June 1994.
[3] Department of Defense Directive 5000.1, "DefenseAcquisition", 15 March 1996.
[4] Herzburg, Frederick, (1968), "One More Time: howDo You Motivate Employees?", Harvard BusinessReview, January/February.
Summary of Advantages and Disadvantages of Acquisition Streamlining Initiativesfor This Project.
Advantages Disadvantages
Flexibility.
Efficiency; elimination of unnecessaryitems.
Forces developers to think about thesoftware development process.
SQA is able to concentrate on moreimportant process issues, rather thancompliance to standards.Since developers define their ownprocesses, SQA is no longer in anadversarial role.Developers tend to follow their plansbecause of a feeling of ownership.
Developers understand the softwaredevelopment process better.
Project schedule is shortened.
SQA input to the process is given seriousconsideration.
"What do we do now?" syndrome
Discovered that some "unnecessary" itemsare necessary.
Forced to write a lot of new standards andprocedures ourselves; very timeconsuming.SQA must be prepared to address issues ontheir technical merits; not becausesomething is required by a standard.SQA has much less clout when thedevelopers shortcut their own processes.
Developers still take shortcuts.
None.
Total SQA review time increases.
SQA must be prepared to have input andsuggestions turned down.
American Institute of Aeronautics and Astronautics
![Page 1: [American Institute of Aeronautics and Astronautics AIAA Defense and Civil Space Programs Conference and Exhibit - Huntsville,AL,U.S.A. (28 October 1998 - 30 October 1998)] AIAA Defense](https://reader031.documents.pub/reader031/viewer/2022020408/575095211a28abbf6bbf218f/html5/thumbnails/1.jpg)
![Page 2: [American Institute of Aeronautics and Astronautics AIAA Defense and Civil Space Programs Conference and Exhibit - Huntsville,AL,U.S.A. (28 October 1998 - 30 October 1998)] AIAA Defense](https://reader031.documents.pub/reader031/viewer/2022020408/575095211a28abbf6bbf218f/html5/thumbnails/2.jpg)
![Page 3: [American Institute of Aeronautics and Astronautics AIAA Defense and Civil Space Programs Conference and Exhibit - Huntsville,AL,U.S.A. (28 October 1998 - 30 October 1998)] AIAA Defense](https://reader031.documents.pub/reader031/viewer/2022020408/575095211a28abbf6bbf218f/html5/thumbnails/3.jpg)
![Page 4: [American Institute of Aeronautics and Astronautics AIAA Defense and Civil Space Programs Conference and Exhibit - Huntsville,AL,U.S.A. (28 October 1998 - 30 October 1998)] AIAA Defense](https://reader031.documents.pub/reader031/viewer/2022020408/575095211a28abbf6bbf218f/html5/thumbnails/4.jpg)
![Page 5: [American Institute of Aeronautics and Astronautics AIAA Defense and Civil Space Programs Conference and Exhibit - Huntsville,AL,U.S.A. (28 October 1998 - 30 October 1998)] AIAA Defense](https://reader031.documents.pub/reader031/viewer/2022020408/575095211a28abbf6bbf218f/html5/thumbnails/5.jpg)
![Page 6: [American Institute of Aeronautics and Astronautics AIAA Defense and Civil Space Programs Conference and Exhibit - Huntsville,AL,U.S.A. (28 October 1998 - 30 October 1998)] AIAA Defense](https://reader031.documents.pub/reader031/viewer/2022020408/575095211a28abbf6bbf218f/html5/thumbnails/6.jpg)
![Page 7: [American Institute of Aeronautics and Astronautics AIAA Defense and Civil Space Programs Conference and Exhibit - Huntsville,AL,U.S.A. (28 October 1998 - 30 October 1998)] AIAA Defense](https://reader031.documents.pub/reader031/viewer/2022020408/575095211a28abbf6bbf218f/html5/thumbnails/7.jpg)
