SOFTWARE PROJECT MANAGEMENT AND QUALITY ENGINEERING PRACTICES FOR COMPLEX, COUPLED MULTIPHYSICS, MASSIVELY PARALLEL COMPUTATIONAL SIMULATIONS: LESSONS LEARNED FROM ASCI
D. E. PostR. P. Kendall
LOS ALAMOS NATIONAL LABORATORY, LOS ALAMOS, NM, USA ([email protected])
AbstractMany institutions are now developing large-scale, complex,coupled multiphysics computational simulations for mas-sively parallel platforms for the simulation of the perform-ance of nuclear weapons and certification of the stockpile,and for research in climate and weather prediction, magneticand inertial fusion energy, environmental systems, astro-physics, aerodynamic design, combustion, biological andbiochemical systems, and other areas. The successful de-velopment of these simulations is aided by attention tosound software project management and software engineer-ing. We have developed “lessons learned” from a set ofcode projects that the Department of Energy National Nu-clear Security Agency has sponsored to develop nuclearweapons simulations over the last 50 years. We find thatsome, but not all, of the software project management anddevelopment practices (rather than processes) commonlyemployed for non-technical software add value to the de-velopment of scientific software and we identify those thatwe judge add value. Another key finding, consistent withgeneral software industry experience, is that the optimalproject schedule and resource level are solely determinedby the requirements once the requirements are fixed.
AcknowledgmentsThe authors are grateful for discussions with and sugges-tions from Tom Adams, Marvin Alme, Bill Archer, DonaldBurton, Gary Carlson, John Cerutti, William Chandler,Randy Christiansen, Linnea Cook, Larry Cox, Tom De-
Marco, Paul Dubois, Michael Gittings, Tom Gorman, DaleHenderson, Joseph Kindel, Kenneth Koch, Robert Lucas,Tom McAbee, Douglas Miller, Pat Miller, David Nowak,James Rathkopf, Donald Remer, Richard Sharp, AnthonyScannapieco, Rob Thomsett, David Tubbs, Robert Weaver,Robert Webster, Daniel Weeks, Robert Weaver, RobertWebster, Dan Weeks, Don Willerton, Ed Yourdon, MichaelZika, and George Zimmerman.
1 Introduction
In the middle of 1996, the Department of Energy (DOE)launched the Accelerated Strategic Computing Initiative(ASCI) to develop an enhanced simulation capability forthe nuclear weapons in the US stockpile. The Los Ala-mos National Laboratory (LANL) and Lawrence Liver-more National Laboratory (LLNL) were tasked withdeveloping this capability for the physics performance,and the Sandia National Laboratory (SNL) for the engi-neering performance of weapons systems. The ASCI pro-gram is now almost eight years old and now has beenrenamed to Advanced Simulation and Computing (ASC).It is an appropriate time to assess the progress and todevelop “lessons learned” to identify what worked andwhat did not. This paper presents the “lessons learned”for successful code development during the ASCI projectso far. The major points are summarized in Table 1.
In the absence of testing, improved nuclear weaponssimulation capability is needed to sustain the US defen-sive capability. Following the fall of the Soviet Union andthe cessation of testing nuclear weapons by both Russiaand the US in the early 1990s, the US inaugurated the“Stockpile Stewardship” program to maintain its nuclearstockpile. Even though the Russian Federation poses amuch reduced threat to the US compared to the SovietUnion, history, particularly the history of the twentiethcentury, has amply demonstrated that any nation thatdoes not possess a strong defense based on modern mili-tary technology can – and often will – fall victim to anaggressor. The US and Russia have been in the process ofreducing their stockpiles from the level of tens of thou-sands of warheads needed to counter a “first strike” to thethousands of warheads needed for deterrence. Thenuclear weapons mission is to sustain and maintain theUS reduced stockpile for the foreseeable future. Theexisting stockpile consists of weapons systems highlyoptimized for specific missions and for the maximumyield to weight ratio. They were designed for a 15–30year shelf life with little consideration given to possiblelonger-term aging issues. The weapons program now hasthe challenge of adapting the existing warheads for dif-ferent missions, and extending their lifetimes to 40 to 60years without the ability to test the nuclear performance.The strategy developed for “Stockpile Stewardship” hasfour major elements:
1. active surveillance of the stockpile to identifyproblems and issues so that they can be fixed;
2. revival of the capability for manufacturing andrefurbishing weapons;
3. development of enhanced fidelity computer simu-lations for nuclear weapons;
4. development of an active experimental program tovalidate the new simulations.
The nuclear weapons community has been developingand using complex codes since 1943. Indeed, weaponssimulations were among the first applications for comput-ers. A typical mature nuclear weapons simulation codehas about 500,000 lines of Fortran or C code (Post andCook, 2000). Its development and support usually involvesan investment of about 200 to 400 man years spread outover 10–20 years. The code development and supportgroup usually has 5–15 members who generally stay withthe same group for 10 years or more. The code is useddaily by a group of 10–50 weapons designers to analyzevarious weapons systems and experiments (Table 2).
The scale of the code development task is truly immense.The “legacy” weapons simulations codes were only ableto model variations in one or two spatial dimensions,provided under-resolved solutions to severely simplifiedequations, and used physical data and materials modelswith a largely semi-empirical basis. They were success-fully used to provide interpolated results between experi-mental results from underground nuclear tests. In a sense,they were highly sophisticated regression fitting algo-rithms. Without the ability to field test weapons to deter-mine the impact of new conditions due to aging andmodifications, the DOE turned to simulations. The out-growth of this new emphasis on simulation was the ASCIprogram. Clearly new tools were required with a muchmore reliable prediction capability.
The new ASCI codes thus have two and three spatialdimensions, provide adequately resolved solutions of
exact equations, and employ more accurate physical dataand materials models. The increase in computing powerfrom 1995 to 2004 required to achieve these advances isabout 105. The ASCI nuclear weapons simulation devel-opment program is complemented by a strong effort in thedevelopment of solution algorithms, physical data, mate-rials data, code development tools and massively parallelcomputer platforms and operating system software. Leg-acy codes generally took 10 to 20 years or more to matureand were often used for 30 to 40 years (Figure 1). Theywere developed by small teams – often just 3 to 5 profes-sionals. The ASCI codes are needed in about 10 years orless, and, since they have as many or more components –each with more complexity – than the fully mature legacycodes, parallel component development is required. Thisresults in larger code teams, up to 20 or 30 staff.
2 Build on the Successes of Your Institutional Software Development History
One of the most successful approaches is to look at yourorganization and similar organizations and see what hasworked and what has not. This not only appears to be a suc-cessful approach at LANL and LLNL but is recommendedby the authors of myriad software books and courses (e.g.McConnell, 1997; Remer, 2000). A quantitative databaseof successful software projects is required for good esti-mation for resources and schedules (Jones, 1998).
LLNL has been successfully developing the capabil-ity for simulating the performance of nuclear weaponssince the late 1950s (Post and Cook, 2000). They have suc-cessfully developed state-of-the-art simulations for everymajor supercomputer platform since the 1950s (Figure 1).They have successfully coped with massive changes incomputer languages, operating systems, platform archi-tectures, and memory structures. It is not uncommon fora nuclear weapons code to have a useful life of 30–40
Table 1Code development “lessons learned” from the ASCI program at LANL and LLNL.
• Build on the successful code development history and prototypes for your organization • Good people in a good team are the most important item• Software project management: run the code project like a project• Risk identification, management, and mitigation are essential• Schedule and resources are determined by the requirements (goals, quality, team survival and building, and added
value)• Strong customer focus is essential for success• Better physics is much more important than better computer science• Use modern but proven computer science techniques; do not be a computer science research project• Train the teams in project management, code development techniques and the physics and numerical techniques used
in the code• Software quality engineering: use best practices to improve quality rather than processes• Validation and verification are essential
401LESSONS LEARNED FROM ASCI
years. The major elements of their success appear to bethe following.
• Strong emphasis on building and supporting codedevelopment teams and expertise.• Long term (5–10 to 20 years or more) support of
code teams with low turnover rates.• Code teams composed of a mixture of senior, expe-
rienced staff and younger staff, with the senior staffmentoring the less experienced staff.
• Computer scientists as an integral part of codedevelopment teams, usually through a long-termmatrix assignment or direct hire into the code devel-opment team.
• Three-phase life cycle (Figure 2). • Development of initial capability and initial use by
customers: 5–10 years.
• Enhancement of initial capability and support ofheavy use by customers: 10–30 years.
• Retirement and phase-out, support of declining userbase: 5–10 years.
• Strong customer focus and continuous direct interac-tions with customers, either by embedding the codedevelopment teams in the user organization, or in organ-izations that are closely coupled to the user organization.
• Requirements come from customers (the users),requirements creep minimized.
• Prior projects serve as prototypes for new projects.• Code development proceeds in steps: develop a core
capability with a small team; let the users try it; if suc-cessful, add more capability (i.e. incremental delivery).
• Strong emphasis on the improvement of the physicscapability.
• Conservatism with regard to computer science.
Table 2Characterization of nuclear weapons simulations (Post and Cook, 2000).
Property LANL/LLNL
Code complexity 20–50 independent packages to simulate different physics phenomena; massively parallel, iterative physics
Product Working code delivered to local users, support for individual users common
User base Small, homogeneous and collocated, tight coupling of users and developers
Code size 100,000 to 1,000,000 lines of executable code (loc); typically ~ 500,000 loc
Code updates releases Major, 1–2; minor, 20–100 or more per year
Technology risk Algorithm R&D is necessary to develop new methods to solve physics problems
Funding Level of effort, only loosely determined by scale of task, resource elements only roughly esti-mated by historical trends
Fault tolerance level High; users can recognize and filter faults and defects and get rapid fixes from code groups; often users suggest fixes
Size of code groups 3–25 professionals (computational physicists, engineers, programmers, computer scientists, computational mathematicians, theorists, etc.)
Project lifetime 10–35 years, usually ongoing continuous development of codes
Responsiveness Code groups must respond rapidly to user requests, hours to days for simpler fixes, months to years for new algorithms and physics
Requirements Largely captured in prior codes and corporate experience, also users can set requirements
Multiple use codes Codes are used for both research and production; design experiments, analyze problems with stockpile
Code evolution Continual replacement of code modules as better techniques are developed
Module coupling Modules are tightly coupled across disparate time and distance scales, usually operator split-ting is adequate, but iterative solvers are beginning to be used for closely coupled physical phenomena
Verification Comparison of calculated results with analytic test problems, comparison with other codes, checks on conserved quantities, regression test suites, infrequent convergence tests
Validation Comparison with data from underground nuclear tests; comparison with past and current above ground experiments
402 COMPUTING APPLICATIONS
Fig. 1 LLNL computer history.
403LESSONS LEARNED FROM ASCI
A perhaps surprising element of the LLNL experience isthe lack of formal requirements for new projects. Theneed for a formal set of requirements is mitigated by theclose interactions between the code developer and users,the use of prior simulation codes as prototypes (each newcode provides improved capability with relatively fewcompletely new features), and the corporate knowledgeof the code developers. As the code projects have becomelarger and the average experience level has dropped, theimportance of more formal requirements has grown.
LANL has had similar experiences, although they havestarted fewer codes than LLNL. Their strategy has gener-ally been to import codes with initial capability fromother institutions, then modify and enhance the importedcode to provide the capability needed by the users.LANL’s efforts to develop new codes from scratch havebeen unsuccessful. The customer focus has generallybeen provided by embedding the code teams in the usergroups and by collocation. LANL has attempted to havea stronger emphasis on innovative computer science (e.g.Oldham, 2002), but has not realized the expected benefitfrom that emphasis. In fact, the code projects with aheavy emphasis on advanced computer science havebeen among the least successful. LANL has experi-mented with different organizational structures, includ-ing separating the code developments from the usergroups, but has been most successful with collocatedcode development teams and user groups. This hasproven to preserve a strong customer focus.
3 Good People in Good Teams are the Basis of Successful Code Development Programs
DeMarco (1997) states that there are four essentials ofgood management:
• “get the right people”;• “match them to the right jobs”;• “keep them motivated”;• “help their teams to jell and stay jelled”
(“all the rest is administrivia”).
The experience at LANL and LLNL confirms this.DeMarco’s statements are true for software projects ingeneral, and are even more true for technical softwareprojects such as large-scale, multiphysics complex com-putational simulations. Such simulations consist of dif-ferent physics modules that interact strongly. There shouldbe at least one team member who is an expert in eachrequired physics and computer science capability, andpreferably more than one to provide risk mitigation and acritical mass for discussion and resolution of difficultissues. All of the team members must have the full pro-fessional respect and trust of the other team members.The team members must be able to openly discuss issuesand difficulties with each other and with management.
Management needs to trust and support the team andteam members (DeMarco and Lister, 1999; Cockburn andHighsmith, 2001; McBreen, 2001). The job of manage-ment is to facilitate, guide, and coach the team. Manage-ment needs to provide resources, solve problems that theteam cannot solve, and provide a stable work environ-ment. Thomsett (2002) puts it as, “People, not resources,
Fig. 2 Weapons code lifecycle.
404 COMPUTING APPLICATIONS
work on projects.” The care and feeding of project teamsis the main task of upper-level management for successfulcode development organizations (DeMarco and Lister,1999). Good teams of competent people are vastly moreimportant than good processes (Cockburn and Highsmith,2001; McBreen, 2001). Good teams are characterized bythe ability to work together informally and share ideas.Management has the duty to provide the type of environ-ment that nurtures and supports good teams. Finally, theteam provides the continuity of corporate knowledge thatforms a basis for future code projects. Thomsett quotesBill Gates as stating that “every code project at Microsofthas two deliverables: a working code and a solid codedevelopment team” (Thomsett, 2002). Without good teams,an institution cannot maintain and support its existingcodes or build new ones. Many, if not most, of the presentASCI code project teams are led by “heroes”. As Freder-ick Brooks points out in “The Mythical Man-Month”, allgood code projects need “conceptual integrity”, a clearvision for the code and how to realize that vision. A coher-ent vision can be developed only by a few individuals, orjust one individual: the standard role for “heroes”. While“heroes” have been essential to successful code projects,particularly code projects with less than ten staff, larger-scale code projects need to evolve into more of a struc-tured organization. As the ASCI projects grow, the myr-iad tasks become more than one person, no matter howtalented and hard working, can accomplish. The hero needsto be able to bring in others to help realize the vision. Atsome scale, the hero becomes a single point of failureinstead of a single point of success. The “heroes” need toevolve into senior mentors and code architects who com-municate their vision and expertise to the other code teammembers, and mentor the more junior staff.
While the central importance of teams may seem obvi-ous, a number of senior managers at LANL and LLNLhave questioned the value of teams and have ended updestroying successful teams. They did not appreciate thatit takes years to build up a good team, but only minutes todestroy one.
4 Sound Software Project Management is Essential for Success
Each code project should be managed as much like a stand-ard software project as possible (DeMarco, 1997; Petersand Pedrycz, 2000; Remer, 2000; Highsmith and Cock-burn, 2001; Humphrey, 2001; Pressman, 2001; Thomsett,2002). However, due allowance has to be given to the dif-ferences between highly complex scientific softwareinvolving a healthy dose of research to develop improvedsolution algorithms and projects that do not require suchresearch. Contingency allowances and planning and riskmanagement and mitigation are essential for success. The
project leader needs to have control of the budget, thestaff, and other resources to continually adjust the taskassignments so that the project goals can be met in a rea-sonable time. The project leader must be able to ensurethat the code team members cooperate and work togetherconstructively. The leader must have the support of theproject sponsors and the stakeholders, and must havetheir strong backing to manage the project. Responsibilityand authority go together. The project leader must be ableto remove disruptive team members. If the project leaderdoes not have this general level of authority, the projectleader is not a project manager, but only a project “cheer-leader” (Remer, 2000). The project should have clearlyunderstood goals, a stable budget and resources, a realis-tic schedule, and support from the external stakeholdersfor elements of the project that are essential for success,but are to be provided by external stakeholders. The statusof the project should be measurable, and each team mem-ber should have access to the status at all times. Trainingin software project management has proved to be veryuseful as well. All of this seems obvious, but it has beensurprising how often one or more of the above tenets havebeen violated at LANL and LLNL.
5 Risk Identification, Risk Management and Risk Mitigation and Key Elements of Success
DeMarco lists five major risks for software projects(DeMarco and Lister, 2002):
1. uncertain or rapidly changing requirements, goalsand deliverables;
2. inadequate resources or schedule to meet therequirements;
3. institutional turmoil, including lack of managementsupport for code project team, rapid turnover, unsta-ble computing environment, etc.;
4. inadequate reserve and allowance for requirementscreep and scope changes;
5. poor team performance.
To these we add two more:
6. inadequate support by stakeholder groups that needto supply essential modules, etc.;
7. tackling a problem that cannot be solved with theavailable resources in the available time.
It is revealing that only item 5 is the responsibility of theteam. It has been frequently assumed that most unsuc-cessful software development efforts fail because theteam did not perform, but that is not the experience in thesoftware industry or at LANL and LLNL. DeMarco and
405LESSONS LEARNED FROM ASCI
Lister (2002) state that uncertain requirements and inade-quate resources and schedule are the most common causesof project failure, and that is the experience at LANL andLLNL. This is where the generally informal requirementsspecification process at the labs is dangerous. It has workedas well as it has so far because the users have been able toadjust their expectations to what is actually delivered andhave been strongly involved in the process at every level.The labs have over 50 years of experience in nuclearweapons modeling, and thus know in considerable detailwhat physics needs to be in the codes. The only real issueis the degree of improved capability. The degree of involve-ment lessens as the code project teams become larger, anda more formal requirements process would add value.Some flexibility in the requirements specification phaseis essential because it is difficult to predict when (or if) aresearch program to invent or discover a new algorithm ormodel will be successful (e.g. successful development ofa new materials model that provides better fidelity).
Inadequate schedule and resources have also been projectkillers at the labs. Often, resources are picked at a levelthat can be obtained or are in the externally determinedbudget and the schedule is picked to match the desires ofthe program for a capability. If these are picked withoutregard to what can actually be accomplished, failure usu-ally occurs. In his book, “Death March”, Yourdon (1997)documents that “overly ambitious schedules” have killedmany code projects and destroyed many code teams, mostlyunnecessarily. We discuss the importance of schedule andresource estimation in the next section. Remer (2000)states that annual code team turnover rates of 15% or morewill doom a multiyear complex code project because thecorporate memory will vanish and too many team mem-bers are taken away from the development work to trainthe new team members. Poor team performance is oftenblamed for project difficulties, but close examinationalmost always shows that, while it may be a factor, theother risk items usually dominate, and usually contributeto poor team performance. For example, several majorsecurity incidents at LANL led to much institutional tur-moil that contributed to poor morale and poor perform-ance for the code projects at LANL. However, this isreally an institutional turmoil issue rather than a team per-formance issue. Also, poor team performance does notsuddenly become evident overnight. Management has aresponsibility to identify poor performers and disruptiveteam members and remove them from the team. Poor per-formers not only do not complete their work but alsoimpede and discourage their team members. Disruptiveteam members will destroy the team. Also, if the codeproject is relying on another organization to provide anessential package or module, and the other organizationdoes not deliver a workable package, the project will fail.Early involvement by upper-level management is usually
crucial to avoid this problem. This cannot be fixed by theproject manager or the project team. It can only be fixedby the project sponsor who is part of upper level manage-ment (Thomsett, 2002). Finally, it must be possible tosolve the equations that represent the system being mod-eled with available techniques and computing power.Tackling a problem that cannot be solved will not lead toa successful code project.
The team needs to identify its risks and build contin-gency into the development plan to increase the chancesof success in event of problems. An example is the needto pursue multiple approaches for the development of algo-rithms and modules on the critical path. If one approachturns out not to be feasible, there is a second approachalready being developed. Similarly key staff members needbackups.
6 Requirements, Schedule and Resources Must Be Consistent
Successful completion of a software project, indeed anyproject, requires that the project have a consistent andrealistic set of requirements, schedule and resources (Postand Kendall, 2002). A cardinal rule (Verzuh, 1999) forsuccessful technical projects is that one can specify atmost two but not three of these. What is not appreciatedas well is that software development projects are muchmore restrictive. For these projects, one can specify onlythe requirements (Jones, 1998; Highsmith, 2000). Speci-fication of requirements determines the optimum sched-ule and resource level. One can do worse than the optimum,but not better. The general experience at LANL andLLNL and in the commercial software industry bears thisout. In particular, we judge that this is one of the mainreasons that many of the ASCI projects have often failedto complete their prescribed milestones on schedule. Thislack of success in meeting their milestones has generallygiven the false impression that these projects were fail-ures when in fact the prescribed schedule was unrealisti-cally optimistic. We are beginning to find that, givenadequate time, most of projects could meet their require-ments. The ASCI milestones were set at the beginning ofthe ASCI program in 1996 with the implicit assumptionthat the project requirements, schedule and resourcescould be independently specified. An approach bettersuited to ultimate success would have been to specify theproject requirements and then develop the project sched-ule, milestones and necessary resources as part of a plan-ning process for each code project. The ASCI program isnow in the process of reformulating the milestones so thatthey incorporate a realistic schedule and are better reflec-tions of overall programmatic goals.
Accurate estimation of software project schedules andresource requirements requires quantitative data that char-
406 COMPUTING APPLICATIONS
acterize code capability and performance, and the timeand resources required to develop that capability. Withoutsuch data, accurate estimation is difficult for new projectsand estimates must be derived during the initial stages ofthe project (McConnell, 1997). Data from similar projectscan also be used. Unfortunately, such data exist at an onlyapproximate level within the ASCI program. In lieu ofdetailed historical data, we have adapted empirical scalingsfrom Jones (1998) calibrated using the LANL and LLNLweapons code history. Jones and others measure the capa-bility of software in terms of “function points” (FPs), aweighted total of inputs, outputs, inquiries, logical files andinterfaces (Symons, 1988; Jones, 1998). While FPs do notcapture all of the complexity of scientific software, theyare the best metric available in a simple form. Single lines ofexecutable code can be converted to FPs (e.g. equation (1)).Jones (1998) lists the equivalent single lines of code (SLOC)per FP for the common computer languages, since com-puter languages have different information densities:
(1)
schedule (months) = FPx ; 0.4 < x < 0.5; use x = 0.47 (2)
real schedule = contingency × function point schedule
+ delays (3)
(4)
Using FPs, Jones presents semi-empirical scalings (equa-tions (1)–(4)) for the time required to complete the project(schedule) and the recommended average team size thatwe modified.
The use of these scalings requires mapping from thecommercial software environment to the environments atLANL and LLNL. We therefore added the additional timeit takes to recruit, hire, train and process security clear-ances for code development staff as compared to condi-tions in the commercial software industry. We estimatethis additional time to be at least one year and probablymore like two to three years. It is less for staff recruitedfrom other parts of the laboratories, and at least two yearsor more for new hires. A reasonable average is about1.5 years. Commercial software companies typically havemuch shorter times of three to six months or less.
A contingency factor is also necessary. We used a stand-ard contingency model developed by the LLNL engineer-ing directorate and described in the Course on Software
Project Management by Remer (2000). The contingencyaccounts for the additional “viscosity”, risks and uncer-tainties involved in developing codes for the complex andchallenging LANL and LLNL computing environmentscompared to the commercial software development envi-ronment, typically on mature single processor Windowsor Unix boxes with stable compilers and operating systems,less stringent security requirements, and relatively straight-forward algorithms. Examples of items that add viscosityfor LANL and LLNL weapons code projects include:
• computing on two disjoint and unconnected comput-ing systems (classified and unclassified);
• delays and low worker efficiency due to the instabilityand immaturity of the new ASCI platforms and thesoftware for code development (compilers, operatingsystems, debuggers, etc.);
• the extra work required because the platforms, operat-ing systems and code development environments changeevery two to three years;
• the paradigm shift from single processor systems tomassively parallel platforms;
• the general complexity of the multiphysics models inthe codes;
• the transition from two-dimensional to three-dimen-sional models;
• the need for extensive algorithm research and develop-ment.
The LLNL contingency model with the assumptions madefor each of these factors yields a contingency factor of 1.6for new projects. A contingency of 1.6 is somewhat highfor standard engineering projects, but is not atypical ofengineering projects that have significant technical riskand require a substantial amount of technology R&D(Remer, 2000).
To develop semi-empirical scalings to estimate scheduleand team size on LANL/LLNL simulation code projects,we modified the form of the Jones scalings to account forthe LANL/LLNL environment and then incorporated theLLNL contingency factor. Then we calibrated these scal-ings using the experience of seven weapons code projectsfrom LLNL and LANL, including six ASCI code projectsand one legacy code (Table 3, Figure 3). We also modi-fied the scaling for team size to provide a better fit to thedata (equation (3)). The minimum team size of three is areflection of the complexity of multiphysics projects, andthe factor of 0.6 reflects the challenges of integratingcomplex multiphysics codes that limit the size of the codeteam and the high degree of specialized training for thecode team (cf. Brooks, 1987). The team size is the peakstaffing level for the code project (Figure 3).
We analyzed seven code projects, three at LLNL andfour at LANL (Table 3). We have identified the LLNL
FPC ++ SLOC
53------------------------------ C SLOC
128---------------------- F77 SLOC
107----------------------------+ +
=
150---------FP
team size = 3 + * 0.6.
407LESSONS LEARNED FROM ASCI
codes with the letters A and B to avoid classificationissues. Table 3 lists the size of the code in FPs, the timeestimated by equation (4) to develop the initial capabilityof the code project, the actual age of the code at the pointit was expected to accomplish its first milestone, whetheror not the project succeeded, the optimal code team sizeestimated from equation (3), the actual size of the team,and the estimated man years and actual man years for theproject. The ASCI planning and milestone schedule aresummarized in Figure 3. The scalings indicate that codesof the ASCI class should take at least seven to nine yearsto develop, consistent with the ages of the projects thateither succeeded in meeting their prescribed milestonesor did not. The scalings for the time required for a codeproject to succeed (i.e. meet its ASCI milestones) areconsistent with the observed historical data if a contin-gency factor of 1.6 is used. The codes that were success-
ful were all older than the age that was estimated to benecessary by equation (2) (generally about eight years).Those that were unsuccessful in meeting their milestoneson schedule are all younger than the estimated requiredtime. The average team size is between 15 and 25, closeto the observed team sizes.
Four of the ASCI codes were started before ASCI beganin 1996 (ASCI B, Legacy A for LLNL, and the Crestoneand Blanca code projects for LANL). ASCI B was startedin 1992 and had a working prototype in 1994. The Cre-stone code project was started before 1992. ASCI A andthe Shavano code project were started in late 1996 andearly 1997. The Antero and Blanca code projects had rootsthat extended back before 1992. The core of the Blancacode project was a working parallel Fortran code importedfrom another institution. The Blanca team decided to“modernize” it by completely rewriting it in C++ with
Fig. 3 ASCI planning, milestone and code project schedules.
408 COMPUTING APPLICATIONS
admixtures of “advanced computer science” techniquesthat were then being developed. LANL retained the origi-nal Fortran imported code with minimal support (gener-ally 0.5 to 1 staff). It outperformed the codes from theBlanca code project during the entire life of the Blancacode project. The Antero code project team sought to join
two existing separate mature code packages that had beendeveloped in the late 1980s and early 1990s. Neither theBlanca nor the Antero code project was successful inmeeting their milestones because they violated many ofthe “lessons learned” in Table 1. These lessons were expen-sive. The LANL ASCI program spent about $50M on theAntero code project and close to $100M on the Blancacode project. Since we are able to match the history ofweapons codes with scalings derived from the experienceof the commercial software industry, we can also con-clude that the constraints, computer science practices andmanagement issues that generally apply to the commer-cial environment apply to the development of weaponscodes (i.e. there is no “silver bullet” that can radicallyreduce the development time; Brooks, 1987).
The historical evidence and the estimation proceduresindicate that it takes generally a minimum of eight yearsto develop an initial capability for a weapons code. Therequirements for a weapons code are fixed by the physicsnecessary to simulate a nuclear weapon. LANL and LLNLhave over 50 years of experience in this area, and knowthese requirements in detail. The requirements are thusnot very flexible. In terms of FPs, weapons codes requireat least 3000 FPs and some require up to 6000. For manyreasons, it is difficult to field code teams of more thanabout 20 staff, but this is adequate according to the esti-mation procedures and historical evidence. These pointsare reflected in Figure 5.
Experience with the ASCI code projects is consistentwith the general experience of the software industryembodied in standard software estimation methods whencontingencies are included. ASCI B at LLNL and theCrestone code project at LANL have been very success-ful in meeting their mileposts. In each case, the age of the
Table 3Software resource estimates for the LLNL and LANL code projects. Items in bold denote computednumbers (equations (1)–(5)) and items not in bold are historical data.
LLNL LANL
ASCI A ASCI B Legacy AAnteroCode
Project
ShavanoCode
Project
BlancaCode
Project
CrestoneCode
Project
Single lines of code 184,000 640,000 410,550 300,000 500,000 200,000 314,000
Fig. 4 Time required to complete a project and averagecode team size as a function of code capability meas-ured in FPs (Symons, 1988; Jones, 1998).
409LESSONS LEARNED FROM ASCI
successful project at the time they met their milepostsexceeded eight years. ASCI A at LLNL and the Anteroand Shavano code projects were not successful at meet-ing their initial milestones. In those cases, the age at thetime the milepost was due was several years less than therequired time estimated from equation (4). In fact, at thetime they failed to meet the relevant mileposts, their agewas generally half of the age of the successful projectsand of the required time estimated from equation (4).
The various projects had different computer languages,framework structures, project organizations, degrees ofsoftware quality processes, platform vendors, laboratorymanagement structures, staff maturity levels, and manyother factors. These factors undoubtedly played somerole, but success seems to correlate consistently with ageof the project. Adequate time was a necessary, but notsufficient, condition. The Blanca code project failed eventhough it had adequate time.
The software project management literature and soft-ware development experience (DeMarco, 1997; McCon-nell, 1997; Remer, 2000) stress the need to start a projectwith a small number of staff who can develop the con-cept and plan. Once the concepts and plan are developed,the team can then be staffed up to the level needed toaccomplish the project. Too large a staff at the beginningleads to a confused plan (e.g. too many cooks spoil thesoup) and locks the project into directions that often donot contribute to project success (Figure 5).
To determine the sensitivity of the estimates to theassumptions for the estimation formulae, we varied theexponents in equation (2) and the contingency (Figure 6).The estimates changed by 10–20%, but the conclusionswe draw from Table 3 remain valid. Indeed, the initialASCI milestones were shorter than even the estimateswith no contingency.
A number of senior managers at LLNL and LANL havesuggested that had the projects been more “aggressively”
their milestones on schedule (cf. DeMarco, 1997). In factsenior management subjected much pressure on all of thecode projects that did not meet their milestones. Exten-sive overtime was authorized. During the period justbefore the milestone, senior management monitored theprogress on a weekly and sometimes daily basis. Thecode teams worked very hard, but were not able toachieve a sustained increase in their rate of progress. Thisis consistent with the experience in commercial softwaredevelopment (DeMarco, 1997; Yourdon, 1997). DeMarco(1997) notes in his books that “aggressive” managementand management pressure are never successful in acceler-ating the code development schedules for more than avery short time interval. A basic reason is that the limitingfactor in software development is the rate at which soft-ware developers think – the rate at which they can solveproblems. “Management pressure and aggressive manag-ers do not make people think faster” (DeMarco, 1997). Infact, excessive pressure slows projects down and retardsdelivery of the project, if it does not kill the project (Fig-ure 7; Esque, 1999; DeMarco and Lister, 2002). Para-phrasing Yourdon (1997), “overly ambitious softwaredevelopment schedules are the leading cause of softwareproject failure.”
Fig. 6 Variation of estimated project times for differ-ent assumptions for the contingency and for the expo-nent in equation (2).
managed, they would have been successful at meeting
410 COMPUTING APPLICATIONS
Another way to state the lesson to be drawn is that soft-ware development time has been very inelastic for theASCI code projects, as it has been for the software industryin general. When resources have been added to a projectthat is late, it has not helped very much and has sometimeshurt. For one thing, the new personnel need to be trained andintegrated into the team, and this often slows things down.In addition, the complexity of the project may not lenditself well to a large staff. The empirical scaling quoted byRemer (2000) is that the resource level, R, required tospeed up a project behaves as R ~ t –4 where t is the time fordevelopment. Decreasing the schedule by 20% requires a150% increase in resources. This assumes that theresources can be utilized efficiently, something that isvery challenging for complex, technical software projects.
History – both in the ASCI program and in the softwareindustry in general – shows that software project require-ments, schedule and resources are not independent. Theschedule and resources are determined by the projectrequirements. This important fact must be recognized iffuture ASCI and other large, complex multiphysics codedevelopment projects are to be successful and meet theirmilestones. Milestones must be determined as part of thedetailed project plans for each project, not set independ-ently. If projects have different maturity levels and simi-lar requirements, they will succeed at different times, notat the same time. As noted before, there is also strongevidence that “aggressive” management retards success,rather than hastening it (DeMarco, 1997; Yourdon, 1997;Esque, 1999). The DOE ASCI program has, over the lastyear, worked with LLNL and LANL to modify the origi-nal milestone plan (Figure 3) to make the plan more con-sistent with the requirements of the weapons program andto ensure that the milestone schedule is realistic. This is a
key step forward since it is essential that milestone sched-ules be consistent with sound software project planningand what the code development projects can actuallyaccomplish, rather than what management thinks theyought to be able to accomplish. Otherwise, the ASCI codeprojects that do not have the time or resources to meet themilestones on schedule will not be able to do so. Theseprojects will then unjustly and inaccurately be judgedfailures and the ultimate success of those projects willunnecessarily be jeopardized.
The need for consistent requirements, schedules andresources is actually more demanding for software projectsthan for normal construction projects or than stated above.Thomsett (2002) points out that there are in reality sevenelements for a project, not three. A successful projectshould:
• meet the project’s objectives/requirements/goals/deliv-erables;
• meet an agreed budget – resources, capital, equipment;• schedule – deliver the product on time;• satisfy the stakeholders;• add value for the organization;• meet quality requirements;• provide a sense of professional satisfaction for the team.
Meeting requirements, budget and schedule are custom-ary project criteria. Satisfied stakeholders (those outsidethe project team who sponsor or support the project or arethe customers) are important for acceptance of the product.If the customers are not satisfied with the final product,the project will not be a success even if the requirementswere met. If the expectations of the stakeholder supportgroups or the sponsor are met, then the project really hasnot succeeded in the eyes of the institution. Managing andmeeting the expectations of the stakeholders is essential.Management and the project team both own this issue.Adding value to the institution is also an important crite-rion. For nuclear weapons simulations, this means that thecode is a substantial improvement over prior codes, andengenders greater confidence in the customers, e.g. users,lab management, DOE, Department of Defense (DoD), etc.,that they can make more accurate predictions. The codemust also meet quality requirements. Not only should thecode be relatively free of bugs and errors, but must be easyto use, and be robust and reliable. The DoD and DOE arebeginning to impose software quality standards on codesdeveloped with their funds. Improving software quality isexpensive, and meeting objective standards for softwarequality is even more expensive. Finally, an essential out-come is a satisfied code project team. Otherwise, the insti-tution will be unable to maintain the code and will nothave good teams to develop the next code. Resource andschedule estimates need to include these additional issues.
Fig. 7 Schematic illustration of the effect of manage-ment pressure on the schedule to complete a softwareproject (DeMarco and Lister, 2002).
411LESSONS LEARNED FROM ASCI
7 Code Project Team must have a Customer Focus
One of the most important criteria for success, mentionedfrequently above, is the customer focus of the code project.The most successful code projects have been collocatedwith the user groups and have interacted with the users onan almost daily basis. This helps to keep the code teamsresponsive and motivated, and helps to develop trustamong the code development team, the users and man-agement. The continuous interest by the users in theprogress of the codes helps sustain the enthusiasm anddedication of the code teams. It is easy to lose motivationon a project that lasts years and has few incrementaldeliverables.
There are many examples of code projects that havefailed because they lacked customer focus. The codedevelopers (and possibly upper-level management) feltthat they knew what the users needed better than theusers. Almost always they were wrong and the new codewas never used. In most cases it failed to deliver what theusers actually needed. In the few cases where the code didhave value, there was no user buy-in because of theadversarial relationship that had been established. Valu-ing and supporting the customers has been an essentialpart of every successful code project at LANL and LLNL.
Another reason for customer involvement is that thecustomers do much of the initial testing and almost all ofthe validation. Without their involvement, the code teamwill need to do all of the testing and validation. The codedevelopers will not have the same level of credibility thatthe users do. They also will not be as familiar with howto carry out validation, will not have the same level offamiliarity with the data, etc.
We observed that good teams, supported and nurturedby management, are normally much more focused on cus-tomers than teams that lack management support and nur-turing – points made in Cockburn and Highsmith (2001).In particular, heavy-handed top-down management seemsto discourage customer focus because the team then seesupper-level management as the customer rather than thereal customer.
The development of critical modules for multiphysicscode projects is often the responsibility of stakeholdergroups not part of the project team. It is vital that thestakeholder groups understand that their customers arethe code project teams. It is absolutely essential that thesupport groups view the success of their module withinthe integrated code project as more important as the suc-cess of the independent module on its own. The inte-grated code project will fail if critical modules do notfunction properly in the integrated code project. Achiev-ing this has been especially challenging for “discipline”or “functional” organizations such as LANL. Several key
LANL code projects have been seriously delayed orthreatened with failure due to problems with delivery ofmodules from stakeholders outside the core project teamor in other organizations. LLNL is organized more alongproject and matrix lines than LANL, but even LLNLoccasionally has problems with support groups providingcritical components.
8 Better Physics is the Key to Successful Prediction
The predictive value of the weapons simulations (or anyphysics simulation) depends on the quality of the physicsin the codes (the right equations, good quality physicaldata and materials models) and accurate solution algo-rithms for the equations and adequate spatial and tempo-ral resolution. To achieve the desired improvement inpredictive ability, the ASCI program has supplementedthe effort to develop better weapons simulation codeswith programs to develop improved linear and non-linearsolvers, better materials models, better physical data forequations of state, opacities and neutral particle cross-sections, better models for turbulent mix, and more accu-rate transport algorithms. This effort to develop betterdata and modules for the simulation codes has also beensupplemented with a strong experimental program to pro-vide experimental data to help improve the physics basisof the models and to validate both the detailed models andthe integrated multiphysics calculations. When the weap-ons program was conducting underground nuclear tests,the weapons design codes could be benchmarked and cal-ibrated with the test data from an actual weapons test. Thetest data usually provided information in integral form,i.e. the total performance of the device. Partially becauseof the hostile physical environment in the vicinity of anuclear explosion, data on detailed effects were difficultto obtain. Since many of the materials models and othermodels had a semi-empirical basis and the physical datahad uncertainties, it was generally possible to adjust thedata and models within the range of uncertainties to fit theintegral data. This was generally adequate for interpola-tion within the general problem domain bounded bynuclear test data, but was not adequate for extrapolationand prediction outside of that domain.
The paradigm shift from interpolation to prediction hasbeen fundamental (Laughlin, 2002). Simulation errors arepotentially limited when the domain is bounded andspanned by experimental data, so interpolation has somevalidity. Parameter adjustments can be made in order tomaximize the cancellation of errors due to inadequate treat-ments of different effects. However, when predictions aremade outside the domain of the test data, the errors arethe sum of the errors due to each element of the calcula-tion. It is not possible to rely upon compensating errors.
412 COMPUTING APPLICATIONS
The only way to improve the predictive capability is toimprove the quality of every part of the calculation. Thesame considerations generally apply to scientific software.
This paradigm shift has involved a major culture change.Managing the culture change in an evolutionary fashionhas also proved to be a challenge. The code developmentprograms have had to continue to support the old toolswhile developing the new tools. The funding agenciesand review boards have had difficulty understanding theimportance of maintaining the old tools that the usersneed until the new tools are fully functional. It is first nec-essary for the new codes to achieve the capability of theold tools before moving forward to better capability. Oth-erwise the connection to the validation database and theuser community is not maintained. This issue needs to beexplicitly featured in any strategy for massively upgrad-ing the quality of a computational science effort. Other-wise, the code developers are caught in a “catch 22”situation where they spend much time in limbo trying tokeep the old codes alive and healthy while trying to developthe capability to replace them with better ones and beingunable to get sympathy for this from their sponsors.
9 Use Modern, but Proven Computer Science
An important element of success is minimization of risks.The general experience in the labs is that the major payoffcomes from better physics. Developing a better treatmentof the physics usually is sufficiently challenging that con-servatism for the other aspects of the code developmenteffort is essential. This particularly applies to the role ofcomputer science in an application code project. Good,sound computer science is essential for a successful project,particularly those in the ASCI program. The codes haveto run on the very latest massively parallel platformswith thousands of processors. The platforms and platformarchitecture change every two or three years. Soundconfiguration management is essential for a complexmultiphysics code involving as many as 20, 30 or moreseparate, large-scale modules being developed and inte-grated by a team of 10–30 members of staff. Obtainingreasonably efficient performance on the complex plat-forms is a demanding computer science task. Visualiza-tion of large data sets (up to terabytes) is essential fordebugging, problem generation, and analysis of theresults. The computer scientists need to be fully inte-grated into the team and treated as professionals withequal professional status to the physicists. Use Fortran 90,not Fortran 2000 which has not been standardized. If youuse C++, do not use templates and inheritance classes inall their glory. Do not use MPI II until it has been shakenout and so on. Do not participate in the latest computerscience fad. Let the new ideas mature, and let someone
else get all the glory and the pain of using the latest andgreatest computer science. You will have enough chal-lenges.
Given the challenges associated with successfullydeveloping and integrating a complex, multiphysics codefor multiple massively parallel new and unstable plat-forms, getting the physics right, and porting it to a newplatform every two years, our record indicates that mix-ing in a strong component of new and unproven computerscience usually makes the total task much more difficultand is often fatal. For instance, almost all of the success-ful ASCI projects have used Fortran or C as the basic lan-guage. Those using C++ and other advanced languages (ifthey succeed at all) have been slower to develop capabil-ity and have exhibited lower computer performance effi-ciencies than the C and Fortran codes. One factor is thatC++ and many other advanced languages have a steeplearning curve, and are more complex. Indirect address-ing often diminishes performance to unacceptable levels.The successful code projects have emphasized modular-ity, transparency, simplicity, and portability rather thanperformance optimization.
There have been several efforts to develop advancedbackplanes and code development environments, e.g.POOMA (Oldham, 2002) and Sierra (Stewart and Edwards,2001). The promise of such tools is great, but none hasbeen an unqualified success so far. LANL unsuccessfullyattempted to use the POOMA framework as the basis ofthe Blanca code project. As noted before, LANL spentover 50% of its code development resources (approxi-mately $100M) on the Blanca code project without suc-cess. Only when the computer science research elementswere replaced by an emphasis on physics and customerfocus was some success realized. The generalized frame-work had poor performance and was not sufficiently flex-ible to accommodate all of the different types of modulesand physics algorithms and mesh types that were neededfor the code to be a success. The Sierra framework hasbeen more successful for engineering applications, buthas taken a long time to mature. The prudent approach isto support computer science research, but wait for theresearch to mature and produce a reliable product.
10 Develop the Team
As noted above, software is developed by teams, not sys-tems, processes or organizations. The most effectivetraining is mentorship of the newer team members by themore experienced team members. In addition, formaltraining has proven very useful to help the teams jell andto improve their skills. DeMarco and Boehm (2002)stress that “part of our 20-year-long obsession with proc-ess is that we have tried to invest at the organizationallevel instead of the individual level… (we should be)
413LESSONS LEARNED FROM ASCI
investing heavily in individual skill-building rather thanorganizational rule sets.” LLNL and LANL have broughtin highly experienced software development leaders suchas Tom DeMarco, Rob Thomsett, Ed Yourdon and DonRemer to give courses in risk management, softwareproject management, estimation techniques, and softwarequality issues. This introduces a sense of best industrypractices into the work. Not only does this train the teamin the use of tools and methods, but also it helps to givethe team members a sense of perspective on how thingsare done at other institutions. The experience of partici-pating in a course with other members from the sameteam helps bond the team. Participating in a course withmembers of other teams helps develop bridges to theother teams, and enhances the sharing of experiences.
Team members should be encouraged to view codedevelopment as a professional activity. They should beencouraged to join and attend the conferences of the IEEEComputer Society and other computer-related organiza-tions, and to subscribe to the relevant journals. We havefound that forming an in-house library of software devel-opment books and journals for the staff also helps. Train-ing in the tools, languages, and methods for computing isalso essential. Even if the team members do not useadvanced languages, every computer literate code devel-oper should be trained in C, C++, PERL, Python, Unixand LINUX, etc. Similarly, training in the relevant phys-ics issues and numerical techniques is important. Semi-nars and colloquia with internal and external speakers arealso important. Inspiring speakers such as Fred Brooks,Tom DeMarco, and others make a positive difference. Inaddition to informal contacts, it has proven useful to havethe customers formally address the team and describe howthe simulation tools will be used and what the issues are.
The code development teams are the major asset of acode development organization and need to be nurturedand encouraged to grow and develop (Cockburn andHighsmith, 2001). Anything that can be done to improvethe skills of the team members and management isworthwhile. Team building retreats and planning retreatshave also proven useful.
11 Software Quality Engineering is Important: Practices and Processes
Software quality engineering (SQE) and software qualityassurance (SQA) are major issues for the commercial soft-ware industry, especially industries developing softwarefor the DoD. In the mid-1980s the Air Force and otherparts of the DoD were experiencing major overruns andfailures due to problems with delivery of software fromcontractors. Planes were crashing, rockets were not work-ing, and satellites were failing due to software problems.The Air Force set up the Software Engineering Institute
(SEI) at the Carnegie Mellon Institute of Technology todevelop a set of standards and methods for prospectiveDoD software vendors (Paulk, 1994). The SEI developedthe Capability Maturity Model (CMM) for vendors toadopt. The SEI surveys show that the CMM does improverepeatability and software delivery on schedule (Herb-sleb et al., 1997). However, there are costs associated withimplementing the CMM to improve the developmentprocess. In general, it takes of the order of two years toimplement the first level of improvement, and much effort(Herbsleb et al., 1997; Remer, 2000). The CMM is oftennot recommended for ongoing projects (DeMarco, 1997).The strong emphasis on reliability and repeatabilitycomes at a sacrifice of flexibility and innovation (Rifkin,2002). It is tough to explain to others why everyone shouldnot adopt the CMM or similar standards (e.g. ISO 9000).Who can be against quality? The problem is that onemethod does not fit all situations. Adopting SQA requiresresources and time in addition to reducing flexibility andinnovation, so a cost/benefit trade-off is essential.
We have found two ways of examining these issuesthat are helpful for developing a balanced perspective.The first is based on the taxonomy of Rifkin (2002) fortechnology development. Rifkin places software projectvalues and goals into the three categories or attributesdefined by Treacy and Wiersema (1995): operationalexcellence, product innovative, and customer intimate.In order to succeed, every project must have all threeattributes, but must concentrate and excel in only one.This attribute must be the element of the product thatmatches the desires and needs of the customers. An exam-ple of an operational excellent product is a simple butrobust computer program that must have no bugs becauseit must be 100% reliable, e.g. the software that controlsa jet fighter. Innovative software has new capability thatis essential, but necessarily entails some research anddevelopment, e.g. improved nuclear weapons simulationcodes. Customer intimate products are focused on thedetailed needs of the individual customer and must beflexible to respond to these varied needs of their custom-ers. They typically feature large menus and many options.An example might include the latest version of MS Word,with so many features that a person can go crazy trying toturn most of them off. Rifkin sees process driven systemsas being good at training an organization to produce oper-ationally excellent software, but poor at producing theother types of software. Our needs are more in the productinnovative category, and we need more flexibility than isachievable under highly process driven systems.
It is essential to use an organized approach to codedevelopment. It is necessary to appreciate that innovativeproducts are developed by creative, highly trained scien-tists who are quite independent minded, highly skepticaland not very willing to accept things on authority. What
414 COMPUTING APPLICATIONS
the code projects are trying to accomplish is very ambi-tious: the schedules are wildly optimistic, the code teamsare too small, and the users want the new code yesterday.We would never get them to be enthusiastic about layingeverything aside for months or years to “improve theirprocesses”. Scientists learn early in their careers to iden-tify and use things that add value to the way they carryout their work, and to avoid doing things that do not addvalue. Otherwise they would not have survived a PhDprogram in a scientific field. Scientists are trained toquestion authority and believe only what they can verifythemselves. This is precisely what we value in them. Wecannot expect them to adopt a different point of viewwhen it comes to how they do their work. As Roache(1998) states, creative software developers and scientistsand orderly process people are at the opposite ends of theMyers–Briggs personality scale and are naturally antago-nistic. We have found that it is much more successful toadvocate and employ “best practices” instead of “bestprocesses” (Phillips, 1997). The LANL software require-ments document (Cox et al., 2002), drawn up by the codeproject teams themselves, lists the set of best practicesthat the teams have judged useful for improving the waythey do business (Table 4; Cox et al., 2002). Getting the
teams to develop their own list of the practices is a keyelement in getting them to adopt the practices. It is also agood way to encourage the code teams to share experi-ences and expertise among different teams. Teams usu-ally cooperate and share practices more readily thanalgorithms and other “proprietary” packages where thereis sometimes more competition.
Some of the distinctions above are a matter of percep-tion. In fact, if we look closely, there is much in commonamong the CMM processes and best practices listed inTable 4. Table 5 lists the intersection of what we think isimportant for the code teams and the best people practicesspecified by the SEI. Table 6 lists the overlap of our bestpractices and the CMM level 2 processes/practices. Table7 lists the overlap of our best practices and higher-levelCMM processes/practices. The key difference is that thecode teams can decide what adds value and what does not.
It is interesting that the SEI has begun to emphasizesound software project management as an important ele-ment of successful projects (Humphrey, 2000; 2001).Indeed, sound management practices appear to be as suc-cessful as institutional process improvement for reducingthe defect rate (Humphrey, 2001). This matches ourexperience. There is no substitute for organizing the teamand the work, monitoring the work as it progresses andrearranging the tasks as the project evolves.
Table 4Some essential elements for software projectmanagement
Software requirements management (documented and controlled)Careful software designSoftware project planning and trackingWork breakdown structure for the project – lists of tasks to be accomplishedEstimation of the resource and schedule requirements for the projectDevelopment of a plan for accomplishing the tasksDevelopment of a schedule for accomplishing the tasksIdentification of the resources needed to accomplish the tasksMonitoring and tracking the planRisk assessment and management and contingency planningDefinition and control of the interfaces between project elementsConfiguration managementExtensive regression testingIntegrated system testing and verification tests Documentation (including requirements, functional specifications, critical software practices, physics and algorithm description, and a user manual)Incremental delivery of capabilityContinual interaction with customersConstruction of prototypes
Table 5Best practices: people issues
Users – understand themBuy-in and ownership by everyoneTechnical performance related to value for the businessExecutive sponsor and support by stakeholdersFewer, better people (management and technical)Use of specialistsClear management accountability
Table 6Best practices: contained in the CMM level 2
Document everythingUser manuals (as system specifications)Documented requirements Fight featuritis and creeping requirementsCost estimation (using tools, realistic versus optimistic)Planning and use of planning toolsQuality gates (binary decision gates)Milestones (requirements, specifications, design, code, tests, manuals)Visibility of plans and progressProject trackingDesign before implementingRisk managementQuality controlChange management
415LESSONS LEARNED FROM ASCI
Attention to software quality is important for otherreasons too. The sponsor is accountable for the success ofthe project. If he does not think the team is giving ade-quate attention to quality, he may impose processes thatare probably less well suited to the team than those theyidentify themselves.
12 Verification and Validation are Essential for an Accurate Simulation Capability
Verification and validation are essential elements for sci-entific codes (Lewis, 1992; Roache, 1998). All of us haverefereed computational physics papers that we could notprove were right. At best, the results were plausible. Infact, if a code is not validated and verified, the users haveno reason to believe that the results have any connectionwith reality (Laughlin, 2002). We define verification asensuring that the code solves the equations in the codecorrectly, i.e. that there are no coding errors or mistakesin the code. Validation is defined as ensuring that thecode results are a faithful reproduction of the naturalworld, i.e. that the models expressed in the code are correct.Verification is a mathematical exercise. Validation con-sists of comparing the code results with experimental data.Benchmarking is comparing the code results with theresults of other codes. Benchmarking is useful during thecode development process but does not give the same levelof assurance of accuracy as verification and validation.
Both verification and validation are essential if thecode results are to have credibility with the customers.For the nuclear weapons codes, validation is largely car-ried out by the users who compare the code results withthe results from underground tests for full systems andabove ground tests for specific physics features. Verifica-tion is carried out by comparing the code results for modelproblems that have analytic answers or by convergencerate tests. A third verification technique is “manufacturedsolutions”. This involves the creation of hypothetical ana-lytic solution that is turned into a real solution by addinga source term to the original equation that makes the man-ufactured solution the exact solution of the modified equa-tion (Salari and Knupp, 2000; Roache, 2002). Validationproceeds at different levels. Each module must be validatedby single purpose experiments, and then the integrated code
must be validated. Integral experiments (ones that, forexample, produce a single value to represent a complexsystem) are not sufficient. They do not test for compen-sating errors and effects. The nuclear weapons program iscarrying out both small-scale experiments for validatingsingle physics effect modules and large-scale experimentssuch as the National Ignition Facility at LLNL (Lindl, 1998)to provide validation data for multiphysics calculations.
13 Summary
After eight years of the ASCI program, a number of keyconclusions and lessons we have learned are evident, asfollows.
• Good teams of highly competent staff are essential.Everything else is second order.
• Schedules and resource levels are determined by therequirements. Setting them independently will wreckthe code projects.
• Base the schedule and resource estimates on yourinstitution’s code development experience and history.
• Run the code project as an organized project.• Identify the risks and provide mitigation. In particu-
lar, set clear requirements, insist on management andstakeholder support, and obtain adequate schedule andresources with contingency.
• Maintenance of customer focus is essential for success.• Better physics is the most important product.• Minimize risks, use modern but proven computer sci-
ence; computer science research is not the goal.• Invest in your people with training and support.• Emphasize “best practices” instead of “processes”.• Develop and execute a verification and validation pro-
gram.
AUTHOR BIOGRAPHIES
Douglass Post is a physicist in the Physics Division atthe LANL. He was the Deputy Division Leader for Simu-lation in the Applied Physics Division at LANL from 2001and 2002. From 1998 to 2001, he was the Associate Divi-sion Leader for Computational Physics for “A” and “X”Divisions at the LLNL. He graduated from Stanford Uni-versity with a PhD in physics in 1975. He has 30 years ofexperience with the development of technical softwareand computational physics and project management inmagnetic fusion, atomic and molecular physics, transportphenomena and nuclear weapons at LANL, LLNL, andthe Princeton University Plasma Physics Laboratory. Dougwas leader of the tokamak modeling group at the PlasmaPhysics Laboratory from 1975 to 1993. He was head of thePhysics Project Unit for the International ThermonuclearExperimental Reactor Conceptual Design Phase (1988–
Table 7Best practices: contained in the CMM higherlevels
Reviews, inspections, and walkthroughsReusable itemsTesting early and oftenDefect trackingMetrics (measurement data)
416 COMPUTING APPLICATIONS
1990) and head of the In-Vessel Physics Group during theEngineering Design Phase (1993–1998). He is the AssociateEditor-in-Chief of the IEEE/AIP publication “Computingin Science and Engineering”, and a fellow of the Ameri-can Physical Society and of the American Nuclear Society.His current professional interests include the developmentof software engineering methodologies for scientific com-puting as Team Leader for Analysis of Existing Codes forthe Defense Advanced Research Projects Agency(DARPA) High Productivity Computing Systems Project.
Richard Kendall recently retired as the Chief Informa-tion Officer at LANL. He graduated from Rice Universitywith a PhD in mathematics in 1973. He has over 30 yearsof experience in the development of technical software inthe upstream oil and gas industry and in computer infor-mation and computer security technology. He started hisprofessional career as Senior Research Mathematician atExxon Production Research Company in 1972. In 1982he left to join a start-up petro-technical software venture,J.S. Nolen & Assoc., as Vice-President. This companyspecialized in reservoir simulation codes for the thenemerging vector supercomputer market. This companywas acquired by Western Geophysical where Kendallbecame Chief Operating Officer of the Western AtlasSoftware division. He joined LANL in 1995. He is a con-tributing member of the Society of Petroleum Engineers(SPE) and the Society for Applied Mathematics (SIAM).
References
Brooks, F. P. 1987. No silver bullet: essence and accidents ofsoftware engineering. Computer 20(4):10–19.
Cockburn, A. and Highsmith, J. 2001. Agile software develop-ment, the people factor. Computer 34(11):131–133.
Cox, L. et al. 2002. LANL ASCI Software Engineering Require-ments. LA-UR-02-888. Los Alamos National Laboratory,Los Alamos.
DeMarco, T. 1997. The Deadline. Dorset House, New York.DeMarco, T. and Boehm, B. 2002. The agile methods Fray.
Computer 35(6):90–92.DeMarco, T. and Lister, T. 1999. Peopleware, Productive
Projects and Teams. Dorset House, New York.DeMarco, T. and Lister, T. 2002. Risk Management for Soft-
ware. The Cutter Consortium, Arlington, MA.Esque, T. J. 1999. No Surprises Project Management. ACT
Publishing, Mill Valley, CA.Herbsleb, J. et al. 1997. Software quality and the capability
maturity model. Communications of the ACM 40:30–40.Highsmith, J. A. 2000. Adaptive Software Development. Dorset
House, New York.Highsmith, J. and Cockburn, A. 2001. Agile software develop-
ment: the business of innovation. Computer 34(9):120–127.Humphrey, W. S. 2000. Introduction to the Team Software
Process. Addision-Wesley, Reading, MA.Humphrey, W. S. 2001. Winning with Software: An Executive
Strategy. Software Engineering Institute, Pittsburg, PA.
Jones, T. C. 1998. Estimating Software Costs. McGraw-Hill,New York.
Laughlin, R. 2002. The physical basis of computability. Com-puting in Science and Engineering 4(3):27–30.
Lewis, R. O. 1992. Independent Verification and Validation: ALife Cycle Engineering Process for Quality Software.Wiley, New York.
Lindl, J. 1998. Inertial Confinement Fusion. AIP Press, NewYork.
McBreen, P. 2001. Software Craftmanship. Addison-Wesley,Reading, MA.
McConnell, S. C. 1997. Software Project Survival. MicrosoftPress.
Oldham, J. D. 2002. Scientific computing using POOMA. C/C++ Users Journal 20:6–22.
Paulk, M. 1994. The Capability Maturity Model. Addison-Wes-ley, Reading, MA.
Peters, J. and Pedrycz, W. 2000. Software Engineering: AnEngineering Approach. Wiley, New York.
Phillips, D. 1997. The Software Project Manager's Handbook.IEEE Computer Society, Los Alamitos.
Post, D. and Cook, L. 2000. A Comparison of Software Engi-neering Practices used by the LLNL Nuclear ApplicationsCodes and by the Software Industry. UCRL-MI-141464.Lawrence Livermore National Laboratory, Oakland, CA.
Post, D. and Kendall, R. 2002. Estimation of Software ProjectSchedules for Multi-Physics Simulations. LA-UR-02-7159. Los Alamos National Laboratory, Los Alamos.
Pressman, R. S. 2001. Software Engineering: A Practioner'sApproach. McGraw-Hill, Boston.
Remer, D. 2000. Managing Software Projects. UCLA Techni-cal Management Institute, UCLA Extension Courses, LosAngeles, CA.
Rifkin, S. 2002. Is process improvement irrelevant to producenew era software? Software Quality – ECSQ 2002 7thEuropean Conference, Helsinki, Finland. Springer-Verlag,Berlin.
Roache, P. J. 1998. Verification and Validation in ComputationalScience and Engineering. Hermosa, Albuquerque, NM.
Roache, P. J. 2002. Code verification by the method of manu-factured solutions. Transactions of the ASME 124:4–10.
Salari, K. and Knupp, P. 2000. Code Verification by the Methodof Manufactured Solutions. SAND2000-1444. SandiaNational Laboratories, Albuquerque, NM.
Stewart, J. R. and Edwards, H. C. 2001. Parallel AdaptiveApplication Development using the SIERRA Framework.First MIT Conference in Computational Fluid and SolidMechanics. Elsevier, Boston, MA.
Symons, C. R. 1988. Function point analysis: difficulties andimprovements. IEEE Transactions on Software Engineer-ing 14(1):2–11.
Thomsett, R. 2002. Radical Project Management. Prentice-Hall, Englewood Cliffs, NJ.
Treacy, M. and Wiersema, F. 1995. The Discipline of MarketLeaders: Choose Your Customers, Narrow Your Focus,Dominate Your Market. Perseus Books, Reading, MA.
Verzuh, E. 1999. The Fastforward MBA in Project Manage-ment. Wiley, New York.
Yourdon, E. 1997. Death March. Prentice-Hall, Upper SaddleRiver, NJ.
