Research organisations pay a lot of attention to the quality of their work.However often no such attention is paid to the quality of the software theyproduce within research projects. This is not a healthy situation since researchorganisations are becoming more and more dependent on softwaredevelopment. This article describes the results of a project on software processimprovement (SPI) at a Dutch research organisation. The aim of the project isto increase the technical quality of software produced by researchers and toincrease the degree of control over the development process. This is done bydeveloping guidelines, introducing tools and supporting researchers in the use ofthese products.
1. Introduction
Traditionally research organisations have always paid considerable attention tothe quality of their work and there is a great deal of interest in the use of correctresearch models to streamline the research process. There is also a lot of interestin the quality control of research products which is implemented by referees andeditors of magazines reviewing research papers. Quality control of importantresearch-related activities, such as the use of statistical methods and chemicalanalysis, is standardised by the NKO/STERIN/STERLAB [1] criteria for qualitymanagement systems tailored to the particular needs of research laboratories.
Despite the high degree of quality control in these areas, very little attentionis paid to the quality of software produced by researchers. There are scientificpapers about the background and fundamental principles of research software,but often there is very little or no technical documentation. Because of the lackof such documentation, a lot of scientific information is available in source codeonly. Since there is no standard way of organising the source code, this createsa huge maintenance problem once the original authors have left and sometimeseven before that. These observations are also made by Loeve [2].
Needless to say that the situation described above makes it very difficult, ifnot impossible, to guarantee the quality of research software. When software isan important element of research, the lack of software control makes it equallydifficult to guarantee the quality of research output.
The DLO Winand Staring Centre for Integrated Land, Soil and WaterResearch (SC-DLO) is a Dutch research organisation that produces software topredict the consequences of environmental changes for the use and managementof land and water. The software produced by SC-DLO consists of simulationprograms and Geographical Information Systems (GIS). These programs areusually made for internal use, but nowadays more and more customers want tobe able to calculate different "what if scenarios themselves. Therefore the roleof the software is changing towards that of a real software product.
Both the need for quality assurance of research output and the changing roleof software from research tool to research product were reasons for SC-DLO tostart a project on software process improvement (the SPI project). The aim ofthe SPI project is to improve the technical quality of the software and toincrease the degree of control over the software development process. The SPIproject is a co-operative venture with the Dutch Software Engineering ResearchCentre (SERC).
Section 2 describes the theoretical background on which the structure of theSPI project is based. Section 3 describes the organisation, phases and currentachievements of the SPI project. Section 4 discusses the benefits of the SPIproject and some of the problems associated with it.
2. Process improvement in research & development
The situation within research organisations such as SC-DLO is not verydifferent from that of technical industries in the recent past. Similar problemsare now known as the reasons behind the "software crisis" recently described byW.W. Gibbs [3]. Although this article shows that the software industry is notyet able to tackle all of its quality problems, numerous initiatives have beenundertaken. Most of these focus either on software products or on the softwaredevelopment process. Representatives of the product-oriented approach areBoehm [4] and SERC-QUINT [5]. Examples of the process-oriented approachare the Capability Maturity Model (CMM) explained by Humphrey [6] andPaulk et al. [7], the ISO 9000-3 [8] standard made by the International
Organization for Standardization and TickIT [9]. Trienekens [10] shows thatthere are basically three ways of tackling quality improvement:
• product improvement;• process improvement;• integral improvement.
Trienekens states that only Total Quality Management (TQM) incorporates theneed for people management into quality improvement initiatives and cantherefore be considered representative of the intregal improvement approach.The usefulness of TQM in technical environments is demonstrated by Zultner[11]. According to Trienekens integral improvement is the preferred way ofachieving sustainable quality improvement.
Although it is not their primary focus, both TickIT [9] and Humphrey [6]address the need for people management. They both state that software processimprovement projects need to be managed just like any other project involvingorganisational change. TickIT [9, ch. 4 pp. 7] states some typical fears thatpeople have at the start of a quality improvement project:
• "standards take all the fun out of programming";• "we'll have to spend more time on paperwork instead of getting on with the
real work";
• "we'll be swamped by heavy documents and manuals";• "I'm not having anyone else meddling with my code!"
Similar problems are mentioned by Humphrey [6]. These problems are probablymore important in a research environment than in any other organisationbecause of the people-oriented culture. An organisation with such a culture isalmost impossible to manage (see Wijnen et al. [12]). So, to handle softwaredevelopment in a research organisation we need cultural change. TickIT [9,ch. 4 pp. 7] states that cultural change is preferred over enforced change.
An interesting study about the role of culture in an organisation was made byPerry et al. [13]. Perry shows that in an organisation with an ISO 9001certificate and CMM level 2, only 40% of the activities of developers areplanned. Every day each developer spends about 70 minutes on informalcontacts, most of which have a duration of less than 5 minutes. He concludesthat all developers are busy with a number of small parallel jobs which oftenhave more to do with the culture of the organisation than with the technicalnature of their work. It is therefore important that initiatives regardingorganisational change are related to the culture of the organisation. Thefollowing anecdote illustrates this point.
In 1989 SC-DLO made a first attempt to tackle their internal software crisis.This was done together with a large consultancy firm. This organisation tried toconvince the researchers that they need modern technical tools to tackle theirproblems. They demonstrated several CASE tools over a period of a few weeks.As a result nothing happened. Nobody was able to incorporate these products in
their daily work. Only at the end of 1992, a new attempt was made to tackle thesoftware crisis with the SPI project.
It can be concluded that it is of utmost importance that the activities of theSPI project reflect the culture of the organisation. Then, step by step, culturalchange can be made. If the SPI project takes over-hasty steps to change from apeople-oriented culture to a more result-oriented culture, the commitment ofthe researchers will decrease and the project will fail.
3. The SPI Project
Although the project organisation was established as a result of the first projectphase, it is described in this section. The next sections describe the phases of theSPI project in chronological order.
3.1 SPI Project organisationThe previous section explained the need for organisational change. Therefore itwas considered important that a large number of researchers were involved inthe process improvement activities and a rather "heavy" project organisationwas set up (see Figure 1). The role of each group in the project is brieflydiscussed.
Steering committee The steering committee has the final responsibility for theproject: it is responsible for allocating resources and budgets and settingpriorities. Top management is represented to express the commitment of themanagement team. As Humphrey [6, pp. 19] and Gilb [14, pp. 244] point out,commitment by top management is essential for achieving organisationalchange.
Project group The project group is responsible for the day-to-day activities ofthe SPI project. The group is a stabilising factor between all the pilot projectsand serves as a reporting channel from the pilot projects to the steeringcommittee.
Pilot project The pilot projects are the research projects within the SPIproject. There are pilot projects that assist in developing guidelines and thereare pilot projects using and evaluating guidelines. In a sense, this is where thereal work is done.
Support group It was considered important to support the pilot projects in thedevelopment and use of guidelines. Therefore a support group was establishedconsisting of employees of the computer science department of SC-DLO,together with software engineers of SERC.
Reviewer The progress of the SPI project as a whole is reviewed by anexternal reviewer. The reviewer reports to the steering committee about the
technical achievements and the progress made by the SPI project. Reviews areheld about every 6 months.
Figure 1: SPI project organisation
SPI interest group The SPI interest group acts like a forum where peoplefrom within the whole organisation can get information about SPI activities.The group also gives advice to the project group about which IT topics to coverand is informed regularly about the status of the work done within pilotprojects.
3.2 Phase 1. Assessment (end of 1992)During the first phase of the SPI project an assessment was made of the existingsoftware engineering situation at SC-DLO. This was done by SERC byinterviewing researchers and inspecting source code and documentation. Themain results of the assessment were:• SC-DLO is a research organisation that carries out research into agricultural
areas such as land use and land management. The researchers at the institutehave backgrounds in hydrology, mathematics and agricultural science. Formany years they have been producing software (mainly simulation softwareand Geographical Information Systems) that facilitate their research. Thesoftware is built as part of the research activities. There is a kind of informalincremental process consisting of a mixture of doing research and producingsoftware. The process used to build software can further be characterised by"code-and-fix". There are no guidelines for configuration management,project planning and other process related issues;
• Most simulation programs are written in Fortran-77, with an average size ofabout 10,000 lines of code (a line was considered any physical line, withoutcomments and blank lines);
• Simulation programs are batch-oriented. Geographical Information Systems(GIS) are interactive (using a modern event-driven approach). This is a result
of the platform used to develop the systems (Fortran-77 for the simulationprograms, and ARC/Info for GIS);
• An enquiry held among clients of SC-DLO made clear that the software washard to use, but was stable and produces reliable results.
The result of this phase was a detailed action plan for the next phase. The mainrecommendations were:• Install a project organisation as shown in Figure 1. Make sure a large number
of researchers are involved in the SPI activities;• Start developing guidelines. Start with activities specified by the researchers;• Enhance the level of IT knowledge of the researchers. Do not rely on a
separate software development team since the development of software andother research activities are too closely related. Only specific areas such asuser interface development can be tackled by a separate development team;
• Install a software quality manager responsible for the co-ordination of theSPI project and continuity of quality assurance efforts in the future.
3.3 Phase 2. Pilot study (1993)During the second phase of the SPI project a number of guidelines weredeveloped. To stimulate the use of the guidelines, associated tools werepurchased or developed (see Table 1). A guideline for configurationmanagement (CM) was made because it was felt that CM is fundamental to anykind of process or product improvement. Without CM it is not possible to relatecomments, test documents, walkthrough reports etc. to a specific version of aproduct. Humphrey [6], ISO 9000-3 [8] and TickIT [9] mention CM as a basicsupport activity for achieving quality improvement.
The other guidelines were drawn up because the areas that they cover arementioned by the researchers as being of primary concern. Since thecommitment of researchers is essential, it was decided to follow their requests.Some important areas for improvement, such as walkthroughs and inspections,were deferred until phase 3. This was mainly determined by the availablecapacity.
Each guideline was developed in co-operation with one pilot project. Theresearchers working on the pilot project built the relevant parts of their softwarein accordance with the guideline. In this way it was possible to demonstrate theguidelines to the organisation during a colloquium together with a real-lifeexample. Since this was the first time that guidelines had been developed, therewas a regular meeting with all the researchers involved every 2 weeks. Althoughthis was a huge investment in time it produced three important benefits. First,the researchers were informed about the state of all guidelines. Since they hadmeetings at their own departments as well, this information became known to alarge number of employees. Second, the meeting was an excellent instrumentfor ensuring consistency between the various guidelines. Third, the first priorityof every member of a pilot project is always his/her own project(s) and not the
SPI project. Holding regular meetings generates a kind of social pressure whichencourages researchers to devote sufficient attention to the SPI project.
Another important activity during phase 2 was the development of a qualityassurance framework. This framework defines the way quality assurance shouldbe made part of the normal business of software development at SC-DLO. Thebasic elements of the framework are listed in § 3.5, which describes the fourthphase of the SPI project.
Table 1: Guidelines developed during phase 2
GuidelineHow to write aguidelineFile IOConfigurationmanagementFortran-77
Documentation
Test
Short descriptionLayout and structure ofguidelinesHow to handle file IOVersion, derivation, andchange managementGuidelines for code and in-linedocumentation of programswritten in Fortran-77Layout and structure of off-line documentation
3.4 Phase 3. Implementation (1994-)In the third phase of the SPI project, several pilot projects are currently usingthe guidelines drawn up during the previous phase (see Table 3 for an overviewof the pilot projects). By restricting the use of the guidelines to pilot projectsonly, it is possible to keep track of the usefulness of each guideline. Later, whenthe guidelines are sufficiently stable, they will be made available to everydeveloper. All pilot projects receive active support from the support group inthe use of the guidelines.
Another important activity during this phase was the development andimplementation of an education plan for software engineering. Again bothHumphrey [6] and ISO 9000-3 [8] stress the importance of training. There arebasically two kinds of training which are considered to be important. The first isthe training of people in the use of the guidelines and associated tools. Thisbecomes necessary when the guidelines are made broadly available. At this pointthere is a training course in configuration management, since that is the mostabstract issue encountered during this phase. The second kind of training isintended to increase the general level of software engineering knowledge amongthe researchers. For example there is a course on datamodelling and all membersof the support group follow a software engineering course.
During this phase a number of guidelines for new areas are developed (seeTable 2). One of them is a guideline for project organisation. 150 of the 300employees at SC-DLO followed a course on project management. The SPIproject developed a guideline that addresses software engineering aspects ofproject management such as descriptions of documents for requirementsanalysis and test documents.
At the start of this phase a software quality manager was appointed andmade responsible for the implementation of the quality assurance frameworkdeveloped during phase 2.
Table 2: Guidelines developed during phase 3
GuidelineWalkthrough
Projectorganisation
AML
Short descriptionHow to organise awalkthrough. What to do withthe resultsGuidelines for structure andlayout of project plans, teambuilding etc.Style guides for code and in-line comments for programswritten in Arc/Info MacroLanguage
ToolsForms for recordingremarks
Word -processing macrosand styles. Projectplanning tool
3.5 Phase 4. Consolidation (1995,...)This phase (which will start at July 1995) will concentrate on consolidating theachievements so far. This will be achieved by implementing the framework forquality management mentioned in § 3.3. The most important elements of thisframework are described below.
Project-oriented approach Ad hoc projects will be used to develop newguidelines and maintain existing ones. The software quality manager will beresponsible for these activities.
Support Researchers should be able to get support with the use of the variousguidelines. Therefore the support group (see § 3.1) will be permanent. Thisgroup is also responsible for tool inventories.
Training The education plan described in § 3.4 will be implemented.
Quality Assurance The SPI project focuses strongly on organisational change.As a result a lot of attention is given to gaining the commitment of theresearchers. This is however not enough to guarantee the quality of thesoftware. Some form of auditing is necessary. Therefore a checklist isdeveloped that makes it possible to assign a quality level to a software product.
The checklist is developed during the previous phase of the SPI project and willbe used during this phase. The checklist is based on the guidelines so it can beeasily extended in the future when new ones become available.
Table 3: Pilot projects participating during phase 3
Project characteristicCombination of threesimulation models(Fortran-77)Simulation model(Fortran-77)
Simulation model(Pascal)CIS (Arc/Info)
CIS (Arc/Info, Oracle)
Evaluate how to combineGIS and Simulationmodels
NameSIGWAN
SIMGRO
AMMO
LANDUSENUSWAWOFOSTSWAPMOISHE
SCN
DrentseAaBOPAK
LKN
GIS/MODEL
Quality related activitiesDatamodelling, Fortran codeimprovement, documentation,configuration managementConfiguration management
Arc/Info Macro Language codeimprovementDatamodelling, Arc/Info MacroLanguage code improvementDocumentation, Arc/Info MacroLanguage code improvement,configuration managementDatamodelling, documentation,configuration managementProject plan
4. Discussion
Software process improvement is expensive in money, time and working-capacity (see Table 4). The SPI project started at the end of 1992 and the firstresults became available in 1993. But it was only in 1994 a reasonable numberof projects had benefited from the activities. This result does not differ from theinvestments in a similar project described by Dion [18]. Dion states that duringa 5-year project about 1 million dollars per year was invested in softwareprocess improvement.
One problem with the current SPI project is the quantification of theachievements so far. Indications of the success of the project are 1) the resultsachieved by pilot projects, which are available to every developer in theorganisation. 2) There is no problem getting pilot projects (all pilot projects areinvolved voluntarily). 3) A number of projects use the guidelines even when notselected as a pilot project and 4) Some of the guidelines are spontaneouslycreated by researchers, following the approach of the guidelines drawn upduring phase 2 of the SPI project. There is however no systematic way ofmeasuring the progress made. Therefore a project has been started in December1994 to evaluate the usefulness of the Capability Maturity Model (seeHumphrey [6] and Paulk et al. [7]) in the context of SC-DLO.
The guidelines provide an excellent way of discussing quality. Since theresult of discussions are written down, the issue of quality becomes less abstractthan would be possible without such guidelines. However, guidelines alone arenot enough. Only when active support for the use of guidelines is provided, areresearchers willing to use them.
In a people-oriented organisation such as SC-DLO, it is of the utmostimportance to obtain the commitment of researchers. Commitment is the onlyway of gaining enough momentum to get things done. Therefore we havechosen activities that are identified by the researchers. Although otherorganisations might make other choices, we think that, to some extent, gettingcommitment is more important than the choice of issues to tackle, especially atthe start of a project on software process improvement. Once there iscommitment, it is more feasible to tackle important but less popular issues. Thedrawback of this approach is of course that some areas that are consideredimportant by the project group are deferred by the researchers (walkthroughsand reviews are examples of this).
Another thing we discovered was that researchers are more interested indiscussing product quality than in discussing process quality. Therefore it is agood idea to start a project on process improvement with product improvement.People will become motivated and feel the need for process improvement as anatural next step.
Our experience with the pilot projects shows that long-term improvement isonly possible when continuous effort is put into quality improvement. Whenthere is no active support, attention to quality diminishes.
4. Boehm, B.W. Characteristics of Software Quality, TRW Series ofSoftware Technology, Vol. 1, North Holland, Amsterdam, TheNetherlands, 1978.
5. SERC-QUINT Met Specificeren van software-kwaliteit, een praktischehandleiding, Kluwer, Deventer, The Netherlands, 1992.
6. Humphrey, W.S. Managing the Software Process. Addison-Wesley,Reading Massachusetts, USA, 1989.
7. Paulk, M.C., Curtis, B., Chassis, M.B. & C.V. Weber CapabilityMaturity Model for Software, Version 7.7, Report No. CMU/SEI-93-TR-24, Software Engineering Institute, Pittsburgh, USA, 1993.
8. ISO 9000-3 Quality Management and Quality Assurance Standards —Part 3: Guidelines for the application of ISO 900! to the development,supply and maintenance of software, International Organization forStandardization, Geneve, Switzerland, 1991.
9. TickIT TicklT— Guide to Software Quality Management SystemConstruction and Certification using ISO 900IIEN 29001, BritishComputer Society, 1990.
10. Trienekens, J.J.M. Tijd voor kwaliteit. Werken aan betereinformatiesystemen, Phd Thesis Technical University Eindhoven, TheNetherlands, 1994.
11. Zultner, R.E. TQM for technical teams, Communications of the ACM,1993,36(10), 79-91.
12. Wijnen, G., Renes, W. & Storm, P. Projectmatig werken, Het Spectrum,Utrecht, The Netherlands, 1988.
