PROMOTER/RENOIR Summer School on Software … PROMOTER/RENOIR Summer School on Software Process...

1

PROMOTER/RENOIR Summer School

on Software Process Technology

30 June – 4 July 1997, Grenoble, France

SPI frameworks: TQM, CMM, SPICE, ISO 9001, QIPExperiences and trendsNorwegian SPIQ project

Reidar Conradi,Dept. of Computer and Information Science (IDI),

Norwgian University of Science and Technology (NTNU)

Phone: +47 73.593444, Fax: +47 73.594466, Email: [email protected]

PROMOTER2/RENOIR Summer School on SPT: SPI frameworks, Grenoble, 30.6-4.7.97 2

Contents

1 Motivation and Basic Ideas 7

1.1 Software Market Sizes, Software Crisis . . . . . . . 8

1.2 Where are we in Software Process Improvement(SPI)? . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.3 Six Approaches to SPI . . . . . . . . . . . . . . . . . 11

1.4 Some reflections on SPI . . . . . . . . . . . . . . . . 12

2 Total Quality Management (TQM) 13

3 Capability Maturity Model (CMM) 16

3.1 CMM survey . . . . . . . . . . . . . . . . . . . . . . 16

3.2 Extended CMM levels . . . . . . . . . . . . . . . . . 17

3.3 CMM, level details . . . . . . . . . . . . . . . . . . . 18

3.4 CMM Process Assessments . . . . . . . . . . . . . . 27

3.5 Software Engineering Process Groups, SEPGs . . . 28

3.6 The Personal Software Process, PSP . . . . . . . . 30

3.7 Further work around CMM at SEI . . . . . . . . . . . 32


4 Quality Improvement Paradigm (QIP) 33

4.1 QIP survey . . . . . . . . . . . . . . . . . . . . . . . 33

4.2 Typical goals of process improvement . . . . . . . . 35

4.3 NASA Software Engineering Laboratory, NASA–SEL 37

4.4 Software Experience Factory, consolidated NASA-SEL 38

5 SPICE, ISO-standardization 40

6 European BOOTSTRAP method 41

7 ISO 9001 standard 42

8 European AMI project for metrification 45

9 Comparing SPI frameworks 46

9.1 CMM Comments . . . . . . . . . . . . . . . . . . . . 46

9.2 QIP Comments . . . . . . . . . . . . . . . . . . . . . 47

9.3 Comparing different SPI approaches . . . . . . . . . 48

9.4 SEI/CMM vs. ISO 9001 . . . . . . . . . . . . . . . . 48

9.5 SEI/CSMM vs. SEF . . . . . . . . . . . . . . . . . . 49


9.6 SEF vs. ISO 9001 . . . . . . . . . . . . . . . . . . . 49

10 Some SPI Experiences 50

10.1 SPI Experiences at SEI . . . . . . . . . . . . . . . . 50

10.2 Siemens Experiences from USA . . . . . . . . . . . 55

10.3 NASA–SEL Experiences . . . . . . . . . . . . . . . . 56

10.4 Misc. Norwegian Experiences . . . . . . . . . . . . . 57

10.5 Norwegian OASIS project: Three Process Assess-ments . . . . . . . . . . . . . . . . . . . . . . . . . . 58

10.6 The Norwegian SISU project . . . . . . . . . . . . . 59

10.7 The ESPRIT project PERFECT . . . . . . . . . . . . 60

10.8 The ESSI project ESPITI . . . . . . . . . . . . . . . 61

10.9 SPI studies by SU group, IDI, NTNU . . . . . . . . . 62

11 Conclusion and how to get started with SPI 63

11.1 How to get started with SPI? . . . . . . . . . . . . . 63

11.2 Threats to SPI Programs . . . . . . . . . . . . . . . 64

12 Appendix: The upstarting SPIQ project in Norway 65

12.1 General on SPIQ . . . . . . . . . . . . . . . . . . . . 65


12.2 SPIQ Profile . . . . . . . . . . . . . . . . . . . . . . . 66

12.3 SPIQ Organization . . . . . . . . . . . . . . . . . . . 68

12.4 Challenges for SPIQ . . . . . . . . . . . . . . . . . . 69

12.5 SPIQ Aftermath . . . . . . . . . . . . . . . . . . . . . 71


Survey:

� Motivation: software market, software crisis.

� Software Process Improvement (SPI): six approaches.

� Total Quality Management (TQM).

� Capability Maturity Model (CMM).

� Quality Improvement Paradigm (QIP).

� SPICE, ISO-standardization.

� European BOOTSTRAP method.

� ISO 9001 standard.

� European AMI project for metrification.

� Comparing SPI frameworks.

� QIP and Software Experience Factory (SEF),

� SPI experiences.

� SPI problems, getting started.

� Conclusion.

� Appendix: The upstarting SPIQ project in Norway.


1 Motivation and Basic Ideas

“There are three ways to loose money:women, gambling and engineers.The two former are the more comfortable ones,the latter the most certain” – bankier Rotschild.

Quoted from Eric J. Hobsbawn:“Industry and Empire”, Pelican Books, 1971, p. 41.

“If builders built like programmers program,a woodpecker could destroy civilization”

Gerald Weinberg’s conjecture.

A system is big when it controls the organization,i.e. the organization is no longer in control of the system.

Manny Lehman (talk at NTNU, June 1995).


1.1 Software Market Sizes, Software Crisis

Software: 5–6% of BNP in OECD countries, 70% in non IT-sector.50 bill. NOK/year on software in Norway.

Still low industrial maturity on SPI.Or is it a “research crisis”: lack of relevance in universities?

Ex. US software crisis.

Christine Comaford, president of Corporate Computing, claims inQBS Software News, p. 14–15, 1995:

� In USA (by Standish Group, 1995): Annual market of 200 bill.$ for tailored software.

– 31% of projects stopped before completion or “dead ondelivery”, with annual losses of 81 bill. $ !!

– 53% of projects have serious overruns (189% in costs),amounting to 59 bill. $ annually.

– 16% of projects according to plan.

� 75% of costs by running client-server systems is in labor overa 5-year period (Gartner Group).

� 10% of developer salaries spent on (re)education, butexpertise is individual not organizational (Gartner Group).


Ex. Failed software projects, cf. Scientific American, Sept. 1994:

� SCS forgets regression tests – stock drops.

� System 12 software doubles – ITT cut in half.

� AT&T, 1992: aborts telephone service in Eastern USA for 5hours.

� Denver airport, 1994: 3 bill. $, delayed by 18 months due toluggage handling software.

� Kreditkassen, 1995: loses 200 mill. NOK on new EDP system.

� Oslo city, Sept. 1993: voter counting system fails inparliamentary election, almost new election.

� TRESS-90 system for Norwegian Social Security(Rikstrygdeverket), 1995: delayed by 5 years, > 1200 mill.NOK lost.

� Road Authority in Norway, 1995: 152 MNOK on inadequatedatasystem.

� National Railways (NSB), 1995: 200 MNOK onnon-functioning system for ticket sales.

� Norwegian police, 1992-95: “Phoenix” system at 300 MNOK isnot yet in use.

� Norwegian stock exchange system, 1995: had to changecontractor.

� etc. etc.

Are other engineering disciplines doing any better;cf. Space-shuttle 1981, Eurotunnel 1995, . . .?

But new road to Trondheim airport: on 367 MNOK budget and twoyear schedule.


1.2 Where are we in Software Process Improvement (SPI)?

� Much recent interest in software process, assuming:

“The quality of a software system is governed bythe quality of the process used to develop it.”

– Watts S. Humphrey.

� Typical SPI goals:Same process: conformance, lower cost, higher predicability.New process: also using new software technologies and BPR.Future processes: experience database, learning, SPI culture.

� Many Process-Centered Software Engineering Environments(PSEEs) and Process Modeling Languages (PMLs), little in use.

� Time to assess progress and set course for future efforts:– What is now ready for deployment?– What needs small adjustments before use?– Where is future effort needed?– Where do we have to revise goals and strategies?

� E.g. applied in Norwegian industrial SPI program: SoftwareProcess Improvement for better Quality, SPIQ, in 1996-2001,by 4 research and 15 software producing organizations.


1.3 Six Approaches to SPI

� A1. The quality certifiers: Use ISO-9001 [Inc94], TickIT, CMM[PWCC95], SPICE [Dor93], BOOTSTRAP [HMK+94] etc.But do ISO-9001 or CMM-level n+1 certification imply SPI?

� A2. The formalists: Better PMLs, cf. European research inPROMOTER book [FKN94].

But Software developers feel enstranged by current PMLs,conceptually and syntactically. Need simplicity, modularity,standardization, and good UI.

� A3. The tool advocates: Computer-assistance by a PSEE orworkflow tool [Fer93] [RP+95].Discuss: to assist, guide, enforce, automate or monitor?Need also good UI and tool interoperability.

� A4. The practitioners: project-level planning and metrics withclassic QA; little process modeling or -enactment.Focus on “hot spots” first, cf. NASA-SEL (QIP, SEF) [Bas93].

But project tools cannot handle process evolution!

� A5. The organization reformers: restructured processes in alearning organization (BPR), with an experience database.“Humanics”: Software Engineering Process Groups (SEPGs).

� A6. The ideologists: SPI paradigms and concepts.Manny Lehman [LB85] on evolution: E- and S-type projects.Terminology work: [CFF94] etc.Partial/incomplete, inconsistent, and evolving process models.

“Armour or mobility” (von Clausewitz, mentioned by [DeM96]).


1.4 Some reflections on SPI

� Many related disciplines [TC95]:– conceptual modeling,– enterprise modeling and BPR,– workflow and CSCW,– distributed and federated info systems,– configuration management,– project management, QA, etc.

� How to achieve demonstrable SPI andto install a SPI culture in SW producing organizations?

Most PMLs and PSEEs have been an oversell.Systematically exploit commercial SW/SPI technologies:– project mgmt,– metrics,– inspections,– reuse,– RAD, ...

Process support requires 10-15% of running costs.But “quality is free” (Crosby)?

� Theory–practice dilemma in software engineering [TLPH95].

Do case-studies, student projects, pilots, ..., full-scale:Measurement-based SPI [BSH86] [IS94].Need insight in technology transfer and social sciences!

� Researchers + vendors + practitioners:– Researchers formulate methods/hypotheses (+ consulting?).– Vendors provide a tool platform.– Practitioners try out this in real setting, and assess.

– SPI researchers get back models and experiences,analyze these in own PMLs; make revised hypotheses etc.


2 Total Quality Management (TQM)

History:

� Quality work: roots in agriculture.R.A. Fisher used statistics to assist UK farmers.

� Walter Shewhart, Bell Labs: picked up the work.

� W. Edwards Deming: lifelong dedication to teaching andimprovement of quality methods.

� J.M. Juran: the number-two quality guru.

� Deming and Juran to Japan in 1950s: industrial quality circlesetc.


Total Quality Management, some details:

� 1945–85: mainly quality on the factory floor in assembly lines,using statistical process control (SPC).

� After 1985: apply quality principles to the entire organization:sales, production, maintenance etc. Need metrics.

� W.E. Deming approach:14 major points mainly on assessment, e.g. create constancyof purpose, institute training, drive out fear, break downbarriers etc.Using simple data analysis tools. See below PDCA cycle.

� J.J. Juran approach:Quality effort into three parts: planning, control, andimprovement.How customers, suppliers, and processes interrelate.

� P. Crosby approach:Culture changes in an organization, human-orientation.Improve awareness, institute teams, set goals, giverecognition, and repeat quality improvement cycle.

� Plan, Do, Check, Act (PDCA), integrates above in TQM:PDCA or Shewart/Deming cycle – a continuous learningprocess.

TQM aimed at larger companies, DoD is active.Total approach: organization and technologies.Reputed for much paper work.


The 7 “old” TQM tools (techniques):

� Pareto diagram: to decide where to start (sorted bar chart).

� Check sheets: check lists, surveys in form of e.g. plots.

� Histogram: for frequency analysis.

� Scatter plot: to show dependencies between variables.

� Ishikawa diagram: used in brainstroming to find causes to orto understand a problem.

� Control chart: to decide if a process is under control.

� Flow chart: to visualize processes.

The 7 “new” TQM tools (techniques):

� Affinity diagrams: to organize abstract thinking on a problem.

� Relations diagram: to show connections between(sub)problems – overlaps functionally with above affinitydiagrams.

� Systematic diagrams: planning tool.

� Matrix diagram: to describe relations between factors,causes, goals etc. E.g. QFD.

� Matrix data analyses: quantifying data in above matrixdiagram.

� Process decision program chart: to document alternativeprocesses, or which processes should execute a certain task.

� Arrow diagram: to analyze process flow.


3 Capability Maturity Model (CMM)

3.1 CMM survey

Developed by Humphrey et al. at SEI [Hum88] [PWCC95], toassess and evolve software-producing organizations.CMM V1.0 from 1991 and V1.1 from 1993.

Originally to assess software subcontractors to DoD, cf. maturitymodels and QA in other fields [Dem84].

Ongoing work on pragmatical process improvement method,IDEAL, based on CMM-ratings:SPI in an organization by moving it through the maturity levels.

Big industrial network: Assessed 440 organizations in 114companies, 515 assessments and 2215 projects, 63reassessments (Sept. 1995).

Top-down installment: start from management.Establish Software Engineering Process Groups (SEPGs).


3.2 Extended CMM levels

Initial (1)

Repeatable (2)

Defined (3)

Managed (4)

Optimizing (5)

Foolish (0)

Stupid (−1)

Lunatic (−2)

Ad hoc, chaotic

Basic project and product mgmtQA in use, metrics collected

Project handbook written

Metrics analyzed,total quality

Feedbacksestablished

Diciplined process

Predictable process

Standard,consistent process

Continuouslyimproving process

Figure 1: Extended CMM levels.


3.3 CMM, level details

Level Focus Key Process Areas, KPAs5. Optimizing Continuous process Defect prevention

improvement Technology innovationProcess change mgt.

4. Managed Product and process Process measurement and anal.quality: measure Quality management

3. Defined Engineering process: Organization process focusUse infrastructure Organization process defn.

Peer reviewsTraining programIntergroup coordinationSoftware product engin.Integrated software mgt.

2. Repeatable Project management: Software project planningEstablish Software project trackinginfrastructure Software subcontract mgt.

Software quality assuranceSoftware configuration mgt.Requirements mgt.

1. Initial Heros Arbitrary

Key Process Areas are divided into 5 Common Features, that againare divided into several Key Practices.

Fig. 2 taken from [Cur], p. 44.


In

Out

Level 5

Level 4

Level 3

Level 2

Level 1

Management Visibility Level

Out

Out

Out

In

In

In

In Out

Figure 2: CMM management visibility.


Process Focus by Level

Level 5: Continuous, measurement-driven process.

Level 4: Analyze measurements.

Level 3: Install defined process, using infrastructure.

Level 2: Establish infrastructure.

Level 1: Undefined (fire-fighting).

Cultural Changes by Level

Level 5: Empowered and innovative culture: individual focus.

Level 4: Culture of measured experience: project focus.

Level 3: Culture of common engineering: organizational focus.

Level 2: Culture of commitment: project focus.

Level 1: Imported cultures: individual focus.


Process Stability by Level

Level 5: Processes are steadily improved:control estimates.

Level 4: Processes perform within quantitative boundaries:move estimates.

Level 3: Common processes are used across organization:reduce variance in estimates.

Level 2: Processes are protected within projects:establish estimates.

Level 1: Individuals do their own thing

Process Measurement Focus by Level

Level 5: Improvement and cost/benefit data

Level 4: Process and product quality data (defect details)

Level 3: Process-related data (defects, delays)

Level 2: Planning and tracking data (size, effort, time)

Level 1: Haphazard data (...)


Level 1, Initial: Immature Software Organizations

General:

� The software process is characterized as ad-hoc andoccasionally even chaotic.

� Processes improvised by practitioners.

� Testing and reviews usually curtailed under stress.

� Quality is unpredictable.

� Costs and schedules are usually exceeded.

� Reactionary management is usually fire-fighting.

� Success rides on individual talent and heroic effort.

� Technology benefits are lost in the noise.

Tool-use requirements:

� No guidelines or standards in the use of tools.

� No metrics gathered.

� No support of reuse.

� High quality produced only by skilled craftsmen.

� Effective software production limited to small teams.


Level 2, Repeatable: Can repeat previously mastered tasks

General:

� Basic project management processes are established to trackcosts, schedule, and functionality.

� The necessary process discipline is in place to repeat earliersuccesses on projects with similar applications.


� Project level communality in use of tools.

� Some team-oriented tool-to-tool integration performed.

� Project standards on tool use specified.

� Training in use of project tools given.

� Some limited reuse of design and code.


Level 3, Defined: Process characterized, fairly understood

General:

� The software process for both engineering and managementis documented, standardized, and integrated into a standardsoftware process for the organization.

� All projects use an approved version of the organization’sstandard software process for developing and maintainingsoftware.


� Tool standards are set across the organization.

� Local tool selection is based on organizational standards.

� Repository for organizational data established,the Software Experience Database (SEDB).

� Metrics gathered based on the organizational standards.

� Metrics stored in org. repository – for trends, profiles.

� Organizational tools training standards established.


Level 4, Managed: Process measured and controlled

General:

� Detailed measures of the software process and productquality are collected.

� Both the software process and products are quantitativelyunderstood and controlled.


� Process improvement is driven by quantitative analysis ofmetric data.


Level 5, Optimizing: Focus on process improvement

General:

� Continuous process improvement is enabled by quantitativefeedback from the process and from pilot innovative ideas andtechnologies.


� Innovative tools and technologies evaluated for, and adoptedinto the organization.


3.4 CMM Process Assessments

What is an Assessment?

An appraisal of an organization’s current software process using:

� Trained assessment team of 4–8 software professionals, e.g.from SEI.

� Review of 4 or 5 representative professionals.

� Interviews with project managers and practitioners.

Takes one week, plus a later report. SEI can be contracted tocarry out the analysis.

Assessment objectives:

� Understand the organization’s current practices.

� Identify needs for improvement.

� Build consensus on most important needs.

� Motivate sponsorship and action:make improvement plan.


3.5 Software Engineering Process Groups, SEPGs

DivisionExecutive

ProgramManager

ProgramManager

ProgramManager

Software Manager

Software Manager

Software Manager

ExecutiveLeadershipCommittee

Software EngineeringProcess Group

TeamLeader

TeamLeader

TeamLeader

SoftwareStaff

SoftwareStaff

SoftwareStaff Action Team

Process

Action TeamProcess

Action TeamProcess

ManagementSteeringCommittee

Figure 3: SPI organizational infrastructure.

Fig. 3 taken from [Cur], p. 72.Cf. Japanese quality circles.


SEPG composition:

� 4–6 experienced software engineering professionals.

� Respected within and knowledgeable about organization.

� Good communication skills.

� Roles: manager, process engineer (make new process),change agent (install this).

� For each SEPG: 3–6 associated Process Action Teams,each of 3–7 people.

� 10–15 SEPG’ers per 100 developers; 30–40 per 1000 ones?

� CMM level change: takes 2 years?

SEPG responsibilities:

� Make periodic assessments and status reports.

� Charter and facilitate Process Action Teams.

� Establish process standards (process models).

� Define and maintain process assets (SEDB).

� Serve as focal point for technology insertion.

� Provide project consultation.


3.6 The Personal Software Process, PSP

In “A Discipline of Software Engineering” [Hum95],Humphrey introduces the Personal Software Process, PSP.

Covers 12 of 18 KPAs, and all from CMM levels 4–5.Simple and persuasive: software developers should discover the“gospel” by their own experiences during a PSP course:

� Size estimates “drive” the process model (plan):Use function points, Putnam or Delphi method?All can be trained to be 20% off, done in 5 minutes.Then estimate effort and defect distributions, per phase andtotal.

� Modest Software Experience Database (SEDB): e.g. to predictabove effort and defect distributions. Suggested, commonmetrics for sharing experiences.

� Design and code inspections: very effective, saves a lot ofeffort by earlier capture of 2/3 of defects.

� Suite of 10 exercises in C++.Simple ADT formalism with assertions for design.Data from 1484 students exists.


Should do code review before compilation, as 9% ofsyntactical/semantic defects escape compilation!Also: Programmers with most compilation defects have the mostdefects found during testing!

Gradual improvement and convergence of student capabilitiesduring course. A 20:1 difference in programmer performance isnot true: most can be much better.

Course of 200 hours in 15 weeks; taught in courses at CMU/SEI,as part of M.Eng. course. Also special course for industrialtutors/advisers. 12 large organizations will start using PSP:Hughes, Motorola, ...

Normally > 100 human-injected defects per 1000 LOC.Table 4 shows defect-cost distribution over lifecycle phases1.

Phases: Design Design Coding Code Compile Unit- System Post- Sum—— review review test test release

Error cost 0.2 1 0.2 1 0.1 0.5 7 40 irrel.in 100$Distrib. initially 5 - 5 - 40 15 25 10 100%Costs initially 1 - 1 - 4 7.5 175 400 589Distrib. after PSP 5 20 5 45 10 2 8 5 100%Costs after PSP 1 20 1 45 1 1 56 200 325!

Figure 4: Table of PSP lifecycle data.

I.e. almost half the costs and better quality – a “win-win”!Now being introduced in 12 large American companies.

1Cost data are from Humphrey’s keynote talk at ESEC’95, Sitges, 28. Sept. 1995.


3.7 Further work around CMM at SEI

� SW-CMM: CMM for Software.V2 ready by 1997, cf. above.

� PM-CMM: People Management CMM,cf. Humphrey’s PSP above.

� SE-CMM: Systems Engineering CMM,for hybrid hardware/software systems.

� SEI guide for step-wise, process improvement.

� SEI guide for process modeling, with a reusable ProcessAsset Library.

� SEI teaching material.

� CMM email-correspondence group of 500 members.Annual SPEG workshop with 900 participants.

� Participation in ISO SPICE effort, see next page.


4 Quality Improvement Paradigm (QIP)

4.1 QIP survey

QIP [Bas93] originates from work at NASA-SEL (4.3).QIP specifies a meta-process with a double-loop:

Characterize

Set goals

Choose model

Execute

Analyze

Package

Figure 5: QIP double loop.

Inner loop similar to Shewhart/Deming’s Plan-Do-Check-Act cycle;may take 24-30 months.

Company-wide Experience Database to accumulate results.

QIP uses Goal-Question-Metrics (GQM) model to tailormeasurement-based, process improvement for a givenorganization.


QIP: six improvement steps (S1–S6) or meta-process [Bas93]:

S0. (added) Build executive support, top and middle level.Build improvement infrastructure:acquire technologies, train people, . . .

S1. Establish baseline: Document and analyse (“assess”) last 5–10projects, using CMM etc. E.g. what is current productivity,reliability, effort/defect distributions etc.?

S2. Formulation of quantitative improvement goals.

S3. Planning of revised and metrified development process.– Identify data to collect (by GQM).– Define data collection procedures.– Assemble a metrics toolset, if needed.– Create a metrics (experience) database, a SEDB.– Define feedback mechanisms to adjust and learn.

S4. Execution and follow-up of process, with measurementcollection.I.e. the inner loop.

S5. Analysis of results, quantitatively and qualitatively.

S6. Packaging of results for future reuse and organizationallearning. I.e. keep project model stable?

Use metricate toolkit from Software Productivity Centre, Seattle?

Bottom-up installment: start with real needs.


4.2 Typical goals of process improvement

Goal 1: Improve “software production process”.

Goal 2: Improve time/cost estimates for projects.

Goal 3: Improve follow-up of projects.

Goal 4: Minimize total time, and deviation fromschedule.

Goal 5: Minimize software development costs.

Goal 6: Improve software quality: reliability, . . .

Goal 7: Improve software performance: efficiency.

Goal 8: Improve software productivity.

And be concrete in goals: improve by 40% in 3 years.


Ex. Goal 1: Improve sw development process, related metrics

� Mean time from defect identification to correction.

� Elapsed time for each activity.

� Number of defects for each activity.

� Number of deviations from planned development process.

� Number of hours (effort) for each activity.

� Number of new/changed requirements during development.

Ex. Goal 2: Improve time/cost estimates, related metrics

� Initial time estimate vs. elapsed time of each activity.

� Initial effort estimate vs. real effort of each activity.

� Initial size estimate vs. real software size (new, reused).E.g. use function points, Delphi or Putnam method, ...E.g. utilize experience database.

� Productivity, e.g. functions or KLOC per person hour.

� Amount of overtime hours.

Both for each activity and total project.


4.3 NASA Software Engineering Laboratory, NASA–SEL

� NASA Goddard Space Flight Center outsideWashington, 12,000 people.

� Flight Dynamics Center: 700 people,275 software engineers.

� Ground-support software for astrophysics,not for manned space flights.

� Emphasis on lead-time and cost,secondary on product quality.

� Typical projects: 18-30 months, 50-300 Klines,30–45 person-years, in FORTRAN, C, or Ada.6–10 projects in parallel.

� Joint Software Engineering Lab (SEL) with Univ. of Maryland,prof. Victor Basili and CSC subcontractor since 1976.

� Has logged 120 projects in 1976–94.Unique experience base!

� Has experimented with many different technologies (CASEtools, Cleanroom, indep. V&V, QA, OO, Ada, reuse.. . ), andrecorded the effects.

� Own Project Handbook, in nine volumes.

NASA–SEL got first IEEE–CS award for software processimprovement in 1994.


4.4 Software Experience Factory, consolidated NASA-SEL

Both a project database and an experience database:

Project organization Experience factory

CharacterizeSet goalsChoose process

Execute process

project/environment characteristics

tailorable goals, processes, tools, products etc. from similar projects

Execution plans

data, lessons learned

project analysis, modification

ExperienceBase

Package

Generalize

Tailor

Formalize

(Synthesis)

Analyze

Projectsupport

Experience factoryProject organization

products, data,lessons learned

direct projectfeedback

products, lessons learned, models

project characteristics

models, baselines,tools, consulting


Software Experience Factory (SEF) [BM]:

Each project has local project database,in dialog with central SPI organization (the SEF) havinga Software Experience Database (SEDB).

Clear roles defined locally and centrally.

SEDB contains reusable information for organizational learning:

� Single components: common ADTs, classes etc.

� Composite components: application frameworks, libraries.

� Process models for quality, methods, cost, . . .

� Project models . . .

For electronic forms, try the URLhttp://fdd.gsfc.nasa.gov/selprods.html.


5 SPICE, ISO-standardization

SPICE, Software Process Improvement and CapabilitydEtermination: run by ISO/IEC JTC1/SC7/WG10 since 1992,scheduled to come out in 1997.ISO DIS 12207-1: Software Life Cycle Process, 1994. Someplanned documents:

� Introductory Guide (IG):– Process Improvement Guide (PIG)– Process Capability Determination Guide (PCDG)– Assessor Training and Qualification Guide (ATQG)

� Process Assessment Guide (PAG):– Assessment Instrument – a Questionnaire (AI)– Baseline Practices Guide (BAG)

SPICE incorporates ideas from:

� SEI’s CMM.

� The ESPRIT projects BOOTSTRAP (1989–93) and AMI.

� BNR’s Trillennium initiative.

� UK MOD standards.

� Company standards from AT&T, BT, HP, . . .

Main actors: large telecommunication companies, UK MOD.


6 European BOOTSTRAP method

BOOTSTRAP was an ESPRIT project in 1989–93, with ownsoftware process assessment method [HMK+94].Among the partners were 2I (D, coord.), Univ. Graz (A), Etnoteam(I), Univ. Oulu (F).

Some highlights:

� Five levels as for CMM, but with decimals: e.g. 2.3.

� Process features are: Organization, Methodology, andTechnology.

� Methodology:Life-cycle independent functions (CM, QA),Life-cycle dependent functions (for design, coding etc.),Process-related features (model, measurements etc.).

65 companies have been assessed by 1994.A self-financed, BOOTSTRAP Institute is continuing the work,lead from Oulu.


7 ISO 9001 standard

See [Inc94]:

� ISO 9001, Quality Systems for Design / Development,Production, Installation and Servicing, 1994.General for all organizations.

� ISO 9000-3, Quality Management and Quality AssuranceStandards – Part 3: Guidelines for the Application of ISO 9001to the development, supply and maintenance of software,1991.

� ISO 9004-4, Quality management and quality system elements– Part 4: Guidelines for quality improvement, 1993.

Proposes double loop for improvement (as NASA-SEL):– Inner: project follow-up and adjustment,– Outer: organizational learning.

Also TickIT: UK guide to ISO 9000 standardization,being much stronger that ISO.

By Aug. 1995: Ca. 40 Norwegian software organizations areISO-certified, whereof 13 by TickIT.


ISO 9001 clauses:

4.1 Management responsibility

4.2 Quality system

4.3 Contract review

4.4 Design control

4.5 Document and data control

4.6 Purchasing

4.7 Control of customer supplied product (not in CMM)

4.8 Product identification and traceability

4.9 Process control

4.10 Inspection and testing


ISO 9001 clauses (cont’d):

4.11 Control of inspection, measuring and test equipment

4.12 Inspection and test status

4.13 Control of non-conforming product

4.14 Corrective and preventive action

4.15 Handling, storage, packaging, preservation, and delivery(not in CMM)

4.16 Control of quality records

4.17 Internal quality audits

4.18 Training

4.19 Servicing

4.20 Statistical techniques

Some experiences [LS95]:Most easy to obtain: 4.17, 4.15, 4.18, 4.7, 4.19, 4.2.Most difficult to obtain: 4.14, 4.20, 4.6, 4.4, 4.12, 4.3.Most useful: 4.14, 4.17, 4.2, 4.3, 4.1.Not useful: 4.7, 4.15, 4.11, 4.19, 4.20.


8 European AMI project for metrification

Application of Metrics in Industry (AMI) an ESPRIT project on aquantitative approach to software management.

AMI is a 12 step supported method for successful measurementof software project – an elaborated variant of QIP + GQMapproach (incl. 6 steps).

� Assess:1) Assess your environment;2) Define primary management goals;3) Validate primary goals.

� Analyze:4) Break down management goal into sub-goals;5) Check the consistency of the resulting goals tree;6) Produce a table of questions to identify metrics.

� Metricate:7) Write and validate the measurement plan;8) Collect primitive data;9) Verify the primitive data.

� Improve:10) Distribute, analyze and review the measurement data;11) Validate the metrics;12) Relate the data to goals and improvement actions.


9 Comparing SPI frameworks

9.1 CMM Comments

Top-down installment: first get management commitment.

18 KPAs, with around 150 “sub”-KPAs:how do they relate and how do they contribute to quality?And is level n+1 really better than n?

Symptom-based, using unweighted KPAs, not adjusted to realneeds.Almost binary scale: missing one KPA and “down to” level 1, thestarting point for all – i.e. failure (flunking).

Little experience with level 4 and 5.

No clear meta-process to improve, but work under way on theIDEAL overall process.

CMM V2.0 planned for 1997.


9.2 QIP Comments

Mainly bottom-up: must know where the “shoe hurts”.

Solid measurements from the start, into a SEDB.

Goal–Question–Metrics (GQM) model: define relevant metrics.

Do not vary too many factors in one project (“experiment”) – bemethodologically sound!

Need 24–30 months to get valid results; be sober and do notoversell.

Always initial “Hawthorne” effect due to enthusiasm the first 6months, may then experience marginal improvement over next2–3 years!!

Two-loop, QIP model: compliant with ISO 9000.

Staged SPI (CMM) and gradual SPI (SEF)?:– mgmnt- (symptoms) vs. development-oriented (need-driven),– pre-structured vs. company-oriented.

Both approaches may be argued for.Initiatives to combine CMM and QIP/GQM/NASA–SEL.


9.3 Comparing different SPI approaches

ISO-9001 SEI’s CMM GQM/QIP/SEFScope General SW development SW developmentOrigin/Spread Europe/World US/World US/WorldFocus Quality control Management process Utilize experiencesView Customer Management AllCertification Yes No (Partly) NoAssessment Yes Yes NoComparative Yes Yes NoRigid One-step/Continuous One-step/Continuous EvolutionaryPrescriptive Yes/no Yes/no NoAdaptive Limited Limited YesApproach – Top-down Bottom-upValidation Survey Survey & case studies Experiments & cases

Figure 6: Classifiation of SPI frameworks, from [Sø97] p. 60

9.4 SEI/CMM vs. ISO 9001

� CMM has unclear goal, partially customer satisfaction.ISO 9001 has customer satisfaction as explicit goal;standard implements conformance to requirements.

� Both approaches are hard to adapt, little flexibility.

� Focus on management responsibility and commitment.

� Both lack validation.


9.5 SEI/CSMM vs. SEF

� SEF evolves according to goals. High degree of flexibility.

� CMM does not focus on measurement until level 3.SEF has a measurement focus right from the beginning.

� CMM depends on assessments for validation. SEF usesbaselining and continuous measurement monitoring.

� CMM takes the view that the same process might be used fordifferent products, while SEF recommends processes to betailored.

� SEF has strong internal validity through Software EngineeringLaboratory at NASA.

9.6 SEF vs. ISO 9001

� SEF has support for flexibility in goals, ISO 9001 is fixed wrt.goals.

� SEF is flexible and adaptable while ISO 9001 has limitedsupport for this.

� Measurements based on organizational goals, but only SEFprovides substantial support.

� ISO 9001 can be used for purposes of certification and forcomparing organizations.

� Both focus on activities directly affecting the quality of theproduct.


10 Some SPI Experiences

10.1 SPI Experiences at SEI

First two companies, then accumulated.

Hughes Aircraft Experience

� Software part of Ground Systems Group

� 500 professionals:– 12% have > 20 years experience– 41% have 10–20 years experience

Software process assessment results:

� 1987: Level 2

� 1990: Level 3


Hughes: Return on Investment Data (cont’d)

Costs:

� $45,000: process assessment, in 1987.

� $400,000: 2-year improvement program.

Benefits:

� Cost perf. index (budgeted/actual cost of work):.94 in 1987, .97 in 1990.

� $2,000,000 annual reduction in cost overruns

Return on investment:

� 5X first year versus initial investment

� 4.5X accrued annually against initial investment

Vs. $75,000,000 budget per year.

Source: Humphrey, Snyder & Willis, IEEE Software, July 1991.


Raytheon Experience

350 professionals.

Software process assessment results:

� 1988: Level 1

� 1990: Level 2

Costs and benefits:

� $1,100,000: 1990 cost of improvement program

� $8,200,000: 1990 non-conformance reduction

Return on investment:

� 7.5X return

Vs. $50,000,000 budget per year.


Raytheon: Cost Analysis (cont’d)

Cost types:

� Performance: cost of building it right first time

� Non-conformance: cost of rework

� Appraisal: cost of testing

� Prevention: cost of preventing non-conformance

Change in average % project time by cost type:

Perform Non-conf. Appraise Prevent1988 34% 44% 15% 7%1900 55% 18% 15% 12%

(up) (down) (no change) (up)

Source: Dion, IEEE Software, July 1992.

Raytheon got second IEEE-CS award for software processimprovement in 1995.


Accumulated Experiences at SEI

Benefit area Companies Annual median Annual range

Productivity growth 4 35% increase 9% - 67%

Time to market 2 19% reduction 15% - 23%

Pre-test defect detection 3 22% increase 6% - 25%

Field error reports 5 39% reduction 10% - 94%

SPI $s per programmer 5 $1269

Return on investment 5 same 5:1 in $s 4:1 - 9:1

Figure 7: Five tables of SEI’s Industrial Results.

Software process improvement in 20 companies initiated andstudied by SEI, e.g. Hughes, Raytheon, Schlumberger, US AirForce, HP, Rockwell etc.

Size of software development organizations: 200 persons.

13 of these are reported by [HCR+94].Above extract is taken from [Cur].

See also Section 3 on CMM.


10.2 Siemens Experiences from USA

Siemens reports annual SPI gain in 1987–93 of [PC94]:

� 35% increased productivity,

� 22% reduced errors,

� 19% reduced time to market.


10.3 NASA–SEL Experiences

� Since 1980: improved productivity by 50%,quality (reliability) by over 80%.

� Reuse: most important single technology since 1985.3X better reliability, 2X bigger productivity.

� Predictability: much improved, giving better total control.

� Tasks are bigger and more challenging.

� Process metrics and improvement: part of culture,“our way” of performing the business.

See [BG94] and [BM].

Ex. Factors affecting reliability at NASA–SEL:

% Improvement % of variationFactor when in use in reliability

“Quality Assurance” 29 10Code reading 29 10Documentation 27 8Chief programmer 8 1Top-down development 6 1Structured code 3 1Tool use 3 1Design Schedule -1 0

Source: Card, McGarry & Page, IEEE Trans. on Software Engineering, July 1987 —

45 flight dynamics projects.


10.4 Misc. Norwegian Experiences

� NOVIT (adminstrative, networked systems): will improve costestimation by 2X and reliability by 3X in two years, e.g. usingsmall experience database.

� Tandberg Data Systems (embedded systems): has won an IBMquality prices for its tape streamers. Using better inspectionand test methods, coupled to small experience base.

� Siemens (defence systems): will increase productivity by 30%in three years, while retaining its reliability.

� Sysdeco Innovation (GIS, 4GL): will reduce configuration errorsby 3X in two years, using a modern CM system.

� Ericsson and Alcatel: CMM assessments.

� ...

See also SISU (10.6), PERFECT (10.7), and ESPITI (10.8).


10.5 Norwegian OASIS project: Three Process Assessments

In OASIS ESSI project, with SINTEF-DELAB.Scores in scale 1..5, where 5 is worst.

Score % Does % DoesVariation not not

OASIS initial assessment 5 4 3 2 1 know apply CommentLevel 5. Optimized � � � � � 0 0 No indication ofProcess Change management � � � � � 0 0 any practicesTechnology change management � � � � � 0 0 at this level.Defect prevention � � � � � 0 0

Level 4. Managed � � � � � 0 0 Only few indic-Software quality management � � � � � 0 0 ations of practicesQuantitative process management � � � � � 0 0 at this level.

Level 3. Defined � � � � � 0 0 Several indicationsPeer reviews � � � � � 0 0 of practices atIntergroup coordination � � � � � 0 0 this level, butSoftware product engineering � � � � � 0 0 mostly existing asIntegrated software management � � � � � 0 0 unformalizedTraining program � � � � � 0 0 practices.Organization process definition � � � � � 0 0Organization process focus � � � � � 0 0

Level 2. Repeatable � � � � � 0 5 Many good indic-Software configuration management � � � � � 0 0 ations of practicesSoftware quality assurance � � � � � 0 0 at this level,Software subcontract management � � � � � 0 33 but mostly existingSoftware project tracking & oversight � � � � � 0 0 as unformalizedSoftware project planning � � � � � 0 0 practices.Requirement management � � � � � 0 0

Level 1. Initial � � � � � 0 0 Indications thatsome practices stillare at this levelfor some projects.

Source: Geir Skylstad, presentation at PROFF seminar, 6.3.95, Oslo.


10.6 The Norwegian SISU project

SISU, Støtte for Integrert SystemUtvikling, 1988–96(Norwegian for: “support for integrated system development”).

Partners: Norwegian Electronics and Telecommunication industry(now 13 companies), SINTEF-DELAB and NR.

Supported by NFR via ITF’s bransjeplan.

Technologies: SDL, quality, process, product management (CM,reuse).

Method book on development with SDL (by Rolf Bræk andØystein Haugen), extra method guidelines developed for PM/QA.

Project leader: Geir Melby, work address at NR.


10.7 The ESPRIT project PERFECT

PERFECT, Process Enhancement for Reduction of softwaredeFECTs: ESPRIT project in 1993–96.

Partners: Cap Gemini Innovation (F, coord.), Univ. Grenoble (F),Univ. Kaiserslautern (D), Daimler-Bentz (D), Bosch (D), Q-Labs(S), Siemens/SINTEF-DELAB (N).

Technologies: Cleanroom, QIP, GSM, Adele CM-system.

Real industrial applications.

Siemens focus: QA process support around CM, using PCMS andProcess Weaver.

PIA method book: the PERFECT Improvement Approach.


10.8 The ESSI project ESPITI

European Software Process Improvement Training Initiative,ESPITI (1994-1996), with participation from NR in Oslo, attn/Karlheinz Kautz.

Ca. 30 partners in 17 countries, 70 MNOK totally with 2 MNOK inNorway.

Goals: Increased SPI awareness and knowledge of QA forindustrial software development in Europe.

Means: Raising of awareness, offering courses, establishing SPIgroups and networks for exchange of experience.

Have sent out SP Questionnaire (see below) to 1250 Norwegian,software-related companies in Feb. 1995. 170 answers (13.5%)[KbL96].See also standard books on technology transfer: [Rog] [All77].

16 companies (9.4%) used SPI methods to some degree:9 used TickIT, 4 CMM, 2 BOOTSTRAP, and 1 own method.

Ca. 40 Norwegian software companies are ISO-9001 certified.


10.9 SPI studies by SU group, IDI, NTNU

Research profile: Software engineering, focussing on softwarequality and software process improvement (SPI). Also processmodeling/enactment, versioning, new transaction models,software complexity.Experiments and case studies, by students and in enterprises.

SPI empirical studies by IDI:

� Software reuse, 15 joint industrial studies [Kar95] [PH96].

� Effect of CM, CMEX ESSI project with Sysdeco Innovation,Kongsberg [Roa95] [CW96] [CW97a] [CW97b].

� Process conformance, student experiment on inspections[Sø97]. See also [BGL+96].

� Process modeling and enactment of CM processes, at SiemensTelecom [Høy97].

� Process modeling, smaller examples [Wes97].

� Process evolution at NOVIT, four projects [Ngu97] [NC96][NWC97].

� Software maintenance, student experiment on value ofdocumentation [Try96] [Try97].

� Enterprise modeling, four projects at Statoil [Tot97].

� Experience databases at NOVIT, Telenor FoU, internal(starting).


11 Conclusion and how to get started with SPI

11.1 How to get started with SPI?

Enormous interest – but who to start, and what is the proof of thepudding [Her93]?

� What organizational strategy: win management, technologytransfer using pilot projects and champions, sustainorganizational latency and policy changes, . . .Use experience networks, e.g. SPINs.

� What general SPI approach to use: CMM, SPICE,BOOTSTRAP, QIP, ISO, AMI, . . .– or revised from SPIQ?Link up with external SPI competence: univ. etc.

� What concrete technologies to “inject”: QA/projectmeasurements and follow-ups, estimation techniques,prototyping and incremental development, inspections, testingmethods, reuse and standard architectures, configurationmanagement, distributed and concurrent development,improved equipment and offices [DL87], re-training, revisedcorporate and team management (BPR), etc. etc..

� What commercial SPI tools to use: workflow, CSCW, CM,PSEE, ...?

� How to organise the experience database, and really validatethat injected SPI technologies are paying off?

If the starting point is poor, any SPI will work! Be precise, andavoid measuring only the initial Hawthorne effect.


11.2 Threats to SPI Programs

In general:

� Lack of senior management commitment.

� Middle management resistance.

� Inadequate support and resources.

� Desire for quick fixes.

� Unreasonable expectations,e.g. level 5 within two years,or 10X “better” in five years!

� Termination before institutionalized.

� Return-on-investment (ROI) fever.

SEPG problems:

� Inadequate planning or SPI management.

� Inexperience with improvement methods.

Acknowledgements

Go to the EPOS team, and the SPIQ and PROMOTERcolleagues.


12 Appendix: The upstarting SPIQ project in Norway

12.1 General on SPIQ

A national method project and -center for software quality throughSPI, cf. equivalents in Australia, Belgia, Tyskland, Canada,Danmark and Sweden (planned), plus SEI / SPR / NASA-SEL inUSA, ESI, og SPINs in many countries.

Goals:1. Better software production: measured and clear improvement:

25% cost reduction in five years?2. Enlarged process awareness and SPI culture, training.3. Adapt and further develop SPI technology.

Better contract between industry and R&D.

Organization:User-driven with 15 companies in inner ring,40-50 companies in outer ring (SPIN network),with couplings to other projects and initiatives (EU etc.).

Duration 5.5 years, 0.5 + 1 + 2 + 2:

1. Exploratory phase, half year: initial framework, status, plans.

2. A start-up phase of one year, increase number of partners.

3. Two main phases of two years.

Budget: 100 mill. kr (40% coverage from govm’t).

One project manager, 3 researchers, 5 PhD students.


12.2 SPIQ Profile

Pragmatical and concrete: take existing SPI tecnologies in use,modify others, develop some new.

Work areas:

� T1: Consolidated SPI framework:PDCA cycle, take from TQM, SPICE, QIP, AMI, own R&D etc.Important not to over-sell, fast effect.

Effort on process modeling, experience database(“processware”), cost/benefit/risk, reuse and evolution,“humanics” og BPR.

� T2: Company projects:Both strategic SPI, and 1-2 concrete pilots.

� T3: Technology clusters around pilots:Ex. OO, reuse, client/server, CM, requirements engineering,incremental delivery, inspections, testing, customer support,. . .

� T4. Dissemination:training with internal and external courses, nationalexperience forum – SPIN, use of www.

� T5. Project management.


Technical results:

� Consolidated, downscaled and validated SPI method:in Norwegian SPI method book.

� Processware: repertoire of reusable process models andcost/quality models, general and adapted forcompany/sw-tech.

� Concrete framework for company experience database,with tools/www.

� High-level PML (e.g. EPOS add-on), mappable ontocommercial workflow/PSEE tools.

� Dissemination material on www: courses, reports etc. – inNorwegian SPIN, on workshops and experience fora.


12.3 SPIQ Organization

Three classes of companies in inner ring:1. Beginners.2. Having defined improvement plans, collects data.3. Needing support for experience database, organizationalchanges.

International projects

International sw org.Norwegian sw org.

SEI

Cap

U.Kais.Bull

ESI

U.Manch.

Outer ring

Other org.

ITF

DND

NFK

SPIQ:

Inner ring ofSW producers

Methodpartners

PROFF

Figure 8: SPIQ: Inner and outer ring.

SDL

sanntid

metodeprosjekt

SISU-2

prosjekterESPITI CMEXPERFECT

ESPRIT-prosjekterESSI- og

ESA-prosjekt

Firma-

PROFF-

SPIQ

SPI-metodeNorsk, tilpasset

Windows95

Objekt-Orientering

C/S-arkitektur

org. utv.SPICE

Andre FoU-prosjekter

Figure 9: SPIQ: couplings towards other projects.


12.4 Challenges for SPIQ

General:

� Need long-term committment!Cannot participate in SPIQ on/off with half-year intervals.

� SPI/QA programs may fail: too ambitious, changing prioritiesand people.

� No easy way: start slowly, build up momentum.

� Make part of company culture.

� Demonstrate cost/benefits: “Quality is free” (P. Crosby).

� Training and dissemination aspects.


Special for SPIQ project:

� Large project: many actors, limited budget.Cf. other joint initiatives by industry and R&D/university.

� Originally SME emphasis, now a mixture.

� Willingness to share both good and bad experiences.

� “Easy” SPI (inspections) vs. “hard” SPI (reuse, incr. delivery).

� SPI framework: consolidate, disseminate, try out, learn andrevise.But how to “validate” such frameworks?

� WWW technologies for modeling, experience databases, andcourses.


12.5 SPIQ Aftermath

SPI researchers and practitioners must find each other:Get demonstrable short- and long-term SPI results.

SPIQ program will define a joint work place for such efforts.

Will also improve software engineering (re)education.

Spin-offs to other fields: BPR, enterprise modeling.

Will be coupled to international projects and researchers.


References

[All77] Tom Allan. Managing the Flow of Technology. MIT Press, 1977.

[Bas93] Victor R. Basili. The Experience Factory and its Relationship to Other Improvement Paradigms. InProc. European Software Engineering Conference (ESEC’93), Garmisch, Germany, pages 68–83.Springer Verlag LNCS 717, September 1993.

[BG94] Victor R. Basili and Scott Green. Software Process Evolution at the SEL. IEEE Software, pages58–66, July 1994.

[BGL+96] Victor R. Basili, Scott Green, Oliver Laitenberger, Filippo Lanubile, Forrest Shull, Sivert Sørumgard,and Marvin V. Zelkovitz. The Empirical Investigation of Perspective-Based Reading, 39 p. EmpiricalSoftware Engineering, (forthcoming), 1996.

[BM] Victor R. Basili and Frank McGarry. The Experience Factory: How to Build and Run One” (foils).ICSE’17 tutorial M1, April 24 1995, Seattle, USA. 165 p. totally.

[BSH86] Victor R. Basili, Richard W. Selby, and David H. Hutchens. Experimentation in SoftwareEngineering. IEEE Trans. on Software Engineering, SE-12(7):733–743, July 1986.

[CFF94] Reidar Conradi, Christer Fernstrom, and Alfonso Fuggetta. Concepts for Evolving SoftwareProcesses. In [FKN94], pages 9–32, 1994. Also as EPOS TR 187, NTH, 9 Nov. 1992, 26 p.,Trondheim.

[Cur] Bill Curtis. Software Process Improvement: Methods and Lessons Learned (foils). ICSE’17 tutorialT1, April 25 1995, Seattle, USA. 163 p.

[CW96] R. Conradi and B. Westfechtel. Configuring Versioned Software Products. In Ian Sommerville (Ed.):“Proc. Sixth International Workshop on Software Configuration Management (SCM’6)”, pages88–109, Berlin, Germany, 25–26 March 1996. Springer Verlag LNCS 1167.

[CW97a] Reidar Conradi and Bernhard Westfechtel. Towards a Uniform Version Model for SoftwareConfiguration Management. In Reidar Conradi (Ed.): “Proc. Sixth International Workshop onSoftware Configuration Management (SCM’7)”, pages 1–17, Boston, USA, 18–19 May 1997.Springer Verlag LNCS 1235.

[CW97b] Reidar Conradi and Bernhard Westfechtel. Version Models for Software Configuration Management,59 p. ACM Computing Surveys, 1997. (Accepted April 1997).

[Dem84] W. Edwards Deming. Out of Crisis. MIT Center for Advanced Engineering Study, Cambridge, MA,USA, 1984.

[DeM96] Tom DeMarco. The Role of Software Development Methodologies: Past, Present and Future. InDieter D. Rombach, editor, ICSE’18 keynote address, 27 March 1996, Berlin. p. 2–5, 1996.

[DL87] Tom DeMarco and Timothy Lister. Peopleware: Productive Projects and Teams. Dorseth House,1987.

[Dor93] Alex Dorling. SPICE: Software Process Improvement and Capability dEtermination. SoftwareQuality Journal, 2:209–224, 1993.

[Fer93] Christer Fernstrom. Process WEAVER: Adding Process Support to UNIX. In [Ost93], pages 12–26,1993.

[FKN94] Anthony Finkelstein, Jeff Kramer, and Bashar A. Nuseibeh, editors. Software Process Modelling andTechnology. Advanced Software Development Series, Research Studies Press/John Wiley & Sons,1994. ISBN 0-86380-169-2, 362 p.


[HCR+94] James Herbsleb, Anita Carleton, James Rozum, Jane Siegel, and David Zubrow. Benefits ofCMM-based Software Process Improvement: Initial Results. Technical report, Software EngineringInstitute, Pittsburgh, USA, August 1994. CMU/SEI-94-TR-13, 53 p.

[Her93] Art Hersch. Where is the Return on Process Improvement? IEEE Software (Special issue on the moveto Mature Processes), pages 12–68, July 1993. (The author is president at SPC).

[HMK+94] Volkmar Haase, Richard Messnarz, Gunther Koch, Hans J. Kugler, and Paul Decrinis. BOOTSTRAP:Fine-Tuning Process Assessment. IEEE Software, pages 25–35, July 1994.

[Høy97] Geir Magne Høydalsvik. Experiences in Software Process Modelling and Enactment. PhD thesis, IDI,NTNU, March 1997. 237 p. (IDI-rapport 6/97, dr.ing. thesis 1997:32 at NTNU).

[Hum88] Watts S. Humphrey. Characterizing the Software Process: A Maturity Framework. IEEE Software,pages 73–79, March 1988.

[Hum95] Watts S. Humphrey. A Discipline of Software Engineering (on the Personal Software Process). SEISeries in Software Engineering. 789 p. Addison–Wesley, 1995.

[Inc94] Darrel Ince. ISO 9001 and Software Quality Assurance. International Software Quality AssuranceSeries, Quality Forum, McGraw Hill, London, 139 p, 1994.

[IS94] IEEE-Software. Special Issue on Measurement-based Process Improvement. IEEE Software, July1994.

[Kar95] Even-Andre Karlsson, editor. Software Reuse: A Holistic Approach (The REBOOT MethodologyHandbook). Wiley Series in Software Based Systems. John Wiley. 510 p., 1995. ISBN0-471-95819-0, REBOOT report no. 8218.

[KbL96] Karlheinz Kautz and Even Aby Larsen. Diffusion and Use of Quality Assurance and Software processimprovement Approaches in Norway: A survey-based Study (from ESPITI project). Technical report,Norwegian Computing Centre, Oslo, Norway, April 1996. Report # 906, 25 p.

[LB85] M. M. Lehman and L. A. Belady. Program Evolution — Processes of Software Change. AcademicPress, 538 p., 1985.

[LS95] Cecilie B. Løken and Torbjørn Skramstad. ISO 9000 Certification – Experiences from Europe, 11 p.In First World Congress for Software Quality, San Francisco, June 1995.

[NC96] Minh N. Nguyen and Reidar Conradi. Towards a Rigorous Approach for Managing ProcessEvolution. In Carlo Montangero (Ed.): “Proc. 4th European Workshop on Software ProcessTechnology (EWSPT’96)”, pages 18–35, Nancy, France, 9–11 Oct. 1996. Springer Verlag LNCS 1149.

[Ngu97] Minh Ngoc Nguyen. Framework and Approach for Managing Software Process Evolution in EPOS.PhD thesis, IDI, NTNU, March 1997. 287 p. (second draft of PhD thesis, finished in May 1997).

[NWC97] Minh Ngoc Nguyen, Alf Inge Wang, and Reidar Conradi. Total Software Process Model Evolution inEPOS. In ACM/IEEE International Conference on SW Engineering (ICSE’97), May 1997. (Acceptedfor ICSE’97, May 1997, Boston, USA, 9 p.).

[Ost93] Leon Osterweil, editor. Proc. 2nd Int’l Conference on Software Process (ICSP’2), Berlin. 170 p.IEEE-CS Press, March 1993.

[PC94] Daniel J. Paulish and Anita D. Carleton. Case Studies of Software-Process-ImprovementMeasurement. IEEE Computer, pages 50–56, September 1994.

[PH96] M. Paci and S. Hallsteinsen. Experience Book. Technical report, ESPRIT Project 9809, SoftwareEvolution and Reuse (SER): Bull SA, ATC, EP-Frameworks, Q-Labs, Sema-Group sae, Sintef, TTB,and TXT Ingegneria, TXT, Milano and SINTEF, Trondheim, January 1996. Case studies from 15European Software Reuse Projects, report no. SER-810.7.


[PWCC95] Marc C. Paulk, Charles V. Weber, Bill Curtis, and Mary B. Chrissis. The Capability Maturity Modelfor Software: Guidelines for Improving the Software Process. SEI Series in Software Engineering.640 p. Addison–Wesley, 1995.

[Roa95] Helge Roald. CMEX: Configuration Management EXperiment (ESSI PIE project descreiption), 18 p.Technical report, Sysdeco Innovation, Oslo, October 1995.

[Rog] Everett M. Rogers. Diffusion of Innovation. ??, ??

[RP+95] David Redmond-Pyle et al. Process Engineer: Reference Manual. LBMS, Stockport, UK, 1995.

[Sø97] Lars Sivert Sørumgard. Verification of Process Conformance in Empirical Studies of SoftwareDevelopment. PhD thesis, IDI, NTNU, February 1997. 252 p. (IDI-rapport 3/97, dr.ing. thesis1997:14 at NTNU).

[TC95] Terje Totland and Reidar Conradi. A Survey and Classification of Some Research Areas Relevant toSoftware Process Modeling. In [Wil95], pages 65–69, 1995.

[TLPH95] Walter F. Tichy, Paul Lukowicz, Lutz Prechelt, and Ernst A. Heinz. Experimental Evaluation inComputer Science: A Quantititative Study. Journal of Systems and Software, 28(1):9–18, January1995.

[Tot97] Terje Totland. Enterprise Modeling as a Means to Support Human Sense-making and Communicationin Organizations. PhD thesis, IDI, NTNU, March 1997. ca. 250 p. (forthcoming PhD thesis June1997).

[Try96] Eirik Tryggeseth. Report from an Experiment of Documentation on Maintenance, August 1996. 4 p.(accepted for workshop after ICSM’96, Monterey, Nov. 1996).

[Try97] Eirik Tryggeseth. Support for Understanding in Software Maintenance. PhD thesis, IDI, NTNU,February 1997. 347 p. (IDI-rapport 4/97, dr.ing. thesis 1997:13 at NTNU).

[Wes97] Per H. Westby. EPOS Software Process Modelling. PhD thesis, IDI, NTNU, January 1997. 319 p.(forthcoming PhD thesis).

[Wil95] Wilhelm Schafer, editor. Proc. Fourth European Workshop on Software Process Technology(EWSPT’95), Noordwijkerhout, The Netherlands. 261 p. Springer Verlag LNCS 913, April 1995.

epos/promoter/summer97-spi.tex

Date post:	13-Mar-2018
Category:	Documents
Upload:	vannhi
View:	218 times
Download:	2 times

PROMOTER/RENOIR Summer School on Software … PROMOTER/RENOIR Summer School on Software Process...

Documents