Juha ItkonenSoberIT

HELSINKI UNIVERSITY OF TECHNOLOGY

T-76.612 Software Project Management

Quality Assurance in Software Project Management

2Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Contents

Software quality and setting quality goalsFrom quality goals to QA practicesQuality in project planningTracking software quality information

3Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Software quality

In general, it is wise to strive for good quality and try to avoid making too many mistakes during the development project

Bad quality leads to rework, bigger costs, delays, and problems in operation

Qualityis meeting the specificationsis meeting the expectations of the customer or useris value to someone

differs depending on who you askis difficult to define and hard to measure

General standards for software qualityE.g. ISO 9126, McCall quality model, …Gives a nice taxonomy, terminology, definitions and conceptual frameworkCannot tell what are the most important quality attributes and how to achieve them in a particular context and situation

4Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Setting the target - Quality goals

General quality characteristics are not enoughCustomer’s requirements and needsProduct characteristicsApplication domainDevelopment technologies and environmentProject characteristics

Specific quality goals are neededSpecific goals for this project (not wish project)UnambiguousConnected to a concrete project, product, and problems

we need to define precisely what qualities we require of a system

5Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

From a general to a specific quality goalExample

AdaptabilityAttributes of software that bear on the opportunity for its adaptation to different specified environments without applying other actions or means than those provided for this purpose for the software considered.

AdaptabilityThe client software must work with all advertised compatible browser and e-mail environments, with or without java, without specific configuration.

6Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Quality goals point out important qualities

What are the most important quality characteristics for this project and productQuality goals show where to focusWith clearly stated goals it is easier to select good practices and methodsQuality goals steer every activity of the project into the right directionGoals reflect the different qualities for different parts of the system

7Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Measuring quality and achieving the goals

Some quality characteristics can be measured quantitatively

e.g. defect counts, performanceIf the goal can be quantified, do so

What measures are usedWho, how and whenHow the measures are used

Some can be assessed qualitativelye.g. usability, conformance to a standardDefine how the goal is assessed

Who, how and whenHow the results are used

Some cannot be measured at all (in a practical situation)e.g. maintainability, reusabilityDefine how to achieve these goals, anyway

8Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Quality Practices

Plan which practices are used to achieve and measure each of the quality goals

Select existing practices in useFind good best practices from outsideInvent new practices

The goals give guidance for performing the practicesMany practices strive for better quality

TestingReviews, inspectionsConventions, guidelinesDocumentingDesign and development methods & techniquesProject management practices

9Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Mapping the goals and practices

1

DB changes break existing clients

RisksQuality goalsPractices

3

Conform-ance to standard X

One developer has no experience in J2EE

Change-ability

UsabilityFunctional correctness

Code reviews

2GUI prototyping

3Functional testing

3 = good practice2 = helps1 = might help

10Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Mapping the goals and practices

3

1

DB changes break existing clients

RisksQuality goalsPractices

3

Conform-ance to standard X

One developer has no experience in J2EE

Change-ability

UsabilityFunctional correctness

1Regression testing clients

21Code reviews

2GUI prototyping

3Functional testing

Documents how quality goals are to be achieved and risks handledFocuses activities into right issuesReveals and communicates quality risks

explicitly identified risksrisks of quality goals with weak practices

11Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Planning the projectwith quality goals and practices

Describe quality goals and major risksMake sure that each quality goal is meaningful and described in the context of this projectMake sure that all required quality practices are planned

Effort estimations and schedule

Plan how to track quality practices and measure achieved quality

What measures are usedWho, how and whenHow the measures are used

Quality assurance takes effort and calendar time

12Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Who is responsible for quality?

Make sure that quality is everybody’s responsibilityand it shows in the project plan

It is good to have a project “QA manager” roleResponsibility to actively track quality and help project manager to make QA activities to happenReports project’s progress from the quality point of view in the project meetings

Tracking quality goalsTries to be less eager sacrificing quality to the new fancy featuresDoes not mean personal responsibility to “create the quality”or to do all quality assurance activities

Keep separate testers tightly in the projectProject meetingsReviewsCollaborating with developers

13Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Testing and project planning

Test releasesMake clear what and when is released for testingPlan what is the required quality level of test releases – and how to ensure / measure itBuilding and delivering the test release takes time

Do not forget that fixing takes time and effortYou don’t know how many bugs will be foundYou must plan how many bugs will be foundPlan the testing timePlan the fixing timePlan the re-testing time (verifying the fixes)

More bugs means more fixing and slower testingTest schedules are relative

Development will be lateCode will be buggyPlan how to handle slipping dates and underestimated workload

14Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Making quality visible in project tracking

Keep quality goals visible in project meetingsTrack the planned metricsPublish the metricsReact to deviations

Find out the reasons behind the metricssmall number of found bugs due to ignored testing taskslarge number of found bugs due to changed specifications -> broken test cases

Connect quality metrics in progress trackingWhat does it mean to get a task done?Reviews, unit tests, functional tests, regression tests, demo, …

Do not be over optimistic“lets check it done anyway, we’ll fix it in a minute”

15Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Test reporting in project management

As a project manager you need test reportsNot bug descriptions, not plain numbers of test cases and bugsTest reports describe what has been tested and assess the current level of achieved quality based on that testing

Important content in test reportsWhat was tested, how thoroughly, what was coverageWhat was found, bugs by severity, issues, metricsAssessment of quality of the systemProgress according to the plan

Report information that is useful for project managerIn a concise and intelligible form

16Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Reporting with weak metrics:Testing Dashboard

6.6.2003 Build 1.34.52

Functional Area Activity Coverage Quality Comments ☺ PerfectFile management exploring 2 ☺ no bugs found, simple functionality HmmMain hierarchy code review, test cases 2 Some weird behavior, still testing Aargghhh!Formatting test cases 2 Some critical bugs foundDrag and drop exploring 1 Don't work with other applications 3 Its testedImports test cases, exploring 3 ☺ all planned formats tested 2 Features checkedExports sample test data 0 Can't test: no specs for export formats 1 We tried it (once)Overall GUI ad hoc 3 A lot of small bugs 0 NothingEditing area feature testing by spec 2 ☺ Looks goodClipboard 0 Not deliveredProperty tables 0 Not implementedPreferences exploring 1 Nothing serious yet…Help review 2 Mainly spelling and grammatic issues

If you cannot have direct quality measures, use indirectUse of certain quality assurance practices gives indirectly hints of the likely quality of the final systemQualitative assessment and the reasoning behind is much better than plain numbers

Modified from: Kaner et al. 2002. Lessons Learned in Software testing.

17Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Use quality information

Quality information is used to steer the projectReview resultsTest results, bug countsDevelopment progressMetrics of different qualities (performance, load, …)

Why you need the information?To re-scope an iterationTo focus resources into right areasTo make sure achieving good enough qualityTo find out we are making the right product

WHEN DO YOU NEED THE INFORMATION?

we need to forecast the likely quality of the final system when it is under development

18Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Quality information and time horizonsRelease time horizon

Not connected to any iterations or milestones – hard to track and manageTypical approach to manage

Informal reviewsSystem testingAcceptance testing

i i i i i i iRelease project

Time

19Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Quality information and time horizonsIteration time horizon

Quality assurance goals for each iterationActivities that provide information by the end of (each) iteration

Not necessary connected to individual development tasks

Information available at the end of each iteration, at latestReview resultsUnit test and integration test resultsFunction test and system test resultsQuality metrics (e.g. performance, bug counts, test reports)

i i i i

i

ii i i i i i ii

Release project

Iteration

Time

20Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Quality information and time horizonsHeartbeat time horizon

Quality assurance activities as part of each development taskInformation created early and continuously

Development progress, bug counts, performed tests, …Code review resultsUnit tests and resultsAutomatic builds and smoke testsFunction test and integration test results

i i i i i i

i

i i i i i i i i i i i i

Release project

Iteration

Heartbeat

Time

21Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Quality information and time horizons

i i i i i i

i i i i

i i i i i i i

ii i i i i i ii

i i i i i i i i i i i i

Release project

Iteration

Heartbeat

Time

Heartbeat quality assurance -Information for steering the iteration and tracking progress

Iteration quality assurance -Information for steering the current and forthcoming iterations

Release quality assurance -Independent activities

22Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Gather information on short time horizons

Plan how you get the quality information before it’s too lateFind actively ways of digging quality information out as early and often as possible

Talk to testers and developersThe best way is to get something completed and tested in short iterations

The risk of poor quality is not delayed to the end of the project

Shorter feedback loop -> easier and faster to fix the problems

Pay attention to heartbeat practicesAssure good enough quality in each iteration

Do not push risks like a growing snowball in front of you

23Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Summary

Define concrete and meaningful quality goalsIdentify practices and activities that are applied to achieve the goals and track progressPlan activities and tracking

quality assurance activities in the schedule and effort estimationshow to react to deviations

Remember the different time horizonsWhen is the information needed in order to be useful?Plan activities to provide the information promptly

Identify also the major quality risksand plan how the risks are handled in the project and whenDo not leave big risks to be revealed at the end of the project