Presentation

April 10, 2023 IDS SEPG

PROCESS … The GLUE that holds

Software Projects Together

A Focus on Software Testing

Dennis A. MorrisonSenior Principal EngineerDD(X) System [email protected]


Presentation Outline

• Software Process Overview

• Software Testing Overview

• Source Code Control

• Trouble Reporting and Bug tracking

• Summary


Software Phases(Waterfall Model)

Requirements

Design

Conceive

Code

Test

Build

Deploy

SupportDeliver


Software Phases(Process Glue)

Requirements

Design

Code

Test

Deploy

Support

Requirements Review – Are the Requirements testableArtifacts – Review documentation; Requirements placed under documentation control

Design Review – Does the design support requirements? Artifacts – Review documentation

Code Review – Does the code support requirements and implement the design??Artifacts – Review documentation; Code under source code control

Test Review – Test plans Test Results – Did the tests pass?Artifacts – Review documentation; Test plans; System Trouble Reports (STR)

Test Review – Customer Acceptance Test Test Results – Did the tests pass?Artifacts – Review documentation; STRs

Customer Survey – Is the system meeting customer expectationsArtifacts - Software Updates; STRs


Software Phases and Loops

Requirements

Design

Code

Test

Deploy

Support

Phases where most rework occurs

Design Errors

New feature requests or feature refinement

Software Bugs

Software Bugs

New Feature Request

Requirement Clarification or changes


Software TestingAn Art or a Science ?

• Testing is a phase of software development that can become, or seem, endless– Classically, the most expensive part of the project

– Schedule overruns are often attributed to the testing cycle

– Testers are usually squeezed into smaller and smaller test cycles

• How much testing is enough?• How do you know when you are finished testing?• How good is the product when you are finished?


Testing Has Many Stages

Test

Unit Test White Box Testing

Integration TestGrey Box Testing

System Test Black Box Testing

Acceptance TestSubset of Black Box Testing


• Phase = Unit Test

• Testing is focused on internal operations of the software component– Inputs/outputs

– Logical operations

– Limit testing

– Limited functional testing with stubs

• Testing methodologies– Formal testing with test plan and review

– Ad-hoc random

– Typically, tests are manually executed

Unit Test = White Box Testing


• Phase = Integration Test• Testing is focused on functional aspects of an integrated

set of software components– Inputs/outputs– Logical operations from outside of the software components– Limit testing (induced error paths from external stimuli)– Stubs are replaced by real software as they become available

• Testing methodologies– Formal testing with test plan and review– Semi-automated and manually executed– Automation yields regression test capability

Integration Test = Grey Box Testing


• Phase = System Test• Testing is focused on system level requirement verification

– User Interface (Human to Computer Interface)– Requirement verification– Limit and normal operational testing– Stress and longevity testing

• Tests are executed on target hardware and software• Testing methodologies

– Formal testing with test plan and review– Typically, tests are semi automated and manually executed– Automation yields regression test capability

System Test = Black Box Testing


• Phase = Acceptance Test• Testing is focused on externally visible features

– User Interface (Human to Computer Interface)– Operational focus– Interfaces with external systems in customer domain

• Other collaborative systems• Customer network, etc.

• Tests are executed on target hardware and software usually at the customer’s site

• Testing methodologies– Formal testing with test plan and review– Typically, tests are manually executed

Acceptance Test = Subset of Black Box Testing


Some Software Testing Models• Developer level testing

– Applicable to small teams

• Community testing– Beta versions of a product are released to a volunteer

community for functionality and user level testing• Used by gaming community prior to product releases

• Microsoft uses this approach as one of their test phases

• Collaborative testing– Developers and customers test together

• Project level testing– Used by most large projects


When Should Software Testing Begin?

• The software test cycle should begin during the project requirements phase

• Requirements review should result in the first level of test specifications

• Similar to the “Agile” software development process, test cases should be written and reviewed before the software is developed


When is Testing Finished?

• When the schedule says you’re finished?

• When you run out of money?

• When the boss says to release it?

• When the tester’s say you’re done?

• When the customer says you’re done?

• When the law suits are filed?


Requirements Verification• Raytheon uses requirement verification and

statistical mechanisms to determine when testing is complete

• Requirements are decomposed and allocated to software components

• Requirements are “sold off” with during formal testing events with stakeholders (customers) present


Something you have

Token Reader

Something you know

Pin Reader

Something you are

Biometric Reader

Requirement Decomposition

Req 1

Req 1.3Req 1.2Req 1.1

Req 1.3.2Req 1.3.1Req 1.2.1Req 1.1.2Req 1.1.1

Operational RequirementParent requirement

decomposes into three child requirements The system shall

provide three factor user authentication


Requirements Role in Testing

• Software is tested against requirements– Before a parent requirement can be “sold off”,

all children requirements must be fulfilled– Before a child requirement can be “sold off”,

all of its siblings must themselves be “sold off”– The failure of any requirements will prevent the

parent from being fulfilled


Something you have

Token Reader

Something you know

Pin Reader

Something you are

Biometric Reader

Requirement Decomposition

Req 1

Req 1.3Req 1.2Req 1.1

Req 1.3.2Req 1.3.1Req 1.2.1Req 1.1.2Req 1.1.1

Operational RequirementParent requirement

decomposes into three child requirements The system shall

provide three factor user authentication

Req 1Req 1


Requirements Traceability• Software test cases are built from system use cases

– Scenarios (rainy day and sunny day) are used to test various threads through the software

– Every scenario is traced to parent requirements• A scenario may have more than one parent requirement

– Test cases must demonstrate interface level testing• All system and software level interfaces are documented and tracked• These are known as IRLLs, IRLEs, and IRLMs

– IRLL = Interface Requirement Logical Level– IRLE = Interface Requirements Electrical Level– IRLM = Interface Requirements Mechanical Level

• Test cases and requirements are stored in DOORS


Software Source Code Control• The ability to save and retrieve versions of

software is a key component to success in medium to large software projects– Software can be “versioned” for new features

or software load lines– Changes made to software modules due to bug

fixes can be isolated– Enables developers to work on next release

while maintaining current versions– Software builds can include or exclude changes


Using Source Control SystemsSource Code

Version 1

Source Code Control System

Source CodeVersion 1

Put (ModA V1)get(ModA V1)


put(ModA V2)

Mod A-V1.1Mod X-V7Mod Z-V2

SW BuildSpecification

SoftwareBuild

Source CodeVersion 1.1


get(M

odA V

1)

put(M

odA

V1.

1)

New Feature

Bug Fix


Trouble Reporting and Bug Tracking• Software bugs are documented using formal mechanisms

– CMTS – Change Management Tracking System (MRs)– WSTR – Web System Trouble Report (STRs)

• Bugs are considered addressed when software has been modified, tested, and submitted to the load-line

• Bugs are considered fixed when they have been verified• Bug fixes are pulled into the software load via the source

code control system• Bug reports and time based metrics guide management in

software release decisions


Software Defect Trend Example

20

60

90

110117

132

144

152 155 157160 162

28

14

24

5460

6670

85

95

115

145

18

52

76

86

63

7278

82

70

62

45

17

0

20

40

60

80

100

120

140

160

180

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Cumulative Reported Cumulative Closed Delta

Steep Increase in bugs found

Bug discovery rate slowing

Developer push to close bugs

Developers on vacation?

End Game push. Nearing delivery

Closing the final gap


Test Methods

• Manual– Usually applied during unit test

• Semi Automated– Applied during integration test

• Fully Automated– System test


Test Automation• Test automation can provide a project with

huge dividends• Test scripts

– Drive test automation engines– Can be used in unit testing– Can be reused by multiple test organizations

• Regression tests can be constructed from reusable test scripts


Metrics Drive Behavior• An organization or individual’s behavior is

directly related to the metrics they are measured by– Defects found and fixed– On time delivery– Productivity

• SLOC counts• Defect rate at delivery

– Complexity indices– Customer satisfaction


Best Practices?• No real silver bullets• Process is the glue that holds a project

together• The right solution depends on the project

but is often a combination of methods/processes and tools

• Just remember that customer satisfaction trumps all!!


BACKUP SLIDES


In the world of software development, process is the “glue” that holds the project and team together. Too little, the pieces fall apart; too much and it can become a sticky quagmire resulting wasted effort, time, and money. The challenge facing many projects is finding just the right balance point on the process continuum. As a project progresses through its life cycle, the balance point will shift and the project will need to continuously review and adjust to maintain optimal performance. A typical software project life cycle is comprised of six distinct phases: Requirements, Design, Code, Test, Deployment, and Support. Most projects place a significant amount of emphasis on the front-end phases of the life cycle e.g. requirements, and design. However, industry data clearly demonstrates that the largest percentage of a projects budget is spent during the final phases, namely test, deployment, and support. These phases encapsulate much of the “rework” experienced by the project. This talk will describe the process philosophies and tools used by Raytheon to tackle large-scale multisided software development projects.

ABSTRACT


Tools

• The right tool can result in huge pay backs

• Source Code Control Tools

• Trouble Report tracking

• Stress testing

• Performance testing


Testing Methodologies• Ad-hoc

– Non structured testing with or without test artifacts

– No test plans

– Non repeatable tests

• Formal testing– Test plans

– Test Artifacts

– Defect tracking


Software Phases(Spiral Model)

Spiral Development:

Experience, Principles, and Refinements Spiral Development Workshop

February 9, 2000

Barry Boehm, edited by Wilfred J. Hansen


The NATO Sea Sparrow Surface Missile System (NSSMS) is a quick reaction,

point defense system designed to protect the ship it is on. It is very,

very effective against enemy aircraft, low flying sea skimming missile and

even bomb laden inflatable boats. It was first developed by Raytheon in

the '70's and installed on over 150 ships world-wide. In the mid '90's the

Navy contracted Raytheon to re-architect the system so that it would be

capable of firing the Navy's newest missile the Evolved Sea Sparrow Missile

(ESSM). The Navy's brand new nuclear powered aircraft carrier , the USS

Ronald Reagan, has 6 Seasparrow radars & 3 Seasparrow missile

launchers. The movie shows the Navy doing Qualification Testing on these

systems.


Software Phases(Process Glue)

Requirements

Design

Code

Test

Deploy

Support

Requirements Review – Do the requirements support the customer’s intent? Are they testable? Are they complete (sunny day/ rainy day)Artifacts – Review documentation; Requirements placed under documentation control

Design Review – Does the design support requirements? Artifacts – Review documentation

Code Review – Does the code support requirements and implement the design??Artifacts – Review documentation; Code under source code control

Test Review – Test plans – Do the tests adequately exercise the requirements?Test Results – Did the tests pass?Artifacts – Review documentation; Test plans; System Trouble Reports (STR) or Modification Requests (MR)

Test Review – Customer Acceptance Test Test Results – Did the tests pass?Artifacts – Review documentation; System Trouble Reports (STR) or Modification Requests (MR)

Customer Survey – Is the system meeting customer expectationsArtifacts - Software Updates; STRs

Date post:	09-Dec-2014
Category:	Documents
Upload:	softwarecentral
View:	485 times
Download:	2 times

Presentation

Documents