Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way...

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in anyway without written permission of Intel Corporation.

Building a Scrum Based Test Strategy

Ray ArellSr. Engineering Manager, Intel Corporation

[email protected]


Course Outline

Key ConceptsPersonas Scrum Lifecycle Strategy Final Thoughts


Part 1: Key ConceptsLearning Objectives:

Establish a base understanding of ScrumUnderstanding of what makes up a general test strategy


Manifesto for Agile Development

Individuals and interactions over processes and toolsWorking software over contract negotiationResponding to change over following the plan

4

We are uncovering better ways of developing software by doing it and

helping others do it. Though this work we have come to value:


Common Agile Methods


More Methods

Automated testingBarely sufficient documentationBottleneck managementCoding standardsCollective code ownership Co-locationContinuous team integration and CMCustomer focus group reviewCustomer onsiteDaily standupDeferred Commitment

RefactoringRetrospectivesRisk managementSelf-taskingSimple, robust designSmall releasesSustainable paceTest-driven developmentTest firstUnit testingUnity statementUse casesUser stories

• Design metaphor• Exploratory spikes• Feature-based

planning• Group design• Information radiators• Inspections• “Intentional” design• Issue tracking• Last Responsible

Moment• Monitor and adjust• Pair programming• Project velocity


Scrum Project Framework

7

Each sprint is a

fixed duration

Team works from a prioritized

product backlog

Short daily team

meetings

Must deliver

working and fully

tested code


Product BacklogOwned by the product ownerFormal list of product requirements written as user stories

Contains the acceptance criteria Estimated by the development team on the effort to completePriority and customer value of the item

Backlog is reprioritized at the start of each iterationDefects are also tracked on the backlog

8


Scrum Roles

Product Owner: Owns the backlog and stakeholder relationshipVoice of the stakeholder to the development teamMakes decisions and priority adjustmentsRemoves obstacles

Scrum Master: Drives the daily processTracks progress and removes obstaclesFacilitates, mediates and protects team

The Team:Programmers, test professionals, and other key peopleSelf managed/breaks down user stories down to tasksReports daily on status in standup meetings (progress, plan, issues)Responsible for demo to stakeholders


User Center Design and Standup Area

10


Example Stand-up

11

Agenda:1. What have you done since yesterday's meeting?2. What are you going to get done today?3. What obstacles do you need removed?

adsfsadfdsafsadfasdfSasdfAdsfa

asdfdasfdsafSdkjfsfsaAsfadf














asdfdasfdsafSdkjfsfsaAsfadf adsfsadfdsafsadfasdf

SasdfAdsfa



Review Release Plans(Developers)

Distribution, review and adjustment of product requirements

(Developers)

Sprint

Develop Features

Test

Internal Review

Adjust

CustomerReview

Review software

Compare backlog with developed software

Editing of backlog requirements

Add new backlog items to teams

Assign backlog items to teams

Plan next review

(Developers)

(Project Stakeholders)

All Project Requirements Complete(Quality, Features, Cost)

Adapted from : Wikipedia Scrum (development) ; Many Authors


Key Points on ScrumSprints are a fixed delivery date

Stories that are not done are pushed to product backlog

Stories and defects live in the backlog togetherSoftware repairs are prioritized with the customer

Scrum does not dictate what processes are used within the 2-4 week sprint

This is left up to the development team

Validation is a part of the development teamValidation and test are not visitors--they are a part of the team


Key Points on Scrum (Continued)Product backlog: is the repository of all the high-level features for a product

Documented user stories

Sprint backlog: are the stories being worked on by the team

They are decomposed into workable tasks

Requirements: are documented in storiesStories are expected to evolve and changeDecomposition of stories into tasks will drive clarificationTasks are manageable pieces of work


User Stories and Acceptance Criteria User stories

Is a software system requirement that is written in everyday languageIt is typically one to two sentencesIt needs to be descriptive and should provide enough information to define the acceptance criteria

Acceptance criteriaList of measurements that will be used to verify “done”Can define both functional and non-functional test requirementsDefined with the customer input

Further Reading: http://en.wikipedia.org/wiki/User_stories


ExamplesUser Story:

Application Start: As a user I expect that when the application is invoked it will display the software license agreement for a default of 5 seconds. Display time can be updated to a max of 100 seconds

Validation Criteria:1. Visually verify that the license agreement is displayed for

the default time period2. Set display time to n seconds and verify that it displayed

for that period of time3. Verify that default time cannot be set less than 0 and no

greater than 100 seconds


What is a Test Strategy?

It is the high-level plan on how testing is going to be done. It defines the type of testing that is

going to be used, when it will be used, resources needed and an articulation of key

constraints and assumptions.


Purpose of Test Strategy

Provides a framework for the entire test process Manages expectations throughout the product development cycleCommunicates strategic choices with the customer and the development team


Primary Objectives

Provide maximum return on investment byFinding the defects that have the highest customer impact and exposure to your company as fast as possibleMaking efficient use of resourcesProviding the most important and accurate information to enable informed decision makingEnabling faster repair


Why it is important to define?

Determine the test costStrategy is translated immediately into time required for testing, and time is converted into cost of testing

Adjust your testing effort to available timeIf testing time is fixed, apply test estimation on test strategy to determine what can be achieved within the given time

Communicate early with the customersTest strategy makes it clear which parts cannot be tested, or cannot be tested fully, and what risks will therefore be incurred


Typical Test Strategy Content

1. Test Levels2. Roles and Responsibilities3. Environment Requirements4. Testing Tools5. Risks and Mitigation6. Test Schedule7. Regression Test Approach8. Test Groups9. Test Priorities10. Test Status Collections and Reporting11. Test Records Maintenance12. Requirements Traceability Matrix13. Test Summary

21

http://en.wikipedia.org/wiki/Test_strategy


Defect Tolerance and Test Reduction

22

Annoyance Loss of Revenue Loss of life


Technical Debt

“Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite.... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.”-- Ward Cunningham

23

The current version of the plugin is 0.2 and uses the following formula to calculate the debt :


Summary of Key Topics Scrum

Development periods have a fixed cadence (2-4 weeks)Teams are self-managedTeams work from a product backlog of user storiesUser stories define the high-level requirements It is readable It is testable

Test StrategyDefines the high-level plan on how testing is going to be doneServes as a communication vehicle with your customer and the development teamIt is focused on finding defects as fast as possibleHelps to identify and remove the risk of shipping a low-quality product


Discussion

25


Part 2: Key ConceptsLearning Objectives:

Understanding of customer personas Create several personas


Different User Representations

Amount of Data

Amount of Detail

User Archetype

User Profile-Single User

MarketData

ActorAnd

Agent

User Role

User Class/Category

Personas


Usability Research

Usability and Customer Research is key to ScrumExpert reviewsOn-line surveys Field research with customersPersona building

Usability Research


What is a Persona?Personas are representations of the group of users of your productIt is an amalgam that documents the behavior—goals, skills, attitudes, and environment—of the end-userIt is based on research and knowledge of the end-usersIt is documented and used in both the development and testing of your product


EdwardApplication Engineer

“There is no ‘One Size Fits All’ with ISVs. Every ISV has a different environment, architecture and

customer needs.”

Ed has been working in this role for 4 years. He was a SW Engineer before that for 10 years. His work focuses on the implementation of

AMT capabilities. Ed’s primary role is assisting ISV engineers in implementing specific features by customizing a solution for their

given environment. Much of his day is spent troubleshooting issues and writing new code to test. Usually, Ed travels to the ISV and

spends time face to face working with their implementation team. Given the economic climate, he has made changes to the way he interacts with his customers. Most of his interaction with ISVs is

done over the phone and via email.

Goals:• Simplify integration for ISV

partners• Solve issues fast• Demonstrate AMT value

Values:• Good customer relationships• Flexible architecture• Good documentation

Obstacles:• Each ISV requires custom solution• Troubleshooting issues remotely• Translating code to meet ISV needs

Design Implications:• Design should demonstrate how a

feature could be integrated--represent believable user experience

• Vanilla design = palatable for all potential customers

• Design should avoid appearing as a competing product


Example Persona: SeanSenior Information Security Analyst

“I am a security nerd.”

Sean is a self proclaimed security nerd. This assessment seems justified considering his 15 years in the IT field—9 of which he has specialized in security. Currently Sean manages the patching pipeline for 110,000 systems and 35,000 servers. His priority is the health of these systems and the data they contain. He must ensure the security of the environment while minimizing interference to the workforce’s productivity. Another part of Sean’s job is to research and evaluate new technology—however, carving out time to do this can be challenging. His team is responsible for deciding what technologies will be implemented into the IT environment. All new technology is thoroughly vetted before releasing into the company.

Goals:• Maintain a secure environment• Stay ahead of the next threat• Minimal interference to workforce

productivity

Values:• Data: securing user/corporate

data; security intelligence; performance metrics

• Speed & efficiency• Control & access

Obstacles:• Keeping confidential user info

protected• Differing platforms (& performance)

across the company• Rogue HDDs with unsecured data • Difficult to manage remote/mobile

systems due to an infrequent network connection

• End users introducing unapproved technology into environment. Design Implications:

• Easy to learn, intuitive interface• Instant gratification• Eliminate error-prone conditions—

stakes are too high (data)• Visualize data – represent client system

status • Reduce/control exposure to confidential

info


Applying the PersonaCustomer satisfaction based testing

Using the goals, motivations, and experience of the customer to set your path in session-based testingHelps to bring a higher priority to defects based on possible bad customer experiences

Scrubbing the acceptance criteria of your user stories

“What would <Persona> care about?”

Focusing the test environmentThe persona lets you know where it is going to be used

32


Why are they useful to product teams?

Provides a detailed and consistent understanding of various audience groups. Addresses how teams supports the needs of a marketFeatures and testing can be prioritized based on how well they address the needs of one or more personas.Provide a human "face" to the customer

33


Personas do not stand aloneAlthough personas have many benefits, they alone will not ensure the success of the product. Business goals and technology constraints also are important to consider.

Business

UserTechnology

Nirvana of SW products


Strategy: Using Customer Personas

Agile is about building high customer value with each release…. but what customer?

Customer personas create an amalgam of your target customersUtilizing this, you have a greater capability of satisfying your end-user marketIn theory, if you satisfy the persona, then you satisfy your market

35


Summary of Key Topics Personas can help you focus your testThey require extensive user researchCustomer satisfaction based testing is a useful addition to your test methodologyRemember that they don’t stand alone


Discussion

37


Part 3: Scrum Lifecycle Strategy Learning Objectives:

Discover how the lifecycle and methods integrate togetherReview example criteria for measuring when a story is complete and when your product is ready to ship


Scrum ConstraintsScrum project framework puts three major constraints on testing

Products are delivered on a fixed cadence and cannot be pushed out All features need to be working and meet the acceptance criteriaNo features are shipped to the customer if it is not tested, repaired, and retested

User stories/features by design are expected to evolveDetails and acceptance criteria in the backlog will evolve overtimeMay be deferred until the maximum amount of information is availableDevelopment of the product itself may fill in the gaps

Customers may shift prioritiesThey are the customer after all!

39

As nerve-racking as it seems, test planning needs to be distributed within each development sprint.


1. Nightly build testing (Smoke Test)2. Weekly regression 3. Story validation

1. Customer acceptance testing

1. Full Testing of new features2. Regression of prior functionality3. Distribution criteria completed

1. Requirements inspection2. Establish story validation criteria3. Customer reviews

High-level Scrum Test Strategy

ProjectBacklogProduct

Backlog

Product Iteration

Develop

Ready For Test

In TestDone

Blocked

Clarify

SprintBacklog

Daily Standup

Defects


Sprinting Over the Entire Project

Slide 41

Project Concept

RequirementsGathering

Adapted from : Agile Project Management, Jim Highsmith

Sprint 1 Theme

Architecture1.1

Feature 3Development


Customer delivery, review, update

requirements

Sprint 2 Theme

Architecture1.2



Sprint n Theme

Architecture1.n

Feature(s) nDevelopment

Sprint 0 Sprint 1 Sprint 2 Sprint n

Customer delivery, review, update

requirements

Customer delivery,

review, done

Hard/Complex/Must Haves Less Complex

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in anyway without written permission of Intel Corporation. 42


Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7 Sprint n

Code GrowthBurn down

Application Scaffolding Application Build-out Cleanup Retrospective and

Customer Survey

Requirement Inspections and Customer Reviews

Regression And Formal Customer Testing

Core Sprint Testing Methods

Customer Engagement and Usability Studies

Story Boarding and Prototypes

Test Strategy


Strategy: Test Scheduling

Wed Thur Fri Mon Tue Wed Thur Fri Mon Tue

Continuous Integration

Release Testing

Atomic / Session-Based Testing

Inspections

Tuesday is a better release

day than Friday

ImportantMake sure the story

is testable!


Do Check

User Story Inspection

ProductProposal and Description

User StoryRev x.x

Develop or Update User Story

Story Testable?

Develop or Update Validation Criteria

Review Criteria with Product

Owner/Customer

Passed Review

Product Design

Document

User StoryRev x.x -1

User StoryRev x.x +1


Example Entry Criteria for “In Development”

User Story is reviewed for testablityIf is not testable, then it is moved to “Blocked”


Example Entry Criteria for “Ready For Test”

Story is implemented per user story description with no blocked issuesCode has been reviewed by the authorIntegrated and works on engineering baselineAuthor has run unit level testing on story and has high confidence of being doneUsage documentation and help files updatedArchitecture document updatedBuild files updated


Exploratory Testing

48

The quality of the testing is dependent on

the tester's skill of inventing test cases and

finding defects. The more the tester knows about the product and different test methods, the better the testing

will be.


Session-based TestingA software test method that combines accountability and exploratory testingDeveloped by Jonathan and James BachTest sessions document:

CharterSessionSession ReportDebriefParsing Results

Planning and debrief sessions

49


Example Entry Criteria for “Done”

Passes peer inspectionPasses integrated build testingCode coverage targets have been achievedMeets the validation criteria established/defined with the customer


Example Criteria: Story CompletedA story is complete when:

Story is implemented per user story descriptionMeets the validation criteria established and defined with the customerPasses the code inspection and peer reviewPasses unit level testingIntegrated and checked into the engineering baselinePasses regression testingUsage documentation and help files updatedEtc…


Test Development and Execution Process Example

Change-Based Test Management (CBTM): Software validation methodology centered on monitoring change to maximize the effectiveness of the validation cycle

Change-Based Test Management is about:Establishing a link between your test cases and regions within your product Tests built to target a region of code Test code coverage analysis Monitoring the changes as your product is being developed Source control delta reportsPriority sorting of tests Critical and changed regions first Unchanged regions last


CBTM

Test Suite Test Suite

Test List Prioritized List

Test 1

Test 2

Test 3

Test n

Test 4 Test n

Test 1

Test 4

Test 2

Test 3

TestCoverage

ProductDelta Report

Test Development and Execution Process

First

LastRemember that the longest test dictates the test process time


Example Distribution CriteriaMinimum Criteria:

Obtain any required external standards compliance certificationComplete smoke test and regression of final build for integrationRemove debugging and testing code from softwareDocument customer-visible defects for release notesInstall/uninstall of product on clean system; ensure instructions are complete and accuratePerform virus scan of all release files, image and mediaVerify the presence/accuracy of copyright noticesVerify that no unintended disclosure of restricted information is presentVerify that all trademarks are correct


Staying “Done”Regression is key to making sure that the prior sprints completed stories stay “done”It is also important that customers don’t see things consistently breakingPick a regression strategy that fits the complexity of the product


Example: A Simple Regression StrategyMaster Done List

Top Customer Features

Defects Repaired

Other

Weekly:• Verify a random pick of 10%Release:• Regress all new features • Passes global product

requirements • Verify a random pick of 30%• Anything you may have a hunch

about

Weekly:• Verify a random pick of 5%Release:• Verify a random pick of 10%

Weekly:• Verify a random pick of 5%Release:• Regress all prior critical defects for

3 sprint cycles

Past Sprints


Prioritize Test Areas

Prioritizing CriteriaHas higher likelihood of having defectsIf broken, will have a higher impact on the customerFrequency of use by customerIf broken, will cause more failures in other requirementsTrust your Spidy-sense


Defect Likelihood ChecklistNew Things

New features, concepts, technology, tools

High ComplexityFunctions with many interfaces (functions that require interfacing with external systems) or high complexity

Frequently changing thingsFunctions that have a lot of updates or bug fixes (frequently changing code)Functions with poor requirements and designAmbiguous specs can lead to incorrect or conflicting implementations

DependenciesIf broken, will cause more failures in other requirements


Defect Likelihood Checklist - 2Rushed work

Some tasks or projects are chronically under-funded and all aspects of work quality sufferFunctions developed under extreme time pressureLate changes: Rushed decisions, rushed or demoralized staff lead to mistakes

Tired ProgrammersLong overtime over several weeks or months yields inefficiencies and errors

Learning CurveFunctions owned by new developer


Features (Not) to be Tested Some of the reasons why a feature may be out of the testing scope

Low priority as a result of risk analysis Tested before and is considered stableWill not be included in the current releaseWill be tested by third partyWill be released but not tested or documented as a functional part of the current version


Summary of Key Topics It is important to establish clear entry criteria to each phase of the scrum development cycle “Done” needs to measured more than onceRegression is key to having continued user satisfaction


Points to RememberAlways review your strategy with your management, customers, subcontractors, and development team members

Test strategies for Scrum need to focus on ensuring high customer satisfaction

Invest in user research and use tools like personas to focus your testing effort and priority

Invest in growing your people. None of this can be done without the talented people who know how to break things


Thank You!

63


References

• Various Authors, Exploratory Testing, Wikipedia • Various Authors, Test Strategy, Wikipedia • Various Authors, Scrum (development), Wikipedia • Various Authors, Session-based testing, Wikipedia • The Scrum Alliance, http://www.Scrumalliance.org/ • Ray Arell, Change-Based Test Management, (ISBN: 0971786127) • James Bach, Heuristic Risk-Based Testing, STQE 11/99• James Bach, Risk and Requirements-Based Testing, Computer, June 1999• Ingrid Ottevanger, A Risk-Based Test Strategy, StarEast 2000• Bret Pettichord, The role of information in Risk Based testing, StarEast 2001• James Bach, Risk-Based Testing Troubleshooter, Paper Draft• Erik Petersen, Smarter Testing with the 80:20 Rule, StarWest 2002• Anne Campbell, Using Risk Analysis in Testing, StarEast 2000• Paul Gerrard, Risk-Based E-Business Testing, System Evolutif• Gregory T Daich, Defining a Software Testing Strategy • Jim Highsmith, Agile Project Management• Ruku Tekchandani, Building a Effective Test Strategy• John Pruitt and Tamara Adlin, The Persona Lifecycle• Pettichord, Kaner, Bach, Lessons Learned in Software Testing, on-line • Jonathan Bach, Session-Based Test Management , http://www.satisfice.com/articles/sbtm.pdf

Date post:	18-Dec-2015
Category:	Documents
Upload:	grace-benson
View:	215 times
Download:	0 times

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way...

Documents