Exploratory Testing Is Now in Session

MM PM Tutorial

9/30/2013 1:00:00 PM

"Exploratory Testing Is Now in

Session"

Presented by:

Jon Bach

eBay

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Jon Bach

eBay, Inc.

With more than eighteen years of experience in software testing, Jon Bach has held technical

and managerial positions in companies including Hewlett-Packard and Microsoft. In his current

role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests

(activity flows) in eBay’s core sites to discover important bugs that threaten its core business.

He is most notable for creating, with his brother James, Session-Based Test Management, a

method to manage and report exploratory testing.

1

eBay Inc. confidential

Exploratory Testing:

Now in Session

Jon Bach

QE Director, eBay

[email protected]

STAR West 2013


Testing is about DISCOVERY…

mailto:[email protected]

2


…finding emerging context


enhanced by conference…

3


...affected by inference


…framed by reference

4


Exploratory testing

is a combination of

Imagination

and

Discovery

Key Idea


CIDTESTDSFDPOTCRUSSPICSTMPLFDSFSCURA

Product Elements

Structure

Function

Data

Platform

Operations

Time

General Test Techniques

Function testing

Domain testing

Stress testing

Flow testing

Scenario testing

Claims testing

User testing

Risk testing

Automatic testing

Project Environment

Customers

Information

Developer relations

Team

Equipment & tools

Schedule

Test Items

Deliverables

Quality Criteria

Capability

Reliability

Usability

Security

Scalability

Performance

Installability

Compatibility

How to Discover with Your Imagination

} } } } }

Development Criteria

Supportability

Testability

Maintainability

Portability

Localizability

5


“The” ET Definition

A style of software testing…

that emphasizes the personal freedom…

and responsibility of the individual tester…

to continually optimize the quality of his/her work…

by treating test-related learning…

test design…

test execution…

and test result interpretation…

as mutually supportive activities…

that run in parallel…

throughout the project.

-- Cem Kaner, 2006


The agile nature of exploration and the ability

of testers to rapidly apply their skills and

experience make exploratory testing a widely

used test approach—especially when time is

short.

But exploratory testing is often dismissed by

project managers who assume that exploratory

testing is not reproducible, measurable, or

accountable.

6


Why this tutorial?

1) Exploratory testers want respect: When testers explore

during testing, they find great bugs. However, since they

often don’t know how to describe their thinking, it’s

considered to be dismissed as “playing around”.

2) The documentation dilemma: Project managers may insist

that all testing be documented, but Agile does not emphasize

that. So how to balance time spent documenting with time

spent testing?

3) Your work might be scrutinized: You may have to give a

report someday about something you did that was

exploratory – like attending this conference.


My missions for you

1) Discuss — and practice — a way to manage

and measure exploratory testing.

2) See some free tools that support session-

based exploration.

3) Learn why exploratory testing can be

thoughtful, structured, and measurable.

4) Have some fun.

7


A structure for exploration

Mission

Tactics

Story


Exploratory exercise

Test this…

../Ideas & Intellect/Exercises/tri_new/tri-bach2.exe

8


Scrutiny

What’s being asked What they may be thinking

What was your mission? Remind me what I told you to do.

Why did you do that?

How did it go? Were you careful or reckless?

What should I be worried about?

How far did you get? Are we closer to shipping?

Can you help me know our status?

Need anything? Can I speed this along?

Do you need more of my time?

When will you be done? Will I get my bonus?

I have a new task for you…


How could you accurately report your testing?

9


Session-Based Exploration

1) Time Box

2) Reviewable Result

3) Debriefing

The “Session”


Time Box: focused effort, fixed duration

– Brief enough:

• for accurate reporting

• to allow flexible scheduling

• to allow course correction

– Long enough:

• to get solid testing done

• for efficient debriefings

Short: 60 minutes (+-15) Normal: 90 minutes (+-15)

Long: 120 minutes (+-15)

10


The deliverable: a session report

• Charter – #AREAS

• Notes CHARTER

-----------------------------------------------

Analyze MapMaker’s View menu functionality and

report on areas of potential risk.

#AREAS

OS | Windows 2000

Menu | et-jsb-010417-c.sesView

Strategy | Function Testing

Strategy | Functional Analysis

Sample

START

-----------------------------------------------

5/30/00 03:20 pm

TESTER

-----------------------------------------------

Jonathan Bach

TASK BREAKDOWN

-----------------------------------------------

#DURATION

short

#TEST DESIGN AND EXECUTION

65

#BUG INVESTIGATION AND REPORTING

25

#SESSION SETUP

20

Metrics – #DURATION

– #TEST DESIGN AND EXECUTION

– #SESSION SETUP

– #BUG INVESTIGATION AND REPORTING

– #CHARTER / OPPORTUNITY

Issues – #ISSUE

Bugs – #BUG


Anticipating Scrutiny

Feelings: How are you feeling about what happened?

Past: What kinds of things did you do?

Results: What were your findings?

Obstacles: What slowed your mission?

Outlook: Where do we go from here?

Agenda: “PROOF” Past Results Obstacles Outlook Feelings

../../../../../sessions/approved/et-jsb-010417-c.ses

../SBTM versions/et-jsb-010417-c.ses







../SBTM versions/Sample

11


Session report

CHARTER

-----------------------------------------------



#AREAS .LINK

OS | Windows 2000

Menu |



START

-----------------------------------------------

5/30/00 03:20 pm

TESTER

-----------------------------------------------

Jonathan Bach

TASK BREAKDOWN

-----------------------------------------------

#DURATION

short


65


25

#SESSION SETUP

20

My session write-up from exploratory testing for

Agile2010…


eBay-related sample

CHARTER

-----------------------------------------------

Analyze MapMaker’s View menu functionality and report on areas of

potential risk.

#AREAS

LINK




START

-----------------------------------------------

5/30/00 03:20 pm

TESTER

-----------------------------------------------

Jonathan Bach

TASK BREAKDOWN

-----------------------------------------------

#DURATION

short


65


25

#SESSION SETUP

20"C:\Documents and Settings\jobach\Desktop\Live

Site\israt_sessions\Session_Listing_Promo.txt"

../Ideas & Intellect/Conferences/Agile2010/Adam_goucher_session.txt

../Ideas & Intellect/Conferences/Agile2010/Agile2010 session report.txt

../Jons eBay files/Live Site/israt_sessions/Session_Listing_Promo.txt







C:/Documents and Settings/jobach/Desktop/Live Site/israt_sessions/Session_Listing_Promo.txt

12


Exercise:

An “exploration and analysis” session


When to use SBTM

Getting support: “We have thousands of test cases and a

huge regression suite, but some of our best bugs come from ad

hoc testing. Why is that?”

Improving skill: “How she finds those great bugs without test

cases, I’ll never know. I guess some people are just natural

explorers -- you either have it or you don’t -- and I just don’t

have her knack for it.”

Earning trust and autonomy: “The triage team needs more

information about these 3 bugs, can you give me that by the

end of the day?”

../Ideas & Intellect/Exercises/keynote.exe

13


First: what’s there to test?


Second: ideas for charters

• What item for sale is nearest to you location? Submitted the most and sold the least? What item appears most with no bids? What are the most recent items posted for any given category? Which category has the most items? Which search strings create null queries? What are 3 examples of broad queries vs 3 examples of ambiguous queries? What queries create invalid results? How do you find "sold" items? What is the most common item? What is the most bizarre item? What is the most expensive item? Is it possible to restrict results to one category without the categories field? Which seller has listed the most items which have not sold? What search string takes the longest to return results? Is there a search that works differently on different browsers? What was the highest bid for an item in support of any charity? Find an item that the seller has no business selling! Find an item you find only on eBay, not Amazon. What item could you find that had the most bids? How many eBay stores are there? What is the oldest object (not item) for sale on ebay? Are results different if you are logged in? Try a search on ebay.com and then try the same on on your mobile. What are the 3 major differences? What item has the greatest difference between the highest and second highest bids? What user has bid the most but won the least? What item has been on eBay the longest? Is there a way to tell the age of the item ID? Who has sold the most items? Is there a way to find the most commonly misspelled item? What item can be found in the most categories?

14


Sources of charters

• Documentation / Specifications

• Web forums

• Previous products

• Team members

• Competing products

• Your expertise

• Help files

• Manuals


High-level sources

Test Plan Evaluation Model

http://www.satisfice.com/tools/tpe-model.pdf

Test Planning Checklist

http://www.satisfice.com/tools/build-the-plan.pdf

Heuristic Test Strategy Model

http://www.satisfice.com/tools/satisfice-tsm-4p.pdf

../Ideas & Intellect/Satisfice/tpe-model.pdf

../Ideas & Intellect/Satisfice/build-the-plan.pdf

../Ideas & Intellect/Satisfice/satisfice-tsm-4p.pdf






15


Charters

• General charters may be necessary at first:

• “Analyze the Insert Picture function”

• Specific charters provide better focus, but take more

effort to design:

• “Test clip art insertion. Focus on stress and

flow techniques, and make sure to insert into a

variety of documents. We’re concerned about

resource leaks or anything else that might

degrade performance over time.”


The power of collaboration

As a London Underground rider,

I want to enter my ticket into the kiosk,

Because I want to go to Heathrow.

What are the possible outcomes?

http://www.faqs.org/faqs/uk/transport-london/section-6.html

../Ideas & Intellect/Exercises/6_ Error codes on UTS gates.txt

6_ Error codes on UTS gates.txt










16


The session report


• Notes CHARTER

-----------------------------------------------



#AREAS

OS | Windows 2000




Sample

START

-----------------------------------------------

5/30/00 03:20 pm

TESTER

-----------------------------------------------

Jonathan Bach

TASK BREAKDOWN

-----------------------------------------------

#DURATION

short


65


25

#SESSION SETUP

20



– #SESSION SETUP



Issues – #ISSUE

Bugs – #BUG


A boring (Boeing) story










17


On October 30, 2006, Alaska Airlines flight 61, a Boeing

737-200, N740AS, took off from runway 34 right at

Seattle-Tacoma International Airport, Seattle,

Washington.

There were no injuries to the 71 passengers or 5

crewmembers and there was no damage to the airplane.

The airplane was operating under the provisions of 14

CFR Part 121 and continued uneventfully to its

destination of Juneau International Airport, Juneau,

Alaska.


The Report

18


Why the report?

1. The government is collecting statistics on runway

incursions, which are potentially deadly.

2. Since the pilots or ground controllers are subject to

reprimand or dismissal for this sort of thing, a legal record that

it happened is necessary.

3. Ground controllers or signage may have caused the

problem. You need a record of the event, and

investigation, to discover if there is another accident

waiting to happen.


Styles I’ve seen

1) Novel Narrative

2) Flight Data Recorder

3) ESPN Highlight

19


Clark’s notes…

Captain Clark, River Dubois opposite the mouth of the

Missouri River, 13 May 1804

“I dispatched an express this morning to Captain Lewis

at St. Louis. All our provisions, goods, and equipage on

board of a boat of 22 oars [party], a large pirogue of 71

oars [in which 8 French], a second pirogue of 6 oars

[soldiers], complete with sails, &c. Men completed with

powder cartridges and 100 balls each, all in health and

readiness to set out. Boats and everything complete, with

the necessary stores of provisions and such articles of

merchandise as we thought ourselves authorized to

procure-though not as much as I think necessary for the

multitude of Indians through which we must pass on our

road across the continent.”


What to write while exploring

Historical Explorer Tester

Observations (To the degree you think they are relevant to stakeholders)

• drawings of flora / fauna

• descriptions of indigenous people

• landmarks

• feature model

• text from log files

• text from dialogs

Conjectures (Inferences based on experiences. After I test, I think I know something)

• what is this thing?

• where should we go today?

• how do we get there?

• new orders from HQ?

• are those people hostile?

• test ideas

• questions

• product and project issues

• concerns

• risks

Project information (Independent of observer)

• mission

• supplies and staff

• latitude / longitude

• death and disease

• supply status

• charter

• test actions

• config info

• build details

• tools used

20


Scrutiny: what happened here?

“(Orderly Book) Camp Mouth of the Kansies June 29, 1804

Ordered --- A Court Martiall will Set this day at 11 oClock... for the

trial of John Collins and Hugh Hall...

John Collins Charged "with getting drunk on his post this Morning

out of whiskey put under his charge as a Sentinal, and for suffering

Hugh Hall to draw whiskey out of the Said Barrel intended for the

party."

...The commanding Officers approve of the Sentence of the Court

and orders that Punishment take place at half past three this

evening, at which time the party will Parrade for inspection.”

June 29, 1804 William Clark


Jon Bach, 2001

Sample test notes from actual exploratory testing :

et-jsb-010416-a.ses

et-jsb-010416-b.ses (for Issues)

et-jsb-010416-c.ses (a map for more ideas)

et-jsb-010417-a.ses (Claims Testing)

et-jsb-010417-b.ses (for Bugs)

et-jsb-010417-c.ses (for referenced notes)

et-jsb-010418-c.ses (more Claims)

../../../../sessions/approved/et-jsb-010416-a.ses







../../../../sessions/approved/et-jsb-010416-b.ses







../../../../sessions/approved/et-jsb-010416-c.ses



































21


How much detail is enough?

1) Sufficient benefits

2) No critical problems

3) Benefits outweigh problems

4) All things being equal, further documentation (or time,

effort, testing, editing, etc.) is more harmful than helpful

All 4 conditions must be met to know when you’re done,

(From brother James’ Rapid Software Testing course)

But ask: “good enough for who, what and when?”


Metrics and Mechanics

Gathering empirical data about the object of your study;

collecting different kinds of data, or data about different

aspects of the object. Designing experiments and establishing

lab procedures.

22


The session report


• Notes CHARTER

-----------------------------------------------



#AREAS

OS | Windows 2000




Sample

START

-----------------------------------------------

5/30/00 03:20 pm

TESTER

-----------------------------------------------

Jonathan Bach

TASK BREAKDOWN

-----------------------------------------------

#DURATION

short


65


25

#SESSION SETUP

20



– #SESSION SETUP



Issues – #ISSUE

Bugs – #BUG


What to measure

Test Design and Execution

Bug Investigation

(and Reporting)

Session Setup










23


Reporting the metrics

• Test, Bug, and Setup are orthogonal categories, but all we’re only tracking

interruptions to testing

• Nearest 5% or 10% is good enough

• If activities are done simultaneously, report the highest precedence activity:

T, B, then S


35%

25%

10%

15%

15%

Test Design & Execution

Bug Investigation

Environment Config

Automation (BLOCKERS)

Learning-Gathering Requirements

TBS Report

24


Sample project

HTML status report


Tools

1) Scan Tool (James & Jon Bach)

2) Session Tester (Jonathan Kohl)

3) Rapid Reporter (Shmuel Gershon)

../../../../sessions/scan-approved-then-run-report.bat

../../../../sessions/scan-approved-then-run-report.bat

C:/Program Files (x86)/Session Tester/SessionTester.exe

../Ideas & Intellect/Conferences/STAR West 2011/RapidReporter.exe

25


Non-Session

61%

Test

28%

Bug

4%

Opportunity

1%

Setup

6%

Diagnosing productivity

• Do these proportions make sense?

• How do they change over time?

• Is the reporting protocol being followed?

0.0

50.0

100.0

150.0

200.0

250.0

300.0

5/26 6/9 6/23 7/7 7/21 8/4 8/18


Estimating a Test Cycle

1. How many perfect sessions (100% on-charter testing)

does it take to do a cycle? (let’s say 40)

2. How many sessions can the team (of 4 testers) do

per day? (let’s say 3 per day, per tester = 12)

3. How productive are the sessions? (let’s say 66% is on-charter test design and execution)

4. Estimate: 40 / (12 * .66) = 5 days

5. We base the estimate on the data we’ve collected.

When any conditions or assumptions behind this

estimate change, we will update the estimate.

../../../../sessions

26


Testing is like looking for words…

???

(10 minutes)


Debriefing

All groups: How did you feel about your mission?

Mission A: What happened during the session?

Mission B: How many words did you find?

Mission C: What obstacles did you face?

Mission D: Do you think you found all the words?

Agenda: “PROOF” Past Results Obstacles Outlook Feelings

Boggle missions.ppt

27


Next steps?

1) Tell the “tactics” story: Practice using terminology for the

skills used during exploration. It has helped managers

understand the sapience and discipline of exploration.

2) Try a three-week SBTM pilot: Frame your explorations in

sessions (time-boxed effort with a report) and see how your

stakeholders react.

3) Use PROOF to anticipate scrutiny: Testing ourselves is just

as important as testing software. It has won testers more

credibility, autonomy, and respect.


More info

• ET Dynamics:

http://www.satisfice.com/articles/et-dynamics.pdf

• Test Heuristics and Planning

http://www.satisfice.com

• Context-Driven Software Testing

http://groups.yahoo.com/group/software-testing

• Center for Software Testing Education and Research

http://www.testingeducation.org/BBST

• Books related to Exploratory Testing skills and tactics

http://www.testingreflections.com/node/view/3190

• Scenario testing examples

http://www.testingeducation.org/a/scenario2.pdf







http://www.testingeducation.org/BBST

http://www.testingreflections.com/node/view/3190

Date post:	11-May-2015
Category:	Technology
Upload:	techwellpresentations
View:	390 times
Download:	0 times

Exploratory Testing Is Now in Session

Technology