MJ AM Tutorial

9/30/2013 8:30:00 AM

"Exploratory Testing Explained"

Presented by:

Jon Bach

eBay

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Jon Bach

eBay, Inc.

With more than eighteen years of experience in software testing, Jon Bach has held technical

and managerial positions in companies including Hewlett-Packard and Microsoft. In his current

role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests

(activity flows) in eBay’s core sites to discover important bugs that threaten its core business.

He is most notable for creating, with his brother James, Session-Based Test Management, a

method to manage and report exploratory testing.

1

Exploratory Testing Explained

Jon Bach

QE Director, eBay

jobach@ebay.com

STAR West 2013

Do you see structure here?

2

3

How about here?

… or here?

http://www.itechnews.net/2008/03/29/steve-jobs-mosaic-portrait/

3

… or here?

http://www.japanquakemap.com/

6

… or here?

4

7

Preamble

Ever use the term "playing around" to describe your testing?

Ever cringe after saying it, wishing there was a better way of

describing what you did than to give the impression it was all

accidental and random?

If so, this workshop may help you understand and explain

exploratory testing as a thoughtful, purposeful approach whose

results stand up under scrutiny.

8

Promises

Participate in exercises that focus on bug isolation and

investigation, risks and vulnerabilities.

Learn frameworks and heuristics of exploration to use in tight

situations

Discover ways to report your exploration so it stands up to

scrutiny.

There is structure and purpose if you know how to

identify it and tell a story about it.

5

Why this talk?

1) Exploratory testers want respect: When testers explore

during testing, they find great bugs. However, since they

often don’t know how to describe their thinking, it’s

considered to be dismissed as “playing around”.

2) The documentation dilemma: Project managers may insist

that all testing be documented, so how to balance time spent

documenting with time spent testing?

3) Your work might be scrutinized: You may have to give a

report someday about something you did that was

exploratory – like attending this conference.

Exercise (ebay Search)

Most bizarre thing for sale on ebay?

Most expensive thing on ebay?

What’s trending?

How can you find completed items?

What’s most common item sold?

How many categories of items for sale?

How many actual items?

6

Exploratory Testing

• Sabourin: “continuous test design as testing continues;

continuous testing as design continues; continuous test

planning as testing continues”

• Hendrickson: a style of testing in which you explore the

software while simultaneously designing and executing

tests, using feedback from the last test to inform the next

(Test-Driven Testing?)

• Bolton: Operating and observing the product with the

freedom and mandate to investigate it in an open-ended

search for information about the program.

• Kaner: Simultaneous learning, design and execution,

with an emphasis on learning.

“The” ET Definition

A style of software testing…

that emphasizes the personal freedom…

and responsibility of the individual tester…

to continually optimize the quality of his/her work…

by treating test-related learning…

test design…

test execution…

and test result interpretation…

as mutually supportive activities…

that run in parallel…

throughout the project.

-- Cem Kaner, 2006

7

“The” ET Definition

A style of software testing…

that emphasizes the personal freedom…

and responsibility of the individual tester…

to continually optimize the quality of his/her work…

by treating test-related learning…

test design…

test execution…

and test result interpretation…

as mutually supportive activities…

that run in parallel…

throughout the project.

-- Cem Kaner, 2006

Analogies

Driving a car

Job Interview

Psychologist

Sports

“20 Questions”

Jam session

Newspaper reporter

Going to a testing conference

Bounty Hunter

8

Missions that inspire ET

• Change test case variables

• Execute a checklist

• Regress a list of bugs

• Confirm a rumor

• Design a test case

• Write some automation

16

Testers light the way.

This is our role. We see things for what they are.

We make informed decisions about quality possible,

because we think critically about software.

9

Testing is…

an infinite process

of comparing the invisible

to the ambiguous

in order to avoid the unthinkable

happening to the anonymous.

Key Idea

Testing is…

an infinite process

of comparing the invisible

to the ambiguous

in order to avoid the unthinkable

happening to the anonymous.

Key Idea

10

19

What is testing?

“Try it and see if it works.”

Learn anything reasonable that matters about

whether it can work and how it might not work.

20

What is testing?

“Try it and see if it works.”

Get it set up

Run it

Run it again, maybe

Choose where to look

See what’s there

See what’s not there

Read specs

See if product matches

Find problems…

…especially the bad

ones

Procedures Oracles Coverage

11

“I want you to test this…”

22

What is testing?

“everything that matters”

If you don’t have an understanding and an agreement on what is

the mission of your testing, then doing it “rapidly” would be

pointless.

12

The “tester freedom” scale

pure scripted freestyle exploratory

charters vague scripts

fragmentary

test cases

(scenarios) roles

To know where a test falls on this scale, the

tester must ask themselves: “to what extent

am I in control of the test, and from where did

the idea originate?”

Exercise

This app asks you for the next

item in a sequence of numbers.

Find the pattern in the

minimum number of tests.

(each line is a test)

Operating rule

13

Exploration is discovery…

?

?

?

?

?

?

…that starts with an idea…

14

and ends with a perception…

After

exploring

Before

exploring

…depending on the mission

15

…mission, mission, mission

If you don’t know your mission,

you’re not testing.

That’s ok, just call it *touring*.

Lewis & Clark, 1802

Mission: Find a water passage across North America…

16

The charter from Jefferson

“The object of your mission is to explore the Missouri river, & such principal stream of it, as, by its course & communication with the water of the Pacific ocean may offer the most direct & practicable water communication across this continent, for the purposes of commerce.” http://www.monticello.org/jefferson/lewisandclark/instructions.html

Chartering

Making your own decisions about what you will work on and

how you will work. Understanding your client’s needs, the

problems you must solve, and assuring that your work is on

target.

17

Sponsors and stakeholders

• Test Manager

• Product Manager

• CEO

• Customer

• Developer

• Marketing

• Tech Writer

• Customer Support

• Other testers

Charter-based method #1

Session-Based Exploration

Think in time-boxed missions to

explore, resulting in a test report

with Notes, Bugs, and Issues.

18

Structure

1) Time Box

2) Reviewable Result

3) Debriefing

The “Session”

“I want you to test this…”

My testing demo…

19

Some sample session charters

Installation: When installed, does Triangle! put any files in the wrong places? Does it leave any files for the uninstall? Check the registry keys, use InCtrl to see what changes are made. Installation is new, so we want to be sure it’s clean.

Boundary testing: We got word from customer support that there are run-time errors when using integers over 32000 but no one can repro it. Best recon is on Win XP Pro with Office 2003 running in the background. Sam K. in CSS says you can use his machine, and he also has customer specs.

Ship drill: Start Triangle! right out of the box. For example, is the readme ready to go? We’re waiting from word on Legal as to the License Agreement, but that shouldn’t hold you up. Also make sure you hit Vista and see what issues arise there.

Claims testing: Triangle is meant for first graders, but we plan to ship a version to General Dynamics in a few months. Try some usability profiles or personas to see what functions become more or less risky. Also, discover the algorithm by which Triangle! reports its results. Is it way off from what a user would expect. Does it cause the user to lower their confidence?

Charter-creation method #2

Open-Book Testing

The act of creating open-ended

questions such that…

20

…are immersed in the product right away, building a model or

mind map.

…learn how they are provoked into critical thinking by being

exposed to many types of questions (test ideas).

… quickly find bugs and raise issues in answering the questions

they are given

…testers…

test ideas

test cases

test scenarios

test plans

test scripts

test designs

test strategies

test heuristics

} Questions

21

test ideas

test cases

test scenarios

test plans

test scripts

test designs

test strategies

test heuristics

} Questions

These comprise the exam to which software will either pass or fail.

A few non-obvious (?) sources for charters

• Bug database

• Testers (paired testing)

• Programmers (different domain expertise)

• Similar (or competing) products

• Customer Support

• Claims made by marketing

• Emails / Meetings / RSS feeds

22

Resources

Documentation / Specifications

Web forums

Previous products

Team members

Competing products

PSS data / KB articles

Your expertise

Heuristics

Help files

Manuals

Questions and answers can originate from the same sources:

IM OPEN

Interrogate: The test manager or tester develops a list of questions to answer.

Manipulate: The testers execute actions to answer the question.

Observe: Testers take notes on what they find.

Plan: Testers determine any follow-up questions (tests) that occur to them, in preparation to debrief their results.

Evaluate: Testers and test manager meet to compare answers (test results).

Negotiate: After the debrief, testers and test managers talk about the appropriate next steps in mission or coverage

23

45

Tests

Project

Environment

Product

Elements

Quality

Criteria

Perceived

Quality

A Heuristic Test Strategy Model

46

Project

Environment

Tests

Product

Elements

Quality

Criteria

Perceived

Quality

A Heuristic Test Strategy Model

24

47

Coverage

Structure

Function

Data

Platform

Operations

Time

Product coverage is the proportion of the product that has been tested.

Capability

Reliability

Usability

Security

Scalability

Performance

Installability

Compatibility

Supportability

Testability

Maintainability

Portability

Localizability

Structural Coverage

platform

input output Test what it’s

made of.

Print testing example

– Files associated with printing

– Code modules that implement printing

– Code statements inside the modules

– Code branches inside the modules

25

Functional Coverage

input output

platform

functions Test what

it does.

Print testing example

– Print, page setup and print preview

– Print range, print copies, zoom

– Print all, current page, or specific range

functions

Data Coverage

functions

&

structure

input output

platform

Test what

it does it to.

Print testing example

– Types of documents

– Items in documents, size and structure of

documents

– Data about how to print (e.g. zoom factor, no. of

copies)

26

Platform Coverage

Test what it

depends upon.

Print testing example

– Printers, spoolers, network behavior

– Computers

– Operating systems

– Printer drivers

functions

&

structure

input output

platform

Operations Coverage

Test how

it’s used.

Print testing example

– Use defaults

– Use realistic environments

– Use realistic scenarios

– Use complex flows

input output

platform

27

Time Coverage

Test how

it’s affected

by time.

Print testing example – Try different network or port speeds

– Print one document right after another, or after long intervals

– Try time-related constraints--spooling, buffering, or timeouts

– Try printing hourly, daily, month-end, and year-end reports

– Try printing from two workstations at the same time

– Try printing again, later.

input output

platform

Exercise

Does it work?

What is the hidden feature?

What story does the data tell?

28

How did you *find* that?

Some Exploration Skills and Tactics

Modeling

Resourcing

Questioning

Recording

Reporting

Exploratory testing is a mindset using this skillset.

Chartering

Observing

Manipulating

Pairing

Generating/Elaborating

Refocusing

Alternating

Branching/Backtracking

Conjecturing

“MR.Q COMP GRABC R&R?”

56

Skills of Exploration

Put the tester's mind at the center of testing.

Learn to deal with complexity and ambiguity.

Learn to tell a compelling testing story.

Develop testing skills through practice, not just talk.

Use heuristics to guide and structure your process.

Be a service to the project community, not an obstacle.

Consider cost vs. value in all your testing activity.

Diversify your team and your tactics.

Dynamically manage the focus of your work.

Your context should drive your choices, both of which evolve over time.

29

Testing ourselves

Chartering is an opportunity for testers and managers to

cultivate and improve testing skill:

How did you arrive at that answer?

What did you see along the way?

Was there anything confusing about the questions?

Any riffs off of questions?

What test ideas did others have with the same question?

What managers might ask

Was this charter reasonable?

What did you find?

How did you spend your time?

Did you need some help / tools?

Do you think there’s more to do here?

Agenda: “PROOF” Past Results Obstacles Outlook Feelings

30

The real message

What’s being asked What they may be thinking

What was your mission? Remind me what I told you to do…

How did it go? What do I worry about next?

How far did you get? Are we closer to shipping?

Need anything? Can I speed this along?

When will you be done? Will I get my bonus?

What to document

Historical Explorer Tester

Observations (To the degree you think they are relevant to stakeholders)

• drawings of flora / fauna

• descriptions of indigenous people

• landmarks

• feature model

• text from log files

• text from dialogs

Conjectures (Inferences based on experiences. After I test, I think I know something)

• what is this thing?

• where should we go today?

• how do we get there?

• new orders from HQ?

• are those people hostile?

• test ideas

• questions

• product and project issues

• concerns

• risks

Project information (Independent of observer)

• mission

• supplies and staff

• latitude / longitude

• death and disease

• supply status

• charter

• test actions

• config info

• build details

• tools used

31

Testing *is* journalism

It involves a story formed by following up on rumors,

tips, leads, conjectures, and questions – in pursuit of

the truth.

It involves consulting sources, references, oracles -- and

taking notes about those details.

It requires communication to an audience who wants

information and who will either scrutinize or trust your

report.

When I was 10, Dad said…

Somebody wants something…

Something stands in their way…

This is what they do about it…

Every story is this simple:

32

Story Elements (Testing)

Characters (Somebody)

Testers

Customers

Stakeholders

Purpose (Wants something)

“How stable are these new features?”

“I want to print all of my recipes.”

“Try to repro this bug.”

Conflict (Something’s in the way)

Limited budget and time

"How does this thing work?”

"We have yet to run <these> tests.”

Actions (What was done about it)

Risks exposed

Techniques used

Features covered

Agility is about the freedom

to create, learn, and adapt,

as we get fast feedback.

[ Responding to change over following a plan ]

Key Idea

33

Exploratory testing is about

the freedom

to discover, learn, and adapt,

while delivering fast feedback.

Key Idea

A report of my exploration

34

Activities to report

Test Design and Execution

Bug Investigation

(and Reporting)

Session Setup

Conclusion

There is structure and purpose in exploration …

know how to identify it.

Management Method: Session-Based tests

Chartering Method: Open-Book Testing

Idea Method: Heuristic Test Strategy Model

Technique inventory: stress, flow, risk, claims,

etc…