MQ PM Tutorial

4/29/13 1:00PM

The Craft of Bug Investigation

Presented by:

Jon Bach

eBay, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Jon Bach

Jon Bach works for eBay (San Jose) as a QA director for the Buyer Experience team. Formerly a manager of corporate intellect and senior test consultant at Quardev, Jon has been in testing since 1995 with experience that includes managing teams at Microsoft, HP, and LexisNexis. The co-inventor (with his brother James) of session-based test management, Jon is an award-winning speaker on test management and exploratory testing. He’s mostly known for his “half-baked” ideas about how to inspire innovation and create a learning culture for test teams. Find Jon on Facebook and Twitter @jbtestpilot. View his presentations at quardev.com/articles and his blog at jonbox.wordpress.com.

The Craft of (black box)Bug Investigation

Jon Bach

QE Director, eBay

jobach@ebay.com

STAR East 2013

eBay Inc. confidential

Preamble

These next few hours are designed with exercises that lead you through ways of thinking that help you narrow down a bug.

We’ll discuss techniques for bug isolation, reproducing intermittent bugs, and skills to help you know when you’ve gone as far as is reasonable before handing it off.

eBay Inc. confidential

A morning of investigation

Psychic Reader

Checkout Checking Out?

Art Show

Mysterious Spheres

Out­of­Bounds

Gettysbug Address

eBay Inc. confidential

CIDTESTDSFDPOTCRUSSPICSTMPLFDSFSCURA

Product Elements

StructureFunction

DataPlatform

OperationsTime

General Test Techniques

Function testingDomain testingStress testingFlow testing

Scenario testingClaims testingUser testingRisk testing

Automatic testing

Project Environment

CustomersInformation

Developer relationsTeam

Equipment & toolsScheduleTest Items

Deliverables

Quality Criteria

CapabilityReliabilityUsabilitySecurity

ScalabilityPerformanceInstallabilityCompatibility

Ways we find bugs

} } } } }

Development Criteria

SupportabilityTestability

MaintainabilityPortability

Localizability

eBay Inc. confidential

How *well* do we find bugs?

Some Investigation Skills and Tactics

Modeling

Resourcing

Questioning

Recording

Reporting

Investigation is a mindset using this skillset.

Chartering

Observing

Manipulating

Collaboration

Generating/Elaborating

Overproduction/Abandonment

Abandonment/Recovery

Refocusing

Alternating

Branching/Backtracking

Conjecturing

“MR.Q COMC GOARABC R&R?”

eBay Inc. confidential

How well do we investigate them?

Good enough quality bar

Further time and effort

is crucial.

Further time and effort

is a waste of resources.

Unacceptable Quality

Too Much Quality

floating  line

Bug Report

Bug Report

eBay Inc. confidential

Abductive Inference

1. Collect data.

2. Find several explanations that account for the data.

3. Find more data that is either consistent or inconsistent with explanations.

4. Choose the best explanation that accounts for the important data, or keep searching.

Abductive inference means finding the best explanation for a set of data.

Jump to conjectures, not conclusions

eBay Inc. confidential 8

This is a dip in bids we saw in Production.

What questions do you have to troubleshoot this?

Is Checkout Checking Out?

eBay Inc. confidential 9

Topics for investigation (FORCOSTTTIP)

• Feature – Do we know what happens when people bid?

• Origin ­­ When did this start happening?

• Recent – Did we just push code to Production?

• Consistency / Correlation ­­ What happened yesterday, tomorrow? Are there other graphs and metrics?

• Overlap – Is there any other process happening?

• Security – Is it malicious? Intentional?

• Terminology ­­ What kinds of “bids”? US, UK, International?

• Third­Party / Silent Partner – What’s not in the room? What sort of information have we NOT heard from?

• Truth is out there – does someone already know about this?

• Instrumentation – is the graph wrong? Can I get more data?

• Prediction – can we test a conjecture with a prediction

eBay Inc. confidential

Exercise: Psychic Reader

eBay Inc. confidential

Lessons

You can be psychic, too

You may be clue-less, for now

Try “defocusing”

Are you just following orders?

eBay Inc. confidential

Exercise: Mysterious Spheres

eBay Inc. confidential

Lessons

If it fails, whose fault is it?

Assumptions need nourishment

Oracles are in “control”

“Elsewhere”, is a useful thought

eBay Inc. confidential

Exercise: Out-of-Bounds

eBay Inc. confidential

Lessons

Be a “model” tester

Stagehands work behind curtains

Outside data is not inside data

Computers are our slaves

eBay Inc. confidential

Exercise: Art Show

What is my Operating Rule?

What do you notice?

What (crazy) test haven’t you tried?

eBay Inc. confidential

Lessons

“You’re so one dimensional”

Jump to *conjectures*

Yes, I *do* look FAT in this. (M vs. O)

Breaking “rules”!

eBay Inc. confidential

Exercise: Gettysbug Address

Does it work?

What is the hidden feature?

What story does the data tell?

eBay Inc. confidential

Lessons

The Rumble Strip heuristic

Missing a mission?

Be “thoughtless” (random is good)

Defocus: do *anything*, *differently*

eBay Inc. confidential

How much investigation?

The answer must be “Yes” to all four criteria,but ask “for whom, what, and when?”

1)  Sufficient benefits

2)  No critical problems

3)  The benefits outweigh the problems

4)  In the present situation, and all things considered, improvement would be more harmful than helpful

eBay Inc. confidential

Intermittent Bugs: Observation

Link

Bad observationIrrelevant observation

Bad memoryMisattribution

MisrepresentationUnreliable oracle

Unreliable communication

eBay Inc. confidential

Intermittent Bugs: System

Purposeful change, and then back to originalAccidental changePlatform changeFlaky hardware

Trespassing systemExecutable corruption

Component competition

eBay Inc. confidential

Intermittent Bugs: Machine State

Frozen conditionalImproper Initialization

Resource denialProgressive data corruptionProgressive destabilization

OverflowOccasional functions

Different mode or option setting

eBay Inc. confidential

Intermittent Bugs: Input

AccidentalSecret boundaries or conditions

Different profileGhost input (alternative source)Simultaneous action as others

Compromised inputTiming issues

Crazy Combinations

eBay Inc. confidential

Intermittent Bugs: You?

You may not be aware of...

…variables of influence…sources of distortion in your observations

…available tools that might help…boundaries and their characteristics

…the system's missing / extra functions…complex / competing algorithms

eBay Inc. confidential

A few (free) investigation tools

Httpwatch­­ displays log of requests and responses

Firebug – inspect HTML

­­ JavaScript debugger

WAVE– accessibility errors

eBay Inc. confidential

More info

• ET Dynamics:

http://www.satisfice.com/articles/et­dynamics.pdf

• Test Heuristics and Planning

http://www.satisfice.com

• Context­Driven Software Testing

http://groups.yahoo.com/group/software­testing

• Center for Software Testing Education and Research

http://www.testingeducation.org/BBST

• Books related to Exploratory Testing skills and tactics

http://www.testingreflections.com/node/view/3190

• Scenario testing examples– http://www.testingeducation.org/a/scenario2.pdf