© 2011 SAP AG. All rights reserved. 1

Exploratory Testing

Haya Rubinstein

Self Evaluation

Extensive testing

• Automated and Manual

• Regression, Integration, Scenarios

• Performance, platform tests

• Validation by a special team

Still have defects

Under 90% of our defects were

discovered before release

Why?

What should we improve?

Why does our software still have bugs?

Highly configurable

Endless possible database formats

Endless possible scenarios

© 2011 SAP AG. All rights reserved. 4

Why Exploratory Testing?

• Test differently every time

• Different people = Different styles = Different bugs

Find more Bugs

• Redesign the test each time you test

• Added Responsibility

Use your brain

• Different Exploratory test types provide different approaches

Vary test types

© 2011 SAP AG. All rights reserved. 5

When and how to use Exploratory Testing?

• Learn

• Design test

• Test Simultaneously

• Personal Freedom

• Personal Responsibility Style

• In parallel w/ other tests

• Throughout the project Timeframe

© 2011 SAP AG. All rights reserved. 6

Challenges

Limited

• Lose focus

• Wider scope - >more potential bugs

• More confusing

• More self empowerment

Detailed

• More focus

• Less scope -> less potential bugs

• Not confusing

• Less self empowerment

Test Data

•What and how much to provide?

Test instructions

•How detailed?

Timeframe

•To limit or not?

© 2011 SAP AG. All rights reserved. 7

How we decide what to test

© 2011 SAP AG. All rights reserved. 8

A sample exploratory test

Load Goodyear Global Vendor Repository 1:

Open a Data Manager with Goodyear Global Vendor repository 1:

Rename a couple of validations. Note which validations you renamed.

Delete some validation groups. Note which validations you deleted.

Rename a workflow. Note which workflow you renamed.

Delete a workflow. Note which workflow you deleted.

Open Import Manager with Goodyear Global Vendor repository 1:

Delete an existing map. Note which map you deleted.

Open Syndicator with Goodyear Global Vendor repository 1:

Delete an existing map. Note which map you deleted.

Unload Goodyear Global Vendor Repository 1.

In Goodyear Global Vendor Repository 2 add a new field to the main table.

© 2011 SAP AG. All rights reserved. 9

Tours we use for new releases and platforms

Back-alley: Not usually used during daily work We use this when testing every new release

• Data recovery

• Preference dialogs

• First time use

All-Nighter: Run it for a long time We use this when testing new platforms & compilers

• Find memory leaks

• Simulate week long runs or runs over the weekend

© 2011 SAP AG. All rights reserved. 10

Tours we use when Testing Customer Scenarios

FedEx: Identify and follow specific pieces of data

• Follow the data through a whole scenario from beginning to end.

• Check that the software interacts with the data. Properly in each step.

• Many of our E2E scenarios follow the customer data through each step

Test one get one free: Multiuser /Multiple instances

• Start multiple instances, and cause each application to manipulate the memory and the disk.

• Run multiple instance scenarios that manipulate the same resources.

• We do exploratory multiuser tests when testing customer scenarios

After-Hours: Tasks that run without user interaction

• Run tasks such as: backup jobs, automated archives, clients polling a backend server.

• This tour makes sure that code that might not be counted as a “feature” will be tested

• We use this when testing customer scenarios for a new release

© 2011 SAP AG. All rights reserved. 11

Tours we use during New Feature Test Phases

Saboteur : Be destructive with necessary resources

• Remove/restrict resources: file access ; memory/disk space

• Change environmental conditions: network connections; power supply.

Obsessive Compulsive: Rerun scenarios many times over

• Rerun steps that manipulate data. Can generate bugs if repeated many times.

• Regenerate error messages repeatedly. Error handling code is not top priority.

Bad Neighborhood: Test the weak areas

• Once you have tested for some time make a list of the "bad neighborhoods"

• Focus on those areas using tours like the Garbage collector to get rid of the bugs

Antisocial: Be contrary and perform negative tests

• Use invalid inputs instead of valid ones. Execute features in the wrong order.

• You should expect to see the a lot of error messages in this tour.

© 2011 SAP AG. All rights reserved. 12

Exploratory Test Guidelines

• Predefine some test parameters

• Record test cases Focus

• Preset test environment

• Provide a time limit Resources

• Analyze defects per iteration/tester

• Good test -> Different defects Results

© 2011 SAP AG. All rights reserved. 13

Questions

© SAP AG 2009. All rights reserved. / Page 14

Thank You