Test case and BUG report

Andrey Oleynik

2014

Agenda

1. Test Case.2. Examples.3. BDD.4. Examples.5. Defects 6. BUG Report7. Examples8. Good BUG9. Additional links

2

About myself

Andrey Oleynik

• About 7 years of work in Quality Assurance area

• QA Automation Engineer

• PhD, Theoretical physics

3

What is a Test Case?

• A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [IEEE]

• Documentation specifying inputs, predicted results, and a set of execution conditions for a test item. [ANSI]

• A test case is a bunch of steps that lead to consider one "item" as successfully working.

4

Why do we write test cases?

• The basic objective of writing test cases is to validate the testing coverage of the application

• Usually QA engineers follow the test cases standards

• So writing test cases brings some sort of standardization and minimizes the ad-hoc approach in testing

5

What information would the test manager want

out of test case document/s?

• Which features have been tested/ will be tested eventually?

• How many user scenarios/ use cases have been executed?

• How many features are stable?• Which features need more work?• Are sufficient input combinations exercised?• Is software ready to ship? Is testing enough?• What is the quality of the application?

6

Test Case design

• Specification based testing;

• Input Domain testing;

• Risk based testing;

• Scenario testing.

7

Test Case Fields(the most important)

• Test Case id

• Title

• Priority

• Description

• Steps

• Expected result

• Feature/Area8

Test Case fields

• Test case status (Idea/Design/Ready/Update/etc.)

• Product version• Responsible person• Preparation notes• Notes to clean the system• Attachments• Examples• Linked feature/requirement/bug• ...

9

HP ALM example

10

ALM example(Test Case steps)

11

Execution example (ALM)

12

Example

Title: Open file.Description: a file can be opened in the Application.

1. Launch the Application.2. Select "File" menu.

File menu pulls down.3. Choose "Open“.

"Open" dialog box appears.4. Select a file to open.5. Click OK.

Expected result: File should be opened.

13

Important:

1. Template

2. Descriptive and specific

3. Reusable

4. Atomic

5. Positive and negative

6. Refactor

7. Test data

8. Setup and tear down

9. Test Case review

10. Screenshots

14

Test Suite

• A test suite is a set of related tests, usually pertaining to a group of features or software component and usually defined over the same database. Suites are combined into groups. [Beizer]

15

Test management tools

• Gemini• HP ALM (QC)• IBM Rational Quality Manager • Meliora Testlab• Seapine TestTrack• TestLink• Google Docs !• …

16

Q & A

17

Testing pyramid

18

TDD

Test-driven development is a software development methodology which essentially states that for each unit of software, a software developer must:

• define a test set for the unit first;

• then implement the unit;

• finally verify that the implementation of the unit makes the tests succeed.

19

TDD

• Java (Junit, TestNG);

• C (Check, CUnit);

• C++ (Boost, CppUnit, CppUnitLite, Google Test);

• C# (xUnit, NUnit, MSTest);

• Ruby (Test::Unit, Shoulda, RSpec);

• Python (pyUnit, py.test, doctest);

• …

20

BDD

• Behavior-driven development is a specialized version of test-driven development which focuses on behavioral specification of software units.

• Behavior-driven development specifies that tests of any unit of software should be specified in terms of the desired behavior of the unit.

21

BDD

22

TitleThe story should have a clear, explicit title.

NarrativeA short, introductory section that specifies

•who •which effect the stakeholder wants the story to have •what business value the stakeholder will derive from this effect

Acceptance criteria or scenariosa description of each specific case of the narrative.

•It starts by specifying the initial condition •It then states which event triggers the start of the scenario. •Finally, it states the expected outcome, in one or more clauses.

BDD example

23

Feature: New accountIn order to have new ready to use accountAs a userI want to get an account with balance $0

Scenario: Create new accountGiven I have no accountWhen I create an accountThen I should have an account with balance $0

BDD

• Jbehave (Java)

• Thucidydes

• Rspec (Ruby)

• Cucumber

• SpecFlow (C#)

• …

24

Q & A

25

What is a BUG(defect)?

• A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.

26

BUG (Harvard Mark II, 1947)

27

Another BUG definition

• A software bug is an error, flaw, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways.[Wikipedia]

28

BUG names• Mistake• Anomaly• Fault• Failure• Error• Exception• Crash• Bug• Defect• Incident• Side Effect• Change request

29

Types of BUGs• UI • Functional issues• Programming logic• Installation • Memory allocation • System resources• Documentation • Requirements • Localization (L10N) • Internalization (I18N) • Performance • Security• …

30

How to find a bug?

• Understand the whole application in depth

• Prepare good test cases and test data before testing

• Execute tests with different test environment

• Compare actual and expected results

31

What to do next?

• Do monkey testing

• Use test data pattern from previous versions/projects

• Check standard scenarios from other products

• Try to break the application!

32

BUG report

Defect/BUG report: A document reporting on any flaw in a component or system that can cause the component or system to fail to perform its required function. [IEEE]

33

“The point of writing problem report(bug report) is to get bugs fixed” [Cem Kaner].

Defect Fields(the most important)

• Bug ID• Summary• Description/Steps to Reproduce• Expected/Actual result• Status• Severity• Priority• Detected by• Assigned to• Module/Area

34

Defect Fields

• Environment

• Reproducible

• Fixed in Version

• Detected Date

• Modified/Last Updated

• Detected in Version/Build Version

• Comments

• Attachments

• Linked testcases/requirements/defects

35

Defect Life Cycle

36

Defect Resolution

37

Defect Resolution

• Fixed

• Won't Fix

• Not Reproducible

• Duplicate

• By Design

• External / 3d party

38

Defect Severity

1 - Urgent• (Critical) • Data loss or corruption. Product is down. Customer operations are

impacted very seriously. There is no any workaround. • For documentation: copyright or offensive content issues,

documentation steps lead to data loss or corruption.

2- High• (Serious) • System / feature is still usable to a high degree but has a serious

issue. Defect has serious customer business impact. • Some data is at risk. No obvious workaround.• For documentation: core functionality is not described.

39

Defect Severity

3 - Medium• Product operational • Application customer data is not at risk. • Obvious/easy workaround available. • Non informative error reporting.• For documentation: Conceptual information is incorrect or

incomplete.

4 - Low• It is more of a cosmetic issue. No serious impedance to

system functionality is noted. Annoyances. • For documentation: typography, formatting.

40

Priority

0. Show-stopper - this needs to be fixed right now, everything else can wait, the build cannot be released with this defect1. Urgent - needs to be fixed before any other high, medium or low defect should be fixed2. High - should be fixed as early as possible3. Medium - should take precedence over low priority defects4. Low - fixing can be deferred until all other priority defects are fixed

41

JIRA example

42

ALM/QC example

43

BUG exampleTitle/Summary: Application crash on clicking the Browse button while file open.

Area: File operations Build Number: x.x.x

Severity: HIGH Priority: HIGH

Status: New/Open/Active

Environment: Windows 2008

Steps To Reproduce:

1) Launch the application

2) Open menu File -> Open.

3) Click on ‘Browse’ button

Actual result: Application crash.

Expected result: On clicking Browse button file browse dialogue should be opened.

Notes: the screenshot and the logs are attached. 44

Good BUG

1. Short and precise title

2. Concise but complete description: Steps to reproduce, consequences of the bug, and suggested behavior

3. Good attachments

4. Complete definition of the categorizing fields

5. Correct Severity & Priority

6. Follow-up and comment

45

Good BUG

7. Include the Results you Expected

8. Include the Results you Actually Observed

9. Explain the Impact on the Customer

10. Avoid speculation

11. Be careful of the tone of your report

12. Avoid duplication - search first (!)

46

Collaboration

• QA engineers and developers are not in fight but are partners

• The main thing is product and its quality

47

BUG Tracking Systems

• Gemini• HP ALM (QC)• IBM Rational Quality Manager • JIRA• Seapine TestTrack• Bugzilla• Google Docs !• …

48

More details

49

• Software testing Help: http://www.softwaretestinghelp.com/

• Про Тестинг: http://www.protesting.ru/• http://software-testing.ru• What is severity and priority in software testing?

http://istqbexamcertification.com/what-is-severity-and-priority-in-software-testing/

• IEEE Std 1044-1993 IEEE Standard Classification for Software Anomalies http://standards.ieee.org/findstds/standard/1044-1993.html

• ISTQB® Glossary http://www.istqb.org/downloads/viewcategory/20.html

More details

50

• Сэм Канер, Джек Фолк Тестирование программного обеспечения

• Роман Савин Тестирование Дот Ком, или Пособие по жестокому обращению с багами в интернет-стартапах

• Луиза Тамре Введение в тестирование программного обеспечения

More details

51

• Introducing BDD by Dan North http://dannorth.net/introducing-bdd/

• Портал об автоматизации тестирования ПО: http://automated-testing.info/

• Конференции sqadays: http://www.sqadays.com

• Software testing Help: http://www.softwaretestinghelp.com/

• Про Тестинг: http://www.protesting.ru/

• http://software-testing.ru

More details

52

• Э.Дастин, Дж. Рэшка, Дж.Пол Автоматизированное тестирование программного обеспечения. Внедрение, управление и эксплуатация

• Matt Wynne and Aslak Hellesoy The Cucumber Book: Behaviour-Driven Development for Testers and Developers

• Криспин, Грегори Гибкое тестирование

Q & A

53

The End

• Thank you!

54