83
BUILD THE “RIGHT” REGRESSION SUITE USING BEHAVIOR DRIVEN TESTING (BDT) vodQA
| Date post: | 16-Jul-2015 |
| Category: |
Software |
| Upload: | anand-bagmar |
| View: | 252 times |
| Download: | 1 times |
BUILD THE “RIGHT” REGRESSION SUITE USING BEHAVIOR DRIVEN TESTING (BDT) vodQA
WHAT DO YOU EXPECT FROM THIS SESSION?
CASE STUDY
• Multiple, long running projects
• Legacy applications
• Integrated
• Limited / flaky / long-running Automation
• Long regression cycle
• Huge cost of fixing defects
FUNCTIONAL TESTING CHALLENGES
• Brittle
• Little / less value
• No visibility into what is tested
• Maintenance nightmare
• Outdated
• Expensive
AS A RESULT
• Trust deficit on the team • In existing Automation
• In manual testing
• Finger pointing / blame game
BIGGEST PROBLEM
My Case Study
Concept #1
Given -> When -> Then
Expected Functionality:
The customer should be able to withdraw money from his account via an ATM machine
BDD Example:
Given the account has balance of 5000
When the customer requests 1000
Then the account is debited by 1000
13
Workshop Begins – Iteration 1
Case Study
PHASE 1 REQUIREMENTS
• Guest User is able to search for a flight for a single traveller
• Select option from Search Results page
• Specify contact information for person booking the flight
A Guest User can …
WORKFLOW
Search Search Results
Contact Information
SEARCH MODULE
SEARCH RESULTS
CONTACT DETAILS
CONTACT DETAILS
PHASE 1 REQUIREMENTS
• Guest User is able to search for a flight for a single traveller
• Select option from Search Results page
• Specify contact information for person booking the flight
A Guest User can …
WORKFLOW
Search Search Results
Contact Information
LETS GET STARTED … WRITE 2 TEST SCENARIOS
Iteration #1
Showcase
Concept #2
Test Specification Styles
• Imperative
• Declarative
TEST SPECIFICATION STYLES
A Guest User can …
AN EXAMPLE
Search Search Results
Contact Information
IMPERATIVE STYLE
Given I am a guest user on the home page
And I choose “round” trip option
And I select “Chicago” from the origin dropdown
And I select “San Francisco” from the destination dropdown
And I select departure date as “5 December 2013”
And I select returning date as “25 December 2013”
When I click on Search
Then I should see the search results page
And I should see at least 1 option for my criteria
...
IMPERATIVE STYLE …
…
When I select the “first” option
Then I am on the Contact Information page
When I enter first name as “Anand”
And I enter last name as “Bagmar”
…
…
And I click the “Select and Continue” button
Then I should be on the next page
DECLARATIVE STYLE
Given I am a guest user
When I search for flight options for a “one-way” trip for “1” “Adult” from “Chicago” to “San Francisco”
And I select the “first” flight
And I enter “valid” contact details for “traveller1”
Then I am able to Save and Continue
• Ben Mabey - http://benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories.html
• Alister Scott - http://watirmelon.com/2010/12/10/cucumber-imperative-or-declarative-that-is-the-question/
REFERENCES
Workshop – Iteration 2
Case Study Continues
• User registration from Contact information page
• Registered user can login from home page (below search criteria)
• For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information
PHASE 2 REQUIREMENTS
Login Search Search Results
Contact Information
• Pre-filled
WORKFLOW … ENHANCED
REGISTERED USER
• User registration from Contact information page
• Registered user can login from home page (below search criteria)
• For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information
PHASE 2 REQUIREMENTS
Login Search Search Results
Contact Information
• Pre-filled
WORKFLOW … ENHANCED
LETS GET STARTED … WRITE 2 TEST SCENARIOS
Iteration #2
Showcase
WHY IS THIS IMPORTANT?
Concept #3
The Test Pyramid
AUTOMATION – A SAFETY NET
• “Succeeding with Agile” – Mike Cohn
• Martin Fowler – Test Pyramid
martinfowler.com/bliki/TestPyramid.html
THE TEST PYRAMID
Time Value Cost
IDEAL TEST AUTOMATION PYRAMID
Manual / Exploratory
UI
Web Service
View
JavaScript
Integration
Unit
Technology-facing Tests
Business-facing Tests
REALITY
UI
Web Service
View
JavaScript
Integration
Unit
Manual / Exploratory
ICE CREAM CONE ANTI PATTERN
Developers Test Pyramid QA Team Test Pyramid
Web Service
View
JavaScript
Integration
Unit
Manual / Exploratory
UI
Web Service
DUAL TEST PYRAMID ANTI-PATTERN
Concept #4
Behavior Driven Testing (BDT)
BEHAVIOR DRIVEN TESTING (BDT)
• How is this functionality going to be used by the end-user?
• What is their thought process going to be when using this product?
• What “core-business-value” should be delivered to the end-user?
THE BIG PICTURE
Manual / Exploratory
UI
Web Service
View
JavaScript
Integration
Unit
Technology-facing Tests
Business-facing Tests
Identify the “right” type of tests that sit on top of the Test Pyramid
My Case Study
HOW TO SOLVE THE PROBLEM?
• Remove the ambiguity
• Same visibility for all
SOLUTION
• Whiteboards
• Mind maps
• Flow charts
• Collaboration!!
TOOLS TO ENABLE BDT
HOW DID WE ACCOMPLISH THIS?
IDENTIFY BUSINESS FLOWS
• Identify business flows based on personas, and how the personas use the business functionality, or,
• Identify business flows, and how different user personas can use the business flows
NOTE: Business flows are different from UI flows
HOW TO IDENTIFY BUSINESS FLOWS ?
PRIORITIZE BUSINESS FLOWS
1st candidate • Automate! • High Value, Low Cost
2nd candidate • Automate! • High Value, High Cost
Can Automate? • Manual Regression? • Low Value, Low Cost
Do Not Automate • Manual Regression! • Low Value, High Cost
High Value
High Cost
Low Value
Low Cost
AUTOMATE THE TOP LAYER USING
• BDD Tools, or,
• Directly in any appropriate programming language / tools
• Identification of Tests
• Automate it!
WHAT’S NEXT?
AUTOMATE TESTS THAT ARE VALUABLE, NOT BECAUSE YOU CAN OR LIKE TO!
CAUTION!
Who is doing the action?
What is the business functionality?
Depicts a user flow / user journey
Actions & Verifications happen implicitly
Given auctioneer creates an auction And 5 vehicles are added to the auction When auctioneer starts the auction And auctioneer sets a starting bid of 5000 And buyer_1 bids And buyer_2 bids Then auctioneer Sells the vehicle And buyer_2 wins the vehicle
EXAMPLE
Delete profile
UPDATE BUSINESS FLOWS
1st candidate • Automate! • High Value, Low Cost
2nd candidate • Automate! • High Value, High Cost
Can Automate? • Manual Regression? • Low Value, Low Cost
Do Not Automate • Manual Regression! • Low Value, High Cost
High Value
High Cost
Low Value
Low Cost
Project Management tool Mingle
Test case repository (manual + automated)
Cucumber .feature files with appropriate tagging
Feature files / Scenarios 33 / 65
# of Manual Tests 0
Tests running on CI (Jenkins) Since the 1st test was automated
Smoke tests execution time 15 minutes
Regression tests execution time
45 minutes
FINAL STATE … IN MY CASE STUDY
• Builds and deployments were automated
• Tests ran on demand, automatically! • Smoke / Regression / Javascript / Unit
• Functional Performance
• Concurrent Testing
ADDITIONALLY …
How did this Technique (BDT) become successful?
CONTEXT IS KING!
Domain
Distributed Teams
Skills
Capabilities Product-under test
Tech Stack
Process
Tools
NOT work in isolation
BDT WILL …
BDT REQUIRES …
Lower layers of Test Pyramid to be well automated!
BDT REQUIRES …
Dedicated and Focused testing (manual / exploratory)
BDT helps identify the “right” type of regression tests!
Executable Specs
BDT Scenarios
Business Idea
Features
Epics
Stories
Iterations
Releases
BDT VALUE PROPOSITION
BDT VALUE PROPOSITION …
Incrementally build
Business Flows
Regress Business Value
Test in-sync with Product Functionality
Effective Tests
BDT VALUE PROPOSITION …
Remove ambiguity
Understand new functionality
Living Documentation
Onboarding
