Post on 02-Aug-2020
transcript
Susmit Bhattacharya, Solution Architect, Asia Pacific
Agile vs Fragile- The need for Automation in Agile
© 2017 Tricentis GmbH. All Rights Reserved.
© 2017 Tricentis GmbH. All Rights Reserved. 2
Critical Physical Limit
Ever-Growing Challenge
Technical Complexity
& Rate of Change
YearsMonths
MonthsWeeks
WeeksDays
DaysHours
HoursMinutes
Delivery
Cycle Time
Perceived DisruptionBeyond
Continuous
Testing
Today
© 2017 Tricentis GmbH. All Rights Reserved. 3
Requirements Design Implementation Testing Acceptance Deployment
Inception Elaboration Construction Transition
Development Operations
DevOps
Agile Event Horizon
Digital Singularity
IncrementalRational Unified Process
4
AgileScrum, Kanban
2
ContinuousDevOps
1
6+
Big BangWaterfall
Development Response = wall of confusion
Flexible Framework
Rigid Rules
Following a Plan
Contract Negotiation
Copious Documentation
Processes & Tools
Individuals & Interactions
Working Software
Customer Collaboration
Responding to Change
© 2017 Tricentis GmbH. All Rights Reserved. 4
Dev Ops
Continuous
Release & DeploymentCollaborative
Development
Continuous
Testing
Continuous
Monitoring
Continuous
Integration
Continuous
Feedback
DevOps Movement
DevOps is no longer a
unicorn’s discipline.
TestDev OpsShift Left Shift Right
of enterprises have
adopted or plan to adopt
81%
DevOps.
© 2017 Tricentis GmbH. All Rights Reserved. 5
Current State Target State
remaining manual tests(exploratory and manual spec-based)
automated UI tests
automated API tests
+
service virtualization(key enabler for high automation)
GUI
API
20-40%
60-80%Specification
driven
5-20%
80-95%
Testing Future
Budgets won’t increase
© 2017 Tricentis GmbH. All Rights Reserved. 6
remaining manual tests(exploratory and manual spec-based)
automated UI tests
automated API tests
+
service virtualization(key enabler for high automation)
GUI
API
Model-Based Test Automation (MBTA) for
UI and APIAutomating the automation framework.Automate
Session-Based Exploratory TestingMake your testing intellectually rich.
Explore
Orchestrated Service VirtualizationMake testing independent.
Stateful Test Data Management Manage
Computer-Aided Test Portfolio DesignRedundancy-free test case portfolio with
maximum risk coverage and fewer tests.Optimize
Integrate
CT Integration (CI, CD)
Distributed ExecutionTarget State
Testing Future
Current State
Budgets won’t increase
© 2017 Tricentis GmbH. All Rights Reserved. 7
remaining manual tests(exploratory and manual spec-based)
automated UI tests
automated API tests
+
service virtualization(key enabler for high automation)
GUI
API
Computer-Aided Test Portfolio DesignRedundancy-free test case portfolio with
maximum risk coverage and fewer tests.Optimize
Target State
Testing Future
Current State
Budgets won’t increase
© 2017 Tricentis GmbH. All Rights Reserved. 8
Low
Risk
High
Risk
Medium
Risk80%20%
Business RiskCoverage
TestCases
The time needed for
testing is infinitely larger
than the time available.
Critical Limit
Risk-Based Testing
Have the right test cases.
Do the right things.
© 2017 Tricentis GmbH. All Rights Reserved. 9
risk
co
vera
ge
60%
80%
100%
40%
20%
0%
𝑴
𝑵=
0
defe
ct r
ate
max
100%
test cases
20%0% 40% 60% 80%
critical defects
M N
𝟏 weight𝟏 𝟏 𝟏 𝟏 𝟏 𝟏… …[%]
Computer-Aided Test Portfolio Optimisation
© 2017 Tricentis GmbH. All Rights Reserved. 10
60%
80%
100%
40%
20%
0%
risk
co
vera
ge
100%20%0% 40% 60% 80%
test cases
critical defects
0
defe
ct r
ate
max
M N
𝒘𝟏 𝒘𝟐 𝒘𝟑 𝒘𝟒 𝒘𝒊 𝒘𝑵… … … … 𝒘𝒊 ≥ 𝒘𝒊+𝟏weight
Computer-Aided Test Portfolio Optimisation
© 2017 Tricentis GmbH. All Rights Reserved. 11
60%
80%
100%
40%
20%
0%
IntuitiveTest Design
MethodicalTest Design
100%20%0% 40% 60% 80%Test Cases
Risk
Coverage
Risk-Based Testing
Achieve maximum risk coverage with
a minimal number of powerful test cases.
© 2017 Tricentis GmbH. All Rights Reserved. 12
remaining manual tests(exploratory and manual spec-based)
automated UI tests
automated API tests
+
service virtualization(key enabler for high automation)
GUI
API
Model-Based Test Automation (MBTA) for
UI and APIAutomating the automation framework.Automate
Optimize
Target State
Testing Future
Current State
Budgets won’t increase
© 2017 Tricentis GmbH. All Rights Reserved. 13
2nd Generation
Test Automation
Framework
1st Generation
Record
Replay
10%Automation
Evolution of Automated Testing
© 2017 Tricentis GmbH. All Rights Reserved. 14
Example:
“…select invoice with the car registration number VIC-3456”
Great for developers who can code
...but is even harder to understand for the business team
Conclusion: Only for developer & Expensive to maintain
Keyword Driven Data Driven
Custom Hybrid
Functionally DecomposedCommercial
Script Based Test Automation Frameworks
© 2017 Tricentis GmbH. All Rights Reserved. 15
Script-Based
2nd Generation
Test Automation
Framework
1st Generation
Record
Replay
30%Automation
10%Automation
+ -Programming Required
- +Business Readable
Model-BasedTest Automation
Next Generation
NoScripting
90%Automation
+
Future of Automated Testing
© 2017 Tricentis GmbH. All Rights Reserved. 16
Technical Information
Application Pages
sampleapp.tricentis.com
Technical Name Value Range
-
-
1;2;3;4;5;6;7;8;9
Petrol;Diesel;Gas;Other
-
-
{CLICK};{RIGHTCLICK}
Private;Commercial
2016;2015;2014;2013
-
Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Insurant Data
Product Data
Next {CLICK}
Start Date 01/03/2016;01/04/2016
Insurance Sum [$] 3Mio;7Mio;10Mio;15Mio
Payment Option Yearly;Quarterly;Monthly
Next >>
Make Audi
Engine Performance [kW] 200
Number of Seats 5
Fuel Petrol
Year of Construction 2016
List Price [$] 35.000
License Plate Number B-CD 123
Annual Mileage [mi] 10.000
Usage Private
Scan Application…. Silver; Gold; Platinum; Ultimate
Quote Details
Price per Year($);Online Claim; ..
© 2017 Tricentis GmbH. All Rights Reserved. 17
Logical Name
Enter Insurant Data
Create Vehicle Insurance
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Value
Audi
200
5
Petrol
35.000
10.000
{CLICK}
Private
{Year}
B-CD 123
Action
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Enter Product Data
Next
Start Date
Insurance Sum [$]
Payment Option
{CLICK}
01/03/2016
7Mio
Yearly
Input
Input
Input
Input
Verify Quote Details
Price List
Price per Year($)
Silver 1.535,22 Verify
Select
Select
Business Information
Technical Name Value Range
Technical Information
-
-
1;2;3;4;5;6;7;8;9
Petrol;Diesel;Gas;Other
-
-
{CLICK};{RIGHTCLICK}
Private;Commercial
2015;2014;2013;2012
-
Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Insurant Data
Product Data
Next {CLICK}
Start Date 01/03/2016;01/04/2016
Insurance Sum [$] 3Mio;7Mio;10Mio;15Mio
Payment Option Yearly;Quarterly;Monthly
Quote Details
Silver; Gold; Platinum; Ultimate
Price per Year($);Online Claim; ..
It’s like playing Lego.
© 2017 Tricentis GmbH. All Rights Reserved. 18
100%
0%
Sprint
Start
Sprint
End
API
UI
Start testing early.
Shift left test automation.
API Testing
Degree of
Completion
Time
Efficiency
Gain
+4xCreation
+6xMaintenance
+20xExecution Future test automation must focus on API testing.
© 2017 Tricentis GmbH. All Rights Reserved. 19
Logical Name
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Next
Start Date
Insurance Sum [$]
Payment Option
Create Vehicle Insurance
Price List
Insurance Sum
Gross Premium [$]
Value
Audi
200
5
Petrol
35.000
10.000
{CLICK}
Private
{Year}
B-CD 123
{CLICK}
01/03/2016
7Mio
Yearly
1.535,22
Action
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
UI Test CaseBrowser
Audi
200
5
Petrol
35.000
10.000
Private
{Year}
B-CD 123
01/03/2016
7Mio
Yearly
1.535,22
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
API Test CaseWebservice
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Start Date
Insurance Sum [$]
Payment Option
Price List
Insurance Sum
Gross Premium [$]
© 2017 Tricentis GmbH. All Rights Reserved. 20
Logical Name
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Next
Start Date
Insurance Sum [$]
Payment Option
Create Vehicle Insurance
Price List
Insurance Sum
Gross Premium [$]
Value
Audi
200
5
Petrol
35.000
10.000
{CLICK}
Private
{Year}
B-CD 123
{CLICK}
01/03/2016
7Mio
Yearly
1.535,22
Action
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
UI Test CaseBrowser
Audi
200
5
Petrol
35.000
10.000
Private
{Year}
B-CD 123
01/03/2016
7Mio
Yearly
1.535,22
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
API Test CaseWebservice
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Start Date
Insurance Sum [$]
Payment Option
Price List
Insurance Sum
Gross Premium [$]
Manual Testers become Automation Specialists
© 2017 Tricentis GmbH. All Rights Reserved. 21TDD vs BDD.
Behaviour-Driven Development (BDD) is a software development process based on Test-Driven Development (TDD)
BDD combines techniques and principles of
• TDD• Domain-Driven Design• Object-Oriented Analysis & Design
BDD gives a clearer understanding as to what the system should do from the perspective of the developer and the customer.
TDD only gives an understanding of what the system should do from a developer perspective.
© 2017 Tricentis GmbH. All Rights Reserved. 22
Better collaboration between developerstest analysts and business analysts
Executable documentation of the software.
BDD/TDD – The promise.
© 2017 Tricentis GmbH. All Rights Reserved. 23
Open Source (Cucumber, JBehave, Specfolw with Selenium, Watir, etc.) • Requires coding! Each GIVEN, WHEN, THEN, AND in the BDD scenario (left) needs a matching function call (right).• No support for systematic scenario identification and definition
BDD – A Working Example.
© 2017 Tricentis GmbH. All Rights Reserved. 24BDD using TCD.
© 2017 Tricentis GmbH. All Rights Reserved. 25
remaining manual tests(exploratory and manual spec-based)
automated UI tests
automated API tests
+
service virtualization(key enabler for high automation)
GUI
API
Session-Based Exploratory TestingMake your testing intellectually rich.
Explore
Optimize
Target State
Testing Future
Current State
Budgets won’t increase
© 2017 Tricentis GmbH. All Rights Reserved. 26
Formal TestingInformal Testing
Exploratory TestingManual
Testing
Automated
Testing
Ad-hoc
Testing
Testing Continuum
Testing Continuum
© 2017 Tricentis GmbH. All Rights Reserved. 27
Risks
Automated Testing
I’m an automated test case.
Usability Issue
Stability Issue
Reliability IssueScalability Issue
Coherence Issue
Understandability Issue
Testability Issue
Convenience Issue
Security Issue
Accessibility Issue
Performance Issue
Functional Issue
Automation is doing, what automation does.
© 2017 Tricentis GmbH. All Rights Reserved. 28
1 2 3 4 5 … … … … … … …
Risks
Automated Testing
© 2017 Tricentis GmbH. All Rights Reserved. 29
I’m an automated test case.
Manual Testing
I’m the same test case executed manually.
© 2017 Tricentis GmbH. All Rights Reserved. 30
Plan as you test!
Story-Based
Motivating
Credible
Exploratory Testing
Exploratory BranchingThe art of deciding on what to test next.
NewTesting
Idea Exploratory TestingPerfect counterpart to specification-based
automated & manual testing.
Test Cases vs. ScenariosPre-specified inputs vs. hypothetical situation
© 2017 Tricentis GmbH. All Rights Reserved. 31
Specification
Based Testing
Exploratory Testing
Main Purpose
Analyze Potential RisksProblem vs. No Problem
Monitor Known RisksPass Result vs. Fail Result
*That’s an illustration. Don’t confuse it with reality.
Mechanical Testing vs. Intelligent Testing≫ Testing as artifact creation vs. testing as performance ≪
© 2017 Tricentis GmbH. All Rights Reserved. 32
Specification
Based Testing
Exploratory Testing
Analyze Potential RisksProblem vs. No Problem
Monitor Known RisksPass Result vs. Fail Result
Agile Testing Law
Checked≫ Efficient Automated Testing ≪
Explored≫ Effective Exploratory Testing ≪
Tested≫ Productive Testing ≪
+
=
𝑅𝜇𝜈 −1
2𝑅𝑔𝜇𝜈 + Λ𝑔𝜇𝜈 =
8𝜋𝐺
𝑐4 𝑇𝜇𝜈
Agile Equation Of Motion
© 2017 Tricentis GmbH. All Rights Reserved. 33Core Conclusion
Automation is a must.
Manual testing is unjustifiable.
Exploratory testing is a necessity.
*We don’t need humans doing something that a machine can do. What we do want the human testers doing is exploratory testing.
We are Passionate about Testing„ „