Date post: | 15-Apr-2017 |
Category: |
Software |
Upload: | smartbear-software |
View: | 201 times |
Download: | 1 times |
How to Make Your UI Tests
Stable, Scalable, and
Maintainable
2
SmartBear’s technology stack
3
Quick Intro
Nikhil Kaul Jeffrey Martin
4
On our plate today
1
Why you shouldn't tolerate flakiness?
2
Common reasons for flaky UI tests
4
What to do with flaky UI tests?
5
How to separate signal from noise?
3
How to overcome these challenges?
5
Flaky tests shouldn't be tolerated
• 300 tests
• Each test has .5% failure rate
• Pass rate for each tests: 99.5%
• Pass rate for the suite: (99.5%)^300 = 22.23%
6
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
7
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
8
Don’t interact with UI elements directly
Test services seperately from the UI
Use page objects
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use UI encapsulation Build modular test design
‒ Identify user journeys
‒ Unit
‒ Service
‒ UI
Automate at three levels Use UI encapsulation
‒ Page Objects
UI Service Unit
9
How to implement in real world
Add Item
Click on an item with name XYZ Clickitem()
Set element with name ABC to quantity Z Set element()
Click on button labeled place in cart clickbutton()
What? How?
10
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
11
Separate the test data from test design
Test services seperately from the UI
Use page objects
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use page objects Build modular test design
– Test specific data
– Reference data
Limit data
dependency
between tests Tests should run
independently
Using databases
‒ Conduct tests
inside a transaction
12
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
13
Limit your dependency on environments
Test services seperately from the UI
Use page objects
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use page objects Build modular test design
‒ Application API
‒ HTML API
‒ Identify user journeys
Limit calls to external systems
Parameterize connections
Hermetic Servers
14
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
15
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use page objects
Build modular test design ‒ Application API
‒ HTML API
Limit calls to external systems
Parameterize connections
Hermetic Servers
Technical complexities can be overcome
AJAX Calls
‒ Callback
‒ Polling Interval
‒ WaitUntil
No Response Calls
‒ Fake objects
‒ Mock
‒ Virtual Service
16
Acceptance test builds should never be red
Separate flaky
from healthy tests
Limit the number of tests as well as the time
17
Use statistics for measuring test stability
Sample Size (N)
• 50
Desired confidence (α)
• 95% confidence
P
• 29/50 = .58
1-p
• 21/50
𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙 𝐶𝐼 = 𝑝 ± z1−α 2
𝑝(1 − p)
n
𝐶𝐼 = .58 ± 1.96.58(1 − .58)
50
𝐶𝐼 = .58 ± .14
𝐶𝐼 = between 44% and 72%.
±z1−α 2 = . ±z.95 = 1.96
18
Takeaways
1
Flaky tests shouldn't be tolerated
2
Understand the nature of flakiness
4
Use statistics for measuring test stability
3
Acceptance test builds should never be red
Questions?