1
Wix AutomationAutomation Manager
Stas Slutsker
2
Introduction
3
What We HaveTeamCity (CI) as test executor
TeamCity (CI) as test executor
4
It Works, But…
5
It Works, But…• Controlling rapidly growing tests amount,
more & better info about them
• Dealing with flakiness in efficient way, technically• Reduce trust and increase annoyance• Poison. How to avoid them and progress
faster? -> Move fast and break nothing
6
TeamCity is NOT test management tool!
It Works, But…
7
So, What’s Missing?
8
• Analyzing tests per configuration, history & statistics for cross-section demands
• Pursue stability
• Risk analysis, better decision making
So, What’s Missing?
9
We need central management of tests for smarter automation
• Progress faster, reaching our release goals• Better development velocity• Quality of our automation service, as a team
Conclusion
10
Automation Manager
11
Automation Manager Goals
• Gathering information about test runs
• Statistics & history, per cross-section demands
• Disabling & approving new or suspected tests, controlling tests statesSmarter Automation
12
States & Stability
13
States & Stability• Each test has a state
• State is changed according to test behavior
• Stable state must be earned!Stable
Beta (new / flaky)
Disabled
14
Beta Tests
15
Statistics
16
Dashboards according to different needs
Statistics
17
Statistics – Ignored Tests
18
Statistics – Test Runs
19
Decision Making
20
Risk analysis & behavior patterns
• Skipping new or non-stable tests
• Discovering non-valuable tests
• Vulnerable product areas (error prune)
Better Decision Making
21
Project Structure
22
Project Structure
API Core Web DB
Client
REST API
Test Observer@ Before@ Failure@ Success@ Skip
GAE
23
Objects Modeling
Test
RunResult
RunResult
RunResult
24
Objects Modeling
RunConfiguration
- browser- experiments- sanity- environment- platform- configurationNa
me- numOfThreads
Test- date- name- state- appearanc
e
RunResult- testName- buildId- runConfigurati
on- passStatus
1
n
25
Thank you!Questions?