<Insert Picture Here>
UI Test Automation EffectivenessAlexandre (Shura) IlineJava SE and JavaFX Quality architect.
The following is intended to outline our general product direction. It is intended for information purposes, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.
This presentation shares experience gotten from testing such products as
JavaFX Authoring Tool
What's effectiveness
• Regular functional testing• Saving human time
• Special groups of tests(Things that could only be done through automated testing)
• Doing it with less human time
UI testing … by Wikipedia
«GUI software testing is the process of testing a product that uses a graphical user interface, to ensure it meets its written specifications.»
UI testing … most often ...
«Checking whether usage of a product UI leads to results expected by the the person who performs testing»
●Start text editor●Push «File/Open»●Verify file chooser directory●Select some file●Verify editor area content●Verify application title●Verify buttons availabilities●....
Sample test scenario
Test automation ... for me
«Building a process which exercises and reports certain product characteristics while run unattended.»
Test automation is like development
• Putting logic in a code• Same lifecycle:
• Requirements• Design• Implementation
• Same set of problems• Bugs • Instabilities
• Scope • defined through test specification/plan
Test automation is not like development
• Big fat dependency – the tested product• vs many libraries and platform
• Many small programs• vs one big program
• Does one thing good – reports status• vs does many thing ... good
• Perfectness is not the goal• other than maintenance cost, ease of use
UI Test automation
• Avoidable• Testing could be done with no automation
• Less obvious• Many misconceptions
• More expensive• Requires specific tools
• Such as the test drivers
Misconceptions
• Automated testing replaces manual testing• it will find all the bugs• it will find more bugs• it does the same amount of work
• Create once use forever• This is easy• This is too hard• No need to create test plan• “Will just buy the XXX tool and it will solve all out
ptoblems”
UI changes
• “Improvements”• Specification/requirement changes• Bug fixes
If UI is not changing ... Why bother testing?
Test base characteristics
• Coverage• How well the test base tests the product?
• Initial creation cost• How expensive it is to initially implement it?
• Sustainability• How expensive it is to support it?
• Reliability• Does it have intermittent failures?
Coverage
• Implementation coverage• Line, block, condition, sequence, ...
• Specification coverage• How well the tests covering the functional specification
• Public API coverage• Whether the public API is tested fully
• UI coverage• Whether the tests cover UI fully
• Combine with bug density• First cover the areas with more bugs
• Combine with code complexity• First cover the more complicated code.
Initial creation
• Record && replay• Inexpensive in creation• Lacks functionality• (More) expensive to support• Disables effective UI test automation
• Coding• Expensive to create• Require experienced engineers• Encourages effective UI testing• (Much) Less expensive to support
Tests code needed to be changed
• Test code needs to be updated every time UI changes
colorComboBox.selectColor(“Grey”)
turns into colorButton.push();...colorChooser.selectColor(Color.GREY);
18
18
Application UI
Product UIProduct UI
19
19
Coordinates
• click(134,32) //selects some record• click(215,122) //hits “Properties”
• sleep(5) //sleeps to let dialog be painted
• click(64,182) //expands color combo• click(235,182) //selects Gray• click(235,212) //hit OK
20
20
Widgets
• Find “Car records” frame• Find table
• Select “1abc234” cell
• Push “Properties” button• Wait for “1abc234” dialog• Select “Gray” color in combo box• Push “OK”
21
21
Prod
uct U
IPr
oduc
t UI
Dom
ain
mod
el
Dom
ain
mod
el
Car record
ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN
Test
Widgets or coordinates
22
22
UI Primitives
• Find car list frame
CarListFrame list = new CarListFrame()• Open properties dialog for car “1abc234”
CarDialog propDialog = list.carProperties(“1abc234”);
• Set color to gray
propDialog.setColor(Color.GRAY);• Apply changes
propDialog.ok();
23
23
LibraryPr
oduc
t UI
Prod
uct U
ID
omai
nm
odel
Dom
ain
mod
el
Car record
Test libraryTest library
ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN
CarListFrame CarDialog
Test
24
24
Domain model
• Set color to gray for a car “1abc234”
new CarRecord(“1abc234”).
setColor(Color.GRAY);
Underneath the cover, CarRecord class does all described earlier
25
25
Domain libraryPr
oduc
tU
IProd
uct
UI
Dom
ain
mod
el
Dom
ain
mod
el Car record
Domain test libraryDomain test library
ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN
UI test libraryUI test libraryCarList CarDialog
CarRecord
Test
The formula?
TD + *T
SN
R
TM * N
RN
C*EA =
NC
*
EA – automation effectiveness
To be used for every particular product.
NR and N
C are unique for a product.
TM is a characteristic of a test suite.
Smaller TD and T
S - higher the E
A.
Coefficient depend on the way you write your tests
Coordinates Widgets UI Library Domain library0
1
2
3
4
5
6
7
8
1.1
3
5
7.5
1
0.50.1 0.05
Td/TmTs/Tm
Td and T
s together
Coordinates Widgets UI Library Domain library0
5
10
15
20
25
30
1.1
3
5
7.5
24
12
2.41.2
25.1
15
7.48.7
TdTsTd+(Ts*Nc*Nr)
TD and T
S for N
C=3, N
R=8, T
M=1
Coordinates Widgets UI Library Domain library0
0.5
1
1.5
2
2.5
3
3.5
0.96
1.6
3.24
2.76
Ea
EA for N
C=3, N
R=8
Other Ts improvements
• Store resources in a property file• Take from the application itself
Testbase reliability
• Select right tool• No sleeps – only waitings• Use event queue• Right lookup criteria
• By ID• By text• By location• Not by index
32
32
Special test types
• Continuous build tests• aka sanity aka acceptance aka smoke
• Pre-integration
• Performance
Development
Continuous build
Commit
BuildExecuted
automaticallyafter commit
Success
Analysis.Rollback!!!
Code changes
NoYes.
PromoteCode is compilableTest further
Continuous build with testing
Commit
Build Success
TestingIs it working?
Passed
Analysis.
Rollback!!!
Code changes
Testchanges
No
No
Test furtherBuild is good
Code line is healthyGo on ...
Yes
Yes. = Compilation
successful
Pre-integration
Commit
Code changes
Testchanges
Testing Passed
No
Yes
How to benefit from UI automation?Use it!
• Sanity• Pre-integration• Attach to bug reports• Make is a quality criteria• Run it for every build• Show it to the boss :)• Don't forget to show it to the dev. team
<Insert Picture Here>
Alexandre (Shura) IlineJava and JavaFX Quality [email protected]
SQE reporting infrastructure improvements