Date post: | 01-Jun-2015 |
Category: |
Technology |
Upload: | spb-sqa-group |
View: | 1,017 times |
Download: | 4 times |
<Insert Picture Here>
Automating JFC UI application testing with Jemmy.
Alexandre (Shura) IlineJava SE and JavaFX Quality architect.
The following is intended to outline our general product direction. It is intended for information purposes, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.
This presentation shares experience gotten from testing such products as
JRockit JavaFX AT
What's in scope
• All sorts of definitions “Testing”, “UI testing”, “Test automation”• Jemmy library• Automation approaches effectiveness and cost.• Bunch of source code
• Basic Jemmy operations• Less basic operations• Switches and settings
• Tricks and tips• Misconceptions
What's not in scope
• The economical effects• A.k.a. what's “quality”• Why to automate
• Automation cost vs. manual testing cost• When to automate• What to automate
• Non-economical benefits of automation
A.k.a. let's leave it for later. :)
UI testing … by Wikipedia
«GUI software testing is the process of testing a product that uses a graphical user interface, to ensure it meets its written specifications.»
UI testing … most often ...
«Checking whether usage of a product UI leads to results expected by the the person who performs testing»
●Start text editor●Push «File/Open»●Verify file chooser directory●Select some file●Verify editor area content●Verify application title●Verify buttons availabilities●....
Sample test scenario
Find Do VerifyPass Pass
Pass
Fail Fail Fail
Failure analysis
Find next controlTo perform operation
Verify that expectedState reached
Perform necessary actions
UI test workflow
Test automation ... for me
«Building a process which exercises and reports certain product characteristics while run unattended.»
Test automation is like development
• Putting logic in a code• Same lifecycle:
• Requirements• Design• Implementation
• Same set of problems• Bugs • Instabilities
• Scope • defined through test specification/plan
Test automation is not like development
• Big fat dependency – the tested product• vs many libraries and platform
• Many small programs• vs one big program
• Does one thing good – reports status• vs does many thing ... good
• Perfectness is not the goal• other than maintenance cost, ease of use
Jemmy
Jemmy history• Started as a tool to tests TeamWare UI (1999)• Used for NetBeans extensions (2000)• Official test tool for NetBeans (2001)• Open-source (2001)• Is used for NetBeans and extensions since then• Adopted as a test tool for Swing (~2003)• Used outside SUN Microsystems (next slide)• Jemmy v3 created (2008)• The test tool for JavaFX SDK (2008 – now)• Extended to support SWT• Test tool for JRMC
JemmyCore
JemmySGJemmy v2
AWT & Swing
JemmyRemote
JemmyFXRemoteJemmyFXJemmySWT
JRMCJavaFX Authoring Tool
JavaFX
LCDUINetBeans
JDK
Jemmy and extensions
Jemmy v3
JemmyAWT
Jemmy v2 JFC
Same VM.
Test code runs in the same VM as the application code• Benefits
• Full access to the application UI objects• As well as the application domain object (sometimes).• Control over event queue
• Drawbacks• Impacts the tested UI
• Options• Run application from test• Run test from application• Use accessibility hook
Using Jemmy
Demo
Operators
Test codeTest code
JFrameOperatorgetTitle(),
...
JFrameOperatorgetTitle(),
...
JComboBoxOperatorgetSelectedItem(),
selectItem(...)
JComboBoxOperatorgetSelectedItem(),
selectItem(...)
JTextFieldOperatorgetText(),
typeText(...), ...
JTextFieldOperatorgetText(),
typeText(...), ...
JButtonOperatorgetText(),
push()
JButtonOperatorgetText(),
push()
Operators
Demo
Threading
“noblocking”operation
periodicalchecks
waiting waitingaction action action
action
WaitingTimeouts
Demo
Event queue
User
UIOtherthreads
UIOtherthreads
Test
Actions
Verifications
Lookup
Event queuewith test
Event queue tools
Demo
Drivers
Test code
ButtonDriverpush(...)
...
TextDrivertypeText()
...WindowDriver
ListDriverslectItem(...)
...
JButtonOperator
JTextFieldOperator
JComboBoxOperator
JFrameOperator
Drivers
Demo
Dispatching modes
• Events• Proper events in a proper order are dispatched to a
component making it thinking some user actions are performed.
• Shortcut• Same as events except the events are posted into the event
queue at once. • Robot
• java.awt.Robot is used
Implemented with driver sets
Dispatching models
Demo
Image
• Fragile• Needed for custom controls
• Comparison• Pixel-to-pixel• %s of pixels• Color distance
• Average• Maximum
Verifying images
Demo
Automationapproaches
33
33
Application UI
Product UIProduct UI
34
34
Coordinates
• click(134,32) //selects some record• click(215,122) //hits “Properties”• sleep(5) //sleeps to let dialog be painted• click(64,182) //expands color combo• click(235,182) //selects Gray• click(235,212) //hit OK
35
35
Widgets
• Find “Car records” frame• Find table• Select “1abc234” cell• Push “Properties” button• Wait for “1abc234” dialog• Select “Gray” color in combo box• Push “OK”
36
36
Prod
uct U
IPr
oduc
t UI
Dom
ain
mod
el
Dom
ain
mod
el
Car record
ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN
Test
Widgets or coordinates
Cars testWidgets
Demo
38
38
UI Primitives
• Find car list frameCarListFrame list = new CarListFrame()
• Open properties dialog for car “1abc234”CarDialog propDialog = list.carProperties(“1abc234”);
• Set color to graypropDialog.setColor(Color.GRAY);
• Apply changespropDialog.ok();
39
39
LibraryPr
oduc
t UI
Prod
uct U
ID
omai
nm
odel
Dom
ain
mod
el
Car record
Test libraryTest library
ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN
CarListFrame CarDialog
Test
Cars testUI Library
Demo
41
41
Domain model
• Set color to gray for a car “1abc234”
new CarRecord(“1abc234”). setColor(Color.GRAY);
Underneath the cover, CarRecord class does all described earlier
42
42
Domain libraryPr
oduc
tU
IProd
uct
UI
Dom
ain
mod
el
Dom
ain
mod
el Car record
Domain test libraryDomain test library
ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN
UI test libraryUI test libraryCarList CarDialog
CarRecord
Test
Cars testDomain Library
Demo
Automation effectiveness
The formula?
TD + *T
S NR
TM * N
RN
C*E
A =N
C*
EA – automation effectiveness
To be used for every particular product.
NR and N
C are unique for a product.
TM is a characteristic of a test suite.
Smaller TD and T
S - higher the E
A.
Coefficient depend on the way you write your tests
Coordinates Widgets UI Library Domain library0
1
2
3
4
5
6
7
8
1.1
3
5
7.5
10.5
0.1 0.05
Td/TmTs/Tm
Td and T
s together
Coordinates Widgets UI Library Domain library0
5
10
15
20
25
30
1.13
5
7.5
24
12
2.41.2
25.1
15
7.48.7
TdTsTd+(Ts*Nc*Nr)
TD and T
S for N
C=3, N
R=8, T
M=1
Coordinates Widgets UI Library Domain library0
0.5
1
1.5
2
2.5
3
3.5
0.96
1.6
3.24
2.76
Ea
EA for N
C=3, N
R=8
Misconceptions
• Automated testing replaces manual testing• it will find all the bugs• it will find more bugs• it does the same amount of work
• Create once use forever• This is easy• This is too hard• No need to create test plan• “Will just buy the XXX tool and it will solve all out
problems”
Resources
jemmy.dev.java.net
<Insert Picture Here>
Automating JFC UI application testing.With Jemmy.The technical aspects. Alexandre (Shura) IlineJava SE and JavaFX Quality architect.
Backup slides
Coverage
• Implementation coverage• Line, block, condition, sequence, ...
• Specification coverage• How well the tests covering the functional specification
• Public API coverage• Whether the public API is tested fully
• UI coverage• Whether the tests cover UI fully
• Combine with bug density• First cover the areas with more bugs
• Combine with code complexity• First cover the more complicated code.
Development
Continuous build
Commit
BuildExecuted
automaticallyafter commit
Success
Analysis.Rollback!!!
Code changes
NoYes.
PromoteCode is compilableTest further
Continuous build with testing
Commit
Build Success
TestingIs it working?
Passed
Analysis.
Rollback!!!
Code changes
Testchanges
No
No
Test furtherBuild is good
Code line is healthyGo on ...
Yes
Yes. = Compilation
successful
Pre-integration
Commit
Code changes
Testchanges
Testing Passed
No
Yes
How to benefit from UI automation?Use it!
• Sanity• Pre-integration• Attach to bug reports• Make is a quality criteria• Run it for every build• Show it to the boss :)• Don't forget to show it to the dev. team