Automating JFC UI application testing with Jemmy

transcript

Automating JFC UI application testing with Jemmy.

Alexandre (Shura) IlineJava SE and JavaFX Quality architect.

The following is intended to outline our general product direction. It is intended for information purposes, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.

This presentation shares experience gotten from testing such products as

JRockit JavaFX AT

What's in scope

• All sorts of definitions “Testing”, “UI testing”, “Test automation”• Jemmy library• Automation approaches effectiveness and cost.• Bunch of source code

• Basic Jemmy operations• Less basic operations• Switches and settings

• Tricks and tips• Misconceptions

What's not in scope

• The economical effects• A.k.a. what's “quality”• Why to automate

• Automation cost vs. manual testing cost• When to automate• What to automate

• Non-economical benefits of automation

A.k.a. let's leave it for later. :)

UI testing … by Wikipedia

«GUI software testing is the process of testing a product that uses a graphical user interface, to ensure it meets its written specifications.»

UI testing … most often ...

«Checking whether usage of a product UI leads to results expected by the the person who performs testing»

●Start text editor●Push «File/Open»●Verify file chooser directory●Select some file●Verify editor area content●Verify application title●Verify buttons availabilities●....

Sample test scenario

Find Do VerifyPass Pass

Fail Fail Fail

Failure analysis

Find next controlTo perform operation

Verify that expectedState reached

Perform necessary actions

UI test workflow

Test automation ... for me

«Building a process which exercises and reports certain product characteristics while run unattended.»

Test automation is like development

• Putting logic in a code• Same lifecycle:

• Requirements• Design• Implementation

• Same set of problems• Bugs • Instabilities

• Scope • defined through test specification/plan

Test automation is not like development

• Big fat dependency – the tested product• vs many libraries and platform

• Many small programs• vs one big program

• Does one thing good – reports status• vs does many thing ... good

• Perfectness is not the goal• other than maintenance cost, ease of use

Jemmy history• Started as a tool to tests TeamWare UI (1999)• Used for NetBeans extensions (2000)• Official test tool for NetBeans (2001)• Open-source (2001)• Is used for NetBeans and extensions since then• Adopted as a test tool for Swing (~2003)• Used outside SUN Microsystems (next slide)• Jemmy v3 created (2008)• The test tool for JavaFX SDK (2008 – now)• Extended to support SWT• Test tool for JRMC

JemmyCore

JemmySGJemmy v2

AWT & Swing

JemmyRemote

JemmyFXRemoteJemmyFXJemmySWT

JRMCJavaFX Authoring Tool

JavaFX

LCDUINetBeans

Jemmy and extensions

Jemmy v3

JemmyAWT

Jemmy v2 JFC

Same VM.

Test code runs in the same VM as the application code• Benefits

• Full access to the application UI objects• As well as the application domain object (sometimes).• Control over event queue

• Drawbacks• Impacts the tested UI

• Options• Run application from test• Run test from application• Use accessibility hook

Using Jemmy

Operators

Test codeTest code

JFrameOperatorgetTitle(),

JComboBoxOperatorgetSelectedItem(),

selectItem(...)

JComboBoxOperatorgetSelectedItem(),

selectItem(...)

JTextFieldOperatorgetText(),

typeText(...), ...

JTextFieldOperatorgetText(),

typeText(...), ...

JButtonOperatorgetText(),

push()

JButtonOperatorgetText(),

push()

Operators

Threading

“noblocking”operation

periodicalchecks

waiting waitingaction action action

action

WaitingTimeouts

Event queue

UIOtherthreads

Actions

Verifications

Lookup

Event queuewith test

Event queue tools

Drivers

Test code

ButtonDriverpush(...)

TextDrivertypeText()

...WindowDriver

ListDriverslectItem(...)

JButtonOperator

JTextFieldOperator

JComboBoxOperator

JFrameOperator

Drivers

Dispatching modes

• Events• Proper events in a proper order are dispatched to a

component making it thinking some user actions are performed.

• Shortcut• Same as events except the events are posted into the event

queue at once. • Robot

• java.awt.Robot is used

Implemented with driver sets

Dispatching models

• Fragile• Needed for custom controls

• Comparison• Pixel-to-pixel• %s of pixels• Color distance

• Average• Maximum

Verifying images

Automationapproaches

Application UI

Product UIProduct UI

Coordinates

• click(134,32) //selects some record• click(215,122) //hits “Properties”• sleep(5) //sleeps to let dialog be painted• click(64,182) //expands color combo• click(235,182) //selects Gray• click(235,212) //hit OK

Widgets

• Find “Car records” frame• Find table• Select “1abc234” cell• Push “Properties” button• Wait for “1abc234” dialog• Select “Gray” color in combo box• Push “OK”

Car record

ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN

Widgets or coordinates

Cars testWidgets

UI Primitives

• Find car list frameCarListFrame list = new CarListFrame()

• Open properties dialog for car “1abc234”CarDialog propDialog = list.carProperties(“1abc234”);

• Set color to graypropDialog.setColor(Color.GRAY);

• Apply changespropDialog.ok();

LibraryPr

Car record

Test libraryTest library

CarListFrame CarDialog

Cars testUI Library

Domain model

• Set color to gray for a car “1abc234”

new CarRecord(“1abc234”). setColor(Color.GRAY);

Underneath the cover, CarRecord class does all described earlier

Domain libraryPr

el Car record

Domain test libraryDomain test library

UI test libraryUI test libraryCarList CarDialog

CarRecord

Cars testDomain Library

Automation effectiveness

The formula?

TD + *T

TM * N

EA – automation effectiveness

To be used for every particular product.

NR and N

C are unique for a product.

TM is a characteristic of a test suite.

Smaller TD and T

S - higher the E

Coefficient depend on the way you write your tests

Coordinates Widgets UI Library Domain library0

0.1 0.05

Td/TmTs/Tm

Td and T

s together

2.41.2

7.48.7

TdTsTd+(Ts*Nc*Nr)

TD and T

S for N

C=3, N

R=8, T

EA for N

C=3, N

Misconceptions

• Automated testing replaces manual testing• it will find all the bugs• it will find more bugs• it does the same amount of work

• Create once use forever• This is easy• This is too hard• No need to create test plan• “Will just buy the XXX tool and it will solve all out

problems”

Resources

jemmy.dev.java.net

Automating JFC UI application testing.With Jemmy.The technical aspects. Alexandre (Shura) IlineJava SE and JavaFX Quality architect.

Backup slides

Coverage

• Implementation coverage• Line, block, condition, sequence, ...

• Specification coverage• How well the tests covering the functional specification

• Public API coverage• Whether the public API is tested fully

• UI coverage• Whether the tests cover UI fully

• Combine with bug density• First cover the areas with more bugs

• Combine with code complexity• First cover the more complicated code.

Development

Continuous build

Commit

BuildExecuted

automaticallyafter commit

Success

Analysis.Rollback!!!

Code changes

NoYes.

PromoteCode is compilableTest further

Continuous build with testing

Commit

Build Success

TestingIs it working?

Passed

Analysis.

Rollback!!!

Code changes

Testchanges

Test furtherBuild is good

Code line is healthyGo on ...

Yes. = Compilation

successful

Pre-integration

Commit

Code changes

Testchanges

Testing Passed

How to benefit from UI automation?Use it!

• Sanity• Pre-integration• Attach to bug reports• Make is a quality criteria• Run it for every build• Show it to the boss :)• Don't forget to show it to the dev. team

Automating JFC UI application testing with Jemmy

Technology