Comparison of the Features of GUI Testing Tools

8/8/2019 Comparison of the Features of GUI Testing Tools

1/13

Abstract- Testing software manual ly is alabour intensiveprocess. Efficient automated testingsignificantly reduces theoverall cost ofsoftware development and maintenance. GUI test

automation is a major challenge for test automation. Differentkinds of automated tools are available in market for various types

of GUI application testing.This paper performs an analysis ofvarious GUI tools based on some specific features so asto

enable the tester to choose an appropriatetool to satisfy hisrequirements and toperform efficient testing of GUI

applications. And also this paper proposes GUI Automation testingtechnique to test GUI-Based java programs as an alternative to theCR technique. This technique develops GUI-event test specification

language for GUI application written using java swing APIs, whichinitiates an automated test engine. Visual editor helps in viewing thetest runs. The test engine generates GUI events and captures event

responses to automatically verify the results of the test cases. This

includes the test case generation, test case execution and test caseverification modules. The testing efficiency is measured bydetermining coverage metric based on Code coverage, while may be

useful during Regression Testing. This paper uses Abbot and JUnittools for test case generation and execution and Clover tool for code

coverage. We have performed tests on various GUI applications andthe efficiency of this technique is provided.

KeywordsAbbot, Capture Reply, Code Coverage, GUI Testing,Unit Testing

I. INTRODUCTION

Graphical User Interfaces (GUIs) have become an

important and accepted way of interacting with todayssoftware. There are different types of GUI testing [4]

v i z . Manual testing, Static Analysis, Usability testing and

Functional testing. The manual testing is userverification of

requirements using some screenshots and

sample inputs and outputs. The static testing is a type of

unconventional testing i.e. testing performed on an

application when it is not executed. It is also called

Verification orin process reviewing. Usability testingsatisfies userrequirements. It captures the user actions and

gets the data. Functional testing satisfiesthe functional

requirements which generally have a separate list of

requirements for GUIs from which test c a s e s a r e

d e v e l o p e d . The t e s t c a s e s a r eexecuted and theoutput is compared with expected results. It is dividedinto

two states namely verifier and level oftesting. Verifier is

used for verification of theactual output with manual

output. The level oftesting includes the functional testing

such as; Changed property set (Operators), GUI relevant

property set (Specification)andComplete property set

(Toolkit/Language). The most reliable way to ensure thata GUI worksas expected is through functional testing.

Unlike unit tests, which focus on testing classes in isolation,

functional tests confirm whether the systembehaves asexpected by t h euser. GUI tests should be able tosimulate

user input to verify that the applicationwill behave as

expected in front of the user [1],[15].GUI tests shouldhave a reliable way to find GUI components. GUI tests need

to tolerate changes in the position, size and layout of anyGUI component. Changes in the appearance ofan

application must not break any GUI tests. Writing GUI

test programs is very tedious (theyare always

lengthy and difficult to read).Thus, CR (Capture and

replay) scripttechnique becomes an attractive

technique. With a CR tool, a testdesigner interacts

with the GUI ofthe Application Under Test (AUT)

and all the events are recorded in a test script. Several

approaches are available for preparing test scripts. Tools

are mainly based on Automation Testingapproaches.

Some of the GUI Testing approaches are manual testing,

and Static analysis and functional testing.

In o r de r t o p e rf o rm f u n c t i o n a l t e s t i n g

different kinds of automated tools [3] are available in

market. It will be quite hard for the tester tochoose an

appropriate tool for satisfying his requirements. Inrunning automatic tests for the GUI application, the tester

saves much time, especially when he is in a huge

production house and needs to be multi-tasking This

paperis organized as follows: Section 2 describes the

different approaches on GUI Testing, includinga

description of the importance ofAutomation testing.

Section 3 presents the categories of Au to ma ti on

Testing. Section 4 presents the di scu ss io n and

analysis of various GUI Testing tools. Section 5

presents the analysis of selected tools for GUI testing.

Section 6 discusses why Abbot tool is the best option

for GUI appl icati on testi ng. Section 7 presents the

implementation of the Abbot tool through which testcase generation, execution, verification and code

coverage are done. Finally, Section 8 gives the

conclusion and future enhancement.

II. APPROACHESIN TESTINGGUIS

Testing a GUI application s eems to beeasy at

first. You dont have to write any code, you just have

to click and see if it works. This native approach works

only for GUIs oflimited complexity. Even quite simple

software has surprisingly complex GUIs and the amount

ofwork required to test the software will grow fast. It

is also quite difficult to envision all waysin which a

GUI can break, and sooner or laterthe testers will grow

so accustomed to testingthe same GUI that they

will start to miss opportunities to make it fail.

A. Automation

Multiple a v e n u e s o f ap p r o a c h h a v ebeen

used in the attempt to conquer the testautomation

problem. Since maintaining GUI tests areexpensive,

som e have tried to automatethe generation of tests.

Other approach to the same problem is to model the test

cases so that they can be automatically repaired when

changes are done to the User Interface (UI)[6]. Both these

solutions are usually based on the modelling of the UI asa set of events and states, which are then combined into

components, which can betested separately. The

common feature is that theseevents sometimes have to

Comparison of the Features of GUI Testing ToolsA.Askarunisa, D.Thangamari

[email protected], [email protected]


2/13

be tested in sequence like cut-paste. Generally in auto

generated tests the event sequences could growto infinite

length. Some limit has to be imposedand according to

studies there is no point in testing very long event

sequences except in special cases. From studies [5 ],[8],[11]

it is clear that event coverage (each event tested at least

once) guarantees very high statement coverage evenat

sequence length of 1. Lengthening the sequence makes the

statement coverage approach 100%.

B. Why write automatedtests?

Automated tests m a k e s u r e t h a t c o d eworks

(specification) and shows the intended use and the behaviour

of the code. Through this things do not get wrong

(integration) and mistakes do not reappear (regression) i .e .

They verify bugfixes. Most of the automated tests detect

unused code (code coverage) and delete obsolete codeor

improve tests. Automated tests generallymeasure the UI

responsiveness (performance). This is the only way to check

the daily performance.

III. CATEGORIESOF AUTOMATION TESTING

There are excess of tools available in the market to test

GUI applications. These tools can be categorized inthe

following ways viz.

A. Random Testing

Random testing is also called adhoc testing. In this

technique, the tester can give the out ofcoverage data and

check the stability ofthe application whether the system

performs smoothly or not. Random testing is blackbox

technique which requires no partitioning ofthe

component's input domain but simplyrequires test inputs

to be chosen from input domain at random. It is good forfinding when systemcrashes. No effort in generating test

cases. The updates of GUI are independent. Itincreases

confidence on the software when running several hours

without finding errors. It is Easy to implement. It is not

good in finding other kindsoferrors. It is difficult to

reproduce the errors. The output is unpredictable for some

test cases.

B Unit TestingFrameworks

Unit test frameworks [18] are akey element of

popular development methodologies such as extreme

Programming (XP) and Agile Development. But unit testing

has movedfarbeyond extreme Programming; it is nowcommon in many different types o f A p p l i c a t i o

development. Unit tests help ensure low-level code

correctness, reduce software development cycle time,

improve developer productivity, andproduce more robust

software. It is very flexible. It has an automatic test

execution. It supportsTDD (Test Driven Development). The

GUI test libraries can be used for GUI testing (essentially

reduce GUI to API testing).The test cases are programmed

manually, but usually has more lines of testing code than

application code.

C. Model based Testing

A model is usually an abstract, partial presentation of thesystem under test's desired behavior. The test cases derived

from this model are functional tests on the same level of

abstraction as the model. These test cases are collectively

known as the abstract test suite. The abstract test suite

cannot be directly executed against the system under test

because it is on the wrong level of abstraction. Therefore

an executable test suite must be derived from the abstract

test suite that can communicate with the system under test.

This is done by mapping the abstract test cases to concrete

test cases that are suitable for execution. In the case of

online testing the abstract test suite exists only as a

concept but not as an explicit artifact. Model testing canbe easily adapted to changes. It requires a forma

specification/model. Small changes to the model can

result in a totally different test suite.

D. Capture and Replay (and script)

Most common testing tools use record playback /

Capture replay techniques [4]. A test designer interacts

with the GUI, generating mouse and keyboard events. The

tool records the user events, captures the GUI session

screens, and then stores the session usually as a script. The

tester later plays back the recorded sessions to recreate the

events with different inputs. An automatic test case

generator can provide a higher level of support, but aprogrammer must code it for all possible decision points

in the GUI. Automated or not, the time consuming record

playback approach easily misses important GUI

decisions.GUI testing requires a clear methodology that

employs tools and techniques integrated to use a

standardized GUI representation. This integration ensures

that all test results are compatible with each other.

Programmatic technique can efficiently reduce the

memory occupied. The drawback of this technique is that

the tester needs to develop his own code for testing which

is a time consuming process. The Testing would be more

efficient if it combines the features of capture/replay and

programmatic techniques. This combination of tool is thebest choice for all GUI applications.

IV. SURVEYOF VARIOUS GUI TESTING TOOLS

We have collected nearly ten different GUI tools[13]

that mainly come under the category of the Capture-

Replay technique and freely downloadable. The details of

the tools are listed below:

Abbot [5]: Abbot is a framework of Java library that

provides methods to reproduce user actions and examine

the state of GUI components. It is an improvement upon

the very rudimentary functions provided by the

java.awt.Robot class (A Better 'Bot'). The framework canbe invoked directly from Java code or accessed without

programming through the use of scripts. It is suitable for

use both by developers for unit tests and Qualit

Assurance for functional testing.

Jacareto [9]: Jacareto is a capture and replay framework

for Java applications and applets. It is very flexible. There

are two front ends (Clever PHL is a GUI front end,

Picorder is a command line front end) included in it.

Jemmy [2]: Jemmy [2] is a Java TM library that is used to

create automated tests for Java GUI applications. It

contains methods to reproduce all user actions which can

perform on swing and AWT components (i.e. buttonpushing, text typing, and tree node expanding).Jemmy

Test is a program written in Java which uses the Jemmy

API to test GUI applications.


3/13

JfcUnit [7]: JfcUnit enables developers to write test cases for

java Swing based applications. JfcUnit provides XML

recording and playback. This allows users to quickly and

automatically generate/edit scripts to drive the testing. The

XML API's are open and allow developers to define their own

XML tag handlers.

Marathon [10]: Marathon is a testing framework for GUI

applications developed using Java/Swing. Marathon composesof recorder, runner and editor. The test scripts are composed

of python code. Marathon focuses on end-user testing. One

need not know Java or Swing to record test scripts using

Marathon. It is kept intentionally simple, flexible and provides

facilities to extend the functionality available through python

scripts or through writing ones own component revolvers.

QF-Test [14]: QF-Test is a highly flexible professional cross-

platform tool for Java and Web GUI test automation (Swing,

Eclipse / SWT+RCP, Web start, Applets, ULC).Supports

regression and load testing tolerant to GUI changes and can

handle even complex custom GUI objects. Modularization

allows creating reusable, robust tests that really pay off.

Q1 [17]:Q1 is a tool for developing GUI and HTML

automated tests. It was designed to provide great control

while maintaining low cost to make it affordable to small

companies and individual developers.

Test Agent [20]: Test Agent focuses on making User

Acceptance Testing more efficient and it enables

developers and QA to meet deadlines without

compromising quality. Some of the key features include

capturing and recording all data on the applications'

pages, automatically detecting and capturing standard and

custom content errors.

GUI dancer[19]: GUI dancer uses a sophisticated, yeteasy-to-use specification method which eliminates all

programming from test creation. A variety of powerful

features allow the extensive reuse of editable test

elements, resulting in tests which are quick to create,

highly adaptable, and which require only minimal

maintenance.

Squish [16]: Squish is the cross-platform, automated GUI

test and web testing tool. It supports testing applications

based on the following GUI technologies: Trolltech Qt

and Qtopia, Web/HTML/Ajax in IE, Firefox/Mozilla,

Safari and Konqueror, Java Swing/AWT, Java

SWT/Eclipse Rich Client Platform (RCP), XView.It

implements dedicated support for the mentioned toolkitsand technologies and its controls to allow creating robust

test scripts.

Thus there are many tools available in the market for

testing GUI applications with varied features.

V. TOOLS SELECTEDFO RANALYSISOF GU I TESTINGBASED

ON SPECIFICATIONS

For the purpose of analysis, we have chosen five

different tools which are freeware tools that can easily be

downloaded. The details of the tools along with its advantages

and disadvantages are given below:

A. Jemmy [2]

Jemmy is a tool that allows one to create automated

tests for Java GUI applications. Tests are written on Java,

using Jemmy as a regular Java library. No recording, no GUI,

no XML, - all the work being done in Jemmy is dedicated

to test stability, so it could be used for big, complicated

and "dynamic" Java GUI applications.

Advantages

Has a Java library that is used to create User

Interface (UI) automated tests.

Provides support covering all Swing

components.

Have two modes of action reproducing (Robotand event dispatching).

Has a Pre/post verification method.

Disadvantages

It does not support System test and Acceptance test.

B. Jacareto [9]

Jacareto is a capture and replay tool for programs

written in Java. One can capture actions on applications

and replay them later on (like macros). Jacareto can be

used for many purposes such as Creation of animated

demonstrations, Creation of macros, Qualitative and

quantitative analysis of user behavior etc.

Advantages

Uses virtual machine registration

Batch file and XML script used to start Java

virtual machine, start the test tool, and load th

applications classes. (Main method of application called

by test tool as required.)

Jacareto has to discover all instances of

subclasses of Component (including Components for

Swing)

Disadvantages

It has difficulty in launching various Java

applications that have dependencies on some external

libraries to handle the look-and-feel of the application. It did not seem compatible with it, as several

applications we selected for test case capture used it and

would fail to launch when we would attempt to perform a

capture in Jacareto.

C. JfcUnit [7]

JfcUnit is an extension to the popular testing

frameworkJUnit [12]. This document assumes that one is

familiar with the usage of JUnit. A JUnit extension that

supports writing GUI tests for Java Swing applications.

Advantages

A library that allows one to test swing applications. It

simulates button presses, ant test focus etc. A nicesolution to a hard problem.

Disadvantages

Tests require writing Java and XML code, which

adds complexity and consumes time. As a result,

applications specially designed to test Swing user

interfaces takes time.

D. Marathon [10]

Marathon is a general purpose tool for both running

and authoring acceptance tests geared at the Java

platform. Included with marathon is a rich suite o

components that helps to interact with the application at

the User Interface (UI) Level. To aid with the regressiontesting of existing applications, Marathon comes bundled

with a recorder to capture events. These events are then
http://www.junit.org/http://www.junit.org/


4/13

converted into a valid Marathon test which can subsequently

be played back.

Advantages

Marathon is the ability to write test cases even before the

application under test is built. Marathon, itself,

adds only a handful of well defined scripting elements to the

script runtime - either Jython or JRuby.

Disadvantages

It develops product that obviously do not mature. Web

testing is not available. Documentation and support is not atsufficient level.

E. Abbot [5]

Graphical user interface (GUI) testing is a potentially

problematic area because constructing effective test cases is

more difficult than the corresponding application logic. The

roadblocks to effective functional GUI testing are:

Traditional test coverage criteria like "80% coverage

of the lines of code" may not be sufficient to trap all the user

interaction scenarios.

End users often use a different user task interaction

model than the one conceived by the development team.

The framework easily integrates with the JUnit testharness and therefore, during application development, the

functional GUI tests can become a part of the test suite. All

these features of Abbot make it an effective framework for

rapidly creating a comprehensive test framework.

Advantages

Abbot builds upon the java.awt.Robot class to

provide an automated event generation and

validation framework for Swing GUI components.

VI. ANALYSISOF TH E SELECTED TOOLS BASEDON SPECIFIC

CHARACTERISTICS

For the analysis of the selected GUI tools, we have

considered tool features that are classified as,

A. General Features

The general features are categorized as

Supported Platforms: Platform Independence for

application

Supported OS:Independent for all operating system

Testing principle: GUI component development as

well as composite GUI development

Testing Structure: Efficient application of planning

by first creating a hierarchical model

Robustness of tests: Improving the entire system's

safety and robustness

Component recognition:Unique mechanism for

specifying and locating components

Test Management: Easy integration with third party

solutions

Reporting: Maintained the bug reports. Bug

Tracking and Reporting

Extensibility / Scripting: The solutions to support

new custom GUI components

B. Usability Features

The usability features include

User friendly: People will tend to buy systems that

are more user-friendly

Accessibility: People should be able to access

GUI objects

Installation: Installing the GUIs tool without

any interrupt Documentation: Extensive manual and tutorial,

online help

Programming Knowledge Required: Requires

any previous knowledge about tool

Rapid script: It will be available for only with

Global GUI map file concept.

User Actions: User interact with GUIs

components

Component finding Technique: Each

component is centralized and can be easily

updated

C. Test Execution Features

The Execution features includes

Test Scripts: It can make test suites easier

Test Case Preparation:Test case must be

complex and follow a path through the GUI

where the operations are performed in a specific

order

Test Modification: Reused for many related

tests.

Object mapping: logical names and physical

properties of GUI mapping tool

Recording: Store the user actions

Play back: Play the stored user actions

Event Mechanism: To implement the

synchronization of user actions

GUI Map Editor: To save unsaved entries in a

local buffer by the test engineer

Error Handling: Maintain all user actions and

reuse for future

Maturity: It is a key variable in deciding what

type of automation is considered

Complexity: Software needs to be tested

because untested code is a potential source ofbugs

Context Sensitive Recording:Records the

operations are perform on specific application by

identifying specific GUI objects work

CSS script: It is a simple mechanism for adding

style

D.Kinds of Testing Performed

This feature includes

Unit Test: It looks similar to the JUnit

GUI.Building unit tests and functional tests

(AWT Unit/ Swing Unit) System Test:client/server and web based

systems deliver system functionality using a

GUI.


5/13

Flexible Testing: It's enough for general desktop

GUI automated testing.

Acceptance testing: It should be done at the

business logic level

Image testing/comparison:It can find image

sample and then focus mouse on it

Running Modes: It is provide Robot class most

basic programmatic support, no support to find

widgets

Component Lookup: It should best way to

guarantee reliable

TABLE 1 ANALYSISOFGUITOOLSBASEDONSPECIFICFEATURES

REQUIREMENTS ABBOT MARATHON JEMMY JFC UNIT JACARETO

General Features

Supported

Platforms

Java, Java swing,

Xml

Java,Xml

Java Swing

Awt /swing

componentsJava, java swing Java, applets

Testing principle

Record/ Play

back,

Programmatic

GUI

Record/Play

back

Programmatic

GUI

Programmatic

GUI

Dynamically

Capture events

play back later

Capture Replay

Testing

Structure

Has a scripted

Java GUI testing

framework.

Has well-defined

scripting

elements to the

script at runtime

Is a Net Beans

independent

module

An extension to

the JUnit

framework

Groups record

elements - build high-

level structures

Testing Robust

ReportingCSV or Xml or

TextHtml or Text Html or Xml Xml CSV or Xml

Extensibility /

Scripting

has Unique

Component

(ID) Scripts -

Call Directly java

Code

Jython/Python

Libraries/Call

Java Code

Rare(new

operator) / high

level xml tool

(jelly)

New events from

basic eventsSpecial event types

Usability Features

User friendly Yes YesDepends on

DesignComplex Yes

Installation/ Size

Very fast - 17

MB

Very fast - 16

MB

Install Jar file

-1MB

Install Jar file

only 8 MBDocumentation Available Available Available Available Available

Programming

Knowledge

Required

Java Swing and

Xml

Java Swing and

XmlJava Java Swing, Xml Java

Test Execution Features

Test ScriptsXml and Java

SwingPython/Jython Xml (Jelly) Xml Xml

Test

Modification

Abbot Molecular

Testing

Usability Net

Performance

testing methods

Usability Net

Performance

testing methods

Not allowed.

Increases the test

complexity

Usability Net

Performance testing

methods

Record/ Play

back

Use Abbot Robot Jython No No Regenerate Events

Event

Mechanism

Robot (Default)/

AWT

AWT Control/

Program control

Awt (Default)/

Robot

Awt (Default)

/RobotAWT/Robot

GUI Map EditorScript Editor

(Costello)

Marathon Map

Editor- JFC Editor Jacareto Editor

Error handling

(Automated)Yes Yes Yes Yes Yes

CSS script Available Available Available Available Available

Context Sensitive

Recording

Script Enviro

-nment and JUnitUsing JUnit Using JUnit Using JUnit Using JUnit

MaturityHigh Degree Higher

Granularity

Higher

GranularityLow Level High level

Types of Testing Performed Features

UnitTest Yes No Yes Yes NoSystem test Default Yes No No

Flexible Testing Yes Yes Yes No

Acceptance test Available Available Not Available Not Available Available


6/13

Image testing/

comparisonSupported Supported Supported Supported Supported

Running Modes 1 2 1 1 2

Component

LookupFuzzy Expects Classes

Component

Specific-

API DesignGUI Object

ScriptJython Operator objects

Custom Event

Classes-


7/13

A detailed comparison of all these features for the five

selected GUI Testing tools are as shown in Table 1. From the

table it is very clear that, Abbot, Jemmy, and JFCUnit provide

ways to unit-test GUI code. Abbot and Marathon provide

higher-level scripting actions more suitable for, Acceptance

testing. JFCUnit has started work on some level of scripting

support. Abbot and of user actions on a system under test for

further manipulation by hand or later playback. Abbot

provides both robot (default) and AWT modes, and throws ina little Programmatic control while the other tools do not

support these features. Jemmy and JFCUnit use AWT mode

by default but provide a robot version if needed. Marathon

uses a mix of AWT control and Javas Accessibility libraries

(programmatic control).

Abbot uses a dedicated class to represent GUI

components (since the components dont necessarily exist

when you first need to reference them). It stores a range of

attributes to be used when looking for a matching component

in the hierarchy, and does a fuzzy lookup to avoid spurious

errors due to component movement or minor changes to the

GUI. The lookup mechanism is very general due to the fact it

is used by the scripting layer which has no apriori knowledgeof the GUI hierarchy or what needs to be looked up. The

framework also provides a number of utilities to facilitate

inspecting the GUI hierarchy itself. Jemmy provides a few

different lookup classes that finds component based on

different criteria built into the class; the programmer needs to

pick the right lookup class based on knowledge of the

hierarchy. JFCUnit provides component-specific lookup

classes. These are used in conjunction with a test helper class

to look up a desired component. Marathon mainly expects to

find components by name.

Abbot also provides for extending the basic tester classes

for new components, and has a well defined method for

naming those new actions so that they are automaticallyavailable at the scripting level. It also provides extensibility

for converting strings from scripts into arbitrary classes, and

introducing new individual steps into scripts.

Jemmy has rather a high hurdle for introducing a new

operator. Jemmy has a higher-level xml-based scripting tool

called Jelly that facilitates higher-level application

actions.JFCUnit requires to define new event types for

components derived from the basic ones. Abbot provides both

a script environment and a JUnit fixture, both of which handle

setup and teardown of the complete GUI environment. Scripts

also support launching a single application instance and

including other scripts.

Thus compared to the other four tools Abbot is

considered best when the applications are mainly Java swing

oriented applications. Using this tool, test cases could be

executed , positive and negative test cases identified, coverage

of the test cases could be found, etc. , which cannot be done

by other GUI tools.

The next section provides the implementation of the

abbot tool through which the generated test cases were

executed and verified.

VII. IMPLEMENTATION

In this paper, the GUI model of Calculator

Application, written using Java Swing is considered. Thescientific Calculator Application contains a collection of

Standard Buttons and Scientific Buttons. It is used to calculate

the Arithmetic and Scientific data values. It performs several

basic operations such as Add, Sub, Mul and Div and also

Scientific Operations such as Sin, Cos, Tan, Log, sqrt ,

etc. The program also contains Radio Buttons like

Hexadecimal, Octal, Decimal and Binary. The calculator

takes inputs from both standard and scientific modes,

where the scientific users may be categorized as technical

users, students and business users. Fig .1 shows the GUI

model application being implemented. All the operations

are performed by click events .

Figure1: The GUI Calculator Application

A. Testing with Abbot

In the Calculator Application, test case generation

makes use of the core components like Component Test

Fixture () and Component Tester () in Abbot and also

writes the test cases using actionClick (), SelectMenuItem

(), actionDelay (),etc. By making use of these

components, test cases are generated. The assert method

compares the data values obtained from calculator

program with the data values obtained by the automatic

execution of the test cases. The click event fo

hexadecimal option button and the corresponding enabled

buttons are shown in Fig 2.

Figure 2: Automatic Execute of User Input Sequence

1). Test Case Generation

Test case Generation and Execution is done

using Abbot and Junit Tool. Abbot is a framework for

driving java UI components programmatically. Costello is

a script editor and a launcher which accompanies Abbot.

They are based on the same capture/ playback/script

principle that marathon uses, but Abbot can also be used

programmatically. The framework can be invoked

differently from Java code. It is suitable for use both by

developers for unit tests and QA for functional testing.

The framework easily integrates with the JUnit test

harness and therefore, during application development, the


8/13

functional GUI tests can become a part of the test suite. All

these features of Abbot make it an effective framework for

rapidly creating a comprehensive test framework. Collections

of test cases are generated by using Abbot and JUnit Tool. The

test cases contain the sequence of user Input Events.

Fig 3 Test Case Generation using Abbot Tool

This framework shown in Fig3 chooses a specific model of

the GUI application. This model satisfies the GUI application

under test which is also the input of the Test Case generation.

Collections of test cases are generated by using JAR (Abbot

and JUnit)..The source code of test case generation for the

calculator application is shown in Fig 4.

Figure 4: The Source code of Test Case Generation

2) Test Case Execution

Testcases from the repository are executed one by one

automatically using Abbot and JUnit Tools as shown in Fig 3.

Fig 3 Test Execution by JUnit Tool

The Test case contains the sequence of user Input events. The

test cases are run with test runner. A test designer interacts

with the GUI and generates mouse and keyboard events.

Depending upon the test cases, the events are executedautomatically. The source code of the test runner is given

below.

Public static void main (String [] args)

{

TestHelper.runTests (args, CalculatorTest.class);

}

3) Test case Verification:

The expected results of various test cases are manually

determined and stored in the GUI model of Testing Oracle.

The Testing Oracle contains the expected state of sequences

for each application. When the test cases start running, the

start timer is initialized. The events are generatedautomatically and the actual results from the tool are verified

with the expected results. Pass/Failure testing report is

generated accordingly.Fig 6 shows the positive and

negative test case verification.

Figure 6: Test case Verification

B.Code Coverage with Clover

Clover [21] uses source code instrumentation,

because although it requires developers to perform aninstrumented build; source code instrumentation produces

the most accurate coverage measurement for the least

runtime performance overhead. As the code under test

executes, code coverage systems collect information about

which statements have been executed. This information is

then used as the basis of reports. In addition to these basic

mechanisms, coverage approaches vary on what forms of

coverage information they collect. There are many forms

of coverage beyond basic statement coverage including

conditional coverage, method entry and path coverage.

1) The Clover Coverage Measurement

Clover uses these measurements to produce a TotalCoverage Percentage for each class, file, and package and

for the project as a whole. The Total Coverage Percentage

allows entities to be ranked in reports. The Total Coverage

Percentage (TPC) is calculated as follows: TPC = (BT +

BF + SC +MC)/(2*B + S + M) where BT - branches that

evaluated to "true" at least once BF - branches that

evaluated to "false" at least once SC statements covered

MC - methods entered B - total number of branches S -

total number of statements M - total number of methods.

Table1 displays the report title and the time of the

coverage contained in the report. The header displays

metrics for the package files or project overview which is

currently selected. Depending on the current selection, themetrics include all or a subset of: Number of Lines of

Code (LOC),Number of Non-commented Lines of Code

(NCLOC),Number of Methods, Number of Classes,

Number of Files, Number of Packages.

C.Event Coverage

Event Coverage is determined by the total number of

click events generated by the user. For Eg table3 shows

that Single event is generated for Basic button, 11 events

are generated for calculating the area of circle (pi*r*r).

D. Code Coverage

Code coverage and event based performanceanalysis are done in the module. Table 4 shows that, Code

coverage tool is used for statements, branches and loop.

Manual analysis is done for event interaction and events


9/13

and the relevant data is collected. Coverage metrics are

collected for different units of application.

TABLE:2 VIEWOFALL TESTCASESVALUEUSING CLOVERCOVERAGE TOOL

AreaTestCase State

ment

Bran

ches

Metho

ds

Classes LOC NCLO

C

Total

Cmp

Cmp

Densit

y

Avg

method

Cmp

Stmt /

methods

Methods /

classes

Total

Coverage

(in %)

Circle 280 132 21 1 625 483 138 0.49 6.57 13.33 21 54.3Rectangle 293 132 24 2 667 513 141 0.48 5.88 12.21 12 53.6

Parallelogram 291 132 24 2 666 511 141 0.48 5.88 12.12 12 53.6

Triangle 298 132 24 2 672 518 141 0.47 5.88 12.42 12 54.1

Trapezoid 297 132 24 2 672 517 141 0.47 5.88 12.38 12 54.1

Total 339 132 28 2 738 570 145 0.43 5.18 12.11 14 55

Surface

Rectangle 312 132 25 2 690 535 142 0.46 5.68 12.48 12.5 41.6

Prims 304 132 25 2 681 527 142 0.47 5.68 12.16 12.5 42.7

Cylinder 303 132 25 2 676 525 142 0.47 5.68 12.12 12.5 53.3

Cone 300 132 25 2 671 522 142 0.47 5.68 12 12.5 53.3

Sphere 294 132 24 2 668 513 141 0.48 5.88 12.25 12 51.7

Total 373 132 28 2 775 602 145 0.39 5.18 13.32 14 53.6

Volume

Rectangle 291 132 24 2 666 511 141 0.48 5.88 12.12 12 55

Prims 293 132 24 2 669 513 141 0.48 5.88 12.21 12 55

Cylinder 296 132 24 2 671 516 141 0.48 5.88 12.33 12 55

Cone 297 132 24 2 672 517 141 0.47 5.88 12.38 12 55

Sphere 299 132 24 2 677 519 141 0.47 5.88 12.46 12 55.5

Pyramid 295 132 24 2 671 515 141 0.48 5.88 12.29 12 55

Total 351 132 29 2 762 586 146 0.42 5.03 12.1 14.5 55


10/13


11/13

TABLE: 3 VIEWOFALL TESTCASES EVENT COVERAGE

Test Plan Event Execution

(in sec)

Execution

Delay

(1000 sec)

View

Basic 1 1.684 2.699

Scientific 1 1.763 2.714

Hex 1 1.342 2.371

Dec 1 1.295 2.309

Octal 1 2.262 2.356

Binary 1 1.342 2.324

AreaCircle 11 3.588 4.555

Rectangle 7 2.434 3.463

Parallelogram 6 2.262 3.26

Triangle 12 3.447 4.446

Trapezoid 12 3.401 4.415

Volume

Rectangle 6 2.278 3.291

Prism 8 2.036 3.65

Cylinder 11 3.525 4.633

Pyramid 10 2.995 4.009

Cones 11 3.588 4.556Sphere 14 4.119 5.085

Surface

Rectangle 26 6.006 6.989

TABLE: 4 VIEWOFALL TESTCASES CODE COVERAGE TOOL

Test Case Stmt Branch Loop Strict

Condition

Basic 59.4 2.3 10.1 6.0

Minus 63.9 10.5 8.7 7.7

Add 63.9 9.3 8.7 7.7

Mul 63.9 11.6 8.7 7.7

Div 63.9 12.8 8.7 7.7

Mod 67.1 15.1 10.1 9.4

Hex 61.6 11.6 14.5 19.7

Dec 60.7 9.3 13.0 7.7

Oct 61.6 11.6 14.5 15.4

Bin 61.6 14 13 9.4

Area

Circle 68.9 36.0 10.1 14.5

Rectangle 63.9 12.8 8.7 11.1

Parallelogram 63.9 12.8 8.7 10.3

Triangle 65.3 17.4 8.7 13.7

Trapezoid 65.8 18.6 8.7 15.4

Surface

Rectangle 64.4 14.0 8.7 13.7Prims 65.3 16.3 8.7 13.7

Cylinder 68.9 34.9 10.1 16.2

Cone 68.9 34.9 10.1 15.4

Sphere 67.6 30.2 10.1 9.4

Volume

Rectangle 63.9 11.6 8.7 8.5

Prims 64.8 15.1 8.7 10.3

Cylinder 68.5 34.9 10.1 14.5

Cone 68.5 34.9 10.1 12.8

Sphere 68.5 33.7 10.1 12.8

Pyramid 64.4 15.1 8.7 11.1

Total 97.7 93 34.8 94.9


12/13

Prims 18 4.524 5.506

Cylinder 17 4.68 5.647

Cones 13 3.916 4.898

Sphere 9 3.183 4.165


13/13

E.Coverage ReportCoverage XML Report

The clover xml report task generates a full HTML report

with sensible default settings. It is also generated prior to

generation of the full report.

Coverage PDF Report

The clover pdf report task generates a PDF report with

sensible default settings. It is also generated prior to

generation of the full report.The different explorer view of coverage report for all

completed test cases is shown in fig 8, 9.

Fig 8: Project Coverage Report in XML Format

Fig 9: Coverage Report in PDF Format

VIII. CONCLUSIONAND FUTUREWORK

In this paper we have analysed the functionalities of

five different automated GUI tools based on some specific

characteristics. This analysis will be of great use to the tester

in order to choose an appropriate tool based on his

requirements. In this paper we have used Abbot tool. This is

easy to learn and use, and provides some unique features that

can make GUI development more productive. We have

developed the method for automatic test case generation,

execution, verification and calculating the coverage based on

statement, method and branch for GUI applications. These

tools, though they are completely automated, are not

generalized for all kinds of GUI applications. In future,

a new tool that satisfies most of the mentioned features,

tests a variety of GUI applications and improves the

coverage for user interactions can be developed.

REFERENCES

[1] A. M. Memon. Gui testing: pitfalls and process. IEEE Computer,35(8):8788, 2002.[2] http://jemmy.netbeans.org

[3] Brad A. Myers. User interface software tools. ACM Transactionson Computer-Human Inter-action, 2(1):64{103, 1995.}

[4] Capture-Replay Tool, http://soft.com

[5] T. Wall. Abbot java gui test framework,2004.http://abbot.sourceforge.net/

[6] Tamas Daboci,Istvan Kollar,Tamas Megyeri,How to Graphical

User Interfaces IEEE Instrumentation & Measurement

Magazine,Sep 2003

[7] The JFCUnit projects homepage http://jfcunit.sourceforge.net/-

[8] A. M. Memon, M. E. Pollack, and M. L. Soffa. Using a goal-driven approach to generate test cases for guis. In 10 Proceedings

of the 21st international conference on Software engineering, pages

257266. IEEE Computer Society Press, 1999

[9] C.Spannagel.Jacareto,2003. http://sourceforge.net/ projects/jacareto

[10] A Thoughtworks production. probably the most recentproject.http://marathonman.sf.net.

[11] A. M. Memon, M. E. Pollack, and M. L. Soffa. Hierarchicalgui test case generation using automated planning. IEEE Trans.Softw. Eng.,

[12] E. Gamma and K. Beck. Gui tools forjunit,2004.http://www.junit.org/news/ extension/gui/index.htm

[13] Atif M Memon. Advances in GUI testing. In Advances inComputers, Ed. by Marvin V.Zelkowitz, volume 57. Academic

Press, 2003.[14] QF-Test - The Tool for Test Automation of Java/Swing and

Eclipse/SWT/RCP applications. http://www.qfs.de/ en/qftest/

download.html.

[15] A. Memon, A. Nagarajan, and Q. Xie, AutomatingRegression Testing for Evolving GUI Software, J. Software

Maintenance and Evolution: Research and Practice, vol. 17,

[16] http://www.froglogic.com/squish/

[17] Q1 is a tool for developing GUI and HTML. http:// www.3d2f.com/programs/11-164-q1-download.shtml

[18] P.Hamill, Unit Test Frameworks, OReilly, 2004[19] http://www.bredex.de/en/guidancer/first.html[20] Clover coverage tool www.atlassian.com/software/
http://jemmy.netbeans.org/http://jemmy.netbeans.org/http://soft.com/http://abbot.sourceforge.net/http://jfcunit.sourceforge.net/-http://jfcunit.sourceforge.net/-http://jfcunit.sourceforge.net/-http://sourceforge.net/%20projects/%20jacaretohttp://sourceforge.net/%20projects/%20jacaretohttp://sourceforge.net/%20projects/%20jacaretohttp://sourceforge.net/%20projects/%20jacaretohttp://www.junit.org/news/%20extension/gui/index.htmhttp://www.qfs.de/http://www.bredex.de/en/guidancer/first.htmlhttp://www.bredex.de/en/guidancer/first.htmlhttp://www.atlassian.com/software/http://jemmy.netbeans.org/http://soft.com/http://abbot.sourceforge.net/http://jfcunit.sourceforge.net/-http://sourceforge.net/%20projects/%20jacaretohttp://sourceforge.net/%20projects/%20jacaretohttp://www.junit.org/news/%20extension/gui/index.htmhttp://www.qfs.de/http://www.bredex.de/en/guidancer/first.htmlhttp://www.atlassian.com/software/

Date post:	09-Apr-2018
Category:	Documents
Upload:	thangamari
View:	219 times
Download:	0 times

Comparison of the Features of GUI Testing Tools

Documents