1
ATHABASCA UNIVERSITY
MARTS - Multi-Agent Based Regression Testing Suite
BY
Khalid Aziz Muhammadi
A project submitted in partial fulfillment
Of the requirements for the degree of
MASTER OF SCIENCE in INFORMATION SYSTEMS
Athabasca, Alberta
April, 2009
© Khalid Aziz Muhammadi, 2009
2
ABSTRACT
Software testing is one of the major phases in Software Development Life Cycle,
and typically adequate budget is allocation for this activity, however software
quality usually does not meet end-users expectations. This paper attempts to
achieve two goals. First, it highlights the fundamental issues with project planning
and management, which directly impact the quality of the end product.
Inadequate software quality and lack of user’s acceptance of the product, is not
only due to technical reasons, but the combination of technical and non-technical
issues, which includes lack of proper management and leadership, decisions
without appropriate user feedback, and incorrect resource allocation. This paper
highlights some underline testing issues, and how Managers, Developers,
System Testers, and End-User’s play its roles accordingly, to make matter
worse. Secondly, it proposes solution of software testing using Intelligent
Software Agents. Regression testing can be repetitive work, but does require
some level of intelligence; Software Agents are the best candidate for the
Automated Regression Testing. Proposed framework; MARTS (Multi-Agent
Based Regression Testing Suite) is a loosely coupled component-based testing
framework, which uses intelligent Software Agents to simulate system testers,
along with some capability to make autonomous decisions. Automated testing
reduces human error possibilities, and overall cost reduction in long run. MARTS
adopts the benefits of proven Distributed Computing solutions, along with the
flexibility and decision making capability of Intelligent Software Agents. For
3
components like Data-Generator and Reporting, preferred solution is to use off-
the-shelf components and integrate into MARTS. Overall goal is not to use latest
technology, and create cutting-edge technology product, but to address real
business problem’s, i.e. how to improve software quality and end-user
experience.
4
ACKNOWLEDGMENTS
Special thanks to Dr. Qing Tan, University of Athabasca, for his supervision in preparing this essay.
5
TABLE OF CONTENTS
CHAPTER I - INTRODUCTION ...................................................................................... 7
Statement of the Purpose .............................................................................................. 16 Research Problem ......................................................................................................... 16 Definition of Terms....................................................................................................... 17 Organization of the remaining chapters........................................................................ 18
CHAPTER II - REVIEW OF RELATED LITERATURE............................................... 19
Related Work ................................................................................................................ 22 Current Research........................................................................................................... 24 Summary ....................................................................................................................... 29
CHAPTER III - METHODOLOGY................................................................................. 30
Software Agents............................................................................................................ 30 JADE Platform.............................................................................................................. 32 Agent Communication Language (ACL)...................................................................... 34 Multi-Agent Based Regression Testing Suite (MARTS): An Overview...................... 34 MARTS Software Architecture .................................................................................... 36 An Online Shopping Store - Example .......................................................................... 47
CHAPTER IV - ISSUES, CHALLENGES, AND TRENDS........................................... 49
CHAPTER V - CONCLUSIONS AND RECOMMENDATIONS.................................. 51
Conclusions................................................................................................................. 51 Suggestions for Further Research .......................................................................... 52
REFERENCES ................................................................................................................. 53
6
LIST OF FIGURES
Figure 1.0: Software Development and Maintenance Costs (Boehm, 1981) 11
Figure 1.1: Software Development and Maintenance Costs in 487 Business
Organizations (Boehm, 1981) 12
Figure 1.2: Software Maintenance Production Function (Boehm, 1981) 13
Figure 2.0: Software Activity Variations – Percentage of Staff Effort by Activity
(Jones, 2002) 21
Figure 2.1: Test Agent System (Choi, 1999) 25
Figure 2.2: Implementing our testing approach upon JADE Platform - a low level
design view. (Coelho, 2006) 28
Figure 3.1: FIPA reference model of an Agent Platform (Bellifemine, 1999) 33
Figure 3.2: Software Architecture of JADE Agent Platform (Bellifemine, 1999) 33
Figure 3.3 MARTS Framework Architecture 37
Figure 3.4 Framework Controller 38
Figure 3.5 Agents Communication with Framework Controller 38
Figure 3.6 Reporting Component with Framework Controller 39
Figure 3.7 Logging Component with Framework Controller 40
Figure 3.8 Framework Controller with Test Data Generator 41
Figure 3.9 Framework Controller with Test Results Analyzer Service 41
Figure 3.10 Framework Controller with Test Scenario Agent 42
Figure 3.11 Test Scenario Agent with Test Data Generator 43
Figure 3.12 Test Scenario Agents with Logging Component 43
Figure 3.13 Test Scenario Agent with Unit Test Wrapper 43
Figure 3.14 Test Scenario Agent with Test Results Analyzer 44
Figure 3.15: SQLEdit DTM Data Generator (SqlEdit, 2009) 45
Figure 3.16: TurboData Data Generator (Turbodata, 2009) 46
7
CHAPTER I
INTRODUCTION
Software testing is one the most important, yet frequently ignored subject area in
IT industry. This is also one the major reason why software maintenance cost is
very high. Third party software providers or (ISP’s) are using better standards
and techniques to improve software quality. However, this is still a major problem
for in-house development. Project Manager’s tend to ignore this phase, or
assign this task to the people with low technical skills. When programmers do not
perform well, they are typically assigned testing work. Writing quality code
requires significant level of technical skills, however, testing the feature requires
in-depth understanding of the business, and the detail study of business
requirements for the application. Unfortunately, software testers usually do not
have adequate business knowledge of the application or understanding of
business requirements. Software testers are given list of test cases for execution,
which are written by the business users and typically reviewed by software
developers. One may think, if test cases are written by the business users and
reviewed by developers, than it should be adequate enough for the software
validation and achieving required software quality.
Writing test cases and creating test plans is the biggest documented blame game
in Software Industry. It’s well structured, where in the end everyone loses,
however no one can be held accountable for the potential disaster. Therefore it
8
can be called as Software Blame Game. Game is played by four major actors,
and each has predefined set of rules, which dictates how they will play the game.
The Project Manager; always concerned about the schedule and budget for the
project. To expedite the development, the manager will assign the testing to the
weakest person on the project. Appropriate time and budget is allocated;
therefore when the incompetent member of the team is not able to test properly,
Project Manager cannot be blamed. However, manager is aware from day one,
that person assigned for testing will not be able to achieve testing goals; hence
software quality will not be according to what is promised to the end-users.
The Business Users; always concerned about how this new application will affect
there day to day tasks. They will make every attempt to make things more
complicated, and sometime provide details which has nothing to do with the
actual application. Business users have day to day tasks, and when new IT
project is initiated, they are expected to contribute and make that project
successful. However, business users day to day tasks should not be impacted
either. Aggressive timelines force them to rush on identifying business
requirements, therefore business requirements tends to be vague, and can
sometime become impossible to create specific test cases to validate those
requirements. Business users are typically aware of IT department’s lack of
information about the business; however they tend to let it slide. If project
encounter issues in production, it can be safely blamed on the IT department.
9
This ensures job security for the business users, and potential re-write of the
project within five to ten years.
Software Testers; execute the test cases according to the predefined test plan
which is created by business users and reviewed by Software Developers. They
will run exactly what is asked for, and will try their best not use their mind and go
extra mile to validate the scenario, which is not documented. If software tester
attempts to go extra mile, they will be discouraged by the Developers and Project
Manager, since tester will spend more time testing, which could impact project
timelines. Therefore, it is not the fault of Software Testers if software quality is
not achieved, because they are not given enough time and development support
to go extra mile.
Software Developers; can also be identified as construction workers, work over
time, and meet their deadline, at the expense of product quality. They tend to
code the software feature without reading the business requirements. They never
accept their fault and can get offended easily when asked to unit test the code.
Overall maintenance cost it increasing exponentially, and regardless of
methodologies, and tools, product quality is not up to the client’s satisfaction. In
large organizations, appropriate best practices and software methodologies are
adopted; however, expected results cannot be achieved using available
resources and timelines. As Stutzke states, “Software processes must produce
10
software that can be gracefully evolved at reasonable costs. The choice of the
software architecture significantly influences modifiability and hence
maintainability.” (Stutzke, 1996). In 2001, “more than 50 percent of the global
software population was engaged in modifying existing applications rather than
writing new applications.” (Jones, 2002)
Since programs are not tested adequately, future enhancements are not easily
doable, and in some cases, software project requires potential re-work during
maintenance. “All programs should be written with the expectation that they will
be changed later. They should be judiciously modularized, well-commented, and
written in proper style so that the effort required to understand them later in order
to make changes is significantly less than that of starting over.” (Conte, 1986)
Software products with higher maintenance usually run into issues of estimation.
It is nearly impossible to accurately estimate the maintenance work on the
application, which was not tested properly in initial release. Some documented
functionality may not exist, and other may or may not work as documented.
“Development estimating is a difficult problem, but estimating maintenance and
enhancements is even more complex because of the very tricky relationship
between the base application that is being modified and the changes being made
to accommodate new features or repair defects.” (Jones, 1998). In order to
ensure that software product is reliable, low maintenance, and ease in future
enhancements, one need to make sure that product is properly tested. "Software
11
testing will remain one of the best tools we have at our disposal to ensure
software dependability." (Briand, 2007)
Figure 1.0: Software Development and Maintenance Costs (Boehm, 1981)
12
Figure 1.1: Software Development and Maintenance Costs in 487 Business Organizations
(Boehm, 1981)
13
Figure 1.2: Software Maintenance Production Function (Boehm, 1981)
There are quite a few methodologies that can be used for software testing. Unit
tests can prove to be very effective, when used for regression testing. This allows
testing to be performed at function level, and each component can be tested at
its core.
"A growing trend in software development is to use testing frameworks such as
the xUnit frameworks (for example, JUnit, JsUnit and NUnit) which allows the
code to conduct unit tests to determine whether various sections of the code are
acting as expected in various circumstances. Those frameworks do not fit for
testing distributed real-time systems." (Yu, 2007). Automated testing using unit
tests can help to improve software quality, and reduce the testing and
14
maintenance cost. For every software build, unit tests can be executed to ensure
no new defects are introduced. Automated testing also guarantees that test
cases are not skipped, and report can be validated to verify the results. As Liu
states that “automated test can run more test cases in a given time, make better
use of resource”. (Liu, 2003). This also reduces the overall testing cost. Test-
driven Development (TDD) is one of the popular evolving agile methods. “TDD
stresses the development of working code over documentation, models, and
plans. The suggestion of TDD is (paradoxically) that developers test the program
before it is fully written. A striking aspect of this approach is the idea that the
code that is implemented may not be behaviorally correct: it just has to pass the
test. The test is therefore the specification against which the code is checked."
(Ostroff, 2007)
As David Kung, in his paper highlights the fact that, Dynamic behaviors,
Heterogeneous representations, Uncertainty in execution and Novel data
handling mechanism in web based applications makes testing extremely difficult
to test. He states that, "web applications [are] much more complicated than
traditional applications. Traditional testing approaches are no longer adequate for
Web applications because they cannot handle such complicated situations. On
the other hand, most available approaches for Web applications testing only
focus on usability, conformance, performance, etc. The lack of mature and
systematic approaches and tools forces Web applications testing to rely on ad
hoc solutions and the experience of developers. This adversely affects the quality
15
of Web applications. Consequently, systematic as well as flexible and extensible
testing approaches and intelligent tools are in urgent demand." (Kung, 2005)
"Software testing remains the primary technique used to maintain quality of the
software products and to gain consumers’ confidence in software. Unfortunately,
it is well-known that testing software is a time-consuming and costly process.
Therefore, techniques that support the automation of software testing will result
in significant cost and time savings for the software industry. Automatic
generation of the test data is essential for the automation of software testing."
(Xu, 2005)
However, one major issue with automated testing is test cases are repeated
without any intelligence. Test Data changes frequently, and sometime results
from the automated tests provides very little or no value to the overall product
quality. Proposed solution to that problem is to utilize Intelligent Software Agents.
"Agents are objects that represent operations; they are effectively closures from
functional programming. Agents can be passed to different software elements,
which can use the object to execute the operation whenever they want. Agents
thus provide a way of separating the definition of a routine from its execution."
(Ostroff, 2005)
Software Agents are autonomous, and can make their own decisions up to
certain extend, based on their Belief, Desire, and Intension (Agent BDI Model).
"Agent technologies facilitate the automated software testing by virtue of their
high-level decomposition, independency, and parallel activation. The informal
16
interpretations of qualitative agent theories are not sufficient to distinguish agent-
based approaches from other approaches in software testing." (Dhavachelvan,
2005)
Statement of the Purpose
Appropriate level of testing not only ensures quality product, but assist in
reducing the maintenance cost. There is no difference of opinion regarding the
importance of testing, however, to achieve the adequate level of testing depends
on project timeline, allocated budget, and resources with appropriate level of skill
set. With new software methodologies, Project Managers allocate sufficient
amount of resources for the testing phase. However, software products are still
error prone with high maintenance cost. There are various testing methodologies,
and depending on the type of software product, different testing strategies can be
used to achieve optimal results. This paper proposes solution by using Intelligent
Software Agents for Regression Testing Framework. It utilizes unit tests and
allows automated testing with extensive reporting capabilities.
Research Problem
Automated testing provided initial results, however over the time; the value of
those results diminishes due to lack of intelligence in the execution. Automated
scripts may not fully accommodate change in environment, i.e. variations in
database schema and business requirements. Proposed Regression Framework
17
- MARTS; uses Intelligent Software Agents for executing unit tests, on JADE
platform (Java Agent Development Framework). Proposed solution only uses
agents for execution of test scenarios, in order to reduce the overhead of
creating every component as agent-based. Therefore proposed solution utilizes
the benefits from the proven distributed applications, along with the flexibility and
autonomous decision making capability of agent-based applications.
Definition of Terms
MARTS: Multi-Agent Regression Testing Suite – The proposed solution, which
uses Software Agents to execute unit test cases.
JADE: Java Agent Development Framework is a software Framework that
implements multi-agent systems through a middle-ware that complies with the
FIPA specifications.
FIPA: Foundation for Intelligent Physical Agents is an IEEE Computer Society
standards organization that promotes agent-based technology and the
interoperability of its standards with other technologies.
BDI: Agent Architecture - Belief, Desire, and Intension. BDI software model
implements the principal aspects of Michael Bratman's theory of human practical
reasoning.
18
Organization of the remaining chapters
Chapter 2 discusses different testing methodologies, and use of Software Agents
for application testing. Chapter 3 discusses Software Agents, brief overview of
JADE platform, and explains in detail the architecture of MARTS – Multi-Agent
Regression Testing Suite. Chapter 4 discusses the issues, challenges, and
trends in software testing, and using agents in regression testing. Chapter 5
discusses conclusion and future work in the field of software testing.
19
CHAPTER II
REVIEW OF RELATED LITERATURE
Software testing has been the field of interest for decades. Significant research
has been conducted, to enhance the over testing process, and especially
Software Testing Automation. With better testing, not only we achieve better
quality product, but also assists in meeting project timelines. As Jones states in
the article on software estimation, “reason software projects run late and exceed
their budgets may be that they have so many bugs they cannot be released to
users.” (Jones, 2002)
There are different approaches, one can adopt for software testing automation
however for the scope of this paper; we will discuss the possible solutions using
intelligent software agents. Software agents have been defined using various
definitions, and as David states that, "although there is no general accepted
definition, an agent can be viewed as an autonomous object that is designed and
implemented to complete certain tasks by cooperating with the environment and
other agents." (Kung, 2005)
Software agents have been used in various projects to automate software
testing. Depending on the type of application, agents can assists in not only
testing the functionality, but to automate the test case generation. As explained in
one research paper, "Agent technologies facilitate the automated software testing
by virtue of their high level decomposition, independency, and parallel activation.
20
Usage of agent based regression testing reduces the complexity involved in
prioritizing the test cases. With the ability of agents to act autonomously,
monitoring code changes and generating test cases for the changed version of
the code can be done dynamically." (Salima, 2007)
Coelho, in his research, used agents to test application in development phase,
an approach also sometime refer as Test Driven Development (TDD). He states
that, "Agent technology has emerged as a prominent technique to address the
design and implementation of these new and complex distributed systems. While
the asynchronous architecture of multi-agent systems (MAS) helps to address
current application requirements, it also brings many threats to software
dependability. [….] there are four complementary means to attain dependability:
fault prevention, fault tolerance, fault removal and fault forecasting. In this work,
we propose a MAS testing approach which aims at removing faults along the
application development." (Coelho, 2006)
Software product goes through an extensive development life cycle, and
depending on the type of software applications, some development phases
require more resources than others. Author Jones; create the comparison chart
for Web Projects, MIS projects, System Projects and Military Projects. Notice,
Military projects, where product quality is extremely important requires even
distribution of efforts at all phases of software development. On other hand, web
projects are usually simpler, and do not usually requires resources at all phases.
21
Figure 2.0: Software Activity Variations – Percentage of Staff Effort by Activity (Jones, 2002)
22
Related Work
Intelligent have been used in the field of software engineering for quite some
time. Agents are used in distributed applications, and in software testing. Since
agents can be enhanced to represent human behavior, they can play very
important roles in the field of software testing. As Coelho states, "an agent is an
autonomous, adaptive, and interactive element that has a mental state. The
mental state of an agent is comprised by: beliefs, goals, plans and actions. Every
agent of the MAS plays at least one role in an organization. One of the attributes
of a role is a number of protocols, which define the way that it can interact with
other roles." (Coelho, 2006)
Regardless of methods of software testing, quality of test cases is very important,
and directly impacts the quality of the software product. "A very important
consideration in program testing is the design and creation of effective test
cases. Testing, however creative and seemingly complete, cannot guarantee the
absence of all errors. Test-case design is so important because complete testing
is almost impossible; a test of any non trivial program must be necessarily
incomplete. The obvious strategy, then, is to try to make tests as complete as
possible. Given constraints on time and cost, the key issue of testing becomes:
What subset of all possible test cases has the highest probability of detecting the
most errors?" (Coelho, 2006). Here author highlights a very important point of
identifying the subset of critical test cases. It is not possible to execute all the test
cases every time new software built is created. Even with test automation, it is
23
very difficult to create the test data to execute the test cases. Once critical test
cases are identified, they can be executed with every software build as part of
built verification. This ensures software quality is maintained, and decreases the
project cost, by reducing the maintenance cost.
Software Agents are frequently used in e-learning systems, as Cabukovski states
that, "different students would have different content view. An idea of agent
based testing subsystem for estimation of the student’s grade level which is the
main measure for dynamic generation of the content view is presented in the
paper. … The agent based testing subsystem was applied in a distance
educational system for learning mathematics and informatics called MATHEIS."
(Cabukovski, 2006)
"MATHEIS is an educational system for learning mathematics and informatics for
pupils and students in primary and secondary schools. ... The system supports
easy XML based import/export between applications, as well as SCORM and
AICC based one-click import/export of the course metadata and content files.
Students use various types of services at MATHEIS: web based material
treasury, e-mail communication, discussion forums, chat, testing services, etc.; It
supports different learning styles, from self-paced material, to scheduled or
synchronous classes offline or online and collaborative and integrated learning."
(Cabukovski, 2006)
24
Current Research
Intelligent Agents are frequently used, especially in e-learning projects. This
allows applications to simulate human behavior, along with artificial intelligence
component. Although main stream industry has not fully adopted the concept of
agents, significant amount of research work has been done in the academic
world. “The concept of an agent has become important in computer science and
has been applied to a number of application domains, such as electronic
commerce and information retrieval.” (Choi, 1999)
There are few reasons why agents are best candidate for software testing. First,
testing has huge component of repetitive task, which is error prone for humans.
Secondly, it requires some sort of decision making component, along with the
repetitive task. Therefore, it is not possible to fully automate software testing
without using Artificial Intelligence. Based on these reasons, intelligent agents
are best fit for software testing. “Automated tests are helpful in performing tests
quickly and precisely, but they don't replace the human tester, who has to run the
tests, check that all is going satisfactorily, and consider what other items need to
be checked to ensure quality. It still takes considerable time to perform all the
selected test cases with an automated test tool. Therefore, we need a test tool
which replaces the role of a tester, one that plans the test, selects test cases,
and performs regression automatically, and also has the properties of an agent,
such as autonomy, social ability, and intelligence.” (Choi, 1999)
25
Figure 2.1: Test Agent System (Choi, 1999)
Figure 2.1 is "composed of three agents, a “User Interface Agent” which
exchanges information with the tester, a “Test Case Selection & Testing Agent,”
which selects test cases, and a “Regression Test Agent,” which performs
regression tests. Also, there are five modules, the “Information Management,” the
26
”Test Case History Management,” the “Test Item Management,” the “Test
Oracle,” and the ”Test Case Suit”.(Choi, 1999)
Choi further explains the system, and states that "the User Interface Agent
interconnects testers and the Test Agent System, allowing testers to exchange
information through this User Interface Agent. The purpose of this agent is to
search for test items from the test target inputted by the tester according to the
test level, and to see the test results. Once the test target program and
information about the function of that test target are input, the User Interface
Agent plans the testing process. The test process includes steps: identifying the
test level according to object-oriented test processes, looking for test items,
selecting test cases for each test item, performing tests, and reporting on the
results. The User Interface Agent identifies the test items according to the test
level. From the “Test History,” the User Interface Agent reasons whether the
input test target needs regression tests or not, based on the rules which will be
described in the next paragraph. The User Interface Agent neither selects test
cases nor performs regression tests. Because of this, the User Interface Agent
recognizes an agent able to select test cases and perform regression tests, and
communicates with that specific agent". (Choi, 1999)
“The Test Case Selection & Testing Agent is able to recognize consistent test
cases without redundancy autonomously through the knowledge base. During
this process, it has to select a test case that can produce greater test efficiency
27
from the numerous test cases made by the automated test tool. The Test Case
Selection & Testing Agent recognizes the agents or modules that can acquire
test items and test case pools, then secures them by corresponding with the
agents or modules.” (Choi, 1999)
Software Agents are also frequently used in web based system. "With the
prevalence of the Internet and the WWW, Web quality assurance becomes
increasingly important. This makes the testing and evolution especially important.
Automatic and intelligent testing is needed so as to improve the testing efficiency.
Intelligent Agent is a self-contained, autonomous software module in the
distributed systems that could perform certain tasks on behalf of its users. These
characters make Agent rather fit for the Internet and WWW platform, and Agent
has made effective progress in Information Retrieval (IR), user personalize
services, and etc." (Jiang, et. al, 2004)
Without artificial intelligence, software testing cannot be fully automated.
"Testers are also needed in the whole process because the tools have no
intelligent and cannot deal with the abnormity. If there are certain of intelligent
entities to assist the test case generation, selection, and execution, the testing
efficiency will be greatly improved. “(Jiang, et. al, 2004)
Figure 2.2 shows another example where mockup agents are used, using JADE
platform. As Coelho explains that "instead of creating a unit testing tool from
28
scratch to exercise the Test Cases defined in our approach, we decided to
extend JUnit framework to support JADE agents’ tests. The reason for that is to
lower the developers’ learning curve providing a simple, and widely used testing
framework architecture. JUnit was already ported to many other languages in
different paradigms (e.g. CppUnit, NUnit, PyUnit, XMLUnit). All these
frameworks, known as xUnit family of tools, have the same basic architecture."
(Coelho, 2006)
Figure 2.2: Implementing our testing approach upon JADE Platform - a low level design view.
(Coelho, 2006)
29
Summary
The purpose of this study was to identify the reasons why software quality is not
achieved, despite of using all latest Software Testing Methodologies and
recommend possible solution for the problem. Testing is important but
unfortunately never done properly. Despite tools and methodology, quality of
software still takes few releases to achieve expected results. Manual testing
cannot be eliminated, and human error/negligence cannot be resolved. Agents
are the best available solution which can represent the human testing,
eliminating human error.
30
CHAPTER III
METHODOLOGY
Software testing is the selective verification of the computer program, based on
the business requirements and pre-defined test cases. These test cases can be
created dynamically, or can be created manually, using the test scenario data. It
is very crucial that proper test cases are created, and executed every time
related code is updated. However, with code changes, it is very hard to predict
how other components are going to respond, until all components are properly
tested. Repeatedly executing test cases requires resources, which impacts the
overall project budget. Automated Regression Testing assist in maintaining the
product quality and keep the budget under control. Initial investment is required
to create the automated testing for the product, but it pays off in maintenance
phase. "Regression testing is the selective retesting of a software system that
has been modified to ensure that any bugs have been fixed and that no other
previously working functions have failed as a result of the reparations and that
newly added features have not created problems with previous versions of the
software." (Salima, 2007)
Software Agents
As defined by Bellifemine, “an agent is essentially a special software component
that has autonomy that provides an interoperable interface to an arbitrary system
and/or behaves like human agent, working for some clients in pursuit of its own
agenda. … Agents may interact with each other indirectly (by acting on the
31
environment) or directly (via communication and negotiation). Agent may decide
to cooperate for mutual benefit or may compete to serve their own interests.”
(Bellifemine, 2006)
Intelligent Software Agents possess certain key characteristics. They can
perform tasks on behalf of the user, and have an ability to communicate with
other agents. Since agents are autonomous, they can operate without direct user
intervention, and can monitor environment and can directly affect surroundings.
“Agent has such primary behavioral attributes as autonomy, cooperation, and
learning. Thus we find it is possible to apply Agent into Web testing so as to
improve the degree of atomization and intelligence of testing. For example, we
need to obtain the accurate and effective user visiting scenes in Web
performance testing and regression testing, and the main aspect of Agent
application is information retrieval, so it is natural to obtain users’ visiting actions
through Agent and determine virtual users’ actions and testing steps by multi-
Agents’ associations. In addition, Agents have the learning ability, and they can
adjust and improve their functions due to users’ feedback, so we can obtain
objective and exact results in Web usability evolution.” (Jiang, et. al)
An agent "is a programmed system that can handle tasks autonomously with
intelligence under some prescribed rules. To benefit from autonomous control
and reduced running costs, system functions are performed automatically. Agent-
oriented considerations are being steadily accepted into the various software
32
design paradigms. Agents may work alone, but most commonly, they cooperate
toward achieving some application goal(s). “(Salima, 2007). Some of the
common properties of intelligent agents are Sociability, Autonomy, and
Intelligence.
JADE Platform
Java Agent Development Framework (JADE) is a distributed middleware system
with a flexible architecture, which supports the development of complete agent-
based applications. It allows easy extensions to available add-on, most popular,
JADE-LEAP (Light Extensible Agent Platform). JADE was initially developed by
Research and Development department of Telecom Italia, however now its open
source community project, code distributed under LGPL license. JADE is fully
compliant with FIPA specifications; Foundations of Intelligent Physical Agents.
FIPA; formed in 1996, a Swiss based organization, produce software standards
specifications for heterogeneous and interacting agents and agent based
systems. As Bellifemine states about JADE framework that,"the goal is to simplify
development while ensuring standard compliance through a comprehensive set
of system services and agents. JADE can then be considered an agent middle-
ware that implements an Agent Platform and a development framework. It deals
with all those aspects that are not peculiar of the agent internals and that are
independent of the applications, such as message transport, encoding and
parsing, or agent life-cycle. (Bellifemine, 1999)
Following figure shows the FIPA reference model of an Agent Platform.
33
Figure 3.1: FIPA reference model of an Agent Platform (Bellifemine, 1999)
Figure 3.2 shows the Software Architecture of JADE Agent Platform
Figure 3.2: Software Architecture of JADE Agent Platform (Bellifemine, 1999)
34
Agent Communication Language (ACL)
Agent Communication Language (ACL), a proposed standard language for agent
communications. It was proposed by Foundation for Intelligent Physical Agents
(FIPA). JADE platform implements the FIPA-ACL for agent communications.
"Agent communication is based on message passing, where agents
communicate by formulating and sending individual messages to each other. The
FIPA ACL specifies a standard message language by setting out the encoding,
semantics and pragmatics of the messages. The standard does not set out a
specific mechanism for the internal transportation of messages. Instead, since
different agents might run on different platforms and use different networking
technologies, FIPA specifies that the messages transported between platforms
should be encoded in a textual form. It is assumed that the agent has some
means of transmitting this textual form. The syntax of the ACL is very close to the
widely used communication language KQML. However, despite syntactic
similarity, there are fundamental differences between KQML and ACL, the most
evident being the existence of a formal semantics for ACL which should eliminate
any ambiguity and confusion from the usage of the language." (Bellifemine,
1999)
Multi-Agent Based Regression Testing Suite (MARTS): An Overview
There are quite a few testing frameworks available which can be adopted, to
perform the regression testing. Most of them are based on distributed computing
concept. The existing frameworks, which utilize the Agent technologies, use
agents very extensively. Most of the components are designs based on the
35
agents, and organizations are hesitant to adopt, due to the fact that not many
organizations have skill set available to understand software agents. In this
paper, we go back to the basic, and attempt to find out why we need agents for
Regression Testing. "Testability can be defined as the property that measures
the ease of testing a piece of code or functionality, or a provision added in
software so that test plans and scripts can be executed systematically." (Yu,
2007)
In regression testing, one important perceptive of using agent would be to allow
agents test the different test scenarios, and make appropriate decisions at
runtime, to simulate testers. This does not mean that we have to create all
services, and entire framework architecture based on agents. We believe,
answer exists in the heterogeneous solution, where we use distributed computing
concept for the entire framework, and the test cases are executed by the test
scenario agents, which simulate the software testers. We will utilized existing unit
test cases, and integrate them into regression framework, so that test scenario
agents can be scheduled for every software release.
MARTS, the proposed framework uses intelligent agents to execute the test
scenarios, which are based on the existing unit test cases. It supports test case
versioning, reporting, logging, and results analysis service, which assists test
administrator to make better decisions, and predicts overall product quality.
Although Test Data Generator is integral part of the solution, it is loosely coupled,
and encouraged to use off-the-shelf third party solutions.
36
MARTS Software Architecture
The first main component is the graphical user interface. Testing administrator
will use this interface to variety of tasks. All user preferences and testing history
will be stored in the framework database. There were two options to store the
framework related data. First was to use xml files, and update them each time
user make changes. Second option, to store all the information in the separate
database, and entire user interface is loaded based on the framework database.
Second option was selected based on few reasons. As requirement for this
framework, user must have access to testing history, scenario agent’s schedules,
common reports, and list of available unit tests along with their metadata, which
includes versioning, parameter lists, expected behavior, and possible actions
based on the results. Since this is only high level design paper, not all
functionality has been discussed in detail; however architecture supports
scalability and flexibility for future enhancements, in order to meet the overall
vision of this framework.
37
Figure 3.3 MARTS Framework Architecture
Framework Controller is the central piece of this framework. Like name suggests,
word controller comes from the concept of Model View Controller (MVC) design
pattern.
Framework Controller. User performs actions using Suite’s user interface, and
framework controller updates user interface based on user’s actions. It stores
user preferences, test results, test scenarios, and other related information in
framework database. Figure 3.1 shows the interaction between controller, user
interface and framework database.
38
Figure 3.4 Framework Controller
User will be able to create test scenario, by selecting available unit tests.
Appropriate unit tests versions will be available for user to select, along with the
brief description. Once test scenario is created, user will have an option to
schedule to test scenario to be executed, or can be executed on demand. When
execution is required, framework will create the test scenario agent to execute
the test scenario. Agent will be able to make its own decisions based on the
results of each test case. At any given time, there can be multiple agents running
testing different test scenarios. Figure 3.5 shows framework interaction with
multiple test agents.
Figure 3.5 Agents Communication with Framework Controller
39
One major problem with test scenarios is it usually passes when executed
individually, however when executed along with other test scenarios, it shows
concurrency and performance issues. Multiple test agents test the application
simultaneously using different test scenarios. Software Agents are based on the
BDI (Belief, Desire, and Intension) software model. As explained on Wikipedia,
"Superficially characterized by the implementation of an agent's beliefs, desires
and intentions, it actually uses these concepts to solve a particular problem in
agent programming. In essence, it provides a mechanism for separating the
activity of selecting a plan (from a plan library) from the execution of currently
active plans. Consequently, BDI agents are able to balance the time spent on
deliberating about plans (choosing what to do) and executing those plans (doing
it). A third activity, creating the plans in the first place (planning), is not within the
scope of the model, and is left to the system designer and programmer."
(Wikipedia)
Figure 3.6 Reporting Component with Framework Controller
Another important functionality of MARTS is reporting component. On demand
reports are created, based on the user defined criteria. All information is stored in
40
the framework database, for future retrieval. This allows user to store the
common reports, and does not have to specify the report criteria again. Reports
can be scheduled, which will execute and user will be notified upon completion.
Here we get two benefits. First, user will receive the information regularly, and
does not have to request for information. For example, in Project Management
software’s, Project managers can execute reports to analyze the overall project
issues, however, most of them either forget to run them, or they don’t know if
certain type of report can be generated. Vision for MARTS is to create the
reports, and information when required. As future extension, agents will have an
ability to initiate the report, to inform user of potential issues that were
encountered during execution. These will be mostly issues that do not fail the test
case; however, it gives enough information regarding possible future issues.
Figure 3.7 Logging Component with Framework Controller
Like all software systems, MARTS logs all agent activities, for troubleshooting
purposes. Logs will be captured in normal text files, however, user will have
access to the logs using user interface.
41
Figure 3.8 Framework Controller with Test Data Generator
For each test scenario, appropriate test data will be required. A stand along Data
Generator service will be integrated with the MARTS, to provide the test data
based on the test scenario. Test Data Generator can either be custom built,
which requires construction for scratch, or can be off-the-shelf test data
generator. Creating test data is a complete topic by it self, therefore details are
left for future enhancements. Framework Controller will create test data for all
new test scenarios, and update data for existing scenarios. Test Data Generator
Service will be integrated with the application database, in order to provide the
appropriate data. Static data causes issues, mostly when database is updated.
Integration of the service with the application database ensures that schema
changes are reflected at execution time of test scenarios.
Figure 3.9 Framework Controller with Test Results Analyzer Service
42
Like Test Data Generator service, Test Results Analyzer will be stand along
service, to analyzes the test results, and provide appropriate feedback. Test
Results analyzer will interact with framework controller, and scenario agents.
Test Scenario Agent. Test Agent will be created by the framework controller at
execution time. For each test scenario, controller will create the agent. The base
structure will be the same, and execution steps will be provided by the
framework. However, for future enhancement, MARTS will have different types of
agents, with different default behavior. User will have an option to select the
agent type from user interface. For example, aggressive type agent will perform
the testing, using more strict rules, and have proactive behavior when interacting
with our agents, whereas defensive type agents will adopt more friendly behavior
when interacting with surrounding agents.
Figure 3.10 Framework Controller with Test Scenario Agent
Once test scenario is completed, agent will update the framework controller,
which will update framework database, and update user accordingly.
43
Figure 3.11 Test Scenario Agent with Test Data Generator
Test Scenario agent will interact with the test Data Generator service to retrieve
the test data for the unit tests. Every test scenario is assigned unique test id. All
unit tests are also assigned unique ids, which represent unit test id along with the
version number. Agent will send the information to test data generator, which
provides the appropriate data for the test agent.
Figure 3.12 Test Scenario Agents with Logging Component
All agents’ activities are logged by logging component, which updates framework
controller.
Figure 3.13 Test Scenario Agent with Unit Test Wrapper
44
Since agents will be using unit tests, it requires certain wrapper, such that an
agent does not have to be specific to test cases. A generic unit test wrapper will
be created to make sure agents can call the wrappers, and can use unit tests,
without getting into specific on JUnit or NUnit.
Figure 3.14 Test Scenario Agent with Test Results Analyzer
Test Scenario agent will frequently interact with the Test Result Analyzer service
to get test results validated. For each test results, a unique id will be created, and
stored in framework database, for future retrieval and tracking purposes.
Reporting component will use that information to create overall progress.
Reporting Component will also be able to generate the quality analysis report, to
analyze the rate of bugs reported by agents, and identify the overall health of the
software product. Since this is regression framework, agents will be scheduled to
run all the times, except for software build deployment and server maintenance.
Unit Test Wrapper. Test Scenario Agents will use the Unit Tests, to execute
the given test scenario. Unit Test Wrappers will be used to make sure generic
agents can be used, for all test scenarios. Test Wrappers can be a service, or
45
web service, depending on the application requirement. For complex
applications, it is better to have stand-alone unit tests wrappers, with no
dependencies to the agents.
Test Data Generator. Test Data Generator service will create test data, which
will be used by the scenario agents. Test Data can be predefined, or can be
create at runtime. This will require more construction time, based on the system’s
requirement; this can be off-the-shelf component, which can be easily integrated
with the MARTS framework. For example, DATA Generator from SQLEdit
provides similar functionality.
Figure 3.15: SQLEdit DTM Data Generator (SqlEdit, 2009)
46
Another third party component which can be potentially used is Turbo Data.
Figure 3.16: TurboData Data Generator (Turbodata, 2009)
In-house data generator will require more resources, whereas third party
components can have some integration issues.
Test Results Analyzer. Test Results Analyzer will receive the input from
agent, and create the analyses report based on pre-defined set of rules. Most of
the system requirements will be validated in this service. Agents will interact with
this, and can make decisions based on the results. For example, if critical test
47
case fails, agent may decide to terminate the test execution. Agent’s decision will
be based on the feedback from Test Results Analyzer.
An Online Shopping Store - Example
For an online shopping store, there can be multiple roles agents can play to test
the system. For example, user creates four test scenarios, first time customer,
existing customer, vendor, and web site administrator. All four agents can start
their test cases simultaneously. Moreover, user can create multiple instances of
same test scenarios, therefore at any given time; we could have n number of
customer agents, shopping online, while few vendor agents can validate their test
cases. Intelligent Agents provides much flexibility, and better simulate the human
testing.
Test Agent - Customer: Existing Customers will already have their accounts
created on the web portal, however for new customers; accounts will have to be
created. For all customers, once there accounts are setup; they will have there
preferences stored, which should customize the web look and feel based on their
choices. Test Agent, would be able to validate the preferences settings, and
continue to verify the available information for data accuracy. Using agent based
testing, all different combinations can be tested, and ensure that services
available on the web portal continue to respond properly, after each software
build. Customer agents will help in analyzing the performance testing, since all
scenario agents can use the web services simultaneously, and simulate the real
environment.
48
Test Agent – Vendor: Vendor agent should continuously monitor the stock
quantities for their respective products, and should be able to bid for the supplies,
as soon as it’s required. Vendor agent would load the vendor information based
on the vendor account information, and simulate the concurrent testing for
application, while customers and vendors are all participating at the same time.
Vendor agent should bid aggressively, and should ensure its goals are met, as
compared to the portals customers. Vendor would also get the runtime
information that is released by the site administrator.
Test Agent - Site Administrator: Site administrator, aside for monitoring the
overall site health and stick piles, should also compare the bids for from different
vendors and approve the lowest quotes, based on the quotes from different
vendor agents. It creates the runtime inventory of items and goods that are not
available on the site, and based on the vendor agent responses, can enhance
the web portal at runtime. For example, lots of customer agents search for the
same electronic item which is not available on the site. Site Administrator agent
would query the vendor agent for the quote. Once response is positive, it will
update the site information, and place orders to the vendor. This will help to
simulate how site administrator will monitor and place orders, and how vendor
will response. In order to perform these testing manually, it will require lots of test
cases, and large amount of human effort to test all possible scenarios. With
agent based-testing, it can run for few days, and log the errors as encountered.
49
CHAPTER IV
ISSUES, CHALLENGES, AND TRENDS
Software Testing has always been the most neglected phase of software
development life cycle. Inadequate resource allocation for software testing phase
tends to be one of the top reasons for high maintenance cost. Business Users for
large Enterprise Applications, typically mainframe users, spend most of the time
understanding new technology, and tends to compare the usability with
mainframe application interface. For example, on mainframe, users perform
specific sequence of steps to perform their task. With newer technology, steps
will definitely change; however, business users tend to accept the change with
the feeling as if IT dictating business. Inexperienced Project Manager and over-
technical IT Senior Management; usually unable to sell the idea appropriately,
results in majority of the user/acceptance testing time, spend in unproductive
meetings. Since technology cannot address these issues, but can help by
automating the testing, and validate the core business requirements.
Agent-based testing can simulate the user test; however, it depends on the
design of the software testing agent. It requires significant amount of time and
effort, to create an intelligent agent, which can make its own decisions, based on
the environment, and interaction with other agent. JADE framework is flexible,
and more important, easy to learn for new developers. More academic projects
tend to use JADE, due to fact that it is open source, and completely written in
JAVA.
50
Although Agent-based systems are more flexible, and scalable, it is quite
complex to develop, and hard to find resources with appropriate skills.
Implementing security model for agent-based system is more complex and error
prone, compared to non agent-based system.
Proposed solution attempts to address common problem of the software industry.
It uses Unit Tests, which are created by developers in construction phase, and
testing scenarios, which are created by tester’s in collaboration with business
users. Since test scenario agents can be scheduled for regular testing, tester’s
gets regular analysis feedback, to ensure software quality is maintained.
51
CHAPTER V
CONCLUSIONS AND RECOMMENDATIONS
Conclusions
In this paper, an agent based regressing testing suite was proposed. It does not
only use agent for the sake of the technology, rather it integrates the benefits of
intelligent agents with the distributed systems. Its very high-level design, and
more detail research will be required to create a product, which can be used for
regression testing mid-size and large enterprise applications. Of all the available
options, Agent-based testing is the better, due to flexibility, autonomous in
decision making, and scalability. Poor quality software is released to customers,
due to excessive schedule pressure, lack of training programs for developers,
lack of code inspections due to time constraints and failure to address defects in
timely manner. With automated testing, regression testing can be performed
regularly, and quality of software is maintained throughout the project.
Proposed solution utilizes the benefits of the agent-based systems, along with
the proven solutions of distributed computing. In proposed solution, only test
scenarios are executed using agents, rather than entire agent-based system.
There were few reasons, why the approach was adopted. Firstly, skilled
developers for agent-based programming are not easily available. Our main goal
was to simulate human testing, along with some level of intelligence and
autonomous decisions making capability. Creating completely agent-based
system for the proposed solution will increase development overhead. Our goal
52
was to achieve simplified solution, using existing technologies, and address real
industry problem, i.e. unacceptable software quality, despite higher budget
allocated for the project.
Suggestions for Further Research
This paper is very high-level design proposal, and requires more detail analysis
of each proposed components. Major work will be required for creating
algorithms for intelligent agents, and agent templates, to allow user to select the
type of agent, to execute the given test scenario. Future implementation of
prototype of the proposed framework – MARTS, and create execute some beta
testing, to create survey results, based on end user’s feedback.
53
REFERENCES
Jones T.C. “Software Cost Estimation in 2002”, CrossTalk June 2002.
Qi, Y., Kung, D., and Wong, E. 2005. An Agent-Based Testing Approach for Web
Applications. In Proceedings of the 29th Annual international Computer
Software and Applications Conference - Volume 02 (July 26 - 28, 2005).
COMPSAC. IEEE Computer Society, Washington, DC, 45-50. DOI=
http://dx.doi.org/10.1109/COMPSAC.2005.42
Xu, B., Xu, L., Jiang, J. 2004. Applying Agent into Web Testing and Evolution.
ISSN 0302-9743 (Print) 1611-3349 (Online) . Volume Volume 3032/2004 ISBN
978-3-540-21988-0 Pages 794-798. DOI=
http://dx.doi.org/10.1007/b97162
Choi, J., Choi, B. 1999. Test Agent System Design Volume: 1, On page(s): 326-
331 vol.1 ISBN: 0-7803-5406-0 INSPEC Accession Number: 6422443
DOI= http://dx.doi.org/10.1109/FUZZY.1999.793259
Liu, L., Zhou, X., Gu, J., Yang, Z. 2003. Agent-based automated compatibility
software test for NLSF. Machine Learning and Cybernetics, 2003
International Conference on Volume 4, Issue , 2-5 Nov. 2003 Page(s):
1986 - 1989 Vol.4 DOI= http://dx.doi.org/10.1109/ICMLC.2003.1259828
Kung, D. 2004. Agent-Based Framework for Testing Web Applications. In
Proceedings of the 28th Annual international Computer Software and
Applications Conference - Workshops and Fast Abstracts - Volume 02
54
(September 28 - 30, 2004). COMPSAC. IEEE Computer Society,
Washington, DC, 174-177.
Dhavachelvan, P. Uma, G.V. 2005. Complexity Measures for Software
Systems: Towards Multi-Agent Based Software Testing. Intelligent
Sensing and Information Processing, 2005. Proceedings of 2005
International Conference on Publication Date: 4-7 Jan. 2005. On page(s):
359- 364 ISBN: 0-7803-8840-2 INSPEC Accession Number: 8703524
DOI= http://dx.doi.org/10.1109/ICISIP.2005.1529476
Xu, D., Li, H., and Lam, C. P. 2005. Using Adaptive Agents to Automatically
Generate Test Scenarios from the UML Activity Diagrams. In Proceedings
of the 12th Asia-Pacific Software Engineering Conference (December 15 -
17, 2005). APSEC. IEEE Computer Society, Washington, DC, 385-392.
DOI= http://dx.doi.org/10.1109/APSEC.2005.110
Cabukovski, V. E. 2006. An Agent-Based Testing Subsystem in an E-Learning
Environment. In Proceedings of the 2006 IEEE/WIC/ACM international
Conference on Web intelligence and intelligent Agent Technology
(December 18 - 22, 2006). WI-IATW. IEEE Computer Society,
Washington, DC, 622-625. DOI= http://dx.doi.org/10.1109/WI-
IATW.2006.34
Zhou, Z. Q., Scholz, B., and Denaro, G. 2007. Automated Software Testing and
Analysis: Techniques, Practices and Tools. In Proceedings of the 40th
Annual Hawaii international Conference on System Sciences (January 03
55
- 06, 2007). HICSS. IEEE Computer Society, Washington, DC, 260. DOI=
http://dx.doi.org/10.1109/HICSS.2007.96
Briand, L.C. A Critical Analysis of Empirical Research in Software Testing.
Empirical Software Engineering and Measurement, 2007. ESEM 2007.
First International Symposium on Publication Date: 20-21 Sept. 2007 On
page(s): 1-8 Location: Madrid, ISSN: 1938-6451 ISBN: 978-0-7695-2886-
1 INSPEC Accession Number: 9875968 DOI=
http://dx.doi.org/10.1109/ESEM.2007.40
Yu, L., Xu, L., Wang, G., Chi, C., Xiao, W., and Su, H. 2007. Testability and Test
Framework for Collaborative Real-Time Editing Tools. In Proceedings of
the Seventh international Conference on Quality Software (October 11 -
12, 2007). QSIC. IEEE Computer Society, Washington, DC, 322-327.
Salima, T. M., Askarunisha, A., and Ramaraj, N. 2007. Enhancing the Efficiency
of Regression Testing through Intelligent Agents. In Proceedings of the
international Conference on Computational intelligence and Multimedia
Applications (ICCIMA 2007) - Volume 01 (December 13 - 15, 2007).
ICCIMA. IEEE Computer Society, Washington, DC, 103-108. DOI=
http://dx.doi.org/10.1109/ICCIMA.2007.182
Ostroff, J., Paige, R., Makalsky, D., Brooke, P. E-Tester: a Contract-Aware and
Agent-Based Unit Testing Framework for Eiffel. Journal of Object
Technology, vol. 4, no. 7, September–October 2005, pp. 97–114, Web
Site: http://www.jot.fm/issues/issues 2005 09/article4
56
Javed, A. Z., Strooper, P. A., and Watson, G. N. 2007. Automated Generation of
Test Cases Using Model-Driven Architecture. In Proceedings of the
Second international Workshop on Automation of Software Test (May 20 -
26, 2007). International Conference on Software Engineering. IEEE
Computer Society, Washington, DC, 3. DOI=
http://dx.doi.org/10.1109/AST.2007.2
Holcombe, M., Coakley, S., Smallwood, R. A General Framework for Agent-
based Modelling of Complex Systems Department of Computer
Science,University of Sheffield North Campus, Broad Lane, Sheffield, S3
7HQ, UK
Coelho, R., Kulesza, U., von Staa, A., and Lucena, C. 2006. Unit testing in multi-
agent systems using mock agents and aspects. In Proceedings of the
2006 international Workshop on Software Engineering For Large-Scale
Multi-Agent Systems (Shanghai, China, May 22 - 23, 2006). SELMAS '06.
ACM, New York, NY, 83-90. DOI=
http://doi.acm.org/10.1145/1138063.1138079
Pugh, W. and Ayewah, N. 2007. Unit testing concurrent software. In Proceedings of the
Twenty-Second IEEE/ACM international Conference on Automated Software
Engineering (Atlanta, Georgia, USA, November 05 - 09, 2007). ASE '07. ACM,
New York, NY, 513-516. DOI=
http://doi.acm.org/10.1145/1321631.1321722
57
Wikipedia. Belief-Desire-Intention software model. Retrieved on March 21, 2009.
Web site: http://en.wikipedia.org/wiki/BDI_software_agent
SQLEdit DTM Data Generator. 2009. Retrieved on March 27, 2009. Web site:
http://www.sqledit.com
TurboData Data Generator. 2009. Retrieved on March 28, 2009. Web site:
http://www.turbodata.ca
Stutzke R.D. “Software Estimating Technology: A Survey”, CrossTalk May 1996.
Conte S.D., Dunsmore H.E. and Shen V.Y. “Software Engineering Metrics and
Models”, Benjamin/Cummings Publishing Company (1986).
Boehm B. “Software Engineering Economics”, Prentice Hall (1981)
Bellifemine, F., Poggi, A., Rimassa, G. 1999. JADE – A FIPA-Compliant Agent
Framework. Proceedings of the Practical Applications of Intelligent
Agents.
Bellifemine, F., Caire, G., Greenwood, D. 2007. Developing Multi-Agent Systems
with JADE. John Wiley & Sons, Ltd. ISBN: 978-0-470-05747-6.