Achieving the Full Potential of Test Automation

7/28/2019 Achieving the Full Potential of Test Automation

1/8

For more information, please see the LogiGear website at www.logigear.com LogiGea r Corporation, 2015 Pioneer Court, Suite B, San Ma teo, CA 94403 USACopyright 2011, LogiGea r Corporation. All Rights Reserved.

White paperPage 1 of 8

Achieving the Full Potential of Test Automation

1. A b stra c t ........................................................................................................................................................1 2. The Be ne fits o f So ft w a re Te st A ut o m a tio n ................................................................................................1 3. Pitfa lls: W hy Te st A ut o m a tio n Pro jec ts Fa il to A c hie ve The ir Po te nt ial ...................................................2 4. G e ne ra tio ns: Te st A ut o m a tio n Evo lutio n ..................................................................................................3 5. A c tio n -Ba se d Te stin g : A Pro ve n A p p ro a c h ..............................................................................................4 6. C o n c lusio n ...................................................................................................................................................8

Diagrams:Figure 1: The A BT Fram e w o rk ................................................................................................................................. 1Figure 2: Ac tio n Base d Te st ing in Te stA rc hi tec t TM ............................................................................................... 7

1. Abstract

Software test automation has the capability to decrease the overall cost of testing and improvesoftware quality, but most testing organizations have not been able to achieve the full potential of test automation. A number of groups have implemented test automation plans but are disgardedand become expensive shelfware. Often such teams continue their automation efforts yetburdened with huge costs in maintaining large suites of a utomated test scripts that are of questionable value.

This paper will first discuss key benefits of software test automation, and then examine the most

common techniques used to implement software test automation. It will then analyze prevalentrea sons why test automation efforts fail to meet their potential. Finally, this paper conc ludes with anexamination in how using a keyword-d r iven approach to test automation allows organizations toavoid problems inherent in other approaches and benefit from test automation.

Action Based Testing TM , the latest methodology from the original architect of the keyword method,and the TestArchitect TM toolset will be presented as proven real-world examples of why testautomation is the optimal solution.

2. The benefits of software test automation

Most software development and testing organizations are well aware of the benefits of testautomation. A quick glance at the Web sites of any test automation tool vendor will point out anumber of the key benefits of test automation. Some of these benefits include:

Re d u c e d t e st e x e c u t io n t im e a n d c o st : Automated tests take less time to execute than manual testsand can generally execute unattended. A tester must simply start the test and then analyze theresults when the test is completed.


2/8



In c re a se d t e st c o v e ra g e o n e a c h t e st in g c y c l e : Automated tests can allow testing teams to executelarge volumes of tests against each build of their application, achieving a level of coverage thatwould not be possible with manual testing. This increased coverage can help teams uncover morequickly than manual testing. Test automation can allow teams to test more features in each cycle(breadth), and a lso to test features using more permutations of inputs (depth).

In c re a se d v a lu e o f m a n u a l t e st in g e f f o rt : So long as applications are meant for human end users, testautoma tion will never entirely rep lac e the need for human testers. Human testers will instantly noticesubtle bugs that are a lmost never detec ted by test automation, particularly usability bugs.Automated test tools are not instinctual and discover bugs using exploratory and a d hoc testingtechniques. By freeing manual testers from having to execute repetitive, mundane tests, testautomation grants them to use their crea tivity, knowledge and instincts to discover important bugs.

3. Pitfalls: Why Test Automation Projects Fail to Achieve Their Potential

Despite the clear benefits of test automation, many organizations are unable to build effective testautomation programs. Test automation is seen as a costly effort finding few bugs and of questionableworth.

There are a number of reasons why test automation efforts are unproductive:

Po o r q u a lit y o f t e st s b e i n g a u t o m a t e d

Mark Fewster explains this problem very well:

It doesnt matter how clever you are at automating a test or how well you do it, if the testitself achieves nothing then a ll you end up with is a test that achieves nothing faster. [Fewster,Software Test Automation, 1.1, (Addison Wesley, 1999)]

Many organizations simply focus on taking existing test cases and converting them into automatedtests. There is a sense that if 100% of the manual test cases can be automated, then the testautomation effort will be a success.

In trying to achieve this goal, many orga nizations may have automated many of their manual tests,but found it to be huge loss in time and money discovering a few bugs. This can be the plain factthat a poor test is a poor test, whether it is executed manually or automatically.

La c k o f g o o d t e st a u t o m a t io n f ra m e w o rk a n d p ro c e ss

Many teams acquire a test automation tool and begin automating as many test cases as possible,with little c onsideration of how they can structure their automation in such a way that it is scalable

and maintainable. Little consideration is given to ma naging the test scripts and test results, crea tingreusable functions, separating data from tests, and other key issues which allow a test automationeffort to progress successfully. After some time, the team rea lizes that they have large quanitities of test scripts, countless separate test result files combined with additional work of maintaining theexisting scripts and continuing to automate new scripts. Ultimately, the organization requires a largertest automation team with higher costs resulting in no additional benefit.

In a b ilit y t o a d a p t t o c h a n g e s in t h e sy st e m u n d e r t e st


3/8



As the team proceeds towa rds their goal of automating a s many existing test cases as possible, theyoften dont consider what will happen to the automated tests when the a pplication under test (AUT)under goes a significa tion change.

In lacking a well conceived test automation framework that considers how to handle c hanges to thesystem under test, teams often find tha t the majority of their test scripts need ma intenance. Theoutda ted scripts will usually result in skyrocketing numbers of fa lse negatives, since the scripts are nolonger finding the behavior they are programmed to expec t.

With teams hurrying to update the test scripts to account for the changes, project stakeholders losefaith in the results of the test automation. Minimal value in test automation causes a dec ision to scrapthe existing test automation effort and start from scratch, using a more intelligent approach that willproduce incrementally better results.

4. Generations: Test Automation Evolution

Software test automation has evolved through several generations of tools and techniques:

Capture/ playba ck tools rec ord the actions of a tester in a manual test and allow tests to be rununattended for many hours each day, greatly increasing test productivity and eliminating themindless repetition of manual testing. However, even small changes to the software under testrequire that the test be rec orded manually aga in. Therefore this first generation of tools is not efficientor scalable.

Scripting, a form of programming in computer languages specifically developed for software testautomation, alleviates many issues with capture/playback tools. However, the developers of thescripts must be highly technica l and spec ialized programmers who work in isolation from the testersactually performing the tests. In add ition, scripts are best suited for GUI testing and dont lendthemselves for embedded, ba tch, or other forms of systems. Finally, as cha nges to the software under

test require complex changes to the associated automation scripts, maintenance of the ever-largerlibraries of automation scripts becomes an overwhelming challenge.

Data-driven testing is often considered individually as an important development in test automation. This approach simply but powerfully sepa rates the automation scripts from the da ta for input andreturned from the software under test. This allows the data to be prepared by testers without relyingon automation engineers, and vastly increases the possible variations and amounts of data that canbe used in software testing. This breaking down of the problem into two pieces is very powerful. Whilethis approach greatly extends the usefulness of scripted test automation, the huge maintenancechores required of the automation programming staff remain.

Keyword-based test automation compartmentalizes work even further in an advanced, structuredand elegant approach. This reduces the cost and time of test design, automation, and execution byallowing all members of a testing team to foc us on what they do best. Using this method, non-tec hnical testers and business analysts can develop exec utable test automation using keywordsthat represent ac tions rec ognizable to end-users, such as login, while automation engineers devotetheir energy to coding the low-level steps that make up those actions, such as click, find text boxA in window B, enter UserName, etc. Keyword-based test design can actually begin based ondocuments developed by business analysts or the marketing department, before the final details of


4/8



the AUT are known. As the test automation proceeds, bottlenecks are removed and the expensivetime of highly trained professionals is used effectively.

The keyword method cost benefits become even more apparent as the testing continues. When thesoftware under test undergoes changes, revisions to the test and to the automation scripts arenecessary. Organizing test design and test automation with the keyword framework, eliminates timepreviously alloc ated to maintaining large libraries of scripts and rewriting entire scripts anew aftermajor changes to the software under test. With the keyword method, the necessary changes are farfewer. Many changes do not require new automation and can be completed by non-technicaltesters or business analysts. When required, changes to automated keywords can be completed byautomation engineers without affec ting the rest of the test.

Hans Buwalda, Chief Architect at LogiGear Corporation, developed the keyword automationconc ept and first presented this subjec t to the software testing community in 1994. Mr. Buwaldabegan implementing his ideas in Europe throughout the rest of the 1990s with the TestFrame TMmethod and tool, and ultimately continuing its development as Action Based Testing TM in the USA. Thismethod is the foundation of LogiGears test automation framework, TestArchitect TM , which not onlyorganizes scripting around keywords, but a lso offers built-in ac tions that make it possible to automate

many tests without scripting of any kind.

5. Action Based Testing: A Proven Approach

Action Based Testing (ABT) provides apowerful framework for organizing testdesign, automation and executionaround keywords. In ABT keywords arecalled actions to make the conceptabsolutely clear. Actions are the tasks tobe executed in a test. Rather than

automating an entire test as one longscript, an automation engineer canfocus on automating actions asindividual building-blocks that can becombined in any order to design a test.Non-technical test engineers andbusiness analysts can then define theirtests as a series of these automatedkeywords, and execute their testsautomatically without creating anyadditionally code.

Traditional test design begins with awritten narrative that must beinterpreted by each tester orautomation engineer working on the test. ABT test design takes plac e in a spreadsheet, with ac tionslisted consecutively in a clear well-organized sequence. Actions, test data and any necessary GUIinterface information are stored in their own spreadsheets from which they can be called by themain test module. Tests are then exec uted from within the sprea dsheet, using TestArchitects built-inautomation or a custom-build test harness.

Figure 1: The ABT Framework


5/8



To achieve the true power of Ac tion Based Testing, it is important to use high-level ac tions wheneverpossible in test design.

High-level actions are understandable by those familiar with the business logic of the test. Forexample, when the user inputs a number they system makes a mortgage calculation or connects toa telephone. A good high-level action may not be specific to the system under test. Enter order is agood high-level step that can be used generically to refer to specific low-level steps that take placein many tests or many different a pp lications.

Automation is then completed through the coding of low level actions. TestArchitect usually providesall the low-level actions necessary through its built-in automation feature, so there is typica lly no needto write any additional code. Crea ting the high-level action required by the test design involvesdragging-and-dropping a few low-level actions to crea te that high-level ac tion. The low-level ac tionsbehind enter order would be the specific steps needed to complete that action via variousinterfaces such as HTML, the Windows, command line, etc. An example of a low-level action wouldbe push button.

Whenever coding by an automation engineer is required, breaking this work down into reusable low-

level actions saves much time and money by making future code changes unnecessary even whenthe software under test undergoes major revisions. A reshuffling of actions is usually all that is required .If more c oding is necessary, it involves only rewriting in d iv id u a l a c t io n s rather than revision of entire te st sc rip ts and the resulting a ccumulation of a vast library of o ld automation.

Action Based Testing allows testing teams to create a much more effective test automationframework, overcoming the limitations of o ther methods.

Full Invo lvem e nt o f the Te st ing Te a m in Te st Auto m a tion

Most testing teams consist primarily of people who have strong knowledge of the applica tion undertest or the business domain, but do not have a background in programming. The team memberswho are fulfilling the role of test automation engineer are often people with a software developmentor computer science background, but without a strong understanding of testing fundamentals, thesoftware under test, or the business domain.

Action Based Testing allows b o t h types of team members to contribute to the test automation effortby allowing each person to leverage their unique skills to create effective automated tests. Testersdefine tests as a series of reusable high-level ac tions. It is then the task of the a utomation engineer todetermine how to automate the necessary low-level actions and combine them to produce therequired high-level actions, both of which can often be reused in many future tests. This approachallows t e st e rs t o f o c u s o n c re a t in g g o o d t e st s, w h ile t h e a u t o m a t io n e n g in e e rs f o c u s t h e t e c h n ic a l c h a lle n g e o f im p le m e n t in g a c t io n s.

Signi fica n t Red uc t ion o f Test Au tom a t ion M a in ten a nc e

Many organizations build a significa nt test automation suite with traditional methods and begin tosee benefits but encounter more maintenance efforts when the application changes. Many testautomation teams spend more time maintaining their existing tests than actually crea ting new tests.

This high maintenanc e burden is due to the fact that automated tests are highly dependent on the


6/8


7/8



Figure 2: Action Based Testing in TestArchitect TM

Fa c ili ta tes Test Auto m a tion Stra teg y

Many testing teams dive into test automation without first considering how they should approach testautomation. A very typical approach is to acquire a test automation tool, and then try to startautomating as many existing test cases as possible. More often than not, this approach is noteffective.

Action Based Testing provides a framework that integrates the entire testing organization insupporting effective test automation. Business analysts, test engineers, automation engineers, testleads and QA managers all work within the framework to complete test planning, test design, testautomation and test execution. With the right framework in place, the organization can respondmost effectively to everything from marketing requirements to software development changes

Ena b le Effec t ive Co l la b orat ion b y Dist ribu ted Tea m s

With testing teams often distributed to low-cost areas across the country and around the world, thechallenge of sharing information, sharing test libraries and sharing automa tion libraries is multipliedmany times over. Ac tion Based Testing provides a strategic framework for orga nizing tests with a veryclear structure enabling a strong measure of control over the disruption caused by distance and timezone differences. TestArchitect TM , as a test automation frameworks supporting the Action Based

Testing methodology, takes this to the next level by remotely sharing da tabase repositories of testmodules, actions and other test assets, and provides clear control and reporting to managers of access, changes and results.


8/8



6. ConclusionAs with other areas of software development, the true potential of software test automation isrealized only within a framework that provides true scalable structure. Since its introduction in 1994,the keyword based method of test automation has become the dominant approach in Europe andis now taking USA by storm prec isely because it provides the best way to achieve this goal.

Ac tion Based Testing offers the latest innovations in keyword-driven testing from the original architec tof the keyword concept. Test design, test automation and test execution are all performed within aspreadsheet environment, guided by a method focused on an elegant structure of reusable high-level actions.

TestArchitect, a test automation framework from LogiGear with features ranging from actionorganization to globally distributed team management, offers the full power of Action Based Testingto the entire testing organization including business analysts, test engineers, automation engineers,test leads and managers.

Date post:	03-Apr-2018
Category:	Documents
Upload:	auxilia-richard
View:	217 times
Download:	0 times

Achieving the Full Potential of Test Automation

Documents