Post on 30-Dec-2015
transcript
AUTOMATED TEST AUTOMATED TEST GENERATION GENERATION
Muhammed İ. KALKANMuhammed İ. KALKAN
Automatic Test Generation (ATG)Automatic Test Generation (ATG)
IndexIndex What is ATG?What is ATG? ATG TypesATG Types Why do we need ATG?Why do we need ATG? What are the main benefits of ATGWhat are the main benefits of ATG ATG MethodsATG Methods Testing MethodologiesTesting Methodologies Automatic Test Pattern GenerationAutomatic Test Pattern Generation Relevant ExamplesRelevant Examples
What is ATG?What is ATG?
Testing an engineered system is the most important part of the product revealed.
Engineers are working on and putting alot of effort into this issue.
Since testing process is required for all obtained engineering products, engineers decided to develop an automated test mechanism instead of generating test cases manually.
What is ATG?What is ATG?
Automated Test Generation includes test cases , scenarios and the evalution of the obtained result such as passed and failed.
Given inputs and obtained outputs are evaluated automatically.
The Main Idea : TEST COMPLETELY TEST COMPLETELY
AUTOMATICALLYAUTOMATICALLY
ATG or ATPG?ATG or ATPG?
ATG is an engineering subject which ATG is an engineering subject which can be applied to both Computer can be applied to both Computer Sciences and other engineering Sciences and other engineering specialities like Electronics or specialities like Electronics or hardware circuits.hardware circuits.
ATG is a term which is generally ATG is a term which is generally used for Computer Sciences used for Computer Sciences especially software testing.especially software testing.
On the other hand for hardware On the other hand for hardware testing Automatic Test Generation is testing Automatic Test Generation is usually referred as Automatic Test usually referred as Automatic Test Pattern Generation.Pattern Generation.
ATG or ATPG?ATG or ATPG?
ATPG ATPG is defined as follows:is defined as follows: ATPGATPG is an electronic pattern is an electronic pattern
automation method/technology used automation method/technology used to find an input (or test) sequence to find an input (or test) sequence
When applied to a digital circuit, it When applied to a digital circuit, it enables testers to distinguish enables testers to distinguish between the correct circuit behavior between the correct circuit behavior and the faulty circuit behavior and the faulty circuit behavior caused by defects. caused by defects.
ATG…….But WHY??ATG…….But WHY??
As years have passed,computer hardwares As years have passed,computer hardwares and its capabilities grow bigger and bigger. and its capabilities grow bigger and bigger.
Result 1: Software types and their Result 1: Software types and their capabilities are varied.capabilities are varied.
Result 2: New software demand on market Result 2: New software demand on market for jobs.for jobs.
Result 3: Bigger income ,more firms to Result 3: Bigger income ,more firms to compete for market ration.compete for market ration.
Result 4: Requirement for strong,bug free Result 4: Requirement for strong,bug free and developed in small amount of time and developed in small amount of time softwaressoftwares
ATG…….But WHY??ATG…….But WHY??
Result 5 : Software Engineering Result 5 : Software Engineering and Testing studies.and Testing studies.
ATG…….But WHY??ATG…….But WHY??
Software testing is importantSoftware errors cost the U.S.
economy about $59.5 billion each year
Improving testing infrastructure could save 1/3 cost
Software testing is costlyAccount for even half the total cost
of software development
BenefitsBenefits
Reduces Testing TimeReduces Testing Time Reduces Testing CostReduces Testing Cost Reduces human errors while Reduces human errors while
testingtesting
ATG MethodsATG Methods
Data-Driven ATGData-Driven ATG xUnit Frameworks for ATGxUnit Frameworks for ATG
Examples : JUnit and NUnitExamples : JUnit and NUnit Model-Based ATGModel-Based ATG Keyword-Driven ATGKeyword-Driven ATG
xUnitxUnit
Various code-driven testing Various code-driven testing frameworks have come to be frameworks have come to be known collectively as known collectively as xUnitxUnit..
Based on a design by Kent Based on a design by Kent Beck, originally implemented for Beck, originally implemented for Smalltalk as SUnit, but are now Smalltalk as SUnit, but are now available for many programming available for many programming languages and development languages and development platforms. platforms.
xUnitxUnit
The overall design of xUnit frameworks The overall design of xUnit frameworks depends on several components.depends on several components.
Test FixturesTest Fixtures A test fixture is the set of preconditions or A test fixture is the set of preconditions or
state needed for a test to run. Also known state needed for a test to run. Also known as a test context.as a test context.
Test SuitesTest Suites A test suite is a set of tests that all share the A test suite is a set of tests that all share the
same fixture.same fixture. Test ExecutionTest Execution
The execution of an individual unit The execution of an individual unit test proceeds as follows:test proceeds as follows:
xUnitxUnit
The setup() and teardown() methods serve to The setup() and teardown() methods serve to initialize and clean up test fixtures. initialize and clean up test fixtures.
AssertionsAssertions An assertion is a function or macro that An assertion is a function or macro that
verifies the behavior of the unit under test. verifies the behavior of the unit under test. Failure of an assertion typically throws an Failure of an assertion typically throws an exception, aborting the execution of the exception, aborting the execution of the current test.current test.
Data-Driven ATGData-Driven ATG
Data-driven testingData-driven testing is a methodology is a methodology used in Test automationused in Test automation
Test scripts are executed and verified Test scripts are executed and verified based on the data values stored in one or based on the data values stored in one or more central data sources or databases.more central data sources or databases.
Databases can range from datapools, Databases can range from datapools, ODBC sources, csv files, Excel files, DAO ODBC sources, csv files, Excel files, DAO objects, ADO objects, etc.objects, ADO objects, etc.
Data-driven testing is the establishment of Data-driven testing is the establishment of several interacting test scripts together with several interacting test scripts together with their related data results in a framework their related data results in a framework used for the methodology. used for the methodology.
Data-Driven ATGData-Driven ATG
Variables are used for both input values and Variables are used for both input values and output verification values: navigation through output verification values: navigation through the program, reading of the data sources, and the program, reading of the data sources, and logging of test status and information are all logging of test status and information are all coded in the test script. Thus, the logic coded in the test script. Thus, the logic executed in the script is also dependant on the executed in the script is also dependant on the data values. data values.
An example of xUnit Framework Automatic An example of xUnit Framework Automatic Testing by using Data-Driven Test Testing by using Data-Driven Test Methodology is as follows:Methodology is as follows:
ExampleExample Put the data that Put the data that
varies from test varies from test to test into the to test into the Data-Driven TestData-Driven Test file that the file that the interpreter reads interpreter reads to execute the to execute the teststests
For each test it For each test it does the same does the same sequence of sequence of actions to actions to implement the implement the
Four-Phase TestFour-Phase Test
Example (FPT)Example (FPT)
Set-up the test fixtureSet-up the test fixture Try the test caseTry the test case Evaluate the resultEvaluate the result Release the fixtureRelease the fixture
Keyword-Driven ATGKeyword-Driven ATG
It is also known as table-driven testing, action-It is also known as table-driven testing, action-word testing word testing
It can be used for manuel testing as well as It can be used for manuel testing as well as automated testingautomated testing
The advantages for automated tests are the The advantages for automated tests are the reusability and therefore ease of maintenance reusability and therefore ease of maintenance of tests which have been created at a high of tests which have been created at a high level of abstraction.level of abstraction.
The Keyword-Driven ATG methodology divides The Keyword-Driven ATG methodology divides test creation into two stages. test creation into two stages. Planning PhasePlanning Phase Implementing PhaseImplementing Phase
Keyword-Driven ATGKeyword-Driven ATG
Planning PhasePlanning Phase
The application (or the The application (or the requirements for the application) requirements for the application) is analyzed to determine which is analyzed to determine which operations and objects will need operations and objects will need to be tested.to be tested.
ExampleExample
A web-questionnaire application A web-questionnaire application will require a large amount of text entries. will require a large amount of text entries. By identifying which operations should be By identifying which operations should be encapsulated into keywords, the encapsulated into keywords, the efficiency and maintainability of the tests efficiency and maintainability of the tests are maximized. are maximized.
Keyword-Driven ATGKeyword-Driven ATG
Implementation PhaseImplementation Phase
The implementation stage differs depending on the The implementation stage differs depending on the tool or framework used.tool or framework used.
Often, automation engineers implement a Often, automation engineers implement a framework that provides keywords like “check” and framework that provides keywords like “check” and “enter”. “enter”.
Testers or test designers (who don’t have to know Testers or test designers (who don’t have to know how to program) write test cases based on the how to program) write test cases based on the keywords defined in the planning stage, which have keywords defined in the planning stage, which have been implemented by the engineers.been implemented by the engineers.
The test is executed using a driver which reads the The test is executed using a driver which reads the keywords and executes the corresponding code. keywords and executes the corresponding code.
Keyword-Driven ATGKeyword-Driven ATG
Other methodologies use an all-in-one Other methodologies use an all-in-one implementation stage.implementation stage.
Instead of separating the tasks of test Instead of separating the tasks of test design and test engineering, the test design and test engineering, the test design design isis the test automation. the test automation.
Keywords, such as “enter” or “check” are Keywords, such as “enter” or “check” are created using tools in which the necessary created using tools in which the necessary code has already been written. code has already been written.
This removes the necessity for extra This removes the necessity for extra engineers in the test process, because the engineers in the test process, because the implementation for the keywords is already implementation for the keywords is already a part of the tool. a part of the tool.
Keyword-Driven ATGKeyword-Driven ATG
BenefitsBenefits
This methodology requires more planning and a This methodology requires more planning and a longer initial time-investment than going directly to longer initial time-investment than going directly to the test creation stage and recording a test.the test creation stage and recording a test.
It does make the test creation and test maintenance It does make the test creation and test maintenance stages more efficient and keeps the structure of stages more efficient and keeps the structure of individual tests more readable and easier to modify.individual tests more readable and easier to modify.
The more abstract keywords are, the more reusable The more abstract keywords are, the more reusable they are, and therefore the easier a test is to they are, and therefore the easier a test is to maintain. maintain.
As well as reducing the cost and time spent As well as reducing the cost and time spent maintaining and updating tests, the modular maintaining and updating tests, the modular structure of Keyword-Driven ATG means that new structure of Keyword-Driven ATG means that new tests can easily be created from pre-existing tests can easily be created from pre-existing modules.modules.
Keyword-Driven ATGKeyword-Driven ATG
Another advantage is the reduction Another advantage is the reduction in technical know-how required for in technical know-how required for the test automation process.the test automation process.
In the first approach, technical know-In the first approach, technical know-how is only required by the how is only required by the engineers that implement the engineers that implement the keywords. keywords.
In the second approach, even this is In the second approach, even this is not required, which means that the not required, which means that the test team is capable of entirely test team is capable of entirely automating tests, even without automating tests, even without programming knowledge.programming knowledge.
Model Based ATGModel Based ATG
Model-based testingModel-based testing is software testing in is software testing in which test cases are derived in whole or in part which test cases are derived in whole or in part from a model that describes some (usually from a model that describes some (usually functional) aspects of the system under test functional) aspects of the system under test processprocess The model is usually an abstract, partial The model is usually an abstract, partial
presentation of the system under test's desired presentation of the system under test's desired behavior. behavior.
The test cases derived from this model are The test cases derived from this model are functional tests on the same level of functional tests on the same level of abstraction as the model abstraction as the model
Can not run abstracted test cases.Need Can not run abstracted test cases.Need executable ones.executable ones.
Model Based ATGModel Based ATG
Diagram description : Diagram description :
Model-Based Test SystemsModel-Based Test Systems
Model Based ATGModel Based ATG
Many ways to produce test cases.Many ways to produce test cases. No best method for producing tests.No best method for producing tests. Use your test interest to define your test Use your test interest to define your test
cases.cases. Design decisions is often known as Design decisions is often known as
"test requirements", "test purpose" or "test requirements", "test purpose" or even "use case" even "use case"
Model Based ATGModel Based ATG
Use model and test requirements to Use model and test requirements to derive abstract test casesderive abstract test cases
Implementation Extra Information to Implementation Extra Information to abstract test suit to produce executable abstract test suit to produce executable test casestest cases
Model Based ATGModel Based ATG
Execute the testExecute the test Evaluate the reportsEvaluate the reports
Model Based ATG Model Based ATG
There are a few ways to deploy the There are a few ways to deploy the Model-Based TestingModel-Based Testing Online TestingOnline Testing: A model-based testing : A model-based testing
tool connects “directly” to a system tool connects “directly” to a system under test and tests it dynamically. under test and tests it dynamically.
Offline Generation of Executable Offline Generation of Executable TestsTests:: A model-based testing tool A model-based testing tool generates test cases as a computer-generates test cases as a computer-readable asset that can be later readable asset that can be later deployed automatically. deployed automatically.
A collection of Python classes that embodies A collection of Python classes that embodies the generated testing logic the generated testing logic
Model Based ATGModel Based ATG
Offline Generation of Manually Offline Generation of Manually Deployable Tests:Deployable Tests: A model-based A model-based testing tool generates test cases as a testing tool generates test cases as a human-readable asset that can be later human-readable asset that can be later deployed manually. deployed manually.
A PDF document in English that A PDF document in English that describes the generated test steps describes the generated test steps
Another Step :Another Step : Test GenerationTest Generation
Model Based ATGModel Based ATG
Model-Based Testing has a good potential in Model-Based Testing has a good potential in aspect of automation.aspect of automation.
Test case generation is one of the issues that Test case generation is one of the issues that ATG is subject toATG is subject to
To find test cases, Model-Based structure is To find test cases, Model-Based structure is interpreted as a kind of Finite State Machine to interpreted as a kind of Finite State Machine to see different states that the system might be insee different states that the system might be in
There are a few technics for generating test There are a few technics for generating test cases.Most knowns are as follows:cases.Most knowns are as follows:
Model Based ATGModel Based ATG
Test case generation by theorem provingTest case generation by theorem proving System model is defined as predicates,kinds of System model is defined as predicates,kinds of
logical expressionslogical expressions These expression are processed and proved These expression are processed and proved
afterwardsafterwards Test case generation by constraint logic Test case generation by constraint logic
programmingprogramming Selecting test cases satisfying specific Selecting test cases satisfying specific
constraints by solving a set of constraints over a constraints by solving a set of constraints over a set of variables set of variables
Model Based ATGModel Based ATG
Test case generation by model checkingTest case generation by model checking We provide a model of the system under test and a We provide a model of the system under test and a
property we want to test to the model checker property we want to test to the model checker Test case generation by symbolic executionTest case generation by symbolic execution
Searching for execution traces in an abstract model. Searching for execution traces in an abstract model. In principle the program execution is simulated using In principle the program execution is simulated using symbols for variables rather than actual values symbols for variables rather than actual values
Test case generation by using an event-flow Test case generation by using an event-flow modelmodel With a graphical user-interface (GUI) front-end With a graphical user-interface (GUI) front-end
is called the event-flow model that represents is called the event-flow model that represents events and event interactions events and event interactions
ExampleExample
TestMaster™ automates the generation of TestMaster™ automates the generation of tests for call processing features developed tests for call processing features developed for the 5ESS®-2000 Switchfor the 5ESS®-2000 Switch
This test uses Model-Based Test GenerationThis test uses Model-Based Test Generation Results obtained here as follows.Results obtained here as follows.
AA test generation productivityimprovement test generation productivityimprovement
of just over 90%of just over 90%
Automatic Test Pattern Automatic Test Pattern GenerationGeneration ATPG is an electronic design automation ATPG is an electronic design automation
method/technology used to find an input (or method/technology used to find an input (or test) sequence that, when applied to a digital test) sequence that, when applied to a digital circuit, enables testers to distinguish between circuit, enables testers to distinguish between the correct circuit behavior and the faulty circuit the correct circuit behavior and the faulty circuit behavior caused by defectsbehavior caused by defects
The generated patterns are used to test The generated patterns are used to test semiconductor devices after manufacture, semiconductor devices after manufacture,
In some cases to assist with determining the In some cases to assist with determining the cause of failure cause of failure
Automatic Test Pattern Automatic Test Pattern GenerationGeneration The effectiveness of ATPG is The effectiveness of ATPG is
measured by the amount of modeled measured by the amount of modeled defects, or fault models, that are defects, or fault models, that are detected and the number of detected and the number of generated patterns.generated patterns.
We can estimate test quality by We can estimate test quality by effectiveness of ATGPeffectiveness of ATGP
Automatic Test Pattern Automatic Test Pattern GenerationGeneration A defect is an error introduced into a device A defect is an error introduced into a device
during the manufacturing process.during the manufacturing process. A fault model is a mathematical description of A fault model is a mathematical description of
how a defect alters design behavior.how a defect alters design behavior. A fault is said to be A fault is said to be detecteddetected by a test pattern by a test pattern
if, when applying the pattern to the design, any if, when applying the pattern to the design, any logic value observed at one or more of the logic value observed at one or more of the circuit's primary outputs differs between the circuit's primary outputs differs between the original design and the design with the fault.original design and the design with the fault.
There are two steps that ATPG should take to There are two steps that ATPG should take to detect fault.detect fault.
Automatic Test Pattern Automatic Test Pattern GenerationGeneration Fault activation: Fault activation: Establishes a signal Establishes a signal
value at the fault model site that is value at the fault model site that is opposite of the value produced by the opposite of the value produced by the fault model fault model
Fault propagation:Fault propagation: Moves the resulting Moves the resulting signal value, or fault effect, forward by signal value, or fault effect, forward by sensitizing a path from the fault site to a sensitizing a path from the fault site to a primary output.primary output.
Sequential ATPGSequential ATPG
Searches for a sequence of vectors to detect a Searches for a sequence of vectors to detect a particular fault through the space of all possible particular fault through the space of all possible vector sequencesvector sequences Vector Sequence: A sequence of values for Vector Sequence: A sequence of values for
circuit inputcircuit input In Sequential Circuits In Sequential Circuits due to the presence of due to the presence of
memory elements, the controllability and memory elements, the controllability and observability of the internal signals (in observability of the internal signals (in general)are much more difficult than those general)are much more difficult than those in a combinational circuitin a combinational circuit
Complexity of sequential ATPG much Complexity of sequential ATPG much higher than that of combinational ATPG. higher than that of combinational ATPG.
Algorithmic MethodsAlgorithmic Methods
Testing very-large-scale integrated circuits with Testing very-large-scale integrated circuits with a high fault coverage is a difficult task because a high fault coverage is a difficult task because of complexity. of complexity.
Many different ATPG methods have been Many different ATPG methods have been developed to address combinatorial and developed to address combinatorial and sequential circuits.sequential circuits.
Some examples:Some examples: Pseudorandom test generation is the simplest Pseudorandom test generation is the simplest
method of creating tests. It uses a method of creating tests. It uses a pseudorandom number generator to generate pseudorandom number generator to generate test vectorstest vectors
Algorithmic MethodsAlgorithmic Methods
The D Algorithm was the first practical test The D Algorithm was the first practical test generation algorithm in terms of memory generation algorithm in terms of memory requirements. The D Algorithm introduced D requirements. The D Algorithm introduced D Notation which continues to be used in most Notation which continues to be used in most ATPG algorithms.ATPG algorithms.
Fan-Out Oriented Algorithm: It limits the Fan-Out Oriented Algorithm: It limits the ATPG search space to reduce computation ATPG search space to reduce computation time and accelerates backtracing. time and accelerates backtracing.
Automatic Test Pattern Automatic Test Pattern GenerationGeneration ATPG can fail to find a test for a ATPG can fail to find a test for a
particular fault in at least two cases. particular fault in at least two cases. The fault may be intrinsically The fault may be intrinsically
undetectableundetectable it is possible that a pattern(s) it is possible that a pattern(s)
exist, but the algorithm cannot find exist, but the algorithm cannot find it it
ReferencesReferences Automated Test GenerationAutomated Test Generation, , (From a Behavioral (From a Behavioral
Model)Model), , James M. ClarkeJames M. Clarke, , Lucent TechnologiesLucent Technologies Automatic Test Pattern Generation , Automatic Test Pattern Generation ,
http://en.wikipedia.org/wiki/Automatic_test_pattern_http://en.wikipedia.org/wiki/Automatic_test_pattern_generationgeneration
Data Driven Tests, Gerrard Meszaros, http://xunitpatterns.com/Data-Driven%20Test.html, 2007
Data-Driven Testing, Data-Driven Testing, http://en.wikipedia.org/wiki/Data-driven_testinghttp://en.wikipedia.org/wiki/Data-driven_testing
ReferencesReferences
Test Automation, Test Automation, http://en.wikipedia.org/wiki/Test_automationhttp://en.wikipedia.org/wiki/Test_automation
Test Automation Framework, Test Automation Framework, http://en.wikipedia.org/wiki/Test_automation_framehttp://en.wikipedia.org/wiki/Test_automation_frameworkwork
A Survay on Automatic Test Case Generation, M.Prasanna, S.N. Sivanandam, R.Venkatesan, R.Sundarrajan, Department of Computer Science and Engineering, PSG College Of Technology, http://www.acadjournal.com/2005/v15/part6/p4/
Keyword-Driven Testing , Keyword-Driven Testing , http://en.wikipedia.org/wiki/Keyword-driven_testinghttp://en.wikipedia.org/wiki/Keyword-driven_testing
ReferencesReferences
Model – Based Testing, Model – Based Testing, http://en.wikipedia.org/wiki/Model-based_testinghttp://en.wikipedia.org/wiki/Model-based_testing
Improving Automation in Developer Testing: Achievements and Challenges, Tao Xie, Department of Computer Science, North Carolina State University
xUnit , http://en.wikipedia.org/wiki/XUnitxUnit , http://en.wikipedia.org/wiki/XUnit
The EndThe End
Thank YouThank You
Q & AQ & A