Date post: | 20-Jun-2015 |
Category: |
Documents |
Upload: | meda-conferences |
View: | 740 times |
Download: | 0 times |
Successful AutomationKDT as a Test Language
Aviram ShottenAP Program ManagerQualiTest
Today’s Menu
• Brief introduction to KDT• Practical KDT – how KDT needs to be implemented –
3 years of experience tells…
• KDT supporting tool – the AutomationPlanner®
• Measuring KDT – RoI and other metrics
• Thumb rules for KDT implementation
• Bad Humor
Introduction to KDT
• KDT is 3rd generation automation approach (after record & replay and functional decomposition)
• Previous approaches had very little success due to the following reasons:– Automation experts required to be also good functional
testers
– Automation became a bottleneck on the way to increase coverage
– Double (manual and automated) maintenance of test cases
– Test management (test leader) didn’t necessarily knew how to manage the automation experts
Introduction to KDT
• In order to better mitigate the drawbacks of automation that were mentioned before, KDT suggests the following:
– Automation experts will develop KDT infrastructure for the functional testers
– Functional testers will use infrastructure for test cases implementation
– Functional testers will mainly investigate failures and incidents rather than execute the test cases
Introduction to KDT
Automation Discipline
Testing Discipline
Developing infrastructure:
• Mapping AUT components
• Imp. components operations
• Imp. system operations
• Imp. business functions
Test engineering tasks:
• Test planning
• Test implementation
• Definition of commonly used sequences for test cases
• Test execution
• Debrief and report
• Continuous improvement
Requirements-KDT
• We, as functional testers, requires from a KDT based solution:
– Simplicity - test cases can be defined without development skills
– Functionality – maintain the option to perform any functional UI operation
– Readability – test cases and test reports will continue to be reviewed by other stakeholders
– Maintenance – test cases will be maintained by functional tester and not by automation experts
Generic KDT Suite
Experience led us to build 4 components to support a generic KDT implementation process among test teams:
– allowing easy test cases
implementation for the functional testers
–executes the test batches
KDT Engine – component that converts KDT test cases to executable format
- integrates the KDT tests to the
standard test process in HP Quality Center®
AP Architecture
Logical Layer
Infrastructure Layer
Testing Tool
AP ENGINE
VBS
Or
SUT
A Test Case
Step #ActionExpected Result
1Execute the “MyApplic” application Log in window appears
2Enter user “JohnB” in username field and “123456” in the password field and press “OK”
Error window appear
3Enter user “JohnB” in username field and “QA2010” in the password field and press “OK”
Login Successful, application’s main window appear
Traditional Test Case
KDT Syntax
• Keywords in use:
– Component Function (CF) – perform a specific operation on a given UI component – click, set value, verify exist etc.
– Service Function (SF) – scripts that enables assisting methods for the test implementation - wait X seconds, write to report etc.
– Business Function (BF) – execute a certain functional operation, that is hard\ non effective to implement by component\ service function
– Sequence (Seq) – a set of keywords (CF, SF, BF, or another Seq) that can be called from different test cases with different parameters.
Test Cases Syntax
Step #
Step Type
ComponentOperationValue
1SF-RunCommand“C:\Program Files\MyApplic.exe”
2CFUserName (Edit)Set“JohnB”
3CFPassword (Edit)Set“123456”
4CFOK (Button)Click-
5CFLoginErrorDialogueVerifyExist-
6CFUserName (Edit)Set“JohnB”
7CFPassword (Edit)Set“QA2010”
8CFOK (Button)Click-
9CFWelcomeWindowVerifyExist-
KDT Test Case
Test Cases Syntax
Step #Step TypeComponentOperationValue
1SF-RunCommand“C:\Program Files\MyApplic.exe”
2SEQLogin-“JohnB”, “123456”
3CFLoginErrorDialogueVerifyExist-
4SEQLogin-“JohnB”, “QA2010”
5CFWelcomeWindowVerifyExist-
KDT Test Case - Optimized
KDT Test Design
Traditional description KDT syntax
Test applicable data
KW type – CF, BF or Sequence
KW specific ID Steps parameters
KDT - Metrics
• Common RoI calculations are commonly defined as:– RoI (Return on Investment)= BENEFIT/COST
• Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost
• Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost
RoI = manual testing cost - automation cost
automation cost
• We suggest otherwise…
Time to Test – where KDT covered 35%
Defect Detection
Breakeven Calculation
• KDT test case implementation takes 3 times more than traditional\ manual TC
• This “extra effort” can be replaced with 3 manual executions of a given TC
• In KDT era, test execution time is not a matter of constraint. A given TC can now be executed numerously with no effort invested
Rules of Thumb for KDT Imp.
• SW\ system under test should be:– Has to be at least medium size dev. effort (12000 dev.
Hours)– Has already began development and it is possible to execute
test on– At least 50% of the TC can be done by a certain UI (the SUT
doesn’t necessarily has UI, can be simulators, etc.)
• Test team\ organization should consider:– KDT is simple yet requires good and talented functional
testers– All beginnings are hard
Conclusion
• Where thumb rules apply, KDT can be implemented with much better chance to succeed
• Where traditional automation applies KDT can increase coverage, effectiveness and reduce cost of automation
• We propose a risk free KDT imp. process using generic tools debugged in many projects of many kinds
• KDT improves the “quality of life” of the functional testers - less tedious execution, more planning and analysis
Personal Experience
• Life is now better for me as a functional tester
• Defects are being unveiled during the KDT test case implementation much earlier than in traditional TC definition
• New options are now available for us i.e 24 hours scenario of functional capabilities
• Assets of the KDT diffused to other teams such as SW integrators build tests, developer unit tests, algorithms test etc