Post on 09-May-2015
transcript
An Examination of Test Automation Use Cases(and the factors behind the examination)
January 19, 2010
1
Mark Meninger
Confidential McAfee Internal Use Only
An Examination of Test Automation Use Cases
Contents
• The Speaker
• Presentation Objectives
Objectives and Perspective
• Objectives of test automation
• GUI vs non-GUI
• A look at 2 products
Test Automation Philosophy
Test Automation Use Cases
How does this all fit in?
January 20, 2010
2
Confidential McAfee Internal Use Only
January 20, 2010
3
Objectives and Perspective
Confidential McAfee Internal Use Only
January 20, 2010
4
The Speaker (Mark_Meninger@McAfee.com)
Objectives and Perspective
• Current role @ McAfee: Driving test automation on Consumer side
• Functional GUI automation (support for L10n) across 3 sites
• Functional non-GUI automation (development focused)
• Automated Performance testing
• Previously @ RIM
• As test automation manager for end-user BlackBerry testing
• RIM Test Automation Conference Chair
• A student of test automation and testing
• Worked (and work) with some very capable individuals
• Made some good mistakes
• Learned from my mistakes
• Little experience with Agile
• Presentations @ StarEast, QAI, KWSQA
• My Blog (as of this Friday): http://automatedtestingblog.blogspot.com
Confidential McAfee Internal Use Only
January 20, 2010
5
Presentation Objectives
Objectives and Perspective
1. Talk about my learnings and philosophy of test automation
2. Hear your perspective on what I’m talking about
3. Evaluate test automation use cases/examples (group-
exercise)
a) Interface complexity
b) Speed of test automation
c) Speed of execution
d) Derived value
e) How this fits into the testing cycle
4. In the context of Agile, I would…
Confidential McAfee Internal Use Only
January 20, 2010
6
Mark’s Test Automation Philosophy
Test Automation Philosophy
Confidential McAfee Internal Use Only
January 20, 2010
7
Manual Testing - A path in the trees
Test Automation Philosophy
• Strategy/Plan (resourcing, infrastructure, approach)
• Need to learn the SUT:
• User-stories
• Use Cases
• Tools
• Technologies
• Testing types & cycles (regression, defect, smoke)
• Sprints
• Apply common techniques
• Scripted test cases (with different methodologies),
• Exploratory testing
• Then testing can begin
• I’ve seen most testing organizations do this well to varying degrees
Confidential McAfee Internal Use Only
January 20, 2010
8
Automated testing – Find a path in the forest
Test Automation Philosophy
• Test automation is less of a cookie-cutter approach than manual testing
• Common tasks in test automation
• Interface, framework, integration
• Very specific considerations:
• The technology of the system under test (SUT)
• The technologies used to test the SUT
• Impact on the development/testing schedule
• These details impact the success of the
implementation, the derived value, the
time to delivery, availability etc
• This makes each test automation
implementation unique and challenging
• Not all orgs do test automation well
Confidential McAfee Internal Use Only
January 20, 2010
9
Evolution of the Approach to Test Automation
Test Automation Philosophy
In the beginning:
Here is a test automation tool… now automate! (seen it)
Confidential McAfee Internal Use Only
January 20, 2010
10
Evolution of the Approach to Test Automation
Test Automation Philosophy
And Then:
Evaluate tools, choose one and automate!
(done it)
Confidential McAfee Internal Use Only
January 20, 2010
11
Evolution of the Approach to Test Automation
Test Automation Philosophy
After much learning (now do it!) :Examine my test automation requirements
a) Look at
• Objectives
• SUT technology, roadmap
• Available budget, resources
• Available tool technologies, capabilities and constraints
• SDLC methodology (Agile, Waterfall etc.)
• Development culture
• Testing culture
• Relationships
b) Put together my business case
c) Start with a careful plan
Confidential McAfee Internal Use Only
January 20, 2010
12
Objectives of test automation
1. Executed successfully
relatively early within the
development/testing cycle
2. Reduction in manual testing
time
Test Automation Philosophy
�Test functionality sooner
rather than later in the cycle
�Interface used is available
and stable
�Automated execution as soon
as a build is released:
� Evenings, Weekends
�Automated execution run in
parallel
� Execution across platforms and
test suites
� Limited by availability of
infrastructure
Confidential McAfee Internal Use Only
January 20, 2010
13
Test Automation Objectives
3. Provides reliable results on
1st run
4. The solution is scalable &
maintainable
Test Automation Philosophy
�Tests can be executed repeatedly
with minimal false negatives
�Regular testing of interface and
framework essential
�Changing interface not breaking
tests
�We can increase coverage of
manual test cases without excessive
framework growth
�Grow test case numbers with
confidence:
� 10’s -> 100’s
� 100’s -> 1000’s
Confidential McAfee Internal Use Only
January 20, 2010
14
Test Automation Objectives
5. Agile Environment
Test Automation Philosophy
�Out-of-the-box, light-weight test
automation supports planned sprints
�Test automation team structured,
organized to support agile testing
requirements
�Solution, approach is flexible and
expandable
Confidential McAfee Internal Use Only
January 20, 2010
15
GUI vs non-GUI Technology Considerations
Test Automation Philosophy
GUINon-GUI(some API)
Confidential McAfee Internal Use Only
January 20, 2010
16
Test Automation – The Big Question
Test Automation Philosophy
Re
UI-Logic Interface
UI
And Another
The Basement
Another Abstraction
Do we start here?
Or here?
Or here?
Where?Where?Where?Where?
SoftwareAbstractions
First Business Logic Layer
Confidential McAfee Internal Use Only
January 20, 2010
17
Test Automation - GUI Fat Client (Generalities)
Test Automation Philosophy
• Interface
• Typically complex - more points of failure
• Changes more frequent
• Increased maintenance
• Automation - dependent upon being stable
• Technology
• Vendor tools provide better support
• Scripting technology usually has limited support (i.e. VBScript)
• Performance
• Management for unresponsive GUI
• GUI adds performance hit each time accessed
• Framework can add considerable overhead
GUI - Fat Client• Skill-set
• Requires solid technical skill-set with good design and programming abilities
• Deeper test automation skill-set and experience required
• More expensive
• Localization
• Running identical functional tests across localized builds
• Language independence support is essential to support L10n automation
Confidential McAfee Internal Use Only
January 20, 2010
18
Test Automation – GUI Web (Generalities)
Test Automation Philosophy
• Interface
• Typically simpler – fewer complexities to manage
• Changes frequently over time
• Automation – dependent upon being stable
• Technology
• Good open source (Selenium, Watir)
• Performance
• Fewer if no performance issues
• Good available open source frameworks
• Skill-set
• Better use of ‘scripters’ rather than developers
• Less investment required in more experienced test automation skill-sets
• Less expensive
• Localization
• Language independent testing can be supported
GUI - Web
Confidential McAfee Internal Use Only
January 20, 2010
19
Test Automation – non-GUI (Generalities)
Test Automation Philosophy
• Interface
• Stable and reliable earlier in the dev/testing cycle
• Less changes over time
• Fewer points of failure
• High-level testing can be supported
• Usually needs to be modified for increased testability over time (fits Agile well IMO)
• Technology
• Choose scripting technology with robust library support
• Integrate into development cycle
Non-GUI (API) Automation
• Performance
• No GUI impact - much faster
• Choose integrated framework with little overhead
• Skill-set
• Requires knowledge of underlying business logic, application architecture and design
• More expensive
• Localization
• Issues around language independence (localized strings integrated into GUI abstractions)
Confidential McAfee Internal Use Only
January 20, 2010
20
GUI Automation
Test Automation Philosophy
• Automation Implication
• Coverage
• Good top-to-bottom testing solution
• Broader
• # of automated test cases dependent upon GUI comlexity
• Development (GUI complexity)
• Higher maintenance costs
• More effort to write tests
• Longer development times
• Execution
• Dependent upon stable GUI
• Slower execution times
• More expensive (resources, equipment and license cost)
• Value-add later in dev/testing cycle for products with major GUI changes
GUI
Confidential McAfee Internal Use Only
January 20, 2010
21
Non-GUI Automation
Test Automation Philosophy
• Automation Implication
• Coverage
• Not top-to-bottom
• Deeper testing
• Larger # of automated test cases
• Development
• Easier to write tests
• Shorter development times
• Lower maintenance costs
• Execution
• Ready when the API are ready
• Test much sooner in the dev/test
• Test results faster; test more often
• Gate before release to testing
• Cheaper (resources, no licenses)
• Add quality paradigm to development organization (code for testability)
• Value-add sooner in dev/testing cycle
Non-GUI Automation
Confidential McAfee Internal Use Only
January 20, 201022
RTQA Build Release Cycle
API Automation
Quality AssuranceStarts
Automated
API Testing
Manual
Testing
TestsPass?
Automated
API Testing
Automated GUI
Testing & Manual
Testing
TestsPass?
!!!??
Manual TestingGUI Automation
Start End
No No
YesYes
Confidential McAfee Internal Use Only
January 20, 2010
23
Automated Testing Coverage
Cost
Complex GUI - Cost for Coverage
• Complex GUI applications will incur an increasing cost to
automate larger #’s of test cases
• Cost: Effort, Infrastructure, Dollars, Time for Execution
Automation Ceiling
Costs become prohibitive
Test Automation Philosophy
GUIGUIGUIGUI
Confidential McAfee Internal Use Only
January 20, 2010
24
Automated Testing Coverage
Cost
Automation Philosophy Review – Cost for Coverage
• Non-GUI Automation has lower cost for coverage
• Cost: Effort, Infrastructure, Dollars, Time for Execution
Automation Ceiling
Less expensive
Test Automation Philosophy
GUIGUIGUIGUI
NonNonNonNon----GUIGUIGUIGUI
Confidential McAfee Internal Use Only
r
Consumer Test Automation Framework (CTAF)/QTP/MAGI
MAGI FrameworkServices
Core Functions
CTAF Extensions
HP Quick Test Pro
GUI HTMLControl
ID HTMLControl
ID HTMLControl
ID
CTAF External LibrariesGUI/Table
LanguageDependentAssert
VBScript Test Script
CTAF Internal Libraries
Unit Test
Helper
GlobalMPF MVS
Lots of librariesto build(and support)
Complex Framework(yet very helpful)
Our ownextensions
Confidential McAfee Internal Use Only
Python py.test/nose(?) Test Automation Framework
Python
Libraries
COM InterfaceImplement IDispatch
High-Level Class
High-Level Class
High-Level Class
Python Test Script
CTAF API Python
Libraries
Example API Test Automation – Using Python
Python hasnumerousrich and diverse libraries
Framework exists which nicely integrates with Python
Confidential McAfee Internal Use Only
January 20, 2010
27
Test Automation Use Cases
A look at some interfaces
• Google Search
• McAfee Security Center (2010 release)
Disclaimer
• These thoughts are my own and are not necessary correct
• Part of the process would be for me to understand the
technology and determine the best approach
(Finding a path through the forest I find myself in)
Confidential McAfee Internal Use Only
Test Automation Use Cases
Points to Evaluate
• What approach would you take?
• What are the risks of doing it this way?
• What are the costs/investment requirements of doing this? (be
specific)
• What are the gains of doing this?
• Why would you do this?
• How would you integrate this into a testing cycle?
• Why would you integrate this into the testing cycle this way?
• How could this approach be fit into an Agile/Lean
methodology?
• What type of buy-in would you need to achieve this?
• What relationships would you need to establish to be
successful?
January 20, 2010
28
Confidential McAfee Internal Use Only
Test Automation Use Cases
What I will focus on
• Interface complexity
• Speed of test automation
• Speed of execution
• Derived value
• How this fits into the testing cycle
January 20, 2010
29
Confidential McAfee Internal Use Only
January 20, 2010
30
Confidential McAfee Internal Use Only
Test Automation Use Cases – Google Search
•Google Search
– From: http://en.wikipedia.org/wiki/Google_search
– PageRank Logic
– synonyms, weather forecasts, time zones, stock quotes,
maps, earthquake data, movie showtimes, airports, home
listings, and sports scores
– prices, temperatures, money/unit conversions ("10.5 cm in
inches"), calculations ( 3*4+sqrt(6)-pi/2 ), package tracking,
patents, area codes,[6] and rudimentary language
translation of displayed pages
– ‘I’m feeling lucky’
– Privatization logic (Switzerland)
– Ajax code
– ….
January 20, 2010
31
Confidential McAfee Internal Use Only
Test Automation Use Cases – Google Search
Our job to automate Search logic • Core functionality
• Page rank is an algorithm that evaluates an index using
supposedly over 200 factors
January 20, 2010
32
Confidential McAfee Internal Use Only
Test Automation Use Cases – Google Search
•What I’d do– Back-end (non-gui) automation
– Focus just on the engine and it’s data
– Evaluate the probability and weighting of each factor
– Generate a list of results and probably would fit into
some level of confidence of accuracy
– Rendering of data could be easily verified manually
January 20, 2010
33
Confidential McAfee Internal Use Only
Test Automation Use Cases – Google Search
•How• Work with developers to build a testing engine that could handle
‘plug-ins’ of new testing factors
• A common testing pattern would be for each factor
• Determine how each factor would return results on its own or in
interaction with another factor
• Integrate this all into the testing engine
• Establish a common mechanism for executing tests, gathering
expected results and comparing actual results
• Utilize the common testing pattern
• Utilize the same pattern for evaluation in the testing engine
• Drive a common testing interface across those adding new ranking
functionality to the google search engine
January 20, 2010
34
Confidential McAfee Internal Use Only
Test Automation Use Cases
Interface complexity
• Interface would be simple – an API
• The combination of algorithms (factors) would make this
complex
Speed of test automation delivery
• Fast
– Could start writing tests for each factor
– Could start building in some relationships for each factor
• Slower
– Integration of everything together
– This would also include the test engine
January 20, 2010
35
Confidential McAfee Internal Use Only
Test Automation Use Cases
Speed of execution
• Very fast
• Could run very often
Derived value
• Substantial
• Could fully automate the algorithm
• If the integration of factors together could be done
successfully, this would be substantial
How this fits into the testing cycle
• Immediately
• Write code, test code
• No throw-over to SV&V/QA
January 20, 2010
36
Confidential McAfee Internal Use Only
January 20, 2010
37
Confidential McAfee Internal Use Only
January 20, 2010
38
Cialis - erectile dysfunction drug
Radical Prostatectomy - is an operation to
remove the prostate gland and some of
the tissue around it Very fast
Confidential McAfee Internal Use Only
January 20, 2010
39
Confidential McAfee Internal Use Only
Test Automation Use Cases – McAfee Security
Center
•McAfee Consumer Product (2010)
– Security Center
– Firewall, AntiVirus, AntiSpam, Parental Controls…
– Focus on AntiVirus
– Verify GUI and Functional behaviour
January 20, 2010
40
Confidential McAfee Internal Use Only
Test Automation Use Cases
• McAfee Consumer Product (2010)
– GUI – thick GUI
– Verify GUI and Functional behaviour
January 20, 2010
41
Confidential McAfee Internal Use Only
Test Automation Use Cases
• How
• Vendor tool
January 20, 2010
42
Confidential McAfee Internal Use Only
Test Automation Use Cases
• McAfee Consumer Product (2010)
– GUI – thick GUI
– Verify GUI and Functional behaviour
January 20, 2010
43
Confidential McAfee Internal Use Only
Test Automation Use Cases
• McAfee Consumer Product (2010)
– GUI – thick GUI
– Verify GUI and Functional behaviour
January 20, 2010
44
Confidential McAfee Internal Use Only
Test Automation Use Cases – McAfee Security
Center
•What I’d do & How– Get a good GUI automation tool
– Find a manageable way to integrate with the GUI DOM
• Re-usability
• Maintainability
– First go: limit my automation to easiest test cases
– Find a good framework to integrate with my GUI
automation tool
• Automate the automation
– Aim would be reduce manual testing with an eye to
reduce maintenance
January 20, 2010
45
Confidential McAfee Internal Use Only
Test Automation Use Cases – McAfee Security
Center
Interface complexity
• Interface is very complex
– Lots of objects to manage (lots of points of failure)
– How we work with the interface is complex under the
covers (constrained by tool)
Speed of test automation delivery
• Slow!!
– Most off-the-shelf have limited library support
– Have common set of libraries across the products
– Need to build libraries for each product at the GUI & functional
level
– Integration into the framework could take even longer
January 20, 2010
46
Confidential McAfee Internal Use Only
Test Automation Use Cases – McAfee Security
Center
Speed of execution
• Slow!
• Fat client GUIs are usually slower
• Hooks from application driving the GUI adds overhead
• Integrating a framework that drives all this adds additional
time
Derived value
• Some!
• New GUI; some changes – later in the cycle
• Same GUI; little changes – earlier in the cycle
• Need more infrastructure to support
January 20, 2010
47
Confidential McAfee Internal Use Only
Test Automation Use Cases – McAfee Security
Center
How this fits into the testing cycle
• If no GUI changes - > Right Away
• GUI changes - > Later
• Testing cycles will take longer
• Slower to execute
January 20, 2010
48
Confidential McAfee Internal Use Only
January 20, 2010
49
Test Automation – The Big Question
Test Automation Use Cases – McAfee Security Center
Re
UI-Logic Interface
UI
And Another
The Basement
First Business Logic Layer
Another Abstraction
Do we start here?
Or here?
SoftwareAbstractions
Confidential McAfee Internal Use Only
Test Automation Use Cases – McAfee Security
Center
•What I’d do & How– Examine the layers below the GUI
– Can I use any of them to automate testing?
– Work to drive this idea within the organization
– Most likely some developer has thought the same thing
– That is my foothold and I build on that
January 20, 2010
50
Confidential McAfee Internal Use Only
Test Automation Use Cases – McAfee Security
Center
Interface complexity
• Interface is less complex
– New technology learning curve
– Easier to call some API than manage GUI objects
– Less change; more stability
Speed of test automation delivery
– Fast!!
– Available frameworks (don’t need to build your own) (i.e. py.test)
– Access to rich scripting environments & libraries (i.e. Python) – don’t
need to build your own
– Less points of failure to manage
January 20, 2010
51
Confidential McAfee Internal Use Only
Test Automation Use Cases – McAfee Security
Center
Speed of execution
• Fast!! No GUI overhead
• Integrated framework will also add little overhead (i.e.
TestNG, Py.test)
Derived value
• Higher value
How this fits into the testing cycle
• Almost always test earlier in the cycle
• Test more frequently
• Integrate into the development cycle rather than QA
– Quality moving upstream
January 20, 2010
52
Confidential McAfee Internal Use Only
January 20, 2010
53
Confidential McAfee Internal Use Only
January 20, 2010
54
How does this all fit in?
Objectives and Perspective
• Have your test automation specialist begin to develop a methodology that
will fit your agile development cycle
• Tell her/him what your requirements are and ask for a solution that will
meet this
• Optimize your test automation execution for Agile!
• This will most likely be an out-of-the-box approach to test automation
• All pay-offs should be well understood:
• Light-weight, quick to execute, easy to develop, not-too-deep solution
• Heavier, complex, longer-to execute, harder to develop, deeper test
automation solution
Confidential McAfee Internal Use Only
January 20, 2010
55
How does this all fit in?
Objectives and Perspective
• They should have a good understanding of the available technologies to
use and what the trade-offs are for each
• What solution will best meet the above requirements?
• Let that test automation specialist know what’s needed and why. This will
hopefully inspire them to build the solution that meets these needs
• Integrate automated testing into your testing cycles
• Fit the automated testing for a given sprint/release
• Each automated testing approach will have a given set of benefits and
costs
• Choose the automated testing items that make most sense
Confidential McAfee Internal Use Only
Consumer Applications QATest Automation Strategy – Detailed Review
January 20, 2010
56
Thank-youMark_Meninger@McAfee.com
As of this Friday
http://automatedtestingblog.blogspot.com