It’s Automation, Not Automagic! …or proving Einstein was right about insanity Presented by Jim Hazen
Transcript
Slide 1
or proving Einstein was right about insanity Presented by Jim
Hazen
Slide 2
Introduction Jim Hazen Veteran of the Software Testing Trenches
Experience in software testing, both commercial and consulting
work. Working with automation and tools since 1989. Implemented
Automation Tools on DOS, OS/2, Windows and Web platforms.
Experience with tools; including HP/Mercury, IBM/Rational, and
other scripting languages.
Slide 3
Sage Words to Remember Insanity: doing the same thing over and
over again and expecting different results. Albert Einstein
Slide 4
Agenda What is Test Automation A Brief History Different types
of test automation, tools and uses Automation Myths How to correct
them Resources & Skills needed Setting expectations for
Management & Others Summary Q&A
Slide 5
What is Test Automation?
Slide 6
Its a Big Topic to Discuss
Slide 7
Bigger Than We Think!
Slide 8
Is It This?
Slide 9
Or More Like This?
Slide 10
Who Builds the Automation?
Slide 11
Some Are Seen This Way
Slide 12
Most Look Like This Instead
Slide 13
What Test Automation Really Is Automation definition Websters
the technique of making an apparatus, a process, or a system
operate automatically In Software Testing we use tools to drive our
processes People only think of tools that execute a test as
Automation. Other tools can be used for Test Design, Test Data
Generation, Defect Tracking, Test Management, etc. Automation is
using a tool during any part of the process of testing.
Slide 14
Who Test Automators Really Are
Slide 15
A Brief History Test Execution tools have been around a long
time! Mainframes (pre Mid-80s) PC DOS (Mid 80s 90s) PC GUI &
Web (1990s present) Mobile (2008 present) Mainframe Home grown
macro tools 1, Hyperion PC DOS AutoTester, SuperKey (macro tool) PC
GUI Automated Test Facility (ATF), Rational Robot, QA Run,
Winrunner, etc. PC Web QTP, SilkTest, Selenium, TestComplete, etc.
Mobile MonkeyTalk, Robotium, etc.
Slide 16
Other Types of Test Automation Tools Other Test Execution Tools
xUnit (JUnit & NUnit), SOAPUI, FitNesse, etc. Test Design &
Test Data Generation Hexawise, AllPairs, Tosca Test Suite, etc.
Test Execution Management HP ALM, MS Team Foundation Server,
TestLink, etc. Test Environment Management VMWare vCloud Director,
etc. Defect Tracking & Reporting Bugzilla, JIRA, etc.
Slide 17
Automation Myths & Misconceptions
Slide 18
Old Myths You can automate 100% of your tests Automation will
cure all problems with testing Automation tools are usable by
anyone Record/Playback is all you need Automation allows Management
to cut test staff One tool is all you need Automation will speed up
testing process * Oldies, but goodies!!
Slide 19
New Myths You dont have to know how to program Opensource tools
are free Commercial tools are expensive GUI level automation is a
waste of time Codeless and Scriptless tools Commercial tools cant
support agile Developers only need to focus on Unit Tests
Slide 20
How to Correct Them 100% Automation Certain tests shouldnt or
cant be automated. 80/20 Rule; automate 20% of functionality used
80% of time. Automation solves all testing problems it helps, but
it creates new problems too. Automation can be done by anyone Not
everyone has the mindset, it is both learned and inherent (logic).
Record/Playback is all you need Proven time & time again to be
false, and to fail.
Slide 21
How to Correct Them cont. One tool is all you need There isnt a
one-size fits all. Automation speeds up testing process The
Illusion of Speed problem. Both execution and creation. You dont
need to know programming Anyone can Automate problem. Opensource
tools are free Costs due to ramp up time (money) and skilled staff
to do it (more expensive). 2 Commercial = Expensive Not really in
grand scheme of things. 2
Slide 22
How to Correct Them cont. GUI level automation is a waste of
time Not really
Slide 23
How to Correct Them cont. Codeless/Scriptless tools Just a
different UI. Youre still writing a test which is a set of
instructions (code). Commercial tools dont do Agile Yes they do,
its how you use the tool that counts. Developers only create Unit
Tests Other tools require someone to build the fixtures/code to
talk to the SUT.
Einstein was right! We keep repeating history, why? Always a
new group of management and testers. Companies always looking to do
testing faster, cheaper and with less resources. Vendors and others
who sell Automagic. Instant gratification mindset. Automation takes
time. Not learning from past mistakes. Software not designed with
testability in mind. Testers not involved early with decisions, not
included by management.
Slide 26
Resources for Automation Dedicated Staff Skilled &
experienced staff Dedicated Equipment/Environments Reduce risk of
competition of resource or teams stepping on each other Dedicated
Budget Automation isnt cheap, plan appropriately Dedicated
Management Support Cant have any of the other 3 without it
Slide 27
Skills for Test Automation Its a different view and purpose for
the script code. Creating software to test software; i.e.,
PROGRAMMING! Need basic understanding of programming methods.
Understand different types of technology layers GUI, API, System,
Web, Services, Database, etc. You need to know how to program in
different languages when needed. C#, Java, VBScript, Jscript, XML,
SQL, etc.
Slide 28
Test Automation Skills Cont. Need to understand how to hook the
software under test and interact with it, and at what level. API,
or Object, or Service, or Protocol, etc. Understand how tool
interacts with the software under test; what functions/methods are
to use. GUI Object methods (functions) Ex:
SwfWindow(LoginForm").SwfEdit(ID).SET Jim Need to understand how to
pick apart the software and drive it programmatically.
Slide 29
Axioms to remember SMART Standalone Test/Script able to run by
itself, minimize dependencies Modular/Maintainable Scripts and code
need to be modular in design and easy to maintain Appropriate Build
according to need for project, dont try to automate 100% Reusable
Build for reuse; keep script code compact and focused. Testable
& Traceable Make it easy to debug your code, and be able to
link together.
Slide 30
Axioms to remember cont. SEARCH 4,5 Setup Test/Script needs to
check for pre-test conditions and flag if needed. Execute The logic
and data needed to run the test itself Analyze Validations, Error
Handling, etc. Report Progress, Error, Summation of run information
(Logfile) Cleanup Test/Script needs to cleanup (temp files, reset
data, etc.) Help (or Home)
Slide 31
Setting Realistic Expectations Management & Other Groups
100% automation is an unreachable goal. 6,7 Automation is its own
form of software development. Ramp-up time will be required, longer
than you think. Benefits realized over time, no immediate payback
for investment.
Slide 32
Expectations cont. Management & Other Groups No tool
supports all systems & technologies out of the box. Not all
testers can write scripts. Specialized resources are a must and
need to be dedicated to the work. Nothing is Free, there are always
costs involved. Dont believe the marketing hype! Dont buy the snake
oil. 8 There is no Magic!
Slide 33
Otherwise, the result
Slide 34
Summary What is Test Automation A lot more than you think!
History Lots of tools over the last 30 plus years. Different types
of test automation tools Not just for execution of a test only.
Automation Myths Old and New; lots of failures due to them. How to
fix the problems Be realistic, dispel the myths. Resources and
Skills needed for test automation Again, a lot more than you think.
It is Programming! Setting expectations for Management & Others
Without proper expectations you are painted into a corner.
Slide 35
Remember Its automation, Not Automagic!
Slide 36
Q&A
Slide 37
Contact Info Jim Hazen Company: Freelance Contractor Home
email: [email protected][email protected] LinkedIn
profile:
http://www.linkedin.com/profile/view?id=2835000&trk=tab_pro
http://www.linkedin.com/profile/view?id=2835000&trk=tab_pro
Slide 38
References 1) Experiences of Test Automation: Case Studies of
Software Test Automation, Chapter 5, Graham & Fewster,
Addison-Wesley, 2012 2) Test Automation - Let's Talk Business, Igor
Gershovich, Connected Testing, 2006. 3) Implementing Automated
Software Testing, IDT Survey, Elfriede Dustin et al.,
Addison-Wesley, 2009. 4) How We Test Software At Microsoft, page
187, Alan Page, Microsoft Press, 2009. 5) How to Automate Testing:
The Big Picture, Keith Stobie & Mark Bergman, 1992. 6) Seven
Steps to Test Automation Success, Bret Pettichord,
http://www.io.com/~wazmo/papers/seven_steps.html, 2001.
http://www.io.com/~wazmo/papers/seven_steps.html 7) Success with
Test Automation, Bret Pettichord,
http://www.io.com/~wazmo/succpap.htm, 2001.
http://www.io.com/~wazmo/succpap.htm 8) Test Automation Snake Oil,
James Bach, 1996
Slide 39
Thank you for attending this session. Please fill out an
evaluation form.