Date post: | 17-Nov-2014 |
Category: |
Technology |
Upload: | towo-toivola |
View: | 283 times |
Download: | 2 times |
Test AutomationTest Automation
Pondering Workshop for FYI on Winter 2011
Towo Toivola Attribution (Tekijä mainittava) http://creativecommons.org/licenses/by/3.0/
Welcome to the Workshop
I hope we all have an appetite for arguing and learning about test
automation
The slides are in English just to be sure, but I will present in Finnish if the audience prefers it.
Many ThanksMany ThanksMuch of this material is heavily
indebted to Maaret Pyhäjärvi, Erkki Pöyhönen, Petri Kuikka, Risto Kumpulainen, Dean Leffingwell..
Thanks also go to my friends at work and in the software engineering community
Towo Toivola 2009
About MeAbout MeExperiences define viewpointsMSc (eng) from HUTIn F-Secure Corp for 11 years
◦COTS software, custom systems◦Many many projects, large and small
A year in Kosovo◦IT administration, system building
Seminars, workshopsInteractive speaker
Towo Toivola 2009
About YouAbout YouHow many are test engineers?How many managers?Any programmers?How many have some
experience on testing tools?What is your industry?What are your most important
requirements for this workshop?
Towo Toivola 2009
DisclaimerDisclaimerAs with everything, nothing is
absoluteI shall not discuss load-testing
much, or test process automationNot an expert of the particular toolsYour terminology may differAttending this course may not be
enoughIt may be that you should not do
test automationThere are two worlds, I will try to
balance in between themTowo Toivola 2009
Agenda for TodayAgenda for Today
Towo Toivola 2009
0900 Welcome (this)
0920 What kind of TA system would you like?
1000 What is the goal of your TA initiative?
1105 Where does all the testing time go?
1255 How would you choose your TA tool?
1345 What is the right level for doing TA in your organization?
1430 How to take TA into use?
1530 How will you change your products to ease TA?
1600 Resourcing TA in the long run?
Lunch
BreakBreak
About the ScheduleAbout the ScheduleThere is a lot of ground to coverThis is a new format, bear with
meWe can change things
You can check what I am planning to leave out..
There should be enough breaks, we should be able to concentrate well on work◦What rules should we have?
Towo Toivola 2009
Format for this Workshop
We will use this format to deal with the questions in the agenda:
Presenting a question (5min)Pondering in small groups
(15min)Discussion with all together
(10min)Presenting of prepared material
(10min)This should make
it into about 30-50 mins per problem if we don't
get carried away.
Why this Format?Traditional teaching (providing answers)
does not promote thoughtThought and discovery promotes
learningStruggling with a problem first and
receiving guidance after that promotes understanding and application (I hope)
Interaction promotes understanding and information dissipation
I am trying to give you the most valuable things well rather than a lot of things
Technical details are not as important as people think
Question #1
What kind of TA system would you like to have?
(Outline your vision or what you want to have in one years time)
0920
Please take 15 to think about thisFirst quickly by yourselfThen within your groupDocument ideas, keywords and
major agreements on provided office supplies
Be prepared to present your views and findings, even if you are not so sure
Results and Discussion for Question #1
What kind of TA system would you like to have?
(Outline your vision or what you want to have in one years time)
Why do You Want What You Want?Tools and technical solutions often
dominate unduly in the decisionsOrganizational boundaries are often
taken as granted and TA may lead to local optimization
You have probably made a lot of implicit assumptions before even starting to actively think about the problem
People tend to focus on fixing what they understand best instead of that which is most important.
Now that you have already discovered what should be the end
result of today, we can start working in light of that assumption ;)
Question #2
What is the goal of your TA initiative?(Why are you here, why do you want to
do TA, what do you expect to gain?)
1000
Group Work Time
First quickly by yourselfThen within your groupDocument Be prepared to present
Results and Discussion for Question #2
What is the goal of your TA initiative?
What’s the Use of Test What’s the Use of Test AutomationAutomation
Cost savings◦Ability to run the same tests cheaper
Faster development◦Ability to implement more features in
the same amount of timeFaster delivery
◦Ability to do snap changesBetter quality
◦Ability to reach and maintain higher quality
Towo Toivola 2009
Differences to Manual Differences to Manual TestingTestingSome tests that are manually
possible are not possible to automate
Some tests that are possible to automate are not possible to run manually
Development takes much more time, execution much less
Much more attention to detail, much less common sense
Requires more maintenanceTowo Toivola 2009
Enabling the Fast Enabling the Fast Feedback LoopFeedback LoopA common and useful use of test
automationMinimize the time from
programming to quality information◦Code – test – fix – retest in hours◦If not in minutes!
Continuous IntegrationFully automatic tests and
reporting”The most important limiting
factor for the velocity of an agile team” -DL
Towo Toivola 2009
VISIBILITY!
Regression and Regression and AutomationAutomation
Is the software still OK?Manual regression will get rid of
your good test engineersAutomation does not get boredThe courage to do changesThe ability to do quick releases
Towo Toivola 2009
VISIBILITY!
The Machine is Superior to The Machine is Superior to ManManAutomation is fasterAutomation has more attention
to detailAutomation can run through
more dataAutomation can execute more
combinationsAutomation can measure more
exactlyAutomation does not get tiredAutomation works through the
nightAutomation provides exact logs
Towo Toivola 2009
Man is Superior to the Man is Superior to the MachineMachineWorks around problemsCommon senseCreativityUnderstanding of the userUnderstanding of prioritiesUnderstands changeLooking outside the boxThe most important bugs are
found in the requirements and assumptions
Towo Toivola 2009
What Do You Want?What Do You Want?You Need to Prioritize, seriouslyIf you don’t set clear goals, don’t
expect clear benefits, or much results of any kind
Test faster?Test more often?Test better?Test in new ways?Test cheaper?
Towo Toivola 2009
Common MistakesCommon Mistakes(that cost a lot of money)(that cost a lot of money)
Saving a projectAiming for X% automatedWorking with untestable
softwareRunning a lot of tests, producing
little valueOrganizational issuesSkills of people working with
automation#1: Unrealistic assumptions
◦Automation will find loads of new bugs
◦No real resourcing needed
Towo Toivola 2009
VISIBILITY!
Real Benefit of TA
TA is only useful when it provides information that..
Gets a bug fixed
Helps you make a decision
It does this..More if you start
early in the project
More if you run it constantly
Only if you react to the results
Only if you believe the results
These require (re)action!
Test automation, when done sensibly, is hugely beneficial. It can
give you decisive competitive advantage over your competitors both financially and in terms of
employee motivation and satisfaction.
It's hard, but it's worth it.
Break TimeBreak Time
Towo Toivola 2009
1050
Question #3
Where does all the testing time go?(Testing software is an expensive
business, why does it cost so much? What do all those people spend all their
time on?)
1105
Group Work Time
First quickly by yourselfThen within your groupDocument Be prepared to present
Results and Discussion for Question #3
Where does all the testing time go?
The Nature of a TestThe Nature of a Test
Which part can the automation Which part can the automation handle?handle?
Towo Toivola 2009
Where Does Your Testing Where Does Your Testing Time Go?Time Go?Do you use too much time
testing?Not enough time?Test planning?Test environment building?Regression testing?Testing new features?Testing large data collections?Testing with numerous
environments? Towo Toivola 2009
Things You Should Things You Should AutomateAutomateThings that are run very often
◦Core regression testsThings that are easy to automate
and bring valueThings that are tough to test
manually◦Performance◦Race conditions◦Linear combinations◦Large amounts of data◦Reliability
Things that have to be run quicklyROI !!!!11!!!!
Towo Toivola 2009
Things You Should Things You Should Not Not AutomateAutomateTests that are not importantTests that are easy to do
manually and hard to automateTests that are run seldom
◦Unless they are critical or very hard to run manually
Does-it-feel-rightTests where automation results
are not reliable
Towo Toivola 2009
But that is all the Sensible StuffIt is good to think about how to
balance between explorative and regression testing, but I bet your testers also use time on..
Meetings, overhead, bureacracyWaiting (for fix, hardware, decision,
info..)Arguing, pleading, fuming..Testing wrong thingsTesting wrong builds..
In the Insensible Reality..How much will making execution
faster actually help?Should you look at the whole
process?Maybe you should fix something
else (as well)?Could you use TA implementation
as a driver for new development culture?
Lunchtime!Lunchtime!
I have reserved one hour for lunch.
Towo Toivola 2009
1155
Question #4
How would you choose your TA tool?(Since you want to do TA, you need to somehow decide which tooling to use. How do you decide? How do you make
the decision an informed one?)
1255
Group Work Time
First quickly by yourselfThen within your groupDocument Be prepared to present
Results and Discussion for Question #4
How would you choose your TA tool?
Finding a ToolFinding a ToolRemember that a tool is not the
same as test automation“A fool with a tool is a more
dangerous fool”There are many vendorsThere are many free alternativesTool finding must be considered
real work◦With goals◦With resources Towo Toivola 2009
Cost, What is it?Cost, What is it?Many costs
◦Cost of the tool (in the receipt)◦Cost of taking the tool into use
(mainly hidden) Including maintenance
◦Cost of not doing something else with the time
◦Cost of maintenanceYou should consider all costsCost of tool is usually overrated
Towo Toivola 2009
Impact of Tool on WorkImpact of Tool on Work
Towo Toivola 2009
Time
Now, what is expensive?
Cheap unsuitable tool
Big tool project
Small tool project
Project end
My AdviceMy Advice
Always take the one that is most suitable for you, taking into account:
Cost of toolIntended usePersonnel skills and attitudesGoals of the tool projectFuture plans for test automationDon't standardize for the sake of
it Towo Toivola 2009
Plan the Tool FindingPlan the Tool FindingMake it a proper project
If you still do projects
Plan – set goalsResource explicitlyHave many people join inScope out the tools out thereSelect a couple for trials
◦Do real work with them◦Not important, busy projects!
Towo Toivola 2009
RememberRememberYou will need support in using
the toolYou will need changes to the toolYour software will changeThe platform your software will
run on will change as wellYou may need changes to your
softwareOr your development process..
Towo Toivola 2009
What Should it be Like?What Should it be Like?Relatively easy to take into useEnables automatic testing, not
just playing with automation◦Including reporting
Supports your technology wellDoes not seriously restrict your
testersCan be run by a single person
and as a central systemFits your organization
Towo Toivola 2009
Or do You Need a Tool?You could make one, afterall you
make software Just right for you Intergrates nicely with your product and
development process
You could just use common utilities to create a mash-up tool
Build server SSH, remote exec Common scripting languages (with TA
libraries..)
In a Modern, Efficient, Agile OrganizationIn a galaxy far, far away..
The best people to make the trade-off decisions between tools are seasoned software development engineers who work in agile teams that have holistic responsibility for developing the software
Trust your professionals to make the right choice and support them
Their knowledge, interest and motivation are priceless in the implementation work of the TA
Success will truly inspire others
Question #5
What is the right level for doing TA in your organization?
(Should the SWE:s do UT/TDD? Should there be paired test engineers
perfoming them? Or module tests? Should there be multi-tier integration
tests? Can you do acceptance-level TA? Should you?)
1345
Group Work Time
First quickly by yourselfThen within your groupDocument Be prepared to present
Results and Discussion for Question #5
What is the right level for doing TA in your organization?
What is Module- and Unit What is Module- and Unit testing?testing?
Towo Toivola 2009
Graphical User
Interface (GUI)
Module testing environment
Unit Testing Framework
When significant part of functionality
available in user interface.
When functionality is available from
below GUI or from side (components / modules and their
interfaces)
When each unit can be used to
catch the problems
“tester territory” “developer territory”
GUI Testing Tool
Application Programming
Interface (API)
Testing LevelsTesting Levels
Towo Toivola 2009
Cost / found
bug
Bug finding potential
System
Integration
Module
Unit
0% 100%
Can you find the silver bullet?
Why Test Automation on Why Test Automation on Many LevelsMany LevelsFaster automationFaster feedback loop
◦Ability to do snap changesBetter quality
◦Ability to reach and maintain higher quality
Faster (= cheaper) implementationLess maintenance work
◦Again, = cheaper
End result: Cost efficiencyTowo Toivola 2009
Differences to GUI Differences to GUI AutomationAutomationAPI:s are more straightforward to use
◦ Implementation is faster and cheaper◦ Tools are often free of charge
API:s change much more seldom than a GUI◦ Requires less maintenance
Some tests that are possible through API:s are not possible to run using the GUI
Some tests that are possible through the GUI are not possible to run using the API◦ You will test a bit less of your application◦ You can, probably, also module-test your GUI
Running takes typically less time Code – test – fix – retest in minutes
Detected bugs are usually easier to fix
Towo Toivola 2009
Question #6
How to take TA into use?(Who should do it? Where do we get the resources? Do we have the skills? Which project? Which product? How to create the culture? How to ensure the long-
term commitment of the organization?)
1430
Group Work Time
First quickly by yourselfThen within your groupDocument Be prepared to present
Results and Discussion for Question #6
How to take TA into use?
Slide of Truth About Everything
Seriously, did you expect that I would have answers to that
question for you?Some thoughts will follow, but remember that every case is
unique.
Test Automation Test Automation ISIS Software Software System DevelopmentSystem DevelopmentGood test automation requires
that you understand testing well,And software developmentAnd system
implementation/administrationAnd preferably test automation
too.
Towo Toivola 2009
Most people don't reeeally get this
What Does That Mean?What Does That Mean?Source control and versioningBug trackingRequirements definitionProject management
◦Iterative and incremental if you want to be successful
ResourcingTesting!Fast feedback loop!Long-term commitmentETC.
Towo Toivola 2009
Don’t Leave the 20% Don’t Leave the 20% UndoneUndone
◦Usability◦Reliability◦User base◦Significance◦Reporting◦Analysing problems◦Maintenance
Towo Toivola 2009
Make sure you generate value ( = $$$$ )not just run tests
My serious main recommendation:One test case. Full chain. Beginning of
project.
Remember: TA is only useful when it
provides information that..Gets a bug fixed
Helps you make a decision
VISIBILITY!
Ugly slide..
Question #7
How will you change your products to ease TA?
(Should you? How? Should you change your process? Why?)
1530
Group Work Time
First quickly by yourselfThen within your groupDocument Be prepared to present
Results and Discussion for Question #7
How will you change your products to ease TA?
A Word of WarningA Word of WarningAutomating an untestable
software can be ruinously expensive
Prepare for organization struggles when you need the software changed
And the processPrepare for the organization to
ignore any bugs found by automation unless they are reproduced by hand
Towo Toivola 2009
FurthermoreDeveloping untestable software is
ruinous anyway, stop doing itThere are N people changing the
software, how will n TA people keep up?
Who investigates the problems found in TA?
Who fixes? The bugs in software? The changes needed for TA?
Question #8
Resourcing TA in the long run?(Who will make sure TA keep generating value next year and the one after? How? Changing software! Maintenance hell!)
1600
Group Work Time
First quickly by yourselfThen within your groupDocument Be prepared to present
Results and Discussion for Question #8
Resourcing TA in the long run?
Maintenance Maintenance If you get your tests running fine,
don’t think that’s most of the work done
The rest of your R&D works constantly – to change your software
Maintenance is in top 2 killers of test automation◦The other is organizational issues
You must be able to maintain your test code, constantly and easily
Towo Toivola 2009
Which Bit Would You Like to Which Bit Would You Like to Maintain?Maintain?
Towo Toivola 2009
How the Math Goes..How the Math Goes..You invest 3 people for automationYou invest 2 months for finding the
toolYou invest 6 months for developing
a good suite of testsChanges in the application make
the tests unreliable, so they will not be run, the organization will not believe in bugs found by the system
You just lost more than 3x(2+6)x5000e = 120 000e worth of investment
I did not include the price of the tool.. Towo Toivola 2009
What Should We LearnWhat Should We LearnImplementing a test automation
system is implementing a constantly supported service system based on software
Good system building and programming practises must be followed◦Duplicate code is your arch enemy
Resources must be allocated to maintain the system at all times
Towo Toivola 2009
08/03/10 Towo Toivola 2009
Something I Have Learned Something I Have Learned Since the 90’sSince the 90’sModel-based test automation
techniqueImportance of visual controlsSocial mass is more important
than technical massJust do it
Things we (probably) didn't have time for..
Technical detailsData-driven vs keyword-driven vs
model-basedDiscussing individual toolsDiscussing how to make software
in generalMuch else..
Debrief of the DayDebrief of the DayWhat do we remember from today?Was this day useful?
◦ What needs to be improved?◦ What was best?
What will you do differently in the future?
The topic is free, let’s have a discussion!
Towo Toivola 2009
BRoA #53 If you learn something and change nothing, you didn't learn anything.
Thank You for Your TimeThank You for Your Time
Towo Toivola 2009
We made it to the finish!
Next is a bunch of reserve slides that may come in handy at some
point...
RobustnessRobustnessAutomation scripts must run
reliablyWhen not testing something you
should perform operations in the most robust way possible
Error handling logicDuplicate code causes
unreliability and maintenance hell
Common operations belong to libraries◦Maintained, high-quality
Towo Toivola 2009
LibrariesLibraries
Towo Toivola 2009
Testing value
Required quality
Amount maintenance Test case
scripts(dozens or hundreds)
Test drivers(handful)
Utility libraries(handful) Use-case
libraries(handful)
Robustness exampleRobustness example
Towo Toivola 2009
Data-DrivenData-DrivenIf automation can do something,
it makes sense to do it a lotLoop over data, run
combinations◦Example: Log in with 10 incorrect
accounts and many correct accounts
Towo Toivola 2009
Keyword-DrivenKeyword-DrivenBasically a new language, just
for your testingReading in commands and data
instructions from a simple, human readable format
Enables non-technical people to understand and generate test cases◦Bring in the business knowledge
Requires quite a bit of workTowo Toivola 2009
(a very simple)(a very simple) Example Example
Go: EM_mainscreenDo: Enter message: “Hello work”Go: LogoutGo: EM_mainscreenDo: Verify message: “Hello work”
Towo Toivola 2009
ATDDATDDAcceptance/automated test
driven developmentAgree on the test first, on a level
understandable for allKeywords implement the testSoftware is developed to meet
the testHighly advanced development /
automation method
Towo Toivola 2009
(a very simple)(a very simple) Example Example
Given at effort manager login screen
When login with incorrect account
Then incorrect login reported
Towo Toivola 2009
Model-Based TestingModel-Based TestingCreating a state-machine that
models (some parts) of your software
Verifying the software behaves according to the state-machine
Combining transitions and data in many ways◦Different algorithms exist◦Random is pretty effective
Challenging, very advancedTowo Toivola 2009
(a very simple)(a very simple) Example ExampleState: emanager-login-screenInput: login-incorrectResultstate: badlogin-complaintInput: login-correctResultstate: emanager-mainscreenState: emanager-mainscreenInput: logoutResultstate: emanager-login-screen
…
Towo Toivola 2009
Automatic TestingAutomatic TestingBig difference between computer
aided testing and automatic testing
Automatic is automatic, human intervention is not needed◦Starts automatically ◦Runs automatically◦Analyses results automatically◦Reports automatically
Could your tests do that?Towo Toivola 2009
Exercise: AutomaticExercise: Automatic
Make your tests run so that they can be executed automatically
• Cmdline usage of TestPartner
• Driver Script
We should have about a half an hour.
Towo Toivola 2009
ReportingReporting
How would you like to get the test results?
Opening the tool?Going to the DB?By email?On a web page?In a file?As SMS?In public view?
Towo Toivola 2009