Pitfalls and tips with Pitfalls and tips with browser based browser based
testIngtestIng
My Background
several projects with functional testing
consulted at organizations to introduce functional testing and builds
heavily involved in trying to improve effectiveness and reduce maintenance
developed a framework over several projects that made use of my learnings
Common uses of functional tests
smoke/sanity tests for most valuable and risky features
black box/regression tests
post deployment checks and rollbacks
multi browser testing
what do we want from tests?
correctness (when my tests work, my application is not broken)
robustness (when my tests break, my application is broken)
what do we focus on?
what’s valuable? (it matters to us that people can do this)
what’s risky? (if this breaks, it really affects us)
so whats the so whats the problem?problem?
five problemsfive problems
1. slow tests1. slow tests
BecauseBecause
uses the uses the entire stackentire stack
low spec hardwarelow spec hardware
tons of setuptons of setupand teardownand teardown
talking to the talking to the browser is slowbrowser is slow
soso
very long feedback very long feedback looploop
can delay can delay deploymentsdeployments
2. Brittle tests2. Brittle tests
UI UI changes changes
a lota lot
tests over tests over specifyspecify
3. Random failures3. Random failures
race conditionsrace conditions
unreliable unreliable dependenciesdependencies
4. tests are hard to 4. tests are hard to debugdebug
too many too many tests breaktests break
error messageserror messagestell you nothingtell you nothing
5. test code becomes 5. test code becomes unmaintainableunmaintainable
insufficient insufficient commitment to commitment to
testingtesting
inadequate inadequate understanding understanding
of testingof testing
what happens
there is an initial stream of tests
tests break unnecessarily
tests become hard to maintain
team loses confidence and motivation
some teams even abandon testing via the UI
tips and trickstips and tricks
Treat test codelike real code
speed
maintainability
A lot of the techniques we A lot of the techniques we use with real code work use with real code work
with test codewith test code
similar levels of similar levels of commitmentcommitment
test test creation/maintenancecreation/maintenancecan be part of the dev can be part of the dev
cyclecycle
Use Ruby
maintainability
development time
use a headless browser when possible
speed
Use selenium RC (if you use selenium)
avoid brittleness
maintainability
Don’t record and replay
(if you use selenium)
avoid brittleness
maintainability
development time
Introduce an Abstraction layer
speed
maintainability
Categorize/tag tests
speed
maintainability
Use the page object pattern
maintainability
Parallelize
speed
separate data for tests
speed
avoid random failures
clump data into common location
maintainability
Push setup and teardown out of tests
speed
avoid brittleness
development time
debuggability
Only exercise parts of the application you
want to testspeed
avoid brittleness
debuggability
Build stub services
speed
avoid random failures
don’t over specify
avoid brittleness
debuggability
run on better hardware
speed
slice vertically
debuggability
Use the right locator
maintainability
avoid brittleness
speed
learn xpath/css selectors
avoid brittleness
maintainability
fail later
speed
use polls over sleep
speed
avoid random failures
use existing waits of selenium
speed
avoid random failures
avoid brittleness
avoid random
Don’t use assertions to identify transitions
avoid brittleness
debuggability
avoid brittleness
avoid random
screenshots and html dumps
debuggability
xpath/css selector tester
development time
firebug or w/e
development time
maintainability
talk to me if you’re talk to me if you’re testing tabular datatesting tabular data
I’m always available I’m always available for questionsfor questions
Questions?Questions?
thank youthank you
