How do you implement

Continuous Delivery?

Part 4: Automated Testing

Implementing

automated testing

Agile Testing Quadrants Implementing automated testing

support programming critique the product

tech

nolo

gy fa

cing

bu

sine

ss fa

cing

http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2 http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/

Agile Testing Quadrants Implementing automated testing

support programming critique the product

tech

nolo

gy fa

cing

bu

sine

ss fa

cing

functional tests prototypes simulation

showcases exploratory testing

usability testing

performance tests load tests

security tests

unit tests component tests

system tests

http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2 http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/

Agile Testing Quadrants Implementing automated testing

Balance your tests

Software Testing Pyramid

Unit tests

Service tests (API, Integration,

Component)

UI tests

Implementing automated testing

Manual Tests

http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid

End-to-end Business facing

Localized Technology facing

Software Testing Pyramid Implementing automated testing

Unit tests

§  Tests at the bottom of the pyramid focus on

smaller sections of code, e.g. unit tests.

§  These tests are the foundation of a good test

automation strategy, they are quick to run and

there should be many of them.

§  They run at the earlier stages of the pipeline.

Software Testing Pyramid Implementing automated testing

Service Tests

§  Tests in the middle of the pyramid cover larger

aggregation of code - components, services, etc.

§  Service tests provide many advantages of end-

to-end tests while avoiding UI complexities.

§  They run only after the build has passed unit

level tests.

Software Testing Pyramid Implementing automated testing

§  Tests at the top cover the "full stack” and are the

slowest to run.

§  Don’t write a test for every acceptance criteria

(antipattern), instead use a few journeys to cover

main areas of the code. 

§  They run only after the build has passed both

the unit level and service level tests.

UI

tests

Implementing automated testing

Working practices

ü  Testers and developers should collaborate to write, run and maintain

tests.

ü  Siloed testing where development hands over tests to QA not only

creates long feedback loops, but also leads to testers duplicating

automated tests with manual tests.

ü  Expensive automated testing tools tend to make the feedback loop

worse. Developers should be able to run all tests, including performance

tests, to help them reproduce and diagnose any issue reported by QA.

Anti-Pattern: Ice-cream cone

http://watirmelon.com/2012/01/31/introducing-the-software-testing-ice-cream-cone/

Unit

tests

Service tests

(API, Integration, Component)

UI tests

Implementing automated testing Manual Tests

Anti-Pattern: Ice-cream cone Implementing automated testing

x  Avoid inverting your test pyramid

x  Testing like this through the UI is slow and leads to brittle tests.

x  Avoid using only a UI-oriented testing tool, as that focuses effort on

writing UI-level automated tests.

x  If a bug is found by users, manual testing or high level testing, push a

test to catch that lower down the pyramid. 

x  The only tests at a given level should be to test something that can't be

caught at a lower level, i.e. when testing multiple components together,

your tests should only check component integration, not each

component. That should have be done by lower-level tests.

q  Flaky tests

q  Tests take too much work to maintain

q  Too much effort to add tests for legacy codebases

Challenges Implementing automated testing

Test data

11150 23958945 8594594359 8498343 0 940938493 -9059894892728937 90808908 323232

Test Data Types of test data

q  Test-specific data: This is the data that drives the behaviour

under test. It represents the specifics of the case under test.

q  Test reference data: Data that needs to be there but actually

has little bearing upon the behaviour under test.

q  Application reference data: Irrelevant to the behaviour

under test, but needs to be there to allow the application to

start up.

References

Rather than using database dumps, use the application’s API to set up the correct state.

Don’t use production data.

Avoid dependencies between tests.

Best practices

Deployment Patterns

Stay tuned for Part 5…

Deploy a great product faster. Agile teams deliver working software early and often. Go automates and streamlines the build-test-release cycle for worry-free, continuous delivery of your product.

Learn More See how Go can help you in your CD journey

goContinuous Delivery