Applying generative testing in the browser with Selenium

transcript

Generating & Running 1 million tests per hour

#Seconf16

Sean Grove VP Eng @ PayGarden

sean@bushi.do @sgrove

http://github.com/sgrove

Caveats• Thought experiment • Greenfield projects • Functional stack (language, database, etc.) • Assume basics covered: Fully-parallelized test setup

• Browsers • Servers • Runners/reporters

What’s testing for?Exploring state-transition space for invalid states

Step Transition State Valid?

1 Add item {:item-ids [1]} TRUE

2 Remove item {:item-ids []} TRUE

nonexistent item

{:item-ids [nil]} FALSE

Cart state transitions

Testing Today• Largely example based

• Hand-Written unit tests (including Page object, etc.) • Recorded examples

• We tend to think we can manually enumerate all important state transitions. We’re very optimistic in that way (combinatorial explosion!)

Example app: Osmium

Book summarizing site

OSMIUM ACTIONS

• Signup • Update account • Login • Logout • View account

ACCOUNTS:

• List • Submit book • View book • Rate book • Update book

Books:

Site flow

View book

List books

Site flow

View book

List books

If not logged inAlways availableSite flow

View book

List books

Sign up

If not logged inAlways availableSite flow

View book

List books

Sign up

If not logged inAlways available

If logged inSite flow

View book

List books

Sign up

Login View account

Update account

Logout

View book

List books

Sign up

LoginSubmit book

View account

Update account

Logout

View book

List books

Sign up

LoginSubmit book

Rate book

View account

Update account

Logout

View book

List books

Sign up

LoginSubmit book

Rate book

Update book

View account

Update account

Logout

Example user flows

Action

- When logged in

- When logged out

List books

Example user flows

Action

- When logged in

- When logged out

List books

Example user flows

View book

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Action

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List booksLogout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

View account

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

View book

View account

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

View book

View account

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

View book

View account

Logout

List books

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…?List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…?…?…?

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…?…?…?

…?…?

…?…

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…?…?…?

…?…?

…?…

? …?…

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…

…?…?…?…?

…?…?

…?…

? …?…

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…

…?…?…?…?

…?…?

…?…

? …?…

…………

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…

…?…?…?…?

…?…?

…?…

? …?…

…?…

…………

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…

…?…?…?…?

…?…?

…?…

? …?…

…?…?

…?…

…………

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…

…?…?…?…?

…?…?

…?…

? …?…

…?…?

…?…?…?

…?…?

…?…

…………

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Example user flows

View book

Sign up

Sign up Login

List books

View book

List books

…?…?

…?…?…?

…?…?

…?…

…?…?…?…?

…?…?

…?…

? …?…

…?…?

…?…?…?

…?…?

…?…

…………

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Action

- When logged in

- When logged out

List books

Test cases trace specific paths

View book

Sign up

Sign up Login

List books

View book

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

List books

View book

Sign up

Sign up Login

List books

View book

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

List books

View book

Sign up

Sign up Login

List books

View book

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

List books

View book

Sign up

Sign up Login

List books

View book

List books

Submit book

Logout

View book

View account

Logout

List books

View account

Update description

Rate book

Logout

Even worse when considering concurrent accessing to

shared resource

shared resourceCan introduce state transitions not possible in linear flow

Scenario, two tabs open in a browser:

shared resource

TAB A TAB B

Can introduce state transitions not possible in linear flow

shared resource

TAB A TAB BLogin

shared resource

TAB A TAB BLogin

Go to “Add book page”

shared resource

TAB A TAB BLogin

Go to “Add book page”Fill out form

shared resource

TAB A TAB BLogin

Logout

shared resource

TAB A TAB BLogin

LogoutSubmit form

shared resource

TAB A TAB BLogin

LogoutSubmit form

What’s the resultant state? Book created? Did we write a test for it?

Alternative:

Don’t write (or record) tests

Alternative:

tests, generate them!

- John Hughes

Alternative:

tests, generate them!

or….

“Automate all the things!”

- Priti Biyani, 1 hour ago

or….

“Automate all the things!”

- Priti Biyani, 1 hour ago

Even writing tests…

or….

Core prop: Use infrastructure instead of human time.

Nearly same effort (with practice) as traditional example-based tests, but scales

to millions of examples

Constant effort vs linear effort

What properties can we assert over the whole

graph?• Page

• Should never have a JavaScript error • Should never have a 200, 30*, 401, 426

response • If a user is logged in, they should always

see a “logout” link (and vice-versa)

List books

List books Login

View book

Sign up

Sign up Login

List books

View book

List books

Submit book

Logout

View book

View account

List books

View account

Update Rate

Logout

What properties can we assert over specific

transitions?• State

• User must have been logged in • Number of ratings for the book should be

increased by 1 • updated_at timestamp should have changed

• Page • Logout link must be visible

Update book

• State • User must have been logged in • updated_at timestamp should have changed

• Page • Logout link must be visible • Must be at the view_book url for the reviewed book

Rate book Logout

• State • User must no longer be logged in

• Page • Login link must be visible

Submit book

• State • Book must have a submitter_id • User therefore must be logged in • Book must have certain properties

• Page • Must be at the view_book url

How can we teach a computer to navigate our

app?• Each page has allowed inbound edges and outbound edges

• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).

• Bit clunky and prone to rot.

• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use

that as a model for generating valid transitions?

that as a model for generating valid transitions?• Render page to intermediate data structure.

that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action

that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action• Automatically tag page structure with a .osmium-action class

that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action• Automatically tag page structure with a .osmium-action class• Walk down an edge and recur to generate possible paths

that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action• Automatically tag page structure with a .osmium-action class• Walk down an edge and recur to generate possible paths• At each step along the walk, run the relevant assertions

Example path

Example path[

Example path[[:to “demo.localbox.info:3005"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click “#osmium-1"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click “#osmium-1"][:osmium.user/email "#osmium-14"]

Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click “#osmium-1"][:osmium.user/email "#osmium-14"]]

Example test (we should never show errors)

(deftest no-errors-test (testing "We should never show an error page" (let [driver (sl/new-sauce-labs-browser) db (:db @o/system) sessions (-> @o/system :session-store .session-map)] (eval! driver [:to "demo.localbox.info:3005"])

(let [actions (walk-n-steps! driver 5 (fn check-step [a path] (let [title (taxi/title driver) error (re-find #"Internal Error" title)] (record-failing-path-when! (not (nil? error)) path))))] (reset! replay {:actions actions :step 0})) (taxi/quit driver))))

• Assumptions • Fully immutable database (Datomic) • Functional, referentially-transparent application (explicit

state)

Challenges in Generative Testing

• Testing the right properties

• Testing the right propertiesBig shift in approach takes time and practice to internalize

• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)

• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here

• Long tests

• Long testsFailing tests may contain thousands and thousands of retro steps, many of which are irrelevant.

Answer: Shrinking!

Answer: Shrinking!• Extreme scale

• Depth of tests + desired coverage + shrinking process = massive demands on infrastructure (both server and browser side)

Shrinking

ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]

• Good news: Found a failing path!

• Good news: Found a failing path!• Bad news, it’s 2000+ steps long

(imaging a user reporting this case, then trying to debug it…)

• Take out some steps, rerun path, see if it still fails

• Repeat until you have a minimum repro case

Minimum case: 7 steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-16"] ]

Recording has its place, but it’s not in the browser.

It’s in the database.

Record state at each step of the test, then run very fast tests over the state

So why 1,000,000 assertions per hour?

• Very cool, but not the point:

• Very cool, but not the point:• Depth (longer sessions, more changes in state)

• Very cool, but not the point:• Depth (longer sessions, more changes in state)• Breadth (desired coverage over branches)

• Very cool, but not the point:• Depth (longer sessions, more changes in state)• Breadth (desired coverage over branches)• Shrinking process

Infrastructure is a challenge, but tractable

• Generative testing has seen uses in many areas, but almost nothing in Selenium/Appium world - infrastructure is the biggest challenge

• CircleCI - scale out your build process automatically

• CircleCI - scale out your build process automatically• Sauce Labs - Scale out browser infrastructure as on

demand

Future Areas

Concolic Testing

Concolic TestingConcrete Tests

Concolic TestingConcrete Tests Symbolic Execution

Terrible name

Concolic TestingConcrete Tests Symbolic Execution

Terrible nameMind-blowingly cool idea

(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))

• We have to generate a very specific value (42) to trigger the exception. We can run infinite tests in either directions and still miss it.

• Bring in code analysis!

Three code paths(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))

Three code pathsGuarded by three conditions

• (< internet-points 42) <- Constrain generator for input to super-user?

• Repeat for (> internet-points 42)

• Repeat for (> internet-points 42)• Finally repeat for (= internet-points 42)

Predictive Testing

Predictive Testing• Record paths your users take through your site

• If an error is triggered, email devs with repro steps automatically

Predictive Testing• Record paths your users take through your site

• If an error is triggered, email devs with repro steps automatically

• Before pushing new code, re-run previous e.g. 10,000 sessions on your site, see if you’ve broken anything your users actually do in your app

Summary

• Humans are terrible at manually enumerating all possible state-space

Summary

• Computers are very good at finding surprising edge cases (just like users!) given some help

Summary

• Computers are very good at finding surprising edge cases (just like users!) given some help

• Don’t write tests - generate them!

CreditAlmost all the ideas/experimentation/actual

work done by Sebastian Bensusan

@sebasbensu http://github.com/bensu

Credit

https://www.paygarden.com/

Thank you!Questions?

sean@bushi.do https://www.riseos.com

Resources

• John Hughes - Testing the Hard Stuff and Staying Sane [Clojure/West 2014]

• Kostis Sagonas - A CutEr Tool [The Erlang Factory 2016] • Reid Draper - Powerful Testing with test.check [Clojure/

West 2014]

Applying generative testing in the browser with Selenium

Software