172
Generating & Running 1 million tests per hour #Seconf16
| Date post: | 11-Apr-2017 |
| Category: |
Software |
| Upload: | sean-grove |
| View: | 166 times |
| Download: | 1 times |
Generating & Running 1 million tests per hour
#Seconf16
Caveats• Thought experiment • Greenfield projects • Functional stack (language, database, etc.) • Assume basics covered: Fully-parallelized test setup
• Browsers • Servers • Runners/reporters
What’s testing for?Exploring state-transition space for invalid states
Step Transition State Valid?
1 Add item {:item-ids [1]} TRUE
2 Remove item {:item-ids []} TRUE
3Add
nonexistent item
{:item-ids [nil]} FALSE
Cart state transitions
Testing Today• Largely example based
• Hand-Written unit tests (including Page object, etc.) • Recorded examples
• We tend to think we can manually enumerate all important state transitions. We’re very optimistic in that way (combinatorial explosion!)
Example app: Osmium
Book summarizing site
OSMIUM ACTIONS
• Signup • Update account • Login • Logout • View account
ACCOUNTS:
• List • Submit book • View book • Rate book • Update book
Books:
Site flow
View book
List books
Site flow
View book
List books
If not logged inAlways availableSite flow
View book
List books
Sign up
Login
If not logged inAlways availableSite flow
View book
List books
Sign up
Login
If not logged inAlways available
If logged inSite flow
View book
List books
Sign up
Login View account
Update account
Logout
If not logged inAlways available
If logged inSite flow
View book
List books
Sign up
LoginSubmit book
View account
Update account
Logout
If not logged inAlways available
If logged inSite flow
View book
List books
Sign up
LoginSubmit book
Rate book
View account
Update account
Logout
If not logged inAlways available
If logged inSite flow
View book
List books
Sign up
LoginSubmit book
Rate book
Update book
View account
Update account
Logout
If not logged inAlways available
If logged inSite flow
Example user flows
Action
Action
- When logged in
- When logged out
List books
Example user flows
Action
Action
- When logged in
- When logged out
List books
Example user flows
View book
Action
Action
- When logged in
- When logged out
List books
Example user flows
View book
Sign up
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List booksLogout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Submit book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View account
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?…?
…?…?
…?…?List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?…?
…?…?
…?…?
…?…?
…?
…?
…?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…?…?…?
…?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…?…?…?
…?…?
…?…
?…
?…
?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?…
…………
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
…?
…?…
?…
?…
?
…………
…
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
…?…?
…?…?
…?
…?
…?…
?…
?…
?
…………
…
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
…?…?
…?…?
…?…?…?
…?…?
…?
…?
…?…
?…
?…
?
…………
…
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
…?…?
…?…?
…?…?…?
…?…?
…?…?
…?…?
…?…?
…?
…?…
?…
?…
?
…………
…
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
List books
Test cases trace specific paths
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
List books
Test cases trace specific paths
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
List books
Test cases trace specific paths
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
List books
Test cases trace specific paths
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Even worse when considering concurrent accessing to
shared resource
Even worse when considering concurrent accessing to
shared resourceCan introduce state transitions not possible in linear flow
Even worse when considering concurrent accessing to
shared resourceCan introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
Even worse when considering concurrent accessing to
shared resource
TAB A TAB B
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”Fill out form
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”Fill out form
Logout
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”Fill out form
LogoutSubmit form
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”Fill out form
LogoutSubmit form
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
What’s the resultant state? Book created? Did we write a test for it?
Alternative:
Don’t write (or record) tests
Alternative:
Don’t write (or record) tests
Alternative:
tests, generate them!
Don’t write (or record) tests
- John Hughes
Alternative:
tests, generate them!
or….
“Automate all the things!”
- Priti Biyani, 1 hour ago
or….
“Automate all the things!”
- Priti Biyani, 1 hour ago
Even writing tests…
or….
Core prop: Use infrastructure instead of human time.
Nearly same effort (with practice) as traditional example-based tests, but scales
to millions of examples
Constant effort vs linear effort
What properties can we assert over the whole
graph?• Page
• Should never have a JavaScript error • Should never have a 200, 30*, 401, 426
response • If a user is logged in, they should always
see a “logout” link (and vice-versa)
Login
List books
List books Login
View book
Sign up
Sign up Login
List books
View book
Login
List books
Submit book
Logout
View book
View account
List books
View account
Update Rate
Logout
What properties can we assert over specific
transitions?• State
• User must have been logged in • Number of ratings for the book should be
increased by 1 • updated_at timestamp should have changed
• Page • Logout link must be visible
Update book
• State • User must have been logged in • updated_at timestamp should have changed
• Page • Logout link must be visible • Must be at the view_book url for the reviewed book
Rate book Logout
• State • User must no longer be logged in
• Page • Login link must be visible
Submit book
• State • Book must have a submitter_id • User therefore must be logged in • Book must have certain properties
• Page • Must be at the view_book url
How can we teach a computer to navigate our
app?
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action• Automatically tag page structure with a .osmium-action class
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action• Automatically tag page structure with a .osmium-action class• Walk down an edge and recur to generate possible paths
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action• Automatically tag page structure with a .osmium-action class• Walk down an edge and recur to generate possible paths• At each step along the walk, run the relevant assertions
Example path
Example path[
Example path[[:to “demo.localbox.info:3005"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click “#osmium-1"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click “#osmium-1"][:osmium.user/email "#osmium-14"]
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click “#osmium-1"][:osmium.user/email "#osmium-14"]]
Example test (we should never show errors)
(deftest no-errors-test (testing "We should never show an error page" (let [driver (sl/new-sauce-labs-browser) db (:db @o/system) sessions (-> @o/system :session-store .session-map)] (eval! driver [:to "demo.localbox.info:3005"])
(let [actions (walk-n-steps! driver 5 (fn check-step [a path] (let [title (taxi/title driver) error (re-find #"Internal Error" title)] (record-failing-path-when! (not (nil? error)) path))))] (reset! replay {:actions actions :step 0})) (taxi/quit driver))))
Demo:
• Assumptions • Fully immutable database (Datomic) • Functional, referentially-transparent application (explicit
state)
Challenges in Generative Testing
Challenges in Generative Testing
• Testing the right properties
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalize
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long tests
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long testsFailing tests may contain thousands and thousands of retro steps, many of which are irrelevant.
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long testsFailing tests may contain thousands and thousands of retro steps, many of which are irrelevant.
Answer: Shrinking!
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long testsFailing tests may contain thousands and thousands of retro steps, many of which are irrelevant.
Answer: Shrinking!• Extreme scale
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long testsFailing tests may contain thousands and thousands of retro steps, many of which are irrelevant.
Answer: Shrinking!• Extreme scale
• Depth of tests + desired coverage + shrinking process = massive demands on infrastructure (both server and browser side)
Shrinking
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
(imaging a user reporting this case, then trying to debug it…)
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
(imaging a user reporting this case, then trying to debug it…)
• Take out some steps, rerun path, see if it still fails
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
(imaging a user reporting this case, then trying to debug it…)
• Take out some steps, rerun path, see if it still fails
• Repeat until you have a minimum repro case
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
(imaging a user reporting this case, then trying to debug it…)
• Take out some steps, rerun path, see if it still fails
• Repeat until you have a minimum repro case
Minimum case: 7 steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-16"] ]
Recording has its place, but it’s not in the browser.
Recording has its place, but it’s not in the browser.
It’s in the database.
Record state at each step of the test, then run very fast tests over the state
So why 1,000,000 assertions per hour?
So why 1,000,000 assertions per hour?
• Very cool, but not the point:
So why 1,000,000 assertions per hour?
• Very cool, but not the point:• Depth (longer sessions, more changes in state)
So why 1,000,000 assertions per hour?
• Very cool, but not the point:• Depth (longer sessions, more changes in state)• Breadth (desired coverage over branches)
So why 1,000,000 assertions per hour?
• Very cool, but not the point:• Depth (longer sessions, more changes in state)• Breadth (desired coverage over branches)• Shrinking process
Infrastructure is a challenge, but tractable
Infrastructure is a challenge, but tractable
• Generative testing has seen uses in many areas, but almost nothing in Selenium/Appium world - infrastructure is the biggest challenge
Infrastructure is a challenge, but tractable
• Generative testing has seen uses in many areas, but almost nothing in Selenium/Appium world - infrastructure is the biggest challenge
• CircleCI - scale out your build process automatically
Infrastructure is a challenge, but tractable
• Generative testing has seen uses in many areas, but almost nothing in Selenium/Appium world - infrastructure is the biggest challenge
• CircleCI - scale out your build process automatically• Sauce Labs - Scale out browser infrastructure as on
demand
Future Areas
Concolic Testing
Concolic TestingConcrete Tests
Concolic TestingConcrete Tests Symbolic Execution
Concolic TestingConcrete Tests Symbolic Execution
Terrible name
Concolic TestingConcrete Tests Symbolic Execution
Terrible nameMind-blowingly cool idea
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
• We have to generate a very specific value (42) to trigger the exception. We can run infinite tests in either directions and still miss it.
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
• We have to generate a very specific value (42) to trigger the exception. We can run infinite tests in either directions and still miss it.
• Bring in code analysis!
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
Three code paths(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
Three code pathsGuarded by three conditions
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
• (< internet-points 42) <- Constrain generator for input to super-user?
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
• (< internet-points 42) <- Constrain generator for input to super-user?
• Repeat for (> internet-points 42)
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
• (< internet-points 42) <- Constrain generator for input to super-user?
• Repeat for (> internet-points 42)• Finally repeat for (= internet-points 42)
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
Predictive Testing
Predictive Testing• Record paths your users take through your site
Predictive Testing• Record paths your users take through your site
• If an error is triggered, email devs with repro steps automatically
Predictive Testing• Record paths your users take through your site
• If an error is triggered, email devs with repro steps automatically
• Before pushing new code, re-run previous e.g. 10,000 sessions on your site, see if you’ve broken anything your users actually do in your app
Summary
Summary
• Humans are terrible at manually enumerating all possible state-space
Summary
• Humans are terrible at manually enumerating all possible state-space
• Computers are very good at finding surprising edge cases (just like users!) given some help
Summary
• Humans are terrible at manually enumerating all possible state-space
• Computers are very good at finding surprising edge cases (just like users!) given some help
• Don’t write tests - generate them!
CreditAlmost all the ideas/experimentation/actual
work done by Sebastian Bensusan
CreditAlmost all the ideas/experimentation/actual
work done by Sebastian Bensusan
CreditAlmost all the ideas/experimentation/actual
work done by Sebastian Bensusan
@sebasbensu http://github.com/bensu
Credit
https://www.paygarden.com/
Resources
• John Hughes - Testing the Hard Stuff and Staying Sane [Clojure/West 2014]
• Kostis Sagonas - A CutEr Tool [The Erlang Factory 2016] • Reid Draper - Powerful Testing with test.check [Clojure/
West 2014]
