32
Scaling Selenium Noah Sussman Etsy Friday, September 23, 2011
| Date post: | 23-Jun-2015 |
| Category: |
Technology |
| Upload: | noah-sussman |
| View: | 2,061 times |
| Download: | 0 times |
Scaling Selenium
Noah SussmanEtsy
Friday, September 23, 2011
2006: Test JavaScript and CSS locallyFriday, September 23, 2011
2011: Test the whole Web application stack. Friday, September 23, 2011
Why would we need to scale our Selenium testing?Friday, September 23, 2011
MTBF: Mean Time Between FailuresMTTR: Mean Time To Recover
MTTR > MTBFFriday, September 23, 2011
It’s not whether you get knocked down; it’s whether you get back up. ~Vince Lombardi
Friday, September 23, 2011
Recovery usually requires deploying new codeFriday, September 23, 2011
faster deploys = smaller changesets = fewer change-related outages
Friday, September 23, 2011
End-to-end tests are slow as hellFriday, September 23, 2011
Friday, September 23, 2011
Scaling communication, concurrency and hardwareFriday, September 23, 2011
22:05:58.241 INFO - Command request: type[listing[quantity], 3] on session f01069b424224b7ebe85040fd2ee8e9d22:05:58.775 INFO - Got result: OK on session f01069b424224b7ebe85040fd2ee8e9d22:05:58.777 INFO - Command request: select[css=#shipping select, United States] on session f01069b424224b7ebe85040fd2ee8e9d22:05:59.376 INFO - Got result: OK on session f01069b424224b7ebe85040fd2ee8e9d22:05:59.386 INFO - Command request: type[country_shipping[209][primary], 1.00] on session f01069b424224b7ebe85040fd2ee8e9d22:05:59.919 INFO - Got result: OK on session f01069b424224b7ebe85040fd2ee8e9d22:05:59.921 INFO - Command request: type[everywhere_shipping[primary], 2.00] on session f01069b424224b7ebe85040fd2ee8e9d22:06:00.462 INFO - Got result: OK on session f01069b424224b7ebe85040fd2ee8e9d22:06:00.466 INFO - Command request: attachFile[css=input[id=image-upload], http://ny-image2.etsy.com/il_170x135.262709802.jpg] on session f01069b424224b7ebe85040fd2ee8e9d22:06:01.063 INFO - Got result: OK on session f01069b424224b7ebe85040fd2ee8e9d22:06:01.066 INFO - Command request: attachFile[css=input[id=image-upload], http://ny-image0.etsy.com/il_170x135.262774260.jpg] on session f01069b424224b7ebe85040fd2ee8e9d22:06:01.614 INFO - Got result: OK on session f01069b424224b7ebe85040fd2ee8e9d22:06:01.616 INFO - Command request: attachFile[css=input[id=image-upload], http://ny-image1.etsy.com/il_170x135.262743857.jpg] on session f01069b424224b7ebe85040fd2ee8e9d22:06:02.160 INFO - Got result: OK on session f01069b424224b7ebe85040fd2ee8e9d22:06:02.164 INFO - Command request: isElementPresent[css=#image_0, ] on session f01069b424224b7ebe85040fd2ee8e9d22:06:02.689 INFO - Got result: OK,true on session f01069b424224b7ebe85040fd2ee8e9d22:06:02.691 INFO - Command request: isElementPresent[css=#image_1, ] on session f01069b424224b7ebe85040fd2ee8e9d22:06:03.217 INFO - Got result: OK,true on session f01069b424224b7ebe85040fd2ee8e9d22:06:03.220 INFO - Command request: isElementPresent[css=#image_2, ] on session f01069b424224b7ebe85040fd2ee8e9d22:06:03.745 INFO - Got result: OK,true on session f01069b424224b7ebe85040fd2ee8e9d22:06:03.748 INFO - Command request: isElementPresent[css=input[value="Preview Listing"], ] on session f01069b424224b7ebe85040fd2ee8e9d22:06:04.274 INFO - Got result: OK,true on session f01069b424224b7ebe85040fd2ee8e9d22:06:04.276 INFO - Command request: click[css=input[value="Preview Listing"], ] on session f01069b424224b7ebe85040fd2ee8e9d22:06:04.849 INFO - Got result: OK on session f01069b424224b7ebe85040fd2ee8e9d22:06:04.851 INFO - Command request: waitForPageToLoad[90000, ] on session f01069b424224b7ebe85040fd2ee8e9d
==> Failed to boot the Selenium Server... exiting! Given I am logged into my Etsy account When I click the 'Your Account' link And I click the 'Add New Item' link And I enter a title for my new listing And I enter a description for my new listing And I add tags And I enter materials for my new listing And I select a maker from the drop down list And I select what it is from the drop down list And I select when it was made from the drop down list Then the item information is displayed in my listing When I select a few categories from the drop down list Then the category and tag information is displayed in my listing When I add selling information When I load several images Then the image is displayed in my listing And I click the Preview Listing button Timeout::Error (Timeout::Error) /usr/lib64/ruby/1.9.1/net/protocol.rb:140:in `rescue in rbuf_fill' /usr/lib64/ruby/1.9.1/net/protocol.rb:134:in `rbuf_fill' /usr/lib64/ruby/1.9.1/net/protocol.rb:116:in `readuntil' /usr/lib64/ruby/1.9.1/net/protocol.rb:126:in `readline' /usr/lib64/ruby/1.9.1/net/http.rb:2219:in `read_status_line' /usr/lib64/ruby/1.9.1/net/http.rb:2208:in `read_new' /usr/lib64/ruby/1.9.1/net/http.rb:1191:in `transport_request' /usr/lib64/ruby/1.9.1/net/http.rb:1177:in `request' /usr/lib64/ruby/1.9.1/net/http.rb:1170:in `block in request' /usr/lib64/ruby/1.9.1/net/http.rb:627:in `start' /usr/lib64/ruby/1.9.1/net/http.rb:1168:in `request' /usr/lib64/ruby/1.9.1/net/http.rb:978:in `post' /usr/lib64/ruby/1.9.1/timeout.rb:57:in `timeout' /usr/lib64/ruby/1.9.1/timeout.rb:87:in `timeout' ./features/step_definitions/list_item.rb:106:in `/^I click the Preview Listing button$/' features/prod_list_item.feature:24:in `And I click the Preview Listing button'22:07:34.874 INFO - Command request: testComplete[, ] on session f01069b424224b7ebe85040fd2ee8e9d22:07:34.874 INFO - Killing Firefox...22:07:34.888 INFO - Got result: ERROR: Got a null result on session f01069b424224b7ebe85040fd2ee8e9d22:07:34.889 INFO - Got result: OK on session f01069b424224b7ebe85040fd2ee8e9d When I review my listing And I publish my listing Then my item listing is posted on Etsy
Failing Scenarios:cucumber features/prod_list_item.feature:7
Most people have a hard time interpreting the results of one failing test
Friday, September 23, 2011
Aggregating and reporting on test results becomes increasingly challenging.
Friday, September 23, 2011
Friday, September 23, 2011
graphs > emailsFriday, September 23, 2011
Friday, September 23, 2011
Use Jenkins to run tests concurrently.Friday, September 23, 2011
There are other ways to do it, but Jenkins was the simplest (for us).
Friday, September 23, 2011
Jenkins Matrix Build pluginFriday, September 23, 2011
Master Build
######
######
########
Friday, September 23, 2011
Friday, September 23, 2011
Most regressions are catastrophicFriday, September 23, 2011
Walk the happy pathFriday, September 23, 2011
early 2010: Headless Firefox on Linux, raw EC2 instances
Friday, September 23, 2011
late 2010:Physical Hardware
Friday, September 23, 2011
2011: SauceLabsxxxx
xxxx
Friday, September 23, 2011
hybrid cluster
Friday, September 23, 2011
early 2010: about 10 people making 5 deployments a day ~150 Selenium tests run every daymid 2011: about 20 people making 30+ deployments ~700 Selenium tests a day
Friday, September 23, 2011
Lessons Learned
Communication and discoverability remain our biggest challenges.
Jenkins’ concurrent build capacity is a bottleneck.
Even with a vendor in front of it, the cloud is still the cloud.
Friday, September 23, 2011
etsy.com/careers
Friday, September 23, 2011
photo creditsparkour guy: loki_v http://www.flickr.com/photos/loki_v/268938583/easy button: crd http://www.flickr.com/photos/crd/283156445sleeping cat: binibix http://www.flickr.com/photos/binibix/6090137450etsy bike rack: cubicle spy http://mashable.com/2010/10/13/etsy-office-picsetsy infrastructure diagram: john allspawetsy deployment velocity graph: erik kastnerupside down server: chris munns http://www.flickr.com/photos/teknogeek/5316475923/
Friday, September 23, 2011
![[320] Web 3: Selenium · for Selenium Java module for Selenium Ruby module for Selenium JavaScript mod for Selenium Chrome Driver Firefox Driver Edge Driver. Examples. Starter Code](https://static.documents.pub/doc/80x56/5eadce82cc4f0d7405687f01/320-web-3-selenium-for-selenium-java-module-for-selenium-ruby-module-for-selenium.jpg)
