TaaS - Test Automation as a Service
Oren Rubin
What have the cloud done for us lately?
Testim.io CEO
Applitools Director of R&D
WixWeb Architect
Cadence Compiler Engineer
IBM Cloud Engineer
Google Developer ExpertOrenRubin
Mentor Google Launchpad
External LecturerTechnion University
Meetup Co-organizerSelenium-IL, GDG, Ember
...
Testing Theory
La järɡən
Testing Theory
La järɡən
● Scope
○ Unit○ Integration○ End to End
● Type
○ Functional○ Visual○ Performance○ Load○ Stress
● Accuracy
○ Smoke○ Acceptance○ Coverage
Black Box
Check against an API
White Box
Look inside
BeholdThe Interweb!
So simple..
So simple.. Not!
E2E Path of Requests and Cache Hierarchy
BeholdThe Interweb!
FRONT END BACK END
Load/Stress Testing
Why linear? Why not hard start/stop?
CommonScenario
Metrics: CPU / Memory / Response
CommonScenario
Response Metrics: Throughput, Average, Median, Deviation
CommonScenario
Testing the Backend
Number of Threads | Ramp-Up | Loop Count
JMeter
JMeter in the Cloud
JMeterIn The Cloud
||Equivalent
Why Front-End/E2E
Matters?
Depends on the app
Front EndMatters
More Than Ever
Who Moved My Code?
FRONT ENDBACK END FRONT ENDBACK END
1997 2017
How to test Front End?
Meet Selenium
Many Languages
Many OS
BrowsersAll major browsers
On all major (desktop) operating systems
Can anyone suggest a good architecture?
HTTP as a universal language
?TheWire
Protocol
Binding(SDK) Driver
One (W3C) Standard to rule them all!
https://code.google.com/p/selenium/wiki/JsonWireProtocol http://www.w3.org/TR/webdriver/
Selenium != Selenium IDE
Meet Selenium's Hello World
Selenium
Meet Selenium IDE
Why Record/Playback
Sucks
Unreliable, Unreadable, Uneditable
Use Machine Learning to find elements
MeetTestim.io
Selenium Grid
new Chrome(); chromedriver.exe
Browser
HTTP Request
localhost:8989/click
driver = new Chrome();...submitBtn.click();
driver (Binding)
What We Want
What We Want
new Chrome();
chromedriver.exe
Browser
HTTP Request
localhost:8989/click
driver = new Chrome();...submitBtn.click();
driver (Binding)
Machine 1 Machine 2
java -jar selenium.jar -port 4444 -role hub
selenium hub
Node 0 on 10.0
Selenium Grid
Selenium Grid
Node 1 on 10.1
Node 2 on 10.2
selenium node
java -jar selenium.jar -role webdriver -browser firefox -hubHost 10.0 –port 8989
java -jar selenium.jar -port 4444 -role hub
selenium node
java -jar selenium.jar -role webdriver -browser chrome -hubHost 10.0 –port 8989
selenium hub
Node 0 on 10.0
chromedriver.exe
firefox-webdriver.dpi
Selenium Grid
driver (Binding)
new Chrome();
BrowserHTTP Request
10.0:4444/chrome
hub = new URL("10.0:4444/wd/hub")cap = new Capabilities ("chrome", "v3", "windows")driver = new RemoteWebDriver( hub, cap );...
my computer / CI server Node 2 on 10.2
selenium node
java -jar selenium.jar -role webdriver -hubHost 10.1 –port 8989
java -jar selenium.jar -port 4444 -role hub
selenium node
java -jar selenium.jar -role webdriver -hubHost 10.1 –port 8989
selenium hub
Node 0 on 10.0
HTTP Request
10.2:8989/chrome
chromedriver.exe
firefox-webdriver.dpi
Node 1 on 10.1
Node 1 on 10.1
new Chrome();
BrowserHTTP Request
10.0:4444/click
driver = ...
submitBtn = driver.find(By.id("submit"));submitBtn.click();
driver
my computer / CI server Node 2 on 10.2
selenium node
java -jar selenium.jar -role webdriver -hubHost 10.1 –port 8989
java -jar selenium.jar -port 4444 -role hub
selenium node
java -jar selenium.jar -role webdriver -hubHost 10.1 –port 8989
selenium hub
Node 0 on 10.0
HTTP Request
10.2:8989/click
chromedriver.exe
firefox-webdriver.dpiSelenium Grid
You need to get the drivers :(
?TheWire
Protocol
Binding(SDK) Driver
Protractor to the rescue
> npm install -g > webdriver-manager
> webdriver-manager update
> webdriver-manager start
Don't you lovedocker?
$ docker run -d -P --name selenium-hub selenium/hub:3.4.0-einsteinium
$ docker run -d --link selenium-hub:hub selenium/node-chrome:3.4.0-einsteinium$ docker run -d --link selenium-hub:hub selenium/node-firefox:3.4.0-einsteinium
Load Balance
Selenium Grid works well for only 25 nodes
https://github.com/seleniumkit/gridrouter
Selenium in the cloud
Browsers in the cloud
SaaS Open Source
Visual Validation
Can you spot the differences?
1 image is worth a 1000 assertions
Pixels Matters
BTW, CSS is Crosscutting
Use Machine Learning to find elements
MeetTestim.io
