Continuous Quality Assurance using Selenium WebDriver

transcript

Quality Assurance Continuous

…using Selenium WebDriver

San Francisco Selenium Meetup Group - May 16, 2013

Fabrizio Branca Lead System Developer at

San Francisco, CA

Janine

that’s me

It’s a boy!

E-Commerce: Magento

CMS: TYPO3

Portals: ZF, FLOW,…

Mobile Searchperience: SOLR

120 people in 7 offices world-wide

High Performance

/Scale

Global Enterprise Projects

Continuous Integration

Deployment

Development

Delivery

Improvement

Quality Assurance Continuous

Everything sounds better with “continuous”

“Continuous” == “Doing stuff over and over again”

Builds Deployment Tests Infrastructure

Devbox Setup

== “Automation”

task size

does it manually

gets annoyed

writes script to automate

runs script

Geeks vs. Non-Geeks

Makes fun of geek’s complicated method

— Ovidiu Platon

We are not shipping your machine! if it works on your machine! I don’t care

Picture The BIG

Vagrant Boxes

Awesome stuff

Every single line potentially

can break production!

Bratwurst (Germans)

Donut (Americans)

Pizza (Italians)

Version Control

GIT, SVN,…

PHP lint

Version Control Code Reviews

“Gerrit”

Version Control Code Reviews Build

Installer

Settings

Triggered by commit, by time

or manually

Jenkins

minify js/css

.tar.gz

“virtual package” (revision)

Multiple Release Branches?*

Multiple Build Pipelines!

… *http://nvie.com/posts/a-successful-git-branching-model/

Installer

Settings

Installer

Settings

Static Code

Analysis

CodeSniffer

php-pmd

php-cpd

php-depend

Installer

Settings

Static Code

Analysis

Install on

“Integration”

First system where code from all

developers/teams meets each other

unstable system. Might be broken. Will be rebuild

without warning

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests

PHPUnit

Code coverage

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests

AOE PMD (for Magento)

broken links, HTTP headers, W3C validity,…

Test features that rely on other parts (e.g. framework, external

services,…) External components (SSL, Redis, Varnish,…)

Basically this is also PHPUnit, but

addressing other stuff

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Here we go:

Selenium!

Installer

Settings

Static Code

Analysis

Install on

“Staging”

Behat (Cucumber)

Same setup as production, but different machines

Client QA / approval

Installer

Settings

Static Code

Analysis

Install on

“Staging”

*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning

Stress

Jmeter

Spin up EC2 instances in the cloud to produce

traffic*

Again: multiple servers or autoscaling setup

Installer

Settings

Static Code

Analysis

Install on

“Staging”

Stress

Install on

“Production”

A/B Deployments on different

scales

Plan B: rollback

Zero Downtime

Varnish Array EC2

Frontend Array Backend Array EC2 EC2

Worker Array EC2 EC2

Deployment x

ELB Load Balancer

Route 53 DNS-Service

shop.angrybirds.com

Deployment

Varnish Array EC2

Deployment x

ELB Load Balancer

Varnish Array EC2

Deployment x+1 Release Y

ELB Load Balancer

Route 53 DNS-Service

shop.angrybirds.com

Deployment

Installer

Settings

Static Code

Analysis

Install on

“Staging”

Stress

Install on

“Production”

Deployment != Feature Release

Installer

Settings

Static Code

Analysis

Install on

“Staging”

Stress

Install on

“Production” Enable Features

Version

Control Code

Reviews Build Static Code

Analysis

Install on

“Integration”

Unit Tests Integration

Acceptance

Install on

“Staging”

Install on

“Production” Enable

Features

Pipeline Visualization

Build Static Code

Analysis Unit Tests

Install on “Integration”

Integration Tests

Acceptance Tests

for team dashboards

Acceptance Tests

test automation is a developer’s task

we need to provide them some good tools…

Selenium 1 vs. Selenium 2

What’s out there? Selenium for PHP

PHPUnit_Selenium (Sebastian Bergmann)

WebDriver-PHP (Chibimagic)

Php-webdriver-bindings (Lukasz Kolczynski)

php-webdriver

PHPUnit_Selenium (Giorgio Sironi)

php-webdriver Selenium 2

08/2011 Facebook/php-webdriver (Justin Bishop)

Element-34/php-webdriver (Adam Goucher)

instaclick/php-webdriver (Anthon Pang)

php-webdriver

“A very thin wrapper of WebDriver”

So what?

A thin layer is good! But not enough…

no test logic

no higher level components

dealing with Selenium specific tasks

not compatible to legacy test cases

No abstraction / structure

Architecture

HTTP AUT

No direct access (DB, classes,…)

REST* (over HTTP)

*http://code.google.com/p/selenium/wiki/JsonWireProtocol

Project test cases

PHPUnit

Project Components

Components Library

php-webdriver

reusable

project specific

Menta Selenium 2 Testing Framework for PHP

Who is “Menta”?

What is “Menta”?

Why “Menta”?

Features Menta The framework.

Not the cat…

Components Session Management

? Bonus Features :) Event / Observer Configuration Management

PHPUnit integration

✔ ✔ ✔

Data Providers

✔ ✔ ✔

Dependent Tests

✔ ✔ ✔

Test Infrastructure

Menta for PHPUnit Abstract test class

PHPUnit_Framework_TestCase Menta_PHPUnit_Testcase_Selenium2 ProjectY_Abstract_TestCase

Very thin layer on top of PHPUnit

ProjectY_TestCase_Registration ProjectY_TestCase_Checkout the “magic” happens in the components

Legacy Selenium 1 tests PHPUnit_Framework_TestCase PHPUnit_Extensions_SeleniumTestCase

ProjectY_TestCase_Registration ProjectY_TestCase_Checkout

Legacy Selenium 1 tests PHPUnit_Framework_TestCase Menta_PHPUnit_Testcase_Selenium2

ProjectY_TestCase_Registration ProjectY_TestCase_Checkout

Menta_PHPUnit_Testcase_Selenium1

Selenium1Facade

Compatible with PHPUnit_Extensions_SeleniumTestCase

HTML Report Screenshots (on demand

and on failure)

test description from phpdoc

additional messages

error details

HTML Report “Duration Heatmap”

shows long running tests

Menta for PHPUnit Text Result Printer

PHPUnit 3.7.20 by Sebastian Bergmann. Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml > SUITE: > SUITE: > SUITE: Acceptance_Tests_Account_Register > TEST: Acceptance_Tests_Account_Register::register SUCCESS. (Duration: 8.18 sec) > TEST: Acceptance_Tests_Account_Register::login SUCCESS. (Duration: 6.5 sec) > TEST: Acceptance_Tests_Account_Register::testRegistrationMail FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22 sec) < Duration: 36 sec < Duration: 36 sec > SUITE: > SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest > TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Failed asserting < Duration: 9.87 sec > SUITE: Acceptance_Tests_Checkout_CheckoutTest > TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView SUCCESS. (Duration: 1.79 sec) > TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::checkout" to pass. (Duration: 5.34 sec)

Components?

Components!

Manager Component

Menta_ComponentManager::get(‘PageObject_Cart’) PageObject_Cart

Rewrites Component

PageObject_Cart ProjectZ_PageObject_Cart

extends

change locators

add functionality

change behavior

overwrite labels

Manager Component

Menta_ComponentManager::get(‘PageObject_Cart’) ProjectZ_PageObject_Cart

Menta_ComponentManager::addRewrite(

‘PageObject_Cart’,

‘ProjectZ_PageObject_Cart’

Components

Common Assert Wait Screenshot Selenium1Facade

Imap WebService W3C Validator Curl …

Some might not even require a Selenium session

Gallery Screenshot

Components! PageObjects?

PageObjects Components

{ } }⊇{

PageObjects are

components

PageObjects?

represents services offered by a page or feature

is the only thing with a deep knowledge of

the HTML

clickCheckoutButton() isInCart($product) getTotals()

✔ ✔ ✔

Component Libraries

Magento

(your library)

Project-

specific

Magento

(community-provided)

General

(your library)

Session Management

Let Menta manage your sessions

auto-close

auto-start

incl. on exceptions and ctrl+c

Reusing sessions

Cleaning sessions

Configuration Management

Fallback mechanism for global settings

Create multiple configurations for different browsers,

application instances or Selenium Servers

Compatible to PHPUnit (xml)

Event / Observer Menta_Events::dispatchEvent('after_session_create');

Menta_Events::addObserver('after_session_create', function($session) {

$session->window('main')->postSize(array('width' => 1024, 'height' => 768));

Bonus Features

Sauce Labs Support

*configuration: http://username:accesskey@ondemand.saucelabs.com:80/wd/hub

works fine*

update test result (REST)

Sauce Labs Support

Speaking name Test results

Perceptual diffs Build X

Build X-1

Perceptual Diff

github.com / AOEmedia / Menta github.com / AOEmedia / Menta_SampleProject

It’s composer based and you’ can get started in a few minutes!

Thank you! Any questions?

http://www.aoemedia.com

http://www.fabrizio-branca.de

@fbrnc Follow me on twitter!

My blog

Continuous Quality Assurance using Selenium WebDriver

Technology