WRITINGWELL-ABSTRACTED SELF-TESTING AUTOMATIONON TOP OF JELL-O®

DAN CUELLARLEAD SOFTWARE ENGINEERZOOSK - QA TEAM

TERMINOLOGY

Test or Automation Scripta series of steps and expected output

Hook or Elementa means of identifying a piece of UI (think the By class in WebDriver)

Controllera program which executes basic commands (think WebDriver or SeleniumRC Server)

PHILOSOPHY• Selenium is a controller program, it exists to perform actions on

a platform (web browsers)

• Click a link• Read some text

• Libraries exist to coordinate these small actions into larger logical actions on a particular platform

• Sign In• Send A Message• Get All Error Messages

• Interfaces exist to remove the platform from the context of libraries

• Automation scripts should coordinate the highest level of abstraction whenever possible to maintain agility and robustness

A BIT ABOUT ZOOSK• Zoosk is a romantic social network with tens of millions of users that

operates in 25 languages and in over 70 countries.

• Products include:

• web site• mobile web site• Facebook app• iPhone app• iPad app• Android app• Adobe Air app

• The development pace is rapid, features are churned out in around 2 weeks time (including testing)

• Product management is always running experiments and adjusting plans based on data from instrumentation

• Developer to quality assurance ratio hovers somewhere around 10:1

SO WHAT DO YOU MEAN BY JELL-O®

Jell-O [jel-oh]

Site wide A/B tests where ½ of users will see a complete site redesign and the other ½ will use the site as it was before.

Facebook app users can also split between different A and B designs at times.

Each A & B may have its own JS, CSS, and business logic to go along with it.

Not to mention that different versions of the API were being used by the web, Facebook app, and mobile apps.

All the while, different features were being enabled and disabled for different subsets of the A/B test groups

A B

WHAT CAN GO WRONG?• General

• Not enough information is retrieved to reproduce the failure• Elements

• The type can change (e.g. dropdown becomes a checkbox)• An element can change location• An element can have multiple ways of identifying itself

depending on an A/B test• Business Logic

• Logic can change for one platform• Logic can change for some users and not others (A/B test)• Logic can change for all platforms

HOW DO WE COMBAT THESE PROBLEMS?• General

• Not enough information is retrieved to reproduce the failure

• Separate test actions from controller code, add abundant logging, and unify the patterns by which common operations are accomplished (e.g. selecting from dropdown) Don’t access selenium directly in any scripts. Create a wrapper that logs everything and performs basic checks. (e.g. is the choice available on the dropdown)

• If libraries are used then you can log just about everything possible every time without having to re-paste all the logging code

HOW DO WE COMBAT THESE PROBLEMS? (CONT.)• Elements

• The type can change• Abstracting away all uses of the item into a library

• An element can change location• Centralize all hooks into the UI, you can change it once

and it will be fixed for all• Using reflection, you can verify that all of your hooks are

functional

• An element can have multiple ways of identifying itself depending on an A/B test

• ID’s are great, but you can also use an XPath Or expression in your centralized hook

HOW DO WE COMBAT THESE PROBLEMS? (CONT.)• Business Logic

• Logic can change for one platform• Update the library for that platform

• Logic can change for some users and not others (A/B test)• Detect the A/B test in the library

• Logic can change for all platforms• Generic Libraries or test scripts which coordinate platform

specific libraries are a great place to address this

HOW DO YOU DEAL WITH THAT AGAIN?• Separate logical actions From calls to controller (in this case

Selenium) code into libraries

• Centralize your site model and logic, so that all automation runs through one place. When you have to fix something, you can fix it once for everything.

• Write code that can test the test automation. Investigating test script failures can be costly, but if you test code can test itself you will be able to pick out certain kinds of problems easily.

• Write platform-agnostic test scripts when possible. This leaves you with one copy of site-logic and test-logic for all platforms. When things change, you’ll be happy its in one place.

SEPARATION OF CONTROL FROM SCRIPTING• An extra layer is great to log information and perform

additional actions that Selenium can’t assume it needs to do

• We use a layer between that wraps logging, error handling, and some other things on top of most WebDriver commands

Test Script •IsSignedIn()

Action Library

•Log(“Retrieving Cookie Auth”);•var authCookie = GetCookie(auth);•Log(authCookie);•return authCookie != null;

WebDriver Wrapper

•Log (“Getting Cookies”);•var c = Driver.Manage.().Cookies;•Log(c);•var a = Driver.Manage().Cookies[“auth”];•Log(“Cookie value: “ + a.ToString());return a;

CENTRALIZING YOUR SITE MODEL FOR SELF -TESTING• All hooks (means by which we retrieve WebElements) are stored in

a single class, namespace, or something similar

• Elements are stored lazily; the means to retrieve them are stored, but the actual WebElement is not retrieved until run-time. We use classes that wrap calls to By or methods which will return a WebElements

• Hooks are grouped relevantly and each group has private self-test setup methods which will navigate the client (e.g. selenium) to where the hooks exist

• Reflection is used to iterate through all of these groups and run the private self-test setup methods and then validate the functionality of the elements

• Annotations (Java) or Attributes (C#) are used to exclude elements from the tests

ABOUT INTROSPECTION& REFLECTION

Introspection (Java) and Reflection (C#) is the process by which a computer program can observe, do type introspection, and modify its own structure and behavior at runtime.

In the example to follow we store information about how to use various hooks (references to pieces of user interface) in the code so that later at runtime we can use the information to determine how to test our library of hooks.

CENTRALIZED HOOKS

[C#]namespace Automation.Hooks.Web{ public static class LoginPage { public static WebHook EmailField = By.Id(“email”); public static WebHook PasswordField = By.Id(“password”); public static WebHook SubmitButton = By.Id(“submit”);

[OmitFromSelfTest] public static WebHook Spinner = By.Xpath(“//div[@class=’wait’]”);

private static void _StartSelfTest() { _SelfTestSeleniumInterface.Open(“http://www.zoosk.com”) } }}

SIMPLE SELF-TEST[PseudoCode]

// iterate in depth-first-search orderforeach(class c in Automation.Hooks.Web){ if c.hasMethod(“StartSelfTest”) c.executeMethod(“StartSelfTest”);

foreach(WebHook h in c) Log.Verify(IsElementPresent(h));

if c.hasMethod(“StopSelfTest”) c.executeMethod(“StopSelfTest”);

}

MORE ADVANCED TECHNIQUES• Subclass WebHook to things like TextBoxHook and have the

test check for reading and writing from the text field using Selenium

• Do the same for Links, Images, etc. and write custom test methods

• Randomize the order of which elements are tested to check for robustness

• Add random data to the tests, e.g. type a random string into the text box.

• Write custom test methods for structures that require it

CONSOLIDATING AND ABSTRACTING PRODUCT LEVEL ACTIONS• Product Level Actions (e.g. Sign In To The Site) are stored in

libraries, so that there exists one good copy that everyone can use

• The libraries implement generic interfaces so the platform can be abstracted away (i.e. the native iPhone app and the website both have libraries with a signup function that conforms to an interface that can be used in any test script)

• If the process changes for a particular platform, you can update it in a single place

• If the process change across all platforms, you can use platform agnostic (generic) libraries and update the logic in a single place

ABSTRACT LIBRARIES

[C#]namespace Automation.Libraries{ public SeleniumActionLibrary WebSignUpLib : ISignUpLib { public void SignIn(string email, string password) { if (IsSignedIn())

return;

if (!Driver.Url.contains(“login.php”); Driver.Navigate().GoToUrl(“https://www.zoosk.com/login.php”);

Hooks.Web.LoginPage.EmailField.type(email);

Hooks.Web.LoginPage.PasswordField.type(password);

Hooks.Web.LoginPage.SubmitButton.click();

WaitForElementToNotExist(Hooks.Web.LoginPage.Spinner); } }}

LIBRARY INTERFACE

[C#]

namespace Automation.Libraries{ interface ISignUpLib { void SignIn(string email, string password); bool IsSignedIn(string email, string password); void SignOut(); }}

PLATFORM AGNOSTIC AUTOMATION SCRIPTS• Many tests that need to be run are logically identical

across all platforms.

• Deactivate an account, attempt to sign in, expect failure• Send a message, verify it shows up in sent items• Take down some web service, verify an error is produced

• Implemented these tests at an abstract level allows you to have one test for all platforms

• When the logic changes, the logic only exists one place in your automation code

AN ABSTRACTED TEST

[C#]namespace Automation.Tests

{ public static class SignInWithDeactivatedAccountTest: Test

{ public static void Run()

{

AutomationClient [] clients = new AutomationClient[] {

new WebClient(), new iPhoneClient() }

var account = APIUtility.Accounts.CreateTestAccount();

APIUtility.Accounts.Deactivate(account);

foreach(AutomationClient client in clients)

{

client.SignUpInterface.SignIn(account.email, acount.password);

Log.VerifyFalse(!client.IsSignedIn());

} } } }

CONCLUSION

Developers and Designers frequently have to be able to adapt to rapidly changing code, automation writers can be just as nimble.

If you design and abstract your code well, you can build smaller, more robust and agile automation scripts.

If anyone is interested in taking up work on bringing iOS support to Selenium… Call Me.

MORE INFO

Dan CuellarLead Software Engineer – QA Teamdanc@zoosk.com

(Email me to retrieve related source code available for sharing)

See Also

Self-Generating Test Artifacts for Selenium WebDriverTue 12:20PM – Track B – Marcus Merrell

Building Quality on Foundations of MudTue 2:40PM – Track A – Kristian Rosenwold

The Restless Are Going NativeWed 2:20PM – Track A – Dante Briones