QUALITY ENGINEERING CHALLENGES IN E-COMMERCE

Yotam Sharan,

Quality Engineering ManagerMay 2013

Ver. 1.3 for publish

WHAT CHALLENGE?

WHAT’S THE BIG DEAL?

WHAT CAN YOU TELL US?

QA documentation

Run sanity tests

Full-regression time

COTS test tools

Test Automation

Mocks and tools

No

Developers

Max 1 day

No

X-org F.W.

QE does

Quality Engineering Challenges 4

Enable merchants to succeed in new commerce env’t

Enable consumers to buy anything, anywhere, anyhow 110M

ACTIVE USERS

25M SELLERS

$175B COMMERCE VOLUME

EBAY MARKETPLACES

Quality Engineering Challenges 5

DATA CHALLENGE

• Big Data

• Various languages

• Numerous processes

• Subsystems dependencies

• QA env. is partial & detached from live data

• Daily ETL / deployments

• Manual is too long

Quality Engineering Challenges 6

AUTOMATED GOLDEN SET

• Copy Production data once, as baseline

• Use/reuse static dataset & manipulate variables (e.g. ID, timestamps)

• Sample specific data, evaluate large sets/volumes

• Automate predefined data input – check against expected output

• Mocking

• Monitors

• Reporting: test results, logs, asserts, notifications

• Smoke test and monitor Production

Tools we use: Selenium/WD, SoapUI, Jbehave, Jmeter

• Exploratory testing

• Regression testing

• Compatibility testing

• Automation testing

• Scenario testing

• Unit testing

• Code review

• Paired testing

• Test-driven development

• Bug bashes

• Capacity planning

• Load and Performance testing

• Accessibility testing

• Mocks and stubs

• Analytic testing tools (FindBugs, HTTPWatch, CAL, FireBug, Sonar)

TEST INFECTION • Integration testing

• E2E testing

• Smoke testing

• PRD review

• Test case review

• Dogfood testing

• Sprint demo

• Security testing

• Usability testing

• Definition of Done

• CI monitors

• Production monitors

• Live Site Testing

• Crowdsourcing

Automation testing

Automate everything, regression first. Plan the right automation approach.

Business Value

Providing thin vertical slices of functionality with business value, every iteration.

Continuous Integration

Constant integration testing to assure build integrity and quality, every drop.

Design for Testability

Ensure that it is easier to develop and apply tests for the designed product.

Early Start

The whole-team approach engages test engineers at early design phases.

AGILE DEV & TESTING PRINCIPLES

Everyone needs to let go of titles and self-image and engage in a constant

feedback, open, transparent, punishment-free work environment.

Developer test and Testers develop

Analysts, programmers and PM write test scenarios

All engineers help define requirements

Quality is everyone’s responsibility

QUALITY DRIVEN APPROACH

Quality Engineering Challenges 10

Sequential Execution

Test execution: 7000 minutes

WEBDRIVER GRID

Parallel Execution

Test execution: 60 minutes

150 VMs/Browsers on WD Grid

Quality Engineering Challenges 11

WEBDRIVER GRID

Selenium Grid Hub

Log

Analyzer:

test logs,

stack trace,

server logs,

configurat’n

WD

Listeners

Quality Engineering Challenges 12

INTEGRATED QE PROCESSES

• Automation Framework for component reuse • Generic test cases for data-driven testing• Fully-integrated env. for anyone to run tests easily

“In God we trust; all others must bring data” – W. Edwards Deming

Gut feelings…

Engineering clockspeed

Adopting/Adjusting best practices

Instant feedback: CI, Pair-x, RT dashboards, Quality indicators (bugs, code QI)

Constant feedback: Stories, Planning, Test results, Reviews, Retrospective, Surveys

FEEDBACK & FINE-TUNE

CONTINUOUS INTEGRATION & CODE QUALITY

Quality Engineering Challenges 15

I’d love to hear from you www.ebay-m.com/yotam

Win the draw!

Work @eBay:

jobs-il@ebay.com

Questions?