Testing the BBC News Website

Mark Micallef

The timeline…

Aug 09 Aug 10

FebNov MayFebNov May

The timeline…

Aug 09 Aug 10

FebNov MayFebNov May

Winter

Olympics

UK

Elections

FIFA

World Cup

Starting Point

Product

Manager Spec

Dev Team

Manager

Project

ManagerTester

Spec

“Use the project as a platform

for transitioning to Agile”

The timeline…

Aug 09 Aug 10

FebNov May

Project Setup Stabilization Delivery Mode Post-Launch

FebNov May

Winter

Olympics

UK

Elections

FIFA

World Cup

Risk mitigation

Aug 09 Aug 10

FebNov May

Rel A Rel B Rel C

FebNov May

Winter

Olympics

UK

Elections

FIFA

World Cup

On launch day…

• Developers in at 4:30am

• Very calm launch

• No real major public facing issuesissues

• Some internal issues (mostly integration issues with downstream systems)

• Ongoing public feedback

Let’s talk process

Remember this?

Spec

Scope

Project

Teams

CoreOutput -Indexes

Output -Stories

Site Building

6

8

10

12

14

16N

um

ber

of

teste

rs

Project Testers

0

2

4

6Nu

mb

er

of

teste

rs

Initial Process

• Agile

• Extensive automated testing

• TDD

• 1 week iterations

• Daily standups

• Weekly Showcases

IdentifiedIdentifiedIn

AnalysisIn

AnalysisReady for

DevReady for

DevIn DevIn Dev

Ready for QA

Ready for QA

In QAIn QAReady for Sign Off

Ready for Sign Off

Signed Off

Signed Off

Process for each user story

Initial Process

• Agile

• Extensive automated testing

• TDD

• 1 week iterations

• Daily standups

• Weekly Showcases

Initial Process

• Agile

• Extensive automated testing

• TDD

• 1 week iterations

• Daily standups

• Weekly Showcases

• Retrospectives

• Twice-weekly QA meetings

• Weekly Management Meeting

IdentifiedIdentifiedIn

AnalysisIn

AnalysisReady for

DevReady for

DevIn DevIn Dev

Ready for QA

Ready for QA

In QAIn QAReady for Sign Off

Ready for Sign Off

Signed Off

Signed Off

Process for each user story

Review Acceptance

Criteria

Review Acceptance

Criteria

Prepare Cucumber

Feature Files

Prepare Cucumber

Feature Files

Dev Machine Demo

Dev Machine Demo

Test / Bug Fix CycleTest / Bug Fix Cycle

Demo to Product Manager

Demo to Product Manager

Process Change Funnel

Test Automation

Technologies

Technologies

CucumberCucumber

Ruby ModelsRuby Models

Analysts

Automated Testing Technology Stack

Ruby ModelsRuby Models

WhiteWhiteSelenium

GridSelenium

GridW3C

ValidatorW3C

ValidatorDB

LibrariesDB

LibrariesXML

ParsersXML

ParsersJMeterJMeter

Engineers

@smoketest

@editions

Scenario: International user accessing the news website

Given I am a user outside the UK

Cucumber Example

Given I am a user outside the UK

When I visit the News website

Then I should be redirected to the International Edition

And I should see adverts on the page

Given /^I am a user outside the UK$/ do

pending

end

When /^I visit the news website$/ do

pending

end

Then /^I should be redirected to the international

edition$/ doedition$/ do

pending

end

Then /^I should see adverts on the page$/ do

pending

end

Given /^I am a user outside the UK$/ do

@helpers.set_proxy(INTERNATIONAL)

end

When /^I visit the news website$/ do

@browser.open(“/”)

end

Then /^I should be redirected to the international

edition$/ doedition$/ do

assert_equal(true, @page.is_international?)

end

Then /^I should see adverts on the page$/ do

assert_equal(true, @page.ads_present?)

end

Test Environments

Test (Manual Exploratory Tests)(Manual Exploratory Tests)

Int(Dev & Unit Testing)

Stage(NFR + Config Testing)

Test XE(Automated Acceptance Tests)

Test Data Management

Create Test

Data

Execute

Automated

TestsRepublish

Test XEQA

Workbench

Test Data

Copied

Over

Communication and Measurement

Communication Tools

• Daily standups

• Weekly Showcases

• Retrospectives

• Walls and Boards• Walls and Boards

• Seating arrangements

Bug Management

Bug Management

Signoff Velocity

David McCandless on Visualisation (TED)

Thank you…