Crowd Testing Mike Bartley - CEO

Test and Verification Solutions

Crowd Testing

Mike Bartley - CEO

Delivering Tailored Solutions for

Hardware Verification and Software Testing

Copyright TVS Limited | Private & Confidential | Page 2

TVS - Global Leaders in Test and Verification

India - 2011

UK - 2008

Germany - 2011

France - 2012

Singapore - 2014

China

South Korea

Continuous

geographical

expansion…

USA - 2014

0

50

100

150

200

Q3-13 Q4-13 Q1-14 Q2-14 Q3-14 Q4-14 Q1-15 Q2-15 Q3-15 Q4-15 Q1-16 Q2-16(Est.)

Q3-16(Est.)

Q4-16(Est.)

EM

PL

OY

EE

S

CALENDAR YEAR

Number of Employees by quarter


The rise of the crowd

Users rate a service or products • Airbnb, Amazon, travel websites, ….

Using the Crowd as an Innovation Partner • Apple use large numbers of users and developers around the

world to create apps and podcasts.

Crowdfunding • In 1997 American fans of Rock band Marillion used the internet

to raise $60,000 so they could play in the US.

• In 2012, there were over 500 crowdfunding platforms

• Global crowdfunding market to reach $34.4B in 2015 (Massolution 2014 prediction

The crown as testers • uTest incorporated in August 2007 with $2.3M in seed funding


99 Tests

Partnership deal with T&VS

Over 15K testers

Over 50 countries

Over 200 clients worldwide

Over 85K bugs found


Crowd Testing Basics

“No matter who you are, most of the smartest people work for someone else”

Joy’s Law – by Sun Microsystems co-founder Bill Joy

“Given enough eyeballs, all bugs are shallow”

Linus Law – by Eric S Raymond in “The Cathedral and the Bazaar”


Crowd Testing Basics

Testing by anonymous ‘crowd’ of testers

Testers are rewarded by identifying bugs quickly, first to find the bug gets paid.

Testing can be limited by device, or open to any platform

Testers are keen to work out of hours 24/7

Limit testing by geographical region, or global

Pay for the bugs only


Crowd Testing Advantages - Speed

Test quickly and conveniently, 24/7 if required

Drive down testing times

Ramp up and down test teams quickly

Have test teams the right size for each stage of development


Crowd Testing Advantages - Costs

Reduce your test costs

Improve Cash Flow

Ramp up and down test teams quickly

Have test teams the right size for each stage of development

No need to invest in different platforms and devices


Crowd Testing Advantages - Quality

Improves Test platform coverage

Testing by real users in the real world

Testing in multiple languages and cultures

Pay Testers for Testing, Developers for Developing

Standard bug reporting

Remove internal bias


Crowd Testing Disadvantages

Mainly for customer facing applications

Limited control of staff experience

Not for Unit or Integration testing

May need manuals to be ready earlier

May need support teams in place

Control of Piracy

Management of the Services


Crowd Testing Hidden Costs

Management of the Crowd Test Process

Control of builds

Regression testing

Earlier ‘deployment’ Support teams

Better documentation may be required


Some questions to consider

Do you need Unit Testing?

Do you need Integration Testing?

Do you need a Strategic Test Plan?

How will you manage retesting?

How bug-free does it have to be?

What will you do with Test Results?


Test Results

Testing Approach • To test the web application, exploratory testing approach was used.

• Testers from various locations accessed the web application and tested the whole application using different exploratory approaches.

To ensure coverage of the major features • We shared the critical workflows of the website


Test Results


Test Results


Test Results


Enterprise v Small companies

• Flex testing teams without

having to employee

additional staff

• Reach out to a broader

cross section of users

• Reduce the testing cycle

• Test Apps and Web Products on

every conceivable device in

days

• No need to divert development

teams to carry out User Testing

• Add independence to the

testing cycle

• Control testing costs

Enterprise Small/Startups


Major Companies are using it

Crowdtesting has already been used by likes of eBay, Amazon, GE, Microsoft, Google and Facebook to develop user-centric products.

Google regularly deploys crowd testing for 14 of their major product lines.


Typical questions to answer

Combination of devices i.e. OS & Browser versions, mobile platforms, Tablets to be tested.

Testing timeframe, number of days per cycle or total duration for testing being allocated.

Number of crowd sourced testers would you like testing your site, egg 25 – 50, 50 – 100

Estimate the number of testing hours you would like to achieve in the days allocated

Estimated number of defects you believe will be found as a result of testing

List bugs that you are already aware of

19


Typical questions to answer

Do you require a Test Plan to be written?

Will you provide Test Cases to be executed?

Would you like the crowd to provide you with: • Fully Detailed Steps - screen shots of each tests performed by Crowd (Screenshots / Video

capture of tests)

• Screen Capture - attached to defects only

Select specialist testers with a specific product / domain skills and expertise?

Confidentiality agreement / NDA with the Crowd testers required ?

Do you require Usability testing of your site? • Formal – Crowd actively is testing for usability and raising defects

• Basic – a Questionnaire is filled in at the end of the cycle to gauge testers feeling about usability Defects will not be raised.

20


Case Study of a Large MNC: Intuit

Intuit provides accountancy packages for small

companies, mainly its flagship application

Quickbooks

Intuit has lots of Small Teams building software

products. Each team is given limited resources. i.e.

their critical resource for them is time.

So, each team is tasked with creating innovative

software product that can potentially reach millions

of customers.

Crowd Testing frees up time for these teams. Thus

running a Crowd Testing campaign gives them

ability to compress their release cycles & also

improve the quality of their application.

21


Case Study of a medium MNC: Teligenz

Teligenz is a successful technology company that provides a range of creative services from web designing to App development.

The successful delivery of their projects depends greatly on the quality of the insights gained through App testing.

We leveraged on our talent pool of professional crowd testers from across the globe to implement effective functional testing for their Apps. The test cycles created were executed by professional crowd testers who were assigned to various projects after matching their skill set with the project requirements.

The efficient strategy enabled the entire test cycle for an App to be covered in just 3-5 days. The quick turnaround time helped the development team at Teligenz to make real-time decisions and focus product development in areas that needed their attention. They were able to improve their App quality exponentially.

22


Case Study of a small MNC: Reverie

Reverie’s cloud-based language-as-a-service platform enables businesses to connect more easily with their target audience by delivering dynamic and static content for apps in real-time in the languages that they need.

Testing the usability for language feature is not an easy task, especially when dealing with highly user-centric E-commerce Apps.

Reverie wanted to test the usability of the App for specific target demographics comprising of Tier 2 and Tier 3 Indian cities whose native language was Hindi.

We provided Reverie with testers across different demographics factoring in parameters like age, city etc. from all over India to gain a real life, end user perspective.

The focus on the human approach, helped developers gain valuable insights on how their target demographics would react to the App.

23


Case studies in progress

MovieR – a UK startup providing a social media multi-user video experience. iOS application being tested by 50 users over a 48 hour window

Pogo – a UK startup providing a photo comparison monitor for social networking. iOS application being tested by 50 users over a 48 hour window

24

Test and Verification Solutions

Questions?

Delivering Tailored Solutions for

Hardware Verification and Software Testing

Date post:	09-Dec-2016
Category:	Documents
Upload:	doanh
View:	218 times
Download:	1 times

Crowd Testing Mike Bartley - CEO

Documents