1

TESTING 4.0THE NEXT GENERATION OF QUALITYPowered by Crowd & Cloud Technologies

2

TestbirdsOur History

Vision: “The World’s Leading Company for Innovative Testing Services”Founded in 2011Origins in the enterprise software testing market Customer satisfaction 9.2 out of 10Quality assurance and UX / usability testing utilising crowd and cloud technologiesTesters based in 193 countriesBroad spectrum of testers from normal users to professional QA testers

3

Testbirds Locations65 Employees in Eight Offices

HQ in Munich, operating in DE, AT & CH

40+ Employees from 10 nations

1,000+ Projects

400+ Clients:

Launched 2013

New Amsterdam office in Jan 2015

7 employees

Launched 2013 London, new

office in June 2015

5 employees

Launched Q2/15 Partnership 5+ employees Turin, IT

Launched offices:2013: Budapest, HU2015: Bratislava, SK2016: Prague, CZ

17 Employees Franchise

partners

Launched 2014

Stockholm office launch in Q3/16

Launched 2Q16 Franchise partnership 5+ employees Moscow, RU

4

Peter BruunBusiness Development Benelux

p.bruun@testbirds.com

Testbirds BeneluxKeizersgracht 1191015 CJ Amsterdamwww.testbirds.nl

Questions?

CROWDTESTINGDon’t ignore, but involve your end-user

5

Device DiversityDo You Have All of the Devices to Test On?

6

Device DiversityHow Do You Validate Localization?

7

Target Group RelevanceDo You Know Your Target Groups‘ Needs?

8

Current in the Market Insufficient QA, Poor Market Analysis and Time Constraints

Organisational blindnessSo is everything really fine? Unbiased users can test differently.

Device diversityDo we really have all of these devices to test with?

Target group relevanceTarget groups have different expectations and abilities.

Lack of resourcesTime pressure, limited budget and priority of other projects.

9

Case studies

10

Nederlandse SpoorwegenNS Spoordeelwinkel

Test type

BugAbility™

5 use cases

Tester profile

26 18-47v.6.x

v.2.x

Results

32 of which 3 critical

v.8

tabletssmartphones

11

Device DiversityHow Do You Validate Localization?

ANWBMobile website

12

CROWD150,000+ people in 193 countries380,000+ devices110,000+ found bugsAny target audience (60+ criteria / demographics)Real world testing without artificial environments Client specific feedback even prior to releaseFlexible and efficient ad-hoc availability

CLOUDAny combination of virtual machine (Win, Mac, Linux) and emulator (iOS, Android)Modern technologies exclusively created by TestbirdsHigh scalability and individual configurationFlexible, efficient and dynamic No maintenance or development costs Servers only hosted in Germany for maximum data security

Testing 4.0Powered by Crowd & Cloud Technologies

13

Our OfferingsOur Services Cover All Stages of Development

Quality Assurance

Usability & UX

Testbirds Exclusives

14

Nest Demo

15

Nest FeaturesCrowd Features

Incentives for testersSmart selection of testers through “slots & constraints” algorithmsLive tracking of the project status, 24/7Ability to test enterprise applications with Bring-Your-Own-Crowd™Confidentiality and security guaranteedExport functionality available to JIRA and othersCustomised “Form Builder”

16

Nest FeaturesCloud Features

Unique tool for creating virtual environmentsSaaS solution in the cloud – no maintenance cost Over 2,500,000 combinations of devices, operating systems and software including your own software/appAutomated and manual testing optionsApp distribution through BirdFlight™Developed, hosted and maintained by Testbirds in Germany

17

Automated TestingPowered by Jenkins, Selenium and TestChameleon™

Video link

18

Our Offerings

Quality Assurance Usability & UX Testbirds Exclusives

19

Quality AssuranceEnabled by Crowd and Cloud Technologies

Test Case Creation

Test Case Execution

Regression Testing

Bug Bounty

Localization Testing

Testing Environments TestChameleon™

Load & Performance Testing

Add-Ons

Exploratory Bugtesting

20

Usability & UXEnabled by Crowd

Usability/UX Study

Card Sorting

Remote Usability Videos

Comparison Study

Competitor Analysis

Pre-App Store Rating

Add-Ons

Prototype Evaluation

21

Testbirds ExclusivesEnabled by Crowd and Cloud Technologies

End-to-End Testing

Bring-Your-Own-Crowd™

On PremisesMyNest

App DistributionBirdFlight™

Training Academy

BugAbility™ Test

22

CrowdtestingHow It Works

Testbirds Project Manager

NestDevelop briefing 1

Selects & supports2

Creates final presentation including recommendations

7

Ensures quality

4

Test protocols &

bugs/defects

6

Customer with new software

Write test protocols and create bugs/defects

5

Testers with own devices

3

Live tracking of the project’s progress

23

Agenda

About UsTesting 4.0DemoServicesContact

24

The CrowdMeet Our Testers

Students Employees OthersBusiness OwnersManagers

62%38%Total150,000+UK18,000+

14%

24%

Consumers Professionals Specialists

62%

Devices:380,000+Mobiles Computers TVsCarsWearables …

Countries

193+Bugs110,000+

Filter Criteria

60

25

Service Levels

Managed ServiceManaged by a project manager Extensive final report (optional)

Service Level

Self-ServicePLUS

Managed Service

Self-Service

Self-ServicePLUS

Set up by our project managers matching your needsConducted by youSelf-ServiceSet up by youConducted by you

26

CloudSee the Value for Your Business

Choose between Manual testing Automated testing

All browsers and related software

All operating systems

Pre-install your own software

Certificate based authentication

Developed and hosted in Germany

Your mobile apps

Your testers

TestChameleonTM BirdFlightTM

27

Current in the Market Insufficient QA, Poor Market Analysis and Time Constraints

Organisational blindnessSo is everything really fine? Unbiased users can test differently.

Device diversityDo we really have all of these devices to test with?

Target group relevanceTarget groups have different expectations and abilities.

Lack of resourcesTime pressure, limited budget and priority of other projects.

Unbiased testers Socio-demographic selection

380,000+ devices in the crowd + access to virtual

devices in the cloudAd hoc availability

28

Why TestbirdsOur Benefits

In-house quality assurance of tester reports

Every deliverable is quality assured: defects, reports, …

Your key contact at Testbirds manages all QA processes

Internal QA

Local teams & test managementLocal offices and teams in DE, NL, UK, SE, HU, SK and US

On-site support during your projects

Our delivery teams speak your language

Full integration into your development processes

Ongoing in-house developmentHighly customisable (overall

process, report forms, defect forms…)

Any OS and device available through cloud and crowd

Compatible with your incident management tools

Platform

You buy a customized package – consume it only when you use it

Package can be used for any testing offered by Testbirds

Testing across all stages of development cycle

Commercial model – BirdCoins

60 profile fields to find your specific target group

In-house community team for recruiting the right people for you

More than 100,000 people having over 250,000 devices (consumers, experts etc.)

Testers

QualityFlexibility

29

PricingMaximum Flexibility to Suit Your Specific Testing Needs

Crowd & Cloud packagesStandard sizes & individual packages Duration: quarterly & yearly with renewalBirdCoins can be used for any crowd serviceRollover of BirdCoins into future periods

30

Peter BruunBusiness Development Benelux

p.bruun@testbirds.com

Testbirds BeneluxKeizersgracht 1191015 CJ Amsterdamwww.testbirds.nl

Questions?

CROWDTESTINGDon’t ignore, but involve your end-user

31

Analysts' ReportsTheme: Crowdtesting

Market Trends: Application Testing ServicesMust Address the Shift to Digital BusinessRequirements, Gartner, 2016More Information

Hype Cycle for Application Development, Gartner, 2015More Information

Market Overview: Mobile App Testing, Forrester, 2015More Information

Cool Vendors in Enterprise Mobility, Gartner, 2014More Information

Uncover Unexpected App Insights Through The Crowd, Forrester, 2016More Information

32

Exploratory BugtestingQuality Assurance

Exploratory Bugtesting ensures the functionality of your software across various devices with different software and browser set ups

Development, Testing and Release

Managed Service & Self-ServicePLUS & Self-Service

10 BC / Tester

33

Exploratory BugtestingCase Study

App store rating of an application is worse in the Google Play Store than in the App Store due to bad reviews about bugsWe start a test using a tester group of 25 Android users from our crowd, each operating Android 4.0 to 6.0 devicesAfter only a few days, the final test report yields 59 bugs, which all can be reproduced on our “Nest” with virtual machinesMost bugs could be easily fixed, so the app is successfully optimized and the rating in the Google Play Store improved

34

Test Case ExecutionQuality Assurance

Test Case Execution helps you ensure the functionality of your software’s core functions manually or automatically

Development, Testing and Release

Managed & Self-ServicePLUS (soon) & Self-Service (soon)

10 BC / Tester

35

Test Case ExecutionCase Study

Manuel Testing: Provider of a cloud- based storage platform brings a new product to the market• 24 testers perform detailed test cases on their own

devices• The results show that moving files from one folder to

another within the cloud is not possible on certain browsers

Automated Testing:A complex website integrates new features• They use TestChameleon, which performs

automated overnight tests on a range of devices by using test scripts

• In the morning the report shows which errors appeared on what kind of device types

36

Test Case CreationQuality Assurance

Your software is complex and requires an elaborate approach to testing it. We support you by creating test cases that can form a solid foundation for systematic and reproducible software testing. Development, Testing and Release

Managed & Self-ServicePLUS & Self-Service

10 Test Case Step Executions (TCSE) / BirdCoin (pending complexity)

37

Test Case CreationCase Study

Two developers from our crowd are writing approximately 50 test cases based on Selenium for biweekly releases for one of our biggest clients in the e-commerce sectorThe test cases are tested overnight in the form of automated test cases on 10 virtual machines running Mac, Windows and a variety of software via TestChameleonThe next morning, the client’s QA department receives the results and can work on the erroneous conditions and bugs that were found during the night

38

Regression TestingQuality Assurance

Regression Testing is an essential step to ensure through the repetition of test cases that modification has not caused any new errors

Testing and Release

Managed & Self-ServicePLUS & Self Service (soon)

8 BC / Tester

39

Regression TestingCase Study

Test subject was an responsive online shop Basics of a structured test:- 30 testers- 20 different device and operating system combinations- 30 test casesResult - 160 revealed bugs, of which 18 were criticalAfter fixing testers performed critical cases againProcedure was repeated twiceNo further critical bugs appearing

40

Bug BountyQuality Assurance

All software has its weaknesses. Some are already known while others are discovered by accident. The crowd is ready to be your bounty hunters

Release

Managed & Self-ServicePLUS & Self Service

Known: 10 BC / Approved ReportUnknown: Setup: 500 BC + 100 BC/month + tester payout

41

Bug BountyCase Study

Reappearing bug caused the company’s app to crash on Samsung devicesBusiness impact- the rating in the Google Play Store loweredHundreds of crowdtesters, each with a different Samsung device and a variety of operating systems, were tasked with reproducing the bug and documenting itThe problem was identified within 24 hoursFinally the bug could be fixed.

42

Localization TestingQuality Assurance

Localization Testing enables you to test your applications by natives located at the specific target markets under real conditions, thus ensuring correct and easily understandable, contextually accurate and graphically flawless translations.Development, Testing and Release

Managed & Self-ServicePLUS & Self-Service

10 BC / Tester

43

Localization TestingCase Study

Web and App

21 Countries / 2-4 testers per country

Web and app differ per country

Scripts differ per country

Managed

44

Testing Environments (TestChameleon™)Quality Assurance

When dealing with complex projects or test automation, it’s common to create a virtualised testing environment. Reduce configuration and maintenance costs with the help of our SaaS solution, TestChameleon.Development and Testing

See cloud packages

45

Testing Environments (TestChameleon™)Case Study

The first version of our SaaS solution, TestChameleon, was created due to the challenges of our client, Jumio. They were looking for a flexible tool to execute automated tests. The client, who specializes in identity management, has trusted in TestChameleon for over two years now to virtualize their testing environments.Jumio performs its Selenium test cases on a dozen virtual machines on a regular basis to ensure that with every new release their software consistently meets their high standards.

46

Load & Performance TestingQuality Assurance

Load & Performance Testing can add value in 2 ways:• Generation of the load• Backend analysis with qualitative feedback

from all the individual crowdtesters who participate in the test.

Testing and Release

Managed

Depending on estimation

47

Load & Performance TestingCase Study

TV MagazineWhat would happen if after broadcasting a TV commercial during prime time, thousands of users begin simultaneously searching for my green sweater with reindeer designs on my online store? Combination of cloud and crowd technologyUsing our external generation tool, the necessary load was created. At the same time, 200 testers on iOS and Android smartphones with differing network providers performed ten previously defined test cases on a 4G connectionAfterwards the testers answered qualitative and quantitative questions, which were then aggregated and evaluated by one of our project managers.

48

Prototype EvaluationUsability & UX

Prototype Evaluation allows you to gain end-user insights at a product’s inception, before the development phase begins.

Prototyping

Managed & Self-ServicePLUS & Self-Service

20 BirdCoins / Tester

49

Prototype EvaluationCase Study

A beverage company wanted to launch an anniversary app. The concept: users could enter raffles and upload photosTestbirds selected 30 target group specific testers to review the concept’s design and navigation featuresResults revealed potential improvements to colour contrast for easier readability, plus suggestions to improve the raffle participation processThis valuable end user feedback was incorporated into the design and re-tested with the same user groupThe result? A well-designed app, validated by end users at the concept phase, setting the company up for a successful launch and adoption

50

Card SortingUsability & UX

Card Sorting supports you by finding out what your target group considers a clear and simple navigation system

Concept

Managed Service

5 BirdCoins / Tester

51

Case Study

A network provider is planning on renovating their website and extending their servicesRequirement: new structure with an extensive menu and downstream design adaptions as well as user-friendly navigation50 testers sorted individual products into five, previously defined, distinct categories to organize the website, with the help from Testbirds supported online toolUsing these results, the network provider created a sitemap and menu structure which supported the user in finding their way around the website in a quick and intuitive manner

Card Sorting

52

Usability/UX StudyUsability & UX

Usability/UX Study supports you by gathering invaluable feedback on the usability of your application and to find out what’s stopping you from providing the best user experience possible! Prototyping, Development, Testing and Release

Managed, Self-ServicePLUS & Self-Service

20 BirdCoins / Tester

53

Case Study

A web only order service for flowers & gifts needs feedback on the user experience of their servicesTogether with a project manager, they define a target group Based on the defined target group with a main focus on DACH, 13 men and 12 women take part that are all over 25 years old and earn an annual salary of more than 38,000 EURTesters provide extensive qualitative feedback concerning what its usability is like and the delivery process in a test report that is then collected and evaluated by TestbirdsUsing these results, the flower service can optimize its website and investigate whether the cause for damaged parcels is internal or whether it lies with the external delivery service

Usability/UX Study

54

Remote Usability VideosUsability & UX

The Remote Usability Videos service gives you a detailed insight into your user’s thoughts by using recorded videos and the think-aloud principle

Concept, Prototyping, Development, Testing and Release

Managed & Self-ServicePLUS & Self-Service

8 or 40 BirdCoins / Tester (without or with evaluation)

55

Remote Usability VideosCase Study

A game developer buys a usability test using Remote Usability Videos for their latest strategy game.Testbirds creates a test with use cases and 20 testers from different European countriesBoth gaming professionals and gamers without previous knowledge participate in the testThe client receives uncut videos from the testers and an overview of the results

56

Comparison StudyUsability & UX

There are a variety of methods how content is being displayed on websites. A Comparison Study supports you to find out what your end users favour and most importantly, why!

Concept , Prototyping, Development, Testing and Release

Managed & Self-ServicePLUS (soon)

30 BirdCoins / Tester

57

Comparison StudyCase Study

One of our clients operating in the sports industry could not decide the main colour of their logo and digital presence during a redesign50 crowdtesters that matched their target group (mostly male, between 20 and 45 years old, sports enthusiasts and with an annual income of over 50,000 euros) were consultedA number of key messages in various designs were presented to the testersAfter evaluating the questionnaires the results clearly indicated the green design was not only preferred by the testers, but was also rated as better matching the company’s key messages

58

Competitor AnalysisUsability & UX

A Competitor Analysis helps you to find out where you have strengths and weaknesses in your digital product compared to your competitors’ by using feedback from your target group

Concept, Prototyping and Release

Managed

30 BirdCoins / Tester

59

Competitor AnalysisCase Study

One of our clients noticed that while family holidays and active tours were booked just as much by them as by their competition, city trips were not30 crowdtesters which matched the clients target group compared their website with their 3 biggest competitors, while not knowing who the client wasCentral characteristics such as navigation structure, filters and search functions were compared. In addition, special attention was given to the booking and payment system.The results showed that the target group requires more flexibility in compiling the trips and that the offered payment methods did not cover the customer’s wishes. Also, the competitor analysis provided valuable graphical insights for the upcoming relaunch.

60

Pre-App Store Rating Usability & UX

Pre-App Store rating allows you to evaluate your target group’s opinions, just like they would in the app store, without the risk of public exposure

Testing

Managed, Self-ServicePLUS & Self-Service

3 BirdCoins / Tester

61

Case Study

An agency creating a new fitness app was looking to ensure that it would be well received in the app store prior to launchThey opted for Testbirds’ Pre-App Store rating service and a crowd of 100 testers were given access to the app. The exact target group, which consisted of running enthusiasts between 18 to 40 years old, were selected with various devices. After using the app for two days they gave their feedback. The results were remarkable: an average rating of 4.6 out of 5 stars. The design and easy navigation were especially well received by the crowd. However, users also suggested changing to a bigger save icon for recording fitness achievements. After the test, nothing stood in the way of this fitness app’s success

Pre-App Store Rating

62

BugAbility™ TestTestbirds Exclusives

A BugAbility™ Test is an all in one service that helps you test the functionality of your application and get usability feedback with the help of our crowd

Development, Testing and Release

Managed & Self-ServicePLUS

15 BirdCoins / Tester

63

BugAbility™ TestCase Study

A Dutch company that offers services in the traffic and travel industry wants to test their mobile website for functionality and user-friendliness issuesTestbirds creates a BugAbility™ Test with 25 Dutch testers, who also need to rate a new navigational concept on the site 42 bugs are found, of which 15 are rated as critical or high severityThe company uses the feedback of the testers and manages to get a consumer rating of 9.2/10 during the launch

64

End-to-End TestingTestbirds Exclusives

The line between the real and digital world continues to grow blurry. Test the interactions between all the different components with the help of the crowd under real world conditions! Development, Testing and Release

Managed Service

> 30 BirdCoins / Tester

65

End-to-End TestingCase Study

One of the world’s largest electronic manufacturers wanted to test their baby monitor with camera function, connected to an iPhone appTarget group: 50 mothers and fathers with children under two years oldTesting phase: over four weeks Testers documented entire process in a user diaryExtensive questionnaire with open and closed questions

66

App Distribution (BirdFlight™) Testbirds Exclusives

Is your beta app not yet ready for the public, but you already want to share it with colleagues or potential clients and perform tests?

Development and Testing

Free

67

App Distribution (BirdFlight™)Case Study

Launched in 2014Supports all OSsOver 4,000 developers registered 14,000 downloads every single month150,000 crash logs protocolled

68

Training AcademyTestbirds Exclusives

Through our Training Academy, you gain access to a team of quality assurance and usability/UX experts that provide you with practical and applicable knowledge with an emphasis on testing using the Testbirds core competencies in crowd and cloud technologies.Concept, Prototyping, Development, Testing and Release

Dependent on each course

69

Training AcademyCase Study

3 Days3 PeopleIn-House (Munich)

MyNest• Testers View• Project Management View• Client View• System Management View

70

Bring-Your-Own-CrowdTM Testbirds Exclusives

Intranet, enterprise and consumer apps or expert portals – have your employees or customers be your testers!

Concept, Prototype, Development, Testing and Release

Managed Service & Self-ServicePLUS

Same Rate

71

Bring-Your-Own-CrowdTM Case Study

A client in the automotive industry wanted to improve the management and interconnectivity of international sales by using a new web portal, or an online platform and mobile app with more than 3,000 retailers worldwide, in 15 languages.To test the portal and its corresponding apps, dealers from all relevant countries were invited to participate in the test. The test took place in two cycles. First, it focused on user-friendliness with an extensive usability study where dealers were able to express their opinion on additional functions.After the feedback was evaluated and the platform was adapted accordingly, the sales partners began to hunt for functional issues by performing typical tasksThanks to this, when launched, the platform was successfully receiving with an internal rating of 9.3 out of 10

72

Private Secure CrowdAdd-Ons

Security is of utmost importance to Testbirds. All of our testers need to accept our extensive terms and conditions which oblige them to respect the confidential and sensitive nature of the projects they participate in. In addition, our clients have the option to let testers participate in projects that require explicit NDAs. Included on request

73

Private Secure CrowdCase Study

Various organizational measures to reduce the risk of confidentially and disclosure breachesTesters can sign an NDA directly between themselves and the customerTesters can be identified via online verification of IDs or mail verification (PostIDENT)Extended privacy features in the Nest to anonymize client data

74

VPN & Access ServicesAdd-Ons

Internal firewalls and special security precautions make it hard for you giving external testers access to your systems?

100 BirdCoins for initial setup

75

Bug ApprovalAdd-Ons

Our optional Bug Approval add-on service enables you to verify whether bugs found on one device are present on all participating devices

Development and Testing

Managed & Self-ServicePLUS & Self-Service

+30% (max 20 Bugs)

76

Re-TestAdd-Ons

As software testing is an iterative process, our optional re-test service enables you to verify whether previously identified bugs have been resolved

Development and Testing

Managed & Self-ServicePLUS & Self-Service

1 BirdCoin / Bug Re-Test