This presentation was performed at AST 2014 (tech.brookes.ac.uk/AST2014/) the 9th International Workshop on Automation of Software Test (AST’14) on June 1, 2014, in Hyderabad, India. It introduces CrowdTesting and provides some initial thoughts on how Crowd Testing and Software Engineering can be combined to get even better results.
Università degli Studi dell’Aquila Henry Muccini DISIM, University of L’Aquila [email protected], @muccinihenry, www.henrymuccini.com @AST 2014, Hyderabad, India – June 01, 2014
Transcript
Universit degli Studi dellAquila Henry Muccini DISIM,
University of LAquila [email protected], @muccinihenry,
www.henrymuccini.com @AST 2014, Hyderabad, India June 01, 2014
Researcher at the University of LAquila, Italy Research 2
How did I get to Crowd Testing? 3 @AST 2012
@MobileSoft2014
Acknowledgment and Disclaimer This work is conducted with the
collaboration of Fabio Guaiani (University of LAquila) and the
CLARITER company (Rome). This presentation represents my group
initial effort towards the understanding of Software Engineering
needs and opportunities in Crowd Testing. 4
Software Testing Evolution 7 Software Testing No dedicated
testing team Dedicated (in-house) testing team Outsourced testing
(in-house) resources resources in the cloud Software
development
Investment in QA 8 [World Quality Report 2013-2014] [1191
respondents] Right first time apps > $50 billion/year on
applications testing and quality assurance [Top six trends to drive
market for software testing in 2012, TechJournal, citing Pierre
Audoin Consultants, Nov 2011] 18% 23% 28% 0% 5% 10% 15% 20% 25% 30%
1 IT budget allocated to QA 2015 2013 2012
(the Crowd) [Oxford Dictionaries] The mass or multitude of
ordinary people
Crowdsourcing [CrowdSource] The word is a combination of the
words 'crowd' and 'outsourcing'. Work Work Work Work Work Work Work
Work Crowdsourcing is the process of getting work or funding,
usually online, from a crowd of people.
What is Crowd Testing? It is, therefore, a form of outsourcing
testing tasks to a mass of internet users (the crowd).
[Stein_TestExp14] If you are testing software that all kinds of
strangers are going to use, then why not use a bunch of strangers
to test it. (Paul Herzlich) Crowtesting puts software to test under
real-world conditions by real users, allowing companies to gather
real insight and feedback and to identify defects quickly and
cheaply [CSTE2013] 18 HM5 HM6
Diapositiva 18 HM5 Its called the power of the people. I asked
thousands of friends to spend 5 minutes testing rather than
requiring five of our employees to spend thousands of hours
testing. It got us quicker results on more devices than we
typically consider. [Mike Sparks article] Henry Muccini; 30/05/2014
HM6 It consists in asking 1000s of crowd testers to spend 5 minutes
testing rather than requiring 5 employees to spend 1000s of hours
testing. Henry Muccini; 30/05/2014
Examples 19 Tested by a crowd of 9 million people 2 million
valuable comments and insights [Stein_TestExp14] GAP had to scrap
their logo redesign after protests on Facebook and Twitter
[www.pass.ch]
20 diversity/coverage/realistic independent and test as a user
payment system immediate results Principles
21 Crowd Testing
The players and the process 22 Crowd Testers Crowd Testing
Company Customers
Who are the crowd testers? [RV_TestExp14] The testing crowd can
be a pool of: Inter-company testers Intra/inter-project testers in
a company Strategic partners Domain knowledge experts Technology
experts Tool experts Network of end users Social network users 23
Certified Specialist Recreational
Exploratory Crowd Testing + Effective to cover all the many
configurations + Good for non critical systems + Simulating real
load and testing - Outcome not predictable - Discovering basic
failures To help making it more effective: Show them the big
picture of what they are doing Formal process A clear report
25
Enterprise cloud infrastruct ure Enterprise Crowd Testing 26
Large enterprises utilizing their in-house excess capacity for
on-demand crowdsourced testing Secured private cloud infrastructure
Testing Managers Organized Crowd testers screening Project
management, tracking, effort planning Professionalized Standardized
Process and ReportingStandardized
Enterprise (Managed) Crowd Testing Enterprise crowd testing,
although still in its nascent and emerging stage, promises to be
the next wave in the social, mobility and localization era.
[28+_TestExp14] 27 Crowdtesting Platform Test Scheduling Testers
Profiles Knowle dge center Comm. and collaboration Test Managem ent
Reporting
A typical Process 28 Preparation Initiation Execution
Evaluation Completion Identify customers needs Fix price Define
report template Select crowdtesters Prepare the test environment
Initiate testing Supervise and monitor Validate results Prepare
client deliverables Testers payment Presents results to client
Crowdtesters ratings [TestExp14]
Crowd TestingGlossary 29 Crowd Crowd Sourcing Outsourcing Crowd
Testing Exploratory Crowd T. Managed Crowd T. Beta Testing
Enterprise Crowd T.
30
- 1.5 months of full-time job for one mobile developer novice +
0.5 months for an expert mobile developer (client side) - 13.000
products in the DB - 400 MB of storage into the device - DB update
31
- The annulla feature (when the button is visualized during the
data download) does not work on IOs - In case the product image is
missing, a default image needs to be loaded - The loading % screen
is not fluid - On IOs, if I download a product with an already
existing image, I get an error - In the product list, when tapping
on a product imagine, while looking at the magnified image, I can
still scroll the page (shall be disabled). 32
SOFTWARE ENGINEERING FOR CROWD TESTING 33 Software Testing
Trends Crowd Testing Software Engineering for Crowd Testing
Most of the reports come from CrowdTesting companies, so there
could be a BIAS!! 35
Still a number of big companies working on it [BenchmarkQA] 36
are 55.000 testers better than 1.600 ? What types of testing can be
run with the crowd? Big companies!
Data from Utest (now, Applause) 37 Randy Perry. Trulia
Leverages uTest's "Crowdsourced" Testing Solution to Speed
Deployment of High-Quality Mobile Apps. Sponsored by uTest. January
2013
Goldman Sachs Bets Millions On uTest
[http://www.forbes.com/sites/petercohan/2014/01/22/goldman-sachs-bets-millions-on-utest/]
38
Challenges in Crowd Testing 39 Security breaches Duplicate
defect management Measurement Coverage critical features coverage
No control Lack of ownership Reward mechanism Communication and
mgmt. Lack of standards Continuous involvement
Opportunities for SE research in crowd testing: COMMUNITIES
Crowdsourcing in SE Global Software Engineering Social Software
Engineering Empirical studies Software Testing 40
1. Opportunities for SE research Effectiveness of different
testing methods Whenshall we test with the crowd, when not to? 41
Crowd Testing Cloud Testing Lab Testing Which sw. domain Which type
of tests Ongoing Work
2. Opportunities for SE research Complementing different
testing methods 42 Crowd Testing Cloud Testing Lab Testing How
crowd and lab testing can be used together? Ongoing Work Need:
clients want to get the best from the combination of both
Efficiency
2. Complementing different testing methods (cont.) 43
Systematic Mobile Testing Crowd Testing Solution to ?
2. Complementing different testing methods (cont.) 44
Systematic Structural Testing Crowd Testing Solution to ? Coverage
Issues Test Selection Test Execution
2. Complementing different testing methods (cont.) 45 Model-
based Mobml Testing Crowd Testing Model- based Mobile
Principle 46 Device independent tests (in LAB) + device
dependent in crowd Test HTML 5 PhoneGap (in LAB) + test Phone Gap
app in Android, iOS, Microsoft Test MobML mobile (in LAB) + test
Phone Gap app in Android, iOS, Microsoft Impl. 1 Impl. With MBT 2.
Complementing different testing methods (cont.)
3. Opportunities for SE research Engineering the CrowdTesting
Platform 47 Crowdtesting Platform Test Scheduling Testers Profiles
Knowle dge center Comm. and collaboration Test Managem ent
Reporting
4. Opportunities for SE research Crowd Coverage Criteria 48
Measurement Coverage critical features coverage No control System
Coverage Crowd Coverage Metrics
5. Opportunities for SE research Social Software Engineering
How to keep Crowd Testers motivated How to keep them involved
Appropriate reward mechanisms 49
In summary 50 Crowd Testing is growing exponentially Crowd
Testing may bring big benefits to QA Still, there a number of
challenges to be managed
If you are interested to this topic, please 51 Stop by after
the presentation Contact me at [email protected] Tweet
@muccinihenry Skype me at henry.muccini Call me H. Muccini Is Crowd
Testing (relevant) for Software Engineers? @AST20114
References Testing Experience, 25/2014 issue on Crowd testing
[TestExp14] Testing Experience, 25/2014 issue on Crowd testing
[RV_TestExp14] Rama Murari & Venkat Ramesh Atigadda, Panorama
of Crowd Testing, in Testing Experience, 25/2014 [Stein_TestExp14]
Markus Steinhauser Crowd Testing An Addition to Traditional
Software Testing, in Testing Experience, 25/2014 [34+_TestExp14]
Dieter Speidel, Mayank Mittal & Mithun Sridharan Enterprise
Crowd Testing, in Testing Experience, 25/2014, page 34+
[28+_TestExp14] Saisubramanian Sivasailem & Dilipkarthik
Jayamohan, The Next Wave in the Social, Mobile and Localization Era
- Enterprise Crowd Testing, in Testing Experience, 25/2014, page
28+ 52
References [BenchmarkQA] http://www.benchmarkqa.com/ [CSTE2013]
Crowdsourced Software Testing for Enterprises, August 2013 (by
passbrains.com) [World Quality Report 2013-2014] By Capgemini,
Sogeti, Hp, 5th edition Crowdsourced Software Testing for
Enterprises, August 2013 (passbrain) Dagstuhl Seminar 13362,
Cloud-based Software Crowdsourcing, Edited by Michael N. Huhns, Wei
Li, and Wei-Tek Tsai 53