CrowdREquire: A Requirements Engineering Crowdsourcing Platform Adedamola Adepetu, Khaja Altaf Ahmed, Yousif Al Abd, Aaesha Al Zaabi and Davor Svetinovic Masdar Institute of Science and Technology Abu Dhabi, United Arab Emirates Email: {aadepetu,kahmed,yalabd,asalzaabi,dsvetinovic}@masdar.ac.ae Abstract This paper describes CrowdREquire, a platform that supports requirements engineering using the crowd- sourcing concept. The power of the crowd is in the diversity of talents and expertise available within the crowd and CrowdREquire specifies how requirements engineering can harness skills available in the crowd. In developing CrowdREquire, this paper designs a crowd- sourcing business model and market strategy for crowd- sourcing requirements engineering irrespective of the professions and areas of expertise of the crowd in- volved. This is also a specific application of crowd- sourcing which establishes the general applicability and efficacy of crowdsourcing. The results obtained could be used as a reference for other crowdsourcing systems as well. 1 Introduction This paper is aimed at developing the requirements for a conceptualized requirements engineering platform based on a crowdsourced workforce. Requirements engineering in- volves different stages (elicitation, analysis, specification and quality assurance (van Lamsweerde 2010)) which if properly organized, might be crowdsourced. The main idea of CrowdREquire is to develop requirements for projects submitted by individuals, corporations or any other external bodies. Also CrowdREquire would be designed to be web- based in order to maximize the advantages of crowdsourcing on the Internet. Labor markets such as InnoCentive (Innocentive.com ), iStockPhoto (iStockPhoto.com ) and prospectively, Crow- dREquire, require specializations and specific talents. Crow- dREquire is similar to Amazon Mechanical Turk but a major difference is that the tasks on Turk are relatively simple and can be performed by anyone while the process of developing requirements, as is the aim of CrowdREquire, is more com- plex. This explains the reason why Amazon Turk has been described as “crowdsourcing for the masses” (Howe 2006). CrowdREquire uses a contest model (Vukovic and Bartolini 2010) in which the final solution to a task is selected based on competition among members of the crowd. This is closely related to InnoCentive which receives requests from external Copyright c 2012, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. corporations, to provide innovative solutions to problems. It is important to create an atmosphere of healthy competition among the users as this has the potential of producing ex- cellent results from the crowd. Cash rewards act as a very important incentive for the crowd as does the sense of pres- tige among peers, i.e., the CrowdREquire crowd in this case. It is also important to note that the success of crowd- sourcing has been aided by the Internet age as these com- panies have been structured to work with a closely net- worked crowd (Howe 2006). Communication is way easier and faster than pre-Internet times, bringing the crowd to- gether in a global village. Hence, Web 2.0 has been a funda- mental factor responsible the success of the crowdsourcing revolution. 2 Background and Approach There are four major areas that need to be considered while developing CrowdREquire and these are the concept of crowdsourcing, requirements specification, business model and market strategy. Distributing simple tasks to individu- als is not necessarily a daunting process. However, assign- ing complex tasks require a high level of organization and enhancement especially with the challenge of dividing prob- lems, assigning the sub-problems to individuals and combin- ing the solutions (Zhang, Horvitz, and Miller 2011). The im- portance of the efficient division of tasks into subtasks, i.e., smaller tasks, getting sub-solutions to these subtasks and systematically combining the sub-solutions to give a com- prehensive solution to the original problem can be seen in (Zhang, Horvitz, and Miller 2011). One of the motivations of crowdsourcing is that crowdsourcing cuts out a signifi- cant percentage of capital required to solve a problem us- ing outsourcing. Crowdsourcing is not entirely free but it is cheaper than outsourcing or having conventional employees (Howe 2006). This is highlighted as one of the major rea- sons for the success of crowdsourcing. Questions arise over the accuracy, quality and authenticity of results of crowd- sourcing. Wikipedia is an example of a crowdsourcing plat- form and it was shown (Goodin 2005) that it is accurate de- spite the fact that it is an encyclopedia developed by mil- lions of users, i.e., a crowd. On the issue of disagreements on accuracy of Wikipedia articles, Wales (the founder of Wikipedia) had to settle such cases himself when Wikipedia was still a fledgling platform. The system is however now AAAI Technical Report SS-12-06 Wisdom of the Crowd 2
Transcript
CrowdREquire: A Requirements Engineering Crowdsourcing Platform
Adedamola Adepetu, Khaja Altaf Ahmed, Yousif Al Abd,Aaesha Al Zaabi and Davor Svetinovic
Masdar Institute of Science and TechnologyAbu Dhabi, United Arab Emirates
This paper describes CrowdREquire, a platform thatsupports requirements engineering using the crowd-sourcing concept. The power of the crowd is in thediversity of talents and expertise available within thecrowd and CrowdREquire specifies how requirementsengineering can harness skills available in the crowd. Indeveloping CrowdREquire, this paper designs a crowd-sourcing business model and market strategy for crowd-sourcing requirements engineering irrespective of theprofessions and areas of expertise of the crowd in-volved. This is also a specific application of crowd-sourcing which establishes the general applicability andefficacy of crowdsourcing. The results obtained couldbe used as a reference for other crowdsourcing systemsas well.
1 IntroductionThis paper is aimed at developing the requirements for aconceptualized requirements engineering platform based ona crowdsourced workforce. Requirements engineering in-volves different stages (elicitation, analysis, specificationand quality assurance (van Lamsweerde 2010)) which ifproperly organized, might be crowdsourced. The main ideaof CrowdREquire is to develop requirements for projectssubmitted by individuals, corporations or any other externalbodies. Also CrowdREquire would be designed to be web-based in order to maximize the advantages of crowdsourcingon the Internet.
Labor markets such as InnoCentive (Innocentive.com ),iStockPhoto (iStockPhoto.com ) and prospectively, Crow-dREquire, require specializations and specific talents. Crow-dREquire is similar to Amazon Mechanical Turk but a majordifference is that the tasks on Turk are relatively simple andcan be performed by anyone while the process of developingrequirements, as is the aim of CrowdREquire, is more com-plex. This explains the reason why Amazon Turk has beendescribed as “crowdsourcing for the masses” (Howe 2006).CrowdREquire uses a contest model (Vukovic and Bartolini2010) in which the final solution to a task is selected basedon competition among members of the crowd. This is closelyrelated to InnoCentive which receives requests from external
corporations, to provide innovative solutions to problems. Itis important to create an atmosphere of healthy competitionamong the users as this has the potential of producing ex-cellent results from the crowd. Cash rewards act as a veryimportant incentive for the crowd as does the sense of pres-tige among peers, i.e., the CrowdREquire crowd in this case.
It is also important to note that the success of crowd-sourcing has been aided by the Internet age as these com-panies have been structured to work with a closely net-worked crowd (Howe 2006). Communication is way easierand faster than pre-Internet times, bringing the crowd to-gether in a global village. Hence, Web 2.0 has been a funda-mental factor responsible the success of the crowdsourcingrevolution.
2 Background and ApproachThere are four major areas that need to be considered whiledeveloping CrowdREquire and these are the concept ofcrowdsourcing, requirements specification, business modeland market strategy. Distributing simple tasks to individu-als is not necessarily a daunting process. However, assign-ing complex tasks require a high level of organization andenhancement especially with the challenge of dividing prob-lems, assigning the sub-problems to individuals and combin-ing the solutions (Zhang, Horvitz, and Miller 2011). The im-portance of the efficient division of tasks into subtasks, i.e.,smaller tasks, getting sub-solutions to these subtasks andsystematically combining the sub-solutions to give a com-prehensive solution to the original problem can be seen in(Zhang, Horvitz, and Miller 2011). One of the motivationsof crowdsourcing is that crowdsourcing cuts out a signifi-cant percentage of capital required to solve a problem us-ing outsourcing. Crowdsourcing is not entirely free but it ischeaper than outsourcing or having conventional employees(Howe 2006). This is highlighted as one of the major rea-sons for the success of crowdsourcing. Questions arise overthe accuracy, quality and authenticity of results of crowd-sourcing. Wikipedia is an example of a crowdsourcing plat-form and it was shown (Goodin 2005) that it is accurate de-spite the fact that it is an encyclopedia developed by mil-lions of users, i.e., a crowd. On the issue of disagreementson accuracy of Wikipedia articles, Wales (the founder ofWikipedia) had to settle such cases himself when Wikipediawas still a fledgling platform. The system is however now
AAAI Technical Report SS-12-06 Wisdom of the Crowd
2
Figure 1: Business Model Key Elements and Building Pro-cess
self-regulating (McNichol 2011).Crowdsourcing is more than just creating a website where
people just work. There must be a structure which guides theclients and the crowd in the desired directions, assists theclients in describing the challenges they want solved, andaids the crowd in executing tasks. It is also important thata crowdsourcing platform has a sufficient crowd, must beappealing to encourage participation, and must be relativelyeasy and feasible for the crowd to use.
According to (Chanal and Caron-Fasan 2008; Oster-walder 2007), the important elements required to build andclearly define a business model are customer segments,value proposition, distribution channels, customers relation-ship, revenue streams, key resources and key activities. Therelationship between these elements is shown in Figure 1.These elements are incorporated in the CrowdREquire busi-ness model development.
Due to the dynamic nature of the project and the rel-atively novel application of crowdsourcing to support re-quirements engineering, an agile approach for the develop-ment of requirements for CrowdREquire should be applied.For requirements elicitation, the brainstorming method wasused (van Lamsweerde 2010). This involved multiple ses-sions where the aims, objectives and potential limitationsof CrowdREquire were discussed. In addition, the VolereSpecification Template (AtlanticSystems 2010) which is anindustry-standard requirements specification template waspartly used. This was done in order to improve the viabil-ity of the final requirements specifications.
The market strategy involves using appropriate strate-gies and corresponding methods for connecting with thecustomers. Some of the factors considered in mapping outthe CrowdREquire market strategy are market segmenta-tion strategy, targeting strategy, positioning strategy, corpo-rate message and image, product and service strategy, pric-ing strategy, distribution channels, promotion and advertis-ing strategy, sales strategy and sales forecasts.
3 CrowdREquireThe CrowdREquire requirements specification, businessmodel and market strategy are presented and explained inthis section.
Business Model
The CrowdREquire business approach is detailed showingthe purpose, goals, stakeholders, work context model andtypical business events of CrowdREquire.
The Purpose of CrowdREquire CrowdREquire incorpo-rates the idea of crowdsourcing with requirements engineer-ing. In other words, the crowd submits requirement spec-ifications as solutions to tasks submitted by clients. Theprocess of developing requirements is a daunting task andrequires expertise which is not always available, thereforeCrowdREquire is aimed at providing this expertise throughthe crowd. Also, CrowdREquire has the potential to attractpeople from different fields of interest, knowledge and ex-perience and as a result, clients would be potentially able tochoose one out of many solutions to their defined problem.
CrowdREquire goals CrowdREquire is needed becauseit will aid individuals and companies in finding the bestrequirements specification for their proposed tasks andprojects. CrowdREquire also provides a communication toolto connect requirement engineering professionals with thecompanies that require their services. The aim of Crow-dREquire is to give timely and complete response to clientswho submit a task definition.
The Stakeholders The CrowdREquire stakeholders are asfollows:
1. Client: A client is any individual, company or organiza-tion that uses the crowd to gather their requirements. Also,the requirements process whether implicit or explicit is anessential process in the success of projects in several fieldsof practice. The range of potential clients is very broadwith respect to size, area of specialization and the natureof projects. The client has the final say on acceptance ofthe one of the submitted requirement specifications. Theclient is also encouraged and assisted to divide projectsinto tasks and subtasks whenever feasible.
2. Crowd: The crowd is the group of those who register onCrowdREquire and are willing to participate by submit-ting a solution to any of the projects offered. The crowdmember is ultimately responsible for deciding whether toparticipate in defining the requirement specification of aproject or not. If one crowd member’s requirement speci-fications are chosen by the client, the crowd member shallreceive a corresponding financial reward.
3. CrowdREquire Administration and Staff: The systemadministration shall have the full access to CrowdREquireand have the full authorization to monitor and maintainCrowdREquire. In addition, the CrowdREquire adminis-tration shall be for overseeing payment processes in whichthey receive the payment from the client and distribute itto the rewarded crowd members after deducting charges.
Work Context Model The work context model, shownin Figure 2 summarizes the activities that exist in Crow-dREquire and the role of each actor in CrowdREquire.
3
CrowdREquire Admin/Staff
Crowd Client
Figure 2: CrowdREquire Context Model
Figure 3: CrowdREquire Use Case Diagram
Business Events List Table 1 shows a list of the businessevents on CrowdREquire. The numbers for each BusinessUse Case (BUC) match the numbers of the correspondingCrowdREquire use cases shown in Figure 3. It also detailsthe input and output information for each event.
Market StrategyCrowdREquire shall be a web-based platform and shalltherefore be conceptually open to use around the globe. Thegeneral methods, distribution channels and industry struc-ture for crowdsourcing shall be identified with the mainfunctionality of CrowdREquire being distributed problemsolving. Also the primary market research channel shall bethe Internet while secondary market research shall also beconducted to collect important market details from resourcespublished by government and financial agencies. The cost ofthe research shall also be taken into account.
In addition, industry analysis shall cover a comprehen-sive study on industry segments, current trends, marketsize, growth rate, rise of other crowdsourcing-oriented busi-nesses, etc. Potential competitors shall be identified and thelevel of competition shall be established. Legal, political
and economic factors that influence the industry and mar-ket shall be identified and monitored. Furthermore, tech-nological breakthroughs such as advancements in Web 2.0shall be anticipated and swiftly incorporated where possibleinto CrowdREquire. Existing customer relationships shall bemaintained and improved where possible.
CrowdREquire Requirements SpecificationThe CrowdREquire functional and nonfunctional require-ments are shown in Table 2. These are general Crow-dREquire requirements rather than specific and detailed re-quirements. It should be noted that ‘the system’ in Table 2refers to CrowdREquire.
4 EvaluationAs seen in Table 2, necessary modifications were made inorder to achieve a general requirements specification setfor CrowdREquire. The organization and structure of theCrowdREquire specification defines and describes businessuse cases, presents a CrowdREquire use case diagram, ex-plained a work context model, and specified the generalfunctional and non-functional requirements. The require-ments were also checked for conflicts. In addition, the re-quirements specification was summarized for this paper asthe fit criteria and rationale for each requirement could notbe included.
With respect to the completeness and correctness of therequirements, the CrowdREquire requirements are writtenat a consistent level of detail. Clear and consistent languageis used in describing the requirements. Classes of users andtheir characteristics are also defined, with specifications onhow the system is supposed to serve the users. Differentforms of product requirements are detailed, with securityand legal requirements as well. The business side and mar-keting requirements are also highlighted. Industry standardmethods were employed in developing the CrowdREquirebusiness model and marketing strategy, and the system re-quirements were analyzed together with the business andmarket requirements.
There are certain challenges that could be faced by Crow-dREquire such as Quality Assurance (QA), government andlegal issues, and business viability. These challenges are notuncommon to crowdsourcing systems (Vukovic and Bar-tolini 2010). However, there are some methods that havebeen explored to tackle these challenges.
The quality challenge traverses all crowdsourcing mod-els, crowd types and sizes (Vukovic and Bartolini 2010). QAin crowdsourced search results was researched in (Le et al.2010), evaluating the impact of training question distribu-tion for the crowd. The results obtained show an improve-ment in the relevance of search results. This use of tests issimilar to the optional client-defined tests incorporated inthe CrowdREquire task definition requirements. These testscan ensure that each task is undertaken by qualified mem-bers of the crowd working. Also, the contest crowdsourcingmodel employed by CrowdREquire encourages a competi-tive environment and this naturally ensures that members ofthe crowd produce solutions to the best of their abilities. QA
4
Table 1: Business Events ListNo. Event Name Input/Output Summary of BUCs1 Client or crowd member registra-
tionCustomer details (in) Users are updated on CrowdREquire
database2 Client submits a task for require-
ments developmentProject specification and details(in)
Requirements task is uploaded on Crow-dREquire website
3 Qualified crowd member appliesfor task
Project specification and details(out)
Requirements task and corresponding toolsare made available to the crowd member
4 Client makes payment to Crow-dREquire
Secure Payment (in/out) Client has funds required for submittingtasks on CrowdREquire
5 Crowd member’s requirement spec-ification is selected by client as win-ning proposal
Winner selection (in) and securepayment (out/in)
Crowd member is informed and fundstransferred to the crowd member’s account
7 Crowd member requests further in-formation on task details
task detail inquiry (in) and furtherclarification (out)
Client receives questions and providesfeedback to the crowd member
8 Client or crowd member submitscomplaints or suggestions aboutthe platform or about crowd mem-ber/client
Complaints/Suggestions (in) andfeedback from administration (out)
Complaints or suggestions are put intoconsideration and necessary adjustmentsmade.
in crowdsourcing has also been researched by (Kern et al.2010) and the use of decision matrices in selecting a QA ap-proach was suggested. Based on the CrowdREquire modeland due to the non-determinacy of results, it is the responsi-bility of the client to select final requirements specifications,hence, the validation review method is used by the client.In addition, the modularization of tasks as recommended byCrowdREquire improves the quality of results as a crowdmember could choose to focus on a particular module ratherthan the system that makes up the modules (Vukovic andBartolini 2010).
Due to the subjective nature of gathering requirementsthrough crowdsourcing, the ability of the crowd to under-stand the basics of requirements engineering has a signifi-cant bearing on the quality of results. As a result, it is impor-tant for CrowdREquire to provide resources and informationnecessary for the crowd to understand the requirements en-gineering process.
Potential legal issues for crowdsourcing systems includeIntellectual Property (IP) rights, copyright, employmentlaws, data security, payment structures, etc. and some ofthese issues have been researched by (Wolfson and Lease2011). However, if these issues are properly handled wouldthey would not be obstacles to the success of CrowdREquire.In general, (Wolfson and Lease 2011) recommends thatcrowdsourcing platform providers pay close attention to thelaw ensure that contracts are signed clients and crowd mem-bers, and clients should clearly define their tasks and expec-tations. For example, InnoCentive requires that crowd mem-bers transfer IP rights in order for them to work on crowd-sourcing tasks (Vukovic and Bartolini 2010), and this ap-proach can also be applied in CrowdREquire.
The ability of CrowdREquire to attract clients and thecrowd is of high importance as the purpose of Crow-dREquire is defeated without the stakeholders. As a re-
sult, a business model and marketing strategy have beendefined. Another factor that can contribute to a success-ful crowdsourcing business are ensuring minimal businesscosts while ensuring brand quality (Vukovic and Bartolini2010). (Lakhani and Boudreau 2009) defined different typesof business models for crowdsourcing platforms and Crow-dREquire employs the two-sided platform model. This in-volves a crowdsourcing platform owner acting as an af-filiate to both clients and the crowd while clients interactfreely with the crowd. However, the crowdsouricing plat-form owner still has a degree of control declaring system pa-rameters and regulations. A high level of freedom in stake-holder interaction coupled with minimal but fundamentalplatform control ensure the viability of the CrowdREquiremodel.
5 SummaryWhile the process of developing requirements is not explic-itly stated in all processes, it is essential to the success ofany endeavor. This paper studies a conceptualized crowd-sourcing platform which provides requirements engineeringservices through on the crowd, and systematically providessimplified system, business model and market strategy re-quirements. As a result, the CrowdREquire model and re-quirements can be applied as a reference for new crowd-sourcing systems or used to improve existing crowdsourcingplatforms.
6 Future WorkOne area of further research is to execute a cost assessmentmodel for CrowdREquire in order to better understand itsdirect financial implications. Also, a feasibility study forCrowdREquire can be carried out. This would require a sur-vey of all the target users of the system, both clients and the
Functional Requirements1 The system shall be web-based.2 The system shall accept and register users.3 The system shall provide substantial information for users.4 The system shall require employee-related data and business-partner-related data from crowd members and clients
respectively.5 The system shall contain crowd-member-specific tabs on the crowd member profile, i.e., where the crowd member
can view details of tasks, search for tasks, upload requirements specifications, and update account information andinquiries.
6 The system shall contain client-specific tabs on the client profile, i.e., task description, task management, tasksubmissions, task selections and inquiries.
7 The system shall have a messaging function.8 The system shall provide users with timely updates on task suggestions and changes.9 The system shall enable clients to define and customize task templates.10 The system shall enable clients to define criteria and tests for selection of crowd members.11 The system shall inform the client the number of crowd members that are eligible for tasks based on specified task
criteria.12 The system shall shall contain task description tools such as a text editor, tables, images (upload), video(upload),
numbering and bullets, and document upload.13 The system shall enable the client to modularize a task into subtasks with corresponding payments, with the system
showing the percentage of the payment that goes to CrowdREquire.14 The system client shall store a history of all users’ task details.15 The system shall require payments from a client before a task can be published to the crowd.16 The system shall enable clients to buy CrowdREquire credits.17 The system shall have an Application Programming Interface (API) for developing custom tasks.18 The system shall have a ranking system for users.19 The system shall have desktop and mobile applications where users can work offline.
Non-Functional Requirements20 The system shall be usable on all browser and operating system platforms.21 The system shall have a user-friendly and simple interface.22 The system shall be readily available for usage.23 The system shall be robust in order to accommodate as many users as possible at a time without the system
performance being affected.24 The system shall be efficient in error handling and prevent loss of data.25 The system shall be secure.26 The system shall have flexible design tools.27 The system shall ensure and maintain a high-level of integrity among users.28 The system shall address any form of abuse or breach of legal conditions.
6
crowd, in order to observe the potential of CrowdREquire asa requirements engineering crowdsourcing system. Further-more, the potential impact of the success of CrowdREquireon other task-oriented commercial crowdsourcing systemssuch as Amazon Mechanical Turk and InnoCentive could beresearched.
However, the next step in line with this paper is to runexperiments on crowdsourcing requirements engineeringexperiments using existing platforms such as MechanicalTurk, Crowdflower or InnoCentive. An example experimen-tal setup is the generation of requirements for an advancedand modularized smart grid. In 2009, the Electric Power Re-search Institute (EPRI) of the United States (US) developeda model for smart grid interoperability (Dollen 2009). In thismodel, the smart grid was divided into seven domains andfurther divided to sub-domains. A summary is seen in Table3. Based on the EPRI smart grid model, a requirements spec-ification for developing a smart grid can be developed usingthe CrowdREquire concept. Each sub-domain is presentedas a module in the task and rewards are attached to eachmodule. The task is set such that the requirements for eachmodule should be defined based on smart grid applicationsparticularly with respect to (Dollen 2009). This experimentcan be executed on InnoCentive.com. Also, the challengesfaced and areas of improvement would be documented inorder to understand how CrowdREquire might work as acrowdsourcing platform.
trial Automation, Solar Generation,and Wind Generation.
Market Market Management, Retailing,Distributed Energy Resources(DER) Aggregation, Trading,Market Operations, and AncillaryOperations.
Service Provider Customer Management, Installa-tion and Maintenance, BuildingManagement, Home Management,Billing, and Account Management.
Operations Monitoring, Control, Fault Man-agement, Analysis, Reporting andStatistics, Calculations, Training,Records and Assets, OperationalPlanning, Maintenance and Con-struction, Extension Planning,Customer Support, Meter Readingand Control, Supply Chain and Lo-gistics, Financial, CommunicationsNetwork, Security Management,Premises, Human Resources,Business Planning and Reporting,and Stakeholder Planning andManagement.
Generation Control, Measure, Protect, Record,Asset Management, and Stabiliza-tion and Optimization.
Transmission Control, Measure, Protect, Record,Asset Management, and Stabiliza-tion and Optimization.
Distribution Control, Measure, Protect, Record,Asset Management, and Stabiliza-tion and Optimization.
McNichol, T. 2011. The wales rules for web 2.0.Osterwalder, A. 2007. How to Describe and Improve yourBusiness Model to Compete Better.van Lamsweerde, A. 2010. Requirements Engineering. Wi-ley Press.Vukovic, M., and Bartolini, C. 2010. Towards a researchagenda for enterprise crowdsourcing. In Margaria, T., andSteffen, B., eds., Leveraging Applications of Formal Meth-ods, Verification, and Validation, volume 6415 of LectureNotes in Computer Science. Springer Berlin / Heidelberg.425–434.Wolfson, S., and Lease, M. 2011. Look before you leap: Le-gal pitfalls of crowdsourcing. In ASIST 2011 Annual Meet-ing (New Orleans, LA).Zhang, H.; Horvitz, E.; and Miller, R. 2011. Crowdsourcinggeneral computation.
