- Crowdsourced testing is an emerging phenomenon in software testing which utilizes benefits of crowdsourcing for software testing. It brings cost and quality advantage with faster delivery of software products. Among all the software development activities, testing is one of the primary activities considered for crowdsourcing. Certain types of testing demand for testers from diverse ethnicity, culture, or geography; crowdsourced testing has become a viable option to address such needs. This paper presents some of the initial attempts done at Accenture to adopt crowdsourced testing. We present some of the software development projects where testing had been crowdsourced using a crowd based platform and yielded promising results. We also showcase how crowd wisdom, gleaned through a sentiment analysis performed on the users' feedback, have helped in improving software.
Crowdsourcing is one of the major paradigm shifts that
organizations are witnessing these days [
Due to rapid change in technology landscape, a software need to be tested on verity of hardware and software platforms in the shortest time possible. This can only be achieved by parallelizing software testing tasks as much as possible. Crowdsourced testing is one of the promising alternative to test software in parallel on verity of hardware and software platforms. In addition it also provides capability for testing software for localization needs such as for different languages, and geographies. Some of the advantages of crowdsourced testing are listed as follows: 1. Improved quality: with crowdsourced testing, one can afford a diverse set of test scenario exercised by crowd workers and thus the software can be tested thoroughly in a shorter time span. 2. Workforce flexibility: one can attract and retain testers who do not wish to accept full time position. 3. Reduced time to market: the testing tasks can be parallelized, thus crowdsourced testing reduces time to market. 4. Tester diversity: crowdsourcing platforms provide access to a large pool of testers from diverse geography, languages, etc. 5. Agility: The crowdsourced testing helps in more agility in workforces.
Here we present some of the experiences gained while adopting crowdsourced testing within Accenture. We discuss some of the preliminary attempts of utilizing crowd for different types of testing such as functional testing, usability testing, and localization testing. We further demonstrate the value of crowdsourced testing in terms of reduction in overall cost and time to deliver software applications. Firstly, we present an estimate that we performed to assess the value of crowdsourced testing. This estimation was done on a project that has already been tested by internal testers. Next, we discuss a project where we adopted crowdsourced testing on a real project. Towards the end, we discuss some of the ongoing work on utilizing crowdsourced data for software debugging purposes that collects reviews from public forums to identify use case that can replay the software bugs.
In one of our assessments, we measured cost and time taken in Sandbox testing performed by internal testers and estimated how much cost and time could have been saved when it was performed by crowdsourced testers. In this project the internal testers performed test for six websites which were launched in Japan. The test activity involved Functional testing which was done by the internal testers.
The same test activity when performed by crowdsourced testers was estimated to cost less than half when compared to performed by the internal testers. Also, the duration was estimated to be 10 time lesser when performed by internal tester. In addition, this type of testing has an advantage of being done by the users located in Japan using their own devices and network connectivity; thus the device procurement cost was almost negligible. Table 1 highlights the measured and estimated metrics for internal testers and crowd testers.
PROOF OF VALUE ESTIMATES FOR CROWDSOURCED TESTING
No of Testers 6
The previous activity helped us in estimating the value proposition for crowdsourced testing. To understand better the true benefits of crowdsourced testing, we adopted crowdsourced testing in a project which was for a world’s leading Hospitality Group to test their Booking engine’s Short Message Service (SMS) communication for any new booking, cancellation, or changes in the service requirement. The above testing required recreation of booking process and confirmation to the user with an SMS message. The SMS needed to have personalized detail, type of room booked and the price based on the user location, language, format as per the mobile device used, and for the network service provider of the user. We initially decided to leverage enterprise testers spread across globe to perform the above test. However, we did not have significant presence in countries like Fiji, Nigeria and Egypt along with required combination of mobile devices and language capabilities. Thus, we considered to use a Mobile Test Automation platforms as next alternative. This came out to be more expensive proposition as per the client’s budget. The next alternative we considered was to utilize crowd testing platform. With crowd testing, not only we were able to test the application by a diverse set of users, countries, devices, and languages, but also we were able to perform this testing with 80% lesser cost, when compare to the estimates provided by Mobile Test Automation Platform vendors. In addition the crowd testers were fairly equipped with necessary mobile devices and networks to execute the needed test scenarios. Time to execute this project from signing the scope of the work to execution was mere two weeks. This cost saving was primarily achieved because the testers’ onboarding was much quicker compare to traditional testers’ onboarding. Moreover, it did not bear any procurement cost for necessary hardware.
Another area where we are exploring the use of crowdsourcing
is software debugging to identifying potential use cases that
result in defects. We refer such use cases as ‘unpleasant use
cases’. Often it is difficult to replay the unpleasant use cases
as users do not post the entire scenario that led to an
unpleasant experience. In this work we leverage power of
crowd to understand how the Mobile Applications (Apps) and
software used in (Internet of Things) IoT devices are used and
which use cases led the software to an instable, faulty, or
unpleasant situations. This analysis is done based on users’
feedbacks. Users provide feedback on the services provided by
App but not about specific aspects of an App that led to an
unpleasant experience. Therefore, many a time we do not
know which aspect of an App have led to a poor service
experience. To collect such use cases, we have adopted a
process where we collect user feedback about Apps usage and
identify use cases of an App that led to bad user experience.
This process is termed as “WhatsAppening”. The user
feedback is collected from Apps marketplaces with an analysis
tool. We utilized the ‘App review and scoring system’,
supported by platform Applause [
In this paper we presented some initial experiences gained while adopting crowdsourcing for testing activities in an enterprise context. The initial results indicate that crowdsourced testing is one of the plausible alternatives for testing workforces in certain scenarios as it provides a flexible and diverse workforce to meet organizations test requirements. However, there are several challenges that need to be address. There is a need for standard testing practice that includes crowdsourced testing an integral part of standard testing process. Value proposition analysis for crowdsourced testing needs to be validated on wide number of additional projects. Also crowdsourced testing need to be assessed from the perspective of confidentiality and security.