Crowdsourcing is an emerging, typically online, distributed problem solving and production model where a problem is solved through the involvement of a large number of people. In this study, we investigate the potential of crowdsourcing in aiding Requirements Engineering. Although the whole area is still to be explored fully, we focus on the Requirements Elicitation stage. In this paper, we survey the literature on crowdsourcing in a variety of disciplines and deduce a set of features which characterize its main constructs; the crowd and the crowdsourcers. We then conduct two focus groups to explore the relationship between these features and the quality of requirements elicited via crowdsourcing. The analysis will lead to a number of hypotheses to confirm and enhance in a future research in the area. The ultimate goal is to systematically develop crowdsourcing platforms for Requirements Engineering and guarantee correctness and maximize efficiency.
Nowadays, the uncertainty of the technical and social environments, where software systems operate, has radically increased. This phenomenon has been accelerated by the ever-increasing utilization of web-based systems, e.g. Software as a Service, and mobile applications. In essence, this means that system developers and software engineers encounter a wider audience of users, which we call the general public, or the crowd. To cater for the requirements of the crowd, we perhaps need to involve that crowd; hence, the importance of crowdsourcing.
Eliciting the requirements from users is traditionally carried out as a design time
activity. Recent approaches advocate that it can be also done at runtime involving the
actual users, e.g., via users feedback [
Crowdsourcing in requirements elicitation has the potential to increase the quality
and comprehensiveness and even the economic feasibility of requirements elicitation.
Crowdsourcing gives the software and engineering team access to a wide diversity of
actual and potential users. It could be utilized throughout the life-cycle of the
software. This would allow developers, potentially, to gain a wider, and more up-to-date
knowledge of how users perceive the system role in meeting their requirements, and
to understand how that perception changes over time. Traditional elicitation
approaches, e.g., interviews and focus groups, are too expensive to deal efficiently with
crowd-oriented applications. While Crowdsourcing seems a promising approach, only
a very few attempts have been at utilizing this potential, e.g. [
The goal of this study is to explore the relationships among different styles of Crowdsourcing and the quality of crowdsourced requirements. To this end, we have surveyed the literature in Crowdsourcing and identified a set of its characterizing features. This has aided the design of two focus groups to explore the relation between these features and the quality of elicited requirements. We have identified a set of observations to confirm in a future study by consulting the opinion of researchers and practitioners in the requirements engineering community. 2
Requirements engineering focuses on the development of a system that best suits the
needs of its stakeholders, i.e. the customers who have paid for the system, but since
the customers’ final goal is for their system to be utilized by end users, their
immediate needs should be considered during RE [
Crowdsourcing is an area of study which has recently gained attention in various scientific fields, such as business, management, environmental sciences, social studies, computing and so on. Crowdsourcing could also aid software development, particularly at the requirements engineering stage, since the crowd could be the potential users of software which is designed to meet their requirements. There are a number of studies which attempted to utilize the power of the crowd and end-users to solve requirements engineering problems. These include:
software in meeting its requirements in a dynamic context makes the validation of a
system a hard and lenghty task. In [
Feedback-based requirements engineering: users feedback on software could
help developers to better understand the requirements of the next release of the
system. This feedback could be explicit, e.g. via forums, or implicit, e.g. through
monitoring their patterns of use of the software. Pagano and Maalej [
Stakeholders’ discovery: in complex and dynamic systems, it is hard to identify
the set of stakeholders and their roles and expertise and also their requirements.
Crowdsourcing here would help identifying a comprehensive set of stakholders from
an initial set of stakeholders specified by the analysts. Lim et al. [
Requirements Identification: in software paradigms like Cloud Computing and
Mobile Apps, the users set is highly diverse and unpredictable. This means relying on
an elite group of users to understand what functionality and quality attributes to meet
in the software is limited and also costly. We could harness the power of the crowd to
understand their requirements as part of the requirements elicitation stage.
CrowdREquire [
Empirical Validation: Validation and users testing for implemented systems share
the same difficulties as mentioned above for requirements elicitation and for
requirements-driven adaptation in the sense that it is costy and lengthy and it often
leads to results which are valid only temporarily, especially in a dynamic
environment. This is true as users might not maintain the same opinion when time
passes due to the emergence of competitive solutions and the use of software in
contexts which were not thought of at the engineering stage. There has been also
some research on the use of crowdsourcing for empirical studies in software
engineering [
In this section, we investigate the potential impact of the different settings of crowdsourcing on the quality of crowdsourced requirements. To this end, we followed a three-steps methodology and report on the results of the first two phases.
In the first step, we studied the literature on crowdsourcing in a variety of disciplines and we identified the features of crowdsourcing according to the different constituents of crowdsourcing, i.e. the crowdsourcer (requirements engineers in the case of requirements engineering) and the crowd (potential software stakeholders including end-users in the case of requirements engineering). This list is illustrated in Table 1.
The Crowdsourcer 1. Incentives Provision 1.1. Financial Incentives 1.2. Social Incentives 1.3. Entertainment Incentives
3.1. Open Call 3.2. Feedback to Crowd 3.3 No Harm to Crowd
In the second step, we held two focus groups, consisting of a mixed group of users and software engineers. We introduced to them the concept of crowdsourcing and its different applications, focusing on software development. The first focus group consisted of seven people; three software developers and four users, four female and three male participants. The second consisted of seven people; four software developers and three users, with one female and six male participants. The 14 participants in the two focus groups were from 10 countries and with an age range of 23 to 50. Then we asked them pre-prepared questions about the effect of different settings of crowdsourcing on the quality of requirements obtained from the crowd. At the end of each question, the participants were asked to write their comments and answer a correlated question in a pre-prepared questionnaire,. This helped us to get qualitative and quantitative data. We then analyzed the focus group results, both by listening to the recordings and by studying the filled-in questionnaires and the written comments. The results which were obtained from the focus groups are listed in Table 2.
biased information, Saturation, More difficulty in coordination More honesty, Less credibility
management, More inconsistency, Possible Dominance of some users, More understandability More correctness, More chance of encountering malicious participants
be actively involved, More participants, More possibility of carrying on with the task
Diversity
More creativity, More relevance, More willing to participate More correctness, More relevance, More completeness
be actively engaged, More participants, Misleading the crowd from true involvement More motivation to engage, Disturbing participants, More influence on next stages, Feeling that participant’s idea is important, More willingness to participate in future
In our third step, we aim to collect and analyze the opinion of the experts (researchers and practitioners) in requirements engineering about our findings including those obtained from the focus groups. The final result will be a benchmark for the relationship between different crowdsourcing settings and the various quality attributes of elicited requirements. The expert survey will be conducted as part of the activities of the 20th International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2014). The survey could be found in the following URL: http://www.esurv.org?u=CSfRE. Also, a PDF version of the online survey is accessible from the following URL: http://goo.gl/oIVdbt. 4
This paper reports on initial results of investigating the use of crowdsourcing for requirements engineering, with focus on elicitation activity. We reported on the results of the literature review and two conducted focus groups as a preliminary step to design an expert survey. The goal is to devise systematic engineering approaches to develop crowdsourcing for requirements engineering.
Acknowledgements. The research was supported by an FP7 Marie Curie CIG grant (the SOCIAD Project) and by Bournemouth University through the Fusion Investment Fund (the BBB, BUUU and VolaComp projects) and the Graduate School Santander Grant for PGR Development.