Requirements engineering is a very critical phase in software development process. Requirements can be interpreted as basic decision alternatives which have to be negotiated by stakeholders. In this paper we present the results of an empirical study which focused on the analysis of key influence factors of successful requirements prioritization. This study has been conducted within the scope of software development projects at our university where development teams interacted with a requirements prioritization environment. The major result of our study is that anonymized preference elicitation can help to significantly improve the quality of requirements prioritization, for example, in terms of the degree of team consensus, prioritization diversity, and quality of the resulting software components. RecSys'12 9th - 13th September, 2012, Dublin, Ireland Paper presented at the 2012 Decisions@RecSys workshop in conjunction with the 6th ACM conference on Recommender Systems. Copyright c 2012 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors..
Requirements Engineering (RE) is the branch of software
engineering concerned with the real-world goals for functions of and
constraints on software systems [
Today’s software projects still have a high probability to be
canceled or at least to significantly exceed the available resources [
Requirements can be regarded as a representation of decision
alternatives or commitments that concern the functionalities and
qualities of the software or service [
Quality decisions, e.g., is the requirement non-redundant,
concrete, and understandable?
Preference decisions, e.g., which requirements should be
considered for the next release?
Classification decisions, e.g., to which topic does this
requirement belong?
Property decisions, e.g., is the effort estimation for this
requirement realistic?
Stakeholders are often faced with a situation where the amount
and complexity of requirements outstrips their capability to
survey them and reach a decision [
The focus of this paper will be preference decisions, i.e., we want
to support groups of stakeholders in the context of prioritizing
software requirements for the next release. Typically, resource
limitations in software projects are triggering the demand of a
prioritization of the defined requirements [
Only a systematic prioritization can guarantee that the most
essential functionalities of the software system are implemented in-time
[
Establishing consensus between stakeholders regarding the
prioritization of a given set of requirements is challenging.
Prioritizations do not only have to take into account business process related
criteria but as well criteria which are related to technical aspects
of the software. Especially in larger projects, stakeholders need
a tool-supported prioritization approach which can help to reduce
influences related to psychological and political factors [
The remainder of this paper is organized as follows. In Section 2 we provide an overview of the basic functionalities of the INTELLIREQ requirements engineering environment developed at our university to collect preferences and decisions of stakeholders during the course Object-oriented Analysis and Design at the Graz University of Technology. In Section 3 we introduce the basic hypotheses that have been investigated within the scope of our empirical study; in this context we also provide details about the study design. In Section 4 we report the major results of our empirical study and show the effect of anonymity on the group consensus, the decision diversity and the output quality. With Section 5 we conclude the paper.
INTELLIREQ is a group decision environment that supports computer science students at our university in deciding on which requirements should be implemented within the scope of their software projects. For this task 219 students enrolled in a course about Object-Oriented Analyse and Design at the Graz University of Technology had to form groups of 5–6 members. Unfortunately, it is not possible to evaluate the existing knowledge and experience of the students and the resulting groups. We therefore distributed the resulting groups randomly on the different evaluation pools and assume that the knowledge and experience is equally distributed on each pool. Each group had to implement a software system with an average effort of about 8 man months. Figure 1 shows the anonymized preference presentation of stakeholders in IntelliReq.
In our study, 39 software development teams had to define a set of requirements which in the following had to be implemented. These requirements had to be prioritized and the resulting prioritization served as a major criteria for evaluating the corresponding software components at the end of the project.
The requirements prioritization process consisted of three different phases (see Figure 2) denoted as construction (collection of individual stakeholder preferences), consensus (adaptation of own preferences, see Figure 3), and decision (group decision defined and explained by the project manager). This decision process structure results in about 15.000 stakeholder decisions and 798 corresponding group decisions taken by the team leaders (project managers). On the basis of this scenario we conducted an empirical evaluation with the goal to analyze the effects of supporting anonymized requirements prioritization. The basic settings of this study will be presented in the following section. ings. The group consensus can then be interpreted as the counterpart of dissent (see Formula 2). As the consensus is the simple inversion of the dissent, we will only take into account the consensus in the remaining paper.
dissent(x) =
jRequirementsj consensus(x) =
1 dissent(x) Decision Diversity. The decision diversity of a group can be defined in terms of the average over the decision diversity of individual users in the consensus phase (see Figure 2). The latter is defined in terms of the standard deviation derived from the decision du of a user – a decision consists of the individual requirements prioritizations of the user.
diversity(x) =
jU sersj (1) (2) (3)
Within the scope of our empirical study we wanted to investigate the impact of anonymous preference elicitation on the decision support quality of the INTELLIREQ environment. Consequently, each project team interacted with exactly one of two existing types of preference elicitation interface. One interface (type 1: non-anonymous preference elicitation) provided an overview of the personal preferences of team members (stakeholders) where each team member was represented by her/his name. In the second type of interface (type 2: anonymous preference elicitation) the preferences of team members were shown in anonymized form where the name of the individual team member was substituted with the terms "User 1", "User 2", etc (see Figure 1). The hypotheses (H1– H8) used to evaluate the decision process are summarized in Figure 4. These hypotheses were evaluated on the basis of the following observation variables.
Anonymous preference elicitation. This variable indicates with which type of prioritization interface the team members were confronted (either summarization of the preferences of the team members including the name of the team members or not including the name of the team members).
Consensus and Dissent. An indication to which extent the team members managed to achieve consensus (dissent) – see the second phase of the group decision process in Figure 2 – is provided by the corresponding variables. We measured the consensus of a group on the basis of the standard deviation derived from requirementspecific group decisions. Formula 1 can be used to determine the dissent of a group x which is defined in terms of the normalized sum of the standard deviations (sd) of the requirement-specific vot
Output Quality. The output quality of the software projects conducted within the scope of our empirical study has been derived from the criteria such as degree of fulfillment of the specified requirements. We also weighted the requirements according to their defined priority in the prioritization task. E.g. not including a very high important requirement enormously decreases the quality value. On the opposite, low priority requirements will only have a small impact on the quality value. Therefore, defining a high priority for a requirement which is of minor importance has to be implemented anyway. On the other hand, each group has to implement all important requirements for the user experience and which are important for the functionality. Therefore, the requirements prioritization has a direct impact on the quality value. The quality of the project output has been determined by teaching assistants who did not know to which type of preference elicitation interface (anonymous vs. non-anonymous) the group has been assigned to. These assignments were randomized over all teaching assistants, i.e., each teaching assistant had to evaluate (on a scale of 0..30 credits) groups who interacted with an anonymous and a non-anonymous interface.
Within the scope of our study we wanted to evaluate the following hypotheses.
H1: Anonymous Preference Elicitation increases Consensus. The
idea behind this hypothesis is that anonymous preference
elicitation helps to decrease the commitment [
H2: Anonymous Preference Elicitation decreases Dissent.
Following the idea of hypothesis H1, non-anonymous preference
elicitation increases commitment with regard to already taken (and
published) decisions. It also decreases the probability of detecting
hidden profiles [
H3: Consensus increases Decision Diversity. As a direct
consequence of an increased exchange of decision-relevant information
(see Hypothesis H1), deep insights into major properties of the
decision problem can be expected. As a consequence, the important
differentiation between important, less important, and unimportant
requirements with respect to the next release [
H5: Consensus increases Output Quality. From Hypothesis H3
we assume a positive correlation between the degree of consensus
and the diversity of the group decision. The diversity is an indicator
for a meaningful triage [
H6: Dissent decreases Output Quality. In contrary, dissent leads to a lower decision diversity and – as a consequence – to less meaningful results of requirements triage.
H7: Decision Diversity increases Output Quality. Group decision diversity is assumed to be a direct indicator for the quality of the group decision. With this hypothesis we want to analyze the direct interrelationship between prioritization diversity and the quality of the resulting software.
H8: Anonymous Preference Elicitation increases Output Quality. Finally, we want to explicitly analyze whether there exists a relationship between the type of preference elicitation and the corresponding output quality.
We analyzed the hypotheses (H1–H8) on the basis of the
variables introduced in Section 3.1 We used a Mann-Whitney U-test if
the examined data set is not normal distributed (H1,H2) and the
ttest if the data set is normal distributed (H8). The correlations (H3
– H7) are calculated with Pearson correlations (normal distribution)
and with the Spearman’s rank correlations (no normal distribution).
H1. The degree of group consensus in teams with anonymous
preference elicitation is significantly higher compared to teams with
non-anonymous preference elicitation (Mann-Whitney U-test, p <
0.05). An explanation model can be the reduction of commitment
[
H3. There is a positive correlation between the group consensus
and the corresponding decision diversity (correlation 0.523, p <
0.01). More group discussions can lead to a higher level of
relevant knowledge about the decision problem. In the following this
can lead to a development of a deeper understanding of the need of
requirements triage [
H4. Dissent is an inverse function of group consensus – the higher the dissent, the lower the corresponding decision diversity (correlation -0.523, p < 0.01). A lower degree of group decision diversity (prioritization diversity) can be explained by a lower degree of decision-relevant knowledge.
H5. Consensus in group decision making increases the output quality (correlation 0.399, p < 0.01). An overlap in the personal stakeholder preferences can be interpreted as an indicator of a common understanding of the underlying set of requirements. This leads to a better prioritization and a higher quality of the resulting software components.
H6. The hypothesis can be confirmed (correlation -0.399, p < 0.01), i.e., there is a negative correlation between group dissent and the corresponding output quality.
H7. In our analysis we could detect a positive correlation between group decision diversity (diversity of prioritization) and the corresponding output quality (correlation 0.311, p < 0.01). Decision diversity can be seen as an indicator of a reasonable triage process and reasonable prioritizations result in higher-quality software components.
H8. Groups with anonymous preference elicitation performed significantly better compared to groups with a non-anonymous preference elicitation (independent two-sample t-test, p < 0.05). 5.
Requirements prioritization is an important task in software development processes. In this paper we motivated the application of requirements prioritization and discussed issues related to the aspect of anonymizing group decision processes in requirements prioritization. The results of our empirical study clearly show the advantages of applying anonymized preference elicitation, for example in terms of higher-quality software components, and can be seen as a step towards a more in-depth integration of decisionoriented research in requirements engineering processes. 6.