=Paper= {{Paper |id=None |storemode=property |title=Eliciting Stakeholder Preferences for Requirements Prioritization |pdfUrl=https://ceur-ws.org/Vol-893/paper5.pdf |volume=Vol-893 |dblpUrl=https://dblp.org/rec/conf/recsys/FelfernigNR12 }} ==Eliciting Stakeholder Preferences for Requirements Prioritization== https://ceur-ws.org/Vol-893/paper5.pdf

Eliciting Stakeholder Preferences for Requirements
Prioritization

Alexander Felfernig Gerald Ninaus Florian Reinfrank
Institute for Software Institute for Software Institute for Software
Technology Technology Technology
Graz University of Technology Graz University of Technology Graz University of Technology
Inffeldgasse 16b Inffeldgasse 16b Inffeldgasse 16b
8010 Graz, Austria 8010 Graz, Austria 8010 Graz, Austria
afelfern@ist.tugraz.at gninaus@ist.tugraz.at freinfra@ist.tugraz.at

ABSTRACT basis for all subsequent phases in the development process and high
Requirements engineering is a very critical phase in software devel- quality requirements are a major precondition for the success of the
opment process. Requirements can be interpreted as basic decision project [4].
alternatives which have to be negotiated by stakeholders. In this
paper we present the results of an empirical study which focused Today’s software projects still have a high probability to be can-
on the analysis of key influence factors of successful requirements celed or at least to significantly exceed the available resources [13].
prioritization. This study has been conducted within the scope of As stated by Firesmith [5], the phase of requirements engineer-
software development projects at our university where development ing receives rarely more that 2-4% of the overall project efforts
teams interacted with a requirements prioritization environment. although more efforts in getting the requirements right result in
The major result of our study is that anonymized preference elici- significantly higher project success rates. A recent Gartner report
tation can help to significantly improve the quality of requirements [7] states that requirements defects are the third source of product
prioritization, for example, in terms of the degree of team consen- defects (following coding and design), but are the first source of
sus, prioritization diversity, and quality of the resulting software delivered defects. The cost of fixing defects ranges from a low of
components. approximately $70 (cost to fix a defect at the requirements phase)
to a high of $14.000 (cost to fix a defect in production). Improving
the requirements gathering process can reduce the overall cost of
Categories and Subject Descriptors software and dramatically improve time to market.
D.2 [Software Engineering]: Requirements Engineering; D.2.1
[Requirements/Specifications]: Requirements Negotiation; H.5 Requirements can be regarded as a representation of decision alter-
[Information Interfaces and Presentation]: Modeling Environ- natives or commitments that concern the functionalities and quali-
ments ties of the software or service [1]. Requirements engineering (RE)
is then a complex task where stakeholders have to deal with various
decisions [11]:
General Terms
Human Factors, Experimentation • Quality decisions, e.g., is the requirement non-redundant,
concrete, and understandable?

Keywords • Preference decisions, e.g., which requirements should be con-
sidered for the next release?
Requirements Prioritization, Group Decision Making
• Classification decisions, e.g., to which topic does this re-
1. INTRODUCTION quirement belong?
Requirements Engineering (RE) is the branch of software en- • Property decisions, e.g., is the effort estimation for this re-
gineering concerned with the real-world goals for functions of and quirement realistic?
constraints on software systems [14]. RE is considered as one of the
most critical phases in software projects, and poorly implemented Stakeholders are often faced with a situation where the amount
RE is one major risk for project failure [8]. Requirements are the and complexity of requirements outstrips their capability to sur-
vey them and reach a decision [3]. The amount of knowledge and
number of stakeholders involved in RE processes tend to increase
as well. This makes individual as well as group decisions much
more difficult.

RecSys’12 9th - 13th September, 2012, Dublin, Ireland The focus of this paper will be preference decisions, i.e., we want
Paper presented at the 2012 Decisions@RecSys workshop in conjunc- to support groups of stakeholders in the context of prioritizing soft-
tion with the 6th ACM conference on Recommender Systems. Copyright ware requirements for the next release. Typically, resource limi-
c 2012 for the individual papers by the papers’ authors. Copying permit-
ted for private and academic purposes. This volume is published and copy- tations in software projects are triggering the demand of a prior-
righted by its editors.. itization of the defined requirements [8]. Prioritizations support

27
software project managers in the systematic definition of software
releases and help to resolve existing preference conflicts among
stakeholders.

Only a systematic prioritization can guarantee that the most essen-
tial functionalities of the software system are implemented in-time
[12]. Typically, requirements prioritization is a collaborative task
where stakeholders in a software project collaborate with the goal
to achieve consensus regarding the prioritization of a given set of
requirements. The earlier requirements are prioritized, the lower is
the effort of implementing irrelevant requirements and the higher is
the amount of available resources to implement the most relevant
requirements [12].

Establishing consensus between stakeholders regarding the prior-
itization of a given set of requirements is challenging. Prioritiza-
tions do not only have to take into account business process related
criteria but as well criteria which are related to technical aspects
of the software. Especially in larger projects, stakeholders need
a tool-supported prioritization approach which can help to reduce
influences related to psychological and political factors [12]. Re- Figure 1: I NTELLI R EQ Anonymous Preference Presentation:
quirements prioritization is a specific type of group work which the preferences of users are anonymized by replacing the stake-
becomes increasingly important in organizations [10]. holder names with the terms "User 1", "User 2", ... , "User n".
The order of stakeholders and the assignment to these terms is
Prioritization decisions are typically taken in groups but this task randomly generated.
is still ineffective due to reasons such as social blocking, censor-
ship, and hidden agendas [10]. One balancing strategy is to drop
or defer low priority requirements to a later release [12]. In a study the students and the resulting groups. We therefore distributed the
conducted at the Graz University of Technology during the course resulting groups randomly on the different evaluation pools and
Object Oriented Analyses and Design, the stakeholder part of the assume that the knowledge and experience is equally distributed
customer was impersonated by four course assistants. These as- on each pool. Each group had to implement a software system
sistants were not aware of the study settings and had to review with an average effort of about 8 man months. Figure 1 shows the
the software functionality developed by the different teams. This anonymized preference presentation of stakeholders in IntelliReq.
evaluation did not include a code review. Rather it was supposed
to assess the user experience of the product and which important
functionality was supported. These important functions were par-
tially defined by the exercise given in the course. The result of this
evaluation is represented by a quality value between 0 and 30 cred-
its. The major contribution of this paper is to show how anonymity
in group decision processes can help to improve the quality of re-
quirements prioritizations.

The remainder of this paper is organized as follows. In Section
2 we provide an overview of the basic functionalities of the I N -
TELLI R EQ requirements engineering environment developed at our
university to collect preferences and decisions of stakeholders dur-
ing the course Object-oriented Analysis and Design at the Graz
University of Technology. In Section 3 we introduce the basic hy-
potheses that have been investigated within the scope of our empir-
ical study; in this context we also provide details about the study
design. In Section 4 we report the major results of our empirical Figure 2: I NTELLI R EQ Prioritization (Decision) Process. Con-
study and show the effect of anonymity on the group consensus, struction: stakeholders define their initial preferences; Con-
the decision diversity and the output quality. With Section 5 we sensus: stakeholders adapt their preferences on the basis of the
conclude the paper. knowledge about preferences of other stakeholders. Decision:
project managers take the final group decision.
2. INTELLIREQ DECISION SUPPORT
I NTELLI R EQ is a group decision environment that supports com- In our study, 39 software development teams had to define a set of
puter science students at our university in deciding on which re- requirements which in the following had to be implemented. These
quirements should be implemented within the scope of their soft- requirements had to be prioritized and the resulting prioritization
ware projects. For this task 219 students enrolled in a course about served as a major criteria for evaluating the corresponding software
Object-Oriented Analyse and Design at the Graz University of Tech- components at the end of the project.
nology had to form groups of 5–6 members. Unfortunately, it is
not possible to evaluate the existing knowledge and experience of The requirements prioritization process consisted of three different

28
phases (see Figure 2) denoted as construction (collection of indi- ings. The group consensus can then be interpreted as the counter-
vidual stakeholder preferences), consensus (adaptation of own pref- part of dissent (see Formula 2). As the consensus is the simple
erences, see Figure 3), and decision (group decision defined and inversion of the dissent, we will only take into account the consen-
explained by the project manager). This decision process structure sus in the remaining paper.
results in about 15.000 stakeholder decisions and 798 correspond-
ing group decisions taken by the team leaders (project managers).
On the basis of this scenario we conducted an empirical evaluation
P
r∈Requirements sd(r)
with the goal to analyze the effects of supporting anonymized re- dissent(x) = (1)
|Requirements|
quirements prioritization. The basic settings of this study will be
presented in the following section. 1
consensus(x) = (2)
dissent(x)

Decision Diversity. The decision diversity of a group can be de-
fined in terms of the average over the decision diversity of individ-
ual users in the consensus phase (see Figure 2). The latter is defined
in terms of the standard deviation derived from the decision du of
a user – a decision consists of the individual requirements prioriti-
zations of the user.

P
u∈U sers sd(du )
diversity(x) = (3)
|U sers|

Figure 3: I NTELLI R EQ Preference Adjustment (Consensus): Output Quality. The output quality of the software projects con-
stakeholders can view their initial preferences and preferences ducted within the scope of our empirical study has been derived
of other stakeholder. With this information stakeholders can from the criteria such as degree of fulfillment of the specified re-
adjust their preferences to increase group consensus. quirements. We also weighted the requirements according to their
defined priority in the prioritization task. E.g. not including a
very high important requirement enormously decreases the qual-
ity value. On the opposite, low priority requirements will only
3. EMPIRICAL STUDY have a small impact on the quality value. Therefore, defining a
Within the scope of our empirical study we wanted to inves- high priority for a requirement which is of minor importance has
tigate the impact of anonymous preference elicitation on the de- to be implemented anyway. On the other hand, each group has to
cision support quality of the I NTELLI R EQ environment. Conse- implement all important requirements for the user experience and
quently, each project team interacted with exactly one of two exist- which are important for the functionality. Therefore, the require-
ing types of preference elicitation interface. One interface (type 1: ments prioritization has a direct impact on the quality value. The
non-anonymous preference elicitation) provided an overview of the quality of the project output has been determined by teaching as-
personal preferences of team members (stakeholders) where each sistants who did not know to which type of preference elicitation
team member was represented by her/his name. In the second type interface (anonymous vs. non-anonymous) the group has been as-
of interface (type 2: anonymous preference elicitation) the pref- signed to. These assignments were randomized over all teaching
erences of team members were shown in anonymized form where assistants, i.e., each teaching assistant had to evaluate (on a scale
the name of the individual team member was substituted with the of 0..30 credits) groups who interacted with an anonymous and a
terms "User 1", "User 2", etc (see Figure 1). The hypotheses (H1– non-anonymous interface.
H8) used to evaluate the decision process are summarized in Figure
4. These hypotheses were evaluated on the basis of the following Within the scope of our study we wanted to evaluate the follow-
observation variables. ing hypotheses.

Anonymous preference elicitation. This variable indicates with which H1: Anonymous Preference Elicitation increases Consensus. The
type of prioritization interface the team members were confronted idea behind this hypothesis is that anonymous preference elicita-
(either summarization of the preferences of the team members in- tion helps to decrease the commitment [2] related to an individual
cluding the name of the team members or not including the name decision taken in the preference construction phase (see Figure 2),
of the team members). i.e., changing his/her mind is easier with an anonymous preference
elicitation interface. Furthermore, anonymous preference elicita-
Consensus and Dissent. An indication to which extent the team tion increases the probability of detecting hidden profiles [6], i.e.,
members managed to achieve consensus (dissent) – see the second increases the probability of exchanging decision-relevant informa-
phase of the group decision process in Figure 2 – is provided by the tion [9].
corresponding variables. We measured the consensus of a group
on the basis of the standard deviation derived from requirement- H2: Anonymous Preference Elicitation decreases Dissent. Follow-
specific group decisions. Formula 1 can be used to determine the ing the idea of hypothesis H1, non-anonymous preference elicita-
dissent of a group x which is defined in terms of the normalized tion increases commitment with regard to already taken (and pub-
sum of the standard deviations (sd) of the requirement-specific vot- lished) decisions. It also decreases the probability of detecting hid-

29
den profiles [6] and thus also decreases the probability of high-
quality decisions (see H3).

H3: Consensus increases Decision Diversity. As a direct conse-
quence of an increased exchange of decision-relevant information
(see Hypothesis H1), deep insights into major properties of the de-
cision problem can be expected. As a consequence, the important
differentiation between important, less important, and unimportant
requirements with respect to the next release [3] can be achieved.

H4: Dissent decreases Decision Diversity. From less exchange
of decision-relevant information we can expect a lower amount
of globally available decision-relevant information. As a conse-
quence, the differentiation between important, less important, and
unimportant requirements is a bigger challenge for the engaged
stakeholders.

H5: Consensus increases Output Quality. From Hypothesis H3
we assume a positive correlation between the degree of consensus
and the diversity of the group decision. The diversity is an indicator
for a meaningful triage [3] between important, less important, and
unimportant requirements.

H6: Dissent decreases Output Quality. In contrary, dissent leads to
a lower decision diversity and – as a consequence – to less mean-
ingful results of requirements triage. Figure 4: Hypotheses defined to evaluate the I NTELLI R EQ De-
cision Support.
H7: Decision Diversity increases Output Quality. Group decision
diversity is assumed to be a direct indicator for the quality of the
group decision. With this hypothesis we want to analyze the direct
interrelationship between prioritization diversity and the quality of
H3. There is a positive correlation between the group consensus
the resulting software.
and the corresponding decision diversity (correlation 0.523, p <
0.01). More group discussions can lead to a higher level of rele-
H8: Anonymous Preference Elicitation increases Output Quality.
vant knowledge about the decision problem. In the following this
Finally, we want to explicitly analyze whether there exists a rela-
can lead to a development of a deeper understanding of the need of
tionship between the type of preference elicitation and the corre-
requirements triage [3] which leads to a higher degree of decision
sponding output quality.
diversity.

4. STUDY RESULTS H4. Dissent is an inverse function of group consensus – the higher
We analyzed the hypotheses (H1–H8) on the basis of the vari- the dissent, the lower the corresponding decision diversity (corre-
ables introduced in Section 3.1 We used a Mann-Whitney U-test if lation -0.523, p < 0.01). A lower degree of group decision diver-
the examined data set is not normal distributed (H1,H2) and the t- sity (prioritization diversity) can be explained by a lower degree of
test if the data set is normal distributed (H8). The correlations (H3 decision-relevant knowledge.
– H7) are calculated with Pearson correlations (normal distribution)
and with the Spearman’s rank correlations (no normal distribution). H5. Consensus in group decision making increases the output qual-
ity (correlation 0.399, p < 0.01). An overlap in the personal stake-
H1. The degree of group consensus in teams with anonymous pref- holder preferences can be interpreted as an indicator of a common
erence elicitation is significantly higher compared to teams with understanding of the underlying set of requirements. This leads to
non-anonymous preference elicitation (Mann-Whitney U-test, p < a better prioritization and a higher quality of the resulting software
0.05). An explanation model can be the reduction of commitment components.
[2] and a higher probability of discovering hidden profile informa-
tion which improves the overall knowledge level of the team. H6. The hypothesis can be confirmed (correlation -0.399, p < 0.01),
i.e., there is a negative correlation between group dissent and the
H2. Group dissent is an inverse function of group consensus and corresponding output quality.
– as a consequence – teams with non-anonymous preference elici-
tation have a significantly higher dissent (Mann-Whitney U-test, p H7. In our analysis we could detect a positive correlation between
< 0.05). In this context, non-anonymous preference elicitation can group decision diversity (diversity of prioritization) and the cor-
lead to higher commitment with regard to the orginially articulated responding output quality (correlation 0.311, p < 0.01). Decision
preferences. diversity can be seen as an indicator of a reasonable triage pro-
cess and reasonable prioritizations result in higher-quality software
1
We are aware of the fact that dissent is the inverse function of components.
consensus, however, for reasons of understandability we decided to
explicitly include dissent as a decision variable. H8. Groups with anonymous preference elicitation performed sig-

30
nificantly better compared to groups with a non-anonymous prefer-
ence elicitation (independent two-sample t-test, p < 0.05).

5. CONCLUSIONS
Requirements prioritization is an important task in software de-
velopment processes. In this paper we motivated the application
of requirements prioritization and discussed issues related to the
aspect of anonymizing group decision processes in requirements
prioritization. The results of our empirical study clearly show the
advantages of applying anonymized preference elicitation, for ex-
ample in terms of higher-quality software components, and can
be seen as a step towards a more in-depth integration of decision-
oriented research in requirements engineering processes.

6. REFERENCES
[1] A. Aurum and C. Wohlin. The fundamental nature of
requirements engineering activities as a decision-making
process. Information and Software Technology,
45(14):945–954, 2003.
[2] R. Cialdini. The science of persuasion. Scientific American,
284:76–81, 2001.
[3] A. Davis. The art of requirements triage. IEEE Computer,
36(3):42–49, 2003.
[4] A. Felfernig, C. Zehentner, G. Ninaus, H. Grabner,
W. Maaleij, D. Pagano, L. Weninger, and F. Reinfrank.
Group decision support for requirements negotiation. In
Advances in User Modeling, pages 105–116, 2012.
[5] D. Firesmith. Prioritizing requirements. Journal of Object
Technology, 3(8):35–47, 2004.
[6] T. Greitemeyer and S. Schulz-Hardt. Preference-consistent
evaluation of information in the hidden profile paradigm:
Beyond group-level explanations for the dominance of
shared information in group decisions. Journal of Personality
& Social Psychology, 84(2):332–339, 2003.
[7] G. Group. Hype cycle for application development:
Requirements elicitation and simulation. 2011.
[8] H. Hofmann and F. Lehner. Requirements engineering as a
success factor in software projects. IEEE Software,
18(4):58–66, 2001.
[9] A. Mojzisch and S. Schulz-Hardt. Knowing other’s
preferences degrades the quality of group decisions. Journal
of Personality & Social Psychology, 98(5):794–808, 2010.
[10] A. Pinsonneault and N. Heppel. Anonymity in group support
systems research: A new conceptualization, measure, and
contingency framework. Journal of Management
Information Systems, 14:89–108, 1997.
[11] B. Regnell, B. Paech, C. Aurum, C. Wohlin, A. Dutoit, and
J. ochDag. Requirements means decision! In 1st Swedish
Conf. on Software Engineering and Practice (SERP’01),
pages 49–52, Innsbruck, Austria, 2001.
[12] K. Wiegers. First things first: Prioritizing requirements.
Software Development, 1999.
[13] D. Yang, D. Wu, S. Koolmanojwong, A. Brown, and
B. Boehm. Wikiwinwin: A wiki based system for
collaborative requirements negotiation. In HICCS 2008,
page 24, Waikoloa, Big Island, Hawaii, 2008.
[14] P. Zave. Classification of research efforts in requirements
engineering. ACM Computing Surveys, 29(4):315–321, 1997.