Context: Software organizations working in a market-driven environment have to select requirements from a large pool to be prioritized and put into backlogs for the development organization. Objective: This paper proposes an approach based on study selection in systematic literature reviews and translates the concept to requirements engineering. The rational for doing so is that the selection processes used there have been e↵ective (selecting and finding relevant papers) and ecient (possible to use for a high number of studies, in some cases 10,000 research contributions had to be evaluated). Method: This paper can be classified as a solution proposal, and utilizes hypothetical examples to explain and argue for the method design decisions. Results: The process proposed consists of three main phases, namely establish selection criteria, evaluate selection criteria, and apply selection. On a more fine-grained level, nine activities are specified. Conclusion: Given that the process has been e↵ective and ecient in a similar context, our proposition to be evaluated in future research contributions is that the process leads to e↵ective and ecient decision making in requirements selection.
Requirements selection is a challenging task, as a very high number of
requirements need to be processed. In particular, in a market-driven context this is a
challenge where the requirements may come from market units, and also from
research and development done within the organization. This phenomenon is
referred to as requirements overload [
Copyright © 2015 by the authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
Interestingly, a similar problem to the requirements selection is faced by
researchers conducting systematic literature reviews. In systematic literature
reviews [
The problem translates very well to requirements selection as we have a high
number of requirements, we need to define clear inclusion and exclusion criteria
to not miss any essential requirements, and have to come to an objective
decision in a relative short amount of time for the assessment to be feasible. Thus, in
the context of systematic literature reviews, procedures for study selection have
been identified, and also evaluated [
The remainder of the paper is structured as follows: Section 2 presents the related work. The solution proposal is described in Section 3, followed by a discussion in Section 4. Section 5 concludes the paper.
The focus of this paper is on requirements selection, which is di↵erent from
requirements prioritization. In requirements prioritization a comparative
analysis of requirements should be made, which is a time-intensive activity [
In requirements selection (in particular screening), the goal is to immediately
remove requirements from the initial set that are not relevant from a system and
stakeholder perspective. That is, rather than deciding on the priority the decision
is on whether to include or exclude the requirement (referred to as triage, a term
inspired from the medical domain [
The related work focuses on introducing the concept of requirements triage, and on existing requirements selection and screening approaches. Further details (e.g. specific criteria for selection) will be revisited in the context of the proposed selection process, where applicable.
Requirements triage: As the pool of resources working with the
requirements (e.g. in refining, designing, testing, and implementing them) is limited,
the main goal of triage is to select the most relevant requirements to focus on
[
Simmons [
Also e↵orts have been undertaken to automate the requirements
prioritization process by clustering requirements [
Processes: Khurum et al. [
In selection processes, often multiple stakeholders are involved, which has not
been incorporated in MERTS. An important contribution to mention here has
been provided by Regnell et al. [
Our proposal complements triage processes by prescribing how to
systematically involve multiple stakeholders in the selection to objectify the selection
criteria and with that potentially increase the e↵ectiveness on making the “right
decision” when selecting requirements. Following a systematic process has shown
to be beneficial in identifying more relevant studies [
We first describe the selection process conducted in the context of literature studies. Thereafter, we present how the process translates to requirements selection.
The reliability of a systematic literature review relies heavily on the selection
process in identifying relevant studies [
Starting with research questions, study selection criteria are formulated in
a way that they can be objectively assessed. As the research question is
providing the scope of the study, the selection criteria are based on the question.
The selection criteria are reviewed by other reviewers for clarity, objectivity of
assessment and alignment with the research questions. The “think-aloud”
approach of applying the criteria is used on a few papers to externalize a reviewer’s
understanding of the criteria. This helps to highlight misunderstandings or
ambiguities in the criteria. Based on the review and “think-aloud” application of
the selection criteria, it may need to be updated. An example of typical inclusion
and exclusion criteria selected from [
Example inclusion criteria: Overall, eight di↵erent inclusion criteria were defined, for example: – The article is peer reviewed – The article discusses a model/framework of strategic release planning or post release planning analysis of strategic release planning. – The article will be included if it discusses a validation of existing model of strategic release planning or post release analysis.
Three exclusion criteria were included, two examples are: – Articles related to only operational release planning will be excluded. – Articles related to re-planning of a release on operational level will be excluded.
In the example, it is noteworthy that some criteria are easier to evaluate (e.g. peer review), while others require more thought (e.g. assessing whether a model is presented focused on strategic release planning and not operational release planning).
Following this, the reviewers independently apply the selection criteria on a randomly selected subset of papers. This “pilot selection” is a confidence building step to assess the wider applicability of the criteria. For example, we may find out that in a majority of the papers, the information we seek to assess a paper’s relevance is not available in the title and abstract. Hence, the assessment could not be made based on that information in title and abstract of a paper alone, which is usually the first step before reading the full-text.
Furthermore, using inter-rater agreement statistics like Kappa [
Now, we are ready to perform selection on the complete set of the papers.
The recommended practice is to have at least two reviewers assessing the
relevance of each paper [
Describe strategy/goal for the product
Specify RQs Acceptable level of agreement
Yes Perform Selection the typically high number of papers they have to sift through). In systematic literature reviews, the reviewers often use three categories to place each paper in terms of its relevance to the research questions: include (i.e. relevant), exclude (iD.oec.umiernrtelevCaonnttro)l or uncertain (i.e. cannot make a decision based on the available inPfroocremss atioInnfo)rm.ation
stEepven with all the measures that were taken to attain high objectivity in the criteria (which will likely be visible with the high-level of inter-rater agreement between the reviewers), there will still be mismatches between individual reviewer’s judgments. The quality of reporting of the paper, quality of our criteria and also mistakes made by the reviewers will be the likely reasons for these mismatches.
In systematic literature reviews, various rules have been suggested and their
implications on the results have been evaluated [
When approaching the study selection process from the perspective of
requirements selection, the first obvious di↵erence is in the input to the process. In
study selection, it is the research questions that communicate the aim and goal
of the study and guide the formulation and application of the selection criteria.
In case of requirements selection such guidance would come from the product
vision, goals, and strategies to reach the goals. Specifying product strategy and
goals (cf. [
Apart from this di↵erence of input to the process, we argue, that there is
no modification required in activities to be conducted and their order [
Establish criteria Like the study selection process (as exemplified in Section 3.1), when specifying the selection criteria for requirements it is important to specify both inclusion and exclusion criteria. For example, questions detailing elements of the criteria could be like the following: – Include a requirement when it fits customer goals. – Exclude it if it is a↵ecting a large part of the architecture, i.e. requires a relatively large e↵ort.
It is important to highlight that like systematic literature reviews there are no
fixed criteria applicable to all, and that individual companies need to investigate
and set the criteria that are suitable for their context. Some useful input however
can be taken in the form of attributes to be considered,criteria for prioritization,
and guidance on how to formulate criteria from: Khurum et al. [
The review of the criteria can be done in a meeting and employing techniques like “think-alound” application (where one practitioner while applying the criteria also verbalizes his understanding of it). These steps help ensure alignment with goals, clarity of the formulation, and facilitate to align various perspectives (if practitioners have di↵erent roles in the product development e.g. product manager and lead architect).
Thereafter, a small scale dynamic evaluation of the criteria is performed by doing a pilot selection on a subset of requirements. Using inter-rater statistics level of mismatches between individual judgments can be evaluated and thus highlighting the need to further clarify the criteria or in turn the product goals from which these were derived. It is important to emphasize that this is an iterative process and may involve a few iterations before we have sucient confidence in the criteria, and alignment between practitioners about the goals, criteria and their interpretation.
None of the existing selection procedures discussed in Section 2 provide such a systematic and structured approach that is geared to align various perspectives early on in the software requirements selection process and also reduces bias in selection.
Perform selection In these steps, the actual selection is performed. Like the
study selection process it will be beneficial to have at least two practitioners
judge each requirement. We argue that these practitioners should be the product
owner and the lead architect of the product to somewhat represent both business
and technical perspectives, this is very important as often their views are not
aligned and can result in inclusion of unnecessary requirements [
The process in study selection is relatively easier to evaluate in terms of how many irrelevant papers got through the selection process and were later rejected in the study. Similar analysis could be done where the number of requirements that are later closed could be an indication of the quality of the selection process. Such requirements could also be an indication of unclear strategy or a change in it, but it is worth starting the analysis from this measure.
The number of relevant studies that are missed by the process can only be identified if the study is repeated or if another set of researchers independently perform the study. Similarly, a reflection on whether some relevant requirements were missed in the selection process will only be possible after the product has been on the market and through analysis of market success of features in competitors products. This could still however be a reflection on the product strategy and less on the selection process.
As highlighted earlier, some criteria are easier to judge than others. Hence, before checking all criteria, the objective and easy to apply ones should be checked first. This reduces the number of requirements to be checked for the criteria that are harder to evaluate. For example, if a requirement fits to the product strategy may be easier to check rather than determining the maximum e↵ort a requirement should be allowed to consume before going into the backlog of the development organization.
The results of selection with two reviewers and three categories as suggested by Davis, can be visualized in a way similar to as shown in Table 1. Based on this table, we can now define decision rules how we deal with the requirements. We can, for example, say that requirements were both practitioners stated “Must satisfy” go into the backlog and are further refined and implemented. In systematic literature review, a category of “uncertain” has been introduced indicating that no decision could be made without further discussion and reflection. In general, if there are disagreements (see D in Table 1) discussions should take place, such as adding additional practitioners from marketing, product development, etc.
The strength of the proposed process is however in ensuring that the quality of product strategy/vision/goal is translated into only appropriate requirements getting selected through it. The success on the market may still be as good as the market analysis and business strategy, however, the operationalization through this process of the product vision increases the chances of alignment of development with it.
We first discuss the potential impact on the e↵ectiveness of the requirements selection process. The goal of increasing the e↵ectiveness is fulfilled if: – the likelihood of excluding relevant requirements is reduced. – the likelihood of including irrelevant requirements is reduced.
As we observed from utilizing study selection process in a variety of
systematic reviews [
Making criteria explicit: The first step is to make the criteria explicit through review, using “think-aloud” and pilot selection. When there are disagreements this leads to a discussion and reformulation of criteria. It may also lead to the addition of further items to the criteria, or may even indicate that the goals and strategies are not formulated well enough. As a consequence, the criteria are not concrete enough so that multiple practitioners could not come to the same conclusion while deciding on the same requirement.
The importance of checking alignment: A development organization has to be aligned in its goals and understanding of what the system should ideally deliver. Barney et al. investigated the alignment of di↵erent roles with regard to their priorities on functionalities, project metrics, and quality. They found misalignment in the organization. The decisions in Table 1 make this alignment explicit and also traceable, as the decisions of di↵erent people should be recorded for each requirement. Furthermore, the alignment can be measured using Kappa statistics. Only with the misalignment being explicit, organizations can take concrete actions to achieve a higher degree of alignment.
Identify the need to improve requirements: Unclear and ambiguous requirements may go into the “uncertain” category and then specific actions have to be defined for that category (e.g. discussion, refine the requirement). If a very high number of requirements always ends up in the uncertain category, then there would be a clear need to investigate this. The reason may be the lack of clarity of the criteria, but also the formulation of the requirement that may be too abstract and does not provide concrete enough information. This situation can also be found in literature studies, where the abstract does not specify important information to make a decision.
Eciency may also be improved, which is very important when making decisions on a very high number of items. As mentioned earlier, in literature studies it is common to make decisions on 1000 and more studies. For this, we provide the following suggestions.
Use multiple level of criteria: Not all criteria are equally dicult to evaluate. Hence, if a criterion is very easy and objective to apply it should be done first, thereby the total number of requirements to be looked at may be reduced. For example, basic and advanced criteria could be defined.
Use multiple levels of inclusiveness: Based on Table 1 di↵erent strategies could be followed. That is, we could define decision rules which requirements to take to the next step (e.g. detailed analysis of the requirements). The most inclusive strategy would be to take everything to the next step (i.e. A+B+C+D+E+F). Though, as we found in the context of literature studies the most inclusive strategy created overhead, while the overhead did not justify the value added to our results. Hence, finding good decision rules giving a good return (e↵ectiveness) on investment (time spent to investigate requirements) may lead to eciency gains and hence the ability to handle a high number of requirements.
Learning: When using the criteria identified for selection early on, time has to be invested to objectify criteria and spread them in the organization. As time goes on and decisions were made and compared, people in the organization will most likely familiarize with the criteria, which may decrease the time required for decision making.
In order to substantiate the propositions stated above, controlled experiments and industry evaluations of the process are the next steps to take in this research.
Controlled experiments: Khurum et al. [
Industry evaluation: Before conducting a live evaluation of the process, it should first be presented to the industry to gather feedback on the process. Thereafter, the process should be utilized by the practitioners, and feedback should be gathered (e.g. through interviews and questionnaires). It may also be important to participate in meetings taking place in relation to the process. This is useful to see how practitioners achieve agreements and decide to reformulate the inclusion and exclusion criteria, or even their product goals and strategies. Within the scope of the process, it is important to evaluate whether e↵ective selection has been achieved with respect to the goals and strategies. Outside the process, the satisfaction of the customers has to be looked at, even though this leads to other confounding factors playing a role as well.
The proposed process provides a systematic way for requirements selection. The process is inspired by the selection of empirical studies in systematic literature reviews. The challenges in both processes are very similar, i.e. making an e↵ective selection with respect to specified goals/research questions for a very high number of decision items (requirements/studies) This process can potentially contribute in the following ways: – provides a transparent process where a conscious traceable decision is made about relevance of requirements – brings multiple perspectives in the selection process, promotes communication and highlights the need for alignment – reduces the likelihood of excluding relevant requirements – reduces the likelihood of including irrelevant requirements
We have suggested concrete studies on how to proceed with the evaluation of the proposed process using controlled experiments and industrial case studies.