Group Decision Support for Requirements Management Processes R. Samer and M. Atas and A. Felfernig and M. Stettinger Graz University of Technology, Graz, Austria email: {rsamer, muesluem.atas, alexander.felfernig, martin.stettinger}@ist.tugraz.at A. Falkner and G. Schenner Siemens AG, Vienna, Austria email: {andreas.a.falkner, gottfried.schenner}@siemens.com Abstract. Requests for proposal (RFP) trigger company-internal requirements. Recommended stakeholders also need to bring deep requirements management (RM) processes in order to assure knowledge about the corresponding item domain in order to provide that offers comply with a given set of customer requirements. precise evaluations of the requirements. As traditional RM approaches require a deep involvement of the S TAKE N ET [14] is an application that supports stakeholder identi- requirements managers of a RM project especially when it comes to fication on the basis of social network analysis. This approach builds assigning suitable stakeholders to requirements, the quality of the a social network on the basis of a set of stakeholders. In this social decisions and the time effort for making correct decisions mainly network, stakeholders are represented by nodes and recommenda- depends on these experts. In this paper, we present a novel stake- tions articulated by the stakeholders are represented by links. On the holder assignment approach that reduces the overall involvement of basis of such social networks, different social network measures are these experts and also limits the uncertainty of overseeing suitable used for the prioritization of the stakeholders. One example of such stakeholders at the same time. The assignment of responsible a measure is betweenness centrality which measures the priority of a stakeholders is represented as a group decision task expressed in the certain stakeholder s based on the ability of this user to play a role as form of a basic configuration problem. The outcome of such a task a broker between separate groups of stakeholders. Castro-Herrera et is a configuration which is represented in terms of an assignment of al. [1] and Mobasher et al. [17] introduce a content-based recommen- responsible stakeholders to corresponding requirements. dation approach where requirements are grouped by using different clustering techniques. Subsequently, stakeholders are recommended and assigned to these groups on the basis of content-based filtering. In this paper, a novel stakeholder assignment approach is introduced. The presented approach, acting as basic configuration service, lets voters evaluate stakeholders based on different criteria/dimensions 1 Introduction and then aggregates their votes to derive possible configurations Group-based configuration is an important application area of Arti- which are then recommended for the final stakeholder assignment ficial Intelligence [3, 4]. It aims to support a group of users in the decision to the requirements manager. In contrast to the aforemen- configuration of complex products or services. In general, when in- tioned stakeholder recommendation approaches where the generated teracting with group-based configurators, group members first artic- recommendations are directly suggested to the requirements man- ulate their preferences, then adapt inconsistent constraints, and fi- ager, the content-based recommendation service presented in this nally, solutions are generated (i.e, reflecting the given configuration). paper only acts as a single artificial voter in addition to some hu- In particular, when interacting with a configurator in the context of man voters. Hence, the stakeholder recommendations (i.e., possible a typical requirements engineering task, each group member (i.e., configurations) shown to the requirements manager are determined stakeholder) has to evaluate each requirement according to differ- based on a combination of votes reflecting opinions of human voters ent dimensions such as priority, effort, and taken risk. However, for as well as votes reflecting opinions of artificial voters. the definition and evaluation of these requirements, first, suitable The major contributions of this paper are the following. First, we stakeholders have to be identified who are responsible for the de- analyze in detail a real-world scenario of a typical bid project. Sec- velopment of these requirements. In addition, an early involvement ond, we show an approach to identify relevant stakeholders for spe- of these stakeholders in the project is essential for the success of a cific requirements and thus generate a global assignment of stake- project [5, 6, 13, 18]. This is because a low involvement of stakehold- holders to requirements. The remainder of this paper is organized as ers in a project can lead to project failure. Project failures are often follows. In Section 2, we describe a typical application scenario of caused by missing or wrong assignments of stakeholders to require- a bid project applied in an industrial context and provide a practical ments in early phases of the requirements engineering process [14]. view of a traditional requirements management process commonly Stakeholder recommendations can help to identify persons who are used for planning large industry projects. Additionally, a novel so- capable of providing a complete analysis and description of software phisticated approach is explained which further improves and ex- tends the traditional approach by considering group decision sup- Domain Stakeholder port techniques. Section 3 discusses some potential issues and several PM project manager factors this approach depends on. Subsequently, Section 4 explains SA system architect RM requirements manager the implementation of such an approach from a technical viewpoint. RAMS reliability, availability, maintainability, and safety Finally, Section 5 concludes with a brief recap of this paper and S(ignal) engineering department for railway signals presents some ideas for future work. PS engineering department for power supply TVD department for track vacancy detection ETCS department for European Train Control System Test quality management department 2 Application Scenario Supplier1 external supplier, subcontractor Whenever an organizational unit of a large company (e.g., Siemens) Table 1: Examples of domain identifiers for rail automation decides to bid for a Request for Proposal (RFP), a new bid project for that proposal is initiated and the necessary stakeholders of the bid trary comment (called prose). In general, large infrastructure projects project are identified. RFPs for technical systems usually consist of a may contain more than 10,000 (potential) requirements. set of PDF or Microsoft Word documents which describe all require- Each (actual) requirement must be assessed by at least one stake- ments for the requested system covering technical, financial, legal, holder. The requirements manager has to figure out which stakehold- etc. aspects. Examples of stakeholders can be project managers, sys- ers are appropriate for which requirements and needs to assign them tem architects, requirements managers, quality management depart- accordingly. However, other stakeholders may improve such initial ments, legal departments, engineering departments relevant for the assignments later during the assessment phase. The RM tool notifies bid, and potential external suppliers. all assigned stakeholders via e-mail to assess the requirements they Within the context of a bid project, a requirements management are assigned to. (RM) process is initiated at the beginning. The purpose of this pro- Table 2 shows an example of an initial assignment done by the re- cess is to assure that no requirement of the RFP has been overlooked. quirements manager (RM). In this table, each row corresponds to a It involves the extraction of all the requirements contained in the RFP requirement and each column refers to a stakeholder. Each cell rep- documents. The identified requirements must be assessed by the rel- resents a single decision (of a stakeholder) for a stakeholder assign- evant stakeholders. This means that requirements concerning con- ment (to a requirement). At the beginning, only the RM proposes tracts must be assessed by the stakeholder(s) of the legal department, assignments of potential stakeholders to requirements based on the technical requirements must be assessed by the affected engineering manager’s expertise and knowledge. For example, the assignment of department, etc. The assessment may involve statements about vari- {S, P M } to the requirement R5 in the RM column indicates that ous criteria such as compliance, risks, approaches, etc. These state- R5 has been initially assigned to the signal department (S) and to ments are interpreted as evaluation dimensions in the remainder of the project management department (PM) by the requirements man- this paper. At the end, each requirement of the RFP must have been ager (RM). As only the RM makes assignments in this initialization assessed by at least one appropriate stakeholder. phase, the values of all other columns remain empty (i.e., are filled with the ”-” label) until the assessment phase. 2.1 Traditional RM Process Req RM PM RAMS S(ignal) R1 {PM} - - - The traditional requirements management process can be best ex- R2 {PM} - - - plained with an example. In the following, we describe a simplified R3 {S} - - - R4 {S} - - - example of a traditional RM process in a rail automation context R5 {S, PM} - - - based on a conventional RM tool such as IBM DOORS. ... At the beginning, the requirements manager of the bid project cre- ates a new project in the RM tool. After that, the necessary stake- Table 2: Initial assignment of stakeholders to requirements done by holders for the current bid project are defined. In this context, stake- requirements manager (RM). The dash symbol (”-”) indicates that holders do not necessarily correspond to persons but correspond to the other stakeholders have not made a decision yet. roles which are uniquely identified with a unique string (called Do- main). These string-based identifiers are unique within the organiza- Next, in the assessment phase, the affected stakeholders take a tion. Furthermore, the RM tool supports the mapping of existing roles look at each of their assigned requirements in the RM tool and can (i.e., domain identifiers) to concrete persons within the bid project. either accept the requirement and assess it or they can veto the pro- This way, responsible persons are assigned to roles based on their posed assignment. Additionally, they can also propose an alternative skills and domain knowledge. stakeholder for the requirement or suggest (although rarely) an ad- Table 1 presents some examples of domain identifiers which oc- ditional stakeholder for the requirement. For the remainder of this cur in the context of rail automation. For such large bid projects usu- paper, this process is hereinafter referred to as assignment feedback. ally more than 50 different domains are defined with the RM tool. After that, the requirements manager can either accept the veto and However, in practice, most projects only use 20 different domains on assign the requirement to a different stakeholder or decline the veto average. and reassign the stakeholder to the requirement. As a next step, the requirements manager imports all the relevant Table 3 shows an intermediate state during the assignment phase documents of the RFP into the project by using the RM tool. The RM which demonstrates examples of assignment feedback given by the tool automatically converts each paragraph of the documents into a stakeholders P M and S(ignal): (potential) requirement whilst the structure of the documents is pre- served. The requirement manager then classifies the (potential) re- quirements in the project as either an actual requirement or as an arbi- Req RM PM RAMS S(ignal) aggregation functions exist - for further information regarding pref- R1 {PM} {PM} - - erence aggregation functions we refer to [2, 15]. To limit the scope R2 {PM} {RAMS} - - of this paper, we assume that the group decision service is a simple R3 {S} - - {S, RAMS} R4 {S} - - {} group recommender using basic aggregation strategies. R5 {S, PM} {S, PM} - {S, PM} The votes of the artificial stakeholders (i.e., bots) are generated ... by using appropriate content-based recommendation algorithms (see Section 4). This way, the group decision service allows to replace the Table 3: State of assignment during assessment phase traditional mainly manual stakeholder assignment process (see Sec- tion 2.1) with a semi-automatic process. As a key difference to the • Requirement R1 has been accepted by P M traditional approach, the group decision service automatically aggre- • Requirement R2 has been vetoed by P M and RAM S has been gates the decisions of all voters and thereby allows the smart incorpo- proposed by P M as alternative stakeholder ration of additional (automatic) voters, i.e., intelligent recommenda- • Requirement R3 has been accepted by S(ignal), but RAM S has tion services for stakeholder assignments. From an abstract point of been proposed by S(ignal) as an additional stakeholder view, the process can be interpreted as a basic configuration process. • Requirement R4 has been vetoed by S(ignal) Like in the traditional RM process (see Section 2.1), the outcome • Requirement R5 has been accepted by all proposed stakeholders of this process represents a consistent assignment of stakeholders to requirements they are responsible for. It is important to point out the fact that in the traditional scenario, Table 5 illustrates a possible initial state in the presence of a group it is always the main responsibility of the requirements manager decision service (GDS) and a stakeholder assignment recommen- to resolve potential conflicts. Typically, this usually involves some dation service (denoted as RS1). In sharp contrast to the assign- personal discussions with the involved stakeholders and some final ments made by other stakeholders, the recommendation service does decisions made by the requirement manager. These final decisions not provide a binary decision for every stakeholder but a confidence then assure a consistent assignment of all requirements to responsi- value which lies in the range between 1 and 10, whereby a higher ble stakeholders. Table 4 presents such a final state where all conflicts number corresponds to more confidence and a lower number corre- have been resolved. sponds to a lower level of confidence. Req RM PM RAMS S(ignal) The column for the GDS shows the result of the group de- R1 {PM} {PM} - - cision service for each requirement, i.e., the aggregated decision R2 {RAMS} {RAMS} {RAMS} - of all voters (including humans and bots/algorithms). Note that a R3 {S, RAMS} - {S,RAMS} {S, RAMS} clear benefit of the group decision service is that some requirements R4 {S} {S} - {S} R5 {S, PM} {S, PM} - {S, PM} can already be assessed by the assigned stakeholders, even though ... they have not yet been proposed/assigned by the requirements man- ager. In other words, stakeholders are automatically proposed by the Table 4: Final state after assessment phase. Consistent assignment of bots/algorithms based on their skills in the initial phase and can al- stakeholders to requirements. ready evaluate their assignment to the requirements. Hence, much assignment effort is taken away in the initial phase from the time- The requirements manager periodically reminds the assigned pressured requirements managers and the initial phase can be signif- stakeholders about their unassessed requirements. This process is re- icantly speeded up. Moreover, it is necessary to point out that the peated until all requirements have been assessed and the assessment stakeholders GDS (perform aggregation) and RM (perform final phase is finished. Thus, the assignment of stakeholders can be con- decision) can be considered to have a special role in this evaluation sidered as a manual configuration process. The outcome of this pro- process, whereas all other stakeholders only occur as voters in the cess is a configuration in terms of a consistent assignment of stake- process. Consequently, the major responsibility/task of a RM in this holders to requirements they are responsible for. In our current im- process is to review the decision suggested by the GDS and to per- plementation, the overall goal is to achieve consensus regarding the form the final decision about the assignment of the stakeholders to stakeholder assignment. Future versions of our system include fur- the requirements. ther constraints that have to be taken into account in task allocation tasks as discussed in this paper. 3 Potential Issues of Group Decision Support 2.2 RM Process with Group Decision Support The exact behavior of the new system presented in Section 2.2 will depend on various factors. Examples of such factors include the ag- The main idea of our novel requirements management approach is gregation strategy used by the group decision service to aggregate to introduce additional stakeholder votes made by artificial stake- the votes (e.g., majority, average, etc.), the individual weight of the holders (called bots). Additionally, the bots automatically propose voters (e.g., “deciders”/experts count higher than normal stakehold- stakeholders in the initial phase of the RM process. Furthermore, an ers), and the confidence/trust users have in different recommendation intelligent group decision service is included in the RM tool to au- algorithms. tomatically aggregate all votes given by human stakeholders as well Furthermore, the question arises how conflicting decisions (for ex- as artificial stakeholders. On a technical level, such a group decision ample, stakeholder A assigns stakeholder B and B assigns A) can be service represents a group recommender system which generates rec- resolved or supportive advice to manually resolve such conflicts can ommendations based on aggregated votes given by group members be given to the voters by the system. Also, inconsistencies and con- of a group (i.e., the stakeholders) [2]. Basically, there exist different tradictions may occur in the evaluation of stakeholders between the strategies on how to aggregate votes of group members [8] such as voters. These voters can be other stakeholders and artificial stake- majority, average, least-misery, etc. In addition, more sophisticated holders. In particular, for artificial stakeholders textual explanations Req GDS RS1 RM PM RAMS S(ignal) R1 {PM} {PM:9} {PM} - - - R2 {RAMS} {RAMS:8, PM:5} - - - - R3 {S} {S:8, RAMS:6} - - - - R4 {S} {S:5} - - - - R5 {S} {S:6} {S,PM} - - - ... Table 5: State of assignment with group decision service (GDS) and stakeholder recommendation service (RS1). The recommendation service provides a confidence value which lies in the range between 1 and 10. can be presented to the group of voters being in conflict. Such textual or numbers) that are most probably irrelevant to be used as keywords explanations can then express the concrete reason and arguments for are removed. Finally, the remaining tokens of each former require- the votes provided by the artificial stakeholders. ment (which was assigned to the stakeholder) are merged together Moreover, the prediction quality (i.e., performance) of the artifi- into a single user profile. cial stakeholders (i.e., the recommender systems) plays a major role By applying the same procedure to new requirements, keywords in the process. In particular, the generated recommendations should for new requirements are extracted as well. Given the keywords of a be evaluated and examined with respect to completeness. In terms new requirement and the user profiles of the individual stakeholders, of common information retrieval measures (such as precision and re- a similarity between a new requirement and a stakeholder is calcu- call), this would, for example, mean that more emphasis should be lated for every stakeholder provided that the stakeholder has been as- given to the recall of the results rather than the precision achieved by signed to an (already completed) requirement in the past. Formula 1 the recommender. In addition to that, an appropriate recommenda- shows the Dice coefficient formula [9] which is a variation of the Jac- tion algorithm should also be capable of giving negative indication card coefficient and used to compute the similarity between a stake- by telling the RM which stakeholders are definitely not suitable to holder and a requirement. The similarity is measured by comparing be assigned to a requirement at all. Such a negative indication can the overlap of the keywords of the stakeholder’s user profile (denoted be shown as, e.g., RAMS:0. Finally, another important aspect would as Ua ) and the relevant keywords of the respective requirement (de- be to take the availability of stakeholders into account before they noted as rx ) with the total number of keywords appearing in Ua as get finally assigned to a requirement. This adds another complexity well as rx . dimension to the underlying basic configuration problem. 2 ∗ |keywords(Ua ) ∩ keywords(rx )| 4 Group Decision Support for Bidding Processes sim(Ua , rx ) = (1) |keywords(Ua )| + |keywords(rx )| In this section, a slightly modified version of the aforementioned Stakeholders who are most similar to a given requirement are sug- RM process based on Group Decision Support (see Section 2.2) gested by the content-based recommender to the RM. This way, the is described. The description explains the technical implementation initial phase can be speeded up and the chance of overseeing suitable of this process provided by the requirements engineering platform stakeholders for requirements at this early stage of the process, is de- O PEN R EQ MVP1 which is developed within the scope of the Open- creased. In the next step, the O PEN R EQ MVP system shows a list of Req EU Horizon 2020 research project. At the current stage, the im- the initially assigned stakeholders for each requirement. Stakehold- plementation is already in use, however, still ongoing and ready to ers who are assigned to a requirement can either accept or reject their be further enriched with additional features. The remainder of this assignment. In addition, the assignments of the stakeholders for the section describes the current status of the existing implementation. requirement can be evaluated by all stakeholders. In the initial phase, the requirements manager (RM) is asked by the This evaluation of a stakeholder-assignment is done based on the system to propose suitable stakeholders for each requirement. As al- criteria Appropriateness and Availability (see Figure 1). Both criteria ready described in Section 2.2, a content-based recommender system are interpreted as evaluation dimensions and stakeholders are evalu- (RS1) helps the RM to find stakeholders based on keywords extracted ated based on both dimensions. Furthermore, an assigned stakeholder from former requirements those stakeholders have solved. Thereby, can also propose the assignment of further stakeholders to the re- on an abstract level, the automated stakeholder-recommendation al- quirement. These newly assigned stakeholders can then be evaluated gorithm (of RS1) can be interpreted as a text classification task [7] again. After a new vote has been given, the group decision service where the recommendation algorithm exploits several Natural Lan- (GDS) is triggered to compute a utility value for the rated stake- guage Processing [19, 20] techniques in order to correctly classify holder. Formula 2 shows the calculation of the utility value of an stakeholders suitable for a given requirement. evaluated stakeholder s, whereas D is the set containing both dimen- The algorithm automatically extracts relevant keywords from the sions, i.e., D = {Appropriateness, Availability}. title and description text of all former requirements which a stake- holder was assigned to, in order to build a user profile for the re- P P eval(s,r,d,t)·weight(d) d∈D spective stakeholder. First, the title and description text is cleaned by t∈T P weight(d) d∈D removing special characters (such as “.”, “,”, “;”, “#”, etc.). Next, the utility(s, r) = (2) text is split into tokens (which, basically, represent the words in the |T | text) and stop words such as prepositions (e.g., “in”, “on”, “at”, etc.) or articles (e.g., “the”, “a”, “an”) are removed. After applying Part- of-speech tagging, tokens/words of classes (such as verbs, adjectives, The formula describes the stakeholder s to be voted by other stake- holders, whereby T represents the set of stakeholders t ∈ T who 1 OpenReq MVP: http://openreq.ist.tugraz.at evaluated s. More formally expressed, T is a set which contains the knowledge concerning the details of a requirement can easily dele- gate their votes to more well-informed and experienced experts. With respect to conflicting decisions (see Section 3), future work should also include mechanisms to automatically resolve such con- flicts or mechanisms which provide supportive advice to the voters, showing how they can manually resolve such conflicts. Furthermore, the configuration approach can be enriched with further constraints taking resource management aspects of stakeholders into considera- tion, in order to optimize the overall allocation of human resources in release planning. Finally, there is also still plenty of room for improvement regard- ing the extraction of keywords used by the discussed content-based recommender system (i.e., artificial stakeholder). For example, a more descriptive and characteristic representation of the key- words can be obtained by using more sophisticated content-based approaches such as Latent Semantic Analysis (LSA) [11, 16] or Figure 1: Evaluation of stakeholders in O PEN R EQ MVP. Each word2vec algorithms [12, 16]. stakeholder-assignment is evaluated by two evaluation dimensions (appropriateness and availability). The utility value of an evaluated stakeholder is calculated by using Formula 2. Acknowledgment stakeholders (including s) who evaluated stakeholder s, i.e., T ⊆ S. Furthermore, the O PEN R EQ MVP platform allows the requirements The work presented in this paper has been conducted within the manager to define different importance levels for both dimensions. scope of the Horizon 2020 project O PEN R EQ (732463). In Formula 2, the importance of a dimension d ∈ D is expressed by the function weight(d). Moreover, eval(s, r, d, t) refers to the REFERENCES dimension-specific rating given by stakeholder t for stakeholder s for the requirement r. Finally, the result of utility(s, r) represents [1] Carlos Castro-Herrera, Chuan Duan, Jane Cleland-Huang, and the aggregated utility of a stakeholder s for requirement r. Bamshad Mobasher, ‘Using data mining and recommender systems to Once all assignments have been evaluated by a sufficient number facilitate large-scale, open, and inclusive requirements elicitation pro- cesses’, 165–168, (09 2008). of stakeholders, a stable state of the assignment utilities is achieved. [2] A. Felfernig, L. Boratto, M. Stettinger, and M. Tkalcic, Group Recom- The utility values are then used as main feedback source for the re- mender Systems – An Introduction, Springer, 2018. quirements manager to make the final decision about which stake- [3] Alexander Felfernig, M Atas, T Tran, and Martin Stettinger, ‘Towards holder(s) should be assigned to the requirement. group-based configuration’, in International Workshop on Configura- tion 2016 (ConfWS16), pp. 69–72, (2016). [4] Alexander Felfernig, Lothar Hotz, Claire Bagley, and Juha Tiihonen, Knowledge-based Configuration: From Research to Business Cases, 5 Conclusion and Future Work Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1 edn., 2014. Conclusion. In this paper, we discussed common application scenar- [5] Alexander Felfernig, Martin Stettinger, Andreas Falkner, Muesluem ios of requirements engineering in the context of industry projects. Atas, Xavier Franch, and Christina Palomares, ‘Openreq: Recom- These scenarios range from traditional requirements management mender systems in requirements engineering’, pp. 1–4, (10 2017). [6] Hubert F. Hofmann and Franz Lehner, ‘Requirements engineering as a processes where the assignment process of stakeholders is solely success factor in software projects’, IEEE software, 18(4), 58, (2001). controlled by the requirements manager, to more sophisticated auto- [7] Emmanouil Ikonomakis, Sotiris Kotsiantis, and V Tampakas, ‘Text mated approaches where the involvement of the requirements man- classification using machine learning techniques’, 4, 966–974, (08 ager is reduced to a minimum. The latter represents a basic config- 2005). uration service which includes artificial stakeholders as additional [8] Anthony Jameson and Barry Smyth, ‘The adaptive web’, chapter Rec- ommendation to Groups, 596–627, Springer-Verlag, Berlin, Heidel- voters and a group decision support system as a vote aggregation berg, (2007). component in the evaluation of stakeholder assignments to require- [9] Dietmar Jannach, Markus Zanker, Alexander Felfernig, and Gerhard ments. On the basis of this scenario we showed how these two com- Friedrich, Recommender Systems: An Introduction, Cambridge Univer- ponents can be applied in order to improve the requirements manage- sity Press, New York, NY, USA, 1st edn., 2010. [10] Anson Kahng, Simon Mackenzie, and Ariel D. Procaccia, ‘Liquid ment process such that the overall effort and the chance of overseeing democracy: An algorithmic perspective’, in AAAI, (2018). stakeholders suitable for requirements can be reduced for the time- [11] Thomas K. Landauer, Peter W. Foltz, and Darrell Laham, ‘An introduc- pressured requirements managers. tion to latent semantic analysis’, (1998). Future Work. As bidding processes can be seen as repetitive [12] Jey Han Lau and Timothy Baldwin, ‘An empirical evaluation of processes, mechanisms which are capable of learning stakeholder doc2vec with practical insights into document embedding generation’, CoRR, abs/1607.05368, (2016). weights and taking individual expertise levels of stakeholders into [13] Dean Leffingwell, ‘Calculating your return on investment from more ef- account can be considered as potential ideas regarding future work. fective requirements management’, American Programmer, 10(4), 13– Moreover, the set of existing evaluation dimensions can be further 16, (1997). extended such that more fine-grained control is given to the evalu- [14] S.L. Lim, D. Quercia, and A. Finkelstein, ‘Stakenet: Using social net- works to analyse the stakeholders of large-scale software projects’, in ation process as well as to the group decision service. Additionally, Proceedings of the 32Nd ACM/IEEE International Conference on Soft- the concept of liquid democracy can be integrated into the evalua- ware Engineering - Volume 1, ICSE ’10, pp. 295–304, New York, NY, tion process [10]. This way, stakeholders who do not have sufficient USA, (2010). ACM. [15] J. Masthoff, ‘Group recommender systems’, Recommender Systems Handbook, 677–702, (2011). [16] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jef- frey Dean, ‘Distributed representations of words and phrases and their compositionality’, in Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, pp. 3111–3119, USA, (2013). Curran Associates Inc. [17] B. Mobasher and J.Clehand-Huang, ‘Recommender systems in require- ment engineering’, 81–89, (2011). [18] Bamshad Mobasher and Jane Cleland-Huang, ‘Recommender systems in requirements engineering’, AI magazine, 32(3), 81–89, (2011). [19] Kevin Ryan, ‘The role of natural language in requirements engineer- ing’, in [1993] Proceedings of the IEEE International Symposium on Requirements Engineering, pp. 240–242, (Jan 1993). [20] J. Winkler and A. Vogelsang, ‘Automatic classification of requirements based on convolutional neural networks’, in 2016 IEEE 24th Interna- tional Requirements Engineering Conference Workshops(REW), vol- ume 00, pp. 39–45, (Sept. 2016).