Analyzing Trade-offs for Sustainability Requirements: A Decision-Making Process

Abstract

-Software engineers have a responsibility to add sustainability as a consideration while developing a software system because of the critical role of software-intensive systems in society. When sustainability composed of strongly dependent sustainable development dimensions is considered, a possibility to have conflicting requirements cannot be avoided. A decision-making process for solving the conflict with sustainability consideration should have the capability to address multiple requirements and objectives. Unfortunately, there is an inadequate study on sustainability trade-offs assessment. One of the challenges in sustainability engineering is the abstractness and complexity nature of sustainability. The lack of a comprehensive understanding of the impact of each decision in the sustainability dimension results in an unreliable outcome. By utilizing the Analytical Hierarchy Process, we propose a multi-criteria sustainability trade-offs analysis for the decision-making process. We aim to analyze the trade-offs between conflicting sustainability requirements based on the domain knowledge represented as quantified importance of stakeholders and domain-specific sustainability criteria. This approach helps to decide which alternative can remove the conflicts and minimize negatives impacts in the sustainability dimension.

I. INTRODUCTION

In the current society, the advancements of softwareintensive systems lead to a strong dependency between human daily life and technology. Unfortunately, the fast growth of the software-intensive system was not followed by an adequate analysis of the software systems’ impact on society and environments. Global E-waste monitor reported that the global quantity of e-waste generation in 2016 was around 44,7 million metric tonnes (Mt)[ 1 ]. Another frequently mentioned problem is the accumulating technical debt of the software system. Software Engineering Institute (SEI) mentioned that technical debt increases the total burden of investment due to high maintenance cost [ 2 ]. Those problems raise because there is a lack of long-term thinking in the current software engineering (SE) practice. Moreover, engineers mostly focus on the technical aspect of a software system.

Therefore, software engineers need to add sustainability as one of the software’s emergent properties while designing a software system. Due to the complex nature of softwareintensive systems, we should consider sustainability as the quality attributes of a system as a whole. Moreover, sustainable development does not only cover the environmental issue but also the economic and social aspects. Consequently, sustainability efforts should focus on finding the balance between those aspects so that we can achieve a stable relationship between human activities and the natural world [ 3 ]. However, we should realize that finding the balance in sustainability dimensions is hardly possible in real-world scenarios. Therefore, sustainability should be considered as an integrated concept that accommodates the entire dimension. Becker et.al in [ 4 ] argued sustainability has multiple dimensions in which the scenario should consider cross-disciplinary expertise.

Realizing the importance of software system sustainability, various studies have been conducted to address a way to incorporate sustainability issues in SE practice [ 5 ], [ 6 ], especially in requirements engineering (RE) field. RE plays an important role in sustainability engineering because the task is to understand the nature of the system and focuses on stakeholders [ 7 ]. Alharti et al. argued that even though there are many RE tools none of them can analyze sustainability requirements involving stakeholders [ 8 ]. Stakeholders play an important role in this step because sustainability is strongly dependent on the domain application. Moreover, there is an inadequate study on sustainability trade-offs assessment in RE. Huber et al. argued that RE contributions considering sustainability frequently focused on minimizing environmental aspects [ 9 ]. Since sustainability is an integrated concept, considering one dimension can result in ineffective outcomes.

Unfortunately, most of the existing works only analyze the sustainability in high-level abstraction. Without sufficient support of sustainability trade-offs assessment, it can lead to an inefficient decision-making process. It is important to quantify the impact of the software system in the sustainability dimension so that we can create a mitigation plan to reduce this impact. Moreover, there is a strong dependency between dimensions in the sustainability concept, each requirement in a particular dimension can affect requirements in different dimensions [ 10 ]. As a result, a conflict may raise within the elicited requirements. The decision-making process for solving the conflict with sustainability consideration should have the capability to address multiple requirements and objectives. It is also important to understand that software sustainability solution can be varied depending on the domain application. Chitchyan et al. in [ 11 ] argued that sustainability engineering should consider the requirements trade-offs and risks. Therefore, the decision-making process should also be made based on specific multi-criteria based on organizational strategy including its business goal and related stakeholders.

By understanding these problems, this paper proposes a multi-criteria sustainability trade-offs analysis for the decisionmaking process using Analytical Hierarchy Process (AHP) method. The purpose of this approach is to analyze the tradeoffs between conflicting sustainability requirements based on the domain knowledge represented as quantified importance of stakeholders and domain-specific sustainability criteria. This approach helps to select the alternatives which can remove the conflicts and minimize negatives impact in sustainability dimension.

The rest of the paper is organized as follows. Section II covers related works. In Section III, we describe the proposed trade-offs analysis approach. The discussion related to the evaluation result is described in Section IV. In section V, we discuss the threats of validity in this study. Finally, Section VI summaries the paper and discusses the future work of this study.

II. RELATED WORKS

The importance of sustainability has been recognized various RE aspects focusing on how to construct requirements of software system which support sustainable development efforts [ 12 ], [ 13 ]. Penzenstadler et al. argued that sustainability should be treated as important as safety and security[ 14 ]. As a result, sustainability requirements analysis should incorporate risk analysis and impact assessment. Venters et al. proposed a method to understand the concept of sustainability requirements[ 15 ]. They argued that sustainability requirements should be structured more tangibly.

Gibson proposed a practical approach to assess sustainability trade-offs[ 16 ]. Even though this study does not specifically focus on software engineering, it covers the key sustainability issues and their interconnection. The author argued that trade-offs analysis depends on the agreement on the contextspecific sustainability objectives and the awareness of relevant conditions.

Alharti et al. implemented a methodology as a software tool called SuSoftPro for analyzing requirements’ impact on sustainability[ 17 ]. By utilizing the Fuzzy Rating Scale and Technique for Order Preferences by Similarity to Ideal Solution (TOPSIS), the tool was able to discover the overall impact on sustainability for each high-level stakeholders’ requirements. The stakeholders are involved in the profiling process by rating their requirements from sustainability dimensions. However, this methodology merely discussed how to optimize the impact.

Sustainability is a complex concept that has tenacious interaction within social, economic, and environmental dimensions[ 18 ]. Any change in a particular dimension can bring an immense impact in other dimensions which may be contradicting with each other. Consequently, there is an increasing possibility of having conflicts between requirements.

Seyff et al. proposed a requirements negotiation method to understand requirements’ impact on sustainability[ 19 ]. This proposed approach extended WinWin Negotiation Model with sustainability consideration. Their industrial case study evaluation showed that they were able to identify the affected requirements and to find alternative requirements to minimize the impacts. The evaluation results suggested having a domain expert with in-depth knowledge of sustainability design.

De Magalhaes et al. proposed an approach called TODeM for managing trade-offs in complex scenarios[ 20 ]. They argued that considering sustainability increases the complexity of a system because different actors and individual’s mechanism generates collective consequences. This approach used the Fuzzy-AHP method to prioritize the project objectives in the environmental and economic sustainability dimensions. Once the trade-offs were identified, the relevance of each conflicting sustainability objective based on defined characteristics was presented to decision-makers.

III. PROPOSED APPROACH

One of the challenges in sustainability engineering is abstractness and complexity of sustainability concept. Moreover, the capture requirements are analyzed based on their impact on each sustainability dimension. Therefore, we propose an approach for systematically analyzing sustainability tradeoffs presented in Fig. 1. We divide the method into four phases which are the preparation phase, impact analysis phase, alternatives analysis phase, and negotiation phase. For easier application, we utilize an excel worksheet to implement the proposed approach.

A. Preparation Phase

The purpose of this phase is to analyze the domain application from the perspective of stakeholders and their business goals. Therefore, we determine the priority in each stakeholder and sustainability dimension in this phase. In this study, we do not consider the component with infinity weight. If there is a component that should not be violated such as law, then its weight should be assigned with 5.

Step-1: Assigning stakeholders’ weight

The importance of stakeholders varies depending on domain application and organizational business goals. Therefore, in the first step, the requirements engineers have the responsibility to assign a weight for each stakeholder. The weight is ranged from 0 to 1. The total weight is 1 because we assume the weight as relative importance for each stakeholder.

Step-2: Validating stakeholders’ weight

To avoid the subjectivity of the engineering, domain experts are asked to validate the score. By validating the stakeholders’ weight, we can remove the bias from engineers. Domain experts play an important role in this step because sustainability is strongly dependent on the domain application. Once there is a disagreement about the weight, the domain expert should negotiate with the requirements expert to revise the weight.

Step-3: Assigning dimension’s priority

In the third step, the representatives of the stakeholder determine the priority of the sustainability dimension. We adopted the five-dimensional sustainability concepts discussed in [ 4 ] such as Economic(Ec), Social(So), Environmental(Ec), Technical(Te), and Individual(In). The priority of each dimension is quantified by assigning value with a Likert scale [ 21 ] point ranged from 1 (least important) to 5 (most important).

Step-4: Calculating weighted priority

Once the weights and priorities were identified, the final priority score for each dimension (Pd) is calculated using equation 1. pdi and wi are the priority of dimension d given by stakeholder i and weight of stakeholder i, respectively.

Pd = n X pdi ⇥ wi i=1 (1)

B. Impact Analysis Phase

The purpose of this phase is to assess the impact of the gathered requirements in terms of the sustainability dimension. Rather than analyzing the three-order sustainability effect, we focus our study on the direct impact of requirements.

Step-5: Determining requirements’ influence

In the fifth step, the requirements engineers identify the influence of each defined requirements based on the sustainability dimension. There are three types of influence such as support, hurt and neutral. In this step, the requirements engineers also need to identify the related sustainability criteria for each requirement. The sustainability criteria are determined based on the business model and goals. For example, in a smart home system for elderly people, the requirements for ”reduce software development live cycle cost” (R1) has criteria total cost (C1), man-hour(C2), maintenance cost(C3), and hardware cost(C4).

Step-6: Validating requirements’ influence

Similar to step-2, the domain experts cross-check the influence defined by requirements engineer for the validation purpose. Therefore, once there is a different opinion between domain expert and requirements engineering, the negotiation should be conducted to solve the disagreement.

Step-7: Identifying conflicting requirements

In the seventh step, the engineer checks the influence of certain requirements in every dimension. When there is a ”hurt” influence found, the engineer should identify the related requirements in that dimension. For example, we found that the requirement for ”energy efficiency”(R2) hurts ”reduce software development live cycle cost” in the economical dimension.

Step-8: Identifying trade-offs

After the conflict is identified, the engineer needs to identify the trade-offs based on the impacted criteria. Based on the previous example, we can describe the trade-offs as the choice to buy eco-friendly hardware with the same performance can increase hardware cost.

C. Alternatives Analysis Phase

The purpose of this phase is to explore the alternative and determine its contribution using the AHP method to solve the identified conflicts.

Step-9: Exploring alternative for each conflict

When there is a conflict, the requirements engineer and domain expert should explore alternatives for the mitigation plan in the ninth step. Based on the sustainability criteria, R2 is more important than R1. Therefore, we should explore the alternatives for R1. In this example, we identified three requirements alternatives to software development live cycle cost which are optimize memory space (A1), rent eco-friendly hardware (A2), and reduce number of vendors(A3).

Step-10: Assigning importance for each alternative Then, each alternative should be analyzed to see whether it has more contribution in each dimension compared to other alternatives. The analysis is not only conducted in the dimension which has ”hurt” influence but also in every dimension. To do this, we range the importance level from 1 to 5 as seen in Table I. From Table II, we can see that A2 is strongly preferred more than A1. As a result, A1 is less important by 0.3 than A2.

Step-11: Calculating contribution with AHP

Once the score is determined, we calculate the contribution of each goal alternative using the priority vector or eigenvector adopted from Saaty’s AHP [ 22 ]. It is a value to show the relative value of each goal. In this study, we do not consider the goal of infinity weight. If there is a goal that should not be violated such as law, then its priority should be assigned with 5. We choose AHP over other requirements prioritization such as TOPSIS[ 23 ] because AHP provides a comparison matrix based on specific criteria. It also allows us to give different priorities to each criterion. By using the AHP method, the ideal solution for requirements prioritization is not required. With AHP, we are able to solve the conflict among the requirements based on its priority. The work proposed by Perini in [ 24 ] shows that AHP gives a remarkable performance in solving requirements prioritization problems because of its ability to explore the trade-offs between the accuracy of the result and the time spent in the prioritization process.

Table III presents the result for calculating contribution for each alternative using AHP. From this table, we can see that in environmental dimension A2 give more than 5 times contribution (62.77%) compare to A3 (13.84%) and more than twice contribution compares to A1 (23.38%).

D. Negotiation Phase

The purpose of the negotiation phase is to choose the alternative which can minimize the number of requirements conflicts based on their impact on each sustainability dimension.

Step-12: Prioritizing alternatives

In the twelfth step, the alternatives are prioritized based on the impact factor. The impact factor (IFa) score of alternative a is calculated based on the submission of the contribution score (Cad) of the goal alternative a in dimension d multiplied by priority of each dimension (Pd) seen in equation 2. Table IV presented the impact factor for each goal. We can see that the priority rank of the requirements alternative to solve the conflict of R1 is A2, A1, and A3.

IFa =

5 X Cad ⇥ Pd d=1 (2) Step-13: Selecting solution

Lastly, the alternatives are analyzed based on the sustainability criteria in the thirteenth step as seen in Table V. The engineer should justify that the updated requirements meet with sustainability criteria and improve its influence in sustainability dimension impact. For example, the solution for solving the conflict between R1 and R2 is A2 because it has the highest impact factor with similar positive and negative impacted criteria compare to other solutions.

Conflict R1 & R2 R3 & R7

Trade-offs buying eco-friendly hardware with same performance can increase hardware cost providing real-time data transmission from temperature sensor can increase power usage

The weight of five stakeholders were captured (UA1): user(0.3), enterprise(0.25), developer(0.2), expert (0.1), and government(0.15)

Domain expert argued that weight of sustainability expert should be increased. The weight were updated and validated by domain expert: user(0.3), enterprise(0.2), developer(0.2), expert (0.15), and government(0.15) Priority of each sustainability dimension were captured based on stakeholders preferences with the range from 1 - 5. Therefore, different stakeholder can put different priority in particular dimension. For example, User gave 3 in environmental dimension but Government gave 5.

Step-4: Calculating weighted priority

The weighted priority for each dimension were calculated (UA2): Economic(4.7), Social(3.25), Environmental(3.8), Technical(2.55), and Individual(3.9).

Step-5: Determining requirements’ influence

The influence of requirements in sustainability dimension were identified based on three category which are ”Support”, ”Hurt”, and ”Neutral”. In this case study, we identified the influence of 22 requirements Step-6: Validating requirements’ influence

The influence of requirements in sustainability dimension were identified by domain expert.

There is no disagreement between requirements engineer and domain expert in this step. Step-7: Identifying conflicting requirements

The conflicted requirements were identified (UA4). We found seven requirements has at least one ”hurt” influence. Many of the requirements have conflict in environmental dimension and economical dimension.

Step-8: Identifying trade-offs based on sustainability criteria

The trade-offs for seven requirements were identified. In this step the number of impacted sustainability criteria was also identified.(UA3) GP, SP1, SP3 Step-9: Exploring alternatives for each conflict Step-10: Assigning importance for each alternative Step-11: Calculating contribution score with AHP Step-12: Prioritizing alternatives Step-13: Selecting solution

The alternatives for seven conflicted requirements were identified. For each requirement, at least two alternatives should be identified (UA5). For example, we identified three requirements alternatives to software development live cycle cost which are optimize memory space , rent eco-friendly hardware, and reduce number of vendors.

The importance of each alternatives were identified with the range from 1 to 5. By using AHP method, the eigen value for each alternative was calculated. We found that difficulty when this step was done manually. Therefore, we provided an excel worksheet to calculate this value.

Impact factor in sustainability dimension for each alternatives were calculated by multiplying eigen value and sustainability dimension priority (UA6). The impact factor for R1’s alternatives are 6.24(A1), 8.22(A2), and 3.68(A3).

The solution was selected based on impact factor and impacted criteria. For example, renting eco-friendly hardware was chosen because its highest priority and least negative impact in sustainability criteria.

GP, SP1, SP2 GP, SP1, SP2 GP, SP1, SP2 GP, SP1, SP2 GP, SP1 GP, SP1 GP, SP3 GP, SP4 GP, SP4 GP, SP4, SP5 GP, SP4

IV. EVALUATION AND DISCUSSION

This section discusses the evaluation result of implementing the proposed approach in the smart house for elderly people case study. This case study is chosen due to its strong impact on the dimensions of sustainability. This domain highly depends on government rules that concern their environmental and community impact.

A. Domain Application Description

The proposed approach was implemented in a green system of home automation that is targeted to address elderly people’s challenges such as limitations of movement, visual, and hearing problem. The system should provide an intelligent service to support and assist elderly people so that they can stay in their home environment while being remotely monitored by the emergency staff. Moreover, the system should be able to detect and alert the user to health emergencies. The system should provide an alert button for the elderly user to notify the medical emergency staff or family member in case the user feels unsafe. The system should also provide a monitoring system for the selected family member to remotely check the condition of the elderly people. The system should provide an effective and efficient feature to understand individual requirements. These requirements should be used to automatically control the registered home appliances and interior. Moreover, the system should allow the user to have social interaction with other people. Since the system implements cloud technology, the system should provide secured access and connection including door lock and security.

B. Evaluation Procedure

We follow the validation process proposed by Lee and Rine [ 25 ] based on the case study research method by Yin [ 26 ] for evaluation. The purpose of this evaluation is to systematically capture the valid inference from the case study and collect evidence as a means to generalize the outcome of the observed case study by performing each step of the proposed approach in the case studies.

1) Study Questions: The study question (SQ) must be defined precisely to ensure a reliable validation process. The following SQ must be answered.

• how can the proposed approach help to analyze multidimensional sustainability trade-offs from requirements? (SQ1) • how can the proposed approach minimize the subjectivity in the decision-making process? (SQ2) 2) Study Propositions: The proposition becomes the assertion that should be examined using a distinct measurement that can answer the study questions. In this study, the General Proposition (GP) is ”The proposed approach achieves research goal because it provides multi-criteria sustainability trade-offs analysis using quantified assessment.”. This GP derives the following specific proposition (SP).

• The proposed approach helps the requirements engineer to identify the possible conflict by allowing them to determine the impact of requirements in the sustainability dimension. (SP1) • The proposed approach allows the requirements engineer to determine the priority of each dimension validating by the domain expert. (SP2) • The trade-offs between conflicting requirements are analyzed based on sustainability criteria. (SP3) • The proposed approach helps the engineer to ease the decision-making process by providing the impact score for each alternative. (SP4) • With the AHP method, the proposed approach quantifies the contribution of each alternative in the sustainability dimension. (SP5) 3) Units of Analysis: The selected resources that must be examined during case study evaluation are presented as Units of Analysis (UA). UA demonstrates how the proposed approach supports the study proposition. The following UA is used in this study.

• stakeholder weight (UA1) • sustainability dimension priority (UA2) • sustainability criteria (UA3) • conflicting requirements (UA4) • alternatives list (UA5) • alternatives’ impact score (UA6) 4) Evidence Collection: We linked the implementation result in each step of the proposed approach to the defined propositions. Table VI presents the summary of collected evidence to systematically show the way proposed approach support all of the proposition. The evidences were collected using semi-structured interview method.

C. Discussion

In this subsection, we discussed the result of the case study evaluation based on the study questions.

1) how can the proposed approach help to analyze multi-dimensional sustainability trade-offs from requirements? (SQ1): From a conducted case study evaluation, we can prove that the proposed approach is capable of analyzing sustainability trade-offs and minimizing the negative impact of requirements in the sustainability dimension with the provided trade-offs analysis worksheet. In this study, we focus more on the sustainability dimension context. We can address the complex nature of sustainability by considering multiple sustainability criteria. We can reduce the number of conflicting requirements by analyzing the sustainability criteria. With the provided approach, we can guide the engineer to formally analyze the sustainability trade-offs. We were able to solve seven out of nine identified conflicts in smart home systems for elderly people domain application. These conflicts were solved with the identified alternatives which were analyzed based on their contribution to the sustainability dimension. We also found that even though we were able to reduce the number of negative impacts, we cannot remove all negative impacts. For example, the best alternative to address the trade-offs in the conflict between R1 and R2 has one negative impact which is ”complexity” related to the technical dimension.

We faced some challenges during the evaluation session. Firstly, there is a difficulty to calculate eigenvector for the AHP method. We need to spend more time to explain the steps for applying Saaty’s AHP. Another challenge that we faced is the stopping criteria. Ideally, we need to stop analyzing the tradeoffs when there is no more conflict between requirements. However, it is almost impossible to have a software system without conflicting requirements. Addressing this threat, we assume that the trade-offs analysis can be stopped when 2/3 of conflicting requirements had been solved.

2) how can the proposed approach minimize the subjectivity in the decision-making process? (SQ2): We also can prove the proposed approach was able to minimize subjectivity in the decision-making process by providing a cross-validation process among domain experts and engineers. With the proposed approach, the domain expert validated the stakeholders’ weight, dimension priority, and requirements influence defined by domain experts. We found that the negotiation among the domain expert and engineer is helpful to explore more alternatives to solve the identified conflicts

During our case study evaluation, there are some disagreements between domain experts and engineers. Domain experts argued that sustainability experts should be given higher weight compare to enterprise because the main objective is to build a green smart home system for the elderly. However, there is still a challenge related to the validation process conducted by the domain expert. This process can be threatened by the knowledge of the domain expert. The validation process in step-2 and step-6 is strongly dependent on how deep the domain expert is. This issue is also related to the capability of requirements engineer and domain expert to explore the alternative for each conflict.

V. THREATS TO VALIDITY

This section presents the threats to the validity of the study including the addressed threats and unaddressed threats. We followed the qualitative case study guideline proposed by Baskarada in [ 27 ]. The quality of the study is analyzed based on construct validity, internal validity, external validity, and reliability adopted from the study proposed by Edmonds and Kennedy in [ 28 ].

A. Addressed Threats

Construct validity focuses on whether a test is to be interpreted as a measure of some attributes that are not operationally defined. We addressed this threat by following the formal case study guideline. With this guideline, the units of analysis used as pieces of evidence were determined based on study questions and study proposition. We ensure there is no reactivity bias by conducting open question semi-structured interview methods to collect the data. Internal validity focuses on the direct impact of dependent and independent variables. We conducted one group pre-test and post-test to ensure the causality of the proposed approach. In order to minimize the interaction effects, the instruction was given at the beginning of the case study and no interruption during the study. We cross-checked the result with the group to address the conclusion bias. A case study can be said as a reliable case study in which it produces the same result in consistent conditions[ 29 ] based on data collection procedure. We addressed this threat by collecting various sources of sustainability reports. We also collected relevant documents on the smart home system for elderly people.

B. Unaddressed Threats

One of the limitations of this study is the treats related to external validity because we conducted a case study in one domain application. Therefore, one of the future works for this study is applying the proposed approach in different domain applications.

VI. CONCLUSION

Sustainability studies in SE have been conducted in the past few years. However, there is a lack of works analyzing the trade-offs between requirements based on sustainability aspects. We propose a novel approach for analyzing the sustainability trade-offs between requirements. We provide an approach for helping the engineer to decide which alternatives are capable of removing requirements conflict. AHP method is utilized to prioritize the requirements based on weighted sustainability dimensions and criteria. By using the case study methodology designed research, we demonstrate the application of our guided proposed approach in minimizing negative sustainability impact and reducing conflicting requirements.

Among the limitation of this work is the difficulty of applying AHP to calculate eigenvector for prioritizing alternatives as a mitigation plan. Therefore, the focus of future work is developing a tool that can aid the engineer in incorporating the AHP method for sustainability analysis. By providing automatic tools, the requirements engineer and domain expert can create traceability links and automate the validation process. We also plan to extend the work by exploring effective and efficient stopping criteria. The future work of this study includes conducting empirical study on various domain application to assess the generalization of the proposed approach.

ACKNOWLEDGMENT

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF2017R1D1A1B03034279)

References

[1] C. P. Balde´ , V. Forti , V. Gray , R. Kuehr , and P. Stegmann , The global e-waste monitor 2017: Quantities, flows and resources . United Nations University, International Telecommunication Union, and . . . , 2017 .

[2] I. Ozkaya and R. Nord . ( 2018 , Oct.) Data-driven management of technical debt . [Online]. Available: https://insights.sei.cmu.edu/sei blog/ 2018 /10/data -driven-management-of-technical-debt .html

[3] S. Garcia , Y. Cintra , S. Rita de Ca´ssia, and F. G. Lima , “ Corporate sustainability management: a proposed multi-criteria model to support balanced decision-making,” Journal of Cleaner Production , vol. 136 , pp. 181 - 196 , 2016 .

[4] C. Becker , R. Chitchyan , L. Duboc , S. Easterbrook , B. Penzenstadler , N. Seyff , and C. C. Venters , “ Sustainability design and software: The karlskrona manifesto ,” in Proceedings of the 37th International Conference on Software Engineering-Volume 2 . IEEE Press, 2015 , pp. 467 - 476 .

[5] S. Druskat and D. S. Katz , “ Mapping the research software sustainability space ,” in 2018 IEEE 14th International Conference on e-Science (eScience) . IEEE, 2018 , pp. 25 - 30 .

[6] N. C. Fernandez , P. Lago , M. R. Luaces , A. S. Places , and L. G. Folgueira , “ Using participatory technical-action-research to validate a software sustainability model , ” in 2019 6th International Conference on ICT for Sustainability (ICT4S) , 2019 .

[7] C. Becker , S. Betz , R. Chitchyan , L. Duboc , S. M. Easterbrook , B. Penzenstadler , N. Seyff , and C. C. Venters , “Requirements: The key to sustainability, ” IEEE Software , vol. 33 , no. 1 , pp. 56 - 65 , 2015 .

[8] A. Alharti , M. Spichkova , and M. Hamilton , “ Towards tool-support for sustainability profiling .” in RE4SuSy@ RE, 2018 , pp. 6 - 14 .

[9] M. Z. Huber , L. M. Hilty , and M. Glinz , “ Uncovering sustainability requirements: An exploratory case study in canteens .” in RE4SuSy@ RE, 2015 , pp. 35 - 44 .

[10] L. Zhang , Y. Xu , C.-H. Yeh , Y. Liu , and D. Zhou , “ City sustainability evaluation using multi-criteria decision making with objective weights of interdependent criteria , ” Journal of cleaner production , vol. 131 , pp. 491 - 499 , 2016 .

[11] R. Chitchyan , C. Becker , S. Betz , L. Duboc , B. Penzenstadler , N. Seyff , and C. C. Venters , “ Sustainability design in requirements engineering: state of practice,” in Proceedings of the 38th International Conference on Software Engineering Companion. ACM , 2016 , pp. 533 - 542 .

[12] M. Mahaux , P. Heymans , and G. Saval, “ Discovering sustainability requirements: an experience report ,” in International Working Conference on Requirements Engineering: Foundation for Software Quality . Springer, 2011 , pp. 19 - 33 .

[13] A. Raturi , B. Penzenstadler , B. Tomlinson , and D. Richardson , “ Developing a sustainability non-functional requirements framework ,” in Proceedings of the 3rd International Workshop on Green and Sustainable Software. ACM , 2014 , pp. 1 - 8 .

[14] B. Penzenstadler , A. Raturi , D. Richardson , and B. Tomlinson , “ Safety, security, now sustainability: The nonfunctional requirement for the 21st century,” IEEE software , vol. 31 , no. 3 , pp. 40 - 47 , 2014 .

[15] C. C. Venters , N. Seyff , C. Becker , S. Betz , R. Chitchyan , L. Duboc , D. McIntyre , and B. Penzenstadler , “ Characterising sustainability requirements: A new species red herring or just an odd fish? ” in 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Society Track (ICSE-SEIS) . IEEE, 2017 , pp. 3 - 12 .

[16] R. B. Gibson , “ Sustainability assessment: basic components of a practical approach,” Impact assessment and project appraisal , vol. 24 , no. 3 , pp. 170 - 182 , 2006 .

[17] A. D. Alharthi , M. Spichkova , and M. Hamilton , “Susoftpro: Sustainability profiling for software,” in 2018 IEEE 26th International Requirements Engineering Conference (RE) . IEEE, 2018 , pp. 500 - 501 .

[18] B. Penzenstadler , L. Duboc , C. C. Venters , S. Betz , N. Seyff , K. Wnuk , R. Chitchyan , S. M. Easterbrook , and C. Becker , “ Software engineering for sustainability: Find the leverage points!” IEEE Software , vol. 35 , no. 4 , pp. 22 - 33 , 2018 .

[19] N. Seyff , S. Betz , L. Duboc , C. Venters , C. Becker , R. Chitchyan , B. Penzenstadler , and M. No¨bauer, “Tailoring requirements negotiation to sustainability,” in 2018 IEEE 26th International Requirements Engineering Conference (RE) . IEEE, 2018 , pp. 304 - 314 .

[20] R. F. de Magalha ˜es, Aˆ . d. M. F. Danilevicz , and J. Palazzo , “ Managing trade-offs in complex scenarios: A decision-making tool for sustainability projects , ” Journal of cleaner production , 2018 .

[21] G. Albaum , “ The likert scale revisited,” Market Research Society . Journal., vol. 39 , no. 2 , pp. 1 - 21 , 1997 .

[22] T. L. Saaty , “ Decision-making with the ahp: Why is the principal eigenvector necessary ,” European journal of operational research , vol. 145 , no. 1 , pp. 85 - 91 , 2003 .

[23] N. Kukreja , S. S. Payyavula , B. Boehm , and S. Padmanabhuni , “ Valuebased requirements prioritization: usage experiences,” Procedia Computer Science , vol. 16 , pp. 806 - 813 , 2013 .

[24] A. Perini , A. Susi , F. Ricca , and C. Bazzanella , “ An empirical study to compare the accuracy of ahp and cbranking techniques for requirements prioritization ,” in 2007 Fifth International Workshop on Comparative Evaluation in Requirements Engineering. IEEE, 2007 , pp. 23 - 35 .

[25] S. W. Lee and D. C. Rine , “ Case study methodology designed research in software engineering methodology validation .” in SEKE , 2004 , pp. 117 - 122 .

[26] R. K. Yin , Case study research and applications: Design and methods . Sage publications , 2017 .

[27] S. Baskarada , “ Qualitative case study guidelines,” Basˇkarada , S. ( 2014 ). Qualitative case studies guidelines . The Qualitative Report , vol. 19 , no. 40 , pp. 1 - 25 , 2014 .

[28] W. A. Edmonds and T. D. Kennedy , An applied guide to research designs: Quantitative, qualitative, and mixed methods . Sage Publications , 2016 .

[29] W. M. Tellis , “ Application of a case study methodology,” The qualitative report , vol. 3 , no. 3 , pp. 1 - 19 , 1997 .