-Software developers are gradually becoming aware that their systems have effects on sustainability. The identification of potential effects software-intensive systems can have on different sustainability dimensions over time is yet in its infancy. Researchers are currently exploring approaches which strongly make use of expert knowledge to identify potential effects. In this work in progress paper, we are looking at the problem from a different angle: we report on the exploration of a machine learning-based approach to identify potential effects. Such an approach allows to save time and costs but increases the risk that potential effects are overseen. First results of applying the machine learning-based approach in the domain of home automation systems are promising, but also indicate that further research is needed before our approach can be applied in practice. Furthermore, we have learned that even providing the ground truth for training the algorithms is a challenging task.
Software-intensive systems do not operate in isolation but in
complex socio-technical contexts. Therefore, they have an
impact on this context, manifesting itself in different dimensions
such as the environmental, economic, social, individual, and
technical dimension [
In previous research [
Requirements are the key to sustainability [
Copyright 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). certain aspects of a system are modified in the context of system evolution. This, in part, also makes the identification of potential effects a hypothetical endeavour, which often needs to be based on expert opinions rather than facts. Only after development, when the system is used in its application context, one can eventually validate its effects on the different sustainability dimensions over time.
Researchers have started to build methods and tools to
support the identification of sustainability effects [
In this work in progress paper, we follow a current trend in software and requirements engineering and propose the use of Machine Learning (ML) for the early identification of potential sustainability effects. In this paper, we present this idea in more detail and also report on a first application experiment in the home automation domain. Based on requirements for home automation systems, we have identified potential sustainability effects and have used these results as ground truth for training our algorithms. Early results indicate that using ML for the identification of sustainability effects is promising, which motivates us to continue with this research.
In Section 2 we discuss existing approaches for identifying sustainability effects in requirements. In Section 3 we present our ML-based approach and report on a first experiment we conducted in Section 4. In Section 5 we conclude the paper and present an outlook on next steps we plan.
II. EXISTING SUSTAINABILITY EFFECT IDENTIFICATION
APPROACHES AND TOOLS
The work presented in this paper is motivated by research in the field of requirements engineering, where researchers aim at identifying potential sustainability effects.
In previous work on tailoring requirements negotiation to
sustainability [
Recent work presents a question-based framework for
raising awareness of the potential effects of software systems on
sustainability [
Alharthi et al. [
We conclude that researchers have identified the need for identifying sustainability effects and that first promising methods and even tool-supported approaches are appearing. However, all presented methods strongly depend on human involvement and might require time-intensive discussions, the involvement of experts or a large number of people to derive results. The quality of the produced results might further vary a lot depending on different factors such as the complexity of the domain and the level of expertise of the people involved. Nevertheless, bespoke methods can raise awareness and can help, at least in part, to improve the sustainability of softwareintensive systems.
III. MACHINE LEARNING-BASED EFFECT IDENTIFICATION
The goal of our ongoing research is to automate the identification of potential sustainability effects by analysing the requirements of a software-intensive system. In contrast to existing methods in place, we expect that such an approach will result in the significant reduction of the effort needed for the analysis. However, we also see the risks that such an approach might result in overlooking potential effects.
To achieve this goal, we follow a recent trend in software
and requirements engineering and explore the application of
ML [
To automatically identify the impact of requirements on
sustainability we follow the ML workflow [
The first key step is data preprocessing that helps to avoid data incompleteness and inconsistency issues. This data is used as an input for the ML algorithm, which means that the ML algorithm learns from existing data. On the basis of this learning, a learning model is produced as output, which can be used to make predictions on a different dataset than the one used for training. The learning model can be evaluated based on its performance. For measuring the performance, we use well-known ML parameters such as precision, recall, accuracy, and F-score.
In the next section, we will describe how we have performed the above discussed steps in our experiment.
We performed a first experiment in the domain of home automation systems to evaluate our ML-based approach. In the next subsection, we describe the setup of our experiment and in the subsequent subsection we present our preliminary results.
Data: The dataset used for training and evaluating our
MLbased approach is comprised of publicly available smart home
requirements1. The requirements were collected as part of
research on crowd-RE [
In a first step, three annotators manually classified 200 randomly selected requirements from the total set of available requirements (around 2900). All three annotators had proficient knowledge and expertise for sustainable systems in software engineering. For each requirement, each annotator independently marked which sustainability dimension(s) it had an effect on. To support this classification, a literature review has been performed on sustainability dimensions and created a classification guideline based on the results of this analysis. The guideline contained for each dimension a set of influence factors and for each factor a description, rational, example requirements, and literature references for further reading. Table I shows an example influence factor in the environmental dimension.
The annotators used the guideline as a reference during the manual classification of the 200 smart home requirements. Each requirement was independently classified according to its influence on the five dimensions of software sustainability as positive, negative, or neutral. The plus sign (+) was assigned for positive influence, a minus sign (-) for negative influence, and no sign for indicating no influence as shown in Table II. The ratings were merged and cases in which the researchers did not agree were discussed until consensus was reached.
1Smarthome Crowd Requirements Dataset, https://crowdre.github.io/murukannaiah-smarthome-requirements-dataset/ Dimension Factor Description Influence/Rational Example
Pre-processing: We applied natural language processing techniques on our data before applying the different ML algorithms. First, we applied text tokenization on each requirement. Then we eliminated all stop words and converted the text into small characters. We applied stemming as our next step. As the last step, we converted pre-processed text as a vector space model using Term Frequency-Inverse Document Frequency (TF-ID or TF-IDF) as a weighting scheme: tf idf (t; d) = tf (t; d) idf (1)
Here t is a term in a vector and d is a requirement in a
collection of requirements [
Classification: For the automated classification of
sustainability requirements and their dimensions, we trained our
model using the annotated requirements dataset. We
implemented Nave Bayes (NB), k-Nearest Neighbor (KNN),
Decision trees (DT), Support Vector Machine (SVM), Logistic
regression (LR), and Neural Network (NN) algorithms and
also trained our classifier with and without stemming the data,
as discussed in [
The results for six different classifiers from our experiments are shown in Table III. For choosing the best classifier, we evaluated the performance of all these classifiers on the basis of commonly used ML metrics i.e., accuracy, precision, recall, and F1-score. These metrics can be calculated using the following formulae:
Accuracy = T P + T N=T P + F P + F N + T N P recision = T P=(T P + F P ) (2) (3)
Recall = T P=(T P + F N ) F 1 score = P recision=Recall (4) (5)
Here, TP (True Positive) is the number of requirements correctly classified as belonging to a category. TN (True Negative) is the number of requirements that are correctly classified as not belonging to a category. FP (False Positive) is the number of requirements incorrectly classified as belonging to a category, and FN (False Negative) is the number of requirements that are incorrectly classified as not belonging to a category.
In simple words, accuracy is the ratio of correctly classified data over total data. This helps to predict the model performance whereas high accuracy results in a better performance of the model. However, data can be asymmetric and thus parameters other than accuracy need to be evaluated. The precision metric refers to the ratio of correctly predicted positive values to the total number of predicted positive values. On the other hand, recall is the ratio of total predicted positive values to the actual number of positive values. It is not possible to maximize both recall and precision metrics at the same time, as one comes at the cost of another. To consider both, F1-score is used which is the harmonic mean of precision and recall.
The highest accuracy and F1-score decide which algorithm is the best among others. We achieved the highest accuracy with DT classifier (70% precision) followed by SVM (69% precision). Our dataset was not balanced, meaning that the five different sustainability dimensions were not equally represented in the dataset (see Fig. 2). The economic, environmental, and social dimensions were almost equally represented. The individual dimension had high occurrences and the technical dimension was almost inexistent. To overcome this problem, we used the weighting technique by assigning more weight to fewer data. After applying this setting, we achieved the highest accuracy for SVM (75%), as shown in column SVM (b) of Table III. Recall and precision are 63% and 57% respectively, which is acceptable according to our accuracy. We also calculated the F1-scores, and the highest value was achieved with SVM (60%).
The results from this initial experiment can be improved further as we observed some issues that are impacting our results. The structure of our dataset varied with respect to the length of the textual requirements. For example, one requirement consisted of 20 words, and another one consisted of 200 words. Due to the significant difference regarding their length, our ML classifier features were sparse, which might have lead to an underfitted model.
Moreover, our negative and positive influence values on the sustainability dimensions were also not equally distributed. Our data only contained 12 requirements with negative influence, the rest were positive influences.
The goal of our ongoing research is the automation of the identification of requirements, which potentially have effects on the sustainability of a software-intensive system.
In this paper, we present the application of a state-of-the-art ML approach to support effect identification. Our first results indicate that ML can be successfully used for the identification of potential sustainability effects. However, we have learned that the results from our first experiment can be improved further. This starts with the dataset. The current dataset can be improved to generate a more suitable learning model for the classifiers.
As a next step, we plan to increase the size of our dataset. We have already designed a web-based solution where experts can update the categorized requirements and add additional requirements. This web-based tool will help us to improve our labeling and support us in getting more data. It will also allow us to provide a more balanced dataset.
Our current results indicate that there is the risk of overlooking requirements which have potential effects. As a next step, we plan to focus on optimizing recall to minimize this risk. We envision that our approach could be used to complement existing methods. Instead of discussing each requirement manually, our approach could provide a list of relevant requirements, which should be discussed further by human stakeholders. High recall might result in lower precision, which means that human stakeholders are confronted with a larger number of false positives. However, we expect that providing a reduced set of requirements for discussion will enable practitioners to save time compared to a full manual analysis.
Similar to other work in this field, we have also experienced that manually identifying potential effects can lead to different opinions amongst the annotators. Although, the three annotators were able to agree on a ground truth used for training our classifier, we would like to highlight that our results might reflect a rather subjective viewpoint of the annotators.
Overall, we would like to explore how we can integrate our
work into existing studies for requirements classification. In
particular, we envision to use our approach within planned
work on crowd-focused semi-automated requirements
engineering for evolution towards sustainability [
The authors would like to thank Robert O¨rdo¨ g for implementing the tool support for our ML-based approach and for conducting the experiment presented in this paper. This research was partially funded by the European Unions Horizon 2020 research and innovation program under the Marie Skodowska-Curie grant agreement No. 674875.