Using Participatory Technical-action-research to validate a Software Sustainability Model Nelly Condori Fernandez∗† Patricia Lago∗ , Miguel R. Luaces† , Ángeles S. Places† , Leticia González Folgueira‡ ∗ Vrije Universiteit Amsterdam, The Netherlands n.condori-fernandez@vu.nl, p.lago@vu.nl † Universidade da Coruña, Database Laboratory, Spain n.condori.fernandez@udc.es, miguel.luaces@udc.es, angeles.saavedra.places@udc.es ‡ Enxenio, Spain lgonzalez@enxenio.es Abstract—[Context and Motivation] In the last years, Lago et al. [8] and Venters et al. [7] agree on software engineering researchers have contributed to defining software sustainability in terms of multiple and defining the notion of sustainability-aware software as interdependent dimensions (e.g. economic, technical, a quality requirement. [Question/problem] The field is, social, environmental, individual). However, despite this however, still missing instruments supporting the design multidimensional nature of sustainability, most of the and assessment of software sustainability. current efforts have been put on understanding what [Objective] This research aims at providing a validated Sustainability Assessment Framework (SAF) through a software aspects can impact on the environmental sus- long-term empirical study in close-collaboration with the tainability dimension [13]. software industry. [Methodology] By using the partici- patory technical action-research method, we validate the In order to provide a better characterization of the sustainability-quality model, one of the instruments of the sustainability dimensions, Condori-Fernandez and Lago SAF framework, by means of investigating its applicability [9] proposed a preliminary sustainability-quality model in an industrial software project and detecting potential for software-intensive systems. It consists of quality at- improvements. tributes that contribute to each sustainability dimension, [Results] Our results confirm the effective applicability of our model as most of its quality attributes (QAs) have been and their corresponding direct dependencies. According either addressed in the software project or acknowledged to the authors [9], software sustainability is defined in as relevant. The action-research method was also very terms of four dimensions: the social dimension aims to useful for enriching our model by identifying QAs missing allow current and future generations to have equal and in the model (e.g. regulation compliance, data privacy). equitable access to the resources in a way that preserves [Contribution] The sustainability-quality model can their socio-cultural characteristics and achieve healthy be effectively used as an instrument for identifying and modern society. The environmental dimension seeks sustainability-quality requirements, and creating aware- to avoid that software-intensive systems harm the en- ness on the relevance of the multidimensional sustainabil- vironment they operate in. The technical dimension is ity nature of certain quality attributes. concerned with supporting long-term use and appropri- Keywords—sustainability-quality model, quality require- ate evolution/adaptation of software-intensive systems ments, action research, software-intensive systems. in constantly changing execution environment. And, the economic dimension aims to ensure that software- intensive systems can create economic value. It is taken I. I NTRODUCTION care of in terms of budget constraints and costs as In recent years, the concept of sustainability has well as market requirements and long-term business been recognized as i) a timely important concern for objectives that get translated into requirements for the researchers from different disciplines in computing (e.g. system under consideration. artificial intelligence [1], human computer interaction [2], software engineering [3], [4]; and ii) a key driver As shown in Figure 1, the Sustainability-quality of innovation at software companies [5]. Thanks to this model is one of the key instruments of the Sustain- growing acknowledgment, several efforts have been put ability Assessment Framework (SAF) proposed by the for understanding what software sustainability means authors and consisting of three main components: the (e.g. [6], [7], [8], [9] ) and how software engineering sustainability-quality model [9], [14], the architectural can support sustainability (e.g. [10], [11], [12]]). decision map [15], and the metrics [16]. In order to validate SAF in close collaboration with and incremental research cycles are carried out at the software industry, we have designed a long-term em- product level, where each increment corresponds to pirical study supported by action-research methodology the instruments of the SAF framework (Sustainability- [17], [18]. In this paper, we present the overall overview quality model, the Decision maps, and the Metrics of our empirical research strategy, as well as the first catalogue). Finally, the results emerging from the stage validation results as a consequence of applying the Reflection (at the product level) are fed back to the sustainability-quality model in a software project that participating software company (at the product-family focuses on the achievement of sustainability goals. level). Notice that the involvement of any other project from the product-family in the subsequent iterations (action, feedback collection and reflection) is allowed as long as the participants agree as part of the diagnosis stage. The following subsections present the research ques- tions shaping our study, the empirical research context (including the project and participants) and the unit of Figure 1: Timeline of the research cycles applied to analysis. validate the SAF framework A. Research Questions In order to validate the SAF framework, the follow- The following sections provide a detailed account ing research questions are formulated: of our study. Section 2 describes the research method. Section 3 reports our main findings. Then section 4 (RQ1) How applicable is the SAF framework to discusses the validity threats and section 5 the related assess the software architecture at hand? works. Section 6 concludes the paper and discuss further (RQ2) Which improvements to the SAF frame- work. work are realized as an outcome of imple- menting the action research? II. R ESEARCH METHOD In this paper, we focus on answering both research Our research strategy is inspired by the combi- questions with a special focus on the sustainability- nation of two types of action research: (i) technical quality model as illustrated in Figure 1. As such, the an- action research because we aim to scale up the action swer to RQ1 aims at (i) identifying sustainability-quality (treatment) to conditions of practice by actually using attributes present in the model and already addressed in it in a particular problem [17]; and (ii) participatory the selected project, and (ii) discovering sustainability- because the researchers are active in making informed quality requirements that are not yet included in the decisions throughout all aspects of the research process, model but are relevant for the selected project. as the participating organization shares experiences of applying the action [18], [19]. According to Petersen et The initial sustainability-quality model proposed by al. [19], the action is the treatment introduced by the Condori and Lago [9], [14] was used as the starting researcher to induce a positive change in the company. point for answering RQ1. In our study, the action is the SAF framework. Naturally, its iterative application to the selected The research strategy we use to validate the SAF products can enrich the model itself with new insights framework consists of four stages (grey-colored in and/or new attributes, and hence help answering RQ2. Figure 2): diagnosis, action, feedback collection and Figure 3 summarizes the possible states that can be reflection. These stages are carried out at two different determined as a result of applying the sustainability- levels. We start the diagnosis at the product-family quality model: Quality Attribute (QA) discovered (or- level with the purpose of understanding the common ange cell), QA covered (green cell), QA missing (red characteristics of the family of products and identifying cell), and n/a when a QA is not observable (grey cell). sustainability-related issues. The output of the diagnosis These states will be used in Section III and Tables II is the selection of a product (software project), which is and III to present the study results. then used in the following three stages. At the product level, we plan the research to apply the SAF to the B. Research context selected software project, then we collect the feedback from the participants and reflect on it to refine the design The research context is characterized by six software of the SAF framework. As shown in Figure 1, iterative products developed by the Database Laboratory (LBD), 2 Figure 2: Participatory Technical-Action Research Process a research group of the University Of A Coruña, as them acted as responsible for planning and executing part of the GIRO Project1 . A spin-off software company, the application of the SAF framework. Both participants Enxenio (ENX), will maintain the software products in are researchers, who played three different roles in the the future. action research [17]: (i) Designer: designing the instru- ments of the SAF framework; (ii) Helper: using the 1) The GIRO project: The GIRO Project, funded by instruments of the SAF framework to help the software the FEDER Interconnecta program, has a reference ar- company in getting awareness on the sustainability- chitecture, reused and adapted to addresses the specific quality requirements that were addressed in the project; requirements of the individual customer companies. and (iii) Researcher: drawing lessons learned about the The customer companies of the GIRO consortium instruments of the SAF framework. are leaders in the following market areas: Two participants were practitioners from the • Company A: treatment of meat by-products not Database Laboratory and the software company (Enx- intended for human consumption, particularly enio), who used the sustainability-quality model to collecting dead animals in the northwest of identify relevant QAs that had not been considered in Spain. the project, as well as missing QAs that were not present in the Model. The roles of these participants were • Company B: management and valorization of software analyst and software project manager. Besides organic waste by means of its transformation the GIRO project counted with a software architect, in biogas. the third practitioner from Enxenio, who contributed solving some doubts in the QAs identification process. • Company C: installation and management of alarm systems in companies and domiciles. C. Unit of analysis and Procedure • Company D: health and well-being services The unit of analysis of the study is the prod- offered for elderly people residing in their own uct family developed under the GIRO project, whose home. documentation was analyzed and discussed in several • Companies E and F: prevention of occupational meetings as part of the diagnosis stage. The software risks regarding health at work. company is interested in knowing how sustainable is the reference software architecture used for the GIRO 2) Participants: The participatory-technical action product family. research team consisted of five participants. Two of In this paper, we focus on the software product re- 1 Acronym used for the project name: ”Generating, Managing and quired by the company A, which is a Mobile Workforce Integrating Routes using OLAP” Management (MWM) System developed to support the 3 Table I: Focus groups conducted during the first research III. R ESULTS AND D ISCUSSION cycle A. RQ1: Applicability of the model FG Purpose LBD/ENX ENX The following presents our study results. They refer FG1 Social sustainability 2 to Table II for the QAs already covered by the Model FG2 Technical sustainability 2 1 (i.e. addressed in the project or relevant but not yet FG3 Economic-Environmental sustainability 2 addressed), and Table III for the new QAs that were FG4 Generic 1 1 missing in the Model. 1) Quality attributes covered in the project: These dead animal collection. Its selection was as a result are: of the activities carried out at the diagnosis stage. In Compatibility. This QA in terms of interoperability order to answer our research questions, the next re- and co-existence was addressed in the project since search activities (action planning and -taking, feedback there was a need of sharing information with some collection and reflection) were carried out at the product existing systems (i.e. Libra, active directory) used by level. Through an action plan defined for allowing the customer company. The relevance to the social the application of our sustainability-quality model [14] sustainability dimension was acknowledged because of (action/treatment) to the selected product, we carried the greater access of information (from different sys- out action taking (i.e. technical-documentation analysis tems) that can then be used by users (e.g. administrator, and four focus group meetings [20]). Each focus- planner). Moreover, the compatibility’s relevance to group meeting was planned by the researchers (the first the technical dimension was also confirmed since as author played the moderator’s role). The purpose of the long as the MWM system performs efficiently, while corresponding focus groups and participants are shown both environment and resources are shared with other in Table I. Although these focus groups were small, the software system, its use will last longer. participants reflected on the analyzed QAs, by explain- ing their relevance for the selected project and giving Effectiveness. This QA was addressed in the project examples on how some of the QAs were addressed. and its relevance to the technical, social and economic Because of this active discussion among participants dimensions was confirmed. Thanks to the 1-month (practitioners and researchers), we considered the four testing period, developers found that the MWM system sessions as a focus group and not an interview. Each provides a good support to accurately perform user focus group had an average duration time of 60 minutes, tasks that achieve all specified stakeholders goals such and notes were taken during the focus group meetings as: the planner is able to schedule new requests from by the moderator (feedback collection). The following the insurance company; the driver is able to store/track section presents the results obtained from the reflection requests until such time as they are complete, etc. It is on the collected feedback to contribute to the answers technical because effectiveness contribute to the long- to RQ1-RQ2. term usage of the MWM system. It is social because of the accurate performance of all user tasks contribute to their own well-being (e.g. drivers less stressed using the MWM system for tracking new requests). Moreover, the achievement of stakeholders goals contribute to one of the long-term business objectives of the software company (i.e. customer satisfaction), as well as the saving costs for fixing failures (economic dimension). Efficiency. Relevance of Efficiency to the environ- ment dimension is confirmed and also partially ad- dressed, by having a good usage of certain resources when users perform their tasks supported by the system. For instance, the average time used for completing user tasks (e.g. administrator/planner for scheduling new requests, drivers register attended requests) was according to the expectations of the customer company. However, other type of resources, like the amount of energy, used by the MWM system when users perform Figure 3: Specific research questions investigated certain tasks, has not been yet evaluated. Similarly to through the action research effectiveness, this QA also contributes to the economic dimension because of the saving costs for fixing failures. 4 Freedom from risk. It is addressed in terms of: Performance efficiency. This requirement in terms environmental risk mitigation, by means of the timely of time behaviour was acknowledged as a good contrib- use of the MWM system, which aims to provide support utor to both technical and environmental dimensions. for collecting dead animals under the European regula- While the system shows an efficient performance (e.g. tions (CE 999/2001). By doing so, the MWM system processing and response time) in delivering the main helps (i) avoid exposing people to potential disease- functionalities (e.g. finding optimum route, allocating causing pathogens (social dimension); and (ii) reduce new requests in the planning), its use will last longer environmental concerns like potential contamination of (technical sustainability dimension). air, soil, surface and sub-surface water (environmental Reliability. There was a consensus that the MWM dimension). In a similar way, the relevance of Health system must be reliable when perform not only un- and safety risk mitigation to the social dimension was der normal operations (maturity), but also whenever it confirmed. The health risks to people like farmers and is required by end-users (availability). Although both potential meat consumers, can be mitigated by means QAs contribute to the technical dimension, a greater of the timely use of the MWM system. Regarding availability of the MWM system (e.g. visualization the safety risk mitigation, this QA has not been yet of the route planning, tracking driver behaviour) will addressed for certain type of stakeholders that directly contribute not only to the environmental and economic interact with the MWM system. For instance, for drivers sustainability dimensions, but also to the social dimen- who are exposed to road safety risks, the system could sion. It is social because the environmental and social allow an adaptive multimodal interaction to show the sustainability goals of this kind of systems can be route for reaching a target place (e.g. a farm). achieved only if the software services are available. The Functional Suitability. This quality is addressed social contribution was marked with an “+” since it was in terms of functional appropriateness and functional not considered in the original model. correctness. Practitioners confirmed the contribution of Satisfaction. This requirement in terms of trust both QAs to the technical and economic sustainability and usefulness is relevant to address both social and dimensions. economic sustainability dimensions. As practitioners Both quality requirements were verified through one- consider that the direct relation between user satisfac- month testing. All the functionalities implemented as tion and the technology acceptance( [21], [22]) has a part of the MWM system facilitate the accomplishment positive impact on the social sustainability dimension of tasks performed by the planner, driver, and system ad- since satisfied users will be in a much better position for ministrator. For example, the planner, main component getting access to social resources provided by the corre- of the MWM system, consider all relevant parameters sponding software system. Also, the practical relevance to provide an optimum route planning to be used by the of both QAs to the economic dimension was confirmed drivers. since customers satisfaction is considered as one of the Functional suitability is relevant to the technical dimen- primary business objectives of the software company. sion since both QAs contribute to the long-term use of Moreover, as long as the usefulness and trust of the the system. It is also relevant to the economic dimension system are valued by the end-users, the acceptance to because the software company do not have to dedicate use the system will be prolonged, which means that both much effort on corrective maintenance actions. QAs contribute to the technical dimension as well. Maintainability. The relevance of this requirement Security. This requirement in terms of accountabil- is acknowledged with respect to the modifiability, modu- ity, authenticity, confidentiality and integrity was con- larity and reusability attributes. As a common architec- sidered as a good contributor to the social sustainability. ture is planned to be used for the six software systems Confidentiality was addressed thanks to the Role Based of the GIRO project, modifiability and modularity are Access Control implementation, where roles previously key to facilitate the adaptation of the reference software defined are assigned only to group of users that are architecture to be used in the product family and its authorized to have access to the system. Authenticity evolution (technical dimension). was also addressed since the login of each user has Moreover, as the company aims to create a flexible been used as identifier. And as several user actions architecture for addressing all relevant requirements of can be traced, accountability was also addressed but each software system of the product family, modifia- only for certain actions (i.e. updates of new routes). bility and reusability should contribute also to reduce Regarding the integrity of the system, this requirement redesign costs and allow quicker response to company was also considered as a good contributor to the tech- customers. However, as the relevance of both QAs to the nical dimension because the MWM system is able to economic dimension was not identified in the original prevent (in certain extent) unauthorized access to the model, in Table II we mark both contributions with an system, and does so to the extent that security controls “+”. specified for that system cannot be compromised. This 5 new contribution was also marked with an “+”. Testability. It is another relevant attribute that was acknowledge as relevant for the technical sustainability Usability. As expected usability attributes like ap- dimension, but it was not addressed since we could not propriateness recognizability, operability, and user error find any evidence on how testable are the software arte- protection were clearly considered as relevant to the so- facts (MWM system, module) to find (critical) faults. cial sustainability dimension. User error protection was addressed through the use of field-validation methods, Capacity. This QA was acknowledged as relevant mandatory fields, and action confirmation. Protecting for the technical dimension but not addressed. Load or users against making errors it is social because it stress testing could have been carried out for determin- contributes not only to satisfy usability requirements but ing the weak points of the system architecture. also to the quality of user experience [23]. However, this QA was also considered as good contributor to Accessibility. Although the company has knowl- both technical and economic dimensions, which were edge on existing standard accessibility guidelines (e.g. also confirmed by two case studies. As both contribu- ISO/IEC 40500:2012), this QA was not addressed in tions were not considered in the original model, they the project since was not considered as very relevant are marked with an “+”. During the testing period, for the project. However, participants agreed that imple- issues related to operability of the system were fixed. menting some accessibility features that can help certain Moreover, as end-users recognized that the system is users like ”deaf and hearing impaired drivers” could appropriate for meeting their corresponding needs (i.e. be beneficial to our society in long term, by removing drivers need to attend all assigned requests per day), interaction/communication barriers. appropriateness recognizability was also addressed. Learnability. Although this QA was not imple- 2) Quality Attributes discovered in the project: In mented, its relevance was acknowledged not only for this section, we discuss the QAs that were discovered the social dimension but also for the economic dimen- as relevant for the selected project as a result of using sion. By implementing features that aid novice users the sustainability-quality model. to quickly learn and also allow steady progression to expertise, it could results on saving training costs to Context Coverage. Practitioners agreed on the rel- the company. (This new contribution to the economic evance of addressing context coverage requirement in dimension is marked with an “+”). terms of context completeness and flexibility. Despite issues related to effectiveness and efficiency were veri- Robustness. Given that the MWM system could fied during the one-month testing period, it is possible be affected by several unexpected situations (e.g. GPS that some contexts where the system could be used were signal is lost), robustness is consider as relevant to not covered by the testing process. The need of explicit address the technical sustainability dimension. specifications about the different contexts of use was acknowledged as a relevant requirement that should be considered as good contributor to both technical and B. RQ2: Improvements in the model economic dimensions. Moreover, if the system is not able to work at any other potential context of use (not This section reports on the missing QAs that were explicitly specified), a higher effort and costs would be identified as relevant to be included in the Model as well needed to improve the flexibility of the MWS system. as their corresponding contributions to the sustainability dimensions (see Table III): Modifiability and Reusability. Although both maintainability attributes were considered as relevant to 1) Quality attributes missing in the Model: These contribute to the environmental dimension, they were are: not addressed because of the lack of facilities/tools for determining in which extent the software artefacts (e.g. Data Privacy. Given MWM system needs data to be modules) of the system that are modifiable or reusable stored and shared for enabling the management of work contribute to reduce environmental impact. assignments and the tracking of real-time field workers, Resource utilization. This requirement was discov- data privacy is a key requirement that should be con- ered as relevant to the project and its contribution to sidered as good contributor to the social sustainability the environmental dimension was confirmed. However, dimension. Timeliness. With MWM systems, timeliness despite the contribution of the planner module to the op- of information is needed for allowing field workers to timization of resources utilization(e.g. trucks needed for continue with their job successfully (rapid collection of collecting dead animals, gasoline consumed by trucks), dead animals). As this QA concerns on the favourable this requirement was not fully addressed because of the time of having the right information(a social resource), lack of a manager that could help in determining the it has been also considered as a good contributor to the actual amount of resources used by the system. social sustainability. 6 Table II: Sustainability-quality analysis of the MWM system (Green cell= QA is addressed, orange cell= QA is discovered as relevant, light-gray cell= QA is in the model but not relevant for the project, += new contribution) Characteristics Attributes Definition according to [9] TECH SOC ENV ECON Co-existence product can perform its functions efficiently while sharing environment and Compatibility resources with other products. Interoperability a system can exchange information with other systems and use the information that has been exchanged. Context cov- Context com- system can be used in all the specified contexts of use erage pleteness Flexibility system can be used in contexts beyond those initially specified in the require- ments. Effectiveness Effectiveness accuracy and completeness with which users achieve specified goals. Efficiency Efficiency resources expended in relation to the accuracy and completeness with which users achieve goals. Freedom Economic risk system mitigates the potential risk to financial status in the intended contexts of from risk mitigation use. Environmental system mitigates the potential risk to property or the environment in the intended risk mitigation contexts of use. Health and system mitigates the potential risk to people in the intended contexts of use. safety risk mitigation Functional Functional ap- the functions facilitate the accomplishment of specified tasks and objectives. suitability propriateness Functional system provides the correct results with the needed degree of precision. correctness Functional degree to which the set of functions covers all the specified tasks and user completeness objectives. Maintainability Modifiability system can be effectively and efficiently modified without introducing defects + or degrading existing product quality Modularity system is composed of components such that a change to one component has minimal impact on other components. Reusability an asset can be used in more than one system, or in building other assets + Testability effectiveness and efficiency with which test criteria can be established for a system. Performance Capacity the maximum limits of a product or system parameter meet requirements. efficiency Resource uti- the amounts and types of resources used by a system, when performing its lization functions, meet requirements. Time response, processing times and throughput rates of a system, when performing behaviour its functions, meet requirements. Portability Adaptability system can effectively and efficiently be adapted for different or evolving hardware, software or usage environments. Replaceability product can be replaced by another specified software product for the same purpose in the same environment. Reliability Availability system is operational and accessible when required for use. + Fault tolerance system operates as intended despite the presence of hardware or software faults. Maturity system meets needs for reliability under normal operation. Recoverability system can recover data affected and re-establish the desired state of the system is case of an interruption or a failure. Satisfaction Trust stakeholders has confidence that a product or system will behave as intended. Usefulness user is satisfied with their perceived achievement of pragmatic goals. Security Accountability actions of an entity can be traced uniquely to the entity. Authenticity the identity of a subject or resource can be proved to be the one claimed. Confidentiality system ensures that data are accessible only to those authorized to have access. Integrity system prevents unauthorized access to, or modification of, computer programs or data. Usability Appropriateness users can recognize whether a system is appropriate for their needs, even before recognizability it is implemented. Learnability system can be used to achieve specified goals of learning to use the system. + Operability system has attributes that make it easy to operate and control. User error pro- system protects users against making errors. + + tection Accessibility Accessibility system can be used by people with the widest range of characteristics and capabilities. Robustness Robustness Refers to the capability of the sytem to behave in an acceptable way in unexpected situations Survivability Survivability The degree to which a system continues to fulfil its mission by providing essential services in a timely manner in spite of the presence of attacks Regulation compliance. As the main business re- quirement of the customer company is to get that dead 7 animal removal requests can be attended without delays • by adding new QAs that had not been consid- according to the European regulations, the software ered in the model. As shown in Table III, many designers/developers should be aware on them to meet of these QAs were added to the social sustain- this requirement. Regulation compliance can be relevant ability dimension (i.e. data privacy, timeliness, for contributing to i) social sustainability since health regulation compliance, tailorability). Most im- risks are minimized, and ii) environmental sustainability portantly only one QA (tailorability) was added since potential contamination of natural resources (e.g. for the technical dimension. Overall, this result water, air) is reduced. shows how important can be to make explicit which sustainability dimension is relevant for Scalability. As the customer company could be which QA, so that significant metrics can be interested in scaling up number of clients(currently col- identified and monitored. lection services are offered only at the Galician region), this would have an impact on the architecture because it • by uncovering new direct dependency relations2 should meet efficiently the increased workload as well. as a consequence of identifying new contribu- Thus, scalability is considered as a relevant QA for tions. The QAs that were included in this type the economic sustainability dimension because of the of relations are shown in Table II, whose cells significant costs saving. are marked with an “+”. For example, as modi- fiability and reusability are new contributors to Tailorability. Enabling new configuration of func- the economic dimension, the direct dependency tionality as well as control information provision con- between environmental and economic dimen- tributes to the technical and social sustainability di- sions consists of four ordered pairs (whose mensions. It is technical because the environments in QAs are efficiency, availability, modifiability, which users can both interact and engage with software and reusability). can contribute to the long-term usage. And it is social because giving users the tailoring capability in their own Finally, we argue that if a QA is found to contribute to context of use can contribute to get a better access of multiple sustainability dimensions, its definition should the information provided by the system. be specialized for each dimension: in this way, the specific contribution is made explicit and hence helps C. Discussion identifying the best-fitting influencing factors, and met- This study focused on validating the sustainability- rics. This observation addresses both RQ1 (cf. model quality model by means of investigating its applicability applicability) and RQ2 (enriching the model). in a real software project (RQ1) and detecting related improvements (RQ2). IV. T HREATS TO VALIDITY The results shown in Table II confirm the effec- Here we discuss the threats to the validity of the tive applicability of our sustainability-quality model action research study [19] and provides rationale for (RQ1) because of the following facts: (i) the quality related design decisions. requirements addressed in the project are covered by most of the QAs present in the model in the indicated Internal validity. Action research is highly context sustainability dimension/s (cells coloured in green); (ii) dependent. To mitigate this threat, the design of our the participants were able to become aware of the study considered a family of software products that relevance of certain QAs that were already present in are from different application domains, which is useful the model but that had not been addressed in the project to analyze the sensitivity of the SAF framework in with respect to a certain sustainability dimension (cells different contexts. coloured in orange). Construct validity. Action research is subjective as We also found that many QAs (e.g. co-existence, the results highly depend on the reflection of the action efficiency, availability, reusability, modifiability, trust, researcher. Several biases may occur due to: (i) two usefulness) contribute to more than one sustainability participants are also part of the software company and dimension of the model. This finding confirms the they could not have provided an objective/external view multi-dimensional nature of sustainability (one of the of the situation. (ii) The researchers as the designers of principles of the Kaslskrona Manifesto [24]) and the the action may interpret the results positively (selective relevance level of the QAs that can be considered by bias) when reporting the results. Both issues were software engineers when performing certain activities partially solved by involving multiple practitioners in like design, assessment, and prioritization [9]. the iterative discussions. In this first research cycle, a With respect to RQ2, the application to the MWM 2 A direct dependency is defined as a finite set of ordered pairs of system helped us enrich our model in two ways: QAs, which is reflexive, symmetric and transitive [9]. 8 Table III: New quality attributes and corresponding contributions to the sustainability dimensions Characteristics Attributes Definitions TECH SOC ENV ECON Data Privacy Data Privacy privacy concerns arise wherever personally identifiable information is collected, stored, or used. Timeliness Timeliness the fact or quality of being done or occurring at a favourable or useful time. Regulation Regulation allows to draw conclusions about how well software adheres to application compliance compliance related regulations in laws. Scalability Scalability the ability of a computing process to be used or produced in a range of capabilities Tailorability Tailorability system’s capability to allow users to create or enable new configuration of functionality as well as control information provision. third practitioner was involved in some of the meet- which helps scoping architectural concerns and quality ings. Moreover, the researchers reviewed carefully the requirements along the four dimensions above. existing technical documentation to triangulate the data Becker et al. [30] have a similar approach but collected from the focus-group meetings. grounded in requirements engineering instead. In ad- External validity. The action is implemented in a dition to the above four sustainability dimensions, they specific social setting, which can hinder the general- add the individual as a fifth sustainability dimension. ization of the results. However, we could apply our We argue that the social dimension and the individual findings to other projects with similar characteristics. dimension share the same social nature. Differently, Moreover, the transferability within the setting studied the first takes a broader perspective (e.g. organizations, may be high if the context is similar. On the other hand, society, stakeholder types). This is especially relevant in as our initial sustainability-quality model was defined software architecture, which aims at capturing “the big based on the ISO/IEC 25010 standard [25], we consider picture”. The second dimension, instead, is appropriate that its generality could be more easily adaptable to whenever the concerns of the individual (e.g. end- other domains, such as those that were considered by user, citizen) should be addressed. This naturally comes the GIRO project. forward more frequently in requirements engineering and human-computer interaction. V. R ELATED W ORK VI. C ONCLUSIONS AND F URTHER W ORK Being able to identify the relevant quality require- ments on sustainability is the first step towards develop- The present empirical study was designed to validate ing software-intensive systems that fulfill sustainability the Sustainability Assessment Framework (SAF) within concerns by design [9]. an action-research setting. This paper focused mainly on the sustainability-quality model, one of the instruments Venters et al. [26] discussed the notion of software of the SAF framework, that has been applied in one sustainability based on the analysis of the literature. of the software products developed under the GIRO After debating if it should be considered as a non- project. As a result of this application, from a prac- functional requirement or an emergent property, the titioner perspective (RQ1), the proposed sustainability- authors conclude it to be a multi-faceted concept and quality model was found as a useful instrument for (i) argue for a quantitative approach. identifying the relevance level of QAs that contribute to different sustainability dimensions (e.g. trust, modifia- Based on the ISO/IEC 25010 Standard, Calero et bility, efficiency), and (ii) discovering quality require- al. [6] provide a preliminary discussion of which qual- ments that had not yet been addressed in the project ity characteristics should be considered in addressing at hand (e.g.context-completeness, flexibility, testability, software sustainability. As a next step, they propose capacity). From a researcher perspective (RQ2), the the definition of a quality model where sustainability study has helped uncovering new missing QAs that were is part of the quality of software products. In contrast identified as relevant to be included in the sustainability- to our work, Calero et al. defined sustainability only in quality model (e.g. regulation compliance, data privacy). terms of energy consumption, resource optimization and perdurability (reusability, modifiability, and adaptabil- As a further work, we plan to apply the decision ity). Originated in the 2013 GREENS workshop [27], maps and the metrics, by using the same software prod- Lago et al. [8] defined a four-dimensional model that uct, and replicate the validation of the sustainability- extends the social, environmental and economic dimen- quality model, by involving a new GIRO software sions (rooted in the Brundtland report [28]) with the product within the same action-research environment. technical dimension. Later on, Lago introduced the Soft- Our sustainability model will be also enriched with the ware Sustainability Assessment (SoSA) method [29], findings obtained from the case study reported in [31]. 9 ACKNOWLEDGMENT [16] N. Condori-Fernández, A. Bagnato, and E. Kern, “A focus group for operationalizing software sustainability with the This work has received partial support by MEASURE platform,” in 4th International Workshop on Mea- Xunta de Galicia / FEDER-UE CSI (ED431G/01, surement and Metrics for Green and Sustainable Software ED431C 2017/58), and MINECO-AEI/FEDER-UE Systems (MeGSuS), Co-located at ESEM, Oulu, Finland, 2018. with the projects: Datos 4.0 (TIN2016-78011-C4- [17] R. Wieringa and A. Moralı, “Technical action research as a 1-R), ETOME-RDFD3 (TIN2015-69951-R), and validation method in information systems design science,” in Design Science Research in Information Systems. Advances FLATCITY(TIN2016-77158-C4-3-R.) in Theory and Practice, (Berlin, Heidelberg), pp. 220–238, Springer Berlin Heidelberg, 2012. R EFERENCES [18] C. MacDonald, “Understanding participatory action research: [1] M. De-Arteaga, W. Herlands, D. B. Neill, and A. Dubrawski, A qualitative research methodology option,” Canadian Journal “Machine learning for the developing world,” ACM Trans. of Action Research, vol. 13, no. 2, p. 34–50, 2012. Manage. Inf. Syst., vol. 9, pp. 9:1–9:14, Aug. 2018. [19] K. Petersen, Ç. Gencel, N. Asghari, D. Baca, and S. Betz, [2] O. Bates, V. Thomas, C. Remy, L. Nathan, S. Mann, and “Action research as a model for industry-academia collabora- A. Friday, “The future of hci and sustainability: Championing tion in the software engineering context,” in ACM International environmental and social justice,” in CHI EA ’18 Extended Workshop on long-term Industrial collaboration on Software Abstracts of the 2018 CHI Conference on Human Factors in Engineering (WISE), pp. 55–62, 2014. Computing Systems, vol. 2018-April, ACM, 4 2018. [20] R. Krueger, Focus groups: a practical guide for applied re- [3] B. Penzenstadler, L. Duboc, C. C. Venters, S. Betz, N. Seyff, search. Sage Publications, 1994. K. Wnuk, R. Chitchyan, S. M. Easterbrook, and C. Becker, [21] B. H. Wixom and P. A. Todd, “A theoretical integration of user “Software engineering for sustainability: Find the leverage satisfaction and technology acceptance,” Info. Sys. Research, points!,” IEEE Software, vol. 35, pp. 22–33, July 2018. vol. 16, pp. 85–102, Mar. 2005. [4] R. Chitchyan and et al, “Sustainability design in requirements [22] G. Assefa and B. Frostell, “Social sustainability and social engineering: State of practice,” in ICSE Companion, pp. 533– acceptance in technology assessment: A case study of energy 542, 2016. technologies,” Technology in Society, vol. 29, no. 1, 2007. [5] R. Nidumolu, C. Prahalad, and M. Rangaswami, “Why sustain- [23] Y. I. Ormeno, J. I. Panach, N. Condori-Fernndez, and O. Pastor, ability is now the key driver of innovation,” IEEE Engineering “Towards a proposal to capture usability requirements through Management Review, vol. 43, no. 2, pp. 85–91, 2015. guidelines,” in IEEE 7th International Conference on Research [6] C. Calero, M. Á. Moraga, and M. F. Bertoa, “Towards a soft- Challenges in Information Science (RCIS), pp. 1–12, May 2013. ware product sustainability model,” CoRR, vol. abs/1309.1640, [24] C. Becker, R. Chitchyan, L. Duboc, S. Easterbrook, B. Penzen- 2013. stadler, N. Seyff, and C. C. Venters, “Sustainability design and [7] C. Venters and et. al, “The blind men and the elephant: Towards software: The karlskrona manifesto,” in Proceedings of the 37th an empirical evaluation framework for software sustainability,” International Conference on Software Engineering - Volume 2, Journal of Open Research Software, vol. 2, no. 1, 2014. ICSE ’15, (USA), pp. 467–476, IEEE Press, 2015. [8] P. Lago, S. A. Koçak, I. Crnkovic, and B. Penzenstadler, [25] ISO/IEC, “ISO/IEC 25010 - Systems and software engineering “Framing sustainability as a property of software quality,” - Systems and software Quality Requirements and Evaluation Communications of the ACM, vol. 58, no. 10, pp. 70–78, 2015. (SQuaRE) - System and software quality models,” tech. rep., 2010. [9] N. Condori-Fernandez and P. Lago, “Characterizing the con- tribution of quality requirements to software sustainability,” [26] C. Venters, C. Jay, L. Lau, M. K. Griffiths, V. Holmes, R. Ward, Journal of systems and software, vol. 137, pp. 289–305, 2018. J. Austin, C. E. Dibsdale, and J. Xu, “Software sustainability: The modern tower of babel,” in RE4SuSy: Third International [10] B. Penzenstadler, “Infusing green: Requirements engineering Workshop on Requirements Engineering for Sustainable Sys- for green in and through software systems,” in Proceedings of tems, 2014. the Third International Workshop on Requirements Engineer- ing for Sustainable Systems, RE4SuSy 2014, co-located with [27] P. Lago, N. Meyer, M. Morisio, H. A. Müller, and G. Scan- 22nd International Conference on Requirements Engineering, niello, “Leveraging energy efficiency to software users: sum- Sweden, August 26, 2014., pp. 44–53, 2014. mary of the second GREENS workshop, at ICSE 2013,” ACM SIGSOFT Software, 2014. [11] N. Condori-Fernández and P. Lago, “The influence of green strategies design onto quality requirements prioritization,” in [28] G. Brundtland, M. Khalid, S. Agnelli, S. Al-Athel, B. Chidzero, International Working Conference on Requirements Engineer- L. Fadika, V. Hauff, I. Lang, M. Shijun, M. Morino de Botero, ing: Foundation for Software Quality, pp. 189–205, 2018. M. Singh, and S. Okita, “Our common future (brundtland report),” tech. rep., World Commission on Environment and [12] R. Chitchyan, J. Noppen, and I. Groher, “What can software Development, 21 May 1987. engineering do for sustainability: Case of software product lines,” in International Workshop on Product Line Approaches [29] P. Lago, “Software and sustainability [inaugural lecture].” in Software Engineering, pp. 11–14, May 2015. http://dare.ubvu.vu.nl, Jan. 2016. [13] G. A. Garcı́a-Mireles, M. Á. Moraga, F. Garcı́a, C. Calero, and [30] C. Becker and et al, “Requirements: The key to sustainability,” M. Piattini, “Interactions between environmental sustainability IEEE Software, vol. 33, no. 1, pp. 56–65, 2016. goals and software product quality: A mapping study,” Infor- [31] N. Condori-Fernandez and P. Lago, “Towards a software mation & Software Technology, vol. 95, pp. 108–129, 2018. sustainability-quality model: Insights from a multi-case study,” [14] N. Condori Fernandez and P. Lago, A Sustainability-quality in International Conference on Research Challenges in Infor- Model: (version 1.0). VU Technical Report, 11 2018. mation Science, IEEE, 2019. [15] P. Lago, “Architecture design decision maps for software sus- tainability,” in International Conference on Software Engineer- ing, Software Engineering in Society Track (ICSE SEIS), 2019. 10