-Measuring the sustainability of software products is sill in the early stages of development. However, there are different approaches how to assess sustainability issues of software and its engineering - including metrics with a practical orientation as well as more theoretical models covering software sustainability. As an example for one step in moving forward bringing existing approaches together, the paper presents a focus group study conducted to find out in which extent the quality attributes related to the technical sustainability can be measured by using existing metrics available at the MEASURE platform. Our first results show that the extent of measurability varies across the software development phases. Functional correctness, robustness, maturity, and testability are the most measurable quality attributes. Index Terms-technical sustainability, measurement, focus group, software metrics
Assessment based on the notion of sustainability, as a
software quality property, is still emerging and poorly understood
[
This is especially the case when it comes to technical
sustainability of software. According to [
The paper is structured as follows: Section II presents the MEASURE platform and the Software Sustainability Model. These are the basic information of the focus group workshop. The design of focus group, including a description of the participants, research questions, and methods, is introduced in Section III. Section IV illustrates the validity of the research done before summarizing and discussing the results of our study in Section V.
The MEASURE ITEA3 consortium (Softeam R&D, 2017)
[
This European MEASURE ITEA 3 project develops a
framework of metrics [
Lago et al. [
The economic dimension aims to ensure that
softwareintensive systems can create economic value. It is taken care
of in terms of budget constraints and costs as well as market
requirements and long-term business objectives that get
translated or broken down into requirements for the system under
consideration. The social dimension aims to allow current
and future generations to have equal and equitable access
to the resources in a way that preserves their socio-cultural
characteristics and achieve healthy and modern society. The
environmental dimension seeks to avoid that software-intensive
systems harm the environment they operate in. And, the
technical dimension is concerned with supporting long-term
use and appropriate evolution/adaptation of software-intensive
systems in constantly changing execution environment. Based
on these definitions, quality attributes (QA) that contribute
to the corresponding sustainability dimensions of
softwareintensive systems were identified [
The goal of our focus group study, according to the
Goal/Question/Metric template, is as follows:
Analyze metrics from the MEASURE platform and Software
Sustainability-Quality Model [
Our focus group study represents an early assessment exercise of the MEASURE platform. We define the following research question:
useful for measuring technical sustainability?
For determining the potential usefulness of MEASURE for operationalizing the sustainability-quality attributes, from our research question, we set out three specific questions to our participants:
quality attributes as contributors to technical sustainability?
cycle, do you think it would be feasible to measure the list of quality attributes? 1http://eseiw2018.wixsite.com/megsus18
be useful for measuring technical sustainability?
For answering our research question, we considered it advisable that our participants should have a very good knowledge competence on software measurement, as well as interest in any research topic related to software sustainability. Both criteria were successfully satisfied by our eight participants, attendees of the MeGSuS workshop. Two of them were practitioners. All of them contributed to the workshop focusing on software measurement and showed their interest in the topic by that.
The focus group study was organized in four small groups, to run the study, the following instrumentation was distributed among the groups:
Technical sustainability definition List of quality attributes and corresponding definitions of the attributes Metrics from the MEASURE platform, whose definitions were accessible via a wiki website2
After reading and clarifying the definitions, the participants selected the phase of the software development, they felt most familiar with. Regarding our two first specifics questions, verbal data was collected, whereas for our third question, a large sheet of paper containing a grid was used by each focus group.
As shown in Figure 2, participants used an ”X” for representing the relation: ”M can measure QA” or ”QA can be measured by M”.
Those ”X” enclosed by a circle were used to identify a set of basic metrics that can measure a QA.
Table I shows the twenty two quality attributes of technical sustainability that were analyzed by our focus group participants.
2https://github.com/ITEA3-Measure/Measures/wiki
As shown in Figure 3, the procedure of our focus group study involves the following four phases:
1) Preparation phase: This phase has two objectives: i) to get a common understanding on what software sustainability means regarding technical sustainability dimensions, ii) to decide which sustainability dimensions are going to be used in the next phase. This phase has been carried out by the organizers of the focus group, consisting of one moderator and two assistants.
After having a discussion (before realizing the focus group), and considering also the time allocated for this study as part of the MeGSuS workshop, the researchers decided to work with the technical sustainability dimension.
The activities of the next phases were carried out during the focus group.
2) Phase 1: What?: The objective of this phase is to validate the contribution of the corresponding QAs to the technical sustainability dimension. Thus, in this phase, participants answered RQ1:1. The moderator introduced briefly the motivation of the focus group, presented an overview of the sustainability-quality model as well as a plan of activities to be carried out. The outcome of this phase is a list of selected QAs that will be analyzed in the following phases.
The average time taken for this phase was about 10 minutes. 3) Phase 2: When?: The objective of this second phase is to discuss on which phases of the software life cycle the selected qualities could be measured. Thus in this phase, based on their participants experience, RQ1:2 was answered. The average time taken was about 5 minutes.
4) Phase 3: How?: The objective of this third phase is to assess the usefulness of the metrics from the MEASURE platform. Thus, in this phase, participants answered RQ1:3.
It took approximately 25 minutes. All participants of the four focus groups shared their mapping results, by emphasizing the reasoning behind the mappings, difficulties of understanding the purpose of some metrics and discussing open questions on the connection of the issues. In this phase, we were open to new metrics that could be suggested by the participants. However, due to time restrictions, this data was not collected.
External validity. It is the ability to generalize the results from a sample to a population. As focus groups tend to use rather small, homogeneous samples, generalization is the main limitation of our study. Our study involved four mini-groups, with people from different countries, but most of them were researchers. To mitigate this threat, we are going to replicate this first focus group, involving more groups representing a diverse sample of people. Internal validity. It is strengthened by a moderator providing an appropriate amount of guidance without introducing any of his/her own opinion or stifling free expression. In order to reduce this threat, the moderator used an introductory material (Powerpoint-slides) for contextualizing the focus group study.
Construct validity. It is concerned with whether the focus group is actually measuring what they are trying to measure. In our focus group, we focus on investigate the coverage and measurability aspects. By using two different existing approaches - one with a more practical orientation and one theoretical model - having a common focus, the direction of the focus group was specifically predefined. This ensured that the focus of discussion was also set on the technical sustainability dimension.
In order to answer our main research question related to the usefulness of the MEASURE platform for measuring the technical sustainability dimension, we analyzed the collected data from each focus group (see matrix, Figure 6). Usefulness of the platform is analyzed regarding coverage and measurability aspects, which are discussed as follows.
Considering the total of metrics available at the MEASURE
platform [21], [22], [
Next we present the selected metrics by each mini-focus group.
metrics of the specification phase” shows the 10 selected specification phase metrics that were mapped with the QAs of the technical sustainability dimension. 2) Metrics for the design phase: Table IV: ”Selected metrics of the design phase” shows the 18 selected design phase metrics that were mapped with the QAs of the technical sustainability dimension.
3) Metrics for the implementation phase: Table V : ”Selected metrics of the implementation phase” shows the 13 selected implementation phase metrics that were mapped with the QAs of the technical sustainability dimension.
metrics of the testing phase” shows the 21 selected testing phase metrics that were mapped with the QAs of the technical sustainability dimension.
As shown in the matrix (Figure 2 in Appendix), the specification, design, implementation, and testing metrics where associated to the quality attributes presented in Table I.
Considering the twenty-two QAs of the technical sustainability dimension (see Table I), Figure 4 shows the percentages of QAs that can be measured by at least one of the selected metrics. Most of the QAs can be measured at the specification phase (82%, 18 of 22 QAs), followed by design (59%, 13 of 22 QAs), testing (32%, 7 of 22 QAs) and implementation (23%, 5 of 22 QAs).
This indicates that most of the QAs related to the technical sustainability can be qualified as measurable. In order to represent the extent of measurability for each development phase, we calculated the number of available metrics selected from the platform for measuring each QA (see Figure 5). According to these results, we observe that our participants found that functional correctness, robustness and maturity can be measured by using a good number of metrics at the testing phase (13 metrics). In case of the specification phase, especially functional correctness and functional appropriateness are of high importance, meaning covered by many metrics (5 of 10 metrics). Efficiency is connected with most of the metrics of the design phase while the quality attribute is not covered in the other phases analyzed. Overall, functional suitability is connected to many of the proposed metrics for the analyzed software development phases.
In this paper, we describe the result of the the focus group designed to discuss on ”What, when and how to measure software sustainability”. The authors organized the study within the MeGSuS 2018: 4th International Workshop on Measurement and Metrics for Green and Sustainable Software Systems [23].
Through the focus group, we found a good number of metrics that were selected from the MEASURE platform as ”potentially” useful for measuring quality attributes of the technical sustainability dimension along certain phases of the software development life cycle (i.e. design, specification, testing, implementation). This result provides evidence on the coverability of the MEASURE platform for the specification, design, implementation and testing phases.
Moreover, the study has also shown that most of the technical sustainability-quality attributes are measurable. The results can be appreciated and are summarized in Figure 5, where we can highlight the following results: .
Metrics for the specification phase were distributed among the various QAs with higher metrics related to QA1 and QA4.
Metrics for the design phase were distributed among the various QAs with higher metrics related to QA19, QA15 and QA14.
Metrics for the implementation phase focus, according to the results of the focus group, on a limited number of QAs (QA15, QA7, QA22) related to technical sustainability dimension. The subgroup considered that maintenability / testability, maintenability / modifiability, and security were the QAs associated to the higher number of implementation metrics (see Table V) .
Metrics for the testing phase focus on a limited number of QAs (QA1, QA11, QA18) related to technical sustainability dimension. The sub-group considered that functional suitability, robustness, and reliability were the QAs associated to the higher number of testing metrics (see Table VI).
Functional correctness, robustness, maturity and testability are
the most measurable quality attributes considering the four
phases. The focus group acknowledged that the technical
sustainability dimension [
We thank all the participants who took part in our focus group study: Je´roˆ me Rocheteau from the Institut catholique d’arts et me´tiers (Icam) site of Nantes, Birgit Penzenstadler from California State University Long Beach, Shola Oyedeji from Lappeenranta University of Technology, Denisse Mun˜ ante from University of Bordeaux, Diogo Silveira Mendoa from Pontifical Catholic University of Rio de Janeiro, and Thibault Beziers la Fosse from Laboratoire des Sciences du Numrique de Nantes, Software Modeling Group (LS2NNAOMOD).
The research presented in this paper is partially funded by the ITEA3 Project no. 14009 called MEASURE (1st December 2015 and running till 31st August 2019). Sage Publications Newbury Park, Calif, 1988. [Online]. Available: http://www.loc.gov/catdir/enhancements/fy0654/87033413-t.html [21] A. Abherve and A. Bagnato, “Repository of measures specification in smm,” https://github.com/ITEA3-Measure/Measures/wiki, Aug. 2017, last accessed on 2018-12-01. [22] A. Bagnato and A. Abherve, “Repository of measure implementations,” https://github.com/ITEA3-Measure/Measures, Aug. 2017, last accessed on 2018-12-01. [23] E. K. Alessandra Bagnato, Nelly Condori Fernandez, “4th workshop on measurement and metrics for green and sustainable software systems (megsus18) october 9, 2018 - oulu, finland,” http://eseiw2018.wixsite.com/megsus18, Aug. 2018, last accessed on 2018-12-01. ID SM1 SM2 SM3 SM4 SM5 SM6 SM7 SM8 SM9 SM10
Requirement Number of Tests Requirements Satisfaction Quality Indice Requirement Traceability To Implementation Indice Requirement Coverage Indice Requirement Complexity Indice
Business Rules Number Of Goals Requirement Traceability To Test Indice
Total number of requirement defined in the selected scope.
Total number of tests defined in the selected scope. Percentage of requirements that have been satisfied. Percentage of requirements that have been satisfied. The average number of requirements tracing an architecture model.
The average number of sub requirements defined to rafine an existing requirement.
Total number of risks defined in the selected scope. .
Total number of requirement defined in the selected scope.
Total number of goals defined in the selected scope.
The % of requirement of tracing a test model. ID DM1 DM2 DM3 DM4 DM5 DM6 DM7 DM8 DM9 DM10 DM11 DM12 DM13 DM14 DM15 DM16 DM17 The number of direct subclasses of a class.
A class implementing an interface counts as a direct child of that interface.
The average number of dependencies from a package. Total number of methods defined in the selected scope. The number of software compoments identified in an application architecture.
Total number of classes in the selected scope Total number of interfaces in the selected scope. Total number of methods defined in the selected scope. Total number of Components defined in the selected scope. Total number of Packages defined in the selected scope. The average number of dependencies from a class. The average number of dependencies from a package.
divided by the total number of types in a package. Total number of fields defined in the selected scope. Total number of use cases defined in the selected scope. Total number of actors defined in the selected scope. Total number of interfaces component types in the java Modelio model. Along with the total number of data, this metric provides an idea of the functional richness of the modelled application.
Total number of aggregated components in the java Modelio model. Along with the number of composed components, this metric reflects the usage of the software decomposition in the modelled application. Count the number of Interface annotated @ComposedComponent in Java Model.
ID IM1 IM2 IM3 IM4 IM5 IM6 IM7 IM8 IM9 IM10 IM11 IM12 IM13 defining how hard it is to understand the code’s control flow. defining as A = 0 vulnerability, B = at least 1 minor vulnerability, C = at least 1 major vulnerability, D = at least 1 critical vulnerability, E = at least 1 blocker vulnerability Effort to fix all vulnerability issues found on the code changed in leak periods Effort to fix all vulnerability issues.
A = 0 bug, B = at least 1 minor bug, C = at least 1 major bug, D = at least 1 critical bug, E = at least 1 blocker bug Effort to fix all bug issues.
Effort to fix all bug issues found on the code changed in the leak period Average complexity by file.
Number of code smells.
Number of new issues with severity (blocker,critical, major, minor) Number of new issues Rating given to your project related to the value of your Technical Debt Ratio. Effort to fix all maintainability issues. ID TM1 TM2 TM3 TM4 TM5 TM6 TM7 TM9 TM10 TM11 TM12 TM13 TM14 TM15 TM16 TM17 TM18 TM19 TM20 TM21
Coverage Hits Conditions By Line Covered Conditions By Line
Lines To Cover Lines To Cover On NEw Code Skipped Unit Tests Uncovered Conditions Uncovered Conditions On New Code Uncovered Lines On New Code Unit Tests Unit Tests Duration Unit Test Errors Unit Test Failures Unit Test Success Density Percent TM8
On each line of code containing some boolean expressions, the condition coverage simply answers the following question: ’Has each boolean expression been evaluated both to true and false?’.
This is the density of possible conditions in flow control structures that have been followed during unit tests execution.
On each line of code containing some boolean expressions, the condition coverage simply answers the following question: ’Has each boolean expression been evaluated both to true and false?’.
This is the density of possible conditions in flow control structures that have been followed during unit tests execution.
List of covered conditions.
Number of conditions by line.
Number of covered conditions by line.
It is a mix of Line coverage and Condition coverage.
Its goal is to provide an even more accurate answer to the following question: How much of the source code has been covered by the unit tests? It is a mix of Line coverage and Condition coverage.
Its goal is to provide an even more accurate answer to the following question: How much of the source code has been covered by the unit tests? Restricted to new / updated source code.
On a given line of code, Line coverage simply answers the following question: Has this line of code been executed during the execution of the unit tests?. It is the density of covered lines by unit tests: On a given line of code, Line coverage simply answers the following question: Has this line of code been executed during the execution of the unit tests?. It is the density of covered lines by unit tests Restricted to new / updated source code.
List of covered lines.
Number of lines of code which could be covered by unit tests Number of lines of code which could be covered by unit tests Restricted to new / updated source code.
Number of skipped unit tests.
Number of conditions which are not covered by unit tests.
Number of conditions which are not covered by unit tests.
Restricted to new / updated source code.
Number of conditions which are not covered by unit tests.
Restricted to new / updated source code.
Number of unit tests.
Time required to execute all the unit tests.
Number of unit tests that have failed.
Number of unit tests that have failed with an unexpected exception.
Test success density = (Unit tests - (Unit test errors + Unit test failures)) / Unit tests * 100