Researcher scholarship (DT Level 2) sponsored by CNPq (Brazil)
[Context and Motivation] Satisfying Non-Functional Requirements (NFRs) in software development is challenging. There are many types of NFRs and each one requires specialized knowledge. Moreover, they can be correlated, which means that achieving one NFR can impact the achievement of another, positively or negatively. A common solution in the literature for helping developers to deal with these issues at the beginning of the development is the usage of NFR catalogs. Then, there is a need to know the current NFR catalogs to both support software development and to identify their challenges. [Question/Problem] However, the literature lacks a synthesized study of existing catalogs to help developers and researchers. Then, this work aims to identify what catalogs have been proposed in the literature and how they have been represented, de ned and evaluated. [Principal ideas/results] To achieve our goal, we performed a literature review through a systematic mapping study. We then identi ed 102 catalogs supporting 86 NFRs. They are mostly represented by Softgoals Interdependency Graphs and are built through literature review. [Contribution] Our results provide not only an overview of the area to guide researchers and practitioners who wish to build their catalogs, but also can help developers to reuse the knowledge presented in these catalogs. Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Dealing with NFRs that are quality characteristics (e.g. Usability, Security) during development brings several
issues to be overcome. One is that there are many types of them, and each requires specialized knowledge on
how to support it in software development. This knowledge is not always e ortlessly available for developers
especially for those who have no experience with NFRs [
A common solution in the literature for helping developers to ful ll NFRs is the usage of NFR catalogs [
To do this, we conducted a systematic mapping (SM) study [
According to the NFR Framework [
Subcharacteristics represent a more speci c knowledge about a NFR. For example, Security can be decomposed into three subcharacteristics, Con dentiality, Integrity and Availability. Methods represent development solutions intended to satisfy NFRs in a system. A method can be of any type, such as a function or some component in architecture, a design constraint or a design guideline. These methods can even be re ned into more speci c ones. Correlations represent the interactions in NFRs, which can be de ned in several levels: a) correlation between top-level NFRs, e.g., Usability hurts Security; b) correlation between subcharacteristics; c) correlation between method and top-level NFR; and d) correlation between method and subcharacteristic.
The literature has several NFRs catalogs. For example, there is a catalog for Security, Performance and
Accuracy in [
3.1
To perform our SM study, we followed a process with three main phases proposed by [
The aim of the planning phase is the de nition of a protocol. This document is usually composed of the following data: research goal, questions, databases, selection criteria, screening process and extraction strategy, explained as follows1. 3.1.1
The goal of this SM is to provide an overview of the literature regarding NFRs catalogs. To achieve this goal, we investigate the following research questions (RQs):
1Detailed information is available in https://github.com/great-ufc/SM-NFRsCatalogs/blob/master/ProtocolandResults.pdf
The strategy PICO (Population, Intervention, Comparison, and Outcomes) suggested by [
As databases, Web of Science and Scopus were chosen, since they have good coverage and stability. Also, Scopus cover other bases, such as IEEE. We also added one speci c database from the requirements area: Workshop on Requirements Engineering (WER). We have chosen it for three reasons: it represents an important event in the area where researchers usually publish NFRs catalogs, publications from there are not all indexed in the databases we previously choose, and it provides a search engine of its own, making the search for catalogs easier. 3.1.3
We have de ned the following inclusion (IC) and exclusion (EC) criteria to select the most suitable studies: IC1 - the study presents a NFR catalog; EC1 - the study is not written in English; EC2 - the study is not from Computer Science or Engineering related areas; and EC3 - the paper does not present a NFR catalog. 3.1.4
An extraction form was elaborated to be used for each selected paper in this phase. This form is organized into four parts. The rst one is related to data from the accepted papers and the other parts are related to data to answer the each RQ (RQ1, RQ2, RQ3, RQ4).
The extracted data was analyzed according to the type of data extracted. For RQ1 and RQ2, a quantitative
analysis was performed due to the nature of the data collected, which were numerical in a nominal scale. A
measure of central tendency (mode) and a measure of dispersion (frequency) were used to present the results.
Regarding RQ3 and RQ4, the data extracted from the literature provided a signi cant amount of textual
information. Therefore, we decided to use a qualitative method called Content Analysis (CA) to analyze the data
and, consequently, answer more appropriately the questions [
This phase was performed in ve steps, described as follows. 3.2.1
This step is about applying the search string (see Table 1) into the search machines of the libraries (Scopus and Web of Science). To select the most suitable set of papers after applying the search string, a screening step was performed using the following ve lters: (1) Applying EC1 and EC 2 in the found studies; (2) Excluding duplicate studies; (3) Applying the exclusion criteria in the abstract and title reading; (4) Applying the exclusion criteria in the introduction and conclusion reading, also apply the exclusion criteria by checking if there is an image of a catalog in the study (e.g., graph, table); and (5) Applying the exclusion criteria in the entire paper reading.
The lter 1 was performed in the own search machines (Web of Science and Scopus) whereas the other lters (2, 3, 4 and 5) were performed with supporting of Parsif.al tool2. Filters 3, 4 and 5 were executed by peers 2Free online collaborative tool for systematic reviews: https://parsif. one student and one expert in NFRs catalog. First, a student performed the lter alone. Then, an expert who has been working with NFRs catalogs reviewed only excluded papers. For example, if a student had excluded a paper by reading the abstract and title, the expert should review them to check whether he-she agrees with the exclusion. If not, the paper would be included in the study again. 3.2.2
We have used the backward snowballing procedures to complement the set of papers found by the database search. The backward procedure consists of checking the references list of a set of papers. In our case, the set of papers were the ones selected in the databases searches. We adapted the procedure from [?] and [?], which consisted of four lters. These lters are not totally similar to the lters we used earlier since the selection through backward snowballing is slightly di erent. First, we started by manually reviewing all references from each accepted paper from databases searches. Then, for each reference we applied the following lters: (1) Applying all exclusion criteria in the reference. In this case, we had to be more speci c since we were reading only titles, authors, venue and year. Thus, we accepted papers who presented any keyword from our search string; (2) Applying all exclusion criteria in the abstract reading; (3) Applying the exclusion criteria in the most relevant part of the papers (introduction, conclusion, and images of catalogs); and (4) Applying the exclusion criteria in the entire paper reading. 3.2.3
WER exists since 1998 and has been an advance for the Ibero-American community. This workshop provides a Google search engine3 that explores papers in all WER editions. Therefore, we applied the same search string from Table 1 on it and the same lters for the databases searches. 3.2.4
After performed all searches, the data extraction took place. In this way, rst, the extraction of data was done in the papers found in the electronic databases. Then the extraction was done in the papers of snowballing and, nally, the extraction was carried out in the papers from the workshop.
Regarding to the papers obtained by the databases, the same student who performed the lters during the ltering phase also performed the data extraction. Each extraction was reviewed by the expert. The extractions of the papers obtained by the snowballing and the workshop were made by the expert and reviewed by the student.
During extraction, the form was updated to include a data not considered during planning: \type of correlation". Through one of the obtained papers, it was possible to note that the correlations can appear between NFRs (e.g., Usability and Performance), which is called INTER-NFRs, or within a same requirement (e.g., Performance subcharacteristics con ict with one another), called INTRA-NFR. Thus, all papers that had already been extracted were reviewed again to extract this speci c data. 3.2.5
This phase was about analyzing the extracted data. RQ1 and RQ2 were indeed analyzed with the measures
Mode and Frequency. RQ3 and RQ4 were also analyzed using quantitative measures, but they were also analyzed
through a qualitative method, Content Analysis [
This systematic study obtained 53 papers (38 from databases and 15 from snowballing). The key results from the extraction and analysis of these papers can be viewed in Figure 1, which presents an overview for each RQ.
Answering RQ1, 102 catalogs were obtained. We realized that they cannot be classi ed as mutually exclusive
in three types of catalogs, as previously proposed by [
Subcharacteristics, T2 - Subcharacteristics and strategies, T3 - Subcharacteristics, Strategies and Correlations, T4 { Strategies, T5 - Strategies and Correlations, T6 { Correlations and T7 - Subcharacteristics and correlations.
Initially it was expected that the catalogs could be for a speci c area, but with this study, we realized that they can be proposed to particular areas, domains or artifacts. These di erent views are named as the \focus" of the catalog.
We also analyzed the NFRs supported by these catalogs. In total, 86 di erent NFRs were extracted. Performance was the most cited (34), followed by Security (29), Usability (23) and Reliability (14). Regarding subcharacteristics and methods, we found out 1269 and 1113, respectively.
Regarding correlations, this SM study found 473 positive and 395 negative correlations in total. They can occur not only between NFRs but also within the same NFR because their subcharacteristics or strategies may con ict with one another. In this way, a classi cation of correlation types was found out: INTER-NFRs and INTRA-NFR.
This last nding allowed us to establish a research opportunity related to the investigation of con icting (positive or negative) correlations. For example, when a catalog states two NFRs is con icting, and another catalog states they are cooperating.
Besides, this SM study found out that there are six levels of correlation. This level varies from the most generic, which are correlations directly between NFRs, to more speci c levels, which are correlations between strategies and strategies and characteristics.
The identi ed catalogs are represented in eight di erent ways (answering RQ2 ): SIG (including SIG adaptations), matrices, i* notation, tables, hierarchical structures, list, template, and pattern. With this information, we established a second research opportunity related to the catalog's representation. Future research could investigate these representations in a way that could indicate which would be the most appropriate to deal with each knowledge of a catalog of NFRs: subcharacteristics, strategies and correlations.
Although there are catalogs de ned by the authors themselves based on their experience, this study realized that the de nition of a catalog can be done in two steps (answering RQ3 ). First, it is necessary to collect the information, and then it is necessary to analyze this information to arrive at a more organized knowledge. Concerning the collection, there are six external sources by which the catalog creator can search for information: literature, existing catalogs, existing systems, experts, stakeholders and architects/developers. Also, there are seven techniques to extract the information from these sources: bibliographic review, systematic review, interview, questionnaire, questions patterns, goal-question-operationalization and measurement. Finally, there are ve techniques to analyze data: Collaboration Process with researchers, Consensus Meeting, Clustering Techniques, Content Analysis and a Technique based on Personal Construct Theory.
Therefore, a third opportunity is related to the de nition of a complete catalog. Although some papers explain their ways of constructing the proposed catalogs, a generic framework or process that provides a detailed guideline to create NFRs catalog was not found.
The last ndings were regarding the evaluation of a catalog (answering RQ4 ). Six evaluation approaches were found out: proof of concept, case study, survey, questionnaire, relative validity and controlled experiment. Seven evaluation purposes: designing a speci c system considering a NFR; redesigning an existing system to show improvement with the use of the catalog; building a new model, often reusing knowledge from the existing catalog; remodeling an existing catalog; arguing about the e ectiveness of the catalog; and support a system's implementation. Also, many catalogs are not evaluate themselves but used to support another proposal. Five supporting purposes were found out: to support evaluations, to be used in a model, to be used for comparison study, to be used in a tool, to be used in proposed approaches.
Then, a fourth and last opportunity is related to the evaluation of a proposed catalog. Few catalogs presented detailed evaluation procedures. Thus, there is a need to create a guideline to guide NFRs catalog evaluations. This paper has presented a study about NFRs' catalogs5. Through the SM method, together with quantitative and qualitative analysis (content analysis), it was possible to derive important information about the catalogs that are useful not only for researchers but also for developers who are concerned with satisfying NFRs in their systems. Thus, the contributions of this work have implications regarding both industry and academy.
For industry, this work generated a dataset containing more than 1000 subcharacteristics, 1113 implementation and design methods, 473 positive correlations, 395 negative correlations for 86 NFRs. We believe this base can be useful for developers to search several ways of implementing NFRs and thus reuse the knowledge gained from this research. For academy, this work generated a list of four high-level research opportunities that can improve the NFRs catalogs area. As a next step, we are using the results of this study to propose a process to de ne NFRs catalogs.
As future work, we plan to conduct a study in the industry to understand the usage of NFRs catalog in real scenarios. Then, the current challenges and issues the practitioners have faced in their work could be investigated. Furthermore, we encourage additional searches to possibly expand our database of NFRs catalogs. 5All materials and results from this SM study are available at https://github.com/great-ufc/SM-NFRsCatalogs