Software quality has been a very critical issue in software engineering for the last four decades and still maintains its importance. Various quality models with different characteristics have been proposed for quality evaluation of software products. Also, static code analysis tools have been widely used to measure specific characteristics of software in fixed quality models. In order to combine the isolated views on software quality and its evaluation (SQiE), metamodels that formalize whole or partial aspects of software quality models as a base for tool support or further research have been proposed in literature. In this paper, a Systematic Literature Review (SLR) to comprehensively examine the meta-models proposed for SQiE in scientific literature is reported, with an aim to understand their purpose of use, content, and structure. The most-known seven digital libraries were searched, and 28 studies were identified out of 114 initially selected and 6488 initially retrieved in this area between 1997-2020. Results show that majority of meta-models are for general purposes, take ISO 9126 as reference and propose for various types of software. Most of them evaluate quality objectively using metric data and provide quantitative results. Majority of them are structured to enable extension with new quality models.
It is a challenging task to define the term ‘quality’ for a specific product or service,
since its meaning is different for customer, manager, tester, user, developer, etc. This
is also true in the field of software engineering, because stakeholders have different
expectations from software products or services. There are many definitions of the
term ‘software quality’ by different sources. For instance, IEEE [
Nevertheless, the lack of a common ground to define the term ‘software quality’
neither decreases its criticality nor the need for its evaluation in the field. In addition,
it increases the quality related costs, some of which are catastrophic. Poor quality of
software in sensitive systems, such as real-time systems and control systems, may
lead to loss of human life, permanent injury, mission failure, dissatisfaction of the
users, and increase in the cost of maintenance or financial loss [
Although many quality models have been proposed in the field as outlined above,
they are not comprehensive and complete and there are still many issues which pose
challenges to their adoption [
In coping with the ambiguities and problems mentioned, there seems a need to
represent the concepts of software quality and evaluation more formally, and
metamodeling can be a suitable vehicle to do this. Researchers have proposed meta-models
based on existing quality models, e.g., [
The rest of the paper is organized as follows: Section 2 discusses related work. Research method in conducting this SLR is described in Section 3. Results of SLR are presented in Section 4. Important findings are discussed in Section 5. Threat to validity is discussed in Section 6 and finally, conclusion is stated in Section 7. 2
Studies that systematically analyzed models for quality or its assessment are summarized below, since meta-models are created considering those models.
Nistala et al. [
Tomar and Thakare [
Yan et al. [
Cote et al. [
Miguel et al. [
Consequently, by looking at the studies that overview software quality models in the literature, it is observed that some of them systematically investigate the quality models and some of them are limited to comparing only a few quality models. Meanwhile, no systematic study on the studies proposing meta-models for SQiE has been found. Therefore, this study aims to address this gap by a systematic literature review. 3
In this study, Systematic Literature Review [
Before determining the search string to be used in this study, searches were conducted with various combinations of search keywords to obtain the most relevant studies. As a result, the following search string was obtained: ("Meta model" OR "Meta-model") AND ("software quality") AND
("evaluation" OR "assessment" OR "measurement")
We used this search string to retrieve publications in the following digital libraries:
Google Scholar, ScienceDirect, Scopus, ACM, Web of Science, IEEE Xplore, and
Springer. Among these libraries, Google Scholar was used to increase the coverage of
our search. Studies initially obtained from the digital libraries are shown in Table 2. It
was observed that the number of studies in Google Scholar was the highest before
applying the inclusion/exclusion criteria and Springer followed it. The least number
of studies was obtained from Web of Science database.
By applying the exclusion criteria, many studies from the 1st set of publications were
eliminated. SLR guideline [
Inclusion criteria were applied to ensure that the publications were within the scope of our study. It was considered that the studies are concerned with SQiE, they propose meta-models accordingly, and they provide mature enough graphical representations of the meta-models including a few entities and relationships between these entities.
In order to apply the inclusion/exclusion criteria, both authors assigned numbers to the articles as ‘0’ or ‘1’, independently, by reviewing title, abstract and keywords. The number ‘0’ meant the study was to be excluded, while the number ‘1’ meant the study was to be included. Articles were included when a total of ‘2’ was obtained, excluded when a total of ‘0’ was obtained, and marked for later discussion when a total of ‘1’ was obtained. As a result, 114 studies were identified, including the conflicting studies (the 2nd set of publications). After a series of discussions, conflicts were resolved.
Both authors reviewed the full texts of the studies in the 2nd set of publications,
applying the inclusion and exclusion criteria, and 24 studies were selected (as the 3rd set
of publications). Then, backward and forward snowballing [
Before starting data extraction, the initial categories for each RQ shown in Table 1 were determined by the authors. When there was a need to add a new category during data extraction, this was discussed by the authors and added as a category item in data extraction sheet. The first author answered the RQs by reading the full texts of the studies, and recorded elicited information by the categories in a tabular form. Then, peer-review was performed by the second author and any conflict between the authors was resolved in a series of discussions. Data extraction sheet can be reached by the following link: https://tinyurl.com/ybz2ybky 4 4.1
RQ 1.1: A meta model is defined as “a model of a well-defined language” [
RQ 1.2: As shown in Fig. 2 (b), more than half of the meta-models were proposed to cover all types of software. Only 2 of them were proposed for open source software (OSS), 2 of them for commercial software (COTS), and 1 of them for microservices (MS) software. In addition, 5 meta-models were proposed for web services (WS).
RQ 1.3: Time is a crucial factor in reducing software evaluation costs. Tools that allow automatic evaluation and eliminate manual effort have critical importance during software quality evaluation. As shown in Fig. 2 (c), less than half of the studies developed tools to reduce the effort spent on software quality evaluation. 7, 25%
21, 75% Generic
Specific A Fig. 2. Basic characteristics of meta-models (RQ1): (a) Percent distribution of main purpose, (b) Percent distribution of types of software products targeted, (c) Percent distribution of whether meta-models are taken as the base for tool development. 4.2
RQ 2.1: Existing quality models were taken as reference in creating most of the metamodels. As shown in Fig. 3 (a), more than half of the proposed meta-models took ISO 9126 as reference. Similarly, 6 of them took ISO 25010, 8 of them took McCall’s model, 9 of them took Boehm’s model, 6 of them took Dromey’s model, 3 of them took IEEE 10610 as reference, and 6 of them took other existing meta-models as reference. In 3 studies, quality models referenced are not explicitly specified. It should also be noted that some meta-models took one or more quality models as reference. Earlier meta-models were based on McCall and Boehm models, while later metamodels were based on ISO 9126. All referenced quality models are hierarchical in structure. Therefore, most of the meta-models examined the hierarchical structure of the quality models, and their objective was to capture the knowledge on software quality present in the hierarchical quality models in one comprehensive model.
RQ 2.2: As shown Fig.3 (b), 11 of meta-models serve for SQiE with respect to all models taken as reference while 6 of them do not. The rest of them does not explicitly specify if they serve for SQiE with respect to all quality models taken as reference.
RQ 2.3: As shown in Fig. 3 (b), while 11 studies map the terminology of software quality models taken as reference to the terminology of the meta-models they propose, 5 studies do not perform this mapping. Also, 7 studies do not explicitly specify whether they mapped the terminology of the quality models they referenced, and 5 studies make this mapping only partially.
RQ 2.4: As shown in Fig. 3 (b), 10 studies map the structure of the software quality model taken as reference to the structure of the meta-models they propose, and 6 studies do not perform this mapping. In addition, 10 studies do not explicitly specify whether they map the structure of the quality models they referenced, and 3 studies make this mapping only partially. 16 14 12 10 8 6 4 2 0
RQ2.3
RQ2.4 b Fig. 3. (a) Software quality model(s) taken as reference for meta-models (RQ2.1), (b) Number of studies for RQ2.2, RQ2.3 and RQ2.4. 4.3
RQ 3.1: We observed 11 meta-models use Goal-Question-Metric (GQM) method and 3 meta-models use Factor-Criteria-Metric (FCM) method for quality evaluation. The remaining (50%) studies do not explicitly specify the methods they use.
RQ 3.2: As shown in Fig. 4 (a), 14 meta-models evaluate quality objectively using only metric data. Only 1 study does not explicitly specify whether it uses metric data or user opinion in evaluation. The rest (46%) of studies make both objective and subjective evaluations considering both metric data and user opinions. In almost all studies, meta-models use metric data in quality evaluation. To analyze the metrics, they either develop their own tools or import data using the existing tools.
RQ 3.3: We observe from Fig. 4 (b) that only 1 study gives qualitative result after evaluation, and 2 studies do not explicitly specify whether they provide quantitative or qualitative results. Also, 7 studies give both quantitative and qualitative results after evaluation. The rest of the studies (63%) provide quantitative results only. These findings show that meta-models are generally aimed to produce quantitative values in order to see concrete results after evaluating software quality.
RQ 3.4: As shown Fig. 4 (c), 3 studies use expert decision, 1 study uses survey, 2 studies use fuzzy logic, 1 study uses analytic hierarchy process (AHP), 2 studies use statistical methods, 2 studies use human decision, and 1 study uses machine learning (ML) to analyze data in their meta-models. The remaining studies (42.8%) do not use any of these data analytics techniques.
RQ 3.5: As shown in Fig. 4 (d), 9 studies provide evaluation results as an index in
range [
7, 26% 1, 4%
RQ 3.6: We observed that 12 studies support SQiE in software delivery, and 10 studies do not support it in a specific phase. Only 1 study supports SQiE in implementation and 1 study in requirements; 4 studies do not explicitly specify a phase. Since product delivery is important for end users, it is the most addressed phase.
RQ 3.7: We observed that only 5 studies support SQiE in the evolution of software while the rest supports it at a single point. There are many changing factors in software lifecycle. Since changes are inevitable, the software needs to keep up with those factors. However, majority of meta-models support SQiE at a single point in time. 4.4
RQ 4.1: Most quality models have special structures such as hierarchical. Like
quality models, meta-models also can have special structures. For instance, the meta-model
in study [
RQ 4.2: Meta-models are diagrams that contain entities and relationships between entities. Entities for SQiE were sometimes defined by different terms in meta-models, even though they were defined for the same concept. Therefore, there seems a confusion in literature in the terminology proposed for SQiE. The terms used for concepts represented as entities in meta-models were analyzed in detail and assigned to category groups shown in Table 3. Accordingly, ‘quality attribute’ and ‘measure’ are the most commonly used entities, while ‘requirement’, ‘scale’ and ‘instrument’ are the least frequently addressed entities in the meta-models.
RQ 4.3 Meta-models for SQiE are expected to cover all aspects of quality models and be flexible enough to apply with modifications. They should be organized according to the needs of users and enable inclusion of new models. Majority (24) of the meta-models were structured to enable extension with new quality models when needed. Also, we observed that only 2 studies do not allow to define new models and 2 other studies do not explicitly specify if they enable to define new quality models. 4.5
Some studies developed their own tools based on the meta-models as addressed in response to RQ 1.3. In addition, the majority of the meta-models use metric data for evaluation as specified in response to RQ 3.2, and obtain quantitative values as a result as addressed in RQ 3.3. In order to obtain quantitative values, the tools based on the meta-models are related to some external analysis tools. Among the meta-models examined in this SLR, 6 use batch import to acquire data from external analysis tools, while 6 analyze data after automatic direct transfer from other repositories. Also, 7 meta-models allow manual data entry. Some of the meta-models support one or more data acquisition methods mentioned above. However, many studies (15) do not explicitly specify how their meta-models acquire data. 4.6
Validation of the proposed meta-models is one of the most important steps in the studies, since research validation is vital to ensure the research is clean, correct and useful. According to the findings obtained, 13 meta-models were validated by designing case studies and 6 meta-models by performing toy experiment. Also, 4 studies conducted peer reviews by experts and 1 study used pilot project application to validate its meta-model. While 5 studies did not explicitly mention the method of validation for their proposals, only 1 study did not use a validation method. A study might have been validated with peer reviews along with a case study or toy experiment. 4.7
RQ 7.1: Research methods employed in proposing the meta-models were classified
according to the scheme by Wieringa [
RQ 7.2: In the studies examined, it is mentioned that many challenges were faced
while creating the meta-models. These challenges, which arose especially from the
quality models taken as reference, were classified based on the suggestions of [
C2: Partially defined: Most quality models are outlined but not fully developed. All
define measurable concepts, some of them also attributes, few of them include (most
often partial) measures and scarcely any defines decision criteria or indicators [
C4: Lack of clarity in interdependencies and measure interpretations: In most quality
models that are not based on theory, the degree of influence of individual internal
quality factors on the quality in use of the application, as well as their interdependencies, are
not well established [
11 (30%)
Regarding the basic characteristics (RQ1), 75% of meta-models were proposed for generic purposes. This situation coincides with the purpose of proposing meta-models in that they should have a general feature shaped according to the purpose of the user. At the same time, the vast majority of meta-models were proposed to evaluate custom software products. However, when the empirical studies for testing the validity of meta-models are examined, it is observed that almost no meta-model used the specific metrics of OSS such as mailing list, user reviews, etc. Therefore, it is not clear whether these meta-models are fully applicable to evaluate the quality of OSS products.
One of the most important contribution of the studies is tool support. Since code analysis tools measure specific features of software quality in fixed quality models, it might be necessary to combine the heterogeneous results of code analysis tools to measure quality more comprehensively and completely. In such a case, inconsistent results can be obtained. To eliminate this, tool support for meta-models is essential. However, more than half of the proposed meta-models did not provide tool support.
Regarding the quality models taken as reference while developing the meta-models
(RQ2), ISO 9126 was the leading model in 57% of the studies since: 1) it provides a
comprehensive specification and evaluation model for software product quality, 2) it
explicitly addresses user needs of a product by allowing a common language for
specifying user requirements by various stakeholders, and 3) it evaluates quality of
software products objectively based on observation and not opinion [
Regarding the basic characteristic of SQiE (RQ3), Goal-Question-Metric method was used for quality evaluation in the meta-models. Also, metric data was very important in evaluating software, since majority of the meta-models support objective and quantitative evaluations and obtain numerical values as the result of their evaluations. However, there are very few meta-models that used the well accepted data analytics methods such as ML, Fuzzy, AHP etc. Generally, meta-models analyzed data independent of these important methods.
There are many factors that change throughout the software lifecycle. Therefore,
changes to software are inevitable, as the software needs to keep up with these
changing factors. However, most of the meta-models (82%) did not consider the evolution
of the software product and only focused on the last version when evaluating quality.
This may be desirable for end users, but it creates a difficulty for developers in
monitoring the quality while maintaining the software product. Considering that 60% of
the total cost in software projects is spent for product maintenance [
Regarding the structure of the meta-models (RQ4), 43% of them are layered with a special structure, and the rest does not have a special structure. Majority (86%) of the meta-models are structured to enable extension with new quality models, which is in line with the purpose of meta-modeling. In addition, regarding the content of the meta-models, ‘quality attribute’ and ‘measure’ are the most commonly used entities.
Regarding data acquisition (RQ5), while almost half (13) of the meta-models adopt batch import (6), automatic direct transfer from other repositories (6), and manual entry (7), the rest of them (15) do not explicitly specify how they acquire data. Please note that a meta-model may support more than one data acquisition method.
With respect to the empirical evidence (RQ6), meta-models were validated by either hypothetical case studies or toy experiments. A real-world case was not used for validation of any meta-model. Therefore, it is observed that there is no empirical evidence sound enough to demonstrate usefulness of the meta-models for SQiE.
Regarding the research methods employed (RQ7) in developing the meta-models,
almost half of the studies (47%) are classified as solution proposal. It means that
meta-models were generally proposed as solutions for SQiE and argued for its relevance,
without a full-blown validation. Therefore, they offered either a weak, hypothetical
example or demonstration. Also, regarding the challenges faced in creating the
metamodels, it was observed that majority of the challenges arose especially from the
quality models taken as reference. Because most quality models are not based on
theory, the degree of influence of individual internal quality factors on the quality in
use of the application as well as their interdependencies are not well established [
The purpose of this SLR was to focus on and analyze specific studies in literature. This situation may lead to problems related to the acquisition of all relevant studies and hence the validity of the results. Main threats might have been due to the choice of search string as well as biases in study selection, data extraction, and classification.
In order to select the search string, first we piloted the searches and reviewed the results in multiple iterations; then, we updated search keywords after the reviews. To alleviate the threat of the search string, backward and forward snowballing were applied to obtain potentially relevant publications that were missed by the search string.
There might have been researcher bias in identifying the exclusion/inclusion criteria for choosing the studies. To alleviate the threat in study selection, a systematic voting process was carried out by the authors independently to decide which articles should be included in the pool. In this way, it was aimed to minimize subjectivity and bias in selecting the publications. The studies that the authors agree on were included in the pool. Other studies were marked for later discussion, and conflicts were solved between the authors after a series of discussions. Non-agreed studies were excluded.
To cope with the threat of data extraction, the first author answered the research questions by reading the full texts of the studies in the final pool, and recorded elicited information by the categories in a tabular form. Then, peer-review was performed by the second author in detail and if there was a conflict between authors, it was resolved after a series of discussions among the authors.
Some studies were used as reference to alleviate inconsistency in the schemes of
classification. For example, research methods used in developing the meta-models
were examined in RQ 7.1. While answering this research question, studies were
classified according to the classification scheme by Wieringa [
In addition, during SLR process depicted in Figure 1, search string, search process and selection criteria were piloted first and refined in a series of iterations. Whenever an update was required after an iteration, the whole process was repeated again. 7
In this paper, the results of an SLR study are reported in order to examine comprehensively the content and structure of the meta-models proposed for SQiE in scientific literature. The most-known seven digital libraries were searched, and 28 studies out of 114 initially selected and 6488 initially retrieved were identified for further analysis with respect to the inclusion/exclusion criteria. These primary studies were analyzed with respect to seven research questions. To the best of our knowledge, this is the first SLR study conducted on the meta-models for SQiE.
This SLR study might help researchers and practitioners in understanding the state of the art on the meta-models proposed for SQiE. The set of 28 studies in the pool serves as a reference catalogue for researchers, and the detailed analysis provides a kind of guide in seeing the weak aspects to propose further meta-models or design further studies. The studies that propose tools based on the meta-models, on the other hand, might be useful for practitioners for investigation and adoption for their use.