Introduction
Quality models are used in software engineering to enable a structured assessment
of the quality of a system according to quality attributes deemed important
for it [
12
]. To do so, quality models typically include a theory of how certain
software characteristics are related to higher level quality attributes and how
such characteristics and attributes can be measured and combined to enable a
quantitative quality assessment [
1
]. An example would be the theory used by
Bansiya et al. [
4
] that in an object-oriented system the characteristic of coupling
impacts extendability in the sense that high coupling has a negative impact on
extendability. Coupling is stated to be quantitatively measurable by counting
for each class to how many other classes it is directly related [
4
]. Such theories
therefore constitute the inner basis of a quality model and the degree to which
they are able to represent the reality ultimately determines the applicability and
usefulness of a quality model. AL-Badareen et al. [
1
], however, also state that
quality models are often formulated in a subjective manner and refer to a finding
from Kitchenham and Pfleeger [
25
] that “software quality models sufer from a
lack of rationale for the relationships between quality characteristics and how the
lowest levels properties are composed into an overall assessment of higher level
quality characteristics” [
1
]. The question of how quality models can be validated,
that means how well they represent reality, is therefore relevant and important
for providing quality models that are useful in practice.
In a recent study [
33
], we have formulated a quality model for cloud-native
application architectures that is based on the Quamoco quality meta-model
[
48
]. Quality attributes are called factors in the Quamoco context and they can
be either higher-level quality aspects or lower-level measurable product factors.
To formulate factors and their interrelationships, called impacts, we relied on
the ISO 25010 standard [
19
] in combination with suggestions from practitioner
books. Nevertheless, the resulting quality model includes a subjective notion. A
validation of the model and especially of the stated impacts would be beneficial
for future work that uses the model. In this work, our aim therefore is to review
existing approaches for validating quality models, especially in an empirical way.
Our quality model for cloud-native applications [
33
] serves as a use case for
which we want to derive implications for how a validation of such a newly created
quality model could be performed and which aspects need to be considered. The
contribution of this work is an overview of how and how often quality models
proposed in literature are validated. In addition, we review approaches that exist
to ensure the validity of quality models with implications for the formulation of
new quality models. To summarize this, we aim to answer the following research
questions:
RQ1: To what extent and how are quality models proposed in literature
validated?
RQ2: Which implications can be derived for a proper validation of newly
formulated quality models?
In the following, we provide some foundations on quality models in Sect. 2,
discuss related work in Sect. 3 and present our methodology in Sect. 4. We
describe our results and answers to our research questions in Sect. 5, before
concluding our work with an outlook in Sect. 6.
2
Hierarchical Software Quality Models
The term quality model is to some extent used ambiguously and can for example
also refer to a list of rules checked through static code analysis [
34
] where quality
is measured directly based on the number of rule violations found in a software.
In this work, however, the focus is on so-called hierarchical quality models [
4
]
where a hierarchy exists from lower-level measures to higher-level quality aspects.
Hierarchical quality models can integrate and interrelate multiple quality aspects
and enable a more detailed evaluation of software quality. In turn, theories are
needed to state the relationships between lower-level measures and higher-level
quality aspects mediated by software characteristics.
In broad fields, such as software engineering, theories are dificult to generalize
and often apply well only within certain contexts. Therefore, diferent quality
models exist for diferent domains, for example object-oriented systems [
4
],
embedded systems [
35
], Web services [
42
], or SOA architectures [
16
]. A contrast
to these specializations are the emerged standards for quality in software: ISO 9126
[
18
] and its successor ISO 25010 [
19
]. The consequence, however, is that these
standards mainly cover higher level quality attributes and advice on how to
measure and evaluate software quality. Nevertheless, the standards provide a
theoretical basis which has been validated through the structured definition and
refinement process involving a group of experts. But for context-specific quality
models including lower level quality attributes and measures the question of
how well the underlying theory maps to reality remains. The methodological
approach for ensuring the validity of a theory, and therefore of a quality model,
is referred to as validation. We distinguish validation from evaluation in this
work by considering evaluation as the approach of using a quality model for
evaluating the quality of a software. Although in literature, these two terms
are used inconsistently. Another distinction which is important for discussing
diferent validation approaches, is that between internal characteristics of a
software which are evaluated by analyzing the internal implementation of a
system and external characteristics of a software which can only be evaluated at
runtime when observing its behavior. This distinction is in line with the ISO 9126
[
18
] and ISO 25010 [
19
] standards which consider internal and external quality,
or Kitchenham et al. [
26
] who diferentiate between “externally visible properties”
and “internally visible properties”. For both types of characteristics impacts on
quality aspects can be stated, but typically impacts of external characteristics are
more intuitive. An example would be the externally visible uptime of a system for
which it can be stated that a high uptime has a positive impact on the availability
quality aspect. Especially considering internal characteristics, a clear rationale
for the relationships between quality attributes is therefore important, as also
stated by Wagner et al. [
48
] who developed Quamoco, a meta-model for quality
models, in which relationships between quality attributes are defined as impacts.
When formulating a quality model based on their meta-model such impacts need
to be stated based on valid reasoning, which can for example rely on logical
reasoning, previous literature, or empirical methods where empirical evidence
is considered to be statements from people with experience in the domain of a
quality model, collected through interviews or surveys. Empirical methods in
specific are also found important for the acceptance of quality models in practice
by Moody [
39
] who reviewed quality attributes of conceptual models (which is
a superset of models used in software engineering and therefore also includes
quality models). In conclusion, that means that also a validation of a hierarchical
software quality model should put a focus on the validity of the stated impacts
for a quality model which describe the relationships between the diferent factors,
also considering their importance in relation to each other. On the basis of these
properties of hierarchical quality models, we designed our approach for reviewing
quality models and validation approaches applied to them.
3
Methodology
To get an overview of quality models proposed in literature and have a basis
for our investigation, we rely on review papers that have already searched the
literature for software quality models in a structured way and that reported the
quality models they have found as results. As recent review papers, we found
the one by Galli et al. [
13
] (23 results) who aim to measure the relevance of
quality models, as well as the systematic mapping studies by Nistala et al. [
41
]
(40 results), focusing on types of model elements used, and by Yan et al. [
49
]
(31 results), focusing on the scope and maturity of quality models. Additionally,
we included the mapping study by Oriol et al. [
42
] (47 results), because of their
thematically related focus on web services. Because these review papers present
their results in diferent ways, we hereby introduce the term entry to consolidate
the results of these review papers in a generic way. An entry refers to a research
undertaking which may span one or several publications and may or may not
explicitly report a quality model. This way we can consider a quality model that
has been presented in one paper, but validated, applied, or evolved in additional
papers, as a single entry. And we can also include papers that do not explicitly
report a quality model, but for example methods for validation which have been
proposed independently from a specific quality model. After merging the results
from the review papers and removing duplicates, we had 121 entries as an initial
set for our investigation. Next, we classified all these entries according to the
following criteria:
– Type of contribution: Not all studies present hierarchical quality models
with explicit factors and impacts, only those that do were classified as
contributing a model. Others contribute a meta-model for quality models,
just a taxonomy which cannot be used as a quality model, or any kind of
method in the context of quality models (for example how to create a quality
model or apply it within an organization). Studies that present a specific
method for validating quality models were classified as validation-method.
– Characteristics considered: For each quality model we diferentiate between
the characteristics it considers, namely internal characteristics and external
characteristics of a software as described in Sect. 2.
– Rationale for non-trivial relationships between quality attributes: This
criterion covers the rationale which is used for stating impacts between factors
and their relative strength, however considering only non-trivial relationships
(In contrast, a trivial relationship would be that a lower latency positively
impacts performance eficiency). A rationale can be argumentation simply
based on logical implication or by relying on existing work (literature-based).
Empirical evidence can be provided by relying on a small set of experts
(empirical-experts), for example through interviews, or a structured survey
among a larger set of participants (empirical-survey) can be used. For a
quantification of the relative strengths of impacts, algorithmic evaluations are
sometimes used. If no rationale is provided or it is not possible to determine
it, we classified an entry as none.
It has to be noted that for each criterion multiple values could be assigned,
for example when both internal and external characteristics are considered or
a model together with a method is presented. To ensure that we do not miss
validations of quality models published separately after the publication of a
quality model, we performed a forward search by looking at the citations for each
entry. However, we restricted the forward search to a filtered list of entries which
only includes entries where the type of contribution includes a model or
evaluationmethod and the considered characteristics include internal characteristics. Our
focus on internal characteristics is due to our use case of the quality model
for cloud-native application architectures [
33
] which aims to evaluate software
architectures at design time based on architectural models. Formulating impacts
on quality aspects from internal characteristics is more dificult, because the
actual behavior of a system can only be observed at runtime. Therefore validations
for such stated impacts are especially important. For the forward search we used
SemanticScholar1 and searched the citations with the keywords evaluation or
validation. The forward search lead to an additional set of 11 publications.
Together with the filtered list of entries our final list which forms the basis of
our investigation thus consists of 50 entries and can be found online2. Detailed
information on the used literature, the search process, and classifications can also
be found in the corresponding repository for this site3. To answer our research
questions, we then quantitatively and qualitatively investigated these 50 entries
to gain insights and provide implications for validations of quality models.
1 https://www.semanticscholar.org/
2 https://r0light.github.io/qualitymodel-validations-review/
3 https://github.com/r0light/qualitymodel-validations-review
Results & Implications
Overall, we were interested in the types of rationale on which the theoretical
concepts of quality models are based. In Fig. 1 it can be seen that the majority of
quality models relies on previously published literature or uses logical implication
to infer conceptual relationships between factors. From a historical perspective it
can be seen that early quality models for example from Boehm [
5
], McCall [
37
],
or Dromey [
11
] have been formulated and described in comprehensive research
reports using experience and logical implication. Later works then relied on them
[
4, 43
] until the ISO standards [
18, 19
] became available and were again frequently
used as a basis [
3, 8–10, 14, 17, 21, 22, 24, 28, 32, 40, 44–47
]. This shows a
tendency to rely on existing literature for a sound theoretical foundation so that
a further validation is less important. Empirical approaches are nevertheless
also frequently used, especially for more recent domain-specific quality models
[
15, 35, 36, 38
] which are more dificult to cover with the more general standards.
To answer RQ1 we classified each entry according to whether an explicit
approach has been used to validate the proposed quality model. Out of the 50
entries in our result set, 40 do present a quality model and from these we found
18 which included an explicit validation approach. The approach and scope of the
validations however are diverse and we therefore further classified the validation
approaches according to the scope in focus. For this classification of validations
based on their scope we also considered the remaining 10 entries presenting
validation methods, independently from a specific quality model An overview
of the diferent validation approaches is provided in Fig. 2. On the left side of
Fig. 2 the elements of a quality model in the sense of the Quamoco meta-model
are shown while on the right side diferent perspectives on a software system are
shown. The arrows represent relations and the stethoscopes ( ) signify measures
attached to the diferent perspectives on a system. So, for example, a measure at
the source code level could be used to measure the degree to which a product
factor is present and the product factor impacts a quality aspect which in turn
might impact a higher level quality aspect. The numbered magnifiers ( ) show
the diferent scopes of validation approaches depending on which elements or
relations are in focus. Generally, a diferentiation can be done based on the amount
of information needed for a validation and the point in time when a validation is
suitable. In Table 1 additional details for the diferent validation approaches are
provided, including which elements of a quality model are required. It can be seen
that the validation of factors themselves (V1), the impacts between factors (V2)
and relative weights of impacts (V3) can be done solely based on factors proposed
for a quality model, and therefore also early in the design phase of a quality model
(early in the sense that only factors are defined). Using interviews with experts,
Gerpheide et al. [
15
] have defined and validated factors (V 1) and Mayr et al. [
35
]
early validated their model (V1, V2, V3) regarding comprehensibility, appropriate
level of abstraction, and consistent classifications through conducting multiple
workshops. Lampasona et al. [
31
] present an approach to rate the minimality
and completeness of factors (V1) using interviews with experts. Surveys among
practitioners have been used by Mehmood et al. [
38
] and Gerpheide et al. [
15
] to
validate impacts (V2) and their weights (V3) from product factors on quality
aspects by calculating the agreement of respondents regarding the existence and
type of impacts. In a similar way, also Khomh et al. [
23
] validated impacts and
their weights (V2, V3), although they investigated design patterns instead of
product factors. In our opinion, however, product factors are just a more general
construct through which also patterns can be expressed. An additional frequently
used approach for assigning weights to impacts on quality aspects (V3) is the
Analytical Hierarchical Process [
2, 7, 29
] in which experts compare impacts for
a quality aspect pairwise and based on that weights are calculated.
Quality model
Quality
aspect
Quality
aspect
Quality
aspect
Quality
aspect
V2
Product
factor
V1
Product
factor
Quality
aspect
V3
Product
factor
V7
V6
Expert
rating
V5
V4
Users
use
Deployment
App
is deployed
Source Code
src
Class
Lib
models
Architecture Model
Service
Service
DB
external
characteristics
internal
characteristics
In contrast to that are validations for completely defined quality models, that
means quality models with factors, measures, and relationships between them:
Kläs et al. [
27
] validated factors based on diversification (V 1) of measures and
overall validity through a comparison with expert ratings (V7). A comparison
with expert ratings (V7), also for relations specifically (V 3, V4), has been done
by Bansiya et al. [
4
] and Mayr et al. [
35
]. Braeuer et al. [
6
] validated measures
by comparing them with previously gained measurements (V4). Finally, also
considering external measures, Jung et al. [
20
] compared external measures with
user measures (V6) while Kvam et al. [
30
] and Yu et al. [
50
] correlated internal
measures with external measures (V5), such as productivity or performance.
Regarding RQ2 and the context of our quality model [
33
] we can therefore
state that in an early phase of quality model formulation, where not all elements of
a quality model are defined yet, surveys and interviews can be used for validation.
This fits the context of cloud-native applications, because it is a comparatively
new topic where less existing literature to rely on exists. Therefore, there is also
a lack of measures focusing on the architectural level of service interactions and
cloud deployment options [
33
] which makes validations where such measures are
needed dificult. Nevertheless, when all elements of a newly formulated quality
model are defined, the quality model should also be validated by comparisons
of complete evaluations with earlier evaluations or independent evaluations by
experts. In addition, it is common to rely on standards as a foundation which
can therefore also be recommended for new quality models. A challenge that
remains is the large number of factors [
33
] for which no implications can be
derived from the literature, because the considered quality models contained less
factors. Finally, an interesting observation is that we did not find any validations
for quality models taking architectural models into consideration.
6