=Paper=
{{Paper
|id=Vol-2800/paper-09
|storemode=property
|title=About the Framework of Quality Evaluation Using SQuaRE
|pdfUrl=https://ceur-ws.org/Vol-2800/paper-09.pdf
|volume=Vol-2800
|authors=Tsuyoshi Nakajima
|dblpUrl=https://dblp.org/rec/conf/apsec/Nakajima20
}}
==About the Framework of Quality Evaluation Using SQuaRE==
https://ceur-ws.org/Vol-2800/paper-09.pdf
About the framework of quality evaluation
using SQuaRE
Tsuyoshi Nakajima
Shibaura Institute of Technology
Tokyo, Japan
tsnaka@shibaura-it.ac.jp
Abstract—Of the five core divisions of the SQuaRE series, the can perform quality evaluation adequately for itself. This may
revision study of the quality evaluation division has started. As lead to a project failure.
systems and software have become more complicated and To overcome this situation, it is important to have a
diversified in recent years, it is important to have a mechanism mechanism to ensure the quality evaluation of a system and
to ensure the quality evaluation of them by a whole society. In
order to do so, this revision aims to provide a framework for
software by a whole society, including objective comparison
systematic quality evaluation using quality models and quality techniques for its quality with other products in the industry,
measures in the SQuaRE series. Furthermore, the framework and third parties’ reliable quality certification.
will expand its scope of application to include four purposes: This paper proposes a quality evaluation framework as a
comparison, qualification to standard, conformity checking to basis to support such a mechanism, including systematic
requirements, and suitability evaluation in the market. This quality evaluation in three stages: quality measurement,
paper organizes and reports on the idea. quality rating, and quality evaluation, using quality models
and quality measures in the SQuaRE series. Furthermore, the
Keywords- quality evaluation, framework, quality measurement,
framework will expand its scope of application to deal with
quality rating
four purposes: comparison, qualification to standard,
conformity checking to requirements, and suitability
I. INTRODUCTION evaluation in the market. This paper organizes and reports on
ISO/IEC 25000 (SQuaRE) series is a set of international the idea.
standards for quality requirements and evaluation for a wide
range of systems and software [1]. Fifteen standards have II. REVISION OF QUALITY EVALUATION DIVISION
been published since 2006 to cover all the five core divisions.
A. Quality evaluation division in SQuaRE
Among them, the quality evaluation division (ISO/IEC
2504n) is currently studying its revision in the study group. SQuaRE divides the 25000 series into SQuaRE divides
In the development of a system or software, there are no the 25000 series into five core divisions and one extension
absolute criteria for its quality to be achieved[2]. This is division. Figure 1 shows how quality evaluation division
because the quality goal is a relative one needed in the context relating to the other division in SQuaRE. At the center are the
of social demands and competition with others. Actual
products need to fulfill different types of quality, such as Quality
Quality Quality
Requirements
performance efficiency and security, and so difficult trade- Definition
Engineering Evaluation
offs are often required to achieve those qualities in a balance.
On the other hand, if a system or software operates as an Support Support Support
important service in the society or works with the other
systems already operating in the society, the quality Quality Requirement Quality Evaluation
division (2503n) division (2504n)
evaluation may exceed the responsibility of one company Use Use
because of the significance of its impact on society[3] [4] .
Nevertheless, many of the quality evaluations for systems Quality Measurement division (2502n)
and software, even if it drives the society, are carried out Based on
based on the empirical knowledge of development
organizations for testing and ambiguous pass/fail criteria. Quality Model division (2501n)
Although it may be managed by an experienced development
Figure 1 Quality evaluation division in SQuaRE
organization with high quality awareness, a small
organization with little experience in quality control hardly
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
quality model division in 2501n and the quality measurement However, there exist many cases in which the quality
division in 2502n. The quality model division currently has evaluation based on the current ISO/IEC 25040 and 25041,
four quality models: quality in use and product quality cannot be applicable because it requires a full set of quality
(ISO/IEC 25010), IT service quality (ISO/IEC TS 25011), requirements for the target. Therefore, we decided to expand
and data quality (ISO/IEC 25012), each of which defines its scope so that it can deal with the cases without fixed
characteristics and sub-characteristics. The quality requirements. We categorize the assumed situations that
measurement division defines the quality measures to require quality evaluation and reconstructed the quality
measure the quality (sub)characteristics defined in each evaluation framework so that such cases can be included.
quality model. The quality evaluation division provides a We have identified the following four types of quality
framework and usage guides to support the quality evaluation evaluation:
process (partly, validation and verification activities in quality (T1) Quality evaluation for comparison
engineering) using the pairs of quality model and quality This is a type of quality evaluation:
measurement. - for the purpose of obtaining information for
B. Revision of quality evaluation division product/component/data selection
The SQuaRE Future Direction Study Group (2018-2019) - to finds out:
in ISO/IEC JTC1/SC7 WG6 concludes has improved the ISO Ø whether the candidate entities meet the setting
/ IEC 2504n quality evaluation department as follows[5]. criteria,
Ø which entities are better overall, and
[Problems] Ø what are the strengths and weaknesses of each
There is a great demand from the industry for methods entity,
and techniques to support how to plan inspections and testing - by comparing multiple entities based on quality
on quality, and concrete assessment based on their results. In requirements or general guidelines (including industry
addition, due to the development of 2502n and the revision standards)
of 25030, the quality requirements and their measurement - mainly performed by the acquirer or development
have been clarified, so modifications aligning with these will organization (integrator).
also be necessary. (T2) Quality evaluation for qualification to quality standard
[Proposed solution] This is a type of quality evaluation:
Ø Ensuring consistency with 2502n and 25030R - for the purpose of obtaining quality certification or
Ø Improving the concept of evaluation modules (EVs) (and information for quality improvement of the target
encouraging industries to provide ANNEXs) entity
Ø Guidelines for the following activities: - based on the standards in the industrial domain
² Organizing quality testing including inspections, - to find out:
aligning with 29119 (WG 26) Ø whether the entity meets the setting standards
² Comprehensive quality evaluation (e.g., for (mostly minimum set) and,
judgment of delivery) based on measurement Ø what are its strengths and weaknesses from the
results quality perspective
- mainly performed by the development organization
⁻ How to devise a set of quality measure suitable
(quality assurance) or an independent evaluation and
for evaluation certification organization.
⁻ Concept of evaluation (analysis of testing results,
(T3) Quality evaluation for conformity checking to
etc) and rating
requirements
² Selecting the right quality characteristics from some
evaluation goal This is a type of quality evaluation:
² Choosing an appropriate evaluation module for the - for the purpose of confirming the satisfaction of the
contract items
characteristics or to make a new evaluation
- based on the agreed requirements
module. - to confirm whether the product satisfies the quality
Based on this policy, we are currently studying a study requirements
group within WG6 for revision. - mainly performed by:
III. FRAMEWORK OF QUALITY EVALUATION Ø the ordering party (at its acceptance) or
Ø the development organization (quality
A. Quality evaluation and its types assurance)
(at the time of final inspection before delivery).
Quality evaluation is a systematic examination of the
extent to which an entity is capable of fulfilling stated and (T4) Quality evaluation for suitability to the market
implied needs. This is a set of activities to perform objective This is a type of quality evaluation:
confirmation work on the target to produce quality evaluation - for the purpose of obtaining information for
results. management decisions
- on in-house products / services
Ø based on the requirements derived from the target to identify the important quality characteristics and sub-
assumed stakeholder needs characteristics for the quality evaluation. For ICT products,
Ø to check how well the product meets the needs important quality (sub)characteristics differ depending on the
of the assumed market category of its system type. For example, user interface
- mainly performed by the development organization aesthetics is important for consumer products while
(quality assurance). operational operability and user error prevention are required
The revision of ISO/IEC 25040 needs to provide the more for mission-critical systems.
framework which covers these four types of quality Furthermore, even if the same quality characteristics/ sub-
evaluation. characteristics are selected, the quality properties to be
evaluated differ depending on the target category. For
B. Relationships between Quality models/measures and example, even with the same time behavior, it is necessary to
quality valuation evaluate "efficiency in processing requests" (throughput) on
Quality evaluation can be performed objectively, the server and "quickness of response to input events"
quantitatively, and comprehensively by using SQuaRE's (response time) on the client. Even with the same learning
quality model and quality measures. As shown in TABLE I, the ability, it is "the degree of operation without a manual" for
SQuaRE series defines a pair of quality model and quality consumer products, and "a lot of support for learning
measures for each entity type to be evaluated. operations" for mission-critical systems. These are measured
by different quality measures.
TABLE I Quality models for entity types and related documents
C. Flow of quality evaluation
Standard for Standard for
Entity Quality model
model measures Use of the SQuaRE quality model and quality measures,
Information Quality in use ISO/IEC ISO/IEC quality evaluation can be systematically performed
system 25010 25022 according to the flow shown in Figure 3, including three
ICT product Product quality ISO/IEC ISO/IEC
25010 25023 activities: quality measurement, quality rating, and quality
IT service IT service quality ISO/IEC TS ISO/IEC TS evaluation. These activities are described below.
25011 25025 (1) Quality measurement
Data Data quality ISO/IEC ISO/IEC
25012 25024 Quality measurement is an activity to measure the quality
property of the target entity using a quality measure under a
As shown in Figure 2, the Quality model classifies the certain measurement condition. Since a quality measure
quality that the target entity should have as quality defines the measurement method, whose concrete tasks
characteristics and subcharacteristics. Quality property is an includes testing or inspection (including static analysis) on the
attribute (information needs) of the target entity which you target entity, detailed measurement conditions (how many
want to measure. Quality measure measures a specific quality subjects to use, what to do with the parameters of the
property. In general, one quality property corresponds to execution environment, etc.) at that time must be given.
multiple quality measures. ISO/IEC 2502n provides quality measures corresponding
The quality evaluator must first select a quality model for to each quality model defined in ISO/IEC 2501n. If no
quality evaluation based on the type of the target entity. suitable quality measures for the target entity are found, a new
Furthermore, all the quality characteristics and quality measure that meets the requirements specified in
subcharacteristics defined in the model are examined for the ISO/IEC 25020 can be used.
(2) Quality rating
Quality rating is an activity to classify the values of
quality measures into preset rating levels. The rating level
gives the meaning of the degree of quality to the measured
value. An example of rating levels that can be used to
determine pass/fail.
Figure 4 shows an example of a typical pre-determined
rating level, in which four rating levels are set for the
measurement scale of quality measurements. Acceptable is a
level that has an unconditionally acceptable value as the lower
limit with the target value as the upper limit. When scoring,
the level of Acceptable can be divided into multiple levels. In
the case of Type 3, the allowable value uses the value
determined by agreement with the customer. Boundary is not
unconditional but acceptable by setting conditions. Not-
acceptable is a level that means unconditional fail. Excessive
quality (optionally defined) represents a level above the target
Figure 2 Relationship between quality model and measures value, which means unnecessarily high quality. In Figure 4,
results of 𝕡, and the evaluation value 𝐸(𝑐) for the quality
characteristic c is the minimum value among the evaluation
result of 𝕤. The quality evaluation value 𝐸!"#$%&& of the target
are calculated as the minimum value among the evaluation
results of 𝕔.
With this calculation method, the lowest level of rating
results for all quality properties goes to the overall result. If
all are boundary or higher, it passes (conditionally), and if
there is even one Not-acceptable, the whole fails.
In case of obtaining a score for quality of the target entity,
Figure 4 Typical preset rating levels
it is reasonable to assign a value to the quality level. This is
the lower limit values of Excessive quality, Acceptable, and because a quality measure basically has a value ranging from
Boundary are respectively 0.008, 0.02, and 0.025. 0 to 1 but cannot be used as a universal scale. To interpret the
levels into some score value, it is recommended that
(3) Quality evaluation Boundary should be 0, Acceptable should be further divided
Quality evaluation is an activity to determine pass/fail or into multiple levels and integer values of 1 or more should be
score regarding quality (sub)characteristics using the quality assigned to the levels, and Not-acceptable should be assigned
rating results for each quality property. The quality evaluation values of -1 or less.
is the integration of the quality rating results for the lower From the quality rating results, the quality evaluation
quality properties, or the integration of the quality evaluation score can be calculated for each stage by the following
results for the quality subcharacteristics immediately below. calculation. Here, wi, wj, and wk are weighting coefficients for
In case of determining pass/fail, the following calculation each stage. It is good to normalize each (adding up to 1 for
can be used. all).
𝐸(𝑠) = min (𝐿(𝕡))
𝐸(𝑠) = 6 𝑤' ∗ 𝑆𝑐𝑜𝑟𝑒(𝐿(𝑝' ))
𝐸(𝑐) = min-𝐸(𝕤)/
'
𝐸!"#$%&& = min (𝐸(𝕔))
𝑤ℎ𝑒𝑟𝑒 𝐸(𝑐) = 6 𝑤( ∗ 𝐸(𝑠( )
s : a quality subcharacteristic (
c : a quality characteristic 𝐸!"#$%&& = 6 𝑤) ∗ 𝐸(𝑐) )
𝕡 : a vector of all quality properties under s
)
𝕤: a vector of all the selected subcharacteristics under 𝑤ℎ𝑒𝑟𝑒 𝑆𝑐𝑜𝑟𝑒(𝑙) is a function to map from quality level to
c score.
𝕔 : a vector of all the selected characteristics
𝐸(𝑠 𝑜𝑟 𝑐): evaluation result of s or c IV. DIFFRENCES IN FOUR TYPES OF QUALITY
𝐿(𝕡): a vector of rating result for each element of 𝕡 EVALUATION
𝐸!"#$%&& : total quality evaluation result for the target Quality evaluation is classified into four types according to
its purpose, which is useful to clarify variation points on how
The evaluation value 𝐸(𝑠) for the quality
to conduct quality evaluation activities. We identified these
subcharacteristic s is the minimum value among the rating
points as below:
a) Model: Which quality model should be applied, Quality
in use, Product quality, Data quality, or IT service
quality (TABLE III “Model to use”)
b) Charas & measures: Which quality characteristics and
sub-characteristics are important, which quality
properties and quality measures should be used.
c) Criteria: Quality criteria to set quality levels, including
weights for scoring for quality properties, quality
subcharacteristics and quality characteristics.
TABLE II Variation points of quality evaluation and their factors
Variation QE type Entity type Industrial
point & category domain
a) Model V V
b) Chars & V V
measures
c) Criteria V V
d) Output V
Figure 3 Flow of quality evaluation_
TABLE III Four types of quality evaluation and their differences
Model to use Expected results
Number of Strength
Type Pass/ Candidate Source of quality criteria
targets QiU PQ DQ Score and
Fail selection
weakness
Quality requirements or
QE for
T1 Multiple M M M V V V general guidelines (including
comparison
industry setting standards)
QE for
determining to Standards in the industrial
T2 Single S M M V V
achieve quality domain
standard
QE for
T3 conformity to Single S M M V V Agreed requirements
requirements
Requirements derived from
QE for the assumed stakeholder
T4 suitability to Single M S S V V needs
the market （established during
development）
M: major, S: supplementary
quality rating, and quality evaluation, using quality models
a) Output: What kind of output are needed (TABLE III and quality measures in the SQuaRE series. Furthermore, the
“Expected results”) framework will expand its scope of application to include four
TABLE II shows variation points of quality evaluation purposes: comparison, qualification to standard, conformity
and their factors. The factors to determine these variation checking to requirements, and suitability evaluation in the
points includes the type of the quality evaluation, entity type market. This paper organizes and reports on the idea.
of the quality evaluation, and industrial domain of the target We hope to get many feedbacks from SQuaRE users on the
entity. ideas in this paper to successfully complete the revision of the
TABLE III shows each type has its own pattern of quality quality evaluation division.
model selection to use and expected results. Only T1 can
have multiple targets while the others cannot. Concerning REFERENCES
model to use, T1 can use all types of quality models, T2 and [1] Esaki, Kazuhiro, Motoei Azuma, and Toshihiro Komiyama.
"Introduction of quality requirement and evaluation based on ISO/IEC
T3 mainly use product quality (PQ) and data quality (DQ)
square series of standard." International Conference on Trustworthy
model, but supplementarily use quality in use (QiU) model. Computing and Services. Springer, Berlin, Heidelberg, 2012.
Contrary to that, T4 mainly use QiU model. Expected results [2] Cavano, Joseph P., and James A. McCall. "A framework for the
have four possibilities: pass / fail, candidate selection, score, measurement of software quality." Proceedings of the software
and strength and weakness. Which results are needed also quality assurance workshop on Functional and performance issues.
1978.
depends on the evaluation types, as shown in this table.
[3] Nakajima, Tsuyoshi, and Toshihiro Komiyama. "Applying Quality
Source of quality criteria describes what kinds of information Requirements Framework to an IoT System and its
source should be used to determine the quality criteria. Evaluation." International Journal on Advances in Internet
Technology Volume 12, Number 1 & 2, 2019 (2019).
V. CONCLUSION [4] Vargas, Iohan Gonçalves, Thiago Gottardi, and Rosana Teresinha
Vaccare Braga. "Approaches for integration in system of systems: a
This paper proposes a quality evaluation framework, which
systematic review." 2016 IEEE/ACM 4th International Workshop on
will be hopefully used for the revision of the quality Software Engineering for Systems-of-Systems (SESoS). IEEE, 2016.
evaluation division (ISO/IEC 2504n). The framework would [5] Nakajima, Tsuyoshi. "Study group report on square future direction."
serve as a basis to support systematic quality evaluation, CEUR Workshop Proceedings. Vol. 2545. CEUR-WS, 2020.
which consist of three activities: quality measurement,