About the framework of quality evaluation using SQuaRE Tsuyoshi Nakajima Shibaura Institute of Technology Tokyo, Japan tsnaka@shibaura-it.ac.jp Abstract—Of the five core divisions of the SQuaRE series, the can perform quality evaluation adequately for itself. This may revision study of the quality evaluation division has started. As lead to a project failure. systems and software have become more complicated and To overcome this situation, it is important to have a diversified in recent years, it is important to have a mechanism mechanism to ensure the quality evaluation of a system and to ensure the quality evaluation of them by a whole society. In order to do so, this revision aims to provide a framework for software by a whole society, including objective comparison systematic quality evaluation using quality models and quality techniques for its quality with other products in the industry, measures in the SQuaRE series. Furthermore, the framework and third parties’ reliable quality certification. will expand its scope of application to include four purposes: This paper proposes a quality evaluation framework as a comparison, qualification to standard, conformity checking to basis to support such a mechanism, including systematic requirements, and suitability evaluation in the market. This quality evaluation in three stages: quality measurement, paper organizes and reports on the idea. quality rating, and quality evaluation, using quality models and quality measures in the SQuaRE series. Furthermore, the Keywords- quality evaluation, framework, quality measurement, framework will expand its scope of application to deal with quality rating four purposes: comparison, qualification to standard, conformity checking to requirements, and suitability I. INTRODUCTION evaluation in the market. This paper organizes and reports on ISO/IEC 25000 (SQuaRE) series is a set of international the idea. standards for quality requirements and evaluation for a wide range of systems and software [1]. Fifteen standards have II. REVISION OF QUALITY EVALUATION DIVISION been published since 2006 to cover all the five core divisions. A. Quality evaluation division in SQuaRE Among them, the quality evaluation division (ISO/IEC 2504n) is currently studying its revision in the study group. SQuaRE divides the 25000 series into SQuaRE divides In the development of a system or software, there are no the 25000 series into five core divisions and one extension absolute criteria for its quality to be achieved[2]. This is division. Figure 1 shows how quality evaluation division because the quality goal is a relative one needed in the context relating to the other division in SQuaRE. At the center are the of social demands and competition with others. Actual products need to fulfill different types of quality, such as Quality Quality Quality Requirements performance efficiency and security, and so difficult trade- Definition Engineering Evaluation offs are often required to achieve those qualities in a balance. On the other hand, if a system or software operates as an Support Support Support important service in the society or works with the other systems already operating in the society, the quality Quality Requirement Quality Evaluation division (2503n) division (2504n) evaluation may exceed the responsibility of one company Use Use because of the significance of its impact on society[3] [4] . Nevertheless, many of the quality evaluations for systems Quality Measurement division (2502n) and software, even if it drives the society, are carried out Based on based on the empirical knowledge of development organizations for testing and ambiguous pass/fail criteria. Quality Model division (2501n) Although it may be managed by an experienced development Figure 1 Quality evaluation division in SQuaRE organization with high quality awareness, a small organization with little experience in quality control hardly Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). quality model division in 2501n and the quality measurement However, there exist many cases in which the quality division in 2502n. The quality model division currently has evaluation based on the current ISO/IEC 25040 and 25041, four quality models: quality in use and product quality cannot be applicable because it requires a full set of quality (ISO/IEC 25010), IT service quality (ISO/IEC TS 25011), requirements for the target. Therefore, we decided to expand and data quality (ISO/IEC 25012), each of which defines its scope so that it can deal with the cases without fixed characteristics and sub-characteristics. The quality requirements. We categorize the assumed situations that measurement division defines the quality measures to require quality evaluation and reconstructed the quality measure the quality (sub)characteristics defined in each evaluation framework so that such cases can be included. quality model. The quality evaluation division provides a We have identified the following four types of quality framework and usage guides to support the quality evaluation evaluation: process (partly, validation and verification activities in quality (T1) Quality evaluation for comparison engineering) using the pairs of quality model and quality This is a type of quality evaluation: measurement. - for the purpose of obtaining information for B. Revision of quality evaluation division product/component/data selection The SQuaRE Future Direction Study Group (2018-2019) - to finds out: in ISO/IEC JTC1/SC7 WG6 concludes has improved the ISO Ø whether the candidate entities meet the setting / IEC 2504n quality evaluation department as follows[5]. criteria, Ø which entities are better overall, and [Problems] Ø what are the strengths and weaknesses of each There is a great demand from the industry for methods entity, and techniques to support how to plan inspections and testing - by comparing multiple entities based on quality on quality, and concrete assessment based on their results. In requirements or general guidelines (including industry addition, due to the development of 2502n and the revision standards) of 25030, the quality requirements and their measurement - mainly performed by the acquirer or development have been clarified, so modifications aligning with these will organization (integrator). also be necessary. (T2) Quality evaluation for qualification to quality standard [Proposed solution] This is a type of quality evaluation: Ø Ensuring consistency with 2502n and 25030R - for the purpose of obtaining quality certification or Ø Improving the concept of evaluation modules (EVs) (and information for quality improvement of the target encouraging industries to provide ANNEXs) entity Ø Guidelines for the following activities: - based on the standards in the industrial domain ² Organizing quality testing including inspections, - to find out: aligning with 29119 (WG 26) Ø whether the entity meets the setting standards ² Comprehensive quality evaluation (e.g., for (mostly minimum set) and, judgment of delivery) based on measurement Ø what are its strengths and weaknesses from the results quality perspective - mainly performed by the development organization ⁻ How to devise a set of quality measure suitable (quality assurance) or an independent evaluation and for evaluation certification organization. ⁻ Concept of evaluation (analysis of testing results, (T3) Quality evaluation for conformity checking to etc) and rating requirements ² Selecting the right quality characteristics from some evaluation goal This is a type of quality evaluation: ² Choosing an appropriate evaluation module for the - for the purpose of confirming the satisfaction of the contract items characteristics or to make a new evaluation - based on the agreed requirements module. - to confirm whether the product satisfies the quality Based on this policy, we are currently studying a study requirements group within WG6 for revision. - mainly performed by: III. FRAMEWORK OF QUALITY EVALUATION Ø the ordering party (at its acceptance) or Ø the development organization (quality A. Quality evaluation and its types assurance) (at the time of final inspection before delivery). Quality evaluation is a systematic examination of the extent to which an entity is capable of fulfilling stated and (T4) Quality evaluation for suitability to the market implied needs. This is a set of activities to perform objective This is a type of quality evaluation: confirmation work on the target to produce quality evaluation - for the purpose of obtaining information for results. management decisions - on in-house products / services Ø based on the requirements derived from the target to identify the important quality characteristics and sub- assumed stakeholder needs characteristics for the quality evaluation. For ICT products, Ø to check how well the product meets the needs important quality (sub)characteristics differ depending on the of the assumed market category of its system type. For example, user interface - mainly performed by the development organization aesthetics is important for consumer products while (quality assurance). operational operability and user error prevention are required The revision of ISO/IEC 25040 needs to provide the more for mission-critical systems. framework which covers these four types of quality Furthermore, even if the same quality characteristics/ sub- evaluation. characteristics are selected, the quality properties to be evaluated differ depending on the target category. For B. Relationships between Quality models/measures and example, even with the same time behavior, it is necessary to quality valuation evaluate "efficiency in processing requests" (throughput) on Quality evaluation can be performed objectively, the server and "quickness of response to input events" quantitatively, and comprehensively by using SQuaRE's (response time) on the client. Even with the same learning quality model and quality measures. As shown in TABLE I, the ability, it is "the degree of operation without a manual" for SQuaRE series defines a pair of quality model and quality consumer products, and "a lot of support for learning measures for each entity type to be evaluated. operations" for mission-critical systems. These are measured by different quality measures. TABLE I Quality models for entity types and related documents C. Flow of quality evaluation Standard for Standard for Entity Quality model model measures Use of the SQuaRE quality model and quality measures, Information Quality in use ISO/IEC ISO/IEC quality evaluation can be systematically performed system 25010 25022 according to the flow shown in Figure 3, including three ICT product Product quality ISO/IEC ISO/IEC 25010 25023 activities: quality measurement, quality rating, and quality IT service IT service quality ISO/IEC TS ISO/IEC TS evaluation. These activities are described below. 25011 25025 (1) Quality measurement Data Data quality ISO/IEC ISO/IEC 25012 25024 Quality measurement is an activity to measure the quality property of the target entity using a quality measure under a As shown in Figure 2, the Quality model classifies the certain measurement condition. Since a quality measure quality that the target entity should have as quality defines the measurement method, whose concrete tasks characteristics and subcharacteristics. Quality property is an includes testing or inspection (including static analysis) on the attribute (information needs) of the target entity which you target entity, detailed measurement conditions (how many want to measure. Quality measure measures a specific quality subjects to use, what to do with the parameters of the property. In general, one quality property corresponds to execution environment, etc.) at that time must be given. multiple quality measures. ISO/IEC 2502n provides quality measures corresponding The quality evaluator must first select a quality model for to each quality model defined in ISO/IEC 2501n. If no quality evaluation based on the type of the target entity. suitable quality measures for the target entity are found, a new Furthermore, all the quality characteristics and quality measure that meets the requirements specified in subcharacteristics defined in the model are examined for the ISO/IEC 25020 can be used. (2) Quality rating Quality rating is an activity to classify the values of quality measures into preset rating levels. The rating level gives the meaning of the degree of quality to the measured value. An example of rating levels that can be used to determine pass/fail. Figure 4 shows an example of a typical pre-determined rating level, in which four rating levels are set for the measurement scale of quality measurements. Acceptable is a level that has an unconditionally acceptable value as the lower limit with the target value as the upper limit. When scoring, the level of Acceptable can be divided into multiple levels. In the case of Type 3, the allowable value uses the value determined by agreement with the customer. Boundary is not unconditional but acceptable by setting conditions. Not- acceptable is a level that means unconditional fail. Excessive quality (optionally defined) represents a level above the target Figure 2 Relationship between quality model and measures value, which means unnecessarily high quality. In Figure 4, results of 𝕡, and the evaluation value 𝐸(𝑐) for the quality characteristic c is the minimum value among the evaluation result of 𝕤. The quality evaluation value 𝐸!"#$%&& of the target are calculated as the minimum value among the evaluation results of 𝕔. With this calculation method, the lowest level of rating results for all quality properties goes to the overall result. If all are boundary or higher, it passes (conditionally), and if there is even one Not-acceptable, the whole fails. In case of obtaining a score for quality of the target entity, Figure 4 Typical preset rating levels it is reasonable to assign a value to the quality level. This is the lower limit values of Excessive quality, Acceptable, and because a quality measure basically has a value ranging from Boundary are respectively 0.008, 0.02, and 0.025. 0 to 1 but cannot be used as a universal scale. To interpret the levels into some score value, it is recommended that (3) Quality evaluation Boundary should be 0, Acceptable should be further divided Quality evaluation is an activity to determine pass/fail or into multiple levels and integer values of 1 or more should be score regarding quality (sub)characteristics using the quality assigned to the levels, and Not-acceptable should be assigned rating results for each quality property. The quality evaluation values of -1 or less. is the integration of the quality rating results for the lower From the quality rating results, the quality evaluation quality properties, or the integration of the quality evaluation score can be calculated for each stage by the following results for the quality subcharacteristics immediately below. calculation. Here, wi, wj, and wk are weighting coefficients for In case of determining pass/fail, the following calculation each stage. It is good to normalize each (adding up to 1 for can be used. all). 𝐸(𝑠) = min (𝐿(𝕡)) 𝐸(𝑠) = 6 𝑤' ∗ 𝑆𝑐𝑜𝑟𝑒(𝐿(𝑝' )) 𝐸(𝑐) = min-𝐸(𝕤)/ ' 𝐸!"#$%&& = min (𝐸(𝕔)) 𝑤ℎ𝑒𝑟𝑒 𝐸(𝑐) = 6 𝑤( ∗ 𝐸(𝑠( ) s : a quality subcharacteristic ( c : a quality characteristic 𝐸!"#$%&& = 6 𝑤) ∗ 𝐸(𝑐) ) 𝕡 : a vector of all quality properties under s ) 𝕤: a vector of all the selected subcharacteristics under 𝑤ℎ𝑒𝑟𝑒 𝑆𝑐𝑜𝑟𝑒(𝑙) is a function to map from quality level to c score. 𝕔 : a vector of all the selected characteristics 𝐸(𝑠 𝑜𝑟 𝑐): evaluation result of s or c IV. DIFFRENCES IN FOUR TYPES OF QUALITY 𝐿(𝕡): a vector of rating result for each element of 𝕡 EVALUATION 𝐸!"#$%&& : total quality evaluation result for the target Quality evaluation is classified into four types according to its purpose, which is useful to clarify variation points on how The evaluation value 𝐸(𝑠) for the quality to conduct quality evaluation activities. We identified these subcharacteristic s is the minimum value among the rating points as below: a) Model: Which quality model should be applied, Quality in use, Product quality, Data quality, or IT service quality (TABLE III “Model to use”) b) Charas & measures: Which quality characteristics and sub-characteristics are important, which quality properties and quality measures should be used. c) Criteria: Quality criteria to set quality levels, including weights for scoring for quality properties, quality subcharacteristics and quality characteristics. TABLE II Variation points of quality evaluation and their factors Variation QE type Entity type Industrial point & category domain a) Model V V b) Chars & V V measures c) Criteria V V d) Output V Figure 3 Flow of quality evaluation_ TABLE III Four types of quality evaluation and their differences Model to use Expected results Number of Strength Type Pass/ Candidate Source of quality criteria targets QiU PQ DQ Score and Fail selection weakness Quality requirements or QE for T1 Multiple M M M V V V general guidelines (including comparison industry setting standards) QE for determining to Standards in the industrial T2 Single S M M V V achieve quality domain standard QE for T3 conformity to Single S M M V V Agreed requirements requirements Requirements derived from QE for the assumed stakeholder T4 suitability to Single M S S V V needs the market （established during development） M: major, S: supplementary quality rating, and quality evaluation, using quality models a) Output: What kind of output are needed (TABLE III and quality measures in the SQuaRE series. Furthermore, the “Expected results”) framework will expand its scope of application to include four TABLE II shows variation points of quality evaluation purposes: comparison, qualification to standard, conformity and their factors. The factors to determine these variation checking to requirements, and suitability evaluation in the points includes the type of the quality evaluation, entity type market. This paper organizes and reports on the idea. of the quality evaluation, and industrial domain of the target We hope to get many feedbacks from SQuaRE users on the entity. ideas in this paper to successfully complete the revision of the TABLE III shows each type has its own pattern of quality quality evaluation division. model selection to use and expected results. Only T1 can have multiple targets while the others cannot. Concerning REFERENCES model to use, T1 can use all types of quality models, T2 and [1] Esaki, Kazuhiro, Motoei Azuma, and Toshihiro Komiyama. "Introduction of quality requirement and evaluation based on ISO/IEC T3 mainly use product quality (PQ) and data quality (DQ) square series of standard." International Conference on Trustworthy model, but supplementarily use quality in use (QiU) model. Computing and Services. Springer, Berlin, Heidelberg, 2012. Contrary to that, T4 mainly use QiU model. Expected results [2] Cavano, Joseph P., and James A. McCall. "A framework for the have four possibilities: pass / fail, candidate selection, score, measurement of software quality." Proceedings of the software and strength and weakness. Which results are needed also quality assurance workshop on Functional and performance issues. 1978. depends on the evaluation types, as shown in this table. [3] Nakajima, Tsuyoshi, and Toshihiro Komiyama. "Applying Quality Source of quality criteria describes what kinds of information Requirements Framework to an IoT System and its source should be used to determine the quality criteria. Evaluation." International Journal on Advances in Internet Technology Volume 12, Number 1 & 2, 2019 (2019). V. CONCLUSION [4] Vargas, Iohan Gonçalves, Thiago Gottardi, and Rosana Teresinha Vaccare Braga. "Approaches for integration in system of systems: a This paper proposes a quality evaluation framework, which systematic review." 2016 IEEE/ACM 4th International Workshop on will be hopefully used for the revision of the quality Software Engineering for Systems-of-Systems (SESoS). IEEE, 2016. evaluation division (ISO/IEC 2504n). The framework would [5] Nakajima, Tsuyoshi. "Study group report on square future direction." serve as a basis to support systematic quality evaluation, CEUR Workshop Proceedings. Vol. 2545. CEUR-WS, 2020. which consist of three activities: quality measurement,