=Paper=
{{Paper
|id=Vol-2545/paper5
|storemode=property
|title=Test Management Based on Quality Characteristics (short paper)
|pdfUrl=https://ceur-ws.org/Vol-2545/paper-05.pdf
|volume=Vol-2545
|authors=Daiju Kato,Ayako Okuyama,Hiroshi Ishikawa
|dblpUrl=https://dblp.org/rec/conf/apsec/KatoO019
}}
==Test Management Based on Quality Characteristics (short paper)==
https://ceur-ws.org/Vol-2545/paper-05.pdf
Test management based on quality characteristics
Daiju Kato Ayako Okuyama Hiroshi Ishikawa
Nihon Knowledge Co., Ltd. Nihon Knowledge Co., Ltd. Tokyo Metropolitan University
Tokyo, Japan Tokyo, Japan Tokyo, Japan
d-kato@know-net.co.jp a-okuyama@know-net.co.jp ishikawa-hiroshi@tmu.ac.jp
Abstract—By using SQuaRE in product development
projects, one can achieve quality traceability based on quality
characteristics. Conveying the quality of the software product to
stakeholders can be done objectively. This paper introduces
quality characteristics and to the use of SQuaRE, with
explanation of quality requirement definitions and evaluation
methods using quality characteristics.
Keywords—Evaluation process, Quality characteristics,
SQuaRE, Test management, Traceability Fig. 1. International standards used in the development process
I. REASON FOR USING QUALITY CHARACTERISTICS used in this way as traceability of test cases and incidents from
quality requirements, the number of test viewpoints, the
We have investigated ways for stakeholders or our number of test cases, and the number of bugs can be
products’ customers to understand software quality aggregated for each quality characteristic to ascertain which
objectively. To date, many companies have used their own quality characteristic is problematic. In other words, quality
quality data to explain the quality of deliverables. For example, characteristics are useful as an indicator of traceability, Fig. 2.
the IT Development White Paper published by the
Information Technology Promotion Agency (IPA) of Japan Quality
Test Viewpoint Test Case Incident Report
provides survey results elucidating the bug density of projects Requirement
Test Type
Test Viewpoint Test Case Incident Report
in various industries [1]. However, bug density metrics often evaluation
Tracebility using by Quality Characteristics
vary greatly depending on development languages, coding
rules, and development processes. It is difficult to ascertain Fig. 2. Quality characteristics as an indicator of traceability.
whether the quality of products is good or bad simply using
numerical values alone.
III. QUALITY REQUIREMENTS CATEGORIZED BY QUALITY
Therefore, as a method to conveying the software product CHARACTERISTICS
quality to anyone using an easy-to-understand methodology, To use quality characteristics as an indicator for the quality
we have examined the use of Systems and Software Quality of software under development project, one must define
Requirements and Evaluation (SQuaRE) [2] to represent quality requirements for each quality characteristic. Instead of
understandable quality objectively. simply writing quality requirement definitions for each quality
II. COMBINATION OF INTERNATIONAL STANDARDS IN THE sub-characteristic, it is better to refer to quality requirements
included in ISO/IEC 25051 [4].
DEVELOPMENT PROCESS
Using SQuaRE in a software development project is According to our developed rules, when writing quality
insufficient for simple description of the quality requirements requirements for product quality, the function or operation of
as quality characteristics. An evaluation result must indicate the software is written as the subject, whereas, when writing
that the quality requirements classified by the quality quality requirements for quality in use, the definition is written
characteristics are satisfied. To take advantage of the quality with the users as the subject. Product quality is the quality
characteristics defined in the ISO/IEC 25010 quality model possessed by the product. Quality in use is the quality that
[3], relevant international standards were identified. The people feel. By dividing the subject clearly, one can be aware
related standards were identified during the development of differences in quality. Only the quality required for the
project, as shown in Fig. 1. target software needs to be defined for both product quality
and quality in use, and quality requirements are not defined
By assigning a quality characteristic that can be for all quality sub-characteristics.
guaranteed by a corresponding test types to a test type that
satisfies each quality requirement, the same quality Once the definition of the quality requirement is presented,
characteristic can be assigned to test viewpoints constituting the test type which can evaluate the quality requirement is
the test types. At this time, in the case of a functional test, mapped. If possible, we recommend identification of the
several quality characteristics such as functional suitability required quality characteristics at each test level or sprint and
and functional accuracy are assigned to the test type, but a recommend mapping of test types that meet the test level or
single quality sub-characteristic is assigned to the test the sprint quality.
viewpoint. Test cases derived from the test perspective will
also be mapped to a single-quality sub-characteristic. When
the created test case is executed and an incident is found, the
incident has the quality sub-characteristic assigned to the
executed test case as an attribute. If quality characteristics are
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
TABLE I. QUALITY REQUIREMENTS CLASSIFIED BY QUALITY Because the created test case also has an inherent quality
CHARACTERISTIC
sub-characteristic as an attribute, the bug found in the
Characteristic
Sub-
Target Quality
execution of the test case is mapped to the same quality sub-
Characteristics characteristic as the test case. Therefore, our bug management
• Development requirements of system prepares quality sub-characteristic items for each
new features have been verified.
Functional
• Migration from past versions incident, and uses the quality characteristics in bug analysis.
completeness If many bugs exist for a particular quality trait, increased
is possible; compatibility is
collateral. testing related to that quality trait can be done to enhance the
• Results of new functional review issued by the development team.
requirements are correct. They
have been verified. V. CONCLUSION
Functional • Each product definition file
suitability developed by the past version Using quality characteristics as an indicator of software
Functional
functioned properly. quality of the final product of a project not only enables
• All functions under English traceability by quality characteristics. It also makes it easier to
correctness
and Chinese environments have analyze quality objectively and to balance the software
the same operation quality as in
the Japanese environment. quality with other quality characteristics.
• Scenario test results present
no difficulty under the estimated
By classifying bugs found after a product release and
users’ operations. inquiries received from the support center based on quality
• Each test level has an analysis characteristics, one can analyze the evaluation work validity
action for test coverage and during development and can build a feedback process.
turn-around time to fix bugs.
• Each test level has a quality To realize software development that uses quality
Maturity improvement action derived characteristics, all project members must have knowledge of
from quality analysis of the quality characteristics. However, considering that
previous test level. international standards are useful and that objective quality
• All bugs verified at the RC
stage. information can be provided to stakeholders and customers,
Reliability • Operations have perfect quality characteristics are used. We believe that using quality
continuity even though one characteristics can provide excellent results. We are
node is stopped under a investigating more effective use of SQuaRE.
Fault tolerance clustering environment.
• Migration programs continue
running when some definition
files have some errors. REFERENCES
• Prior versions of definition
[1] “Software Development Data White Paper 2018-2019”, Information-
Recoverability files can be saved even though
Technology Promotion Agency (IPA), Japan, ISBN 978-4-905318-64-
the files include some errors.
4, 2018, pp. 283-285.
• Performance is less than or
equal to a maximum 3% of the [2] ISO/IEC 25000:2014, “Systems and software engineering – Systems
prior version. and software Quality Requirements and Evaluation (SQuaRE) – Guide
• Concurrent multi-access to SQuaRE”, March, 2014.
Time behavior [3] ISO/IEC 25010, Systems and software engineering -- Systems and
testing is done with no error.
Performance • There is no high CPU load or software Quality Requirements and Evaluation (SQuaRE) -- System
efficiency I/O load condition under some and software quality models, March 2011.
functional operations. [4] ISO/IEC 25051, Quality Requirements and Evaluation (SQuaRE) --
• No memory leak occurs under Requirements for quality of Ready to Use Software Product (RUSP)
Resource usual operations. and instructions for testing, February 2014.
utilization • Memory and I/O resources are [5] D. Kato, H. Ishikawa, “Develop Quality Characteristics based quaity
used effectively. evaluation process for ready to use software products”, JSE-2016,
February, 2016.
[6] D. Kato, A. Okuyama, H. Ishikawa, “Use proactive evaluation process
IV. EVALUATION PROCESS for ‘Quality in Use’”, Seventh World Congress for Software Quality”,
In the test design of each test type, the test design is done Seventh World Congress for Software Quality, March, 2017.
considering the assigned quality characteristics. As described [7] D. Kato, H. Ishikawa, “Development of quality assurance process for
Ready to Use Software Product with using JIS X 25051:2016 and
above, the test viewpoint allows a single quality sub- benefit of the process”, Software Quality Symposium, September 2016,
characteristic to be assigned. We aggregate all test viewpoints in Japanese.
for each quality characteristic so that the test viewpoint [8] D. Kato, H. Ishikawa, “Feedback process of inquiry analysis conscious
regarding functional compatibility is less than 70%. This is of recurring business”, Software Quality Symposium, September 2016,
true because the test is not only a functional test; it also in Japanese.
assesses the completeness of quality.