=Paper=
{{Paper
|id=Vol-2800/paper-04
|storemode=property
|title=Quality Evaluation and Testing Method of Industrial Application Based on ISO/IEC 25000 SQuaRE Standard
|pdfUrl=https://ceur-ws.org/Vol-2800/paper-04.pdf
|volume=Vol-2800
|authors=Xiuming Yu,Wenpeng Li,Yangyang Zhang,Zengzhi Liu
|dblpUrl=https://dblp.org/rec/conf/apsec/YuLZL20
}}
==Quality Evaluation and Testing Method of Industrial Application Based on ISO/IEC 25000 SQuaRE Standard==
https://ceur-ws.org/Vol-2800/paper-04.pdf
Coverage analysis method using quality
characteristics
Daiju Kato Hiroshi Ishikawa
Nihon Knowledge Co., Ltd. Tokyo Metropolitan University
Tokyo, Japan Tokyo, Japan
d-kato@know-net.co.jp ishikawa-hiroshi@tmu.ac.jp
Abstract— Requirement of Must-Be Quality for software has -Quality with the user in mind (Effectiveness -
changed over time, and the awareness of software quality has Quality in Use-)
become increasingly strong. Therefore, an objective and easy- -Safe and secure to use (Freedom for risk – Quality
to-understand method for analyzing software quality is in Use-)
necessary. By utilizing SQuaRE (ISO/IEC 25000 series) in
software development projects, it is possible to proceed with
*Quality characteristics in parentheses
software development with comprehensive quality in mind.
However, in order to utilize these quality characteristics, the Ten years ago, Must-Be Quality indicated that a function
project is needed to have traceability of quality in various worked exactly as required and that there were no
phases or sprints. Therefore, analyzing the quality using by performance issues. However, this alone is worrisome in a
various evidence under development project provides an world where a variety of devices are connected to the Internet.
effective means of explanation of the quality logically and Every week, numerous cyber incidents are distributed by the
enables us to judge the quality under standard rules. In this CERT Coordination Center[2] , and Microsoft delivers
paper, we propose a quality coverage analysis method using security updates on the second Tuesday of every month[3]. In
quality characteristics for software within a general software addition, antivirus patterns are being delivered by anti-virus
lifecycle. (Abstract) software vendors quickly. The ability to use software with
peace of mind, i.e., to use software without being aware of
Keywords- quality analysis, software engineer, quality vulnerabilities and security as well as features and
characteristics,SQuaRE (key words) performance, has become a minium requirementment of
quality these days.
ISO/IEC 25010[4] defines two quality models. A system
I. INTRODUCTION
must meet the explicit and implicit needs of the various
The Kano Model[1] is a type of customer satisfaction people who use the system, called stakeholders, and
model and the model uses data obtained from questionnaires satisfying them is considered to be quality. This quality is
to rank product quality (function and performance) in terms categorized by characteristics, and the standard defines two
of attractiveness (differentiation) and affordability models: product quality and Quality in Use. There are other
(indispensability), etc., to visualize the characteristics of data quality models defined in ISO/IEC 25012[5], which
products from the customer's point of view and stimulate defines the quality of the data used in the system.
discussion in design and development. The quality model consists of characteristics and sub-
In the world of software, I feel that this Kano model of characteristics that make up the characteristics. The product
quality has been broadly interpreted and used in the same way quality model classifies the quality of a software or system
as minimum requirement of quality. However, we believe that product into eight characteristics. This product quality
this quality requirement is changing with the times like below provides the 'quality of things'. It means the quality of a thing
bullet. The quality characteristics in parentheses at the end because it is a quality that the product itself has.
indicates those that match the product quality. On the other hand, quality in use is the quality that people
receive or feel when they use the product. It is called as "user
• Quality that was commonplace 10 years quality" and it is a quality that is perceived mainly by the user.
ago Quality in use is closely related to product quality, and quality
-Functions work correctly as required (Functional in use will not improve unless product quality is exhaustive.
Suitability) SQuaRE consists of five divisions and defines the means
-No response problems (Performance Efficiency) to achieve quality-conscious software development, Figure.1.
• Quality that is commonplace today Utilizing SQuaRE in your development projects will enable
-Safe to use (Security) you to develop software with quality characteristics in mind.
• Quality as a matter of course in the future However, to make full use of quality characteristics,
SQuaRE can be used from the quality requirement phase of a
development project, or all test work must have test cases or
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
test criteria that are mapped to quality characteristics to is work for all quality characteristics and the second ‘Risk
classify the ensured quality into quality characteristics. Since Base Priority’ is work for functional suitability, reliability and
quality characteristics provides essentially a classification of performance efficiency.
quality, they are effective in measuring the coverage of the Table 1 shows the results of mapping common test types to
quality of the final product. Therefore, we developed a quality characteristics. You can refer to this table to map the
method to achieve coverage analysis of quality with quality characteristics of the test types written in the test plan.
classification of the quality of software during a development
project with ISO/IEC/IEEE 12207 [6] and ISO/IEC/IEEE TABLE I. MAPPING OF QUALITY CHARACTERISTICS TO TEST TYPES
15288[7]. Those standards define software life cycle model Test types Quality Characteristics
and classification of activity at development process with Accessibility Testing Usability
mapping of product quality characteristics. Compatibility Testing Compatibility
Conversion Testing Functional Suitability
Disaster Recovery Testing Reliability
Functional Testing Functional Suitability
Instalation Testing Portability
Interoperability Testing Compatibility
Localization Testing Functional Suitability
Usability
Portability
Maintainability Testing Maintainability
Performance-Related Testing Performance efficiency
Portability Testing Portability
Reliability Testing Reliability
Security Testing Security
Usability Testing Usability
Stress/Load Testing Performance efficiency
Reliability
Screen Transition Testing Functional Suitability
Figure 1. Organization of SQuaRE series Usability
II. CLASSIFICATION OF ACTIVITIES BY QUALITY From the above approach, it will be possible to summarize
CHARACTERISTICS which quality characteristics the development project is
performing for from the activities listed in the project plan or
This section proceeds with the quality coverage analysis sprint plan and the test plan.
from the following artifacts created in a typical software
development project. III. MAPPING QUALITY CHARACTERISTICS TO
• Project plan and test plan REQUIREMENTS
• Various documents: requirement spec, The next step is to map the quality requirements to quality
design spec, test cases and test procedures characteristics to identify the quality that the final product, the
• Bug information during the project software, will require. If the quality requirements are not clear,
• Review results and Test analysis report it is a good idea to categorize them into functional
• Software and test data requirements, performance and load requirements, user
The project plan generally describes goal of the quality interface requirements, etc., and map each requirement to a
requirements. Also, the document indicates how to fulfill the quality characteristic. If there is a service specification, the
quality requirement even though the project use waterfall listed items can also be classified as quality characteristics.
development process or agile process. The project plan Maintainability requirements are generally not included in
includes test process to list the test types to be conducted and requirement definitions. Of course, You can refer to the
will contain the idea of the criteria for each test type. project plan or project rules to determine how software
Since the quality requirements written in the project plan maintenance will be performed. The quality characteristic
are the quality goals of the final deliverable software, it is probably contains coding style rules for programming or
possible to define the required quality by classifying the testing way for using some stab or debugging technique.
quality requirements by quality characteristics. It is necessary Ideally, the requirements derived from each requirement
to analyze both the process and the deliverables to see if the should also be classified as a quality characteristic. Typically,
developed software has the quality that is the goal. requirement items are mapped to a single quality
The first step to find out if the process was a quality- characteristic so that the requirements required by the final
building process is to find out which quality characteristics product, the software, can be counted for each quality
the various activities during the project have an effect on. characteristic. Since the number of man-hours of
Although it is very time-consuming to do this task from classification work increases proportionally with the size of
scratch, ISO/IEC 30130[6] summarizes which quality the software, requirements can be sampled or, at worst, only
attributes are mapped to various activities, TABLE 3. For requirements can be classified into quality characteristics.
example, ‘Test Design’ described in the first part of this table
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
This work completes the classification of quality V. CREATING A COVERAGE ANALYSIS OF QUALITY
characteristics of the quality required for the software.
Development
IV. MAPPING VARIOUS EVIDENCE TO QUALITY Quality Requirement Software
CHARACTERISTICS
Mapping to Activity Checking of PQ
Then, classify all the review evidence over the life of the
project, e.g., design reviews, code reviews, test design Design/Manufacture/Test
reviews, etc., into quality attributes. Rather than simply Activity during project
mapping activities to quality attributes, we classify criteria
into quality attributes. For example, in the case of API design, Mapping to PQ Mapping to PQ Mapping to PQ
the criteria and review reports show whether the reviewers
check not only the implemented functionality itself, but also Review Criteria Test Criteria
Bug Reports
error sequencing and recovery methods, performance and Test Reports
Review Results Reference Reference
load considerations, authentication methods, and
vulnerability responses and so on. Evidence of built in quality
The findings that are identified and corrected during the
review are also classified by quality characteristics. Corrected Figure 2. Coverage analysis method using quality characteristics
bug information is important evidence to ensure the quality of USING QUALITY CHARACTERISTICS
the corresponding quality characteristic.
Finally, we refer to the criteria for each test type performed Once all the classification work for completeness analysis
to see if the quality characteristics mapped to the test type are using quality characteristics is completed, the results of
ensured by having met the criteria. For example, suppose the categorizing the quality requirements are compared with the
criteria are set for performance testing, Table 2. If the criteria results of categorizing the final deliverable, the software, to
are too vague to map the quality characteristics of the criteria, ensure that the quality is on target. The broad analysis is
the test cases are checked and classified. In addition, bugs determined by using a radar chart as like Figure 3 to visualize
found and fixed in the test as well as in the review report are the differences between requirement and results.
classified according to each quality characteristic to check The areas that have a large difference between
whether the quality characteristics are maintained or not. requirements and results indicate possibility of the unclear
qualities, which can be pursued by checking the specific
TABLE II. MAPPING QUALITY CHARACTERISTICS TO TEST CRITERIA classified results and investigating the causes.
AT STRESS TESTING In general, functional conformance requirements are often
No Test Criteria Quality fulfilled. But reliability is not ensured if the scope of impact
Characteristics of a bug fix is not adequately checked or if the functional
1 Single The CPU load should return to Performance conformance test is not re-run even though effect of fixed
operation normal after each process. efficiency some bug found in a test related to performance efficiency has
2 Single Memory is released after each Performance an impact on functionality. It is also a good idea to analyze
operation process efficiency
whether their impact on relevant quality characteristics has
3 Use-case If a new process occurs while Reliability
multiple processes are in been verified or not, when the bugs which are classified as
progress, it should not result in non-functional conformance have been fixed.
an error. If there is a large difference between the two, you can run
4 Use-case The CPU load becomes 100% Reliability some of the tests and check the results of the analysis with
and other processes are not your team members to improve the validity of the results.
interrupted while multiple
processes are running.
5 Peak The process is carried out even if Performance
operation the load is twice the expected efficiency
workload. Reliability
If there is a description of quality in the test report, we
consider whether the description or results can be mapped to
some quality characteristics as well.
Categorizing the evidence of these activities into quality
characteristics will clarify the comprehensiveness of the
quality of the final product.
Figure 2 shows the way of mixed up classify for coverage
analysis method using quality characteristics. PQ in the figure
stands for Product Quality.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
an exhaustive analysis of whether the quality requirements
Coverage Analysis are being met or not. Although it is most desirable to use
SQuaRE from the beginning of a development project, quality
Functional
Suitability analysis using quality characteristics can be performed even
20 after the project is over by using the method described here.
Maintenability 15 Performance In addition, it is possible to identify the quality that is
Efficiency considered to be unsecured from the analysis results, and
10
therefore, it is also possible to consider policies for
5 strengthening weaknesses.
Thus, quality comprehensiveness analysis using quality
Portability 0 Reliability
characteristics not only visualizes the quality of software, but
it is also effective as a means to improve quality.
It is also possible to classify test techniques, process
transfer criteria and test reports in sprints by quality
Security Compatibility
characteristics. In the case of derived development, it is
possible to judge the appropriateness of quality quantitatively
Usability for each quality characteristic by using quality data from past
Requirement Results development and quality characteristic quantities defined in
ISO/IEC 25022. We intend to further improve the quality
Figure 3. Coverage analysis by radar chart comprehensiveness analysis method using these quality
characteristics so that it can be used as a quality monitoring
VI. CONCLUSION method that enables objective visualization of quality.
Quality characteristics enable us to objectively and
logically classify the quality of software product, and conduct
[6] ISO/IEC/IEEE 12207: Systems and software engineering — Software
REFERENCES life cycle processes(2017)
[1] Kano,N., Seraku, N., Takahashi, F., Tsuji, S. : Attractive Quality and [7] ISO/IEC/IEEE 15288: Systems and software engineering — System
Must-Be Quality; Quality Vol.14(2), pp147-156, JSQC, 1984, life cycle processes(2015)
(Japanese) [8] Kato, D., Okuyama, A., Ishikawa, H. : Introduction of test
[2] CERT Coordination Center, management based on quality characteristics, IWESQ 2019, 2019.
https://www.sei.cmu.edu/about/divisions/cert/index.cfm [9] Kato,D., Ishikawa, H. : Develop Quality Characteristics based quaity
evaluation process for ready to use software products, JSE-2016,
[3] Microsoft Security Update Guide, February, 2016.
https://portal.msrc.microsoft.com/en-us/security-guidance [10] Kato,D., Okuyama, A., Ishikawa, H.: Use proactive evaluation
[4] ISO/IEC 25010: Systems and software engineering — Systems and process for ‘Quality in Use’, Seventh World Congress for Software
software Quality Requirements and Evaluation (SQuaRE) — System Quality, 2017.
and software quality models (2011)
[5] ISO/IEC 25012: Software engineering — Software product Quality
Requirements and Evaluation (SQuaRE) — Data quality model(2008)
.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
TABLE III. SUMMARY OF CAPABILITIES WITH CHARACTERISTICS IN ISO/IEC 30130
Categories Characteristics
Test
Dynamic test execution environm Code analysis Test management software quality characteristics Granularity
ent
Capabilities
Input for Dynamic Test data Test Input for Code Test Test Quality Test Ver if Test Functional Reliability Usability Performance Maintain Portability Compatib Security Smallest Intermedia Largest
dynamic test repository environm code analysis plan asset record completio ication status suitability efficiency ability ility unit te units unit
test execution ent analysis report n report and report
execution validation
Test design 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
Risk Based Priority 〇 〇 〇 〇 〇 〇 〇 〇
Test execution
controll,
〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
Automated test
execution
Caputure,Playback 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
Keyword driven test
〇 〇 〇 〇
case
Test comparator 〇 〇 〇
Debugging 〇 〇 〇 〇 〇
Dynamic analysis 〇 〇 〇 〇 〇 〇
Monitoring 〇 〇 〇 〇 〇 〇 〇
Coverage
〇 〇 〇
measurement
Security testing 〇 〇 〇
Test data
preparation, 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
Test data generation
Stress testing,Load
〇 〇 〇 〇
testing
Performance testing 〇 〇 〇 〇 〇
Data validation and
〇 〇 〇 〇 〇 〇
verification
Database validation
〇 〇 〇 〇 〇 〇
and Verification
Emulators,Simulator
〇 〇 〇 〇 〇 〇
s
Unit test framework 〇 〇 〇
Automated
〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
environment set up
Runtime
envrionment 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
management
Review 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
Code analyzer 〇 〇 〇 〇 〇 〇
Codebased security
〇
testing
Test management 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
Test Asset
configuration 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
management
Incident
〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
management
Defect
management,
〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
Defect tracking,Bug
tracking
Test monitoring 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
Relating of data that
serves as the basis
〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
for Verification and
Validation Reports
Verification and
〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇
Validation Report
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).