Coverage analysis method using quality characteristics Daiju Kato Hiroshi Ishikawa Nihon Knowledge Co., Ltd. Tokyo Metropolitan University Tokyo, Japan Tokyo, Japan d-kato@know-net.co.jp ishikawa-hiroshi@tmu.ac.jp Abstract— Requirement of Must-Be Quality for software has -Quality with the user in mind (Effectiveness - changed over time, and the awareness of software quality has Quality in Use-) become increasingly strong. Therefore, an objective and easy- -Safe and secure to use (Freedom for risk – Quality to-understand method for analyzing software quality is in Use-) necessary. By utilizing SQuaRE (ISO/IEC 25000 series) in software development projects, it is possible to proceed with *Quality characteristics in parentheses software development with comprehensive quality in mind. However, in order to utilize these quality characteristics, the Ten years ago, Must-Be Quality indicated that a function project is needed to have traceability of quality in various worked exactly as required and that there were no phases or sprints. Therefore, analyzing the quality using by performance issues. However, this alone is worrisome in a various evidence under development project provides an world where a variety of devices are connected to the Internet. effective means of explanation of the quality logically and Every week, numerous cyber incidents are distributed by the enables us to judge the quality under standard rules. In this CERT Coordination Center[2] , and Microsoft delivers paper, we propose a quality coverage analysis method using security updates on the second Tuesday of every month[3]. In quality characteristics for software within a general software addition, antivirus patterns are being delivered by anti-virus lifecycle. (Abstract) software vendors quickly. The ability to use software with peace of mind, i.e., to use software without being aware of Keywords- quality analysis, software engineer, quality vulnerabilities and security as well as features and characteristics,SQuaRE (key words) performance, has become a minium requirementment of quality these days. ISO/IEC 25010[4] defines two quality models. A system I. INTRODUCTION must meet the explicit and implicit needs of the various The Kano Model[1] is a type of customer satisfaction people who use the system, called stakeholders, and model and the model uses data obtained from questionnaires satisfying them is considered to be quality. This quality is to rank product quality (function and performance) in terms categorized by characteristics, and the standard defines two of attractiveness (differentiation) and affordability models: product quality and Quality in Use. There are other (indispensability), etc., to visualize the characteristics of data quality models defined in ISO/IEC 25012[5], which products from the customer's point of view and stimulate defines the quality of the data used in the system. discussion in design and development. The quality model consists of characteristics and sub- In the world of software, I feel that this Kano model of characteristics that make up the characteristics. The product quality has been broadly interpreted and used in the same way quality model classifies the quality of a software or system as minimum requirement of quality. However, we believe that product into eight characteristics. This product quality this quality requirement is changing with the times like below provides the 'quality of things'. It means the quality of a thing bullet. The quality characteristics in parentheses at the end because it is a quality that the product itself has. indicates those that match the product quality. On the other hand, quality in use is the quality that people receive or feel when they use the product. It is called as "user • Quality that was commonplace 10 years quality" and it is a quality that is perceived mainly by the user. ago Quality in use is closely related to product quality, and quality -Functions work correctly as required (Functional in use will not improve unless product quality is exhaustive. Suitability) SQuaRE consists of five divisions and defines the means -No response problems (Performance Efficiency) to achieve quality-conscious software development, Figure.1. • Quality that is commonplace today Utilizing SQuaRE in your development projects will enable -Safe to use (Security) you to develop software with quality characteristics in mind. • Quality as a matter of course in the future However, to make full use of quality characteristics, SQuaRE can be used from the quality requirement phase of a development project, or all test work must have test cases or Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). test criteria that are mapped to quality characteristics to is work for all quality characteristics and the second ‘Risk classify the ensured quality into quality characteristics. Since Base Priority’ is work for functional suitability, reliability and quality characteristics provides essentially a classification of performance efficiency. quality, they are effective in measuring the coverage of the Table 1 shows the results of mapping common test types to quality of the final product. Therefore, we developed a quality characteristics. You can refer to this table to map the method to achieve coverage analysis of quality with quality characteristics of the test types written in the test plan. classification of the quality of software during a development project with ISO/IEC/IEEE 12207 [6] and ISO/IEC/IEEE TABLE I. MAPPING OF QUALITY CHARACTERISTICS TO TEST TYPES 15288[7]. Those standards define software life cycle model Test types Quality Characteristics and classification of activity at development process with Accessibility Testing Usability mapping of product quality characteristics. Compatibility Testing Compatibility Conversion Testing Functional Suitability Disaster Recovery Testing Reliability Functional Testing Functional Suitability Instalation Testing Portability Interoperability Testing Compatibility Localization Testing Functional Suitability Usability Portability Maintainability Testing Maintainability Performance-Related Testing Performance efficiency Portability Testing Portability Reliability Testing Reliability Security Testing Security Usability Testing Usability Stress/Load Testing Performance efficiency Reliability Screen Transition Testing Functional Suitability Figure 1. Organization of SQuaRE series Usability II. CLASSIFICATION OF ACTIVITIES BY QUALITY From the above approach, it will be possible to summarize CHARACTERISTICS which quality characteristics the development project is performing for from the activities listed in the project plan or This section proceeds with the quality coverage analysis sprint plan and the test plan. from the following artifacts created in a typical software development project. III. MAPPING QUALITY CHARACTERISTICS TO • Project plan and test plan REQUIREMENTS • Various documents: requirement spec, The next step is to map the quality requirements to quality design spec, test cases and test procedures characteristics to identify the quality that the final product, the • Bug information during the project software, will require. If the quality requirements are not clear, • Review results and Test analysis report it is a good idea to categorize them into functional • Software and test data requirements, performance and load requirements, user The project plan generally describes goal of the quality interface requirements, etc., and map each requirement to a requirements. Also, the document indicates how to fulfill the quality characteristic. If there is a service specification, the quality requirement even though the project use waterfall listed items can also be classified as quality characteristics. development process or agile process. The project plan Maintainability requirements are generally not included in includes test process to list the test types to be conducted and requirement definitions. Of course, You can refer to the will contain the idea of the criteria for each test type. project plan or project rules to determine how software Since the quality requirements written in the project plan maintenance will be performed. The quality characteristic are the quality goals of the final deliverable software, it is probably contains coding style rules for programming or possible to define the required quality by classifying the testing way for using some stab or debugging technique. quality requirements by quality characteristics. It is necessary Ideally, the requirements derived from each requirement to analyze both the process and the deliverables to see if the should also be classified as a quality characteristic. Typically, developed software has the quality that is the goal. requirement items are mapped to a single quality The first step to find out if the process was a quality- characteristic so that the requirements required by the final building process is to find out which quality characteristics product, the software, can be counted for each quality the various activities during the project have an effect on. characteristic. Since the number of man-hours of Although it is very time-consuming to do this task from classification work increases proportionally with the size of scratch, ISO/IEC 30130[6] summarizes which quality the software, requirements can be sampled or, at worst, only attributes are mapped to various activities, TABLE 3. For requirements can be classified into quality characteristics. example, ‘Test Design’ described in the first part of this table Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). This work completes the classification of quality V. CREATING A COVERAGE ANALYSIS OF QUALITY characteristics of the quality required for the software. Development IV. MAPPING VARIOUS EVIDENCE TO QUALITY Quality Requirement Software CHARACTERISTICS Mapping to Activity Checking of PQ Then, classify all the review evidence over the life of the project, e.g., design reviews, code reviews, test design Design/Manufacture/Test reviews, etc., into quality attributes. Rather than simply Activity during project mapping activities to quality attributes, we classify criteria into quality attributes. For example, in the case of API design, Mapping to PQ Mapping to PQ Mapping to PQ the criteria and review reports show whether the reviewers check not only the implemented functionality itself, but also Review Criteria Test Criteria Bug Reports error sequencing and recovery methods, performance and Test Reports Review Results Reference Reference load considerations, authentication methods, and vulnerability responses and so on. Evidence of built in quality The findings that are identified and corrected during the review are also classified by quality characteristics. Corrected Figure 2. Coverage analysis method using quality characteristics bug information is important evidence to ensure the quality of USING QUALITY CHARACTERISTICS the corresponding quality characteristic. Finally, we refer to the criteria for each test type performed Once all the classification work for completeness analysis to see if the quality characteristics mapped to the test type are using quality characteristics is completed, the results of ensured by having met the criteria. For example, suppose the categorizing the quality requirements are compared with the criteria are set for performance testing, Table 2. If the criteria results of categorizing the final deliverable, the software, to are too vague to map the quality characteristics of the criteria, ensure that the quality is on target. The broad analysis is the test cases are checked and classified. In addition, bugs determined by using a radar chart as like Figure 3 to visualize found and fixed in the test as well as in the review report are the differences between requirement and results. classified according to each quality characteristic to check The areas that have a large difference between whether the quality characteristics are maintained or not. requirements and results indicate possibility of the unclear qualities, which can be pursued by checking the specific TABLE II. MAPPING QUALITY CHARACTERISTICS TO TEST CRITERIA classified results and investigating the causes. AT STRESS TESTING In general, functional conformance requirements are often No Test Criteria Quality fulfilled. But reliability is not ensured if the scope of impact Characteristics of a bug fix is not adequately checked or if the functional 1 Single The CPU load should return to Performance conformance test is not re-run even though effect of fixed operation normal after each process. efficiency some bug found in a test related to performance efficiency has 2 Single Memory is released after each Performance an impact on functionality. It is also a good idea to analyze operation process efficiency whether their impact on relevant quality characteristics has 3 Use-case If a new process occurs while Reliability multiple processes are in been verified or not, when the bugs which are classified as progress, it should not result in non-functional conformance have been fixed. an error. If there is a large difference between the two, you can run 4 Use-case The CPU load becomes 100% Reliability some of the tests and check the results of the analysis with and other processes are not your team members to improve the validity of the results. interrupted while multiple processes are running. 5 Peak The process is carried out even if Performance operation the load is twice the expected efficiency workload. Reliability If there is a description of quality in the test report, we consider whether the description or results can be mapped to some quality characteristics as well. Categorizing the evidence of these activities into quality characteristics will clarify the comprehensiveness of the quality of the final product. Figure 2 shows the way of mixed up classify for coverage analysis method using quality characteristics. PQ in the figure stands for Product Quality. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). an exhaustive analysis of whether the quality requirements Coverage Analysis are being met or not. Although it is most desirable to use SQuaRE from the beginning of a development project, quality Functional Suitability analysis using quality characteristics can be performed even 20 after the project is over by using the method described here. Maintenability 15 Performance In addition, it is possible to identify the quality that is Efficiency considered to be unsecured from the analysis results, and 10 therefore, it is also possible to consider policies for 5 strengthening weaknesses. Thus, quality comprehensiveness analysis using quality Portability 0 Reliability characteristics not only visualizes the quality of software, but it is also effective as a means to improve quality. It is also possible to classify test techniques, process transfer criteria and test reports in sprints by quality Security Compatibility characteristics. In the case of derived development, it is possible to judge the appropriateness of quality quantitatively Usability for each quality characteristic by using quality data from past Requirement Results development and quality characteristic quantities defined in ISO/IEC 25022. We intend to further improve the quality Figure 3. Coverage analysis by radar chart comprehensiveness analysis method using these quality characteristics so that it can be used as a quality monitoring VI. CONCLUSION method that enables objective visualization of quality. Quality characteristics enable us to objectively and logically classify the quality of software product, and conduct [6] ISO/IEC/IEEE 12207: Systems and software engineering — Software REFERENCES life cycle processes(2017) [1] Kano,N., Seraku, N., Takahashi, F., Tsuji, S. : Attractive Quality and [7] ISO/IEC/IEEE 15288: Systems and software engineering — System Must-Be Quality; Quality Vol.14(2), pp147-156, JSQC, 1984, life cycle processes(2015) (Japanese) [8] Kato, D., Okuyama, A., Ishikawa, H. : Introduction of test [2] CERT Coordination Center, management based on quality characteristics, IWESQ 2019, 2019. https://www.sei.cmu.edu/about/divisions/cert/index.cfm [9] Kato,D., Ishikawa, H. : Develop Quality Characteristics based quaity evaluation process for ready to use software products, JSE-2016, [3] Microsoft Security Update Guide, February, 2016. https://portal.msrc.microsoft.com/en-us/security-guidance [10] Kato,D., Okuyama, A., Ishikawa, H.: Use proactive evaluation [4] ISO/IEC 25010: Systems and software engineering — Systems and process for ‘Quality in Use’, Seventh World Congress for Software software Quality Requirements and Evaluation (SQuaRE) — System Quality, 2017. and software quality models (2011) [5] ISO/IEC 25012: Software engineering — Software product Quality Requirements and Evaluation (SQuaRE) — Data quality model(2008) . Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). TABLE III. SUMMARY OF CAPABILITIES WITH CHARACTERISTICS IN ISO/IEC 30130 Categories Characteristics Test Dynamic test execution environm Code analysis Test management software quality characteristics Granularity ent Capabilities Input for Dynamic Test data Test Input for Code Test Test Quality Test Ver if Test Functional Reliability Usability Performance Maintain Portability Compatib Security Smallest Intermedia Largest dynamic test repository environm code analysis plan asset record completio ication status suitability efficiency ability ility unit te units unit test execution ent analysis report n report and report execution validation Test design 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 Risk Based Priority 〇 〇 〇 〇 〇 〇 〇 〇 Test execution controll, 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 Automated test execution Caputure,Playback 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 Keyword driven test 〇 〇 〇 〇 case Test comparator 〇 〇 〇 Debugging 〇 〇 〇 〇 〇 Dynamic analysis 〇 〇 〇 〇 〇 〇 Monitoring 〇 〇 〇 〇 〇 〇 〇 Coverage 〇 〇 〇 measurement Security testing 〇 〇 〇 Test data preparation, 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 Test data generation Stress testing,Load 〇 〇 〇 〇 testing Performance testing 〇 〇 〇 〇 〇 Data validation and 〇 〇 〇 〇 〇 〇 verification Database validation 〇 〇 〇 〇 〇 〇 and Verification Emulators,Simulator 〇 〇 〇 〇 〇 〇 s Unit test framework 〇 〇 〇 Automated 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 environment set up Runtime envrionment 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 management Review 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 Code analyzer 〇 〇 〇 〇 〇 〇 Codebased security 〇 testing Test management 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 Test Asset configuration 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 management Incident 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 management Defect management, 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 Defect tracking,Bug tracking Test monitoring 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 Relating of data that serves as the basis 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 for Verification and Validation Reports Verification and 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 〇 Validation Report Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).