A Method of Software Quality Comparison based on PCA Wenpeng Li , Wei Su, Jianxun Guo, Kuan Feng, Xiuming Yu* China Electronics Standardization Institute, Beijing, China Abstract The platform software has a large number of functional and performance efficiency quality indicators, as well as differences in fixed basic hardware and software environments, making it impossible to effectively compare the quality. Considering that different users have different concerns about the product indicators of platform software, it brings certain difficulties to the selection of users. This paper proposes a software quality comparison method based on PCA, which extracts principal components by analyzing the correlation between data, reasonably allocates and evaluates software quality through dimensionality reduction and weighting, avoids errors caused by subjective experience, and can effectively adapt to changes in evaluation dimensions and the number of software products. Achieve the goal of horizontal comparison between products through a score. Keywords quality comparison, quality models, PCA 1 functional characteristics: Functional Suitability, 1. Introduction Performance efficiency, Compatibility, Usability, Reliability, Security, Maintainability, and Portability, With the rapid advancement of information along with their respective sub-characteristics and technology, the level of information digitization in all properties [4]. aspects of social life is continuously improving, and software has emerged as a modern infrastructure. However, software is a product of human intellectual labor with poor visibility in terms of quality. The complexity and fuzziness of software make it challenging to quantify, thereby complicating users' ability to objectively comprehend its quality. The construction of a software quality model establishes a framework for measuring software quality attributes, establishing the relationship between measurable attributes and software quality, thereby providing a basis for evaluating and comparing the quality of software products. Figure 1: Product quality model In 1991, ISO/IEC JTC1/SC7 issued the ISO/IEC 9126 standard, which is based on McCall and Boehm's ISO/IEC 25023 provides quality measures for sub- quality model. It reformulated the quality of software characteristics of the software product quality model, into 6 main attributes and 21 sub-attributes, marking which are widely used for measurement functions. a significant milestone in the standardization of These include mean response time and mean software quality. turnaround time for time behavior measurement; In the research of software quality evaluation, mean processor utilization, mean memory utilization, methods such as Delphi method, fuzzy fuzzy mean I/O utilization, and bandwidth utilization for comprehensive evaluation, topsis, evidential theory, resource utilization measurement; transaction and so on are often used. Most of these methods rely processing capacity, user access capacity, and on subjective experience or fuzzy theory to construct adequacy of user access increase in capacity measures quality measurement models, making them [5]. susceptible to human subjective influence in The platform software has a large number of determining weights, introducing a level of functional and performance efficiency quality uncertainty [1, 2, 3]. indicators, as well as differences in fixed basic ISO/IEC 25010 provides the software product hardware and software environments, making it quality model (as shown in Fig. 1), offering eight impossible to effectively compare the quality. 5th International Workshop on Experience with SQuaRE Series andits Future Direction, December 04, 2023, Seoul, Korea liwp@cesi.cn (W. Li); yuxiuming@cesi.cn (X. Yu) 0009-0001-4065-2527 (W. Li) © 2023 Copyright for this paper by its authors. The use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 13 Considering that different users have different coefficients of indicators in the linear combinations of concerns about the product indicators of platform principal components, with the weights being the software, it brings certain difficulties to the selection variance contribution rates of the principal of users. In the quality testing of large-scale platform components. Finally, the indicator weights are product, function indicators of platform product normalized. capability and performance indicators are widely In software quality comparison, the matrix used concerned. The results of test records are usually a for PCA can be represented as: numerical value (e.g. how many algorithms the platform supports, and the maximum concurrency  x11 x12 x1m  supported by the performance result is 10,000). x x22 x2 m  X = T Inconsistent data dimensions make it difficult to = x x p  21 x2 compare and analyze products, and inconvenience    1 users in comparing products.   This paper introduces a method for analyzing data  x p1 xp2 x pm  correlation. This approach eliminates the need for Where, p represents the number of software subjective experience and is suitable for conducting products, and m represents the number of quality large-scale comparisons of product quality. characteristics. 3. Software Quality Comparison 2. Principal Component based on PCA Analysis Analyzing data from some tested blockchain Principal Component Analysis (PCA) accomplishes platforms, performance indicators of blockchain the objective of eliminating correlations between platform products, specifically as follows: features by transforming a set of potentially correlated variables into a set of linearly independent variables ⚫ Average response time: the average time it through orthogonal transformation. This process takes for a transaction to be processed and retains crucial features while minimizing information confirmed. This metric is measured by loss. PCA generates two types of coefficients to achieve iterating multiple times (with a 1-second these goals: 'weights' that define the transformation interval) and obtaining the average response from raw data to summary scores, and 'loadings' that time for each iteration. indicate the strength of association between the raw ⚫ Transaction processing rate: the number of variables and the low-dimensional representations [6]. transactions that can be processed per PCA can be represented by the following second. This metric measures the overall mathematical model： performance of the blockchain product in terms of transaction processing speed.  x1 = a11F1 + a12 F2 + + a1m Fm + a11 ⚫ Concurrent user/request count: the  x =a F +a F + +a F +a   2 21 1 22 2 2m m 2 2 maximum number of users or requests that  (1) can be processed simultaneously. This  metric measures the scalability of the  x p = a p1F1 + a p 2 F2 + + a pm Fm + a p p  blockchain product and its ability to handle x ， x ， x ， ，x p multiple concurrent requests. where, 1 2 3 represent p primitive F1，F2，F3， ，Fm ⚫ Data processing volume: the amount of data variables, represent m factor that needs to be processed for each variables, matrix form can be expressed as： transaction. This metric measures the size of the transactions being processed and the X = AF + a (2) overall data processing capacity of the where, F represent common factors, A represent blockchain product. a factor loading matrix, ij represent factor loading. ⚫ CPU utilization: the percentage of CPU For determining the weights of indicators in resources being used by the blockchain principal component analysis, the first step is to product. This metric measures the efficiency b of the blockchain product in utilizing the calculate the coefficients ( ij ) of indicators in the available CPU resources. linear combinations of each principal component and c ⚫ Memory utilization: the percentage of the variance contribution rate ( ij ) of each principal memory resources being used by the component. The coefficient of each indicator in blockchain product. This metric measures different linear combinations of principal components the efficiency of the blockchain product in b a utilizing the available memory resources. ( ij ) equals the ratio of the loadings ( ij ) of each  And, core functional indicators as following: indicator to the square root of the eigenvalues ( i ) of a ⚫ Supported consensus mechanisms: the bij = ij number of different consensus mechanisms i each component, which is . Secondly, the that the blockchain product supports. indicator weight is the weighted average of the 14 ⚫ Supported smart contract development ⚫ Supported key algorithms: the number of languages: the number of different smart different key algorithms that the blockchain contract development languages that the product supports. blockchain product supports. Selecting data from six blockchain platform products, see Table 1 for details. Table 1 Blockchain platform products test data Average Data smart blockchain Transaction Concurrent CPU Memory response processing consensus contract key platform processing user/request utilization utilization time volume mechanisms development algorithms products rate count (%) (%) (ms) (MB) languages 1 7 12882 167272 490.05 74.7 8.3 4 3 6 2 9 44200 45000 1560.99 18 15 1 4 5 3 2 150918 111040 73.35 23 16 3 5 7 4 59 693 17845 16 43 20 5 4 5 5 29 799 27424 9 50 61 3 5 6 6 66 59241 8514 367 10 15 4 4 7 Constructing a 9  6 matrix as following: Transaction processing -0.41 0.762 0.429  product1  rate  product 2  Concurrent   user/request -0.678 -0.335 0.623  product 3  count X =  Data  product 4  processing -0.699 -0.036 -0.674  product 5  volume   CPU  product 6  utilization 0.008 -0.805 0.337 Performing PCA analysis in the above matrix. The Memory 0.57 0.182 -0.191 cumulative variance explanation rate of the first four utilization eigenvalues in table 2 exceeds 95%. Ingredients, also consensus 0.631 -0.43 0.543 known as predictors or independent variables refer to mechanisms the original variables or features in the dataset that smart you want to reduce the dimensionality. Components contract 0.318 0.803 0.009 are the new variables that are created by PCA to developmen represent the original data in a lower-dimensional t languages space. These components are linear combinations of key 0.041 0.54 0.705 the original ingredients and are ordered so that they algorithms capture the most variance in the data. Table 2 Obtain the value of a in formula (1), which shown Variance explained table in Table 4. Cumulative Table 4 Explained explained Ingredient matrix table Ingredients Eigenvalue Variance variance Ingredient Ingredient Ingredient (%) indicators (%) 1 2 3 1 2.621 29.122 29.122 Average 2 2.499 27.762 56.884 response 0.314 -0.021 -0.086 3 1.998 22.204 79.088 time 4 1.535 17.055 96.143 Transaction 5 0.347 3.857 100 processing -0.156 0.305 0.215 6 100 rate Concurrent The factor loading coefficients were calculated user/request -0.259 -0.134 0.312 using the first four eigenvalues, and the results are count shown in Table 3. Data processing -0.267 -0.014 -0.337 Table 3 volume Factor load factor CPU 0.003 -0.322 0.169 Principal Principal Principal utilization indicators componen componen componen Memory 0.218 0.073 -0.095 t1 t2 t3 utilization Average consensus 0.241 -0.172 0.272 response 0.824 -0.053 -0.171 mechanisms time 15 smart The principal component scores and contract comprehensive scores of the six products, which 0.121 0.321 0.004 listing in TABLE 5. development languages Firstly, calculate the product of each principal key component's linear combination coefficient and its 0.016 0.216 0.353 algorithms corresponding variance explained rate, then accumulate these products, and finally divide the total Obtain the formula (2) as follows: variance explained rate by this sum to get the comprehensive score. F=(0.291/0.961)×F1+(0.278/0.961)×F2+(0.222/ 0.961)×F3+(0.171/0.961)×F4 Table 5 Comprehensive score table blockchain platform Principal Principal Principal Principal score products component 1 component 2 component 3 component 4 5 0.584 0.967 0.213 -0.266 1.643 3 0.495 -0.616 1.286 1.187 0.205 6 0.13 0.648 0.576 0.079 -1.415 4 -0.051 1.05 -0.772 -0.258 -0.49 1 -0.472 -0.846 -1.524 0.888 0.108 2 -0.686 -1.203 0.221 -1.63 -0.051 Based on the score in Table 5, product 5 is the best, System and software quality models, and product 2 is the worst. International Standard Organization, 2011. [5] ISO/IEC 25023. Systems and software engineering —Systems and software Quality 4. Conclusion Requirements and Evaluation (SQuaRE) — Measurement of system and software product Blockchain platform products, as a typical software quality, International Standard Organization, system with functional and performance efficiency 2016. indicators, need to comprehensively consider the [6] Yumeng C , Yinglan F .Research on PCA Data support of the platform for algorithms, languages, Dimension Reduction Algorithm Based on consensus mechanisms, and performance efficiency Entropy Weight Method[C]//2020 2nd indicators for scoring. International Conference on Machine Learning, This paper proposed a software quality Big Data and Business Intelligence comparison method based on PCA, which extracts (MLBDBI).2020.DOI:10.1109/MLBDBI51377.20 principal components by analyzing the correlation 20.00084. between data, reasonably allocates and evaluates software quality through dimensionality reduction and weighting, avoids errors caused by subjective experience, and can effectively adapt to changes in evaluation dimensions and the number of software products. Achieve the goal of horizontal comparison between products through a score. References [1] Shi H , Ma J , Zou F . A Fuzzy Comprehensive Evaluation Model for Software Dependability Based on Entropy Weight[C]. International Conference on Computer Science & Software Engineering. IEEE Computer Society, 2008. [2] Bo L , Yang C . An Improved Comprehensive Evaluation Model of Software Dependability based on Rough Set Theory[J]. Journal of Software, 2009, 4(10). [3] Ding S , Yang S L , Fu C . A novel evidential reasoning based method for software trustworthiness evaluation under the uncertain and unreliable environment[J]. Expert Systems with Applications, 2012, 39(3):2700-2709. [4] ISO/IEC 25010. Systems and software engineering —Systems and software Quality Requirements and Evaluation (SQuaRE) — 16