The platform software has a large number of functional and performance efficiency quality indicators, as well as differences in fixed basic hardware and software environments, making it impossible to effectively compare the quality. Considering that different users have different concerns about the product indicators of platform software, it brings certain difficulties to the selection of users. This paper proposes a software quality comparison method based on PCA, which extracts principal components by analyzing the correlation between data, reasonably allocates and evaluates software quality through dimensionality reduction and weighting, avoids errors caused by subjective experience, and can effectively adapt to changes in evaluation dimensions and the number of software products. Achieve the goal of horizontal comparison between products through a score.
1. Introduction With the rapid advancement of information technology, the level of information digitization in all aspects of social life is continuously improving, and software has emerged as a modern infrastructure. However, software is a product of human intellectual labor with poor visibility in terms of quality. The complexity and fuzziness of software make it challenging to quantify, thereby complicating users' ability to objectively comprehend its quality. The construction of a software quality model establishes a framework for measuring software quality attributes, establishing the relationship between measurable attributes and software quality, thereby providing a basis for evaluating and comparing the quality of software products.
In 1991, ISO/IEC JTC1/SC7 issued the ISO/IEC 9126 standard, which is based on McCall and Boehm's quality model. It reformulated the quality of software into 6 main attributes and 21 sub-attributes, marking a significant milestone in the standardization of software quality.
In the research of software quality evaluation,
methods such as Delphi method, fuzzy fuzzy
comprehensive evaluation, topsis, evidential theory,
and so on are often used. Most of these methods rely
on subjective experience or fuzzy theory to construct
quality measurement models, making them
susceptible to human subjective influence in
determining weights, introducing a level of
uncertainty [
ISO/IEC 25010 provides the software product quality model (as shown in Fig. 1), offering eight 5th International Workshop on Experience with SQuaRE Series andits Future Direction, December 04, 2023, Seoul, Korea liwp@cesi.cn (W. Li); yuxiuming@cesi.cn (X. Yu) 0009-0001-4065-2527 (W. Li) © 2023 Copyright for this paper by its authors. The use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
ISO/IEC 25023 provides quality measures for
subcharacteristics of the software product quality model,
which are widely used for measurement functions.
These include mean response time and mean
turnaround time for time behavior measurement;
mean processor utilization, mean memory utilization,
mean I/O utilization, and bandwidth utilization for
resource utilization measurement; transaction
processing capacity, user access capacity, and
adequacy of user access increase in capacity measures
[
The platform software has a large number of functional and performance efficiency quality indicators, as well as differences in fixed basic hardware and software environments, making it impossible to effectively compare the quality. Considering that different users have different concerns about the product indicators of platform software, it brings certain difficulties to the selection of users. In the quality testing of large-scale platform product, function indicators of platform product capability and performance indicators are widely concerned. The results of test records are usually a numerical value (e.g. how many algorithms the platform supports, and the maximum concurrency supported by the performance result is 10,000). Inconsistent data dimensions make it difficult to compare and analyze products, and inconvenience users in comparing products.
This paper introduces a method for analyzing data correlation. This approach eliminates the need for subjective experience and is suitable for conducting large-scale comparisons of product quality. 2. Principal Component
Analysis
Principal Component Analysis (PCA) accomplishes
the objective of eliminating correlations between
features by transforming a set of potentially correlated
variables into a set of linearly independent variables
through orthogonal transformation. This process
retains crucial features while minimizing information
loss. PCA generates two types of coefficients to achieve
these goals: 'weights' that define the transformation
from raw data to summary scores, and 'loadings' that
indicate the strength of association between the raw
variables and the low-dimensional representations [
PCA can be represented by the following mathematical model： (1) x1 = a11F1 + a12F2 + x2 = a21F1 + a22F2 + xp = ap1F1 + ap2F2 + + a1mFm + a11 + a2mFm + a2 2 + apmFm + ap p where, x1，x2，x3， ，xp represent p primitive
F1，F2，F3， ，Fm variables, represent m factor variables, matrix form can be expressed as：
X = AF + a (2) where, F represent common factors, A represent a factor loading matrix, ij represent factor loading.
For determining the weights of indicators in principal component analysis, the first step is to b calculate the coefficients ( ij ) of indicators in the linear combinations of each principal component and c the variance contribution rate ( ij ) of each principal component. The coefficient of each indicator in different linear combinations of principal components b a ( ij ) equals the ratio of the loadings ( ij ) of each indicator to the square root of the eigenvalues ( i ) of a bij = ij each component, which is i . Secondly, the indicator weight is the weighted average of the coefficients of indicators in the linear combinations of principal components, with the weights being the variance contribution rates of the principal components. Finally, the indicator weights are normalized.
In software quality comparison, the matrix used for PCA can be represented as: x11 x X = 21 xp1 x2m = x1
xpm x2 xp
T
Where, p represents the number of software products, and m represents the number of quality characteristics. 3. Software Quality Comparison based on PCA
Analyzing data from some tested blockchain platforms, performance indicators of blockchain platform products, specifically as follows: ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Average response time: the average time it takes for a transaction to be processed and confirmed. This metric is measured by iterating multiple times (with a 1-second interval) and obtaining the average response time for each iteration.
Transaction processing rate: the number of transactions that can be processed per second. This metric measures the overall performance of the blockchain product in terms of transaction processing speed.
Concurrent user/request count: the maximum number of users or requests that can be processed simultaneously. This metric measures the scalability of the blockchain product and its ability to handle multiple concurrent requests.
Data processing volume: the amount of data that needs to be processed for each transaction. This metric measures the size of the transactions being processed and the overall data processing capacity of the blockchain product.
CPU utilization: the percentage of CPU resources being used by the blockchain product. This metric measures the efficiency of the blockchain product in utilizing the available CPU resources.
Memory utilization: the percentage of memory resources being used by the blockchain product. This metric measures the efficiency of the blockchain product in utilizing the available memory resources.
And, core functional indicators as following: ⚫
Supported consensus mechanisms: the number of different consensus mechanisms that the blockchain product supports. ⚫
Supported key algorithms: the number of different key algorithms that the blockchain product supports.
Selecting data from six blockchain platform products, see Table 1 for details.
CPU utilization (%)
Memory utilization (%) consensus mechanisms
Obtain the value of a in formula (1), which shown in Table 4. Constructing a 9 6 matrix as following:
Performing PCA analysis in the above matrix. The cumulative variance explanation rate of the first four eigenvalues in table 2 exceeds 95%. Ingredients, also known as predictors or independent variables refer to the original variables or features in the dataset that you want to reduce the dimensionality. Components are the new variables that are created by PCA to represent the original data in a lower-dimensional space. These components are linear combinations of the original ingredients and are ordered so that they capture the most variance in the data. smart contract development languages 3 4 5 4 5 4 0.762 -0.335 -0.036 -0.805 0.182 -0.43 0.803 0.54 -0.021
F=(0.291/0.961)×F1+(0.278/0.961)×F2+(0.222/ 0.961)×F3+(0.171/0.961)×F4 score
Based on the score in Table 5, product 5 is the best, and product 2 is the worst. 4. Conclusion Blockchain platform products, as a typical software system with functional and performance efficiency indicators, need to comprehensively consider the support of the platform for algorithms, languages, consensus mechanisms, and performance efficiency indicators for scoring.
This paper proposed a software quality comparison method based on PCA, which extracts principal components by analyzing the correlation between data, reasonably allocates and evaluates software quality through dimensionality reduction and weighting, avoids errors caused by subjective experience, and can effectively adapt to changes in evaluation dimensions and the number of software products. Achieve the goal of horizontal comparison between products through a score.
Principal component 1 0.967 -0.616 0.648 1.05 -0.846 -1.203
Principal component 3 -0.266 1.187 0.079 -0.258 0.888 -1.63
Principal component 4 1.643 0.205 -1.415 -0.49 0.108 -0.051
The principal component scores and comprehensive scores of the six products, which listing in TABLE 5.
Firstly, calculate the product of each principal component's linear combination coefficient and its corresponding variance explained rate, then accumulate these products, and finally divide the total variance explained rate by this sum to get the comprehensive score.