=Paper=
{{Paper
|id=Vol-3612/IWESQ_2023_Paper_02
|storemode=property
|title=A Method of Software Quality Comparison based on PCA
|pdfUrl=https://ceur-ws.org/Vol-3612/IWESQ_2023_Paper_02.pdf
|volume=Vol-3612
|authors=Li Wenpeng,Wei Su,Jianxun Guo,Kuan Feng,Xiuming Yu
|dblpUrl=https://dblp.org/rec/conf/apsec/LiSGFY23
}}
==A Method of Software Quality Comparison based on PCA==
https://ceur-ws.org/Vol-3612/IWESQ_2023_Paper_02.pdf
A Method of Software Quality Comparison based on PCA
Wenpeng Li , Wei Su, Jianxun Guo, Kuan Feng, Xiuming Yu*
China Electronics Standardization Institute, Beijing, China
Abstract
The platform software has a large number of functional and performance efficiency quality indicators,
as well as differences in fixed basic hardware and software environments, making it impossible to
effectively compare the quality. Considering that different users have different concerns about the
product indicators of platform software, it brings certain difficulties to the selection of users. This paper
proposes a software quality comparison method based on PCA, which extracts principal components by
analyzing the correlation between data, reasonably allocates and evaluates software quality through
dimensionality reduction and weighting, avoids errors caused by subjective experience, and can
effectively adapt to changes in evaluation dimensions and the number of software products. Achieve the
goal of horizontal comparison between products through a score.
Keywords
quality comparison, quality models, PCA
1
functional characteristics: Functional Suitability,
1. Introduction Performance efficiency, Compatibility, Usability,
Reliability, Security, Maintainability, and Portability,
With the rapid advancement of information along with their respective sub-characteristics and
technology, the level of information digitization in all properties [4].
aspects of social life is continuously improving, and
software has emerged as a modern infrastructure.
However, software is a product of human intellectual
labor with poor visibility in terms of quality. The
complexity and fuzziness of software make it
challenging to quantify, thereby complicating users'
ability to objectively comprehend its quality. The
construction of a software quality model establishes a
framework for measuring software quality attributes,
establishing the relationship between measurable
attributes and software quality, thereby providing a
basis for evaluating and comparing the quality of
software products. Figure 1: Product quality model
In 1991, ISO/IEC JTC1/SC7 issued the ISO/IEC
9126 standard, which is based on McCall and Boehm's ISO/IEC 25023 provides quality measures for sub-
quality model. It reformulated the quality of software characteristics of the software product quality model,
into 6 main attributes and 21 sub-attributes, marking which are widely used for measurement functions.
a significant milestone in the standardization of These include mean response time and mean
software quality. turnaround time for time behavior measurement;
In the research of software quality evaluation, mean processor utilization, mean memory utilization,
methods such as Delphi method, fuzzy fuzzy mean I/O utilization, and bandwidth utilization for
comprehensive evaluation, topsis, evidential theory, resource utilization measurement; transaction
and so on are often used. Most of these methods rely processing capacity, user access capacity, and
on subjective experience or fuzzy theory to construct adequacy of user access increase in capacity measures
quality measurement models, making them [5].
susceptible to human subjective influence in The platform software has a large number of
determining weights, introducing a level of functional and performance efficiency quality
uncertainty [1, 2, 3]. indicators, as well as differences in fixed basic
ISO/IEC 25010 provides the software product hardware and software environments, making it
quality model (as shown in Fig. 1), offering eight impossible to effectively compare the quality.
5th International Workshop on Experience with SQuaRE Series andits
Future Direction, December 04, 2023, Seoul, Korea
liwp@cesi.cn (W. Li); yuxiuming@cesi.cn (X. Yu)
0009-0001-4065-2527 (W. Li)
© 2023 Copyright for this paper by its authors. The use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
13
Considering that different users have different coefficients of indicators in the linear combinations of
concerns about the product indicators of platform principal components, with the weights being the
software, it brings certain difficulties to the selection variance contribution rates of the principal
of users. In the quality testing of large-scale platform components. Finally, the indicator weights are
product, function indicators of platform product normalized.
capability and performance indicators are widely In software quality comparison, the matrix used
concerned. The results of test records are usually a for PCA can be represented as:
numerical value (e.g. how many algorithms the
platform supports, and the maximum concurrency x11 x12 x1m
supported by the performance result is 10,000). x x22 x2 m
X =
T
Inconsistent data dimensions make it difficult to = x x p
21
x2
compare and analyze products, and inconvenience 1
users in comparing products.
This paper introduces a method for analyzing data x p1 xp2 x pm
correlation. This approach eliminates the need for Where, p represents the number of software
subjective experience and is suitable for conducting products, and m represents the number of quality
large-scale comparisons of product quality. characteristics.
3. Software Quality Comparison
2. Principal Component
based on PCA
Analysis
Analyzing data from some tested blockchain
Principal Component Analysis (PCA) accomplishes platforms, performance indicators of blockchain
the objective of eliminating correlations between platform products, specifically as follows:
features by transforming a set of potentially correlated
variables into a set of linearly independent variables ⚫ Average response time: the average time it
through orthogonal transformation. This process takes for a transaction to be processed and
retains crucial features while minimizing information confirmed. This metric is measured by
loss. PCA generates two types of coefficients to achieve iterating multiple times (with a 1-second
these goals: 'weights' that define the transformation interval) and obtaining the average response
from raw data to summary scores, and 'loadings' that time for each iteration.
indicate the strength of association between the raw ⚫ Transaction processing rate: the number of
variables and the low-dimensional representations [6]. transactions that can be processed per
PCA can be represented by the following second. This metric measures the overall
mathematical model： performance of the blockchain product in
terms of transaction processing speed.
x1 = a11F1 + a12 F2 + + a1m Fm + a11 ⚫ Concurrent user/request count: the
x =a F +a F + +a F +a
2 21 1 22 2 2m m 2 2 maximum number of users or requests that
(1) can be processed simultaneously. This
metric measures the scalability of the
x p = a p1F1 + a p 2 F2 + + a pm Fm + a p p
blockchain product and its ability to handle
x ， x ， x ， ，x p multiple concurrent requests.
where, 1 2 3 represent p primitive
F1，F2，F3， ，Fm ⚫ Data processing volume: the amount of data
variables, represent m factor that needs to be processed for each
variables, matrix form can be expressed as： transaction. This metric measures the size of
the transactions being processed and the
X = AF + a (2)
overall data processing capacity of the
where, F represent common factors, A represent blockchain product.
a
factor loading matrix, ij represent factor loading. ⚫ CPU utilization: the percentage of CPU
For determining the weights of indicators in resources being used by the blockchain
principal component analysis, the first step is to product. This metric measures the efficiency
b of the blockchain product in utilizing the
calculate the coefficients ( ij ) of indicators in the available CPU resources.
linear combinations of each principal component and
c ⚫ Memory utilization: the percentage of
the variance contribution rate ( ij ) of each principal memory resources being used by the
component. The coefficient of each indicator in blockchain product. This metric measures
different linear combinations of principal components the efficiency of the blockchain product in
b a utilizing the available memory resources.
( ij ) equals the ratio of the loadings ( ij ) of each
And, core functional indicators as following:
indicator to the square root of the eigenvalues ( i ) of
a ⚫ Supported consensus mechanisms: the
bij = ij number of different consensus mechanisms
i
each component, which is . Secondly, the that the blockchain product supports.
indicator weight is the weighted average of the
14
⚫ Supported smart contract development ⚫ Supported key algorithms: the number of
languages: the number of different smart different key algorithms that the blockchain
contract development languages that the product supports.
blockchain product supports.
Selecting data from six blockchain platform
products, see Table 1 for details.
Table 1
Blockchain platform products test data
Average Data smart
blockchain Transaction Concurrent CPU Memory
response processing consensus contract key
platform processing user/request utilization utilization
time volume mechanisms development algorithms
products rate count (%) (%)
(ms) (MB) languages
1 7 12882 167272 490.05 74.7 8.3 4 3 6
2 9 44200 45000 1560.99 18 15 1 4 5
3 2 150918 111040 73.35 23 16 3 5 7
4 59 693 17845 16 43 20 5 4 5
5 29 799 27424 9 50 61 3 5 6
6 66 59241 8514 367 10 15 4 4 7
Constructing a 9 6 matrix as following: Transaction
processing -0.41 0.762 0.429
product1 rate
product 2 Concurrent
user/request -0.678 -0.335 0.623
product 3 count
X = Data
product 4 processing -0.699 -0.036 -0.674
product 5 volume
CPU
product 6 utilization
0.008 -0.805 0.337
Performing PCA analysis in the above matrix. The Memory
0.57 0.182 -0.191
cumulative variance explanation rate of the first four utilization
eigenvalues in table 2 exceeds 95%. Ingredients, also consensus
0.631 -0.43 0.543
known as predictors or independent variables refer to mechanisms
the original variables or features in the dataset that smart
you want to reduce the dimensionality. Components contract
0.318 0.803 0.009
are the new variables that are created by PCA to developmen
represent the original data in a lower-dimensional t languages
space. These components are linear combinations of key
0.041 0.54 0.705
the original ingredients and are ordered so that they algorithms
capture the most variance in the data.
Table 2 Obtain the value of a in formula (1), which shown
Variance explained table in Table 4.
Cumulative Table 4
Explained
explained Ingredient matrix table
Ingredients Eigenvalue Variance
variance Ingredient Ingredient Ingredient
(%) indicators
(%) 1 2 3
1 2.621 29.122 29.122 Average
2 2.499 27.762 56.884 response 0.314 -0.021 -0.086
3 1.998 22.204 79.088 time
4 1.535 17.055 96.143 Transaction
5 0.347 3.857 100 processing -0.156 0.305 0.215
6 100 rate
Concurrent
The factor loading coefficients were calculated user/request -0.259 -0.134 0.312
using the first four eigenvalues, and the results are count
shown in Table 3. Data
processing -0.267 -0.014 -0.337
Table 3 volume
Factor load factor CPU
0.003 -0.322 0.169
Principal Principal Principal utilization
indicators componen componen componen Memory
0.218 0.073 -0.095
t1 t2 t3 utilization
Average consensus
0.241 -0.172 0.272
response 0.824 -0.053 -0.171 mechanisms
time
15
smart The principal component scores and
contract comprehensive scores of the six products, which
0.121 0.321 0.004 listing in TABLE 5.
development
languages
Firstly, calculate the product of each principal
key component's linear combination coefficient and its
0.016 0.216 0.353
algorithms corresponding variance explained rate, then
accumulate these products, and finally divide the total
Obtain the formula (2) as follows: variance explained rate by this sum to get the
comprehensive score.
F=(0.291/0.961)×F1+(0.278/0.961)×F2+(0.222/
0.961)×F3+(0.171/0.961)×F4
Table 5
Comprehensive score table
blockchain platform Principal Principal Principal Principal
score
products component 1 component 2 component 3 component 4
5 0.584 0.967 0.213 -0.266 1.643
3 0.495 -0.616 1.286 1.187 0.205
6 0.13 0.648 0.576 0.079 -1.415
4 -0.051 1.05 -0.772 -0.258 -0.49
1 -0.472 -0.846 -1.524 0.888 0.108
2 -0.686 -1.203 0.221 -1.63 -0.051
Based on the score in Table 5, product 5 is the best, System and software quality models,
and product 2 is the worst. International Standard Organization, 2011.
[5] ISO/IEC 25023. Systems and software
engineering —Systems and software Quality
4. Conclusion Requirements and Evaluation (SQuaRE) —
Measurement of system and software product
Blockchain platform products, as a typical software quality, International Standard Organization,
system with functional and performance efficiency 2016.
indicators, need to comprehensively consider the [6] Yumeng C , Yinglan F .Research on PCA Data
support of the platform for algorithms, languages, Dimension Reduction Algorithm Based on
consensus mechanisms, and performance efficiency Entropy Weight Method[C]//2020 2nd
indicators for scoring.
International Conference on Machine Learning,
This paper proposed a software quality Big Data and Business Intelligence
comparison method based on PCA, which extracts (MLBDBI).2020.DOI:10.1109/MLBDBI51377.20
principal components by analyzing the correlation 20.00084.
between data, reasonably allocates and evaluates
software quality through dimensionality reduction
and weighting, avoids errors caused by subjective
experience, and can effectively adapt to changes in
evaluation dimensions and the number of software
products. Achieve the goal of horizontal comparison
between products through a score.
References
[1] Shi H , Ma J , Zou F . A Fuzzy Comprehensive
Evaluation Model for Software Dependability
Based on Entropy Weight[C]. International
Conference on Computer Science & Software
Engineering. IEEE Computer Society, 2008.
[2] Bo L , Yang C . An Improved Comprehensive
Evaluation Model of Software Dependability
based on Rough Set Theory[J]. Journal of
Software, 2009, 4(10).
[3] Ding S , Yang S L , Fu C . A novel evidential
reasoning based method for software
trustworthiness evaluation under the uncertain
and unreliable environment[J]. Expert Systems
with Applications, 2012, 39(3):2700-2709.
[4] ISO/IEC 25010. Systems and software
engineering —Systems and software Quality
Requirements and Evaluation (SQuaRE) —
16