=Paper=
{{Paper
|id=Vol-3356/paper1
|storemode=property
|title=SQuaRE Matters: Reflection of Quality Evaluation, Benchmark, and Practioners' Perception through SQuaRE (invited)
|pdfUrl=https://ceur-ws.org/Vol-3356/paper-01.pdf
|volume=Vol-3356
|authors=Hironori Washizaki
|dblpUrl=https://dblp.org/rec/conf/apsec/Washizaki22
}}
==SQuaRE Matters: Reflection of Quality Evaluation, Benchmark, and Practioners' Perception through SQuaRE (invited)==
https://ceur-ws.org/Vol-3356/paper-01.pdf
SQuaRE Matters: Reflection of Software Quality Evaluation,
Benchmark, Pattern Classification and Practitioners’ Perception
through SQuaRE
Hironori Washizaki 1,2,3,4,
1
Waseda University, 3-4-1 Okubo, Shinjuku City, Tokyo, 169-8555, Japan
1
National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda City, Tokyo, 101-0003, Japan
1
SYSTEM INFORMATION CO., LTD., 1-7-3 Kachidoki, Chuo City, Tokyo, 104-0054, Japan
1
eXmotion Co., Ltd., 3-4-3 Kitashinagawa, Shinagawa City, Tokyo, 141-0032, Japan
Abstract
The ISO/IEC 25000 Systems and software Quality Requirements and Evaluation (SQuaRE)
series is a valuable framework to measure and evaluate quality from more multifaceted,
objective, and standardized criteria across products and organizations. This talk introduces
successful use cases of SQuaRE: software systems quality evaluation and benchmarking, and
machine Learning and IoT system design patterns classification with practitioners’ perception.
Keywords 1
Software quality, quality measurement, quality evaluation, standard, software development,
machine learning systems
1. Introduction
2. Quality evaluation and
The ISO/IEC 25000 Systems and software benchmarking
Quality Requirements and Evaluation (SQuaRE)
series is a useful framework to measure and
Conventional quality evaluations of software
evaluate quality from more multifaceted,
concentrate on specific quality characteristics.
objective, and standardized criteria across
Moreover, the measurement data are limited to
products and organizations [1]. SQuaRE is
particular products and organizations.
independent of the domain or product. It
Consequently, the present state of product quality
assembles important quality characteristics,
and quality in use characteristics are not fully
measurement values, and evaluation methods.
understood, preventing effective decision-making
SQuaRE should be a valuable standard for
for software stakeholders. To alleviate this
various use cases, such as software evaluation and
problem, ISO/IEC defined the SQuaRE series for
classification, from the viewpoint of quality
comprehensive quality measurement and
attributes. This talk introduces successful use
evaluation. However, these standards remain
cases of SQuaRE: software systems quality
rather general and abstract, making them difficult
evaluation and benchmarking, and machine
to apply.
Learning and IoT system design patterns
In these papers [2]–[4], we established a
classification with practitioners’ perception.
SQuaRE-based comprehensive software quality
evaluation framework, Waseda Software Quality
Framework (WSQF), which concretizes many
product quality and quality in use measurement
methods originally defined in the SQuaRE series.
4th International Workshop on Experience with SQuaRE Series and
its Future Direction, December 06, 2022, Tokyo, Japan
EMAIL: washizaki@waseda.jp (A. 1)
ORCID: 0000-0002-1417-9879 (A. 1)
© 2022 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Wor
Pr
ks
hop
oceedi
ngs
ht
I
tp:
//
ceur
-
SSN1613-
ws
.or
0073
g
CEUR Workshop Proceedings (CEUR-WS.org)
By applying the WSQF to 21 commercial ready- 17%. Most considered the functional suitability of
to-use software products, we revealed the status of the ML systems and software during design. This
software product quality. A resulted seems natural since the functionality is the most
comprehensive benchmark includes trends of the fundamental attribute of any system and software.
quality measurement values, relationships among In addition, more than 40% of the respondents
quality characteristics, the relationship between considered the maintainability, reliability,
quality-in-use and product quality, and the security, and usability of the ML systems and
relationship between the quality characteristics software. In contrast, portability and compatibility
and product contexts within the limits of an were rarely considered. According to our pattern
application. analysis, maintainability and reliability are well
addressed in existing ML patterns, while security
3. Machine learning design pattern and usability are less addressed; more ML
patterns focusing on security and usability are
classification with practitioners’ anticipated by accumulating more design cases
perception since these characteristics are majorly concerned.
In terms of ML model and prediction quality
Machine learning (ML) software engineering characteristics considered when designing ML
design patterns encapsulate reusable solutions to systems, the top concern was model robustness,
commonly occurring problems within the given followed by model explainability and prediction
contexts of ML systems and software design. accuracy [5]. According to our pattern analysis,
These ML patterns should help develop and model robustness and prediction are well
maintain ML systems and software from the addressed in existing ML patterns, but model
design perspective. However, to the best of our explainability is less addressed. These major
knowledge, there was no study on the characteristics are expected to be reflected in the
practitioners’ insights on the use of ML patterns future development and revision of ML-related
for the design of their ML systems and software. quality model standards, such as the SQuaRE
In these papers [5], [6], we reported the results of quality model for AI systems ISO/IEC DIS 25059
a literature review to identify ML design patterns. and the related AI-specific data quality measures
We also reported a questionnaire-based survey on ISO/IEC AWI 5259-2.
ML system developers’ state-of-practices with
concrete ML patterns. Furthermore, we described 4. IoT system design patterns
most of the identified patterns in previous papers classification
[7]–[10].
Any design pattern should address one or more
quality characteristics that are associated with We have applied a similar analysis to the
design problems. For ML design patterns, we Internet of Things (IoT) design patterns [11], [12].
assumed that the product quality characteristics IoT patterns, including IoT design and
defined in the SQuaRE quality model (i.e., architecture patterns, have been published to
ISO/IEC 25010:2011), as well as ML model and document the successes (and failures) in IoT
prediction quality characteristics, can be systems and software development [13].
addressed. We analyzed the quality characteristics IoT design patterns should mostly address
by reading problems and solutions descriptions of interoperability, which is defined as a sub attribute
the 15 ML design patterns and identifying related of compatibility in SQuaRE since, by definition,
specific descriptions or keywords. Many ML IoT is about ensuring interoperability among
design patterns address maintainability. Most objects. To classify IoT patterns, we used all
operation patterns address model and prediction quality attributes except for functional suitability
quality characteristics [6]. defined in the SQuaRE quality model and selected
Furthermore, we surveyed 300+ software and terms from software engineering: performance,
ML developers who participated in an online compatibility, usability, reliability, security,
seminar on ML patterns in July 2020 in terms of maintainability, and portability. We excluded
perception of quality characteristics considered in functional suitability because certain functional
ML system design and development [5]. Out of requirements are often satisfied by concrete
the 300+ participants, 52 answered our questions, system and software design decisions, including
which corresponds to a response rate of around the reuse of IoT platforms and software libraries,
instead of the reuse of abstract architecture or [2] H. Nakai, N. Tsuda, K. Honda, H. Washizaki,
design patterns. and Y. Fukazawa, “Initial framework for
We observed that some IoT patterns are software quality evaluation based on
dedicated to one or a few quality characteristics, ISO/IEC 25022 and ISO/IEC 25023,” in
while others address many characteristics [12]. QRS Companion. IEEE, 2016, pp. 410–411.
According to SQuaRE, performance, usability, [3] ——, “A square-based software quality
reliability, and security significantly influence the evaluation framework and its case study,” in
quality in use for primary users, while IEEE TENCON. IEEE, 2016, pp. 3704–3707.
compatibility, maintainability, and portability [4] N. Tsuda, H. Washizaki, K. Honda, H. Nakai,
greatly impact quality in use for secondary users Y. Fukazawa, M. Azuma, T. Komiyama, T.
who maintain the system. The former is an Nakano, H. Suzuki, S. Morita, K. Kojima,
important concern of primary users, while the and A. Hando, “WSQF: comprehensive
latter is about the ease of extending a system by software quality evaluation framework and
maintainers in terms of performance. Furthermore, benchmark based on square,” in ICSE (SEIP).
we identified potential additional quality IEEE / ACM, 2019, pp. 312–321.
characteristics for IoT as privacy and scalability. [5] H. Washizaki, H. Takeuchi, F. Khomh, N.
Natori, T. Doi, and S. Okuda, “Practitioners’
5. Conclusion insights on machine-learning software
engineering design patterns: a preliminary
study,” in ICSME. IEEE, 2020, pp. 797–799.
This talk introduced successful use cases of [6] H. Washizaki, F. Khomh, Y. Gu´eh´eneuc, H.
SQuaRE: software systems quality evaluation and Takeuchi, N. Natori, T. Doi, and S. Okuda,
benchmarking and machine Learning and IoT “Software-engineering design patterns for
system design patterns classification with
machine learning applications,” Computer,
practitioners’ perception.
vol. 55, no. 3, pp. 30–39, 2022. [Online].
Future works can include further analysis of Available:
software systems and design patterns from the https://doi.org/10.1109/MC.2021.3137227
quality viewpoints, a relationship model among
[7] H. Washizaki, F. Khomh, and Y.-G.
different quality characteristics, and suggesting
Gu´eh´eneuc, “Software engineering
new quality characteristics to be considered for
patterns for machine learning applications
quality models targeting ML and IoT. (sep4mla),” in 9th Asian Conference on
Pattern Languages of Programs (AsianPLoP
6. Acknowledgements 2020). Hillside, Inc., 2020, pp. 1–10.
[8] H. Washizaki, F. Khomh, Y.-G. Gueheneuc,
This work was partially conducted as a part of H. Takeuchi, S. Okuda, N. Natori, and N.
the Research Initiative on Advanced Software Shioura, “Software engineering patterns for
Engineering in 2015, supported by Software machine learning applications (sep4mla) –
Reliability Enhancement Center (SEC), part 2,” in 27th Conference on Pattern
Information Technology Promotion Agency Languages of Programs in 2020 (PLoP’20).
Japan (IPA). This work was also supported by Hillside, Inc., 2021, pp. 1–10.
JST-Mirai JPMJMI20B8 Engineerable AI (eAI), [9] J. Runpakprakun, S. R. O. Peralta, H.
JSPS JPJSBP 120209936, KAKENHI 21KK0179, Washizaki, F. Khomh, Y.-G. Gueheneuc, N.
and enPiT-Pro Smart SE. Yoshioka, and Y. Fukazawa, “Software
engineering patterns for machine learning
applications (sep4mla) – part 3 – data
7. References processing architectures,” in 28th
Conference on Pattern Languages of
[1] International Organization for Programs in 2021 (PLoP’21). Hillside, Inc.,
Standardization, “ISO/IEC 25010:2011 2021, pp. 1–11.
Systems and software engineering — [10] H. Washizaki, F. Khomh, and Y.-G.
Systems and software Quality Requirements Gueheneuc, “Software engineering patterns
and Evaluation (SQuaRE) – System and for machine learning applications (sep4mla)
software quality models, institution = – part 4 — ml gateway routing architecture,”
International Organization for in 29th Conference on Pattern Languages of
Standardization,” ISO/IEC, Tech. Rep., 2011.
Programs in 2022 (PLoP’22). Hillside, Inc.,
2022, pp. 1–8.
[11] H. Washizaki, N. Yoshioka, A. Hazeyama, T.
Kato, H. Kaiya, S. Ogata, T. Okubo, and E.
B. Fern´andez, “Landscape of iot patterns,”
in SERP4IoT@ICSE. IEEE / ACM, 2019, pp.
57–60.
[12] H. Washizaki, S. Ogata, A. Hazeyama, T.
Okubo, E. B. Fern´andez, and N. Yoshioka,
“Landscape of architecture and design
patterns for iot systems,” IEEE Internet
Things J., vol. 7, no. 10, pp. 10 091–10 101,
2020. [Online]. Available:
https://doi.org/10.1109/JIOT.2020.3003528
[13] H. Washizaki, A. Hazeyama, T. Okubo, H.
Kanuka, S. Ogata, and N. Yoshioka,
“Analysis of iot pattern descriptions,” in
SERP4IoT. IEEE, 2021, pp. 21–26