SQuaRE Matters: Reflection of Software Quality Evaluation, Benchmark, Pattern Classification and Practitioners’ Perception through SQuaRE Hironori Washizaki 1,2,3,4, 1 Waseda University, 3-4-1 Okubo, Shinjuku City, Tokyo, 169-8555, Japan 1 National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda City, Tokyo, 101-0003, Japan 1 SYSTEM INFORMATION CO., LTD., 1-7-3 Kachidoki, Chuo City, Tokyo, 104-0054, Japan 1 eXmotion Co., Ltd., 3-4-3 Kitashinagawa, Shinagawa City, Tokyo, 141-0032, Japan Abstract The ISO/IEC 25000 Systems and software Quality Requirements and Evaluation (SQuaRE) series is a valuable framework to measure and evaluate quality from more multifaceted, objective, and standardized criteria across products and organizations. This talk introduces successful use cases of SQuaRE: software systems quality evaluation and benchmarking, and machine Learning and IoT system design patterns classification with practitioners’ perception. Keywords 1 Software quality, quality measurement, quality evaluation, standard, software development, machine learning systems 1. Introduction 2. Quality evaluation and The ISO/IEC 25000 Systems and software benchmarking Quality Requirements and Evaluation (SQuaRE) series is a useful framework to measure and Conventional quality evaluations of software evaluate quality from more multifaceted, concentrate on specific quality characteristics. objective, and standardized criteria across Moreover, the measurement data are limited to products and organizations [1]. SQuaRE is particular products and organizations. independent of the domain or product. It Consequently, the present state of product quality assembles important quality characteristics, and quality in use characteristics are not fully measurement values, and evaluation methods. understood, preventing effective decision-making SQuaRE should be a valuable standard for for software stakeholders. To alleviate this various use cases, such as software evaluation and problem, ISO/IEC defined the SQuaRE series for classification, from the viewpoint of quality comprehensive quality measurement and attributes. This talk introduces successful use evaluation. However, these standards remain cases of SQuaRE: software systems quality rather general and abstract, making them difficult evaluation and benchmarking, and machine to apply. Learning and IoT system design patterns In these papers [2]–[4], we established a classification with practitioners’ perception. SQuaRE-based comprehensive software quality evaluation framework, Waseda Software Quality Framework (WSQF), which concretizes many product quality and quality in use measurement methods originally defined in the SQuaRE series. 4th International Workshop on Experience with SQuaRE Series and its Future Direction, December 06, 2022, Tokyo, Japan EMAIL: washizaki@waseda.jp (A. 1) ORCID: 0000-0002-1417-9879 (A. 1) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Wor Pr ks hop oceedi ngs ht I tp: // ceur - SSN1613- ws .or 0073 g CEUR Workshop Proceedings (CEUR-WS.org) By applying the WSQF to 21 commercial ready- 17%. Most considered the functional suitability of to-use software products, we revealed the status of the ML systems and software during design. This software product quality. A resulted seems natural since the functionality is the most comprehensive benchmark includes trends of the fundamental attribute of any system and software. quality measurement values, relationships among In addition, more than 40% of the respondents quality characteristics, the relationship between considered the maintainability, reliability, quality-in-use and product quality, and the security, and usability of the ML systems and relationship between the quality characteristics software. In contrast, portability and compatibility and product contexts within the limits of an were rarely considered. According to our pattern application. analysis, maintainability and reliability are well addressed in existing ML patterns, while security 3. Machine learning design pattern and usability are less addressed; more ML patterns focusing on security and usability are classification with practitioners’ anticipated by accumulating more design cases perception since these characteristics are majorly concerned. In terms of ML model and prediction quality Machine learning (ML) software engineering characteristics considered when designing ML design patterns encapsulate reusable solutions to systems, the top concern was model robustness, commonly occurring problems within the given followed by model explainability and prediction contexts of ML systems and software design. accuracy [5]. According to our pattern analysis, These ML patterns should help develop and model robustness and prediction are well maintain ML systems and software from the addressed in existing ML patterns, but model design perspective. However, to the best of our explainability is less addressed. These major knowledge, there was no study on the characteristics are expected to be reflected in the practitioners’ insights on the use of ML patterns future development and revision of ML-related for the design of their ML systems and software. quality model standards, such as the SQuaRE In these papers [5], [6], we reported the results of quality model for AI systems ISO/IEC DIS 25059 a literature review to identify ML design patterns. and the related AI-specific data quality measures We also reported a questionnaire-based survey on ISO/IEC AWI 5259-2. ML system developers’ state-of-practices with concrete ML patterns. Furthermore, we described 4. IoT system design patterns most of the identified patterns in previous papers classification [7]–[10]. Any design pattern should address one or more quality characteristics that are associated with We have applied a similar analysis to the design problems. For ML design patterns, we Internet of Things (IoT) design patterns [11], [12]. assumed that the product quality characteristics IoT patterns, including IoT design and defined in the SQuaRE quality model (i.e., architecture patterns, have been published to ISO/IEC 25010:2011), as well as ML model and document the successes (and failures) in IoT prediction quality characteristics, can be systems and software development [13]. addressed. We analyzed the quality characteristics IoT design patterns should mostly address by reading problems and solutions descriptions of interoperability, which is defined as a sub attribute the 15 ML design patterns and identifying related of compatibility in SQuaRE since, by definition, specific descriptions or keywords. Many ML IoT is about ensuring interoperability among design patterns address maintainability. Most objects. To classify IoT patterns, we used all operation patterns address model and prediction quality attributes except for functional suitability quality characteristics [6]. defined in the SQuaRE quality model and selected Furthermore, we surveyed 300+ software and terms from software engineering: performance, ML developers who participated in an online compatibility, usability, reliability, security, seminar on ML patterns in July 2020 in terms of maintainability, and portability. We excluded perception of quality characteristics considered in functional suitability because certain functional ML system design and development [5]. Out of requirements are often satisfied by concrete the 300+ participants, 52 answered our questions, system and software design decisions, including which corresponds to a response rate of around the reuse of IoT platforms and software libraries, instead of the reuse of abstract architecture or [2] H. Nakai, N. Tsuda, K. Honda, H. Washizaki, design patterns. and Y. Fukazawa, “Initial framework for We observed that some IoT patterns are software quality evaluation based on dedicated to one or a few quality characteristics, ISO/IEC 25022 and ISO/IEC 25023,” in while others address many characteristics [12]. QRS Companion. IEEE, 2016, pp. 410–411. According to SQuaRE, performance, usability, [3] ——, “A square-based software quality reliability, and security significantly influence the evaluation framework and its case study,” in quality in use for primary users, while IEEE TENCON. IEEE, 2016, pp. 3704–3707. compatibility, maintainability, and portability [4] N. Tsuda, H. Washizaki, K. Honda, H. Nakai, greatly impact quality in use for secondary users Y. Fukazawa, M. Azuma, T. Komiyama, T. who maintain the system. The former is an Nakano, H. Suzuki, S. Morita, K. Kojima, important concern of primary users, while the and A. Hando, “WSQF: comprehensive latter is about the ease of extending a system by software quality evaluation framework and maintainers in terms of performance. Furthermore, benchmark based on square,” in ICSE (SEIP). we identified potential additional quality IEEE / ACM, 2019, pp. 312–321. characteristics for IoT as privacy and scalability. [5] H. Washizaki, H. Takeuchi, F. Khomh, N. Natori, T. Doi, and S. Okuda, “Practitioners’ 5. Conclusion insights on machine-learning software engineering design patterns: a preliminary study,” in ICSME. IEEE, 2020, pp. 797–799. This talk introduced successful use cases of [6] H. Washizaki, F. Khomh, Y. Gu´eh´eneuc, H. SQuaRE: software systems quality evaluation and Takeuchi, N. Natori, T. Doi, and S. Okuda, benchmarking and machine Learning and IoT “Software-engineering design patterns for system design patterns classification with machine learning applications,” Computer, practitioners’ perception. vol. 55, no. 3, pp. 30–39, 2022. [Online]. Future works can include further analysis of Available: software systems and design patterns from the https://doi.org/10.1109/MC.2021.3137227 quality viewpoints, a relationship model among [7] H. Washizaki, F. Khomh, and Y.-G. different quality characteristics, and suggesting Gu´eh´eneuc, “Software engineering new quality characteristics to be considered for patterns for machine learning applications quality models targeting ML and IoT. (sep4mla),” in 9th Asian Conference on Pattern Languages of Programs (AsianPLoP 6. Acknowledgements 2020). Hillside, Inc., 2020, pp. 1–10. [8] H. Washizaki, F. Khomh, Y.-G. Gueheneuc, This work was partially conducted as a part of H. Takeuchi, S. Okuda, N. Natori, and N. the Research Initiative on Advanced Software Shioura, “Software engineering patterns for Engineering in 2015, supported by Software machine learning applications (sep4mla) – Reliability Enhancement Center (SEC), part 2,” in 27th Conference on Pattern Information Technology Promotion Agency Languages of Programs in 2020 (PLoP’20). Japan (IPA). This work was also supported by Hillside, Inc., 2021, pp. 1–10. JST-Mirai JPMJMI20B8 Engineerable AI (eAI), [9] J. Runpakprakun, S. R. O. Peralta, H. JSPS JPJSBP 120209936, KAKENHI 21KK0179, Washizaki, F. Khomh, Y.-G. Gueheneuc, N. and enPiT-Pro Smart SE. Yoshioka, and Y. Fukazawa, “Software engineering patterns for machine learning applications (sep4mla) – part 3 – data 7. References processing architectures,” in 28th Conference on Pattern Languages of [1] International Organization for Programs in 2021 (PLoP’21). Hillside, Inc., Standardization, “ISO/IEC 25010:2011 2021, pp. 1–11. Systems and software engineering — [10] H. Washizaki, F. Khomh, and Y.-G. Systems and software Quality Requirements Gueheneuc, “Software engineering patterns and Evaluation (SQuaRE) – System and for machine learning applications (sep4mla) software quality models, institution = – part 4 — ml gateway routing architecture,” International Organization for in 29th Conference on Pattern Languages of Standardization,” ISO/IEC, Tech. Rep., 2011. Programs in 2022 (PLoP’22). Hillside, Inc., 2022, pp. 1–8. [11] H. Washizaki, N. Yoshioka, A. Hazeyama, T. Kato, H. Kaiya, S. Ogata, T. Okubo, and E. B. Fern´andez, “Landscape of iot patterns,” in SERP4IoT@ICSE. IEEE / ACM, 2019, pp. 57–60. [12] H. Washizaki, S. Ogata, A. Hazeyama, T. Okubo, E. B. Fern´andez, and N. Yoshioka, “Landscape of architecture and design patterns for iot systems,” IEEE Internet Things J., vol. 7, no. 10, pp. 10 091–10 101, 2020. [Online]. Available: https://doi.org/10.1109/JIOT.2020.3003528 [13] H. Washizaki, A. Hazeyama, T. Okubo, H. Kanuka, S. Ogata, and N. Yoshioka, “Analysis of iot pattern descriptions,” in SERP4IoT. IEEE, 2021, pp. 21–26