Revisiting the Quality in Use model Raul Martinez IRAM Expert Subcomité de Calidad en Tecnología de la Información Buenos Aires, Argentina rmartinez582@gmail.com Abstract— This paper examines concepts in the Quality in II. ISO/IEC 25010 DEFINITIONS Use model proposed by SQuaRE standards. A more detailed differentiation is made about the results of the interaction The ISO/IEC 25010:2011 defines Quality in Use as the with the system in a production environment and propositions degree to which a product or system can be used by specific are made for future revisions of the model in the standard. users to meet their needs to achieve specific goals with effectiveness, efficiency, and satisfaction and freedom from Keywords— SQuaRE, Quality in Use, Quality Models, risk in specific contexts of use. Quality in Production Environments, ISO/IEC 25023, ISO/IEC 25022. Quality is modelled through five characteristics related to outcomes of interaction with a system: effectiveness, I. INTRODUCTION efficiency, satisfaction, freedom from risk and context coverage. In many cases, Quality in Use model is highly, or only, associated with usability. This vision is at least partial. For Quality in Use model the term usability refers to the subset of quality in use composed of effectiveness, This paper proposes re-examining one of the more efficiency, satisfaction, and context of use coverage. important SQuaRE quality models, the Quality in Use model. Figure 1 is a simplified graphical explanation of Quality in Use model and its relationship with a product in Basic concepts are reviewed, and several suggestions are offered to be considered on future revisions of the model. A. THE QUALITY IN USE MODEL CONCEPT When a system runs in a production environment, Result of the use considerations about its quality could be based at least on: of the system (need fulfilled assumed positive) 1. Usability of the system (if a human user exists) Outputs Usability Things produced (virtual or real) 2. Generation of the correct expected outputs Consequences of the use of the system (assumed 3. Fulfilling of expected positive outcomes (results) neutral or negative) 4. Avoidance of possible negative outcomes (consequences) If these conditions are met within acceptable values the System interactions outcomes system will be considered as a quality (enough) system. Context of execution Usually people describe desired solutions as required outcomes and an unrefined request about how these outcomes are obtained. operation. These elements, at a very early stage, are the first outline Figure 1 Graphical explanation of quality in use model to the Quality in Use model for the system because they contain quality concepts expected and valuable for the System in a production environment sequence: customer. 1 Product execution: This Quality in Use model will then depict the behaviour a. System started, with or without human interaction of the system in a production environment compared with target behaviour established at modelling stage. 2. Resultant outcomes: a. Outputs produced It should be noted this paper is focused on the final user but that, as stated by [4] ISO/IEC 25010:2011, “Other b. Usability attained stakeholders, such as software developers, system c. Results achieved integrators, acquirers, owners, maintainers, contractors, d. Consequences derived quality assurance and control professionals, will also be concerned with the quality.” 3. System running in a specific context of use Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). III. REVIEWING COMPONENTS OF THE QUALITY IN USE • Consequences: ISO/IEC 25022:2016 defines MODEL consequence as an outcome emerging from a negative risk (not an opportunity). A. Interactions and outcomes Different measures are provided by the model for Enumerated risks include threats to economic usability (effectiveness, efficiency, and satisfaction that status, human life, health, or the environment. includes user experience and ergonomics). All these The quality measures could compare the target measures apply if a human user, primary, secondary or value expected for each risk or consequence and indirect exists. the actual value obtained during executions. The model also considers consequences and results, Definitions for Outputs, Results, Consequences and explained later. Figure 2 shows the measures proposed for Outcomes concluded or obtained from [6], [7], [8], [1]. the production environment. C. Points to consider on interactions and outcomes Result Many existing systems operate exchanging activities achieved with other systems, and subsequently all of them share measures responsibility for the results. Result of the use of the system Organizations or individuals using the system will perceive (need fulfilled assumed positive) quality without an exact discrimination of quality of each Outputs (*) intervenient component. Usability Things produced (virtual or real) Consequences of - Should Quality in Use model offer properties and Usability the use of the system (assumed measures for this kind of systems of systems? measures External neutral or negative) quality measure Today we are developing systems that “see”, “hear” and Consequences “know” where they are located, making decisions and derived from use measures executing actions in actual environments based on incoming data and, for example, previously learned behaviours. System interactions outcomes - Will there be a quality measure for bias of the Context of execution decisions of the algorithms? Which will be the target value? Figure 2. Measures proposed for the production environment. - Will there be a quality measure for ethics of the solution? Which will be the target value? (*) Output is outside Quality in Use model proposition - Will it be acceptable to call these “interactions”? but is generated as an outcome during execution on a production environment; for that reason, measures from D. Context of use ISO/IEC 25023:2016, as functional suitability and ISO/IEC 25022:2016 defines context of use as users, tasks, appropriateness, and measures from ISO/IEC 25024:2015 equipment (hardware, software and materials), and the for Data Quality could be used to measure quality of Output physical and social environments in which a system, produced product or service is used. B. Outcomes [SOURCE: ISO 9241‑11:1998, 3.5, modified — With Outcomes are goals, projected or attained, that are the main “product” replaced by “system, product or service”.] reason of the existence of the system. The proposed quality property associated to context is Some clarification is required to distinguish between the Coverage. Completeness and Flexibility are the measures different outcomes: outputs, results and consequences: proposed for this property. • Outputs: Things, virtual or real, produced or Such properties and measures are oriented towards changes changed as effect of the execution of the system. in user skills or capabilities, type of users, and several target scenarios. These is correct for some types of systems where As mentioned above Output is outside Quality in contexts can be enumerated. Use model proposition but is generated as an E. Points to consider on context of use outcome during execution on a production environment, for that reason quality measures - Is Coverage as proposed by the Quality in Use model from ISO/IEC 25023:2016 could be used. applicable to highly variable / unknow contexts? • Results: Short-term or long-term effects obtained - How is quality measured in these cases? How are from the use of the system. completeness and flexibility measured? - Context changes could require user actions or It is assumed that they are related to needs produce diverse outcomes. How could we measure fulfilled and are positive. The quality measures the quality of the produced response to the new could compare the target value expected for each context? result and the actual value obtained during executions, one or many. IV. CONCLUSION product. Fulfilment of first expectations and actual Quality in Use model is a very important model whose behaviour of product in Production conform the user elements, as mentioned, are identified at the very beginning perceived quality. of the development process and completed throughout the REFERENCES product life cycle. [1] ISO/IEC 25022:2016, Systems and Software engineering The paper rises several questions about meaning and content Measurement of quality in use) of the current Quality in Use model. These questions will [2] ISO/IEC 25023:2016, Systems and Software engineering require a deep discussion and opinions exchange, but some measurement of system and software product quality. key ideas are briefed below. [3] ISO/IEC 25024 Systems and software quality requirements and evaluation (SQUARE) - Measurement of data quality • The model should be detached from the usability [4] ISO/IEC 25010:2011. ISO/IEC 25010:2011 Systems and Software centred idea and the value of the model should be engineering System and software quality models emphasised. Possibly a new name for the model [5] ISO 9241‑11:1998 has been revised as ISO9241-11:2018 should be considered showing the relationship [6] Deborah Mills-Scofield It’s Not Just Semantics: Managing between the model and the operation of the system in Outcomes Vs. Outputs Harvard Business Review, 2012 diverse and mutating production environments. A [7] Koopman and Fratrik, How Many Operational Design Domains, suggestion could be Production Quality Model. Objects, and Events?, Safe AI 2019 talk, 2019 [8] INTRAC 2015, Outputs, Outcomes and Impact • ISO/IEC 25022:2016 establishes that the external measures can only be used during “the testing stages of the life cycle process and during any operational stages”. Included in this definition are environments identical to production used for testing process. For that reason, external measures that apply to production environment should be moved to Quality in Use model obtaining a complete picture of system execution. • Current and emerging systems should be considered by the Quality in Use model, as new forms of input, algorithms, processing and communications are constantly evolving and new ones appearing. Nevertheless, the correct level of abstraction should be maintained, and the new characteristics applicable to these kinds of systems must be considered. These characteristics could be allocated on a Technical Report suitable to the specific kind of ICT product without compromising the level of abstraction of the general model. • Measures applicable to Usability, Results and Consequences need to be analysed and/or defined in accordance to the kind of ICT product. • Current Service Quality model should be re-examined considering at least two possibilities. First, if it really is a different model or it is part of the Quality in Use model. Second, consider the Service Quality model as an specialization of the Quality in Use model. • The fundamental concept of Context should be re- examined. Current Quality in Use model includes two basic measures for context, Completeness and Flexibility. The measures embrace important ideas but with completely different meaning depending on the kind of systems. For this reason, context measures should be defined using for example a Technical Report mechanism as stated previously. As a final thought, Quality in Use model appears at the very beginning as quality expectations about the future product and at the end as outcomes: outputs, results and consequences, of the use of the actual product in Production environments. Between expectations and production, other models need to be developed, Product Quality Model and Data Quality Model, stating the quality characteristics required on the