A software quality assessment is a mandatory process in ensuring the required software quality as part of the overall software development process. The constant development of existing information technologies and the emergence of new information technologies (artificial intelligence, cloud computing, virtual and augmented reality, etc.) and systems increase the requirements for the assessment process and software quality assurance. This evolution among the existing approaches and paradigms of software quality assessment has insufficient dynamics, as there are significant problems, including weak formalization in the planning software quality assessment tasks, a high degree of uncertainty in decision-making by responsible participants of the process, lack or excess of necessary initial information, formation of participants group of software quality assessment process. Especially such problems are clearly expressed in the methods of software quality assessment based on the software fault injection.
One of the existing approaches that can formalize the software quality assessment process to the required level is the scenario approach. The analyzed works on the organization and formalization of the software quality assessment process describe some cases in the development of the software quality assessment process [1][2][3][4], and the scenario approach is described in part, at the level of some elements [5][6][7]. There are works on the scenario approach, but it is considered in general as an approach to management [8][9][10][11][12], without taking into account the specifics of the software quality assessment in general. The scenario approach is not conceptually considered in the works on software quality assessment based on the software fault injection [13,14]. In some papers [15] on software quality assessment, the scenario model is considered in general, but it does not take into account important features of this process.
Thus, the aim of the article is to develop an extended model of software quality assessment scenario, which takes into account its elements, features of state change, etc.
We will present and formally describe a scenario-oriented approach to software quality assessment. First of all, consider the concept of the scenario. The word «scenario» comes from the Latin word «scaena», which translates as «scene». Initially, the scenario was considered as a literary and dramatic work, written as a basis for the production of film or television, and other events in the theater and elsewhere. In the twentieth century, Herman Kahn, a leading analyst at the RAND Corporation [16], adapted this term for use in writing possible stories of future developments. Kahn is often cited as a father of scenario planning. Oliver Sparrow, one of the founders of the scenario approach at the Royal Dutch Shell Corporation, distinguished four modern interpretations of this term [17]:
as «a sensitivity analysis» whether in cash flow management, broader risk assessment, or project management; as synonymous with the concept of «contingency plan» in military or civilian contingency planning, determining who and what to do in the event of an emergency;
as synonymous with a contingency plan in corporate or public policy; in the sense of «logically agreed assumptions about the future» in decision-making and strategy formation.
All the main definitions are summarized by the Dutch researcher Philip Van Notten in the following definition [18]: a scenario is a consistent description of alternative hypothetically possible variants for future events, which reflects different perspectives on the past, present, and future, and which can be the basis for action planning.
Adapting the presented definition of the scenario for software quality assessment, we obtain the following interpretation: a software quality assessment scenario is a product of planning and describing a (continuous) sequence of actions aimed at software quality assessment, which includes a description of initial conditions, inputs, expected results (hypotheses), corrective factors and distribution of participant roles in software quality assessment process. Among the process participant roles are the following: organizer (research engineer) of the software quality assessment process (scenario developer), head of the software testing team (quality team leader), tester (quality engineer), user. Thus, the software quality assessment scenario includes the following 7 elements: actions, transition data transmitted from stage to stage, roles, input data, corrective factors, initial conditions, expected result, or hypothesis. Elements of the scenario are presented in general form: graphic (Fig. 1) and formal form: Experimentally, it was found that the scenario during its life cycle (Fig. 2) evolves and is presented in the following three states:
«pilot scenario» (PSAQSW -Pilot Scenario of Assessment of Quality of Software). This is a scenario on paper that runs in test mode. Such a scenario is needed to work out and refine the scenario on a real test case. As a rule, the number of participants involved in the scenario is minimal. Typically, this scenario differs from the scenario on paper by refining the scenario elements. To indicate this state of the scenario for each set an index «P» is added ( 3
(3)
«real scenario» (RSAQSW -Real Scenario of Assessment of Quality of Software). This state of the scenario is used to assess the software quality for the actual object of research. As a rule, it can differ from the pilot scenario due to the clarifications that are made to it in the process of implementation. To indicate this state of the scenario for each set an index «R» is added (4):
Thus, we will refine the general record for the software quality assessment scenario based on the state of the scenario and add a «VOS» index for each set, which indicates the state of the scenario (VOS -Variant Of Scenario). Thus, the index «VOS» can have the following values: S -SPAQSW -Scenario on Paper of Assessment of Quality of Software, P -PSAQSW -Pilot Scenario of Assessment of Quality of Software, R -RSAQSW -Real Scenario of Assessment of Quality of Software) (
During its life cycle, the scenario can be refined, i.e. modified. The article does not consider and analyze examples and reasons in which cases the scenario may change, because such material requires more volume of the article and may claim a separate article. It was found that such changes can be reduced to the following two operations on scenario elements: exclusion (deleting) a scenario element (EOE VOS,VOS,TEOS -Exclusion Of Element); inclusion (adding) a scenario element (IOE VOS,VOS,TEOS -Inclusion Of Element). A scenario element conversion operation is also possible, but it is not considered because a pair of exclusion-inclusion operations can represent it. When entering additional sets for each of them, the index «TEOS» was added, which indicates the variant of the scenario (TEOS -Type of Elements Of Scenario). Thus, the index «TEOS» can take the following values:
For a more formal description of such changes in the scenario, we introduce additional notation -S VOS,VOS , which can be of two types: S S,Pa transition state of the scenario on paper to the pilot scenario, S P,Ra transition state of the pilot scenario to the real scenario. Consider possible variants of inequalities of scenarios and their elements for transition states (Fig. 2).
Formally, we present a description of the operations of exclusion (EOE VOS,VOS,TEOS ) and inclusion (IOE VOS,VOS,TEOS ). To do this, enter the following additional sets: a set of initial elements from the corresponding set (SOE TEOS -Set of Original Elements). This set includes all elements of the initial scenario to which the corresponding operation will be applied; a set of elements that are excluded from the corresponding set (SEXE TEOS -Set of Excluding Elements). Because only one element can be deleted from the set of initial scenario elements when using an exclusion operation, such set will consist of one element. Although such elements in the set can accumulate when the exclusion operation for the initial scenario is used repeatedly; a set of excluded elements from the corresponding set (SEE TEOS -Set of Excluded Elements). Because only one element can be deleted from a set of initial scenario elements when using an exclusion operation, such set will consist of one element, although such elements in the set can accumulate when the exclusion operation for the initial scenario is used repeatedly. That is, the element of the scenario when using the exclusion operation transits from the set of excluding elements to the set of excluded elements; a set of resulting elements from the corresponding set (SRE TEOS -Set of Resulting Elements). Such set is formed as the difference between the set of initial elements and the set of excluding scenario elements, or as the sum of the set of initial elements and the set of included elements; a set of included elements from the corresponding set (SIE TEOS -Set of Included Elements). This is a set that consists of an element or elements that will be added to the set of initial elements, and such combination of sets forms the resulting set of scenario elements.
In general, the operation of exclusion (deleting) an element (EOE VOS,VOS,TEOS ) for each state of the scenario is written as follows ( 6
The life cycle of the software quality assessment scenario includes 3 stages, which correspond to its three states (Fig. 2):
1. Stage of forming the «scenario on paper»; 2. Stage of forming the «pilot scenario»; 3. Stage of forming the «real scenario». Such stages of forming a software quality assessment scenario can be performed only sequentially: at the beginning the stage of forming a scenario on paper, then the pilot and real scenarios.
Moving from stage to stage, the software quality assessment scenario can change during its life cycle. Such changes are a consequence of scenario element exclusion and (or) inclusion operations.
For the transition state S S,P there are two variants of inequalities. The first is when the «scenario on paper» is not equal to the pilot scenario, i.e. SPAQSW PSAQSW
. If we consider such inequality at the level of elements, then there are variants for equality and inequality of such elements. Consider the variants for inequalities at the level of the elements of the «scenario on paper» and «pilot scenario» and present them as an exhaustive simple search, excluding the variant of complete equality (Table 1). For example, one such variant of inequalities (Table 1, line 22) will be described in more detail. This variant consists of the following ratios: . Since the inequalities of the scenarios indicate the use of the operation of inclusion or exclusion, we will describe in more detail the following inequalities for both operations:
Variants of inequalities of sets of elements for «scenario on paper» and «pilot scenario»
if an exclusion operation was applied to elements of the set of initial conditions, actions, corrective factors and roles for the scenario (8-11):
The second variant of inequalities, when the «scenario on paper» is identical to the «pilot scenario», i.e. SPAQSW PSAQSW
. Thus, for each set of «scenario on paper» corresponds to the equivalent set of «pilot scenario», i.e. For the transition state S P,R there are the following two variants of inequalities. The first is when the «pilot scenario» is not equal to the «real scenario», i.e. PSAQSW RSAQSW . Such inequality in the set of variants at the level of scenario elements is similar to the inequality of «scenario on paper» and «pilot scenario», given that as the coefficients for the scenario elements the coefficients in parentheses are considered, i.e. instead of the index «S» index «P» is considered, and instead of the index «P» the index «R» is considered (Table 1).
The second variant of inequalities, when the «pilot scenario» is identical to the «real scenario», i.e. PSAQSW RSAQSW
. Thus, each of the sets of elements of one scenario is equal to the corresponding set of another scenario, i.e.
The proposed model can be used to assess the software quality using software fault injection [19]. In particular, in the development and implementation of Fault Injection Testing (FIT) [20], which is used in the Research-and-Production Corporation «RADIY», different fault injection scenarios have been applied to assess the functional safety of FPGA projects and safety related FPGA based information and control systems of the nuclear power plant. Besides, different fault injection techniques and scenario based tools were used during successful certification of FPGA platform RadICS against requirements of Nuclear Regulatory Committee (US NRC).
To perform FIT, various profiles of defects are formed, which are injected in the electronic project, physical module, top-level software in the form of single and multiple defects. This diversity of profiles gives rise to a variety of FIT and quality assessment scenarios.
It should also be noted scenario approach application for software user interfaces usability assessment [21] with use eye-tracking. Elements of such process (including different scenarios) are input data, initial conditions, actions, transition states, corrective factors, results and participants according to them roles.
Extended model of the software quality assessment scenario can be used for applied intelligent systems as to class information systems.
An extended model of software quality assessment scenario is presented and formally described in the article. Its using will formalize planning (initial conditions, input data, corrective factors, actions, transition states, roles and results) and scenario execution. These processes take into account possible features of scenario states, transition of scenario from state to state considering possible changes of sets of scenario elements.
Further research should be focused on the development and automation of realization of detailed scenarios for the quality assessment of software and FPGA projects considering cyber security issue and possibilities of injecting vulnerability (similar design faults/defects). Such approach will provide more complete assessing safety of FPGA platform based information and control systems in conditions of threats and insider intrusions. Another direction for research is a combination of profileoriented and scenario-oriented approaches to a single paradigm and more detail adaptation such the model for applied intelligent systems.