=Paper=
{{Paper
|id=Vol-3171/paper116
|storemode=property
|title=Extended Model of Software Quality Assessment Scenario: Concept, Operations, Application
|pdfUrl=https://ceur-ws.org/Vol-3171/paper116.pdf
|volume=Vol-3171
|authors=Oleksandr Gordieiev,Daria Gordieieva,Vyacheslav Kharchenko,Inna Kondius,Ievgen Brezhniev
|dblpUrl=https://dblp.org/rec/conf/colins/GordieievGKKB22
}}
==Extended Model of Software Quality Assessment Scenario: Concept, Operations, Application==
https://ceur-ws.org/Vol-3171/paper116.pdf
Extended Model of Software Quality Assessment Scenario:
Concept, Operations, Application
Oleksandr Gordieiev1, Daria Gordieieva1, Vyacheslav Kharchenko2, Inna Kondius1 and
Ievgen Brezhniev2
1
Lutsk National Technical University, Lvivska Str., 75, Lutsk, 43018, Ukraine
2
National Aerospace University "Kharkiv Aviation Institute", Chkalov Str.,17, Kharkiv, 61070, Ukraine
Abstract
A software quality assessment (SQA) is a mandatory process in ensuring the required quality
of software as part of the overall software development process. The constant development of
existing information technologies and the emergence of new information technologies
(artificial intelligence, cloud computing, virtual and augmented reality, etc.) and systems
increase the requirements for the assessment process and software quality assurance.
Existing approaches to a quality assessment have significant problems: a weak formalization
in the planning SQA tasks; a high degree of uncertainty in decision making by the
responsible participants of the process; insufficient or redundant information; determining the
number of participants in the software assessment process.
Recent publications in open access, which consider the scenario approach in general and in
relation to the tasks of assessing the software quality, were analyzed. More detailed attention
was paid to the description of the scenario approach for SQA tasks. Existing approaches to
formalizing the scenario approach do not take into account all the features of SQA. The
purpose of the article is to develop a model of a software quality assessment scenario.
The article proposes a representation and description of the software quality assessment
scenario model, which consists of the following 7 elements: initial conditions, input data,
actions, transition data, corrective factors, roles, and results. It has been found that a scenario
during its life cycle can be in the following states: scenario on paper, pilot scenario, and real
scenario. During the transition to each state, sets of scenario elements can change. To
formalize such changes, additional operations on the scenario were introduced and formally
described: an operation of exclusion and an operation of inclusion. Variants of inequalities of
scenario elements sets were considered for the scenario on paper and the pilot scenario.
As a result, the extended model of the SQA scenario was developed and presented. It can be
used for software quality assessment based on the software fault injection. And can be
considered as universal model for SQA also for applied intelligent systems.
Keywords
Scenario approach, extended scenario model, software quality, software quality assessment,
software fault injection, applied intelligent systems
1. Introduction and related works analysis
A software quality assessment is a mandatory process in ensuring the required software quality as
part of the overall software development process. The constant development of existing information
technologies and the emergence of new information technologies (artificial intelligence, cloud
computing, virtual and augmented reality, etc.) and systems increase the requirements for the
assessment process and software quality assurance.
_____________________
COLINS-2022: 6th International Conference on Computational Linguistics and Intelligent Systems, May 12–13, 2022, Gliwice, Poland.
EMAIL: oleksandr.gordieiev@gmail.com (O. Gordieiev); gordeyeva.daria@gmail.com (D. Gordieieva); v.kharchenko@csn.khai.edu
(V. Kharchenko); innastk@ukr.net (I. Kondius); e.brezhnev@csn.khai.edu (I. Brezhniev)
ORCID: 0000-0003-2517-9388 (O. Gordieiev); 0000-0003-0474-3138 (D. Gordieieva); 0000-0001-5352-077X (V. Kharchenko); 0000-
0003-3416-7506 (I. Kondius); 0000-0003-2073-9024 (I. Brezhniev)
2022 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
This evolution among the existing approaches and paradigms of software quality assessment has
insufficient dynamics, as there are significant problems, including weak formalization in the planning
software quality assessment tasks, a high degree of uncertainty in decision-making by responsible
participants of the process, lack or excess of necessary initial information, formation of participants
group of software quality assessment process. Especially such problems are clearly expressed in the
methods of software quality assessment based on the software fault injection.
One of the existing approaches that can formalize the software quality assessment process to the
required level is the scenario approach. The analyzed works on the organization and formalization of
the software quality assessment process describe some cases in the development of the software
quality assessment process [1-4], and the scenario approach is described in part, at the level of some
elements [5-7]. There are works on the scenario approach, but it is considered in general as an
approach to management [8-12], without taking into account the specifics of the software quality
assessment in general. The scenario approach is not conceptually considered in the works on software
quality assessment based on the software fault injection [13, 14]. In some papers [15] on software
quality assessment, the scenario model is considered in general, but it does not take into account
important features of this process.
Thus, the aim of the article is to develop an extended model of software quality assessment
scenario, which takes into account its elements, features of state change, etc.
2. The concept of scenario
We will present and formally describe a scenario-oriented approach to software quality
assessment. First of all, consider the concept of the scenario. The word «scenario» comes from the
Latin word «scaena», which translates as «scene». Initially, the scenario was considered as a literary
and dramatic work, written as a basis for the production of film or television, and other events in the
theater and elsewhere. In the twentieth century, Herman Kahn, a leading analyst at the
RAND Corporation [16], adapted this term for use in writing possible stories of future developments.
Kahn is often cited as a father of scenario planning. Oliver Sparrow, one of the founders of the
scenario approach at the Royal Dutch Shell Corporation, distinguished four modern interpretations of
this term [17]:
as «a sensitivity analysis» whether in cash flow management, broader risk assessment, or
project management;
as synonymous with the concept of «contingency plan» in military or civilian contingency
planning, determining who and what to do in the event of an emergency;
as synonymous with a contingency plan in corporate or public policy;
in the sense of «logically agreed assumptions about the future» in decision-making and
strategy formation.
All the main definitions are summarized by the Dutch researcher Philip Van Notten in the
following definition [18]: a scenario is a consistent description of alternative hypothetically possible
variants for future events, which reflects different perspectives on the past, present, and future, and
which can be the basis for action planning.
3. Formalized representation of the model
Adapting the presented definition of the scenario for software quality assessment, we obtain the
following interpretation: a software quality assessment scenario is a product of planning and
describing a (continuous) sequence of actions aimed at software quality assessment, which includes a
description of initial conditions, inputs, expected results (hypotheses), corrective factors and
distribution of participant roles in software quality assessment process. Among the process participant
roles are the following: organizer (research engineer) of the software quality assessment process
(scenario developer), head of the software testing team (quality team leader), tester (quality engineer),
user. Thus, the software quality assessment scenario includes the following 7 elements: actions,
transition data transmitted from stage to stage, roles, input data, corrective factors, initial conditions,
expected result, or hypothesis. Elements of the scenario are presented in general form: graphic (Fig. 1)
and formal form:
INCONSCE inconscek k 1 – a set of initial conditions of the software quality assessment
p
scenario (INCONSCE – INitial CONditions of SCEnario), inconsce – an initial condition of the
software quality assessment scenario;
f
INDASCE indasce j – a set of input data of the software quality assessment scenario
j 1
(INDASCE – INput DAta of SCEnario), indasce – input data of the software quality assessment
scenario;
q
ACTSCE actscei i 1 – a set of actions of the software quality assessment scenario
(ACTSCE – ACTions of SCEnario), actsce – an action of the software quality assessment scenario;
Figure 1: Graphical representation of an extended model of software quality assessment scenario
TRANDAT trandatl l 1 – a set of transition data that is transmitted from action to action,
d
i.e. the output data transit and become the input data for the next action (TRANDAT – TRANsition
DATa), trandat – transition data that is transmitted from action to action;
CORFACT corfact w w1 – a set of corrective factors that clarify the actions in the transition
z
from action to action (CORFACT – CORrective FACTors), corfact – a corrective factor;
ROLSCE rolscen n 1 – a set of roles of the software quality assessment scenario (ROLSCE
t
– ROLes of SCEnario), rolsce – a role of the software quality assessment scenario;
RESSCE resscem m 1 – a set of results of the software quality assessment scenario
v
(RESSCE – RESults of SCEnario), ressce – a result of the software quality assessment scenario.
Thus, the software quality assessment scenario (SAQSW – Scenario of Assessment of Quality of
Software) is described as a set of sets (1):
INCONCE , INDASCE, ACTSCE , TRANDAT ,
SAQSW . (1)
CORFACT , ROLSCE, RESSCE
Experimentally, it was found that the scenario during its life cycle (Fig. 2) evolves and is presented
in the following three states:
Figure 2: Software Quality Assessment Scenario Lifecycle
«scenario on paper» (SPAQSW – Scenario on Paper of Assessment of Quality of Software).
This is the first state of the scenario developed by the organizer of the assessment process. To indicate
this state of the scenario for each set an index «S» is added (2):
INCONCES , INDASCES , ACTSCES , TRANDAT ,
SPAQSW ; (2)
CORFACTS , ROLSCES , RESSCES
«pilot scenario» (PSAQSW – Pilot Scenario of Assessment of Quality of Software). This is a
scenario on paper that runs in test mode. Such a scenario is needed to work out and refine the scenario
on a real test case. As a rule, the number of participants involved in the scenario is minimal.
Typically, this scenario differs from the scenario on paper by refining the scenario elements. To
indicate this state of the scenario for each set an index «P» is added (3):
INCONCEP , INDASCEP , ACTSCEP , TRANDATP ,
PSAQSW ; (3)
CORFACTP , ROLSCEP , RESSCEP
«real scenario» (RSAQSW – Real Scenario of Assessment of Quality of Software). This state
of the scenario is used to assess the software quality for the actual object of research. As a rule, it can
differ from the pilot scenario due to the clarifications that are made to it in the process of
implementation. To indicate this state of the scenario for each set an index «R» is added (4):
INCONCER , INDASCER , ACTSCER , TRANDATR ,
RSAQSW . (4)
CORFACTR , ROLSCER , RESSCER
Thus, we will refine the general record for the software quality assessment scenario based on the
state of the scenario and add a «VOS» index for each set, which indicates the state of the scenario
(VOS - Variant Of Scenario). Thus, the index «VOS» can have the following values: S – SPAQSW –
Scenario on Paper of Assessment of Quality of Software, P – PSAQSW – Pilot Scenario of
Assessment of Quality of Software, R – RSAQSW – Real Scenario of Assessment of Quality of
Software) (5):
INCONCEVOS , INDASCEVOS , ACTSCEVOS , TRANDATVOS ,
SAQSW . (5)
CORFACTVOS , ROLSCEVOS , RESSCEVOS
4. Operation on the scenario
During its life cycle, the scenario can be refined, i.e. modified. The article does not consider and
analyze examples and reasons in which cases the scenario may change, because such material requires
more volume of the article and may claim a separate article. It was found that such changes can be
reduced to the following two operations on scenario elements:
exclusion (deleting) a scenario element (EOEVOS,VOS,TEOS – Exclusion Of Element);
inclusion (adding) a scenario element (IOEVOS,VOS,TEOS – Inclusion Of Element).
A scenario element conversion operation is also possible, but it is not considered because a pair of
exclusion-inclusion operations can represent it. When entering additional sets for each of them, the
index «TEOS» was added, which indicates the variant of the scenario (TEOS – Type of Elements Of
Scenario). Thus, the index «TEOS» can take the following values: INC – INCONSCE – INitial
CONditions of SCEnario, IND – INDASCE – INput DAta of SCEnario, ACT – ACTSCE – ACTions
of SCEnario, TRA – TRANDAT – TRANsition DATa, COR – CORFACT – CORrective FACTors,
ROL – ROLSCE – ROLes of SCEnario, RES – RESSCE – RESults of SCEnario).
For a more formal description of such changes in the scenario, we introduce additional notation –
SVOS,VOS, which can be of two types: SS,P – a transition state of the scenario on paper to the pilot
scenario, SP,R – a transition state of the pilot scenario to the real scenario. Consider possible variants
of inequalities of scenarios and their elements for transition states (Fig. 2).
Formally, we present a description of the operations of exclusion (EOEVOS,VOS,TEOS) and inclusion
(IOEVOS,VOS,TEOS). To do this, enter the following additional sets:
a set of initial elements from the corresponding set (SOETEOS – Set of Original Elements).
This set includes all elements of the initial scenario to which the corresponding operation will be
applied;
a set of elements that are excluded from the corresponding set (SEXETEOS – Set of Excluding
Elements). Because only one element can be deleted from the set of initial scenario elements when
using an exclusion operation, such set will consist of one element. Although such elements in the set
can accumulate when the exclusion operation for the initial scenario is used repeatedly;
a set of excluded elements from the corresponding set (SEETEOS – Set of Excluded Elements).
Because only one element can be deleted from a set of initial scenario elements when using an
exclusion operation, such set will consist of one element, although such elements in the set can
accumulate when the exclusion operation for the initial scenario is used repeatedly. That is, the
element of the scenario when using the exclusion operation transits from the set of excluding elements
to the set of excluded elements;
a set of resulting elements from the corresponding set (SRETEOS – Set of Resulting Elements).
Such set is formed as the difference between the set of initial elements and the set of excluding
scenario elements, or as the sum of the set of initial elements and the set of included elements;
a set of included elements from the corresponding set (SIETEOS – Set of Included Elements).
This is a set that consists of an element or elements that will be added to the set of initial elements,
and such combination of sets forms the resulting set of scenario elements.
In general, the operation of exclusion (deleting) an element (EOEVOS,VOS,TEOS) for each state of the
scenario is written as follows (6):
SRETEOS SOETEOS \ SEXETEOS
EOEVOS ,VOS ,TEOS SEETEOS SOETEOS SEXETEOS ; (6)
SIE
TEOS
and the operation of inclusion (adding) an element (IOEVOS,VOS,TEOS) for each state of the scenario is
written as follows (7):
SRETEOS SOETEOS SIETEOS
IOEVOS ,VOS ,TEOS SOETEOS SIETEOS . (7)
SEE
TEOS SEXETEOS
The life cycle of the software quality assessment scenario includes 3 stages, which correspond to
its three states (Fig. 2):
1. Stage of forming the «scenario on paper»;
2. Stage of forming the «pilot scenario»;
3. Stage of forming the «real scenario».
Such stages of forming a software quality assessment scenario can be performed only
sequentially: at the beginning the stage of forming a scenario on paper, then the pilot and real
scenarios.
Moving from stage to stage, the software quality assessment scenario can change during its life
cycle. Such changes are a consequence of scenario element exclusion and (or) inclusion operations.
For the transition state SS,P there are two variants of inequalities. The first is when the «scenario
on paper» is not equal to the pilot scenario, i.e. SPAQSW PSAQSW . If we consider such inequality
at the level of elements, then there are variants for equality and inequality of such elements. Consider
the variants for inequalities at the level of the elements of the «scenario on paper» and «pilot
scenario» and present them as an exhaustive simple search, excluding the variant of complete equality
(Table 1). For example, one such variant of inequalities (Table 1, line 22) will be described in more
detail. This variant consists of the following ratios: INCONCES INCONCEP
,
INDASCES INDASCEP ACTSCES ACTSCEP CORFACTS CORFACTP ,
, ,
TRANDTS TRANDTP ROLSCES ROLSCEP RESSCES RESSCEP Since among the
, , .
inequalities there are equations that indicate the identity of the elements, we will consider and
describe only the following inequalities: INCONCES INCONCEP INDASCES INDASCEP ,
,
CORFACTS CORFACTP , ROLSCES ROLSCEP . Since the inequalities of the scenarios indicate
the use of the operation of inclusion or exclusion, we will describe in more detail the following
inequalities for both operations:
Table 1
Variants of inequalities of sets of elements for «scenario on paper» and «pilot scenario»
INCONCES ( P ) , INDASCES ( P ) , ACTSCES ( P ) , CORFACTS ( P ) , TRANDATS ( P ) , ROLSCES ( P ) , RESSCES ( P ) ,
№ INCONCEP ( R ) INDASCEP ( R ) ACTSCEP ( R ) CORFACTP ( R ) TRANDATP ( R ) ROLSCEP ( R ) RESSCEP ( R )
1.
2. =
3. =
4. = =
5. =
6. = =
7. = =
8. = = =
9. =
10. = =
11. = =
12. = = =
13. = =
14. = = =
15. = = =
16. = = = =
17. =
18. = =
19. = =
20. = = =
21. = =
22. = = =
…
125. = = = = =
126. = = = = = =
127. = = = = = =
128. = = = = = = =
if an exclusion operation was applied to elements of the set of initial conditions, actions,
corrective factors and roles for the scenario (8-11):
SOEINC INCONCES
SREINC SOEINC \ SEXEINC INCONCEP
EOES , P , INC ; (8)
SEEINC SOEINC SEXEINC
SIEINC
SOEIND INDASCES
SREIND SOEIND \ SEXEIND INDASCEP
EOES , P , IND ; (9)
SEEIND SOEIND SEXEIND
SIEIND
SOECOR CORFACTS
SRECOR SOECOR \ SEXECOR CORFACTP
EOES , P ,COR ; (10)
SEECOR SOECOR SEXECOR
SIECOR
SOEROL ROLSCES
SREROL SOEROL \ SEXEROL ROLSCEP
EOES , P , ROL ; (11)
SEEROL SOEROL SEXEROL
SIEROL
if the inclusion operation was applied to elements of the set of initial conditions, actions,
corrective factors and roles for the scenario (12-15):
SOEINC INCONCES
SREINC SOEINC SEXEINC INCONCEP
IOES , P , INC SOEINC SIEINC ; (12)
SEE SEXE
INC INC
SIEINC
SOEIND INDASCES
SREIND SOEIND SEXEIND INDASCEP
IOES , P , IND SOEIND SIEIND ; (13)
SEE SEXE
IND IND
SIEIND
SOECOR CORFACTS
SRECOR SOECOR SEXECOR CORFACTP
IOES , P ,COR SOECOR SIECOR ;
SEE (14)
COR SEXECOR
SIECOR
SOEROL ROLSCES
SREROL SOEROL SEXEROL ROLSCEP
IOES , P , ROL SOEROL SIEROL .
SEE (15)
ROL SEXEROL
SIEROL
The second variant of inequalities, when the «scenario on paper» is identical to the «pilot
scenario», i.e. SPAQSW PSAQSW . Thus, for each set of «scenario on paper» corresponds to the
equivalent set of «pilot scenario», i.e. INCONSCES INCONSCEP , INDASCES INDASCEP ,
ACTSCES ACTSCEP , TRANDATS TRANDATP , CORFACTS CORFACTP ,
ROLSCES ROLSCEP , RESSCES RESSCEP .
For the transition state SP,R there are the following two variants of inequalities. The first is when
the «pilot scenario» is not equal to the «real scenario», i.e. PSAQSW RSAQSW .
Such inequality in the set of variants at the level of scenario elements is similar to the inequality
of «scenario on paper» and «pilot scenario», given that as the coefficients for the scenario elements
the coefficients in parentheses are considered, i.e. instead of the index «S» index «P» is considered,
and instead of the index «P» the index «R» is considered (Table 1).
The second variant of inequalities, when the «pilot scenario» is identical to the «real scenario»,
i.e. PSAQSW RSAQSW . Thus, each of the sets of elements of one scenario is equal to the
corresponding set of another scenario, i.e. INCONSCEP INCONSCER , INDASCEP INDASCER ,
ACTSCEP ACTSCER , CORFACTP CORFACTR , TRANDATP TRANDATR ,
ROLSCEP ROLSCER , RESSCES RESSCEP .
5. Application of the model
The proposed model can be used to assess the software quality using software fault injection [19].
In particular, in the development and implementation of Fault Injection Testing (FIT) [20], which is
used in the Research-and-Production Corporation «RADIY», different fault injection scenarios have
been applied to assess the functional safety of FPGA projects and safety related FPGA based
information and control systems of the nuclear power plant. Besides, different fault injection
techniques and scenario based tools were used during successful certification of FPGA platform
RadICS against requirements of Nuclear Regulatory Committee (US NRC).
To perform FIT, various profiles of defects are formed, which are injected in the electronic
project, physical module, top-level software in the form of single and multiple defects. This diversity
of profiles gives rise to a variety of FIT and quality assessment scenarios.
It should also be noted scenario approach application for software user interfaces usability
assessment [21] with use eye-tracking. Elements of such process (including different scenarios) are
input data, initial conditions, actions, transition states, corrective factors, results and participants
according to them roles.
Extended model of the software quality assessment scenario can be used for applied intelligent
systems as to class information systems.
6. Conclusions
An extended model of software quality assessment scenario is presented and formally described
in the article. Its using will formalize planning (initial conditions, input data, corrective factors,
actions, transition states, roles and results) and scenario execution. These processes take into account
possible features of scenario states, transition of scenario from state to state considering possible
changes of sets of scenario elements.
Further research should be focused on the development and automation of realization of detailed
scenarios for the quality assessment of software and FPGA projects considering cyber security issue
and possibilities of injecting vulnerability (similar design faults/defects). Such approach will provide
more complete assessing safety of FPGA platform based information and control systems in
conditions of threats and insider intrusions. Another direction for research is a combination of profile-
oriented and scenario-oriented approaches to a single paradigm and more detail adaptation such the
model for applied intelligent systems.
References
[1] Yan, M., Xia, X., Zhang, X. et al. (2019). Software quality assessment model: a systematic
mapping study. Science China Information Sciences Journal, Vol. 62, P.1-18.
https://doi.org/10.1007/s11432-018-9608-3
[2] Miguel J., Mauricio D. and Rodriguez G. (2014). A Review of Software Quality Models for the
Evaluation of Software Products. International Journal of Software Engineering & Applications
(IJSEA), Vol.5(6), P. 31-53.
[3] Brandtner, M. (2013) Fostering software quality assessment. In Proceedings of 35th International
Conference on Software Engineering (ICSE’2013). P.1393-1396.
[4] Issa, T., Isaias, P. (2022). Usability and Human–Computer Interaction (HCI). In: Sustainable
Design. Springer, London. P. 23-40. https://doi.org/10.1007/978-1-4471-7513-1_2
[5] Deissenboeck F, Juergens E, Lochmann K, et al. (2020). Software quality models: purposes,
usage scenarios and requirements. In proceedings of the ICSE Workshop on Software Quality,
P. 9-14.
[6] Briggs, Ch. M., Matejova, M. (2019). Scenario Planning and Complex Scenario Approach.
Disaster Security. P. 38-60.
[7] Koppen, P. J., Mackor, A. R. (2019). A Scenario Approach to the Simonshaven Case. Wiley
Online Library. Topics in Cognitive Science. P. 1-20. URL:
https://onlinelibrary.wiley.com/doi/10.1111/tops.12429.
[8] Campi, M. C., Garatti, S. (2018). Introduction to the Scenario Approach: book. Society for
Industrial and Applied Mathematics and the Mathematical Optimization Society. 116 p.
[9] Ramponi, F. A. (2018). Consistency of the Scenario Approach. Journal on Optimization.
Vol. 28(1). P.135-162.
[10] Garatti, S., Campi, M. C. (2019). Learning for Control: a Bayesian Scenario Approach. In
proceedings of the IEEE 58th Conference on Decision and Control (CDC’2019). P.1772-1777.
[11] Shumkov, Y.A., Vidovskiy, L.A. (2017). Scenario approach to project management.
Polythematic Online Scientific Journal of Kuban State Agrarian University. Vol. 134(10). P. 1-9.
[12] Kononenko I., Sushko H. (2021). Mathematical model of software development project team
composition optimization with fuzzy initial data. Radioelectronic and computer systems journal.
Vol. 3(99). P. 149-159. URL:https://doi.org/10.32620/reks.2021.3.12.
[13] Soumya, S., Baiju, A. (2014). Software Faults Emulation by Software Fault Injection.
International Journal of Computer Applications. Vol. 97. P.9-11.
[14] Cotroneo, D., Madeira, H. (2013). Introduction to Software Fault Injection. Innovative
Technologies for Dependable OTS-Based Critical Systems / ed. D. Cotroneo. Springer. P. 1-15.
[15] Oleksandr Gordieiev, Konstantin Leontiev (2020). Software quality assessment scenario
model. Technical sciences and technologies Journal. Vol. 3(21), P. 209-219. URL:
https://doi.org/10.25140/2411-5363-2020-3(21)-209-219.
[16] Kahn, H. (1967). The Year 2000: A framework for speculation on the next thirty-three year. NY:
Macmillan Publishing Company. 432 p.
[17] Sparrow, O. (2000). Making use of scenarios – from the vague to the concrete. Scenario &
Strategy Planning. Vol. 2(5). P. 18-21.
[18] Van Notten, Ph. (2005). Writing on the wall: scenario development in times of discontinuity.
Florida: Boca Raton. 209 p.
[19] Kooli M., Bosio A., Benoit P., Torres L. (2015). Software testing and software fault injection,
10th International Conference on Design & Technology of Integrated Systems in Nanoscale Era
(DTIS 2015), P. 1-6, doi: 10.1109/DTIS.2015.7127370.
[20] Kharchenko V., Sklyar V., Odarushchenko O. and Ivasuyk A. (2013). Fault-injection testing:
FIT-ability, optimal procedure and tool for FPGA-based systems SIL certification. In
proceedings of the East-West Design & Test Symposium (EWDTS 2013). P. 1-5, doi:
10.1109/EWDTS.2013.6673129.
[21] Purwaningsih R., Yenifi I. (2015). Usability Assessment of International Office Website of
Diponegoro University with Scenario-Based Usability Evaluation Method and Wammi Method.
Internatioanl Journal ComTech: computer, mathematics and engineering applications Vol. 6
(3). P. 329-342.