=Paper=
{{Paper
|id=Vol-2318/paper3
|storemode=property
|title=Application of Decision-making Methods for Evaluation of Complex Information System Functioning Quality
|pdfUrl=https://ceur-ws.org/Vol-2318/paper3.pdf
|volume=Vol-2318
|authors=Hryhorii Hnatiienko,Vitaliy Snytyuk,Oleh Suprun
|dblpUrl=https://dblp.org/rec/conf/its2/HnatiienkoSS18
}}
==Application of Decision-making Methods for Evaluation of Complex Information System Functioning Quality==
https://ceur-ws.org/Vol-2318/paper3.pdf
Application of Decision-making Methods for Evaluation
of Complex Information System Functioning Quality
Hryhorii Hnatiienko1, Vitaliy Snytyuk1 and Oleh Suprun1
1
Taras Shevchenko National University of Kyiv, Kyiv, Ukraine
g.gna5@ukr.net, snytyuk@gmail.com, oleh.o.suprun@gmail.com
Abstract. This article presents an approach, which allows to evaluate and,
hence, to improve the functioning of complex intellectual systems. Although
most of the intellectual systems have one goal – to create a trustful and accurate
simulation of real-life program or event, that will allow to make the best tactical
and strategic decisions in long or shirt terms, it’s impossible to design universal
scheme for these systems, since they vary very much, same as real-life practical
tasks. But, to make correct decisions, based on the model and simulation, the
experts must be sure that this system is valid and has high quality. Defining the
system quality level gives an expert the opportunity to adequately perceive the
results of the simulation. To solve this problem, some heuristics are introduced
in the article, to link abstract concept of the model with real-life problems. The
proposed method allows to calculate and to compare risks that appear in every
company, like those, connected to lack of recourses, or inefficient work of its
employees. Besides, according to quality levels, and expert can make decisions
about replacing inefficient links of the system, to transfer responsibilities be-
tween different links, or rank the tasks inside the system according to their im-
portance for the system as a whole. The proposed method can be used with dif-
ferent systems and environments, since it gives an expert the possibility to set
necessary coefficients during the preparation stage. So, the preliminary prepara-
tion requires more time, but it gives much more possibilities to the experts to
verify the system as a whole and prepare its work in different situations.
Keywords: Complex Information Systems, Decision-Making, Quality Evalua-
tion.
1 Introduction
The problem of ensuring the quality of information system functioning is especially
relevant today. This is due to the need of providing reliable and adequate information
right in time to achieve the main tasks of the system functioning in conditions of strict
competition in all spheres of human life. This is especially important in such
branches, as economic, banking and financial systems, for the large corporations and
companies, and even for country future development [1]. All these spheres are unique,
most of the situations are very different from each other in real life, so it is impossible
to use only experience to predict the outcome and make correct decisions. At the same
26
time, a lot of information, that also might be wrong, must be taken into account, and a
single expert, or a team, can’t handle with all of it. Because if this, a reliable and ade-
quate information system, which corresponds to real life situation, must be build,
tested, and adapted to the current requirements.
Even the seemingly small mistake, made at the modeling and designing stage, can
provide great loss in future. This is especially important for big companies that can be
described as complex systems with multiply subsystems, which have different con-
nections between each other. Loosing one of these connections or subsystems may
cause serious damage for the system as a whole, so the risks must be calculated and
the most important nods stated in advance. The intellectual systems simulation is the
best way to make a trustworthy model that will react and change itself same as the
real object. And the quality evaluation is same important as modeling itself.
To explore the systems functioning, theoretically-gaming, probabilistic, graph and
matrix models are traditionally used [2]. To evaluate the quality of the complex in-
formation system functioning, the methods of collective objects arrangement, which
are a wide class of methods for the simulation of practical problems in various subject
areas, will be used [3]. Among the decision-making problems, the task of objects
organizing is highlighted by a large number of specific real-life applications and the
undoubted topic actuality. The problem of searching for the resulting objects ar-
rangement by individual object arrangement is one of the most common problems of
linear objects arrangement.
A complex information system contains hundreds of elements that perform thou-
sands of tasks, and it may have different nature and specification: for example, a map
of organization business processes algorithms of a certain hierarchical system interac-
tion, etc. As the companies scale grows rapidly, more and more information is needed
to simulate it’s work [4]. To calculate and evaluate all this information, different ap-
proaches are used, such as neural systems [5] or evolutional technologies [6]. All this
is made to calculate the possible risks and to avoid them [7], that is impossible with a
intellectual system with low quality.
Besides the great amount of information, that the system has to deal with, all of
this information must be verified, since the information itself is one of the expensive
recourses, and a little mistake (made by chance, or the information can be changed by
the opponents) can cause great consequences [8]. Also, since the system, that must be
tested, is a part of real life world, it is not isolated from other systems. So, it must be
dynamic, to analyze and respond the external impacts, such as changes in prices for
companies, social and political accidents for country development planning, action of
the opponents and other.
Hence, the system, that is build as a model for big real life projects, must have
highly complicated structure, very often to contain smaller subsystems and react to
external impacts. And testing and quality assurance of such system is a task that is
almost as complicated as the system building itself, such as all the systems are unique
and have different requirements.
27
2 Problem Setting
The main problem with testing large real life information models, especially that
represents such brunches as economics, is inability to check it’s work in dynamic
systems, because modeling this dynamic system, that should represent the world or a
certain country, as a whole, is very difficult our days, since there are a lot of different
information, that changes all the time.
Hence, the best and the only way to test such system, is not to evaluate it’s quality
as a whole, but to check the quality of different subsystems. After that, the final
evaluation must be done. But checking each subsystem also can be rather difficult
problem, since these subsystems are mostly unique and automated testing is impossi-
ble. On this step, the expert’s thoughts and conclusions must be considered. A human
being, or a group, can not check all the system, with all information and real time
changing, but they can test different and simple joints of the system.
After gathering results of quality evaluation, made by every expert, these results
must be unified and structured for further usage.
Let some resultant (aggregated, collective) arrangement be given as n prob-
lems R* (ai1 ,..., ain ) , i j I 1,..., n , j I , which is built on logic, that character-
izes the processes of some information system functioning. Arrangement R * is built
on the basis of individual ordering tasks that are performed by k elements of the sys-
tem R i (a1i ,..., ani ), i J 1,..., k , where ni , i J , the number of tasks in indi-
vidual arrangements, that are performed by i th elements of the system. Let the
Ai , i J , be the subset of tasks, performed by j th element of the system.
Since R * reflects the logic of solving a collective problem, tasks in individual
ranking can have indices that do not coincide with the natural series. For example,
tasks a a a belong to the ranking R1 , but tasks a a a a belong to the
3 5 7 1 4 2 6
ranking R 2 . Tasks order indicates the sequence of tasks implementing during the sys-
tem operating.
At the same time, the tasks performed by the elements are not duplicated, ie
n ni – each task in the system is unique and each task in the ranking R * occurs
i J
only once: Ai1 Ai2 , i1 , i2 J .
Each task from the set of tasks A a1 ,...an is characterized by two parameters:
ci0 the nominal price of the execution or the need for the resources, i I ;
ti0 the nominal execution time, i I .
During the performance of i th task by j th element of the system, the follow-
ing is known:
ci j the real price of the task, i I , j J ;
ti j the real time of the task execution, i I , j J .
28
Each element of the system in its regular mode executes its tasks and has limited
ability to perform all subset of its tasks. These limits are:
j j
c C , jJ
j j
i
ai A
, (1)
j
t T , for j, j J .
j j
i (2)
ai A
Restriction (1) is the cost of tasks performing by an element of the system - an em-
ployee's salary analogue in the business processes simulation, and restrictions (2) is
the time limitation - analogue to the monthly norms of the working time duration
during the organizations functioning, although in general, the time constraints may be
different.
During the performance of normative tasks, determined by the nominal tactical and
technical characteristics of the system, the requirements of the system and its ele-
ments in resources (1) - (2) are constant, and the quality of tasks execution by all sub-
systems and the system as a whole is 100 percent.
The nominal tactical and technical characteristics of the system are the resources
requirements:
n
0i 0
c C – system execution budget,
i 1
i
n
0i 0
t T – total time requirement to perform the system functioning.
i 1
i
Since tasks are not duplicated, there is no need for direct redundancy. The redun-
dancy is potential, hidden: the functional moves to another element of the system,
when an element, that should perform the task according to the norm, can not do this.
But this is due to the additional costs of a limited resource.
It is necessary to design a model that will reflect the system's response to various
types of environmental influences and changes in the states of system elements. In
this case, the quality of the system and its elements functioning should be evaluated,
depending on the system elements state.
3 Modeling the Decision-making Situations
In the process of system performance in real conditions, the situation described in the
problem setting, can significantly differ from the normative one. For example, in the
case of a large organization, there are always employees who are currently on sick
leave, on vacation, on business trips, absent for unknown reasons, formally issued
refuses, dismissed from work for various reasons, violate labour discipline etc.
The difference from an external impact is that all these situations happen inside the
model, hence they can be also predicted, and their results are known. If not, they can
be set as some coefficients by the experts, or relying on the previous experience, since
29
they don’t really differ from each other. But the reason must be known to determine
the outcome and overall impact on the system.
All these reasons can be estimated, according to the experts. After that, the follow-
ing steps must be done:
– heuristically determine the current level of performance for each task;
– evaluate the quality of each task at the 100 percent scale, according to the heuris-
tics.
In case of temporary or long-term failure of the system element, all functions that
are performed by this element can not be executed by the system. For their implemen-
tation, it is necessary to make decisions about the functions redistribution or their
replacement. For example, during the temporary absence of the system element, its
tasks can be:
– distributed to perform among other elements of the system;
– passed to perform to one element of the system;
– ignored as such, without which the information system will not significantly lose
its functionality level.
All these 3 situations must be further considered, such as there are different re-
quirements for each of them, and the outcome of the task replacement can be rather
different, according to the maid choice.
3.1 Model 1. Tasks distribution among the elements of the system.
It is clear that the tasks distribution can only be done between the elements that can
perform these tasks, according to their qualifications, the available certificates, etc. In
this case, such heuristics should be taken into account, to ensure the overall system’s
quality:
Heuristic H1. While solving tasks that are not normative for the current system
element, the quality of these tasks performance by current elements that are intended
for the temporary execution is clearly reduced. The level of the tasks performance
quality is set individually for each case and can be, for example, 80%.
Heuristic H2. During the necessity of performing by system element of additional
tasks, the situation of element overload occurs, and therefore the quality is reduced:
a) performance of its own normative tasks, for example, to the level of 90-95%;
b) performance of additional tasks based on Heuristics 1.
Heuristic H3. The price of resource type (1) in case of tasks redistribution due to
the lack of one of the system elements, may increase in the range from 10% to 15% -
to increase the motivation of new elements to perform additional tasks.
After applying heuristics H1-H3 the recount of the resources that are needed to
perform tasks in new circumstances must be made. It is clear that the new values will
differ significantly from the normative ones. At the same time, the quality of the
tasks, and, therefore, and the quality of the system will vary greatly from the ideal
100%. Even the little percentage differ on the lower level may have great impact on
the system as a whole, so all the coefficients must be considered and exactly set by
the experts.
30
Also, when the tasks are distributed to relative elements, the value of these ele-
ments increase, since their absence will have even greater impact, so the risks must be
considered.
3.2 Model 2. Redistribution of the missing element tasks to another for their
execution.
With a significant additional load on the system element, that is assigned to perform
the task of a missing element, the quality of the new tasks implementation, and also
the tasks that it has been normatively performed, is greatly reduced. In this model the
additional heuristic must be used.
Heuristic H4. With an additional load on the system element, the quality of its ad-
ditional tasks implementation significantly decreases, for example, according to a
linear function, the parameters of which can be assigned separately for each situation
of decision-making.
Heuristic H5. The load on the system elements can not exceed some given value,
for example, 2 * T, where T - time constraints established by the formula (2).
During the application of heuristics H4-H5, the definition of new quality levels for
the tasks performance and the quality of the system as a whole is made. In addition,
there are changes in the requirements of resources that are necessary for the system to
perform tasks in the new environment – taking into account the transfer of all tasks of
the missing element to another element.
3.3 Model 3. Ignoring the tasks that are performed by the missing element of
the system.
If it is known that a system element is temporarily absent, and an experienced expert,
that makes decisions, understands that there is no urgent need for the task of this ele-
ment to be performed, a temporary moratorium to perform the relevant tasks may be
made.
Heuristic H6. If the element responsible for performing an autonomous task is ab-
sent, the quality of the task performance drops gradually, during some time. The regu-
larity of the task quality reduction can be set separately for each individual case.
Heuristic H7. If a task for which an executor is not absent is not autonomous, that
is, other tasks depend on its implementation, the function of changing the implemen-
tation quality of the dependent tasks is given separately for each specific decision-
making situation.
Decision to ignore tasks, that are temporarily left without an executor, is very re-
sponsible and requires constant monitoring by the expert or controller appointed by
him. At each monitoring iteration, an assessment of the quality changes must be made
according to the heuristics H6-H7.
This choice may have the greatest impact on the system as a whole. This is espe-
cially important for big companies, where not all links between the elements are
clears and understandable, and the absence of only one element may turn off the per-
formance of the subsystem, that it belongs to, or even some separate subsystem, if the
missing element was responsible for the outcome of his working cell. Therefore, the
31
expert must clearly know, if the element is connected to other systems, and what im-
pact the choice to stop the tasks performing will have.
4 The Results of Information System Functioning Quality
Evaluation
After making decisions about the functions redistribution between system elements or
their replacement, new values of resources for the system tasks and the level of func-
tioning quality are calculated. This information is recorded in the system database.
Based on the obtained values, the affiliation of the system functioning quality lev-
els to the fuzzy set (0,1) is determined. Approaches to the determination of affiliation
functions and algorithms for constructing affiliation functions on the basis of the val-
ues frequency analysis are given in the monograph [3]. That is, the system functioning
quality as a result of the described procedure application will be characterized by the
function of affiliation to the fuzzy set.
It is also possible to design functions for a priori introduced linguistic variables
with such names as, for example, "critically acceptable quality of functioning", "risk
system operation", "sufficient quality", "excellent quality", etc.
5 Knowledge Base for the Functioning Quality Evaluation
The practical significance of the proposed models will greatly increase if the expert
has the tools for evaluating various decision options to ensure that tasks that should be
performed by missing elements of the system are performed. To use the described
models for information system quality evaluation, it is necessary to create a knowl-
edge base with such indicative content:
– interchangeable elements of the system and the degree of their interchange abil-
ity;
– the restriction related to the ability to perform or delegate the tasks performance
associated with hierarchical links in the system;
– tasks decomposition in system elements and potential assignment of tasks for
critical elements;
– functions of changing the system elements working capacity at non-normative
overloads;
– information about the possibility of some tasks duplication by individual ele-
ments of the system;
– the priority of tasks implication by several elements - whenever possible and
necessary;
– the possibility of a temporary moratorium on some tasks;
– formula for calculating the load for system elements;
– the inclusion of tasks in processes, the criticality of the certain tasks perform-
ance, the estimation of the system quality level loss;
32
– an evaluation of the decrease in the functioning quality in the absence of coordi-
nation from the elements that control the hierarchical system;
– taking into account the factors of system quality decreasing: lack of competence
of the element that temporarily performs the task, or overloading the element with
additional tasks.
6 Possibilities of Applying the Different Models Classes to the
Evaluation of the Information System Functioning Quality
Since all of the described heuristics are equally relevant for the most real life models,
at the same time formalizing some aspects of the system functioning, the following
improvements and clarifications may be made, depending on the specific task, that
will improve the system reliability for each current situation.
On the first stage of modeling, system elements may correspond to a non-oriented
graph - only the presence of tasks is indicated, without a detailed description of inputs
and outputs. It is very important while planning the system as a whole, to show all it’s
scale and the number of subsystems without detalization, that will be made on next
steps.
For systems that perform tasks, where the order of execution is essential, like in
most corporations, it is necessary to apply models of strict tasks ranking, described in
this paper. Failure in this may result in great financial and tome losses, since some
subsystems will be idle before others finish their tasks.
If the parallel processes of task execution are modeled, models of non-strict rank-
ing can be applied - for better detailing.
When there are cycles in the interaction between tasks, it is necessary to apply in-
dividual matrices of the tasks sequence - in such cases, the resulting matrix of pair-
wise tasks ranking will be block-diagonal and substantially sparse.
The metric matrix of pairwise tasks ranking is used in cases where it is essential to
specify the terms between the events occurrence or the tasks beginning - for example,
when describing the Gantt chart using matrices.
If these terms of tasks are not clear, then matrices of pairwise tasks ranking with
elements in the form of affiliation functions can be used to simulate such systems.
7 Perspective for Improving the Adequacy of Evaluating the
System Quality
For a more complete consideration of the real systems features, it is necessary to
complicate the described mathematical model. In particular, this can be done by tak-
ing into account the following factors:
– ranking system of the elements, definition of subordination between elements;
– the establishment of hierarchical links between the elements of the system and
the determination of the influence levels of one element to another or the absence of
such influence;
33
– definition of a priori priority of tasks, regardless of their importance in terms of
cost or execution time;
– taking into account the competence coefficients of system elements;
– increasing the level of detail and the level of adequacy of the model by describ-
ing subtasks;
– description of processes that establish the links between tasks and subtasks.
An example of a process consisting of tasks and subtasks that are related to the log-
ic of the system's performance, its organizational and functional structure as a whole,
is the following (Table 1):
Table 1. An example of a process consisting of tasks and subtasks
Process System Elements Tasks Subtasks
1 1 1.1 1.2
2 2 2.1 2.2 2.3
Process Name 1 3 1.3
3 4 3.1 3.2
1 5 1.4 1.5
8 Further Research Directions
Based on the described approach, new tasks can be developed and new approaches to
increase the adequacy of the modeling can be defined:
– a priori assessment of the system reliability;
– evaluation of permissible decrease of the system elements functioning quality
and the level of tasks performance;
– considering the presence or absence of links between tasks: the impact of the task
on the quality of the other tasks functioning;
– solution of optimization problems of forecasting the system functioning quality,
the cost of providing this quality and calculation of allowable time expenditures;
– restoration of the acceptable quality level in case of several elements failure: de-
termination of functioning necessary conditions.
9 Conclusions
In the article the problem setting and different models for system functioning quality
evaluation are proposed. Since the real-life problems may be very different, it’s very
important to establish common problem setting. The variety of proposed models al-
lows an expert to choose one according to the occurred situation without the necessity
to design a new model as a whole.
For the problem solving different methods of decision-making theory, including
the classic ones, or newly-established approaches, such as artificial intelligence, may
be used. This gives the opportunity to choose the best method, according to the prob-
34
lem, and the ability to compare them at the same time. The proposed heuristics dis-
play the whole range of real situations, or at least the main ones.
As the flow of this method, the requirements of long-term preparations can be
stated, such as choosing the priorities, setting the coefficients or determining the main
nodes of the system. At the same time, this allows to configure the system according
to the real-life problem, and, as result, more accurate answer may be obtained.
The perspectives for the future development of such problems are mentioned, same
as possible ways of improvements, using new methods, to enhance models matching
with real objects
References
1. Yan Chen, Lili Qu: The Impact of ICT on the Information Economy. IEEE 2008 4th Inter-
national Conference on Wireless Communications, Networking and Mobile Computing,
12-14 Oct. 2008.
2. A.G. Dodonov, D.V. Lande: Vitality of information systems. - K . Science. Dumka, 2011.
- 256 p.
3. H. M. Gnatyenko, V. Ye. Snytyuk: Expert decision making technologies. - K.: McLaugh,
2008. - 444 c.
4. Tsan-Ming Choi ; Hing Kai Chan ; Xiaohang Yue: Recent Development in Big Data Ana-
lytics for Business Operations and Risk Management, IEEE Transactions on Cybernet-
ics ( Volume: 47 , Issue: 1 , Jan. 2017 ).
5. Miyuan Shan ; Wenxi Zhu: Risk Analysis for Highway Project Based on Multiple-
Dimension Neural Network, 2006 6th World Congress on Intelligent Control and Automa-
tion.
6. V. Y. Snytyuk ; O. O. Suprun: Evolutionary techniques for complex objects clustering,
2017 IEEE 4th International Conference Actual Problems of Unmanned Aerial Vehicles
Developments (APUAVD)
7. Alexander Setiawan ; Adi Wibowo ; Andrew Hartanto Susilo: Risk analysis on the devel-
opment of a business continuity plan, 2017 4th International Conference on Computer Ap-
plications and Information Processing Technology (CAIPT).
8. Zhigang Ji, Qingyu Sun: The analyse of the impact of Information Technology from the
perspective of the Information Economy, IEEE 2009 2nd International Conference on
Power Electronics and Intelligent Transportation System (PEITS), 19-20 Dec. 2009.