This article presents an approach, which allows to evaluate and, hence, to improve the functioning of complex intellectual systems. Although most of the intellectual systems have one goal - to create a trustful and accurate simulation of real-life program or event, that will allow to make the best tactical and strategic decisions in long or shirt terms, it's impossible to design universal scheme for these systems, since they vary very much, same as real-life practical tasks. But, to make correct decisions, based on the model and simulation, the experts must be sure that this system is valid and has high quality. Defining the system quality level gives an expert the opportunity to adequately perceive the results of the simulation. To solve this problem, some heuristics are introduced in the article, to link abstract concept of the model with real-life problems. The proposed method allows to calculate and to compare risks that appear in every company, like those, connected to lack of recourses, or inefficient work of its employees. Besides, according to quality levels, and expert can make decisions about replacing inefficient links of the system, to transfer responsibilities between different links, or rank the tasks inside the system according to their importance for the system as a whole. The proposed method can be used with different systems and environments, since it gives an expert the possibility to set necessary coefficients during the preparation stage. So, the preliminary preparation requires more time, but it gives much more possibilities to the experts to verify the system as a whole and prepare its work in different situations.
The problem of ensuring the quality of information system functioning is especially
relevant today. This is due to the need of providing reliable and adequate information
right in time to achieve the main tasks of the system functioning in conditions of strict
competition in all spheres of human life. This is especially important in such
branches, as economic, banking and financial systems, for the large corporations and
companies, and even for country future development [
Even the seemingly small mistake, made at the modeling and designing stage, can provide great loss in future. This is especially important for big companies that can be described as complex systems with multiply subsystems, which have different connections between each other. Loosing one of these connections or subsystems may cause serious damage for the system as a whole, so the risks must be calculated and the most important nods stated in advance. The intellectual systems simulation is the best way to make a trustworthy model that will react and change itself same as the real object. And the quality evaluation is same important as modeling itself.
To explore the systems functioning, theoretically-gaming, probabilistic, graph and
matrix models are traditionally used [
A complex information system contains hundreds of elements that perform
thousands of tasks, and it may have different nature and specification: for example, a map
of organization business processes algorithms of a certain hierarchical system
interaction, etc. As the companies scale grows rapidly, more and more information is needed
to simulate it’s work [
Besides the great amount of information, that the system has to deal with, all of
this information must be verified, since the information itself is one of the expensive
recourses, and a little mistake (made by chance, or the information can be changed by
the opponents) can cause great consequences [
Hence, the system, that is build as a model for big real life projects, must have highly complicated structure, very often to contain smaller subsystems and react to external impacts. And testing and quality assurance of such system is a task that is almost as complicated as the system building itself, such as all the systems are unique and have different requirements.
The main problem with testing large real life information models, especially that represents such brunches as economics, is inability to check it’s work in dynamic systems, because modeling this dynamic system, that should represent the world or a certain country, as a whole, is very difficult our days, since there are a lot of different information, that changes all the time.
Hence, the best and the only way to test such system, is not to evaluate it’s quality as a whole, but to check the quality of different subsystems. After that, the final evaluation must be done. But checking each subsystem also can be rather difficult problem, since these subsystems are mostly unique and automated testing is impossible. On this step, the expert’s thoughts and conclusions must be considered. A human being, or a group, can not check all the system, with all information and real time changing, but they can test different and simple joints of the system.
After gathering results of quality evaluation, made by every expert, these results must be unified and structured for further usage.
Let some resultant (aggregated, collective) arrangement be given as n problems R* (ai1 ,..., ain ) , i j I 1,..., n, j I , which is built on logic, that characterizes the processes of some information system functioning. Arrangement R * is built on the basis of individual ordering tasks that are performed by k elements of the system Ri (a1i ,..., ani ), i J 1,..., k, where ni , i J , the number of tasks in individual arrangements, that are performed by i th elements of the system. Let the Ai , i J , be the subset of tasks, performed by j th element of the system.
Since R * reflects the logic of solving a collective problem, tasks in individual ranking can have indices that do not coincide with the natural series. For example, tasks a3 a5 a7 belong to the ranking R1 , but tasks a1 a4 a2 a6 belong to the ranking R2 . Tasks order indicates the sequence of tasks implementing during the system operating.
At the same time, the tasks performed by the elements are not duplicated, ie n ni – each task in the system is unique and each task in the ranking R * occurs iJ only once: Ai1 Ai2 , i1,i2 J .
Each task from the set of tasks A a1,...an is characterized by two parameters: ci0 the nominal price of the execution or the need for the resources, i I ; ti0 the nominal execution time, i I .
During the performance of i th task by j th element of the system, the following is known: cij the real price of the task, i I , j J ; tij the real time of the task execution, i I , j J .
Each element of the system in its regular mode executes its tasks and has limited ability to perform all subset of its tasks. These limits are: cij C j , aijAj
j J , tij T , for j, j J . aij Aj (1) (2)
Restriction (1) is the cost of tasks performing by an element of the system - an employee's salary analogue in the business processes simulation, and restrictions (2) is the time limitation - analogue to the monthly norms of the working time duration during the organizations functioning, although in general, the time constraints may be different.
During the performance of normative tasks, determined by the nominal tactical and technical characteristics of the system, the requirements of the system and its elements in resources (1) - (2) are constant, and the quality of tasks execution by all subsystems and the system as a whole is 100 percent.
The nominal tactical and technical characteristics of the system are the resources requirements: n ci0i C0 – system execution budget, i1 n ti0i T 0 – total time requirement to perform the system functioning. i1
Since tasks are not duplicated, there is no need for direct redundancy. The redundancy is potential, hidden: the functional moves to another element of the system, when an element, that should perform the task according to the norm, can not do this. But this is due to the additional costs of a limited resource.
It is necessary to design a model that will reflect the system's response to various types of environmental influences and changes in the states of system elements. In this case, the quality of the system and its elements functioning should be evaluated, depending on the system elements state. 3
In the process of system performance in real conditions, the situation described in the problem setting, can significantly differ from the normative one. For example, in the case of a large organization, there are always employees who are currently on sick leave, on vacation, on business trips, absent for unknown reasons, formally issued refuses, dismissed from work for various reasons, violate labour discipline etc.
The difference from an external impact is that all these situations happen inside the model, hence they can be also predicted, and their results are known. If not, they can be set as some coefficients by the experts, or relying on the previous experience, since they don’t really differ from each other. But the reason must be known to determine the outcome and overall impact on the system.
All these reasons can be estimated, according to the experts. After that, the following steps must be done: – heuristically determine the current level of performance for each task; – evaluate the quality of each task at the 100 percent scale, according to the heuristics.
In case of temporary or long-term failure of the system element, all functions that are performed by this element can not be executed by the system. For their implementation, it is necessary to make decisions about the functions redistribution or their replacement. For example, during the temporary absence of the system element, its tasks can be: – distributed to perform among other elements of the system; – passed to perform to one element of the system; – ignored as such, without which the information system will not significantly lose its functionality level.
All these 3 situations must be further considered, such as there are different requirements for each of them, and the outcome of the task replacement can be rather different, according to the maid choice. 3.1
Model 1. Tasks distribution among the elements of the system.
It is clear that the tasks distribution can only be done between the elements that can perform these tasks, according to their qualifications, the available certificates, etc. In this case, such heuristics should be taken into account, to ensure the overall system’s quality:
Heuristic H1. While solving tasks that are not normative for the current system element, the quality of these tasks performance by current elements that are intended for the temporary execution is clearly reduced. The level of the tasks performance quality is set individually for each case and can be, for example, 80%.
Heuristic H2. During the necessity of performing by system element of additional tasks, the situation of element overload occurs, and therefore the quality is reduced: a) performance of its own normative tasks, for example, to the level of 90-95%; b) performance of additional tasks based on Heuristics 1.
Heuristic H3. The price of resource type (1) in case of tasks redistribution due to the lack of one of the system elements, may increase in the range from 10% to 15% to increase the motivation of new elements to perform additional tasks.
After applying heuristics H1-H3 the recount of the resources that are needed to perform tasks in new circumstances must be made. It is clear that the new values will differ significantly from the normative ones. At the same time, the quality of the tasks, and, therefore, and the quality of the system will vary greatly from the ideal 100%. Even the little percentage differ on the lower level may have great impact on the system as a whole, so all the coefficients must be considered and exactly set by the experts.
Also, when the tasks are distributed to relative elements, the value of these elements increase, since their absence will have even greater impact, so the risks must be considered. 3.2
Model 2. Redistribution of the missing element tasks to another for their execution.
With a significant additional load on the system element, that is assigned to perform the task of a missing element, the quality of the new tasks implementation, and also the tasks that it has been normatively performed, is greatly reduced. In this model the additional heuristic must be used.
Heuristic H4. With an additional load on the system element, the quality of its additional tasks implementation significantly decreases, for example, according to a linear function, the parameters of which can be assigned separately for each situation of decision-making.
Heuristic H5. The load on the system elements can not exceed some given value, for example, 2 * T, where T - time constraints established by the formula (2).
During the application of heuristics H4-H5, the definition of new quality levels for the tasks performance and the quality of the system as a whole is made. In addition, there are changes in the requirements of resources that are necessary for the system to perform tasks in the new environment – taking into account the transfer of all tasks of the missing element to another element. 3.3
Model 3. Ignoring the tasks that are performed by the missing element of the system.
If it is known that a system element is temporarily absent, and an experienced expert, that makes decisions, understands that there is no urgent need for the task of this element to be performed, a temporary moratorium to perform the relevant tasks may be made.
Heuristic H6. If the element responsible for performing an autonomous task is absent, the quality of the task performance drops gradually, during some time. The regularity of the task quality reduction can be set separately for each individual case.
Heuristic H7. If a task for which an executor is not absent is not autonomous, that is, other tasks depend on its implementation, the function of changing the implementation quality of the dependent tasks is given separately for each specific decisionmaking situation.
Decision to ignore tasks, that are temporarily left without an executor, is very responsible and requires constant monitoring by the expert or controller appointed by him. At each monitoring iteration, an assessment of the quality changes must be made according to the heuristics H6-H7.
This choice may have the greatest impact on the system as a whole. This is especially important for big companies, where not all links between the elements are clears and understandable, and the absence of only one element may turn off the performance of the subsystem, that it belongs to, or even some separate subsystem, if the missing element was responsible for the outcome of his working cell. Therefore, the expert must clearly know, if the element is connected to other systems, and what impact the choice to stop the tasks performing will have. 4
The Results of Information System Functioning Quality Evaluation After making decisions about the functions redistribution between system elements or their replacement, new values of resources for the system tasks and the level of functioning quality are calculated. This information is recorded in the system database.
Based on the obtained values, the affiliation of the system functioning quality
levels to the fuzzy set (0,1) is determined. Approaches to the determination of affiliation
functions and algorithms for constructing affiliation functions on the basis of the
values frequency analysis are given in the monograph [
It is also possible to design functions for a priori introduced linguistic variables with such names as, for example, "critically acceptable quality of functioning", "risk system operation", "sufficient quality", "excellent quality", etc. 5
Knowledge Base for the Functioning Quality Evaluation
– an evaluation of the decrease in the functioning quality in the absence of coordination from the elements that control the hierarchical system;
– taking into account the factors of system quality decreasing: lack of competence of the element that temporarily performs the task, or overloading the element with additional tasks. 6
Possibilities of Applying the Different Models Classes to the Evaluation of the Information System Functioning Quality Since all of the described heuristics are equally relevant for the most real life models, at the same time formalizing some aspects of the system functioning, the following improvements and clarifications may be made, depending on the specific task, that will improve the system reliability for each current situation.
On the first stage of modeling, system elements may correspond to a non-oriented graph - only the presence of tasks is indicated, without a detailed description of inputs and outputs. It is very important while planning the system as a whole, to show all it’s scale and the number of subsystems without detalization, that will be made on next steps.
For systems that perform tasks, where the order of execution is essential, like in most corporations, it is necessary to apply models of strict tasks ranking, described in this paper. Failure in this may result in great financial and tome losses, since some subsystems will be idle before others finish their tasks.
If the parallel processes of task execution are modeled, models of non-strict ranking can be applied - for better detailing.
When there are cycles in the interaction between tasks, it is necessary to apply individual matrices of the tasks sequence - in such cases, the resulting matrix of pairwise tasks ranking will be block-diagonal and substantially sparse.
The metric matrix of pairwise tasks ranking is used in cases where it is essential to specify the terms between the events occurrence or the tasks beginning - for example, when describing the Gantt chart using matrices.
If these terms of tasks are not clear, then matrices of pairwise tasks ranking with elements in the form of affiliation functions can be used to simulate such systems. 7
Perspective for Improving the Adequacy of Evaluating the System Quality For a more complete consideration of the real systems features, it is necessary to complicate the described mathematical model. In particular, this can be done by taking into account the following factors: – ranking system of the elements, definition of subordination between elements; – the establishment of hierarchical links between the elements of the system and the determination of the influence levels of one element to another or the absence of such influence; – definition of a priori priority of tasks, regardless of their importance in terms of cost or execution time; – taking into account the competence coefficients of system elements; – increasing the level of detail and the level of adequacy of the model by describing subtasks; – description of processes that establish the links between tasks and subtasks.
An example of a process consisting of tasks and subtasks that are related to the logic of the system's performance, its organizational and functional structure as a whole, is the following (Table 1): In the article the problem setting and different models for system functioning quality evaluation are proposed. Since the real-life problems may be very different, it’s very important to establish common problem setting. The variety of proposed models allows an expert to choose one according to the occurred situation without the necessity to design a new model as a whole.
For the problem solving different methods of decision-making theory, including the classic ones, or newly-established approaches, such as artificial intelligence, may be used. This gives the opportunity to choose the best method, according to the problem, and the ability to compare them at the same time. The proposed heuristics display the whole range of real situations, or at least the main ones.
As the flow of this method, the requirements of long-term preparations can be stated, such as choosing the priorities, setting the coefficients or determining the main nodes of the system. At the same time, this allows to configure the system according to the real-life problem, and, as result, more accurate answer may be obtained.
The perspectives for the future development of such problems are mentioned, same as possible ways of improvements, using new methods, to enhance models matching with real objects