The paper is devoted to identifying bottlenecks in information and communication systems. The problems of analyzing the reliability and fault tolerance of distributed data storage system elements are considered. A system of 10 nodes is considered as an example. Input data is based on monitoring network traffic between nodes for one month. The application of the theory of Markov processes and the estimation of the probabilistic characteristics of system functioning in the stationary mode allowed to determine the most loaded nodes of the system and the direction of its modernization. The algorithm for identifying a bottleneck in a distributed data storage system was carried out on the basis of the developed data transformation mechanism and was used as the basis for creating a specialized application software in C# programming language, which allows solving a whole class of problems in various aspects of information and infocommunication technologies. The algorithm and software are ready to solve tasks of much greater dimensionality.
Modern information technology (IT) infrastructure is a complex system that consists of a number of interrelated components, such as servers, clients, networks, etc. These components are subject to two trends: an increase in the number of elements and the complication of the interrelations between them. System’s functionality determines the complexity of solutions and components. The higher the possibilities, the more complex the system. For example, the IT infrastructure of large corporate-level enterprises is, as a rule, distributed systems with complex interrelations between their elements. The large number and high complexity of the components give rise to problems of monitoring and forecasting the consequences of contingencies. These facts justify the need to work on the system in the direction of increasing its reliability. Copyright © 2019 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The reliability of the IT solution is determined by its fault tolerance. Failure is understood as any violation in the system's operability, which leads to downtime in services’ operations. Therefore, fault tolerance is the ability of the system to continue its normal operation even after the failure of one or more components. A number of activities, including regular and extra-regular audit of the system, accompanies providing fault tolerance in complex systems. The analysis allows determining the direction of system’s further development. However, one of the most important results of the fault tolerance audit of the system is the identification of “bottlenecks”.
To date, many organizations offer IT infrastructure audit services. Typically, the
audit process consists of many stages and is based on empirical approaches and
methodologies [
Thus, the issue of developing a universal methodology for identifying critical vulnerabilities as part of the fault tolerance audit of elements of distributed IT infrastructure subsystems remains urgent.
Building a reliable storage facility that can not only resist hardware failures, but also provide high speed data access for many clients simultaneously working with it, is a serious technical problem that does not have the only right decision. Every decision is a compromise in terms of many factors. Successful achievement of such a compromise is possible only on the basis of theoretical research, simulation modeling and experience in practical implementation of the data storage systems (DSS). The task is further complicated by the fact that there are almost no mathematical models that adequately describe DSS, built on the basis of the principles described above.
Methods of building fault-tolerant DSS based on multiple disk drives originated about 3 decades ago. These were the so-called Redundant Array of Independent Disks (RAID) — a series of disks combined into a single storage. Improving the reliability of such storage was achieved by storing redundant information (checksums) on one or more additional disks.
The first industrial system built on this principle was the Google cluster file system, intended for internal use by specialized Google applications. Similar ideas formed the basis for Azure DSS developed by Microsoft, as well as the open source CEPH system.
In a system consisting of hundreds of servers and thousands of disks, hardware and disk failures in particular are quite ordinary events. Accordingly, the software of such a system should provide automatic recovery after a hardware failure to minimize the chance of data loss. In this regard, the construction of an adequate mathematical model of reliability is a decisive factor for creating a fault-tolerant and scalable data storage system. 2
Thus, a distributed system is a stand-alone computers that work together and combined into a single system. Their benefits include: 1) simplifying the integration of various applications into a single system; 2) scalability.
These benefits lead to more complex software modules, performance degradation and possible security problems.
In a distributed system, one computer is a server, all other machines are remote workstations. The basis of network interaction of distributed systems is OSI/ISO (open systems interconnection model). This model divides the process of interaction between the client and server into seven levels: physical, data link, network, transport, session, presentation and application.
The algorithm of interactions in open systems is described by standard protocols. The main protocol stack is TCP/IP. The transport layer protocol is TCP, and the network layer protocol is IP. The operating system is a link to the transport layer protocol and provides an interface for the upper layers based on sockets. Sockets provide low-level elementary operations for the direct exchange of a bit stream between two processes. There is no session or standard presentation layer in the TCP/IP protocol stack. These sometimes include secure SSL/TLS protocols.
Protocols can be divided into 2 main types: connection oriented and connectionless. When a data connection is established, the sender and recipient establish a connection, and after the transfer is completed, the connection is released. Without connection, the sender immediately sends the message to the addressee.
Socket based TCP/IP is a standard, cross-platform, but low-level service for exchanging data between components.
TCP works at the transport level, it is responsible for sending and receiving packets without data loss and in the correct order. The TCP transport protocol is called the Transmission Control Protocol and belongs to the basic stack of network protocols, since it implements all the services necessary for building network applications.
The TCP/IP protocol stack is a standard for network communications. Internet protocol stacks contain UDP (Universal Datagram Protocol, which does not require a connection), it is a universal datagram protocol and is a modification of IP.
Any interaction between remote components of a distributed system is based on TCP/IP.
A characteristic feature of distributed systems is the possibility of a partial failure, a
failure in one of the components of a distributed system. When creating a distributed
system, it is important that it can be automatically restored after partial failures [
Modern information systems make high demands on software and hardware, and,
in particular, on reliability and fault tolerance [
There are several possible reasons for failure in information systems. Statistics are shown in Fig. 1.
Statistics of the failure causes in information processing
systems 10% 11% 22% 6% 27% 24%
Personnel errors
Solving the scientific task of identifying vulnerabilities in distributed systems with a
further failure risk reduction is a relevant problem, which attracts the attention of
researchers. Various mathematical approaches are used, including analysis and
calculation of probabilistic characteristics, methods for determining the critical route,
methods for solving dynamic programming problems, model-based testing methods,
and algorithmic methods as well as their software implementations [
Owing to its simplicity, clarity and logical consistency, the mathematical apparatus
of the theory of Markov processes found its application in the study of random
processes in medicine and genetics [
The theory of Markov chains takes a worthy place in the course of studying random
processes, usually depending on the parameter, which in most applications is time.
The foundations of the theory were postulated by A. Markov at the beginning of the
20th century, and are valid for a random process whose evolution after any given
value of the time parameter t does not depend on the evolution preceding t, if the
value of the process at that moment is fixed. The apparatus of the theory of Markov
chain processes is intended for a complete description of both the long-term and local
dynamics of random sequences of events, including ones in steady state or limiting
mode [
According to the ergodic theorem, under certain conditions the average over the
ensemble coincides with the mean in time, that is, the same information can be
obtained from long-term observation of the system and from simultaneous (and
instant) observation of many independent copies of the same system [
In the application of the Markov chains to the study of the stability of distributed systems (for example, data storage systems) in order to identify their weaknesses, one of the important factors is a fairly long process, i.e. the course of the process after the termination of the initial conditions affecting it. Under certain conditions, in the end, the final stationary regime of the process is established, in which the probabilities of the states of the system p1 pn no longer depend on either time or the initial probability distribution. The vector ( p1 pn ) whose coordinates are the final probabilities is the final (limiting, stationary) vector and characterizes the average time of finding the system in each of its states.
In the studies [
pn ) can be found from the equation (1) subject to the normalization condition p1 p2
pn 1. ( p1 pn ) ( p1 pn ) * P; (1) p11 P p21 pr1 p12 p22 pr 2 p1r
p2r , prr where Р is the matrix of transition probabilities.
Sufficient conditions for the existence of limit probabilities are formulated in the theorem that if a homogeneous Markov chain with a finite number (n) of states is regular, then there are final probabilities p1 pn , and A regularity condition for a Markov chain is the existence of a natural number m such that any element of the matrix Pm is positive except for elements in columns whose numbers coincide with the unstable states of the system S.
Thus, to find the final probabilities, it is necessary to check the Markov chain for regularity, and if it turns out to be so, then it is possible find the final stationary probability vector from equation (1).
The mathematical apparatus of the Markov chain theory is a good variant of a combination of simplicity and quality for solving such problems. It makes it possible to identify critical nodes in terms of fault tolerance based on the analysis of statistics. The simplicity of the algorithmic implementation of the proposed approach should be noted, which is an important factor in automating the process of identifying critical vulnerabilities in distributed systems. 5
As an example of the application of the theory of stationary processes in the basis of
the methodology for identifying critical vulnerabilities in distributed systems, network
traffic was collected between 10 stationary nodes in a distributed data warehouse
system [
Quantitative characteristics of the amount of information transmitted from node to node were recorded during the month. The obtained data were partially processed and tested for relevance. The averaged statistical data were summarized in a table characterizing the amount of information aij transmitted from node i to node j (Table 1). For the demonstration example, several generalizations are accepted: the size of the files is taken into account, and not their number; data warehouses physical implementation is not taken into account; network infrastructure and remoteness of nodes are not considered.
A B C D E F G H The normalized data in the form of probabilities of transmission of outgoing traffic between nodes in the system are given in Table 2. be arbitrarily small and therefore the time instants t1, t2 tn can be chosen so close to each other that the system does not change its state between them. Consequently, the process taking place in the system can be considered a discrete-time process, and Table 2 in terms of Markov chains can be interpreted as a matrix of transition probabilities.
An analysis of the data in Table 1 and the corresponding labeled graph allows to conclude that the system does not have unstable states [23]. Therefore, in order for the Markov chain to be regular and the system to have final probabilities it is necessary and sufficient that there be a natural number m such that all the elements of Pm are positive. This condition is met when m = 2, which allows to conclude that the Markov chain is regular.
When m = 2 (Table 3), all elements of the matrix are strictly greater than zero, pij 0 , i.e. the necessary and sufficient condition of regularity is satisfied, which allows us to conclude that the Markov chain under consideration is regular and the final probabilities exist and can be determined. In order to identify the potential vulnerabilities of fault tolerance of distributed system nodes, we use (1) to form the equations system (2). The result (2) is a system of linear algebraic equations. The solution of the system can be obtained by one of the known methods: Gaussian elimination, Cramer’s rule, etc.
The solution of system (2) is the vector of final probabilities (3). (2) (3) p1 0,081 p2 0,089 p 0,133 3 p4 0,087 p 0,057 5 p6 0,11 p7 0,101 p 0,093 8 p9 0,091 p10 0,158 The probability values determine the average time of receiving information by the corresponding node [24]. With the same technical characteristics of the nodes, the most loaded elements of the system would be nodes J and C. However, it should be noted that the technical characteristics of the nodes are generally not the same. Therefore, in order to correctly compare the risk of their failure, it is advisable to take into account both the final probabilities of the nodes, which characterize the average time they are in the active state, and the total or average load values on them. The product of these two indicators (Table 3) allows us to estimate the risk of failure or unstable operation of the element, and the correlation of this value with the maximum permissible level of load (technical characteristic of the nodes) suggests a safety margin of each of the nodes. Thus, the nodes J, C and F of the system are in the state of obtaining information on average 15.8%, 13.3% and 11% of the time, respectively, the maximum amounts of received information fall on J (29.6092), F (22.451) and A (17.7714), and node C (10.4272) is not so critically loaded.
As a rule, distributed systems have a significant number of nodes and the calculation of probability indicators characterizing the vulnerability of the system is not an easy task, requiring cumbersome calculations. To eliminate this difficulty, a design was carried out and a software application was implemented for solving the problems of identifying bottlenecks in information and communication systems.
In the course of designing a software application for identifying the most loaded node of a distributed system, the task of developing a single-user application that implements a search algorithm based on an inter-node data transfer matrix was considered and implemented. The following requirements were imposed on the developed application: it should be possible to enter the input data both manually and from an external file; there should be a feature of creating a load chart for the system nodes; there should be a feature of saving the chart as an image.
A functional model of the application was built by the analysis of the subject area. 10 use cases are shown in the UML use case diagram (Fig. 2).
The program user can directly create and open input data files. After opening the file, it is possible to edit it. Two save options are available while editing: save to the current file; save to another file. After making the necessary changes, the user can determine the load on the nodes of the distributed system and view two charts: chart of the load on the system nodes; final probabilities chart.
The chart can be saved to a file. When designing a software system, a sequence of user actions was determined to estimate the load on the system nodes based on the input data file. When opening or creating an input data file, the user gets access to the form of editing the data transfer matrix with the corresponding menu, which allows saving changes to the current or new file. After the validation of the entered data, the application proceeds to the stage of finding a solution to the problem with an indication of the results on the tab with the final calculation table. The user interface of the application allows switching between the following tabs: result table; input data; chart of load on the system nodes; final probabilities chart.
The chart can be saved to a file at runtime. The file can be closed upon work completion and choosing the option of saving information.
As a result of subject area object-oriented analysis, a UML class diagram was made, which contains 6 classes (Fig. 3). The software system is written in the C# programming language using IDE Visual Studio 2017 and .NET Framework 4.5. The developed system supports three input data file formats: comma-separated CSV files; semicolon-separated CSV files; tab-delimited text files.
When saving a file, the decimal separator set in the regional standards of the operating system for the current user is used (in comma-separated CSV files the decimal separator is always a dot).
After opening the input data file, the “Input data” tab will be shown, on which you can edit the data transfer matrix (Fig. 4). Changes can be saved. The number of nodes can be changed in the “Node count” field (the value must be in the range from 2 to 100). Click the “Calculate” button to solve the problem. The “Reset” button is used to reset the changes. After clicking on the “Calculate” button, the load on the nodes of the distributed system will be calculated. The “Analysis results” tab displays the results of solving the problem. The row in the table highlighted in red contains a record of the most loaded node in the system (Fig. 5). The “Load on system nodes” tab displays the load chart for the nodes of the distributed system (Fig. 6). The load is defined as the product of the final probabilities of receiving traffic by system nodes by the average amount of data transferred. The “Final probabilities” tab displays the final probabilities chart (Fig. 7). The final probability represents the probability of failure of a distributed system node. The “Log” tab displays the history of solving the system of linear equations using the method of Gaussian elimination
Software performance testing was carried out on matrices of different sizes. The graph of the average system operation time versus the number of nodes is shown in Fig. 8. The application of the apparatus of the theory of stationary processes makes it possible to determine critical vulnerabilities in distributed systems based on statistical data. The methodology provides a clear and mathematically substantiated answer to the question posed by the definition of a potentially vulnerable component of the system. The application of the methodology is possible to solve a whole class of problems in various aspects of information and infocommunication technologies.
It is planned to adapt the vulnerability search algorithms for large-dimension matrices, to apply numerical solution methods and multithreading in further software development. 23. Kavun, S., Mykhalchuk, I., Kalashnykova, N., Zyma, O.: A Method of Internet-Analysis by the Tools of Graph Theory. Intelligent Decision Technologies. 15, 35-44 (2012). 24. Mikheev, I., Shapovalova, O., Burmensky, R.: Methodology for Identifying Critical Vulnerabilities in Distributed Systems. 2018 5th International Scientific-Practical Conference Problems of Infocommunications. Science and Technology (PIC S&T). 654-658 (2018).