Inter-domain routing method under normalized Quality of Service based on hierarchical coordination

Abstract

The inter-domain routing method under normalized Quality of Service (QoS) based on hierarchical coordination in a Software-Defined Network (SD-WAN, Hybrid SD-WAN) is proposed in the paper. The novelty of the method is that the routing solutions are aimed at ensuring the normalized QoS in terms of average transmission rate and average end-to-end packet delay. The method is based on the use of a decomposition flow-based routing model, which consists of the inter-domain routing interaction and ensuring the normalized QoS conditions that obtained during tensor network modeling. Therefore, the inter-domain QoS routing problem was presented as an optimization problem with a quadratic optimality criterion. A number of numerical examples confirmed the efficiency and effectiveness of the proposed method in terms of providing the normalized QoS within the finite number of iterations. Reducing the number of such iterations helps to decrease the amount of service traffic transferred between routers and SDN controllers at different levels, as well as minimize the time for solving the inter-domain QoS-routing task.

Keywords

• Method
• Inter-domain routing
• Hierarchical coordination
• Quality of Service
• End-to-end delay
• SDN

-

Providing a specified level of Quality of Service for user requests has been and remains the primary purpose of the functioning of modern infocommunication networks (ICNs), in which various technological means and protocols of distribution and reservation of network resources, traffic management, etc. are involved. At the same time, the problem of QoS provision is exacerbated in the conditions of considerable territorial distribution and heterogeneity of ICNs, which significantly influence the scalability of traffic management solutions.

An effective way to increase scalability is to use multi-domain Software-Defined Networks (SDN) when a significant list of traffic management functions is translated to many network operating system controllers, which in turn are built into a clear functional hierarchy [1-6].

Assigning a separate controller for each network domain contributes to a significant reduction in the amount of service traffic circulating on the network, the size of the routing tables, and improving the timeliness of traffic management tasks solutions, among which routing is of an important consideration. On the other hand, full implementation of these advantages requires a radical modernization of models, methods, and protocols of routing with their functional adaptation to the features of the hierarchical architecture of SDN-controllers and the multidomain structure of ICNs. 2

Overview of existing solutions regarding hierarchical routing Currently the most well-known protocols that implement the principles of hierarchical multi-domain routing are considered to be OSPF and Integrated IS-IS used in IP networks, as well as the somewhat outdated PNNI in the ATM network [4-6]. However, these protocols still use Dikstra's combinatorial algorithm to calculate routes, which does not take into account the hierarchical features of network construction. It should also be added that QoS requirements in the same protocols are organized indirectly only through routing metrics, which are usually related to the bandwidth of ICN links and paths. Implementing such an approach does help improve the overall QoS level on the network, but does not guarantee the numerical values of the end-to-end QoS indicators, such as the average packet delay for a particular flow.

A number of solutions have been proposed in scientific papers on hierarchical routing [7-14], which are mainly represented by flow-based models and optimization methods of calculation. In articles [7, 8, 12-14], such solutions are adapted to the features of SDN architectures. An important feature of the results obtained in [9-11] is that they are based on the use of the provisions, principles, and postulates of the theory of hierarchical multilevel systems [15, 16]. First, these solutions impose several hierarchical levels of decision-making on routing in ICNs. Second, they imply a decomposition representation of a network mathematical model, which can be described, for example, by a system of differential equations [17]. Third, a mandatory component of hierarchical routing is the coordination process, which is implemented by the top level in relation to the routing decisions of the lower levels.

From the point of view of QoS routing implementation the approach [18-24] is particularly noteworthy since is based on the use of tensor research methodology, within which it was possible to obtain in an analytical form the conditions for ensuring the Quality of Service on a number of indicators – bandwidth, average delay, probability of packet loss. Therefore, in this paper, we will propose the solution to the problem of hierarchical coordination of inter-domain routing in the Software-Defined infocommunication network with the provision of normalized QoS, that is when the requirements for the level of end-to-end QoS are set for each domain in the form of corresponding norms. The presented solution is a further development and integration of the results obtained in [9-11, 23, 24].

Decomposition model of inter-domain QoS routing in

Software-Defined infocommunication network Suppose that the network structure is represented as an oriented graph G  (R, E) , where R is the set of vertices that model the routers and E is the set of graph edges that describes the communication links of the network. In the general case, a packet flow is generated when providing a particular information service on the network. We denote the number of flows circulating in the network by K , then K  K is the power of the set, which quantitatively characterizes the total number of flows in the k ICN. For each k th flow ( k  K ), its average packet rate (intensity) req is known, which is measured in packets per second (1/s) and defines the requirements for the ICN bandwidth allocated to that flow.

When developing a decomposition model of inter-domain routing, suppose that the network consists of N interconnected subnets – domains. Then let each p th individual domain in the ICN be described by the graph G subgraph G p  (R p , E p ) , where R p  Rip ; i  1, m p  is the set of routers of the p th domain, E p  Eip,j ; i, j  1, m p , i  j is the set of links connecting the routers of the p th domain, mp and n p are the total numbers of the routers and communication links in the p th domain respectively.

During network decomposition, the boundary between domains passed through network routers, as implemented, for example, in OSPF [10]:

R p  Rq  0 and E p  Eq  0 , p  q , ( 1 ) that is, some network routers may belong to several adjacent domains at a time. We also define for each p th domain a set of border routers B p ( B p  R p ). In turn, the entire set of the p th domain border routers can be divided into two subsets: Bipn,k is the subset of the border routers through which the packets of the k th flow income into the p th domain; Bopu,tk is the subset of the border routers through which the packets of the k th flow outgo from the p th domain. For each communication link Eip,j , we denote its bandwidth ip, j , which is measured in packets per second (1/s).

As a result of solving the hierarchical inter-domain routing task for each p th domain, it is necessary to calculate the routing variables xip,,jk that characterize the fraction of the k th packet flow transmitted in the link Eip,j  E p . Then, for each p th domain router, the k th flow conservation conditions must be met to ensure the connectivity of intradomain sections of the inter-domain routes.

If the p th domain is transit for the k th packet flow, then such conditions take the form:                      p,k B p  Eip,jE p Bin Eip,jE p

p,k  xi, j

 E jp,iE

p Bopu,tk B p  Eip,jE p

p,k  xi, j

 E jp,iE

p p,k  0; x j,i p,k  xi, j

 E jp,iE p

p,k  x j,i   1;

p,k  x j,i   1.

      kK  λrkeq xip,,jk  ip, j , p  1, N . p,k   λreq xi, j

k p,k Let us denote as λi, j kK

as the average intensity of the k th packet p flow transmitted in the link Ei, j  E

p . Then, when implementing multipath routing, the routing variables are imposed by the constraints:

The system of equations ( 2 ) must be met separately for each k th packet flow. The first condition of the system ( 2 ) covers all the border routers through which the k th flow arrives at the p th domain. The second condition in ( 2 ) is introduced for internal p th domain routers that are transit for the k th packet flow. The third condition must be satisfied for all border routers through which the k th flow outgoes from the p th domain.

If the k th packet flow arrives at a network through the p th domain, and its source is the router Rip , for example, then for this network the first condition of the system ( 2 ) will be somewhat simplified and will look like:  Eip,jE p

p,k  1 . The rest of the xi, j equations in the system ( 2 ) will remain unchanged. When the router Rip from the p th domain is the k th packet flow receiver, only the last equation of system ( 2 ) will be simplified and will look as follows:

 E jp,iE p

p,k  1 . x j,i

In addition, to prevent links congestion on the p th network domain, it is important to fulfill the following conditions: ( 2 ) ( 3 )

Variables ( 4 ) are the coordinates of the routing vectors xkp that set the result of the solution for the k th flow routing problem in the p th domain. During distributed calculation of vectors xkp across each p th domain, it is important to ensure the structural and functional connectivity of inter-domain routes that are traverse multiple routers of different domains. In order to ensure inter-domain connectivity, the model ( 1 )-( 4 ) introduces the following inter-domain interaction conditions [10]: С kp,q xkp  Сq, p xq , p, q  1, N , p  q , k  K ,

k k k where Сp,q is the interaction matrix of the p th and q th domains of the size m p,q  mxp,k ; m p,q is the number of routers through which the border between the p th and q th domains passes; mxp,k is the number of coordinates xip,,jk of the xkp .

Thus for the network structure shown in Fig. 1 that consists of two domains, the boundary passes through two routers. In the designations of the first domain (Fig. 1) the border routers are R31 and R41 , while in the designations of the second the border routers are R12 and R22 . In the gaps of communication links, their bandwidth is shown (1/s). ( 4 ) ( 5 )

Fig. 1. An example of the ICN structure to be investigated.

Terms of providing normalized Quality of Service over domains in the infocommunication network

For each network flow, its transmission rate, average end-to-end delay, and packet loss probability, certain limits are set with respect to their boundary (minimum and/or maximum) values, which determine the level of customer QoS. In case of support of multidomain ICN architecture, the provision of set values of the end-to-end QoS indicators is often realized on the basis of their previous normalization. Let us introduce the following notation:

k  req is the requirement for the boundary value of the average end-to-end delay of the k th packet flow in the ICN, which is measured between the incoming router of the source domain and the outgoing receiver router of the k th packet flow;  rpe,qk is the normalized requirement for the average delay of packets of the k th flow in the ICN p th domain (  p,k ). k

Thus, for each flow k  K , the requirements req for the average end-to-end delay of packets are distributed (normalized) in some way between separate domains with the following conditions:

N  rpe,qk  kreq and  p,k  rpe,qk . p1 ( 6 ) Within the framework of this study, it is considered that req and rpe,qk are predek termined. This raises the problem of formulating the conditions of normalized QoS on the network in terms of calculating expressions for  p,k . To obtain such conditions it is necessary to use the functionality of tensor analysis of networks. The results presented in [18-25] allowed obtaining analytical expressions for calculating the values of end-to-end QoS indicators, which were evaluated and analyzed between a pair of individual routers. For the case considered in this study, in the structure of an arbitrary network domain, multiple routers may simultaneously belong to each of the sets Bipn,k and/or Bopu,tk , which is a special characteristic of transit domains. The following methodology is proposed to take into account the features of the multidomain ICN architecture. is determined, between which  p,k will be calculated. In the domain that served as the source of the k th packet flow, the Ripn was the router through which the k th flow arrived into the ICN. For the domain that served as the k th packet flow p receiver, the Rout was the router through which the k th flow outwent from the ICN. 2. Additional imaginary routers Ripn and Roput , which become adjacent for routers from Bipn,k and Bopu,tk sets respectively through imaginary communication links, are introduced to the ICN structure at the boundaries of the p th domain that interacts with other domains through two or more routers. If the boundary between the p th and q th domains passes through several routers, as shown, for example, in Fig. 2, then the imaginary routers Ripn and Rout coincide. The introduction of p imaginary routers is conditioned by the fact that value  p,k can now be evaluated and analyzed already between a pair of routers Ripn and Rout on the basis of the p approach proposed in [10, 18]. In order to prevent impact of packet delays  p,k in imaginary communication links that connect real border routers to imaginary ones, their bandwidths must go to +∞ during the calculations. 3. With the aim of further geometrization of the ICN structure, the continuous numbering of the communication links in the p th domain is adopted. For this purpose, the set of network links belonging to the p th domain is denoted as V p  vzp , z  1, n~p  , where n~p and m~p are the number of communication links and routers respectively in the p th domain taking into account the introduced imaginary elements of the network. Thus, for an example of the structure of ICN shown in Fig. 2, n1~  n2~  6 and m1~  m2~  5 . Such an increase in the number of communication links and routers leads to an increase in the number of variables with the condition ( 2 ) being replaced by the following flow conservation xip,,jk conditions on the p th domain routers:     Eip,jE p  

 Eip,jE p   Eip,jE p xip,,jk  xip,,jk  xip,,jk    E jp,iE p E jp,iE p

 x jp,,ik  1, if Rip  Ripn ; x jp,,ik  0, if Rip  Ripn , Roput ; x jp,,ik  1, if Rip  Roput . ( 7 ) r1eq

R11

R12

R31 R41  1,1 1 2 Rout Rin

R12 R22 150

R32  2 ,1

According to the methodology of tensor modeling of ICN [18, 19, 23, 24], the domain structure determines the anisotropic space formed by coordinate paths. Network edges (links), circuits, interpolar paths, and node pairs can act as coordinate paths where the network poles are the routers Ripn and Roput . The dimension of this space is determined by the total number of edges in the network and is equal to n~p [18, 24]. From all possible interpolar (namely, end-to-end from the source to the destination) paths in the p th domain, we choose  p linearly independent i , i  1,  p . Whereas the set of internal node pairs is represented by the set  j , j  1,  p  . These sets form the basis of the n~p -dimensional space of the network structure: ~p  n~p  m~p  2 ; ~p  m~p  2 ; n~p  ~p  ~ . p

When routing the k th flow in the selected space, the p th domain can be represented by a mixed bivalent tensor Q  T   , where  is the tensor multiplication operator and the components of the tensor are a univalent covariance tensor of average packet delays T and a univalent contravariant tensor of the average intensities of flows  in the coordinate paths of the selected domain. Further, the index p in all tensor quantities and their projections will be omitted for greater clarity of information perception, because it is always a question of only physical quantities that are associated with the p th domain. The tensor can be written in the index form: qij   ji , ( i, j  1, n~p ), ( 8 ) ( 9 ) where  j is the average packet delay along the jth coordinate path (s); i is the average packet flow intensity along the ith coordinate path (1/s).

In the selected n~p -dimensional space, the tensor ( 9 ) will be set in one of two coordinate systems. The first is the coordinate system (CS) of the network edges vk , k  1, n~p  , the second one is the CS of linearly independent interpolar paths i , i  1,  p  and internal node pairs  j , j  1,  p , the projections of the tensor in which will be denoted by the index  .

In the case of modeling the operation of the network routers interface by the queuing system M/M/1, the coordinates of the projection of the metric tensor G in the basis of edges Gv represented by diagonal elements of the matrix will be determined by the expression:

gvzz  vz (z  Βvz ) , where G1 (t) is the square ~p  ~p submatrix; G4 (t) is the square ~p  ~p submatrix; G2 (t) is the ~p  ~p submatrix; G3 (t) is the ~p  ~p submatrix.

G  AtGv A , where G is the projection of the metric tensor in the CS of interpolar paths and internal node pairs; A is the n~p  n~p matrix of covariant coordinate transformation in the transition from the CS of interpolar paths and internal node pairs to the basis of edges; t is the operation of the matrix transposition.

As shown in [23, 24], the matrix G can be represented as

G    

G1 G3 |  |

G2    , G4 where vz is the intensity of the k th th flow in the z th communication link when using the continuous link numbering; Βvz is the intensity of the aggregated flow in the z th communication link, which is defined as follows: vz  rkeq xip,,jk , Βvz   rkeq xip,,jk under vzp  Eip,j .

kK

Projections of the twice contravariant metric tensor G when changing the CS of its consideration are transformed by the law (10) (11) (12) (13)

Then, as shown in [22, 23], the conditions for ensuring the normalized QoS in the p th domain when routing the k th flow ( 8 ) take the form:

    G1  G2 G4   1

G3 Treq ,  where 1 is the matrix inversion operation;  is the ~p -dimensional vector of flow intensities in the interpolar paths of the selected domain with coordinates i

The hierarchical coordination inter-domain routing method will be based on solving an optimization problem for the calculation of vectors of routing variables xk p ( p  1, N , k  K ) subject to constraints ( 3 )-( 7 ), (14)-(16) by using the following optimality criterion: min F , F    (xkp )t H kp xkp ,

pN kK

Treq is the ~p -dimensional vector of average packet delays in the interpolar paths of the selected domain, each of which coordinates i corresponds to the condition i  rpe,qk , i  1, ~p . 5

Method of hierarchical coordination inter-domain routing in Software-Defined infocommunication network with provision of normalized QoS that are connected by the following condition   i  rkeq ; i1 where H kp is the diagonal matrix of routing metrics of links of the p th domain.

The goal coordination principle [9-11, 15, 16] will be used to solve the optimization problem (17). Then, moving to the problem at the unconditional extremum, it is necessary to maximize by  the Lagrangian of the form:

N N N L    (xkp )t H kp xkp     ( μkp,q )t (Сkp,q xkp  Сqk, p xqk ) , p1kK p1q1 kK q p (14) (15) (16) (17) (18) where  is the vector of Lagrange multipliers;  p,q are subvectors of the vector  assigned to each of the vector-matrix domain interaction conditions ( 5 ).

Given that within the principle of goal coordination, the Lagrange multiplier vectors  are calculated at the upper level and for the lower level are values known, the N expression (18) can be represented in the following decomposition form: L   Lp , p1 where

N Lp   (xkp )t H kp xkp   kK qp1q kK p  (μkp,q )t С kp,q xkp  N  ( μqk, p ) С p,q xkp , t k

(19) qp1q kK p where K p is the subset of flows incoming to the p th domain from other domains; K p is the subset of flows outgoing from the p th domain ( K p , K p  K ).

Within the framework of the proposed method, the general problem of hierarchical coordination of inter-domain routing is formulated as a two-level optimization problem: 1. At the lower level, SDN controllers of the domains calculate the routing variables represented by vectors xkp during the minimization of Lagrangians (19) under constraints ( 3 ), ( 4 ), ( 7 ), and (14)-(16). The results of the calculations are sent to the top level, namely to the SDN controller of the network. 2. At the top level, the SDN controller of the network coordinates the lower-level solutions to ensure that the conditions ( 5 ) are met by modifying the Lagrange multiplier vectors: μ p,q (a 1)  μkp,q (a)  μkp,q , μ kp,q (x) k  С p,q xkp  Сq, p xqk ,

(20) x  x* where a is the iteration number; μkp,q is the gradient of the function (19). 3. The modified values of the Lagrange multiplier vectors μkp,q are transmitted to the lower level for the calculation of new routing vectors xkp . The calculation process becomes iterative. Inter-domain route connectivity will be ensured when the gradient values (20) approach zero.

From the technological point of view, minimizing the number of iterations of the procedure (20), when obtaining the desired optimal solution, aims to reduce the amount of service traffic transmitted between hierarchical levels about the results of calculations at each iteration, and to decrease the total time of solving the problem of inter-domain routing in the ICN as a whole [9-11]. 6

Investigation of the proposed method of hierarchical coordination inter-domain QoS routing

Let us investigate the proposed method of hierarchical coordination inter-domain routing in the ICN in order to confirm its functionality, adequacy, and efficiency of the obtained calculation results. In the framework of the numerical example, let us analyze the peculiarities of the solutions to the problem of hierarchical coordination inter-domain QoS routing for the variant of the infocommunication network structure shown in Fig. 1. As an example, consider a single-flow case when, in the course of a study, the requirements for the QoS level in a multidomain network were given by the following parameters: req  350 1/s and req  100 ms, 1r,e1q  40 ms, r2e,1q  60 ms. 1 1 (21)

Then Fig. 3 presents a solution to the problem of inter-domain QoS routing prior to the start of the coordination procedure (20). In Fig. 3, the link breaks show the following data (top to bottom): packet flow intensity, bandwidth, and average packet delay in this link.

The characteristic feature of the obtained solution (Fig. 3) is that the conditions for providing the normalized QoS ( 6 ), (21) are fulfilled: the maximum end-to-end delay in the first domain was 38.5 ms and in the second domain it was 48.6 ms. However, the conditions for inter-domain interaction ( 5 ) were not met. Inter-domain route connectivity was ensured after the third iteration of the coordination procedure (20).

Domain 1

R21 350

R11

The coordinated solution to the problem of inter-domain QoS routing is presented in Fig. 4, within which the specified normalized values of the average packet delays in each of the domains (21) were provided: in the first domain the maximum end-toend delay was 38.5 ms, and in the second domain it was 58.3 ms. The paper proposes a method of hierarchical coordination inter-domain routing in SDN, which is a further development of the solutions presented in [9-11, 23, 24]. The novelty of the method is that the routing solutions obtained with it are aimed not only at increasing the scalability of ICN, but also at ensuring the normalized QoS in terms of average transmission rate and end-to-end average packet delay.

The proposed method is based on the use of the decomposition flow-based model of inter-domain routing ( 3 )-( 7 ). The model was supplemented by the conditions for the provision of normalized QoS (14)-(16), which were formulated in an analytical form on the basis of tensor modeling of the ICN ( 8 )-(13).

Within the proposed method, the problem of inter-domain QoS routing was presented in the optimization form with a quadratic optimality criterion (17). The goal coordination principle was used to solve the optimization problem. During the study of the method, its functionality and efficiency were confirmed in terms of ensuring the normalized QoS. It was found experimentally that the method converged to the optimal solution for the finite number of iterations (20). For the structure of the network that was selected as the test (Fig. 1), the number of iterations of the coordination procedure (20) with the proper setting of the gradient search did not exceed three iterations. Reducing the number of such iterations helps to decrease the amount of service traffic that is transmitted across the network between routers and SDN controllers at different levels, as well as minimizing the overall time for solving the inter-domain QoS routing task.

The prospect of further research in this area is that the QoS metrics, which each domain should provide, are not set statically, for example, on the SDN controller, but can be redistributed dynamically between domains with the fulfillment of the conditions ( 6 ) in accordance with their structure, capacity, and utilization. 14.Amin, R., Reisslein, M., Shah, N.: Hybrid SDN networks: A survey of existing approaches. IEEE Communications Surveys & Tutorials, 20( 4 ), 3259–3306. IEEE (2018). https://doi.org/10.1109/COMST.2018.2837161 15.Mesarovic, M.D., Macko, D., Takahara, Y.: Theory of hierarchical multilevel systems. Elsevier (2000) 16.Singh, M.G., Titli, A.: Systems: decomposition optimisation and control. Pergamon (1978) 17.Segall, A.: The modeling of adaptive routing in data-communication networks. IEEE Transactions on Communications, 25( 1 ), 85–95 (1977) 18.Lemeshko, A.V., Evseeva, O.Y., Garkusha, S.V.: Research on tensor model of multipath routing in telecommunication network with support of service quality by greate number of indices. Telecommunications and Radio Engineering, 73(15), 1339–1360 (2014). https://doi.org/10.1615/TelecomRadEng.v73.i15.30 19.Lemeshko, O., Yevdokymenko, M., Naors, Y.: Development of the tensor model of multipath QоE-routing in an infocommunication network with providing the required quality rating. Eastern-European Journal of Enterprise Technologies, 5 ( 2 ), 40–46 (2018). https://doi.org/10.15587/1729-4061.2018.141989 20.Strelkovskaya, I., Solovskaya, I., Grygoryeva, T., Paskalenko, S.: The solution to the problem of the QoS characteristics definition for self-similar traffic serviced by the W/M/1 QS. In: 2016 Third International Scientific-Practical Conference Problems of Infocommunications Science and Technology (PIC S&T), pp. 40–42. IEEE (2016). https://doi.org/10.1109/INFOCOMMST.2016.7905330 21.Strelkovskaya, I., Solovskaya, I., Makoganiuk, A.: Optimization of QoS chracteristics of self-similar traffic. In: 2017 4th International Scientific-Practical Conference Problems of Infocommunications. Science and Technology (PIC S&T), pp. 497–500. IEEE (2017). https://doi.org/10.1109/INFOCOMMST.2017.8246447 22.Strelkovskaya, I.V., Solovskaya, I.N.: Tensor model of multiservice network with different classes of traffic service. Radioelectron.Commun.Syst. 56, 296–303 (2013). https://doi.org/10.3103/S0735272713060058 23.Lemeshko, O.V., Yeremenko, O.S., Hailan, A.M.: QoS solution of traffic management based on the dynamic tensor model in the coordinate system of interpolar paths and internal node pairs. In: 2016 International Conference Radio Electronics & Info Communications (UkrMiCo), pp. 1–6. IEEE (2016). https://doi.org/10.1109/UkrMiCo.2016.7739625 24.Yeremenko, O.: Development of the dynamic tensor model for traffic management in a telecommunication network with the support of different classes of service. Eastern-European Journal of Enterprise Technologies. 6( 9 ), 12–19 (2016). https://doi.org/10.15587/17294061.2016.85602

References

1. Wibowo , F.X. , Gregory , M.A. , Ahmed , K. and Gomez , K.M. : Multi-domain software defined networking: research status and challenges . Journal of Network and Computer Applications , 87 , 32 - 45 ( 2017 ). https://doi.org/10.1016/j.jnca. 2017 . 03 .004

2. Katsalis , K. , Rofoee , B. , Landi , G. , Riera , J.F. , Kousias , K. , Anastasopoulos , M. , Kiraly , L. , Tzanakaki , A. , Korakis , T. : Implementation experience in multi-domain SDN: Challenges, consolidation and future directions . Computer Networks , 129 , 142 - 158 ( 2017 ). https://doi.org/10.1016/j.comnet. 2017 . 09 .005

3. Blial , O. , Ben Mamoun , M. , Benaini , R.: An overview on SDN architectures with multiple controllers . Journal of Computer Networks and Communications , 2 , 1 - 8 ( 2016 ). https://doi.org/10.1155/ 2016 /9396525

4. Medhi , D. , Ramasamy , K. : Network routing: algorithms, protocols, and architectures . Morgan Kaufmann ( 2017 )

5. Misra , S. , Goswami , S. : Network Routing: Fundamentals, Applications, and Emerging Technologies. Wiley ( 2017 )

6. Szigeti , T. , Zacks , D. , Falkner , M. , Arena , S. : Cisco Digital Network Architecture: Intentbased Networking for the Enterprise . Cisco Press ( 2018 )

7. Wójcik , R. , Domżał , J. , Duliński , Z. , Rzym , G. , Kamisiński , A. , Gawłowicz , P. , Jurkiewicz , P. , Rząsa , J. , Stankiewicz , R. , Wajda , K. : A survey on methods to provide interdomain multipath transmissions . Computer Networks , 108 , 233 - 259 ( 2016 ). https://doi.org/10.1016/j.comnet. 2016 . 08 .028

8. Eun , J. , Jung , H.: August. The implementation of domain routing protocol in hierarchical domain network model . In: 2015 17th Asia-Pacific Network Operations and Management Symposium (APNOMS) , pp. 396 - 399 . IEEE ( 2015 ). https://doi.org/10.1109/APNOMS. 2015 .7275350

9. Lemeshko , O. , Nevzorova , O. , Hailan , A.M. : February . Hierarchical method of routing and resource allocation in DiffServ-TE network . In: 2018 14th International Conference on Advanced Trends in Radioelecrtronics , Telecommunications and Computer Engineering (TCSET), pp. 1014 - 1018 . IEEE ( 2018 ). https://doi.org/10.1109/TCSET. 2018 . 8336366 10. Lemeshko , O. , Yeremenko , O. , Nevzorova , O. : Hierarchical method of inter-area fast rerouting . Transport and Telecommunication Journal , 18 ( 2 ), 155 - 167 ( 2017 ). https://doi.org/10.1515/ttj-2017 -0015 11 .Nevzorova, Y.S. , Аrous , K.М. , Salakh , М.Т.R.: Method for hierarchical coordinated multicast routing in a telecommunication network . Telecommunications and Radio Engineering , 75 ( 13 ), 1137 - 1151 . ( 2016 ). https://doi.org/10.1615/TelecomRadEng.v75. i13 . 10 12 .White, R. , Tantsura , J.E. : Navigating Network Complexity: Next-generation routing with SDN, service virtualization, and service chaining . Addison-Wesley Professional ( 2015 ) 13 . Lin , S.C. , Akyildiz , I.F. , Wang , P. , Luo , M.: QoS-aware adaptive routing in multi-layer hierarchical software defined networks: A reinforcement learning approach . In: 2016 IEEE International Conference on Services Computing (SCC) , pp. 25 - 33 . IEEE ( 2016 ). https://doi.org/10.1109/SCC. 2016 .12