The designing and development of a computing clouds is complex process where numerous factors have to be taken into account. For example, size of planned cloud and potential growth, hardware/software platforms, flexible architecture, security, ease of maintenance. Computing cloud is quite often consisted of several data centers (DC). The DC is considered to be a group of hardware and/or virtual servers which is dedicated to run the user virtual machines (VM) and/or storage servers. Each pair of DCs may be interconnected by one or more virtual data transfer links. To manage such cloud to form “Infrastructure as a Service” (IaaS) a distributed operating management system (DOMS) is needed. The proposed architecture for DOMS is a set of software agents. Important advantages of such approach are flexibility, horizontal scalability, independent development/maintenance of any agent. The specially developed protocol to send and receive requests between agents is also discussed. Due to geographical distribution, the requirements for system stability in terms of hardware and software malfunctions are high. Proposed DOMS architecture is addressed operating stability as well. Observation of prototype consisting of several DCs in ~100Km distance from each other and practical results were presented. Potential application fields where this development might be used is also discussed.
The specific demands for computing power are often exceeds abilities of one data centre (DC),
which leads to various types of DC clustering implementation. Currently, in order to ensure efficient
use of the computing resources of individual DCs, approaches are being implemented based on DCs
combination, such as Grid computing and Cloud computing. It eventually fulfilles the requirement of
the redistributing available capacities flexibility. One example is the largest grid-type system
WorldLHC Computing Grid [
Of great importance is the geographical distribution of data centres [
Cloud computing offers tremendous potential benefits in agility, resiliency, and economy. Moreover, when using IaaS cloud model, it is possible to provide various type of infrastructures, including Grid, therefore it is often preferable. Thus, the basic requirements for geographically distributed IaaS could be these: the relative simplicity of the redistribution of resources between local data centres (computing power, data storages, transmission channels), geographically remote from each other; security/reliability of data transmission and long-term data storage; gradual (staged) degradation in case of failure of hardware and/or software components as failures of individual components should not lead to a shutdown of the whole system.
There are important questions about reasons of implementing of specific cloud system, e.g. how clients might avoid being locked in specific cloud system? Besides, there are always several issues about data storage and transmission safety guarantee, i.e. how can one store his data in data storage for many years and how to provide high-level security for data transmission between geographically distributed DCs? All these questions are being considered in the project presented in this paper.
The main goal of the project is the designing and development computing infrastructure, which permits to integrate the resources of Geographically Distributed DCs to form cloud Infrastructure as a Service (IaaS) and attempt to resolve the issues mentioned above.
An analysis of publications devoted to architectural and technical solutions that are used to
create distributed computer infrastructure management systems, for example [
The choice of the following technical solutions was determined by their modernity and rich
component developers experience:
OS NauLinux (RHEL clone)
Ryu SDN controller
CEPH as storage backend
Openstack for virtual resources control [
The components listed above are considered as plain tools only that can help to implement certain approaches to the design of the entire system. Among other things, it is important to avoid any dependence on specific software for IMS. The following approaches were used to create the prerequisites for such independence in the design of the SGDD control system: the architectural solution must be implemented on the basis of software-configurable infrastructure technologies; software components of the system should be implemented using opensource software only, which is configured for specific functionality; system components must be created as software agents that run in an operational-isolated environment, such as virtual machines or containers; to protect the communication lines between the data centres, quantum communication technology at the side frequencies must be applied.
Based on the mentioned approaches, the IMS of the SGDD was created, the architecture of which is shown in Fig. 1.
The interaction between the software agents is implemented through a specially developed
protocol, which includes organizing request queues using the distributed RabbitMQ message queuing
system between agents and storing all requests in a database, which came up for the reasons
mentioned above and was examined in detail in [
An important element of the system that combines all the data centres within the SGDD is the Registry directory (RD), inside which there are subdirectories describing each individual data centre, as well as data channels between the data centres. The RD is automatically replicated to all data centres within the SGDD. Access to the data in the RD is via https, the data itself in the directory is stored in encoded form. Thus, access to the data in the register directory is possible only for authorized software agents.
All software agents of SGDD are launched automatically in the process of loading hardware servers. Each software agent uses configuration parameters from its local RD which he inherited from the main RD via secured channel. Since any change of the SGDD system is reflected in RD, no changes in software agents are required, which is considered to be convenient. The list of IMS agents with their functions is presented in Table 1.
Handling cloud orchestration So-called «agent in the middle», receives requests requests from Registration, (or transaction with several ones) and forwards authorization & billing agent. At them to the specific agent (VM, SDS, virtual links). least one instance per DC and one Orchestration agent is a core agent of IMS. top level instance per SGDD.
Creation and maintaining of VMs via Performing all operations on VMs together with Openstack. At least one instance per orchestration agent and virtual objects database.
DC.
Creation and maintaining of SDSs Performing all operations on SDSs together with via CEPH. At least one instance per orchestration agent and virtual objects database. DC. Working with various types of storage (block, S3, etc) and SLA (encoding, replicas number, etc).
Virtual links Creation and maintaining of virtual Providing virtual links inside single DC or between management agent links via Ryu and Open vSwitch. At two DCs. SDN and NFV were used to provide the least one instance per DC. required SLA: compression, erasure coding, encryption using a quantum key distribution system, parallel communication channels.
Monitoring of virtual resources, Monitoring of virtual CPU and storage, collecting providing statistical data for billing. monitoring history for billing purposes, alert At least one instance per DC. system. All alerts must be sent to orchestration agent in order to start damage evaluation procedure.
Monitoring of power supply, climate Monitoring, security issues, alert system. All alerts control and fire extinguishing must be sent to orchestration agent in order to start systems, DC invasion sensors, video damage evaluation procedure. cameras for video surveillance, etc.
One instance per DC.
Agent Registration, authorization & billing agent Orchestration agent VM management agent SDS management agent Virtual infrastructure monitoring agent Physical infrastructure monitoring agent
It is also necessary to provide special components for agent to operate properly. There are
three main components for such issue:
Database (DB). Since databases are one of the important components of IMSs, fault tolerance
technology based on configuration of high availability architecture (master-master mode,
where two masters are located on separate physical servers) is used to increase the reliability
of operation [
Repository. It contains all software components (both specially designed and borrowed from external sources) that are used in the IMS. For any deployments or upgrades to the IMS, software components are used only from this repository. For the placement of all components in the repository, developers or the administrators of the support of the IMS are responsible at the initial stage. In other words, the repository should be the only source of software components of the IMS. There are several repositories: one production repository in each DC and one-two preproduction repositories. When any package has been added or upgraded and tested in preproduction repository all production repositories are synching with preproduction repository. Each repository is divided by subrepositories. Each subrepository can be developed independently from other subrepositories. Suggested repository architecture permits to build up virtual machines or/and containers to eliminate conflicts between different libraries, packages, etc. The evaluation of new features (technology preview) becomes much easier with such the approach.
Semi-automated deployment procedure consists of two stages. The first one is installation planning where the administrator must allocate all IMS agents throughout hardware servers. Some services like DB needs to be allocated at least on two separate servers for creating an HA cluster. It is necessary to provide information about storage allocation for agents for storing logs, big databases and all configuration parameters for DC network. Afterwards, a special image of virtual machine is used for deployment process. It is used for the OS installation, building up the hardware database, preparing Registry directory and IMS components installation afterwards, one by one in according to the specification prepared in first stage. The deployment of the Registration, authorization & billing agent is implemented as completely separate step.
The significant part of the data transfer functionality is implemented as virtual network
functions to meet SLA. Among SLAs for virtual data lines, there are requirements for encoding the
transmitted data, including the requirement to use the quantum distribution of encoding keys, which is
based on the technology of quantum communications at the side frequencies of modulated radiation in
this paper [
When individual hardware component of the SGDD fails, the orchestration agent starts a damage assessment procedure, the results of which are forwarded to the system administrator through GLPI, a system for handling claims and incidents. Built-in inventory system with the FusionInventory plugin allows tracking the history of each software change and hardware state. Using the knowledge base and information received, administrator will be able to choose some options for minimizing damage.
Experimental studies of IMS were carried out on a prototype of a scalable geographically distributed data centre built in conjunction with the SMARTS company (Russia, Samara). The layout diagram is shown in Fig. 2.
DCs (3 microDCs) are combined into a single system with a fiber-optic cable, one of the fibers is used to transmit quantum keys. Maximum distance between DCs is 100 km (~62 miles). Every DC is provided with the same hardware, which is: HPE DL380 Gen10 8LFF CTO servers with OS NauLinux for VMs and SDSs (2 x Intel Zeon
Gold CPU, 128 GB RAM, 30 TB storage, 6 network interfaces); HP FF 5700-32XGT-8XG-2QSFP+ network switches with OpenFlow protocol support; Special quantum key distribution units.
There were dozens of tests carried out, such as virtual objects creation and deletion tests, read/write tests for VMs and SDSs, data links with OpenFlow rules tests, etc. All of those were successfully completed and were performed on the basis of the test patterns developed within the project. That showed the reliable performance of the adopted technical solutions and created software.
In the process of passing the test, the state of the agents of the system and equipment was monitored by displaying the data entering the Zabbix on the Grafana panel (Fig. 3). Monitoring agents of the system was carried out by sending a special message according to the developed protocol in order to determine the response time of the agent to the request. In this case, the events of both disconnecting one of the agents and disconnecting the data center were investigated.
A description is given of a control system for a geographically distributed DC created using modern architectural approaches and technical means. Particular attention is paid to the flexibility of rebuilding the system to new tasks, efficient use of resources, reliable operation in the event of failure of system components and protecting information from unauthorized access. The use of IMS when conducting experimental research on the SGDD layout confirmed the effectiveness of the main decisions made during the development. Implemented according to the Infrastructure as a Service model, the hardware-software complex demonstrates wide possibilities for creating cloud infrastructures and using it for business purposes. Furthermore, such DC clustering model could provide a solution for several issues concerning specific demands for computing power for a single DC. Thus, such model does make computing power more flexible in terms of aggregation of DC resources.
The main advantages of developed solution are: Opportunities for the rapid reallocation of resources of a geographically distributed DC in terms of computing power, data flows, and engineering infrastructure with IMS.
IMS might use not only bare metal servers but virtual resources from existing clouds. It gives a lot of flexibility because someone can use virtual resources from several existing clouds. With minor corrections IMS start use different underlying components, e.g. vmware instead Openstack.
Horizontal scalability by starting several instances of same type agents in DC.
A high degree of autonomy of the functioning of the components of a geographically distributed DC due to the architecture based on software agents running in operationally independent environments.
Own software repository containing the Linux distribution and all necessary software packages (including source codes), which ensures the autonomy of the system and preservation of its operability even in the case of network isolation of DC.
Secure medium between nodes with the quantum technology of sending encoding keys in combination with the technology of SDN.
Platform for long-term storage and transmission of big data, which is formed by a combination of software components used in the system.
Unique semi-automated deployment procedure of IMS for a single DC. Such procedure was designed for this project, so it has no resemblance with other well-known auto deployment procedures. Such procedure is considered to be convenient when one has to add some extra hardware into the existing and operating DC.
The research has been carried out with the financial support of the Ministry of Science and Higher Education of the Russian Federation under grant agreement No.03.G25.31.0229. [4] Fedchenkov P.V., Khoruzhnikov S.E., Samokhin N.Y., Shevel A.Y. The designing of cloud infrastructure consisting of geographically distributed data centres. Proc. of the VIII International Conference «Distributed Computing and Grid-technologies in Science and Education» (GRID 2018), Dubna, Moscow region, Russia, 2018, pp. 32–36. Available at: http://ceur-ws.org/Vol-2267/32-36paper-5.pdf (accessed: 06.10.2019)