The JINR cloud infrastructure hosts a number of cloud services to facilitate scientific workflows of individual researchers and research groups. Although batch processing systems are still the major compute power consumers of the cloud, new auxiliary cloud services and tools are being adopted by researchers and are gradually changing the landscape of the cloud environment. While such services, in general, are not so demanding in terms of computational capacity, they can have spikes of demand and can dynamically scale to keep the service availability at a reasonable level. Moreover, these services may need to compete for resources due to the limited capacity of the underlying infrastructure. This paper discusses how resource distribution can be managed in such a dynamic environment with the help of a cloud meta-scheduler.
The JINR cloud [
The cloud provides two types of resources: shared resources, which are in common use by all JINR participants, and resources of the so-called Neutrino Platform, which are owned by several neutrino experiments JINR participates in.
The abovementioned cloud services are sometimes underutilized [
As mentioned in the introductory part, the underutilization of cloud resources can occur for various reasons. For example, interactive services (like JupyterHub or interactive nodes of the HTCondor cluster) are usually underutilized at night or during holidays. Figure 2 illustrates a typical CPU usage profile of an interactive machine. It is clearly seen that this machine was not used at all at night and in the morning, then the load increased after lunch and dropped by the end of the working day. Sometimes people can work at night too.
On the contrary, batch clusters are usually better loaded in terms of hardware usage since batch jobs do not need sleep and can run for hours or even days without a stop. However, resources dedicated to individual projects or experiments can also sometimes encounter periods of inactivity (for example, when data production stops during detector maintenance periods), and these periods can be quite long, up to a few weeks (Fig. 3).
Thus, in most cases, underutilization can be considered normal (and expected) because the system efficiency can be defined in different ways depending on the purpose of the system in question. For example, with batch systems we usually want to maximize hardware utilization, while in the case of interactive systems like JupyterHub, we try to keep the system responsive and for this reason it is fine to keep a reasonable amount of resources idle and ready to serve incoming users.
In certain cases, hardware utilization can be easily improved by redistributing resources between different cloud services. For instance, at night, most of the interactive nodes can be stopped in favor of additional batch cluster worker nodes reverting everything back in the morning. The same applies to the owned resources in batch systems: when the experiment knows that its resources will not be used for a long period of time, these resources can be shared with other experiments using the same technique, i.e. scaling in unused resources and scaling out systems that need them. Nevertheless, standard cloud tools do not give us convenient control over the cloud services scaling, taking into account the interests of all services running in the environment, as well as the interests of different working groups that own some fraction of cloud resources. To deal with the issue, we started the Cloud Meta-scheduler project.
The project goal is to provide resource managers and users of JINR’s cloud services with convenient tools for managing and monitoring resource distribution between the services and resource owners, as well as for creating and approving resource lease requests, with a scheduler component in its core, which handles the actual scaling of the services.
Although the general idea of such a system seems simple, the implementation of the actual scaling of cloud services encounters intricate details, as different services (or even their different parts) can be scaled for different reasons. For example, three different components of the HTCondor cluster may be scaled for the following reasons: • • • schedulers (virtual machines that operate the job queue) – to maximize the job submission rate; interactive nodes – to keep them responsive; worker nodes – to improve the throughput of the cluster.
For this reason, we started the development with the prototype of a meta-scheduler component to study possible technical solutions, to discover some potential pitfalls and better understand the requirements for the system under development.
Python was chosen as the primary development language because of its rapidly growing
popularity in data science, which makes it possible to involve data science students in the development
of the project with the potential to apply data analytics for incorporating more complex scheduling
schemes [
Scheduler daemon – runs the scheduling loop; HTCondor API microservice – implements communication with the HTCondor cluster; Cloud API microservice – implements communication with the JINR cloud.
The microservices approach gives the system additional flexibility that may be needed for large-scale deployments. For instance, the HTCondor API implemented as a service can be run on the same machine as the scheduler and can communicate HTCondor via SSH; however, it can also be run on the HTCondor scheduler machine, directly executing shell commands to talk to HTCondor and then communicate back the information to the scheduler over the network using the specialized Pyro wire protocol.
The current early version of the prototype implements only the simple automatic scaling of HTCondor worker nodes, depending on the job queue size: when there are idle jobs in the queue, more worker nodes are created (if there are common resources available), and once the jobs are completed, the nodes are deleted.
• • •
The further development of the prototype is planned in the following stages:
The IT industry and data science are rapidly evolving, new technologies are emerging and becoming popular, and the changing IT landscape sets new challenges in computing infrastructures management. To keep up with the evolution of computing models and environments, we need to develop novel tools to help us efficiently handle their growing complexity. In this paper, we have described the idea and development course of one such tool designed for the dynamic load-balancing of multi-service cloud environments to improve computing resources utilization in cloud environments. with