The article discusses a complex solution for managing traffic recording and storage in data storage systems. In the conditions of modern legislation, the issue of storing a large amount of data becomes acute. Physical storage management avoids the unnecessary costs of scaling storage systems. The article proposes the structure of a hardware and software complex for managing physical data storage for storage systems that can be used by owners of technological communication networks to store traffic. Control mechanisms are considered, such as the distribution of data over various media using Kohonen neural networks and forecasting capacity extension using a statistical model and machine learning methods. traffic, data storage system, data distribution, physical data storage, machine learning, neural The requirements of modern legislation in the field of citizen security pose serious challenges to various organizations, including data storage. The anti-terrorist amendments adopted in 2016 (the socalled “Yarovaya law”) obliged telecom operators to store traffic metadata for three years, and the traffic itself for six months. In addition, in June 2020, the Ministry of Digital Development, Communications and Mass Media of the Russian Federation proposed a bill, according to which the owners of technological communication networks are required to store traffic for three years [1].
B I 2016 2017 2018
2019
2020 Copyright for this paper by its authors.
In October 2020, only “Rostelecom” spent 7.8 billion rubles on data storage equipment. Other
operators also purchase various storage systems (table 1) [
As can be seen from Table 1, telecom operators of the Russian Federation suffer serious financial costs for the purchase of equipment for storage systems. On the other hand, modern information technologies make it possible to manage the resources of data storage systems, use them efficiently and, therefore, avoid unnecessary costs.
The data storage system can manage the recording of the incoming data stream and, firstly, distribute it among different types of media, and secondly, monitor the state of the storage and make a forecast of capacity growth for its timely extension. 2. Physical Data Storage Managing During Recording and Storing Traffic
A research a study has been carried out in which a data storage system is considered as a storage
management system that performs the following functions:
• Distribution of data files on various types of media, depending on the file size and storage
time
• Monitoring the storage state based on snapshots of each media state
• Forecast of storage capacity extension. [
Obviously, for the implementation of such a storage management system, a soft-ware-hardware system is needed that performs the above functions.
It is proposed to include a programmable logic controller (PLC) in this system, which distributes files to media and software that monitors the state of the physical storage and builds a forecast for its extension (Figure 3).
The controller receives an incoming data stream (for example, internet traffic). The controller performs clustering of incoming traffic using Kohonen's neural networks and, in accordance with the resulting topological map, distributes data files to media in the physical data storage.
Physical storage can be organized depending on the information being recorded. In paper [
This solution can be easily adapted to the needs of the owners of technological communication
networks [
The structure of physical data storage is determined by the storage system administrator and can contain, for example, RAID arrays for text files, streamers for audio and video files. In addition, volumes inside a RAID array can have different operating systems with different sizes of the logical data block, which will avoid the "under-filling" of files during writing (Figures 4, 5).
tlim
Vmax tmax 1 1 = ∫ ( )
= ∑ ( ), = ∫ ( ) = ∑ ( ), 1 1 (1) (2) where tlimmn – time to reach limited media capacity; tmaxmn – time to reach maximum media capacity; f(t) – incoming data function;
The forecasting task is to find the timeline point at which the limited capacity and the maximum
capacity of each media are reached [
To solve this problem, it is necessary to predict the amount of incoming traffic in the storage system.
The forecast can be made by various methods, while it is necessary to consider the peculiarities of the data stream entering the recording. Due to uneven user activity associated with weekends and working days, vacation periods, etc. the incoming data stream is heterogeneous and has a seasonal structure (Fig. 7)
In [
a) 1 0,5 B G , 0 V -0,5 -1 B ,G8 V 3 B0 G , V -0,5
-1 10 20 30 40 50 60 70 80 90 100 110 120 130 140 t, hour
The norms of modern legislation oblige the owners of technological communication networks to store a large amount of data using their own data storage systems. This leads to serious costs, which, in the end, fall on the end user of communication services.
At the same time, modern technologies make it possible to create systems for managing physical data storage that can efficiently consume physical storage resources. Existing virtualization technologies make it possible to create structures containing various types of storage media and distribute the saved traffic files over them depending on certain characteristics of the files.
Since there is a need for regular scaling of the data storage, it is necessary to monitor its status and scale only those media whose capacity limits tend to be maximized. Predicting capacity extension allows for timely scaling.
Thus, dividing the total incoming data stream by media, predicting capacity consumption, and monitoring the state of physical data storage allow owners of technological communication networks to rationally use physical storage resources and avoid unnecessary costs when increasing storage. 4. References