This paper investigates the application of machine learning (ML) model to predict data backup needs and optimize backup solutions in any-scale IT environment. By leveraging ML-driven predictive analytics, users and companies can enhance the efficiency and reliability of their data backup processes, increase performance, reduce costs, and minimize data loss in case incident. The paper describes a solution utilizing the RandomForestRegressor model for learning on attributes of existing files in system to predict a priority of processing backup of files and avoid data redundancy via skipping extra files. It will allow to speed up the backup process and reduce the size of the backup. With enough training on metadata of files and filesystem behavior, the solution will help make backup software more resistant to errors, intelligent and dynamic.
Data is a critical asset for companies in the digital age, who use it as the basis of all operations,
decision-making, and strategic planning [
The primary problem is the lack of adaptive, intelligent systems that can predict and
optimize data backup scope needs in real-time. Fixed backup schedules often lead to excessive
resource consumption by frequently backing up low-priority data and neglecting the needed
backup of critical, high-priority data [
Therefore, there is a need for a new approach that leverages new technics to perform
predictive analytics for data backup strategies. For such needs perfectly fit one of machine
learning models [
Every company that processes any data should take responsibility not only for its secure and
reliable storage but also for corresponding data backup solution. Currently, existing backup
software solutions could perform backups of separate files, folders, or volumes of data via a
straight and narrow approach by performing one or few backup schemes or its combinations
(full, incremental, differential). Full data backup means the solution will make a full copy of
source data, it offers the simplest recovery but consumes the most time and storage space.
Incremental backup - only backs up data that has changed since the last backup, saving time and
storage but requiring multiple backups for a full restore. Differential backup saves all changes
made since the last full backup, balancing time and storage between full and incremental
backups [
To avoid weak parts of existing backup solutions and make the system more dynamic, a new machine-learning-driven solution should help. ML algorithms can significantly improve data backup strategies through various approaches, optimizing both the efficiency and reliability of the backup strategy. This allows to create more intelligent schedule of backups, ensuring that critical data is backed up more frequently while less critical data is backed up less often. This reduces the amount of backups, storage usage and maintains efficient storage utilization.
Another area to optimize – make adaptive backup policies based on real-time monitoring of system resources. For instance, if the ML algorithm detects increased activity in certain datasets, it can trigger more frequent backups for those datasets temporarily. By monitoring the usage of machine resources and their trends (e.g., bandwidth, storage, memory, central processing unit), the ML algorithm can predict the best times for backups to ensure minimal disruption to regular operations and efficient use of resources.
Due to the increasing amount of data [
Additionally, ML algorithms can predict potential backup failures by analyzing system events, logs, and metrics. This allows the backup system to take proactive measures, such as rerunning a backup process or fixing underlying issues before a critical backup operation fails. If a system event happens during backup, the ML algorithm can initiate a spare process of verifying backups and checking their integrity. This ensures that backups are complete, uncorrupted, and can be restored successfully when needed.
Every company has its own policies for IT environment, including data storage, its backups and retention periods, security of systems; so the ML algorithm can ensure that data backup strategies comply with organizational policies and regulatory requirements by continuously monitoring backup processes and systems to meet compliance standards.
To overcome the problems of existing backup software, a new machine-learning backup solution is proposed to solve fixed schedule and redundant data problems. For this purpose, RandomForestRegressor model was selected. This model has the resilience to noisy data, no need for extensive data preprocessing, ability to handle non-linear relationships which make it perfect for performing analytics among file system entries and user behavior.
The proposed software solution should solve the problems mentioned above - avoid redundant files, and make the backup schedule by assigning priority for tasks. The main advantage of the proposed solution will be a feature of predicting priority to make backup of important files faster and speed up the backup process.
For that purpose will be enough of a simple console application. It should consist of a few modules: main – which is the entry point of a system; BackupScheduler – a service to fire backup activities (directly or via timer), BackupExecutor – a main component that performs backup activity, FileChangeStatisticsHandler – a module for tracking of file system changes. The system structure diagram is shown in figure 1.
Development of such a solution will require a Python interpreter installed, Pip configured, Pandas and Scikit-learn, NumPy modules imported, text editor on Windows/Linux/MacOS machine. Additional features, like filesystem access were imported from os, shutil, and glob.
To be able to train a RandomForestRegressor model there will be a need for a large set of data, which could be any directory with different types of files, for the proposed solution it will be %userprofile%\Documents.
In the first step, need to define the criteria of files and how they should be preprocessed - the model should build relations to predict “backup_priority” and “backup_type” of file. Preprocessing allows data to make it suitable for training machine learning models. Model training will be performed by such attributes as: “size”, “last_modified”, “created“, “is_system”, “is_hidden”, “is_readonly”. Target attributes that should be predicted are: “backup_priority” and “backup_type”. Both groups of attributes should be defined, so a model could split data into logical relations between source and result and make sufficient decisions in the future.
The process of model training is iterative: which means the developer should pass incoming variables and review model prediction results in every iteration – if the prediction value does not fall into the expected range – adjust incoming values and repeat. All coefficients should be adjusted due to the exact filesystem and case. It could be implemented by simple conditions – for example, if the file was created less than a minute ago – increment priority by 10, or if file size is less than file system block size (for Microsoft Windows its 4kb) – then it could be more efficient to copy it rather then archive. The algorithm of estimating “backup_priority” is displayed in figure 2 (as there are many criteria to estimate – most of them hidden in function).
Start of the software will be from the Main module. It's an entry point of a console program, which takes arguments: training data path, backup source path, backup target path. Main initiate analyzing of scanning files in the training data path and start RandomForestRegressor model training. It includes the following steps: 1. splitting data into training and actual (made by function “train_test_split” from “sklearn.model_selection”) 2. creating multi-target regressor – to be able to predict multiple variables with the same model (class “MultiOutputRegressor”) 3. fit model to train data (function “fit”).
BackupScheduler has no logic except repeated timer and queue which allows to schedule, postpone, and send backup tasks to BackupExecutor. Additionally, it utilizes CPU and RAM metrics of OS to avoid making backup task when OS under load.
FileChangeStatisticsHandler is a handler derived from “FileSystemEventHandler” to be able to track file system modifications. It implements “on_modified” method which fired on any modification of a file. Unfortunately, it fired even if file metadata changed (in Microsoft Windows 10 environment). So, to avoid extra changes of “last_access” attribute of a file, handler should also calculate, save, and compare MD5 hash of file, it can be made by “md5” function of “hashlib”. Additionally, to have full history of file changes its save dictionary of dates by file into a temporary file.
BackupExecutor uses trained model to predict the backup priority of file(s) by its metadata. “Predict” function return array of results, it our case it is 2 target values due to 2 target attributes ”backup_priority” and “backup_type”. Based on the prediction - a corresponding backup action will be made: either ignore the file (e.g. ”backup_priority” have 0 importance for backup), copy the file (e.g. it’s a small file), or zip file (it's a large file).
Evaluation of data backup solutions involves comparing different factors to determine which option best meets specific needs. Data backup solutions should consider the type and volume of data, the speed and frequency of backups, redundancy, storage use and reliability. Need to have a set of metrics to evaluate and compare the performance and redundancy of existing and new backup solutions:
Compressed size: size of compressed backup files [
Space saving: the reduction in the size of original data relative to the uncompressed data size, calculated by the formula (1).
Time spent: The duration of the backup process, measured in s.
Redundancy: The size of extra files that are included in the backup (temporary files made by other software, operating system files, non-accessible files) - redundant and irrelevant files, calculated by formula (2).
Space saving , %=1−Compressed ¿ ¿ Uncompressed ¿ ¿ 100 , ¿ ¿ Redundancy , %=1− Redundant files ¿ ¿ Compressed ¿ ¿ 100 , ¿ ¿ (1) (2)
Attributes "Space saving" and "Redundancy" are the key indicators of the quality of a backup solution. With ideal software, the "Space saving" should be positive, and as large as possible, and the "redundancy value" should lead to 0.
Software solutions perform backup to the same disk as source data to keep hardware input/output delay as small as possible. The technical specification of the test machine is shown in table 1.
Uncompressed size, bytes 250 774 175 250 774 175 250 774 175 250 774 175 250 774 175 250 774 175
Time spent, s 8 90 8 37,55 2,60 5,33
Compressed size, bytes 245 683 200 251 187 200 179 387 221 240 815 723 241 507 281 241 486 408
Space saving,
% 2.03 -0.16 28.47 3.97 3.7 3.7
Redund ant files size, bytes 11 702 11 702 11 702 11 702 11 702 0
As we can see from the results in table 2 most software performs a backup of redundant files (column “Redundancy files size” - 11 702 bytes), while the proposed solution has 0% redundancy, which is the perfect case among backup software. Also, proposed solution has a high level of “space saving”: 3,7%, while the biggest level is achieved by Duplicati: 28,47%. Additionally, the test for the proposed solution took only 5,33 seconds which is second place after the best - 2,60 seconds by WinZip.
The results of the second test displayed in table 3, confirms that the proposed solution has the smallest value of “redundant files size”: 2 429 307bytes (0,04% redundancy), which is ~4 times less than competitors. The value of “space saving” of the proposed solution is the biggest among competitive software - 2,29%, but it took 91,483 seconds.
So, the evaluation of the result of testing the proposed solution shows good conclusions: the smallest size of redundant files, a high value of “Space saving” and a low amount of “Time spent” means that the RandomForestRegressor model successfully fit for our needs. Applying this machine learning model will significantly improve the performance of backup software – so end-customers will see that backup software will check their latest files first and reduce time spent on backup.
The paper provides insights into the potential of ML-driven predictive analytics solutions to improve data backup strategies, ultimately contributing to more robust, efficient, and adaptive systems. By integrating these ML approaches, software companies can develop more sophisticated and responsive data backup solutions to improve existing IT environments.
One of the key areas for improvement could be the use of more advanced multi-target regression models. Specifically, we plan to employ a more complex multi-target XGBoost regressor (in One-Model-Per-Target or Vector Leaf mode) for predicting the target attributes "backup_priority" and "backup_type". One of its primary benefits is its efficiency and scalability, making it suitable for both small and large-scale datasets.
Future steps for the research could be: using data sets with a bigger amount of different files
to teach and train a model, including immunosensors [
Intelligent big data system based on scientific machine learning of cyber-physical systems of medical and biological processes. CEUR Workshop Proceedings, 2021, 2864, pp. 34–48. [15] Martsenyuk, V., Sverstiuk, A., Bahrii-Zaiats, O., Kłos-Witkowska, A. Qualitative and Quantitative Comparative Analysis of Results of Numerical Simulation of Cyber-Physical Biosensor Systems. CEUR Workshop Proceedings, 2022, 3309, pp. 134–149. [16] Martsenyuk V., Sverstiuk A., Klos-Witkowska L., Nataliia K., Bagriy-Zayats O., Zubenko I.
Numerical analysis of results simulation of cyber-physical biosensor systems (2019) CEUR Workshop Proceedings, 2516, pp. 149 – 164. [17] Martsenyuk V., Sverstiuk A., Bahrii-Zaiats O., Kłos-Witkowska A. Qualitative and Quantitative Comparative Analysis of Results of Numerical Simulation of Cyber-Physical Biosensor Systems (2022) CEUR Workshop Proceedings, 3309, pp. 134 – 149. [18] Trysnyuk V., Zozulia A., Lupenko S., Lytvynenko I., Sverstiuk A. Methods of rhythm-cardio signals processing based on a mathematical model in the form of a vector of stationary and stationary connected random sequences (2021) CEUR Workshop Proceedings, 3021, pp. 197 – 205. [19] V. MARTSENYUK; I. ANDRUSHCHAK; N. KOZODII; Y. KRAVCHYK; A. SVERSTIUK; Y.
PALANIZA. COMPARISON OF RESULTS OF NUMERICAL ANALYSIS OF SIMULATION OF CYBERPHYSICAL BIOSENSOR SYSTEMS (2023) Herald of Khmelnytskyi National University. Technical sciences, pp. 202 – 212. https://doi.org/10.31891/2307-5732-2023-3191-202-212