With the advent of Industry 4.0, predictive maintenance techniques have largely spread throughout companies. However, it is still di cult to understand how to implement a predictive maintenance strategy to get satisfactory results. In this research we propose a methodology to de ne a benchmark in terms of performance of machine learning algorithms in the context of prognostic predictive maintenance from a classi cation perspective. In de ning such a benchmark we use three target datasets publicly available over which to compare di erent preprocessing and feature engineering techniques and di erent machine learning algorithms and auto learning tools. Our benchmark shows that it is possible, by following the guidelines delineated in this paper, to select the proper combination of preprocessing, feature engineering and algorithms/tools to get an average F1-score of 98%. Moreover, we propose an innovative approach based on siamese neural networks that shows comparable results with respect to the benchmark de ned, thus showing that also this kind of algorithm has to be tested to be sure to reach the best possible results.
Thanks to the advent of Industry 4.0 and the enhancements in machine learning techniques, in recent years predictive maintenance (PdM) applications have largely spread throughout companies. Since PdM is an active area of research, there are thousands of papers published on the topic, however among all the possibilities of implementing a PdM system, it is still di cult to identify a speci c strategy to get satisfactory results. On the shake of this, this research aims at de ning a technical benchmark to measure the performance of di erent fault prognosis algorithms and auto-learning tools in the context of prognostic PdM. More precisely, this research is driven by the business need of a partner company that produces vertical cutting machines and aims at creating a PdM system to predict breakage events, thus reducing related costs by avoiding them. One of the main issues of the partner company is that they have scarse connectivity throughout the production line, so they cannot stream data in real-time to the nal system that will be cloud-based. In this context, the nal goal is to produce a system capable of monitoring in semi-real-time through sensors some operating parameters of the machines in order to be able to predict their remaining useful life (RUL). Generally, the RUL is a continuous variable that requires a regression problem to predict it. However, the semi-real-time scenario poses the issue of interacting with the machine at discrete time intervals sending a batch of the collected data every hour. Thus, since in the real case scenario we will get data in batch with a certain time delay, in this research we focus on the context of PdM from the classi cation point of view, i.e. we aim at predicting the class of breakage of the last observations collected. Thus, we designed a methodology to compute a benchmark for prognostic PdM based on the main state-of-the-art machine learning algorithms and the available auto learning tools to provide the partner company with a precise methodology to select the best algorithm and determine how it performs with respect to the benchmark. Moreover we compare the results with an innovative approach based on Siamese Neural Network (SNN), which is an algorithm never used in this context. Computing this benchmark will allow the partner company to de ne the best PdM strategy by following the guidelines de ned in this paper and to check whether the results they are obtaining by implementing the PdM system are competitive with the state-of-the-art results. The rest of this paper is organized as follows. In chapter 2 a comprehensive analysis of the state of the art for PdM is presented. Then in chapter 3 the methodology for the creation of the benchmark is de ned together with the innovative approach based on SNN. In this chapter also the dataset used for the computation are described. In chapter 4 the results obtained are shown with a focus on the comparison of the benchmark with the innovative approach and, nally, in chapter 5 some conclusive remarks. 2
Predictive maintenance (PdM) [
The goal of this research is to de ne a benchmark in terms of performance of the
data driven PdM algorithms for fault prediction and compare the results with
an innovative approach based on SNNs. As explained in Chapter 1, given that
in the partner company's scenario data are send to the PdM model in batch
with a certain delay, we focus on the case of multi-class classi cation, where the
aim is that of determining, given a time series of variables, the breakage class.
For the de nition of the benchmark, not only custom algorithms are used, but
also some auto-learning tools such as Google Cloud AutoML [
Three di erent public datasets have been identi ed for the de nition of the
benchmark.
Zenodo predictive maintenance dataset The dataset published by
Zenodo [
NASA's Prognostic CoE [
XJTU-SY Bearing Datasets The XJTU-SY dataset [
In order to de ne a precise benchmark, we decided not only to compare different machine learning algorithms but also di erent preprocessing techniques and di erent feature engineering approaches. In this case, three distinct feature engineering modes have been de ned that share a common preprocessing phase. Preprocessing Firstly data is normalized and the features that have zero variance are eliminated. The target variable is then de ned as RUL, i.e. number of cycles missing from the fault. Since the benchmark to be determined refers to a multi-class classi cation problem we need to convert the RUL from a continuous variable to a discrete variable. Thus, we de ne a methodology to divide the dataset into 3 classes: one containing the cases of RUL between 0 and N, one containing the cases of RUL between N and M with M>N and one with the cases of RUL greater than M. In this way the three classes represent a case of failure in the short term (RUL<N), a case of failure in the medium term (N<RUL<M) and a case of good functioning (RUL>M). The method of selecting the parameters N and M depends both on the use case, i.e. on how long the cycles last and in how many cycles on average a failure case occurs, and on the performance of the models, i.e. on the accuracy of the models in the short, medium and long term. Starting from a given N (the minimum number of cycles that ensure the operator room to maneuver), a series of machine learning models is trained for larger values of N. For each model trained, the performance are calculated and the performance trend is studied as N varies. The idea is to select N as large as possible (to ensure that the prediction occurs in time, guaranteeing the operator room to maneuver) but with the aim of maintaining good performance. A limit deviation of 5% from the maximum performance value is therefore considered. Once parameter N has been selected, M is selected in the same way. Feature Cycle (FC) The rst feature engineering approach consists in using the preprocessed dataset and in creating for each feature and for each pair of features the corresponding second degree feature crosses (for example, given the features x and y, a dataset containing x; y is considered, xy; x2y2). This is a standard feature engineering technique that has to be tested in order to nd the best solution possible. Once the features are generated, since some datasets contain many sensors and the size of the features may explode, the features are reduced through Principal Component Analysis (PCA) , in order to represent the features through an embedding vector. The vector is constructed by taking only the k principal components that describe at least 95% of the variance present in the data.
Feature Rolling (FR) A second method of feature engineering consists in using the preprocessed dataset enriched with the calculation of the second degree feature crosses as described above, then adding a level of temporal aggregation to also take into account the overall trend of the series. In this case, for each detection xi the sequence of the t values preceding xi; (xi t; :::; xi) is taken and it is replaced with the average of the values of the sequence of length t. This allows to engineer historical information and include them in the features, letting some algorithms (which otherwise would not be able to consider historical information) take these historical correlation into account. To determine the optimal value of t, an approach based on the analysis of the performance of the models generated by considering di erent values of t is adopted. In this case we start from a minimum value of t = 2 and proceed by increasing t by 1. For each increment, a classi er is trained and the performance are measured. The choice of t must be made taking into consideration the peak point of the performance trend, but always considering that too large t implies the need to have a certain number of measurements before being able to make the prediction, therefore we take into account the average life cycle of a machine and select t as the minimum value between the peak point and 13 of the maximum number of life cycles. Also in this case follows the reduction of the dimensionality based on PCA and on the principle of 95% of the variance explained.
Feature Rolling Enriched (FRE) The third and last approach of feature engineering starts from the pre-processed database enriched with second degree feature crosses over which FR is performed as described above but, for each batch the features are enriched by calculating statistical indices as the mean value, the median, the minimum, the maximum, the skew, the standard deviation and the kurtosis index. This allows to enrich the dataset with statistically signi cant features that could help in modeling particular relations between variables. Also in this case follows the reduction of the dimensionality based on PCA and on the principle of 95% of the variance explained. 3.3
Once the datasets have been prepared, di erent machine learning algorithms are
trained for each of the datasets described above which, according to the state of
the art, are widely used. Speci cally, the algorithms chosen for the creation of
the benchmark are: Logistic Regression (LR) as baseline algorithm, RF, ANN
and KNN since they are the main used algorithms in this context according to
the state of the art (from which we excluded SVM due to their scalability issues
with large datasets) and LSTM classi er which, among the DL techniques, is
the most consolidated in sequence learning problems [
Once all the algorithms and tools have been trained, the benchmark is computed by averaging the maximum performance obtained in terms of F1 score for each algorithm and for each feature engineering technique applied. More formally, given a set of datasets D : fd1; :::; dDg, algorithms A : fa1; :::; aAg and feature d engineering techniques F : ff1; :::; fF g and considered F 1a;f as the F1-score associated to the ath algorithm, the f th feature engineering and the dth dataset, the benchmark is computed as
PD max d=1 fa2A;f2F g
F 1da;f jDj (1) Thus, this benchmark represents an average result which can be pursued by applying the correct feature engineering technique and the correct algorithm to a speci c dataset.
Guidelines for benchmark computation and evaluation To summarize, given a set of dataset D, to compute the benchmark the following steps need to be performed for each dataset: { De ne the minimum number of cycles N to let the operator intervene. { Preprocess the data and de ne the optimal value of N and M with the methodology de ned in paragraph 3.2-Preprocessing. { Over the preprocessed data run the feature engineering technique de ned as FC, FR and FRE, de ning the optimal parameter t for the FR feature engineering technique with the methodology described in 3.2-Feature Rolling (FC). { Divide the data in train and test with an 80/20 split and train the LR, RF, ANN and KNN algorithms with the training data using Google Cloud vertex AI leveraging the HyperTune algorithm to run cross validation and hyperparameters tuning and Google auto-learning tools that automatically perform all the optimizations. { Test each algorithm trained with the test data and measure the F1-score. { Compute the benchmark as described in paragraph 3.4 by considering the best F1-score for each algorithm trained and feature engineering technique applied.
The benchmark thus obtained represents an average a ordable result. Moreover, by following the same steps over a real dataset, it is possible also to identify the best algorithm for the speci c case and compare the performance over that dataset with respect to a prede ned benchmark. 3.5
In addition to the state-of-the-art algorithms we decided to study the
performance of an innovative approach based on SNN [
Bx2n;1 x2n;T C Xn = @BB ... . . . ... ACC ; 8n 2 f1; : : : ; N g xnF;1 xnF;T (2)
For each feature f and for each time instant t, the centroid k of the class k with the components fk;t are computed as
0 11 1T 1 k = B@ ... . . . ... CA ; fk;t =
F F 1 T 1
X jCkj Xj2Ck xjf;t; 8k 2 f1; : : : ; Kg (3)
Starting from the centroids of each class, we propose to use an approach based on KNN to get the S multivariate time series closest to each centroid and use these as comparison samples to detect the nal class by majority voting. In this way we assure to use as a comparison sample just the series that are more descriptive of each class. The size of S will be determined on the basis of performance, starting from a minimum value of 1, which correspond to an n-way one shot learning, and increasing incrementally. The drivers for the choice of S will be both the performance of the models and the prediction times: since the system must be industrialized it will be necessary that these times remain relatively low, approximately in the order of seconds. Furthermore, since this is a classi cation model, the same metrics used to evaluate the other models that contribute to the de nition of the benchmark are used to evaluate and compare the performance of this innovative approach with the benchmark. 4
In the following the experimental results obtained are shown. In particular, rstly the results obtained in running the methodology for N, M and t selection, as explained in Paragraph 3.2. Then, the actual results of the di erent models and tools trained to compute a benchmark and a comparison between the benchmark and the SNN algorithm are presented. 4.1
The rst step for the de nition of the benchmark is that related to the preprocessing and feature engineering step. It is, indeed, important to de ne the parameters N, M and t as described in section 3.2-Preprocessing, in order to get the best results by keeping the model useful from a business perspective (i.e., the parameters N and M do not have to be too small, otherwise the operator does not have time to stop the machine and avoid breakage event and, similarly, the t parameter does not have to be too large, otherwise it is necessary to have a lot of values in the past to be able to perform a prediction). In order to determine the parameters N and M we train several RF classi ers rstly for subsequent values of N and then, once N has been de ned, of M, keeping N xed. In Figure 1 the results for di erent values of N and M for the Aircraft dataset are shown. Clearly, the selection of N and M varies according to the dataset used. As an example, for the XJTU-SY dataset and the OML dataset, the breakage events happen after thousands of cycles, thus N and M are of the order of 50 to 100 thousands, while for the Aircraft dataset the breakage events happen after hundreds of cycles, thus N and M are of the order of 10 to 100 cycles. After looking at the results, and considering a limit deviation of 5% from the maximum performance value to keep N and M as large as possible, the selected parameters are de ned in Table 1.
Analogous considerations hold for the selection of t. A RF classi er has been trained for subsequent values of t, starting from t=2. Also in this case, the frequency of the breakage events impact the selection of t. In Figure 2 it is shown the trend of the scores register for the Aircraft dataset varying t. In Table 1 we can see the best selected t considering a limit deviation of 5% from the maximum performance value to keep t as small as possible. The selected parameters are those used to train the di erent models and tools.
To train and test each model and tools we applied an 80-20 split of training and test data, performing a strati ed sampling. All the models have been trained on Google Cloud Vertex AI, exploiting the hyperparameters optimization module. As far as the auto learning tools are concerned, they automatically perform the hyperparameters optimization. In order to compare the results and evaluate the performance of each algorithm, we used the F1 score. In Table 2 a comparison of the results obtained is shown. By analyzing the results we can see that generally the performance over the OML dataset are better than the other and the ones over the XJTU-SY dataset are worse. On average, the RF algorithm reaches the best performance over all datasets (with all the feature engineering techniques applied), with an 85.5% of F1 score. Moreover, we can state that the FRE feature engineering technique, in most cases extremely improves the results of the algorithms. This can be seen especially in the case of Aircraft and XJTUSY, where the performance over the other feature engineering techniques are de nitely worse, but also in the case of OML, even though the performance are good also in the other cases. Finally, by considering the best results for each dataset over all the feature engineering techniques and over all the algorithms tested, on average we can state that it is possible to reach a 98% of F1-score, with a 1.7% of standard deviation. This is the benchmark computed as in Equation 1, which describes the target result that one can achieves by properly selecting the algorithm and the feature engineering technique for his own speci c dataset. 4.3
In order to de ne the performance of the SNN algorithm, rstly we need to de ne the optimal number S of comparison series. To de ne the optimal number S, we follow the methodology proposed in Paragraph 3.5. We compute di erent F1 score based on a varying number of comparison series from 1 to 15. We compare the results over each feature engineering and select the smallest S corresponding to the best result. The best results obtained are S = 4 for FC, S = 5 for FR and S = 5 for FRE. Given the benchmark de ned previously, we can compare the results of the SNN based algorithms. In Table 2 the results, in terms of F1 score are shown. To compute the predictions, we get the top S series closest to the centroid, de ned as in Equation 3. To get the top S series we use the approach based on KNN described in Paragraph 3.5. As we can see, also in this case, the best results are obtained with the FRE feature engineering technique. Very good results are obtained for the Aircraft and the OML dataset, while acceptable results are obtained for the XJTU-SY dataset. Computing the average of the best results also for this algorithm, we have an average F1-score of 95.3%. This is slightly under the benchmark previously de ned, however results are comparable, meaning that also this kind of algorithm can be adopted in the context of PdM. 5
In this research we present a methodology for the de nition of a benchmark in terms of performance in the context of prognostic predictive maintenance from the classi cation perspective. In de ning the benchmark we consider di erent preprocessing and feature engineering techniques and di erent machine learning algorithms and auto-learning tools and we compute the benchmark over di erent public datasets. We also de ne some approaches to automate parameters selection that contribute to reach the best performance. In conclusion, we show that, despite the input dataset, it is possible to select the proper feature engineering technique and the proper machine learning algorithm or tool to reach an average F1-score of 98%. Moreover we test an innovative approach based on SNN and we show that it is competitive with the benchmark computed. To enhance the research, it could be interesting to expand the de nition of the benchmark also to real datasets, to understand whether the results obtained with public dataset (some of which are synthetic) can be compared with the results obtained with real dataset.