Analytics based on sensor data is gradually becoming an industry standard in equipment maintenance. However, it involves several challenges, such as sensor data preprocessing, feature engineering and forecasting model development. Due to work in progress, this paper is mainly focused on sensor data preprocessing, which plays a crucial role in predictive maintenance due to the fact, that real-world sensing equipment usually provides data with missing values and a considerable amount of noise. Obviously, poor data quality can render practically useless all the following steps of data analysis. Thus, many missing data imputation, outlier filtering, and noise reduction algorithms were introduced in the literature. Streaming sensor data can be represented in a form of univariate time series. This paper provides an overview of common univariate time series preprocessing steps and the most appropriate methods, with consideration of the field of application. Sensor data from different sources comes in different scales and should be normalized. Thus, the comparison of univariate time series normalization techniques is given. Conventional algorithm quality metrics for each of the preprocessing steps are described. Basic sensor data quality assessment approach is suggested. Moreover, the architecture of a sensor data preprocessing module is proposed. The overview of time series-specific feature engineering techniques is given. The brief enumeration of considered forecasting approaches is provided.
Maintenance costs are a major part of the total operating
costs of any business involving complex equipment.
Conducted surveys of maintenance management
effectiveness indicate that one-third of all maintenance
costs is wasted as the result of unnecessary or improperly
carried out maintenance [
First of all, often collected data is of poor quality,
which can lead to unreliable analysis and ineffective
maintenance. Consequently, data from sensing
equipment needs to be preprocessed before it can be used
for any analysis. Poor data quality means
noncompliance with requirements on at least one of the data
quality assessment metrics. The root of problems can
vary: connection issues, sensor malfunction, transmitting
hardware failure, data processing server downtime,
software crash, measuring equipment inaccuracy and
many more. Common cases of poor data quality involve
unacceptable amount of missing values, outliers, sudden
spikes etc. Simply ignoring these issues can be critical
due to several reasons. For example, some analysis tools,
including popular machine learning algorithms, can’t
handle missing values. The absence of outlier filtration
can dramatically skew the results. Measuring equipment
standard error can be mistaken for an actual pattern in
data. As a result, time series preprocessing involves
several independent steps: missing data imputation,
noise reduction, and data normalization. After these
steps, data can be evaluated in quality and passed further
for analysis. It is clear, that preprocessing should be done
in near real-time to minimize the delay between data
measurement and decision making. Thus, there is a need
for a fast and scalable independent module, that can
preprocess constantly incoming sensor data. This paper
proposes the design of such a module, keeping in mind
the following integration with the existing architecture of
a predictive maintenance system, introduced in [
Secondly, it can be difficult to distinguish the patterns and relationships in the initial data. The process of extracting and generating new characteristics and features out of the available data, commonly referred to as feature engineering, has two main objectives. The first one is to represent the data in such a form, that will make it easier to establish simple yet strong connections between the input and the output variables for the forecasting model, increasing the quality of the forecasts. The second objective is to pick the most useful features out of all the available ones, reducing the amount of computations of the forecasting model.
Finally, a proper forecasting model is to be chosen and implemented. There are various approaches to time series forecasting, from straightforward methods like naive method to way more sophisticated ones like longshort term recurrent neural network. The main complication here is the trade-off between the forecast quality and the ease of model implementation and deployment.
The remaining part of the paper is organized as follows. The preprocessing module architecture is described in Section 2. Section 3 reviews missing data imputation methods. Section 4 is devoted to time series noise reduction. Section 5 briefly overviews data normalization techniques. In section 6, some thoughts on data quality assessment are combined. Section 7 is devoted to time series feature engineering. The brief overview of time series forecasting approaches is given in section 8. Finally, the future directions of presented work are given in section 9.
The preprocessing module is a part of the system for
predictive maintenance, deployed to a Hadoop [
There are many stream data processing frameworks
including but not limited to Apache Storm [
First of all, Spark Streaming is a mature framework with
thorough documentation and huge community. As a
result of long-term popularity, there are plenty of
opensource tools for Spark Streaming, including solutions for
relatively painless integration with Kafka and mentioned
earlier database management systems [
One of the biggest downsides of Spark Streaming is performance degradation on sudden bursts of input data. However, in case of sensor data processing the input data flow intensity remains nearly the same at all time intervals, which mitigates the downside.
The data flow and module components are introduced below in figure 1. The whole module consists of 4 transformation steps and the data quality assessment step.
Sometimes due to a sensor malfunction, unstable internet connection or other technical difficulties the data for some points in time is missing. Simply ignoring those gaps may be not the best strategy, because it can lead to a loss of efficiency and unreliable results of the analysis. Another approach is to try to impute the missing values based on the available information.
The detailed overview of basic imputation methods and
their implementations can be found in imputeTS R
package documentation [
Some simple methods that are applicable not only to time series: median imputation, mode imputation, mean imputation, random imputation. These methods are fast and very straight-forward, but lack accuracy.
Simple time series specific methods include LOCF (last observation carried forward),
NOCB (next observation carried backward), interpolation (linear, polynomial, Stineman) and moving average (simple, weighted, exponential). All of them are rather fast and can work in specific cases, but fall off when there is seasonality in the data or large missing sub-sequences.
More sophisticated approaches like Structural Model &
However,
sensor
data
has
one
unfortunate
characteristic – the gaps of missing data can be too long
for conventional methods to work properly. In this case,
the method proposed in [
Missing data imputation involves 2 types of quality metrics based on the pattern of imputation.
For single value imputations, the metrics coincide with the ones commonly used in time series forecasting – RMSE (Root Mean Square Error)and MAPE (Mean Absolute Percentage Error).
= =
100% ∑ ( ^ − )2,
× ∑ − ^ , where yi is real value, ŷi is the forecasted value and n is the number of forecasts.
However, different metrics are used for long gap imputation. The most popular of them are similarity and Dynamic Time Warping distance.
1 = × 1 +
DTW calculation algorithm can be found at [
Similar to missing data points, sensor data is usually contaminated with noise, which can be mistaken for actual data pattern, which yet again leads to a loss of efficiency and unreliable results of the analysis. The task of noise reduction is to subtract the maximum amount of noise from the initial data, leaving the maximum amount of useful signal.
According to Chen et al. [
Frequency domain approaches are based on signal decomposition into frequency components. The most common approaches involve discrete/fast/short-time Fourier transform either wavelet transform.
Most of the time domain approaches are based on smoothing the signal of each given data point based on the values of its neighbors.
The comparison of the basic noise reduction methods
can be found in the work of Köhler et al. [
simple nonlinear noise reduction in different conditions.
The downside of the approaches listed above is that
they modify almost all the data values, most of which are
initially correct. Song et al. [
However, this method can be used only for large
outlier detection. In some cases, even small errors can be
important and repairing only spike errors is insufficient.
Zhang et al. [
effective and computationally efficient heuristics are also introduced in this work.
Most of the papers involve RMSE, defined earlier, as a denoising quality metric. However, there are several less popular ones, including the Symmetrical Visual Error
Making sure that your data is of uniform scale is key for many
methods, including k-NN, linear models, artificial neural networks and many more.
univariate time series data should be normalized because it might be further used in combination with data of different scale from other sources.
The most well-known and widely used are min-max normalization and z-score normalization.
Min-max implies that you know the minimum and the maximum values in your dataset beforehand, which is often not the case. Z-score is more robust but performs poorly om nonstationary time series.
ŷ = ŷ−
− ( ) =
( )− ( ) − ( ), ( ) where ŷ is a value after normalization, y is a value prior normalization and Y is the set of values being normalized.
Some less popular methods are decimal scaling
normalization, which holds all the drawbacks of
minmax normalization, sigmoid normalization, which is
actively used in neural networks and tanh estimators,
which roughly can be described as a hyperbolic tangent
of the z-score normalization.
Data Quality Assessment (DQA) is the scientific and
statistical evaluation of data to determine if data obtained
from environmental data operations are of the right type,
quality, and quantity to support their intended use [
There is a comprehensive work on time series data
quality assessment done in [
The assessment is to be done for both data prior and after preprocessing to acquire an evaluation of preprocessing module effectiveness. It is also worth keeping in mind, that initially clean data is different to the data, that was made “clean” during preprocessing due to approximations and inevitable errors of the methods involved on each step.
Feature engineering is, probably, the most peculiar step of data processing, as it depends on the initial data type, its origin, quantity, quality, the desired output of the forecasting model and even the nature of the model itself. As it was already mentioned, sensor data can be represented in a form of univariate time series. The conventional approaches to time series feature engineering can be divided into 3 categories: timestamp features, statistical features, and spectral features. The feature extraction step is usually followed by a dimensionality reduction step.
It is worth mentioning, that there are automatic time
series feature engineering tools such as tsfresh [
The idea of this approach is to extract the features from the timestamp of each observation. The most commonly used features are: • minutes elapsed for the day; • hour of the day; • day of the month; • weekend or not; • season of the year; • public holiday or not.
Talking about sensor data, some examples of useful timestamp features are: • time since the last maintenance; • age of equipment; • time since the last failure; • operating time of equipment.
Using just these features alone for predictions will likely result in a poor model. However, in combination with other features, they can boost the quality of forecasts.
This approach involves sliding through a time series with the window of a given width and calculating statistics for each iteration. The most common statistical features are the mean of the previous few values, the median, the mode, the minimal value, the maximum value, the standard deviation and many more. In addition to calculated statistics, we can also use the lagged values of a time series as features.
The biggest challenge of this approach is that the window can be of any width and there is no general algorithm to choose it. Usually, the researchers just try out several values of the width and choose the one that performs best. However, if there is a seasonal pattern in data, it is worth making the width of the window not less than the period of the seasons.
Different variations of Fourier transform and wavelet transform are used to extract spectral features from a nonstationary signal. The basic idea behind those methods is to decompose a given time series into a sum of several basic functions, providing a different representation of the initial signal. The biggest drawback of those methods is that they are relatively computationally expensive.
Feature extraction provides many features, some of which can be useless or strongly correlated with each other. Excessive features not only add unnecessary computations but also can decrease the quality of the model. Thus, several dimensionality reduction methods were introduced to minimize the number of features, at the same time keeping the maximum amount of information. The most common ones are principal component analysis, independent component analysis and partial least squares regression.
There is a variety of methods, that can be used for time series forecasting. Each of them has advantages and drawbacks and can be viable in certain circumstances.
First of all, there are some basic methods such as average method, naive method, seasonal naive method, and drift method, which are very simple yet can be effective when the data pattern is easy.
Secondly, linear regression models can be used for forecasting. In the simplest case, the regression model allows for a linear relationship between the forecast variable and some predictor variables. The biggest downside is inherent linearity, while real-world data is mostly non-linear.
The most common approach is to use stochastic models – ARMA, ARIMA, SARIMAX, etc. One of the biggest drawbacks of those models is that they require fine-tuning of several hyperparameters, which is computationally expensive and not intuitive.
One of the recently popular approaches is to use decision trees. Random forests and gradient boosting methods, which are so widely used in machine learning competitions, can also be used for time series forecasting.
Artificial neural networks approach for time series forecasting gained immense popularity in last few years. Their modifications – recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) are especially effective for this task due to their “memory” component. Although neural networks tend to be the most accurate forecasting method when tuned properly and given enough data, they might be computationally too costly for the considered conditions.
In this study, the overview of sensor data preprocessing steps, methods, and common metrics is held. Some thoughts on sensor data quality assessment are given. The architecture of a fast, scalable preprocessing module is proposed. A brief overview of time series feature engineering techniques and forecasting methods is given. The future goals of the ongoing work are to implement the designed preprocessing module on Spark Streaming engine, integrate it into the existing predictive maintenance pipeline, implement the feature engineering step and to develop a remaining lifetime forecasting model.
Acknowledgments. This work is supervised by Dmitriy Kovalev, Institute of Informatics Problems, Federal Research Center “Computer Science and Control” of the Russian Academy of Sciences.
The research is financially supported by Ministry of Education and Science of the Russian Federation (project’s unique identifier RFMEFI60717X0176).