When dealing with sensor's data, it's important to keep track of what it's really happening in the tracked environments since failures, interruptions and misreadings must be expected at any time. Especially with logging processes involving extremely voluminous reports, an automatic method to detect entries that are not following the normal distribution of data (i.e. anomalies) should be the ideal solution. In the presented work the task performed by the autoencoder is to generate a reproduction error, used as metric for the classification of a sample in one of two classes: anomalous or non-anomalous.
not.
Ideally, the anomalies are a minimal part of a dataset When dealing with sensor’s data, it’s important to keep with generally low probabilities to be drawn from the track of what it’s really happening in the tracked envi- distribution describing the set: this scarcity implies a ronments since failures, interruptions and misreadings big reproduction error from the autoencoder. Moreover, must be expected at any time. Especially with logging the more a gradient descent is performed over a set of processes involving extremely voluminous reports, an inputs, the more the loss should decrease (until it hits its automatic method to detect entries that are not following minimum), vice versa, if a datum is not common, the loss the normal distribution of data (i.e. anomalies) should be is greater with respect to other well known data. the ideal solution. This method is shown to work over a real life, unlabelled Neural Networks can be used in this type of task as de- dataset, posing the problem in the unsupervised learning tectors for the distance of the sample from the natural landscape. distribution underlying the dataset.
In particular, autoencoders [
This type of networks has currently been used
successfully for image denoising [
In this work, the reproduction error (i.e. the error between the input sample and the output of the autoencoder) over a set of sample is exploited to discriminate which samples are anomalous in the given set and which Anomalies detection tasks have been already studied and solved with neural networks exploiting the reproduction error: the diference between a generic sample and a reconstruction of itself performed by some mathematical model.
In [
Similarly, in [
The GAN generates an adversarially learned set of
features used to project the high-dimensional original space
of the dataset into a reduced one. The reduced
representations are then decoded and the reproduction error is
used as an anomaly indicator.
More similar to the approach proposed in this work but
extremely more advanced, [
In the presented work the task performed by the autoencoder is to generate a reproduction error, used as metric for the classification of a sample in one of two classes: anomalous or non-anomalous.
The training procedure is the standard training procedure used for generic neural networks. A sample is drawn from the dataset, sampled from the distribution (). The latent vector (i.e. the output of the bottleneck layer) ℎ is generated by the encoder part of the autoencoder, generating ℎ = (). This representation, at the end of the training, can be used as an approximated reduced representation of the original input. Finally, ℎ is used as input for the decoder , generating ′ = (ℎ), belonging to the same space of . In this case, the L1 error between and ′ is minimized to train the parameters of the network. It’s crucial that the goal of the training is to guarantee the minimum diference between and ′.
An autoencoder is made up by two functions: an encoder and a decoder. The goal of the encoding function is to The key concept of the method is in the notion of scarcity: map (i.e. to encode) the input into a diferent space. Sym- an anomaly, to be defined as such, needs to be a notewormetrically, the decoder must map the encoded vector thy event. In practical cases, it could be an unpredictable back to the original input, without losing information. spike in a time series, a set of burned pixels in a photo, Conveniently, the encoding space is often chosen smaller the saturation of a sensor, an unexpected down time of a in dimensions than the original space, making the au- link and so on. toencoder performing also a dimensionality reduction. It’s worth nothing to say that, when sampling an element More formally, given an encoding function () and a from a distribution, it’s far more probable to sample a decoding function (), (()) must return back to normal entry instead of an anomalous case (if not, other the original . methods must be used or some problems could be in the The two mentioned above functions can be approximated dataset). With these premises, the autoencoder capacby symmetric neural networks, solving the following op- ity to capture a distribution and projecting it in another timization problem: space is exploited: during the training phase, multiple epochs are performed over all the samples in dataset, meaning that the network will experience a gradient min ‖ − (())‖2 descent over the loss generated by the same samples , multiple times but without modifying the ratio between where and are the parameters of the respective non-anomalous samples and anomalous ones. Moreover, neural networks. the reproduction error will be minimized over the most prominent distribution in the dataset, generating larger 3.2. Architecture errors in the anomalous subsets.
As shown in Figure 1, the scarcity is the fundamental The only constraint that need to be taken in consideration parameter that separates a normal sample from an anomais the presence of a bottleneck, a layer smaller than all lous one. When inverting the contamination proportion, the other layers in the network, essential for the dimen- the originally good samples results in a loss distributed sionality reduction. Without a bottleneck, the network higher than the respective anomalous one. This example is not "forced" to ignore useless or non-representative is purely demonstrative of the analysis made on the loss: features in the input, losing the capacity of mapping the a real-life dataset has a percentage of anomalies sensibly input in a denser space. lower than the ones used here.
The networks’ architectures are completely adaptable The last thing needed to define an anomaly is the loss to the problem. Normally, the input (e.g. time-series, threshold: after analyzing the distribution of the autoenimages) is embedded into a vectorial representation and coder’s reproduction loss on all the samples in the dataset, then reduced till the bottleneck. This representation is a threshold must be manually imposed, where a sample the projection of the input into a diferent space carried with a loss beyond the latter is considered anomalous. out by the encoder part. The latent vector (i.e. the output A compromise must be reached, since all the anomaof the bottleneck) is then passed to a generally symmet- lous samples must be included without including nonric network, returning a representation belonging to the anomalous samples. original input space.
The method is fast and with big enough datasets a single training can be performed to perform analysis on future, unseen events.
The preprocessing of the dataset is unavoidable: normalizing data and orders of magnitude is necessarily to have a good, consistent loss analysis. Samples bigger in module can importantly alter the distribution.
Finally, the model is fine-tuned (i.e. autoencoder architecture, bottleneck dimensions and so on) and a standard training procedure must be performed. The loss distribution is analyzed and the threshold is manually selected by the user.
Once new data arrive, there is no need to retrain the autoencoder: the sample is normalized and passed to the model, finally it’s classified according to the threshold.
It may happen that long-term changes in the dataset distribution could completely alter the outcome of the process, requiring a new training and threshold selection. 4.2. Preprocessing • VID: An integer representing a single car in an
unique way. • First Ping: A date in the format yyyy-dd-mm
representing the day of the session • TotalTime: Seconds (integer) elapsed from the ifrst connection to the last disconnection, same for all the sessions of the day. • start: timestamp representing the start of the
session. • end: timestamp representing the end of the ses
sion. • env: An integer representing the client owning
the car in an unique way. • Service: An integer representing the fleet of the
client, unique for the given client. multiple entries will show the multiple sessions.
The dataset is composed by information gathered from real car sensors.
Each car is assigned to a client that can have multiple lfeets and, every day that the car has been used, the total time of connection to the network is logged, saving the duration in seconds, the start date and the end date of the session. In case of disconnections during the same day,
The structure of the dataset could resemble a time-series in which, eventually, each day is made up by more entries for the same car. To tackle this complexity, a reduction to a single entry for each car in a given day is performed in such a way to get a single point for each pair of (VID, First Ping) (First Ping is then dropped at the end of the preprocessing phase). However, to not lose any important information, some fields have been added to record what is implicit in the original dataset while VID, TotalTime, env and Service are kept the same. Since start and end are removed, TotalTime_OFF has been added to record the total time in which the car has been disconnected from the network. Moreover, n_disconnections is added to remember the number of sessions in a single day for the given car. The final attribute added is C_v_off_time, namely the coeficient of variation of the down-times between each session, an index of dispersion.
The coeficient is defined as: () = () () (1) where is a set of data, is the standard deviation of and is the mean. Practically, an higher means that the data are unbalanced, implying a big diference across each element in the set. For example, (10, 10, 10, 10, 10) = 0 meaning no dispersion in the data, while (35, 5, 5, 2, 3) = 1.4, evidencing a set of data more scattered around the mean.
The meaning given to the coeficient of variation can vary based on the needs: in some cases it’s better to have short disconnections than a long one since it could be easier to deduct missing locations or missing data, preferring therefore lower s. Contrary to the last sentence, a near 0 could also represent a set of long disconnection times followed by small ones: this value alone is not enough to get an idea of how a link is performing since it doesn’t contain information regarding the quantities in the set, only a normalized index of dispersion.
After the preprocessing, the dataset has been reduced from 573064 entries to 99386. As for the training process, the autoencoder is trained over TotalTime, TotalTime_OFF, C_v_off_time and n_disconnections normalized between 0 and 1. This choice is justified by the fact that the remaining variables are categorical one hence completely arbitrarily values used only for an identification purpose.
The simplicity of the reduced dataset permits the use
of a fully connected network. An hidden layer (for each
network) is enough to capture the dimensionality, ending
in a bidimensional bottleneck. The nonlinearity is
introduced by a ReLU function and a Sigmoid [
The training process is supervisioned by an early stopping mechanism, keeping the best model before reaching a situation of overfit. The performances are tracked by computing the loss over a set of unseen samples, extracted with a proportion of 30% from the original dataset. where 1803 anomalies are found, the 1.84% of the whole dataset.
As expected, the notion of anomaly in a connection to network domain is in direct correlation with the duration of the down-time. To better see the motivation of the overlapping peaks, a new type of plot can be introduced:
Figure 5 shows a blue peak of the distribution over the In this plot each entry is scattered on a plane, representmaximum value admissible (i.e 86400 seconds in a day), ing TotalTime on the x-axis and TotalTime_OFF on followed by a peak in the same position of the anomalies. the y-axis. An ideal entry has the maximum TotalTime The interesting part of the plot is where the anomalies and the minimum TotalTime_OFF, posing itself on the are distributed more than the dataset itself, hence in the rightmost lower corner of the plot. lower values of the x-axis. As expected, an important The yellow star represents the center of mass of the nonnumber of entries with a low total connection time is anomalous distribution (in blue), very near to the ideal classified as anomaly. point, while the green star represents the center of mass Symmetrically, the plot of the down-time represents a of the anomalies distribution. There are ∼ 80 blue similar situation: points while only ∼ 2 red ones, evidencing a big diference in the concentration.
It’s worth nothing to say that a diferent threshold would have moved the frontier of the two clusters up or down.
The reason of the overlapping peaks lies in the fact that a big number of anomalies is in the vertical line over the maximum of TotalTime (i.e. = 86400) and on the horizontal line over the minimum TotalTime_OFF (i.e. = 0), meaning that one variable is in a good range while the other one not. The worst anomalies are the ones lying near the center of the plot, containing a discrepancy in both the variables.
In the left plot there aren’t any notable results, the two distributions appear the same. The number of disconnections is evenly distributed over each entry in the dataset and in the anomalies.
When computing the distribution over the average disconnection time for each car, the histograms show an important diference: the peak of the anomalies is lower than the peak of the dataset, meaning that the anomalous entries have a lower number of disconnections. However this could cause some confusion since a larger number is expected when talking about this type of variable but analyzing the results paired with the ones obtained in the previous subsection, the lower number of disconnections reveal a longer down-time. This result is confirmed by the following plot where the anomalies distribution is slightly translated to the right, meaning a less uniform number of disconnections that can be caused by a longer disconnection time followed by a set of short times.
answering the practical question: "Are there bad cars or bad clients?".
Following the previous analysis, it’s possible to retrieve an overview on the presence of a single car in the anomalies by plotting the distribution of the VIDs. In Figure 11 (left), the anomalies are concentrated in the first part of the x-axis, showing a peak near 0. A set of car that appear only less than 10.0% in the dataset is responsible of the ∼ 30.0% of the anomalies. A zoomed version of the same plot can be seen on Figure 11 (right), where only the cars with a VID less than 2000 are showed. In a practical way, this result can help focusing more on the subset of car that is more present in the anomalies, helping saving time on the analysis.
From this result it’s possible to derive the conclusion on the final result, the client analysis. In Figure 10 the biggest percentage of anomalies is covered by the client number 1. The duality on the results can show that the most of the first 2000 cars are assigned to the first client, notion that may help with the understanding of the failures.
The method explained here is implemented in PyTorch
[
Finally, every evaluation can be made in real-time (after the training), with times that can vary according to the hardware and the architecture used. On an NVIDIA MX150, the evaluation over all the dataset takes approximately 20 seconds.
It’s possible that the model needs a retraining if the distributions in the dataset changes in an unexpected way (e.g. logging temperatures can require a retraining between summer and winter if the entries are not enough for each season).
The method presented in this paper has shown the ability to perform a classification of an unlabelled dataset. It’s a fast way to identify outliers in datasets or in a real-time data feed (after a first training over a big enough dataset composed by recorded logs).
Its flexibility can be a great incentive to its utilization, being tied only to the use of an autoencoder, architecture that can be expanded and customized at each eventuality. Moreover, the fact that the threshold must be imposed after the training can be exploited to increase or decrease the severity of the system in real-time, following changes in the needs of the user.
However, the manual choice of a loss threshold could be an element of imprecision, looking at the fact that a slightly alteration could extremely change the samples considered as anomalous. Another downside could be the needing of a big enough dataset, since with a little one there could be dificulties in learning the right distribution. However, this is a common problem among all the autoencoder applications.
Finally, the system is sensible to the dataset’s dimensions, requiring an adequate normalization that must be applied also to real-time samples.
In conclusion, as future work the method can be expanded to automatically detect the threshold, removing the manual component that could completely change the outcome of the process.