Recently, the amount of generated time series data has been increasing rapidly
in many areas such as healthcare, security, meteorology and others. However, it
is very rare that those time series are annotated. For this reason, unsupervised
machine learning techniques such as anomaly detection are often used with such
data. There exist many unsupervised algorithms for anomaly detection ranging
from simple statistical techniques such as moving average or ARIMA till complex
deep learning algorithms such as LSTM-autoencoder. For a nice overview of the
recent algorithms we refer to read [
Difficulties with the unsupervised approach are: defining an anomaly score to correctly represent how anomalous is the time series, and setting a threshold for that score to distinguish between normal and anomaly data. Supervised anomaly detection, on the other hand, needs an expensive involvement of a human expert. An additional problem with supervised anomaly detection is usually the occurrence of very low ratio of anomalies, yielding highly imbalanced data.
In this extended abstract, we propose an active learning extension for an
anomaly detector based on a LSTM-autoencoder. It performs active learning
using various classification algorithms and addresses data imbalance with
oversampling and under-sampling techniques. We are currently testing it on the
ECG5000 dataset from the UCR time series classification archive [
LSTM-autoencoder [
Although LSTM-autoencoder works well for time-series with complicated
patterns, setting the anomaly score threshold can be very complicated without
labelled data. Furthermore, with a higher ratio of anomalies present in the
training data, a simple setting of the threshold might produce a lot of false positives
and false negatives. Therefore, we incorporated active learning into building the
anomaly detector that uses the code of a previously trained LSTM autoencoder.
The most related research has been done by Pigmentel [
First, an LSTM autoencoder is trained on unlabelled data. An initial anomaly
detection threshold on reconstruction error distribution is selected. At the initial
value of the threshold, we decided to use the mean plus three times the standard
deviation of the reconstruction error as an initial threshold. Every sample with
the reconstruction error above the threshold is labelled as an anomaly and the
same number of samples below the threshold are labelled as normal.
Furthermore, instead of the original time-series data, we use their representation in the
code layer. This provides us an artificially created, balanced dataset and
converts the anomaly detection to a binary classification task. A classifier is trained
on the created dataset and an active learning loop starts. In each iteration, a
resampler is used to balance the new updated labelled dataset. The resampler can
be either an undersampling or oversampling algorithm. The classifier is fitted
using the resampled data. Uncertainty sampling (US) active learning framework
[
The proposed algorithm was evaluated on a benchmark time-series dataset with
electrocardiogram readings [
L = L ∪ hencoder(xi), 1i U = U \ xi i = i + 1 end // Add normal data samples into the labelled dataset for j = i to 2i do
L = L ∪ hencoder(xj), 0i
U = U \ xj end while stopping criterion is not met do
R = resampler(L) train binary classifier m on R // Find the most informative seqeuence from U and ask for label x∗ = argmaxx∈U φ(x) y∗ = query(x∗) L = L ∪ hx∗, y∗i
U = U \ x∗ end end of the last cell is copied and used as the input of the first cell of the decoder. The decoder has architecture mirrored.
We experimented with 5 classifiers: logistic regression, decision tree,
Gaussian naive Bayes classifier, k-nearest neighbours classifier and support vector
machines. We experimented with 11 under-sampling and 4 over-sampling
techniques taken from the imbalanced-learn python toolbox [
Figure 1 shows how actively asking for annotations can improve the
unsupervised anomaly detection with an LSTM-autoencoder (red dashed line). The
best results were achieved with repeated edited nearest neighbours [
Moreover, a classifier model fed by labels outperformed the LSTM-autoencoder with the initially set anomaly score threshold (red dashed line) and the best possible anomaly score threshold (green dashed line). The source code is available in GitHub repository 3. 4
We presented an active Learning for LSTM-autoencoder-based anomaly detection for time-series data. An experiment on the ECG5000 data set has shown that the proposed method is able to boost the performance of the model significantly with only approximately 200 labelled samples. We plan next to experiment with variational LSTM-autoencoders and to pay attention to the interpretability of the detected anomalies .
The work has been supported by the grant 18-18080S of the Czech Science Foundation (GACˇ R). 3 https://github.com/tsabata/active_anomaly_detection model_type = SVM model_type = KNN model_type = LogReg
Fig. 1: F1 score performance metric in active learning loop. The figure contains 5 models (support vector machines (SVM), k-nearest neighbours (KNN), logistic regression(LogReg), decision tree (DT) and Gaussian naive Bayes(GNB)) and 9 resampling techniques. The greyed area represents standard deviation. Red dashed line represents the performance of the LSTM-autoencoder anomaly detector without active learning and initial setting of the anomaly score threshold. Green dashed line represent the performance of the LSTM-autoencoder anomaly detector without active learning and the best setting of the anomaly score threshold, and blue dashed line represents the best-achieved performance in the last iteration of active learning loop.