Introduction

Anomalies can be defined as examples that significantly deviate from the majority of the data to arise the suspect of being generated by a different mechanism. Anomaly Detection represents a fundamental task in many human activities, including Healthcare, Cyber-security, Industrial Monitoring, Fraud Detection, and many others.

It is possible to identify three different types of settings of Anomaly Detection [1]. In the Supervised setting a dataset whose items are labeled as normal and abnormal is available to build a classifier, typically the dataset is highly unbalanced and the anomalies form a rare class. The Semi-supervised setting, also called one-class, is characterized by the presence in input of only examples from the normal class that are used to train the detector. In the Unsupervised setting the goal is to assign an anomaly score to each object of the input dataset in order to find anomalies in it.

Classical data mining and machine learning algorithms performing the task of detecting outliers include statistical-based [2], distance-based [3,4,5,6], density-based [7,8], reverse nearest neighborbased [9,10,11], SVM-based [12,13], and many others [1].

1st Workshop on Green-Aware Artificial Intelligence, 23rd International Conference of the Italian Association for Artificial Intelligence (AIxIA 2024), November 25-28, 2024, Bolzano, Italy Recently, the approaches that have achieved the most success have been those based on deep learning [14], which can be divided into three main families: reconstruction error-based methods employing Autoencoders (AE), models based on Generative Adversarial Networks (GAN), and SVM-like neural architectures.

At the basis of the application of Autoencoders (AE) and Variational Autoencoders (VAE) [15,16,14] to Anomaly Detection relies the concept of reconstruction error. More in detail, (Variational) Autoencoders are trained to map data into a low dimensional latent space and then map them back into the original space generating in output a reconstruction of the input as similar as possible to it. Since the majority of the data used for training models belongs to the normal class, it is assumed that these networks are able to reconstruct the inliers better than the outliers and, thus, the reconstruction error can be adopted as an anomaly score.

GAN-based models [17,18,19,20] basically consist in the combined, adversarial training of two sub-architectures, the generator and the discriminator. Specifically, the generator network produces artificial anomalies as realistic as possible, and the discriminator assigns an anomaly score to each item.

SVM-like methods [21,22,23] leverage the idea of enclosing normal data into a hypersphere employing a One-Class SVM-like loss function combined with a deep neural architecture. A slightly different approach that can be included in this family, is introduced in [24] where the architecture presents an additional final layer composed of just one neuron that produces an anomaly score that, for anomalies, is as far as possible from a value obtained as the average of randomly sampled normal items anomaly scores.

Moreover, in [25] has been introduced Deep Isolation Forest (DIF), a novel methodology that utilizes casually initialized neural networks to map original data into random representation ensembles, where random axis-parallel cuts are subsequently applied to perform data partition.

Nevertheless, the cost of high power and energy combines with the high accuracy and training speed of the Deep Learning models. This is leading researchers to be aware of the environmental impact of deep neural architectures by trading off accuracy against energy consumption and also to perform characterization in terms of performance, power and energy for guiding the architecture design of DNN models [26,27,28,29].

This paper aims to provide a contribution in this direction, and, in particular, to the field of Anomaly Detection by analyzing the behaviour of recent methods from the point of view of the detection performance as well as from the point of view of their carbon footprint. Specifically, we focus on the Latent𝑂𝑢𝑡 algorithm [30,31,32,33], an anomaly detection framework that applies to any deep neural architecture as a baseline to obtain a refined score, and we compare it with the baseline architecture on which it is applied and deep learning-based competitors from the other families.

The Latent𝑂𝑢𝑡 algorithm for Unsupervised Anomaly Detection

Due to the quite good performances they obtained as well as their versatility, the ones based on (Variational) Autoencoders have become the most widespread Anomaly Detection approaches relying on Deep Neural Networks.

The main issue about them is that they often generalize so well to reconstruct also anomalies [30], thus worsening the capability of detecting anomalies of the reconstruction error.

In [31] Latent𝑂𝑢𝑡 is introduced. It is a methodology that enhances both the reconstruction error and the latent space distribution of the Variational Autoencoder in order to obtain a refined anomaly score. Specifically, the first variant of the Latent𝑂𝑢𝑡 (Figure 1) algorithm considers the enlarged feature space 𝐹 = 𝐿 × 𝐸, where 𝐿 represents the latent space and 𝐸 is the reconstruction error space (usually 𝐸 ⊆ ℝ), and performs a 𝑘-NN density estimation in the space 𝐹.

In Figure 1 the complete workflow of Latent𝑂𝑢𝑡 is showed. Each point of the dataset 𝑥 ∈ 𝑋 is mapped into the latent space 𝐿 of the VAE (blue points represent inliers, red ones represent anomalies) by means of the encoder 𝜙 𝑊 and then reconstructed back in the original space x ∈ 𝑋 by means of the decoder 𝜓 𝑊 . Then, the reconstruction error 𝐸(𝑥) = ‖𝑥 − x ‖ 2 2 is computed, the feature space 𝐹 = 𝐿 × 𝐸 is created, and the 𝑘-NN density estimation is performed in it to compute the Latent𝑂𝑢𝑡 anomaly score. The motivation behind this procedure is based on the observation that anomalies tend to lie in the sparsest regions of the augmented feature space 𝐹. This happens because even when their reconstruction error is not exceptionally large, is still significantly larger than that of their most similar normal items.

In [32] Latent𝑂𝑢𝑡 has been expanded in order to be potentially applied to any neural architecture that has three fundamental properties:

• it outputs an anomaly score,

• it has a latent space 𝐿, • it performs a mapping from the original data space 𝑋 to 𝐿 through an encoder-shaped module.

In particular, the neural models on which Latent𝑂𝑢𝑡 has actually been tested are AE, VAE, GANomaly, Fast-AnoGAN, SO − GAAL, and MO − GAAL.

Moreover, in [33] it has been showed that the separation properties of the enlarged space 𝐹 allow any generic anomaly score (not only the 𝑘-NN) to perform better when applied on it than on the input data space 𝑋.

Experimental results

Experimental setup

In our experiments we consider the tabular datasets cardio, letter, lympho, mammography, pendigits, pima, satellite, satimage-2, speech, thyroid, from the ODDS repository [34] as well as the image datasets MNIST [35], Fashion-MNIST [36], and CIFAR10 [37].

The last three datasets (differently from the ones from the ODDS repository) are multi-class, thus to make them suitable for the anomaly detection task we adopt a one-vs-all strategy, meaning that we consider one class as normal and we randomly sample 𝑠 items from each other class. If not otherwise stated, we set 𝑠 = 10. Specifically, we select the class "0" as normal for the MNIST dataset, the class "Sandal" for Fashion-MNIST, and the class "deer" for CIFAR-10.

As for the implementation details of the algorithm, we consider the original version of Latent𝑂𝑢𝑡 with the VAE as baseline architecture, and the 𝑘-NN with 𝑘 = 50 as estimator of the density of the feature space 𝐹. The latent space dimension ℓ of the VAE is set to ℓ = 2 for tabular ODDS datasets and to ℓ = 32 for image datasets. As for the encoder structure (the decoder is symmetric to it) we adopt the same strategy used in [33], i. e. we insert hidden layers of dimension ℓ 𝑖 = ⌊ 𝑑 4 𝑖 ⌋ between the input 𝑑-dimensional space and the ℓ-dimensional latent space for each 𝑖 ∈ ℕ + such that ⌊ 𝑑 4 𝑖 ⌋ > ℓ. The 𝐶𝑂 2 emissions are estimated by means of the Python library CodeCarbon [38] which bases its tracking on the power consumption and the geographic location where the code is executed.

Evolution of performance and emissions of Latent𝑂𝑢𝑡 and VAE during training

The energy consumption of any Deep Learning model is related to the training phase, and, in particular, to the number of training epochs.

Therefore, it is of crucial importance to understand the behavior of these algorithms as the training proceeds to optimize the trade-off between the maximization of the performance and the minimization of energy consumption.

The quantity of 𝐶𝑂 2 produced by Latent𝑂𝑢𝑡, which we represent as ℰ Latent𝑂𝑢𝑡 , is fundamentally constituted by two terms:

• the emissions ℰ 𝑉 𝐴𝐸 needed for the training of the architecture and the computation, which is shared with the Variational Autoencoder, • the emissions ℰ 𝑘-NN used for the building of the feature space ℱ and the computation of the 𝑘-NN algorithm in it.

Since the two operations are carried out in sequence and independently of each other, we have that

ℰ Latent𝑂𝑢𝑡 = ℰ 𝑉 𝐴𝐸 + ℰ 𝑘-NN

which means that, with equal training epochs, the carbon footprint of Latent𝑂𝑢𝑡 is always greater than the one of the Variational Autoencoder. Thus, for a fair comparison, we train the Variational Autoencoder for 100 epochs and we stop the training earlier for evaluating the Latent𝑂𝑢𝑡 score. In figures 2, 3, 4, we show the performances of both Latent𝑂𝑢𝑡 (in orange) and the standard Variational Autoencoder (in blue) in terms of Area Under the ROC Curve (AUC) as the training proceeds. Observe that on the horizontal axis is reported the 𝐶𝑂 2 emissions (in 𝐾 𝑔), which means that, for the reasons stated above, each value of the AUC of Latent𝑂𝑢𝑡 is obtained with fewer epochs than the relative value of the VAE.

As we can see, in almost every plot the curve of Latent𝑂𝑢𝑡 is placed above the curve of the VAE. Moreover, the trend of Latent𝑂𝑢𝑡 is much more regular than the one of the VAE (see in particular the plots of the datasets cardio, mammography, satellite, satimage-2, mnist, cifar). This implies that if we fix a threshold on the amount of 𝐶𝑂 2 we want to emit, the score of Latent𝑂𝑢𝑡 always outperforms the standard score of the VAE. In other words, Latent𝑂𝑢𝑡 is able to better exploit the emissions produced than the standard architecture on which it is applied. This happens because as the training proceeds the reconstruction capabilities of the VAE improve so much that at some point it becomes able to reconstruct also outliers, thus lowering the anomaly detection performances of the model. On the other side Latent𝑂𝑢𝑡 benefits of the latent space organization that produces a progressively better separation between normal examples and anomalies in the feature space 𝐹.

Comparison with competitors

We consider as competitors some of the neural networks algorithm implemented in the Python library PyOD [39], namely Deep-SVDD [21], from the SVM-like family, AnoGAN [17] and ALAD [20], from the GAN family, and DIF [25]. For the implementation details (number of layers and neurons, training epochs, learning rate, potential hyperparameters), we refer to the default values fixed in PyOD. As for Latent𝑂𝑢𝑡, we consider again the setup described in section 3.1 and we perform a few-epochs training, due to the good convergence properties observed in the last section. Specifically, the VAE is trained for 15 epochs.

As evaluation metrics we adopt the standard Area Under the ROC Curve (AUC) and the ratio

𝐶𝑂 2 𝐴𝑈 𝐶

between the emissions of 𝐶𝑂 2 (in 𝐾 𝑔) produced for the training and the inference of a model, and the AUC. This last value is a measure combining both performance and energy consumption, indeed it indicates how much 𝐶𝑂 2 is needed (on average) to obtain a single percentage point of AUC. Table 1 shows the results in terms of AUC. As we can see, Latent𝑂𝑢𝑡 is the best method for half the datasets, achieving performances close to the best also in the other half. In particular, confirming the observation made in [31], Latent𝑂𝑢𝑡 is especially effective on higher dimensional, structured data (for example speech and the image datasets). In Table 2 are shown the results of the experiment in terms of the ratio 𝐶𝑂 2 𝐴𝑈 𝐶 . Here, Latent𝑂𝑢𝑡 outperforms its competitors in all but one dataset, exhibiting the best trade-off between performances obtained and the emissions of 𝐶𝑂 2 produced.

Conclusion

In this paper, we have focused on the algorithm Latent𝑂𝑢𝑡 for unsupervised anomaly detection in order to evaluate its performances and measure the environmental impact of its executions. When compared to the standard architecture on which it is applied, i. e. the Variational Autoencoder, Latent𝑂𝑢𝑡 shows that low energy-consumptive training can lead it to conspicuously better results. Moreover, in comparison with other neural network-based anomaly detection approaches it has shown superior performances both in terms of absolute AUC and, most importantly, in terms of the ratio between the emitted 𝐶𝑂 2 and the AUC obtained.

As future development, we intend to expand the discussion about the environmental impact of Latent𝑂𝑢𝑡 by including a more profound analysis of all its several variants and an investigation specialized on the hardware type (e.g., CPU vs. GPU), as well as propose novel measures to better capture the trade-off between emissions and performances. Finally, as a more ambitious goal, we aim at introducing a mechanism enabling Latent𝑂𝑢𝑡 to consider the green-aware aspect at training time.