environmental impact. Because of their astonishing performances, Deep Neural Network-based approaches have become pervasive in many human activities. However, they often require a long, energy-intensive training phase, which has a huge In recent years, there has been a significant increase in the emphasis placed on environmental themes across various sectors, driven by growing concerns over climate change and sustainability. This heightened focus has led to many initiatives, policies and discussions aimed at addressing ecological challenges and promoting a more sustainable future. For the reasons stated above, Deep Learning cannot be exempted from such initiatives and the literature is starting to pay attention to these issues. This paper aims at contributing to this field, in particular, concerning the Anomaly Detection Task whose environmental impact, due to its widespread employment, deserves to be addressed.
with other Anomaly Detection Neural Network-based methods and we highlight that it is the one that obtains the best results in terms of a balance between high accuracy performance and low carbon footprint.
Anomalies can be defined as examples that significantly deviate from the majority of the data to arise the suspect of being generated by a diferent mechanism. Anomaly Detection represents a fundamental task in many human activities, including Healthcare, Cyber-security, Industrial Monitoring, Fraud Detection, and many others.
It is possible to identify three diferent types of settings of Anomaly Detection [
Classical data mining and machine learning algorithms performing the task of detecting outliers
include statistical-based [
CEUR
ceur-ws.org
Recently, the approaches that have achieved the most success have been those based on deep learning
[
At the basis of the application of Autoencoders (AE) and Variational Autoencoders (VAE) [
GAN-based models [
SVM-like methods [
Moreover, in [25] has been introduced Deep Isolation Forest (DIF), a novel methodology that utilizes casually initialized neural networks to map original data into random representation ensembles, where random axis-parallel cuts are subsequently applied to perform data partition.
Nevertheless, the cost of high power and energy combines with the high accuracy and training speed of the Deep Learning models. This is leading researchers to be aware of the environmental impact of deep neural architectures by trading of accuracy against energy consumption and also to perform characterization in terms of performance, power and energy for guiding the architecture design of DNN models [26, 27, 28, 29].
This paper aims to provide a contribution in this direction, and, in particular, to the field of Anomaly Detection by analyzing the behaviour of recent methods from the point of view of the detection performance as well as from the point of view of their carbon footprint. Specifically, we focus on the Latent algorithm [30, 31, 32, 33], an anomaly detection framework that applies to any deep neural architecture as a baseline to obtain a refined score, and we compare it with the baseline architecture on which it is applied and deep learning-based competitors from the other families. 2. The Latent
algorithm for Unsupervised Anomaly Detection Due to the quite good performances they obtained as well as their versatility, the ones based on (Variational) Autoencoders have become the most widespread Anomaly Detection approaches relying on Deep Neural Networks.
The main issue about them is that they often generalize so well to reconstruct also anomalies [ 30], thus worsening the capability of detecting anomalies of the reconstruction error.
In [31] Latent is introduced. It is a methodology that enhances both the reconstruction error and the latent space distribution of the Variational Autoencoder in order to obtain a refined anomaly score. Specifically, the first variant of the Latent (Figure 1) algorithm considers the enlarged feature space = × , where represents the latent space and is the reconstruction error space (usually ⊆ ℝ ), and performs a -NN density estimation in the space .
In Figure 1 the complete workflow of Latent is showed. Each point of the dataset ∈ is mapped into the latent space of the VAE (blue points represent inliers, red ones represent anomalies) by means of the encoder and then reconstructed back in the original space ∈̂ by means of the decoder . Then, the reconstruction error () = ‖ − ‖̂ 22 is computed, the feature space = × is created, and the -NN density estimation is performed in it to compute the Latent anomaly score.
The motivation behind this procedure is based on the observation that anomalies tend to lie in the sparsest regions of the augmented feature space . This happens because even when their reconstruction error is not exceptionally large, is still significantly larger than that of their most similar normal items.
In [32] Latent has been expanded in order to be potentially applied to any neural architecture that has three fundamental properties: • it outputs an anomaly score, • it has a latent space , • it performs a mapping from the original data space to through an encoder-shaped module. In particular, the neural models on which Latent has actually been tested are AE, VAE, GANomaly, Fast–AnoGAN, SO − GAAL, and MO − GAAL.
Moreover, in [33] it has been showed that the separation properties of the enlarged space allow any generic anomaly score (not only the -NN) to perform better when applied on it than on the input data space .
In our experiments we consider the tabular datasets cardio, letter, lympho, mammography, pendigits, pima, satellite, satimage-2, speech, thyroid, from the ODDS repository [34] as well as the image datasets MNIST [35], Fashion-MNIST [36], and CIFAR10 [37].
The last three datasets (diferently from the ones from the ODDS repository) are multi-class, thus to make them suitable for the anomaly detection task we adopt a one-vs-all strategy, meaning that we consider one class as normal and we randomly sample items from each other class. If not otherwise stated, we set = 10 . Specifically, we select the class “ 0” as normal for the MNIST dataset, the class “Sandal” for Fashion-MNIST, and the class “deer ” for CIFAR-10.
As for the implementation details of the algorithm, we consider the original version of Latent with the VAE as baseline architecture, and the -NN with = 50 as estimator of the density of the feature space . The latent space dimension ℓ of the VAE is set to ℓ = 2 for tabular ODDS datasets and to ℓ = 32 for image datasets. As for the encoder structure (the decoder is symmetric to it) we adopt the same strategy used in [33], i. e. we insert hidden layers of dimension ℓ = ⌊ 4 ⌋ between the input -dimensional space and the ℓ-dimensional latent space for each ∈ ℕ + such that ⌊ 4 ⌋ > ℓ.
The 2 emissions are estimated by means of the Python library CodeCarbon [38] which bases its tracking on the power consumption and the geographic location where the code is executed. 3.2. Evolution of performance and emissions of Latent and VAE during training The energy consumption of any Deep Learning model is related to the training phase, and, in particular, to the number of training epochs.
Therefore, it is of crucial importance to understand the behavior of these algorithms as the training proceeds to optimize the trade-of between the maximization of the performance and the minimization of energy consumption.
The quantity of 2 produced by Latent , which we represent as ℰLatent , is fundamentally constituted by two terms: • the emissions ℰ needed for the training of the architecture and the computation, which is shared with the Variational Autoencoder, • the emissions ℰ -NN used for the building of the feature space ℱ and the computation of the -NN algorithm in it.
Since the two operations are carried out in sequence and independently of each other, we have that ℰLatent = ℰ + ℰ -NN which means that, with equal training epochs, the carbon footprint of Latent is always greater than the one of the Variational Autoencoder. Thus, for a fair comparison, we train the Variational Autoencoder for 100 epochs and we stop the training earlier for evaluating the Latent score.
In figures 2, 3, 4, we show the performances of both Latent (in orange) and the standard Variational Autoencoder (in blue) in terms of Area Under the ROC Curve (AUC) as the training proceeds. Observe that on the horizontal axis is reported the 2 emissions (in ), which means that, for the reasons stated above, each value of the AUC of Latent is obtained with fewer epochs than the relative value of the VAE.
As we can see, in almost every plot the curve of Latent is placed above the curve of the VAE. Moreover, the trend of Latent is much more regular than the one of the VAE (see in particular the plots of the datasets cardio, mammography, satellite, satimage-2, mnist, cifar ). This implies that if we ifx a threshold on the amount of 2 we want to emit, the score of Latent always outperforms the standard score of the VAE. In other words, Latent is able to better exploit the emissions produced than the standard architecture on which it is applied.
This happens because as the training proceeds the reconstruction capabilities of the VAE improve so much that at some point it becomes able to reconstruct also outliers, thus lowering the anomaly detection performances of the model. On the other side Latent benefits of the latent space organization that produces a progressively better separation between normal examples and anomalies in the feature space .
We consider as competitors some of the neural networks algorithm implemented in the Python library
PyOD [39], namely Deep-SVDD [
As evaluation metrics we adopt the standard Area Under the ROC Curve (AUC) and the ratio 2 between the emissions of 2 (in ) produced for the training and the inference of a model, and the AUC. This last value is a measure combining both performance and energy consumption, indeed it indicates how much 2 is needed (on average) to obtain a single percentage point of AUC.
Table 1 shows the results in terms of AUC. As we can see, Latent is the best method for half the datasets, achieving performances close to the best also in the other half. In particular, confirming the observation made in [31], Latent is especially efective on higher dimensional, structured data (for example speech and the image datasets). In Table 2 are shown the results of the experiment in terms of the ratio 2 . Here, Latent outperforms its competitors in all but one dataset, exhibiting the best trade-of between performances obtained and the emissions of 2 produced.
In this paper, we have focused on the algorithm Latent for unsupervised anomaly detection in order to evaluate its performances and measure the environmental impact of its executions. When compared to the standard architecture on which it is applied, i. e. the Variational Autoencoder, Latent shows that low energy-consumptive training can lead it to conspicuously better results. Moreover, in comparison with other neural network-based anomaly detection approaches it has shown superior performances both in terms of absolute AUC and, most importantly, in terms of the ratio between the emitted 2 and the AUC obtained.
As future development, we intend to expand the discussion about the environmental impact of Latent by including a more profound analysis of all its several variants and an investigation specialized on the hardware type (e.g., CPU vs. GPU), as well as propose novel measures to better capture the trade-of between emissions and performances. Finally, as a more ambitious goal, we aim at introducing a mechanism enabling Latent to consider the green-aware aspect at training time.
We acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), Spoke 9 Green-aware AI, under the NRRP MUR program funded by the NextGenerationEU.