Autoencoders have emerged as powerful tools for unsupervised representation learning, finding applications in various domains such as computer vision, natural language processing, and anomaly detection. More generally, the latent space reconstruction mechanism can be extended to the reconstruction of any type of data. This extended abstract presents an idea of representation learning applicable to data with an initial instant described by a baseline (input data), and an instant in the future referable to a follow-up (output data). The novel approach consists of combining a construction-focused loss with a classification-driven loss. The proposed hybrid autoencoder architecture aims to simultaneously enhance data reconstruction while learning discriminative features for classification tasks. Initial experimental results demonstrate the eficacy of the proposed hybrid autoencoder on a long-covid dataset.
CEUR ceur-ws.org
Traditional data representation techniques often rely on manual feature engineering, which
is labor-intensive, domain-specific, and might miss out on intricate patterns present within
the data. Autoencoders [
The autoencoder architecture includes two basic modules: encoder and decoder. In the following, we will present these two components.
Encoder: this module allows the compression of the initial features space into the latent
representation. In our setting, the encoder is realized by a fully connected layer that maps the
input space into a lower dimensional latent representation. The encoder takes as input a sample
∈ ℝ and outputs a representation ℎ ∈ ℝ ℎ , where ≫ ℎ . During the encoding phase,
we also introduce non-linearity by applying a non-linear activation function immediately after
the fully connected layer. Finally, to reduce overfitting and improve the model generalization
we also adopt a dropout layer [
Decoder: module that reconstructs the output data starting from the latent representation. In most settings, this module is designed to complement the encoder, aiming to reconstruct the input sample from the latent representation ℎ. However, in our experiments, we chose to expand the output dimension to include the features of the patients at follow-up time = + . In this configuration, the encoder needs to map the input features to a latent representation that not only compresses the data but also retains suficient information for accurate reconstruction of the follow-up details during the decoding phase. As a result, the model can potentially learn patterns that span across the hospitalization and follow-up time. Lastly, to address the task of classifying patients with or without long-covid syndrome, we incorporate a classification head into the model.
During the learning process, the model uses a loss function comprised of two separate components. The first part ℒ is responsible for the reconstruction part of the learning task. In particular, we resorted to the Mean Squared Error (MSE) loss that calculates the distance between the reconstructed data and the training samples. The second component denoted by ℒ is used for the classification of long-covid syndrome, using the Binary Cross Entropy loss (BCE). We chose to mix these two losses by introducing two coeficients, denoted as and , enabling us to seamlessly balance between a reconstruction and classification regimen. The complete loss function is described as follows:
ℒ = ℒ + ℒ
During the experiments, we applied the model to a real-world dataset. In the subsequent sections, we will start by giving an overview of the long-covid scenario, presenting dataset statistics, and finally, we will discuss the model configuration and performances.
Following the characterization in [
Concerning patient characterization, baseline data indicate 38 features describing the demographic and medical history of the patient, while hospitalization data (14 features) refer to the patient’s symptoms at hospitalization (acute COVID-19 onset). Baseline data are not directly related to COVID-19 infection but are important factors to take into account in order to make an accurate diagnosis or prediction. Features in the baseline data can be grouped in terms of demographic characteristics (sex, age, smoking attitude, ...) and of prior comorbidities (obesity, chronic liver disease, hypertension, anxiety and depression, ...).
Hospitalization data include the patient’s symptoms at COVID-19 onset (fever, cough, dyspnea, arthralgia, ...), drugs administered (hydroxychloroquine, monoclonal antibodies, glucocorticoids, antivirals, ...), and hospitalization information (duration, oxygen administration, ICU intubation, ..). Baseline and hospitalization jointly form the input space.
The follow-up data (27 features) contains among others the same symptoms present in the hospitalization but at a diferent instant (one year in the future). The follow-up set combined with the input space forms the output space.
The original dataset consisted of 324 entries, representing a very limited data scenario for a
deep learning architecture. To augment our dataset with additional samples, we employed the
Synthetic Minority Over-sampling Technique (SMOTE) [
We conducted a series of tests on the aforementioned dataset to assess whether the latent space constructed by our model could establish a correlation between hospitalization and follow-up while simultaneously maintaining discriminative capabilities for identifying cases of long-covid. Detailed hyperparameters employed during the model training are provided in Table 1. To address the limitations posed by the dataset size, we adopted a shallow dimension for the latent space and relatively high dropout probability. This decision is justified by the dataset’s size, as maintaining a compact representation enhances the model’s generalization capacity when there are limited examples available.
The average accuracy achieved in our experiments is 71% ± 0.9. We applied PCA to the latent space embeddings varying the parameter of the loss, which governs the classification contribution to the loss. As illustrated in Figure 1, it is clear that as the value of alpha increases, the explained variance of the embeddings also rises. This suggests that the clusters become progressively more linearly separable. The result is especially promising considering the scarcity of data and the complexity of the multi-class classification problem.
The dual loss framework enables the creation of a latent space that takes into account the
presence of at least one of the symptoms as a discriminator. This capability is made possible by
the end-to-end architecture of our model. When visualizing the latent space in two dimensions
using t-distributed stochastic neighbor embedding (t-SNE) [
Hidden dimension Latent size Reconstruction Loss Learning rate Dropout
128 16 1 0.7 MSE 0.001 0.5
In conclusion, this paper has introduced a hybrid autoencoder architecture that leverages both construction-focused and classification-driven loss functions to enhance unsupervised representation learning. Autoencoders have already proven their utility in various domains, and this work extends their applicability to data with a temporal aspect. By incorporating both reconstruction and discriminative feature learning objectives, our approach aims to provide a comprehensive solution for a wide range of tasks.
Our initial experiments on a long-covid dataset have yielded promising results, demonstrating the efectiveness of the proposed hybrid autoencoder in capturing meaningful representations from sequential data. These findings pave the way for future research in utilizing autoencoders for time-dependent data and underline the potential impact of this approach in addressing complex real-world problems.
M. Dossena and C. Irwin are supported by the National PhD program in Artificial Intelligence for Healthcare and Life Sciences (Campus Bio-medico University of Rome). We want to thank A. Chiocchetti and M. Bellan for having provided us with the long-COVID data and for several fruitful discussions about the case study.
We want to thank our tutors Luigi Portinale, Annalisa chiocchetti, Luca Piovesan and Stafania Montani for their support in our PhD journey.
This work has been supported by the “Piano Riparti Piemonte”, Azione n. 173 “INFRA-P.
Realizzazione, raforzamento e ampliamento Infrastrutture di ricerca pubbliche–bando”
INFRAP2-TECHNOMED-HUB n. 378-48 [