The paper presents the results submitted by Team SCaLAR-NITK for task 3 of eRisk Lab at CLEF 2024 [1]. The dataset provided by the task organizers consisted of 74 subjects for training and 18 for testing. We begin by describing the data cleaning and preprocessing steps. Subsequently, we outline various approaches used to address the problem, such as Word2Vec, TF-IDF, Backtranslation and Dimensionality Reduction, among others. Finally, we summarize the results obtained from each approach. Our solutions demonstrated strong performance, achieving the best results in 7 out of the 8 evaluated metrics.
Eating disorders, such as anorexia nervosa, bulimia nervosa, and binge eating disorder, are serious
mental health conditions characterized by abnormal eating habits and distorted body image [
In this paper, we discuss methodologies for detecting signs of eating disorders from users’ Reddit posts. The first approach involves creating separate models for each of the 22 questions, with each model learning the distribution specific to its corresponding question. The second approach reshapes the dataset to include the text of the question in a separate column, allowing the use of a single model for both training and predicting, rather than maintaining 22 individual models. The third approach employs Principal Component Analysis (PCA) for dimensionality reduction, ensuring that the importance of the text related to the question is weighted at a ratio of 3:1. These methodologies aim to efectively identify patterns and indicators of eating disorders in user’s reddit posts.
The dataset was given in the form of XML files which had to be cleaned for further processing. The XML files consisted of the user names, the posts they had posted and their timestamps. The answers given by the subjects (true labels) were given as a separate text file.
Several XML had loading issues and were incorrectly formatted. So as a first step, these issues were resolved and the XML file for each user was converted to CSV files for further cleaning and usage. It was also found that column names had a mismatch and was fixed.
Further cleaning involved the removal of emojis from the posts of users using the emoji library. It was also found that several posts were enclosed with starting with b" and ending with ". These were removed accordingly. Unicode representations were replaced with their actual representations. For example, \xe2\x80\x99 means ’ and instances such as these were found and replaced.
After cleaning, all posts from a single user were concatenated into one single chunk and preprocessed using standard preprocessing methods. This text was lowercased and URLs were removed. All kinds of punctuation were removed. Stop words were removed so that the machine learning models are not heavily influenced by the efects of these words. Finally, the words were lemmatized using the WordNetLemmatizer function from the nltk.stem library.
Three diferent approaches are proposed in our solution to the problem. They are described and explained below.
This approach consisted of fitting a model to each question. Since there were 22 questions answered
by each subject, we made 22 diferent models to learn the distribution of each question’s answers.
Thereafter, the Linear Support Vector Machine algorithm was used for classification [
The Linear Support Vector Machine model implemented here utilizes stochastic gradient descent with a hinge loss function for maximum-margin classification. It applies L2 regularization to prevent overfitting and employs hyperparameters for regularization strength, reproducibility, the maximum number of iterations and the tolerance of stopping criteria. It is particularly suited for large datasets and online learning scenarios.
For word embeddings for input to the models, a pipeline was constructed. The first component was the CountVectorizer, which converts the collection of text documents into a matrix of token counts. Then, the standard TF-IDF approach is taken and the corresponding matrix is constructed. These embeddings are taken for each document and fed into the machine learning algorithms and neural network for each question.
This approach demonstrated good results, and got the best score in 2 of the measuring metrics. The reason as to why we went with the simpler approach is because complex models would overfit the data, hence resulting in poorer scores, as observed in the results of the same task last year. Clearly, the approach paid of with good results.
Although a good approach with decent results, the importance of the questions was not given as inputs to the models. It was just a case of learning 22 arbitrary distributions and then predicting the same. This emerged as a drawback as the questions were not given any importance in the predictions. Also, since the training set was small, there was a chance that the training was not suficient for predicting the validation labels correctly.
To overcome the drawback of the previous approach, the dataset was reshaped to include the text of the question in a separate column. This also includes the size of the dataset and makes use of a single model for training and predicting, rather than 22 models as in the previous approach. A pictorial representation of the transformation is shown in Figure 1.
After the transformation, word embeddings for both text and question was constructed using
Word2Vec. Pretrained Word2Vec models, ofer precomputed word embeddings trained on vast amounts
of text data, such as Google News articles as used in our work. The loaded model, pre-trained on
Google News corpus (3 billion running words) word vector model (3 million 300-dimension English
word vectors), captures intricate semantic nuances and gives numerical meaning to text [
Separate word embeddings are generated for the text and question. Each of these are 300 dimensional vectors, and were concatenated to form the final embedding vector which was used for training which turned out to be a 600 dimensional vector. By this way, we ensured that both the text and question are given equal weightage during prediction of labels. An illustration of our methodology is shown in Figure 2.
The imbalance in training data was taken care of by Backtranslation. Backtranslation is the process
of translating text from one language to another and then translating it back to the original language
[
Our implementation handles cases where the input text may exceed character limits specified by GoogleTranslate API (4999 characters) by splitting it into smaller chunks. It utilizes the Google Translator library to perform translation tasks eficiently.
Backtranslation was performed for only the text column and only for those instances with labels 2,3,4 and 5. These were then added to the dataset as new instances, which increased the weightage of the minority classes. This can be pictorially seen in Figure 5.
Using these new embeddings - Linear Support Vector Machine and Gradient Boosting algorithms were used to train the data.
The Gradient Boosting model utilized in this implementation is a powerful ensemble learning
method widely used for classification tasks. It works by sequentially adding weak learners, typically
decision trees, to correct the errors made by preceding models. Each subsequent learner focuses on
the residual errors of the previous model, gradually improving the overall predictive performance [
An innovative method to reduce the weightage of the question embeddings is proposed using dimensionality reduction. Principle Component Analysis (PCA) was used for the purpose of dimensionality reduction.
PCA (Principal Component Analysis) is a dimensionality reduction technique that transforms
highdimensional data into a lower-dimensional representation while preserving most of its variance. It
achieves this by identifying the principal components, which are orthogonal directions in the original
feature space that capture the maximum variance in the data [
This ensured that the importance of text to question is in the ratio of 3:1. The approach is illustrated in Figure 6.
The results of our endeavours are shown in Table 1. For this task, 5 teams took part and 14 diferent solutions or runs were submitted, along with 3 baseline solutions described by the authors. Our solutions performed exceedingly well and we had the best results in 7 out of the 8 given metrics.
Citing previous year results, complex deep learning based solutions like transformers did not give great results - most likely because of the overfitting problem. Hence we decided to go with a simpler machine learning based approach to tackle the problem. It can be inferred that when the size of the dataset is small, simpler solutions ofer better results than complex Deep Learning based solutions.
This paper outlines Team SCaLAR-NITK’s involvement in Task 3 of the eRisk@CLEF 2024 edition, where we investigated diverse techniques for assessing eating disorders across multiple users based on their Reddit contributions. Our investigation focused on two primary approaches to tackle the problem. The first approach employed 22 distinct models, each tailored to a specific question, while the second approach utilized a single model capable of capturing the essence of both the questions and the posts. Innovative approaches such as Backtranslation was used to balance the label distribution and Principle Component Analysis (PCA) was used to better assign the weightages of text and question. Our contributions aim to address complex societal challenges in mental health detection and intervention.