This study is a contribution of the MentalRiskES 2024 task 3 which explores the detection of suicidal ideation using a range of machine learning models encompassing both traditional algorithms and advanced Transformer models. We assess several approaches including Logistic Regression, Naïve Bayes, Random Forest, and variants of the RoBERTuito model, focusing particularly on the adaptation of the RoBERTuito base model for text analysis and sentiment analysis in Spanish. Results indicate that the RoBERTuito base model outperforms other models in accuracy and performance metrics, demonstrating its potential for critical mental health applications in Spanish-speaking contexts.
Mental health and wellness are integral to overall well-being. Mental health research is vital as it
influences our emotional, psychological, and social functioning. Mental illnesses, often perceived
as personal and isolating struggles, are also significant public health concerns. Society must treat
mental disorders with the same urgency and care as chronic medical conditions, recognizing their
high treatability and the potential for recovery in many individuals [
Suicidal ideation involves thoughts of self-harm or ending one’s life and is a critical mental health
challenge with potentially fatal outcomes if unaddressed. This issue afects a diverse range of individuals
and is often linked to mental health disorders, though it can also arise from situational stressors. Early
recognition of suicidal ideation and understanding its root causes are essential for improving patient
outcomes and reducing suicide risks [
The expression and detection of suicidal intent can be facilitated through the analysis of social media
posts using natural language processing (NLP) and machine learning (ML) algorithms [
CEUR
ceur-ws.org
While some studies have begun to analyze mental health based on gender and social status, attention to diferent ethnicities remains limited. Notably, research on suicidal ideation among Hispanic populations is scarce.
This study aims to address this gap by evaluating various ML models in NLP, including traditional classifiers like Logistic Regression, Naïve Bayes, and Random Forest, as well as advanced Transformerbased models such as RoBERTuito, which are specially designed to understand the nuances of mental health discourse in Spanish. The efectiveness of RoBERTuito in detecting suicidal thoughts in social media texts underscores its potential in public health strategies for suicide prevention, highlighting the need for culturally relevant mental health interventions within Spanish-speaking communities.
For the development of this analysis, the task organizers provided only the test dataset for Task 3; the
same procedure was followed in previous corpus [
Given these requirements, the training dataset for Task 1 (Disorder detection) for the MentalRiskES
[
Publicly available data was taken from the published tables from studies on the recognition of suicidal
intent among depressed populations consisting of 20 social media messages in English language [
To adapt the datasets found in the previously mentioned literature, social media content was translated
from its original version in English to Spanish using neural machine translation tools, specifically Google
Translate and DeepL. These tools were selected based on their proven efectiveness and accessibility as
documented in previous studies [
Model training and evaluation involved splitting the data into training and testing sets using train_test_split. The model was trained on a sub-portion (80%) of the collected training set and evaluated on another sub-portion (20%) of the collected training set that was called the testing set.
The analysis was conducted using Python 3.9.6. Below are the primary libraries that were utilized for traditional classifiers such as logistic regression, random forest, Naïve Bayes, and SVC models: • General Libraries: Pandas, NumPy, scikit-learn (version 0.24.1), Nltk, Genism, SciPy (version 1.6.0), os, re, json. • Specific Tools for Text Feature Extraction: TfidfVectorizer, CountVectorizer, Word2Vec (via gensim).
• Miscellaneous: TensorFlow, Keras, transformers, datasets, accelerate.
For the logistic regression model, text data were preprocessed using TfidfVectorizer and CountVectorizer. For the implementation of the Naive Bayes classification model, the Multinomial Naive Bayes algorithm provided by the scikit-learn library in Python was utilized. For the random forest model, text data were converted into numerical vectors using the word2vec-google-news-300 model from gensim, focusing on semantic content. Stopwords were removed from the text data in the random forest model using the nltk library, which provides a comprehensive list of stopwords for various languages. For the LSTM classifier, the TensorFlow and Keras frameworks were employed to construct and train the model.
To address the complexities of language and sentiment analysis inherent in this task, a transition to utilizing transformer-based models was done.
For the classification tasks involving RoBERTa (Robustly Optimized BERT Pretraining Approach), we utilized the transformers library. RoBERTa, an optimized version of BERT, enhances pretraining techniques to achieve more robust performance across a wider range of tasks. The components utilized in this model setup included: • RoBERTa: RobertaTokenizer, TFRobertaForSequenceClassification. • RoBERTuito sentiment analysis and RoBERTuito base pretrained models: AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments. 2.3.1. Model Implementation Multinomial Naive Bayes: This model was used for the classification with discrete features (word counts for text classification). We trained this model with and without smoothing (alpha parameter). The efectiveness of the model was evaluated using accuracy, precision, recall, and f1-score for the standard version and area under the curve (AUC) for the version with smoothing.
Logistic Regression, Random Forest, and SVC: These models were applied after transforming the trained dataset by TfidfVectorizer. For the SVC model, the impact of document length as an additional feature was explored by combining it with the tf-idf matrix using hstack. Both models were evaluated based on the AUC metric, which provides an aggregate measure of performance across all classification thresholds. Additional features like document length and digit count to the feature set for the Logistic Regression model were implemented to observe any potential improvement in model performance. Finally, for a deeper linguistic analysis, pre-trained word embeddings from the word2vec-google-news300 model were used to convert messages into average vector representations. Both Logistic Regression and Random Forest classifiers were trained on these embeddings.
LSTM: TensorFlow and Keras frameworks were employed due to their advanced support for deep learning applications. The model architecture was built using the Sequential API, which included several layers crucial for text processing and sequential learning: • An Embedding layer to transform text inputs into dense vectors of fixed size, capturing semantic information. • An LSTM layer to learn long-term dependencies within the text data. • A SpatialDropout1D layer to prevent overfitting by dropping entire 1D feature maps during training.
• A Dense layer to produce the probability distribution over the target classes.
Preprocessing of text data involved tokenizing the texts and padding them to a uniform length to ensure consistent input dimensions across all samples. This was facilitated by Keras’ text preprocessing utilities. Additionally, categorical target variables were encoded using scikit-learn’s LabelEncoder, adapting them for model training. The model’s performance was evaluated using standard metrics from scikit-learn, including accuracy score and a detailed classification report, providing insights into the precision, recall, and F1-scores across diferent classes.
RoBERTa: The RoBERTa pretrained model on the collected dataset for training aimed at identifying indications of suicidal ideation in social media text achieved an overall accuracy of 66%. The model exhibited a macro precision of 66.07% and a macro recall of 65.85%, resulting in a macro F1-score of 65.81%. The micro-averaged precision, recall, and F1-score all stand at 66%.
RoBERTuito sentiment analysis: The fine-tuning of the RoBERTuito sentiment analysis model on the collected dataset for training aimed at identifying indications of suicidal ideation in social media text achieved an overall accuracy of 74.65%. The model exhibited a macro precision of 74.65% and a macro recall of 75.14%, resulting in a macro F1-score of 74.59%. The micro-averaged precision, recall, and F1-score all stand at 74.65%.
RoBERTuito base uncased: The fine-tuning of the RoBERTuito base uncased model on the collected dataset for training aimed at identifying indications of suicidal ideation in social media text achieved an overall accuracy of 89.61%. The model exhibited a macro precision of 94.59% and a macro recall of 63.64%, resulting in a macro F1-score of 68.57%. The micro-averaged precision, recall, and F1-score all stand at 89.61%.
Traditional classifiers: Multinomial Naive Bayes, Logistic Regression, Random Forest, and SVC demonstrated suboptimal performance. Specifically, the Multinomial Naïve Bayes model achieved a modest accuracy of 57.93%, and the SVC model slightly better at 59.8%. The Logistic Regression model performed somewhat more efectively, reaching an accuracy of 63%. Notably, the Random Forest model outperformed the others with an accuracy of 69%.
LSTM model on the collected dataset for training aimed at identifying suicidal ideation in social media text achieved an overall accuracy of 61.6%. The model exhibited a macro precision of 61.6% and a macro recall of 61.1%, resulting in a macro F1-score of 60.9%. The micro-averaged precision, recall, and F1-score all stand at 61%.
RoBERTa pretrained model on the collected dataset for training aimed at identifying indications of suicidal ideation in social media text achieved an overall accuracy of 66%. The model exhibited a macro precision of 66.07% and a macro recall of 65.85%, resulting in a macro F1-score of 65.81%. The micro-averaged precision, recall, and F1-score all stand at 66%.
RoBERTuito sentiment analysis fine-tuned model on the collected dataset for training aimed at identifying indications of suicidal ideation in social media text achieved an overall accuracy of 74.65%. The model exhibited a macro precision of 74.65% and a macro recall of 75.14%, resulting in a macro F1-score of 74.59%. The micro-averaged precision, recall, and F1-score all stand at 74.65%.
RoBERTuito base uncased fine-tuned model on the collected dataset for training aimed at identifying indications of suicidal ideation in social media text achieved an overall accuracy of 89.61%. The model exhibited a macro precision of 94.59% and a macro recall of 63.64%, resulting in a macro F1-score of 68.57%. The micro-averaged precision, recall, and F1-score all stand at 89.61%.
The efectiveness of various classifiers was evaluated in this study, including both traditional models and advanced deep learning approaches, in detecting indications of suicidal ideation within social media posts. Our analysis highlights the varying degrees of success and points to potential areas for further optimization.
Traditional classifiers, including Multinomial Naïve Bayes, SVC, Logistic Regression, and Random Forest, and even LSTM, demonstrated a range of efectiveness, with accuracies ranging from 57.93% to 69%. These models, while historically robust across various text classification tasks, fell short in this context, particularly in handling the complexities of mental health data. The highest performing among them, the Random Forest model, achieved an accuracy of 69%; however, this performance still highlights significant limitations, underscoring the need to explore more sophisticated deep learning models capable of handling the complexities of mental health data, particularly in a delicate topic such as suicidal ideation. RoBERTuito Base Uncased
The RoBERTa model achieved an accuracy of 66% on the collected training dataset, which indicates that it correctly predicts the binary labels for only two-thirds of the dataset. While this demonstrates some capability to generalize from training data to unseen data, the accuracy suggests that there might be substantial room for improvement.
The RoBERTuito sentiment analysis fine-tuned model achieved an accuracy of 74.65% on the collected training dataset, indicating a relatively high level of correctness in classifying text as either indicative or non-indicative of suicidal ideation. This suggests that the model is efectively leveraging its sentiment analysis training to make accurate predictions in a new closely related domain. Nevertheless, there is still potential for enhancement.
The election of RoBERTuito base uncased after trying RoBERTuito Sentiment Analysis model was due to the nature of the multiclass classifier of RoBERTuito Sentiment Analysis. Since it has been specifically trained to classify text into multiple sentiment categories such as positive, negative, and neutral, it may have still a bias even after indicating a binary classification.
In contrast, RoBERTuito Base Uncased is not specifically a multiclass classifier but a more generalpurpose model pretrained on a language modeling task without specific tuning for sentiment analysis or any other specialized classification task. Here it was taken as a foundational model that was adapted to this specific binary classification.
RoBERTuito Base Uncased resulted in an accuracy of 89.61% on the test dataset of our collected training data, which indicates that the model is efectively generalizing from the training data to the unseen test data, demonstrating robust performance in distinguishing between texts that either suggest or do not suggest suicidal ideation.
The macro-averaged precision of 94.59% indicates that, on average, the model’s predictions of each class are reliable. However, the macro-averaged recall of 63.64% suggests that the model, while precise, is conservative in predicting positive instances and could be missing a proportion of actual positive cases. Nevertheless, the similarity between micro-averaged metrics and overall accuracy suggests that the model performs uniformly across all instances, providing confidence in its general utility.
Testing our RoBERTuito Base Uncased fine-tuned model on the testing data provided by MentalRiskES, the performance of the was evaluated across three runs in comparison to other teams and a baseline model. The results demonstrated that our RoBERTuito Base Uncased fine-tuned consistently achieved an accuracy of 0.691 across all runs. In terms of macro-averaged metrics, the RoBERTuito Base Uncased finetuned attained a Macro_Precision (Macro_P) of 0.345, Macro_Recall (Macro_R) of 0.500, and Macro_F1 score of 0.409. Similarly, for micro-averaged metrics, the RoBERTuito Base Uncased fine-tuned recorded a Micro_Precision (Micro_P), Micro_Recall (Micro_R), and Micro_F1 score all at 0.691. The RoBERTuito Base Uncased fine-tuned performance on the Early Risk Detection Error (ERDE) metrics showed an ERDE5 score of 0.261 and an ERDE50 score of 0.261. Additionally, the RoBERTuito Base Uncased ifne-tuned latency to positive prediction (latencyTP) was consistent at 0.214 with a speed of 1 and a latency-weighted F1 score of 0.817. See Table 1. Overall, our RoBERTuito Base Uncased fine-tuned performance was comparable across all runs, maintaining a balance between precision and recall with strong latency-weighted F1 scores, highlighting the consistency and reliability of their approach in this evaluation.
While traditional classifiers such as Logistic Regression, Naïve Bayes, and Random Forest have demonstrated utility in handling a variety of text classification tasks, the evolution of deep learning has introduced more sophisticated models that can capture intricate patterns in large datasets. This shift marks a significant advancement in the field of natural language processing.
While the transition from traditional classifiers to sophisticated deep learning models like RoBERTuito represents a significant leap forward in our capacity to process and analyze mental health data, future eforts need to continue. Particularly in Hispanic-speaking communities where little research has been performed; the challenge of accurately detecting suicidal ideation cannot be focused on specific demographic groups. The results of this study underscore the pressing need for continued enhancements, especially in improving recall without sacrificing precision, to efectively support suicide prevention eforts and provide timely interventions in Hispanic-speaking populations where such resources are critically needed.
When testing the RoBERTuito base uncased fine-tuned model on the actual test dataset provided by the MentalRiskES committee at IberLEF 2024, the metrics resulted in an accuracy of 69.1%, a macro-averaged precision of 34.5%, a macro-averaged recall of 50%, and a macro-averaged F1 score of 40.9%; and a micro-averaged precision of 69.1%, a micro-averaged recall of 69.1%, and a micro-averaged F1 score of 69.1%. This could be due to the diferences between the training and the test datasets. This approach was guided by the need to train our models on data that closely mirrors the characteristics of the test set provided by the task organizers, albeit the exact content and context might difer.
This research was supported by the National Autonomous University of Mexico. Special thanks to the MentalRiskES 2024 task organizers and the IberLEF 2024 committee.
The sources for the ceur-art style are available via: • GitHub • Overleaf template