This paper presents the methodologies employed by the UC3Mental team in its participation in the MentalRiskES tasks at IberLEF 2025. The competition focused on two key challenges related to the detection of mental illness in Spanish-language social media: addiction detection and addiction type classification. Our approach involved three diferent strategies: (1) a baseline model using a Support Vector Machine (SVM); (2) a two-stage pipeline that first applied a Random Forest classifier to classify addiction type, followed by a specialized SVM trained to detect addiction cases of that specific type; and (3) a model based on a BERT Transformer architecture.
Mental disorders are commonly associated with significant disturbances in an individual’s thoughts, emotions, or behavior. Although awareness of mental health issues has grown, the number of people afected continues to rise, and many still face stigma and insuficient access to treatment. One of the main challenges lies in the early detection of such conditions, which is often hindered by a lack of resources and timely intervention.
In recent years, the analysis of social media content has gained prominence as a complementary
method for identifying mental health risks [
MentalRiskES competition was designed to support research in this area, putting extra value on the
early detection and classification of addiction-related not only on accurate results. The 2025 edition of
the competition comprised two tasks: addiction detection and addiction type classification [
The objective was to assess the efectiveness of combining classical machine learning techniques with more recent deep learning architectures for the task of mental health risk detection on social media platforms.
This is a binary classification task aimed at determining whether a user is at high risk (label = 1) or low risk (label = 0) of developing a gambling-related disorder based on their messages. The objective is to enable early detection and facilitate timely interventions.
In this task, all users are considered to be at some level of risk (either low or high). Regardless of the risk level, the model must identify the specific type of addiction associated with the user’s messages. The available labels for classification are: Betting, Online Gaming, Trading, and Lootboxes.
The dataset provided for the shared tasks consists of 358 annotated user samples for the addiction risk
detection task (Task 1). The same set of users is used for the addiction type classification task (Task 2),
but annotated with one of four specific addiction categories [
The dataset contains user-generated messages sourced from two platforms: Telegram and Twitch. Each user has a varying number of messages, and these messages serve as input features for both tasks. The dataset ofers a realistic setting but presents some challenges related to class and platform imbalance, which are critical to consider during preprocessing and model development.
The initial analysis of the dataset reveals that the binary classification task (Task 1) is well-balanced, with an approximately equal distribution of high-risk and low-risk users. However, a significant platform imbalance is present: Telegram contributes a much larger volume of messages than Twitch. Despite this imbalance, both platforms include users from both risk categories. Additionally, users vary considerably in their message activity, which can be grouped into three general tiers: those with fewer than 20 messages, between 20 and 60 messages, and more than 60 messages, Figure 1. This variability should be considered when designing input representations or aggregating features across user histories. (a) Distribution of high-risk and low
risk users across platforms.
(b) Message length distribution by risk label. (a) Number of messages per user by addiction type label. (b) Message length distribution by addiction class.
In the multi-class classification task (Task 2), we observe a pronounced class imbalance, Figure 2b. The most frequent category is Trading, followed by Online Gaming and Betting, with Lootboxes being the least represented. Furthermore, there is a strong association between platform and addiction type: Telegram is the predominant source for messages related to Betting and Trading, while Online Gaming and Lootboxes are more commonly found on Twitch. Although these patterns are informative, caution must be exercised to avoid developing models that rely on superficial cues such as the platform itself. Overfitting to platform-specific distributions can hinder generalization and reduce the model’s ability to focus on meaningful linguistic and behavioral features indicative of each addiction type. Finally, the hypothesis regarding a possible correlation between message length and addiction classification was addressed by examining the message length distribution across the full dataset, both for low and high risk labels and for addiction classification, see Figure 3a and 3b, respectively. Any exploitable correlation was discarded as no statistically meaningful diference could be appreciated on the distributions.
In addition to the descriptive analysis of the data, and as a consequence of the poor results obtained in Task 1 during the training and testing of the models (to be discussed later), a further investigation was conducted to assess the hypothesis regarding the absence of an underlying pattern governing the message labeling process. Specifically, the study aimed to validate the notion that the target variable for Task 1 is distributed in an almost random manner, which would mathematically preclude the development of models capable of achieving significantly higher accuracy than 50%. The following set of experiments was conducted to attempt to validate the previously stated hypothesis: 1. Train three basic machine learning models and evaluate their performance on the test set. If diferent techniques yield results close to 50% accuracy, this could suggest a random distribution of the labels. However, such results alone are not suficient to conclusively support this hypothesis. 2. Apply a k-means clustering model to the vectorized data (using TF-IDF, as detailed later in the document), where each vector represents a user. The goal is to group similar vectors—hence, similar users—and subsequently analyze the statistical distribution of Task 1 labels within each cluster. The underlying assumption is that similar users should exhibit similar labels. If the label frequencies are comparable across clusters, this could indicate the absence of a clear labeling pattern. 3. Perform a Kullback-Leibler (KL) divergence test to compare the “distance” between the empirical probability distribution of the target variable in Task 1 and a uniform binary probability distribution (i.e., a Bernoulli distribution with 50% probability for each label).
The three models employed were: -Nearest Neighbors (-NN), logistic regression, and a decision tree (ID3). The results in terms of classification accuracy and Area Under the ROC Curve (AUC) are presented in Table 1.
-NN Logistic Regression
Decision Tree
Accuracy 51.38% 70.83% 58.33% f1-score (macro-avg) f1-score (weighted-avg) 0.34 0.35 0.71 0.71 0.58 0.58
AUC 0.58 0.77 0.59
Based on the results, two of the three models exhibit values close to 50%, while logistic regression approaches 70%. These figures should be interpreted with caution, as although most models appear to be operating on a randomly labeled target variable, the notably higher performance of logistic regression may be a consequence of the specific random train-test split used. This could have fortuitously resulted in overly optimistic test performance that is not generalizable.
In this section, as previously mentioned, the aim is to group similar users based on the TF-IDF vectors generated during the text processing phase (further details are provided in the following sections of this article). The -means algorithm was employed for clustering the users, and the optimal number of clusters k was determined using the Silhouette Score (Equation 1). This metric, applied by performing multiple clustering runs with varying values, enables the identification of the most appropriate number of clusters. By applying the algorithm and the silhouette criterion, the optimal number of clusters was found to be 2 (see Figure 4).
() =
() − () max{(), ()} (1)
Where () is the average distance between point and all other points in the same cluster and () is the average distance between point and all points in the nearest cluster to which does not belong. The average silhouette value over all points provides an overall measure of clustering quality.
By characterizing each of the resulting groups, we obtain the results shown in Figure 5.
From this, it can be concluded that no clear pattern exists namely, that one type of user (i.e., one of the two clusters) has significantly more labels of one class than the other suggesting that identical or highly similar inputs yield completely diferent outputs. The Kullback-Leibler (KL) divergence is a measure of how one probability distribution difers from a second, reference probability distribution. It is important to notice that KL divergence is not symmetric, meaning that KL( ‖ ) ̸= KL( ‖ ). So it is very important to note that KL divergence is not a distance, but rather a similarity measure.
For continuous probability density functions () and (), the KL divergence is defined as: KL( ‖ ) = ∫︁ ∞ () log ︂( () )︂ ()
−∞ In both cases, the KL divergence is always non-negative and equals zero if and only if = (or (2) () = () almost everywhere in the continuous case).
For two discrete probability distributions and defined over the same set , the KL divergence from to is given by: KL( ‖ ) = ∑︁ () log ︂( () )︂
() ∈
To obtain a unified KL divergence result, the arithmetic mean of the KL divergence calculated in both directions is applied, as shown in Equation 4.
1 K*L( ‖ ) = [KL( ‖ ) + KL( ‖ )] (4)
2
As a result of applying Equation 4, the bidirectional KL divergence is found to be 0.0001922, a value suficiently small and close to zero to support the assumption that the probability distribution of the target variable in Task 1 closely resembles a uniform distribution (i.e., a random distribution). Therefore, based on the results obtained from the preceding experiments, we can reasonably conclude that the target variable in Task 1 has been labeled in a random manner, or at the very least, lacks an underlying pattern that can be efectively learned by machine learning models. (3)
The following section presents a detailed account of the proposed systems and models employed for the text classification task, ranging from traditional baseline approaches, through their combinations, and culminating in transformer-based methods and large language models (LLMs).
The initial approach employed to address the task of user classification based on their messages was a traditional method relying on Support Vector Machines (SVMs), adopted as a baseline model. This model takes as input the users’ pre-processed texts, as illustrated in Figure 6. The rationale behind this approach is to assess the efectiveness of a classical method before resorting to large language models and transformer-based architectures, given that such traditional algorithms have demonstrated strong performance in past competitions.
The main aspect of the classical SVM-based approach lies in the preprocessing strategy chosen to clean and vectorize the texts in a manner suitable for the SVM model to process the information. As illustrated in Figure 6, the core idea is to tokenize the texts using TweetTokenizer, optionally remove emojis, and eliminate punctuation marks. It is important to note that, despite eforts to thoroughly clean the texts during preprocessing, the data originates from social media conversations. As such, spelling errors and informal or inconsistent semantic and syntactic structures present in the messages significantly hinder the ability to obtain clean, easily processable text—this being one of the major challenges of the task.
Since both tasks involve classifying users based on their texts, all available messages per user were tokenized, cleaned, and concatenated, such that the input for each user corresponds to the concatenation (with whitespace inserted between them) of all their texts (see Equation 5). This input is then processed by the TF-IDF algorithm, which is used to vectorize the information that will be provided to the SVM. ← ⃦ ⃦ ∈ℳ Φ( ) || ; ∀ ∈ | | (5)
Where ℳ denotes the set of messages associated with the -th user, Φ represents the tokenization and data cleaning function, and denotes the whitespace character.
For the training and hyperparameter tuning of the SVM model, an 80–20 split was performed for the training and test sets, respectively, selected at random while ensuring a stratified distribution across all classes. In the case of Task 1, the label distribution between high_risk (1) and low_risk (0) was nearly balanced at 50%, thus eliminating the need for additional class balancing techniques. However, for Task 2, the multiclass classification problem exhibited the class imbalance shown in Figure 7, prompting the use of the SMOTE technique to perform oversampling and achieve a more balanced class distribution. SMOTE was applied exclusively to the training set, as applying it to the entire dataset (including the test set) would result in what is known as data leakage—an issue that introduces unintended knowledge into the model and leads to overly optimistic performance results.
Kernel function
0.1, 1, 2, 3, 7 scale, 0.01, 0.001, 0.0001 linear, rbf Hyperparameter search space for the SVM baseline model.
Finally, for hyperparameter tuning (conducted via grid search), various values of , , and the SVM kernel were tested (see Table 2), using 5-fold cross-validation. The optimal configuration was found to be: = 3, = auto-scaled, and kernel = radial basis function (RBF).
The RF-SVM ensemble constitutes the main contribution of the present work. The rationale behind this ad hoc model, specifically designed to address the early detection task, lies in the observation that both Task 1 and Task 2 are interrelated, as they rely on the same underlying data (i.e., the same users and messages), difering only in the type of labels provided. While the SVM model yielded the best performance for Task 1, for Task 2, the Random Forest model slightly outperformed the SVM, albeit by a narrow margin. Accordingly, the RF-SVM ensemble operates as follows (see Figure 8): a Random Forest model, trained for Task 2, is first employed to classify the message, achieving excellent precision, as will be discussed later. Based on the classification output for Task 2, a second classification is then performed using an SVM model that has been trained exclusively on messages originally labeled with the corresponding addiction type. In this manner, a general Random Forest classifier determines the type of addiction (among the four available classes) to which the current message belongs, allowing a specialized SVM—trained specifically on messages related to that addiction type—to carry out Task 1, which is inherently more challenging.
This strategy aims to enable model training at a finer level of granularity, where, instead of learning all distinctions and characteristics across the entire dataset, the model captures local patterns. For example, it may be significantly more common to observe high-risk mental health indicators in messages associated with a specific type of addiction than in others. 3.3. BERT BERT (Bidirectional Encoder Representations from Transformers) is a language model developed by Google in 2018. Its primary innovation lies in the use of a bidirectional Transformer architecture, which enables the model to understand the context of a word by considering both the preceding and succeeding words. This capability makes it particularly efective for tasks such as sentiment analysis, question answering, and text classification. Within the scope of the problem to be addressed (Task 1 and Task 2), the Transformer-based approach aimed to strike a balance between result quality and the use of models that do not impose excessive computational or memory costs. To meet both requirements, BERT-small [13] was employed. BERT-small is a reduced version of BERT designed to be more resource-eficient. While the base BERT model consists of 12 Transformer layers and approximately 110 million parameters, BERT-small typically includes only 4 layers and around 29 million parameters, resulting in a faster and more lightweight model. Although it trades of some accuracy compared to its larger counterparts, BERT-small is well-suited for deployment on resource-constrained environments such as mobile devices or embedded systems.
The data preprocessing followed the previously described pipeline, with the exception of the steps beginning from the application of the TF-IDF algorithm (inclusive), as the Transformer architecture inherently computes its own embeddings. In other words, external vectorization of the texts is not required. The procedure essentially consists of performing fine-tuning on the pre-trained BERT-tiny base model in order to adapt it to the desired text classification task. Within the fine-tuning process, the following selected parameters are of particular note: • Evaluation strategy: per epoch • Train batch size: 4 • Evaluation batch size: 4 • Training epochs: 25 • Learning rate: 7 · 10− 7 • Weight decay: 10− 3 • Model storage policy: best model 3.4. LLM Additionally, although it was not part of the models used in the competition, an approach based on Large Language Models (LLMs) was tested to address the classification task. The large language model (LLaMA 3, 70B) [14] developed by Meta AI was used to perform two classification tasks on user-generated text. The LLaMA 3 model is a transformer-based architecture trained on a diverse and extensive corpus of publicly available web data, books, academic texts, and code. It is optimized for instruction following, question answering, and general-purpose language understanding tasks. The 70-billion parameter version of the model was used, which represents the most capable publicly released version of the LLaMA 3 family at the time of this study. All classification tasks were carried out using a zero-shot prompting setup — no additional fine-tuning or task-specific training data was provided to the model. Instead, the model was prompted with natural language instructions tailored to each task and expected to infer the appropriate label based on its pre-trained knowledge and language understanding capabilities. The results of the LLM-based approach were not successful in either of the two tasks, as will be shown in the following section on experimental results. Therefore, as previously mentioned, only the first three approaches described in this section were employed for the competition. To conclude this section, the prompts used to address Task 1 and Task 2 are provided below:
You will now be given a series of messages written by a person. Your task is to analyze the content and determine if there are clear and repeated indications of behaviors related to pathological gambling (gambling addiction). Frequent references to gambling or compulsive trading. Expressions of loss of control, anxiety about gambling, or urgent need to win back money. Mentions of significant financial losses due to these games or similar behaviors. Difficulty to stop gambling or thinking about gambling constantly. You should not interpret all comments about gambling or trading as a sign of addiction since the objective is only to detect cases of pathological gambling, addiction and loss of control. Your answer must be exclusively a number: 1 if there are several clear and direct indications of pathological gambling. 0 if there is not enough clear evidence (even if there are vague or isolated mentions). Now evaluate the following messages:
Below you will be given a series of messages written by a person suffering from some form of pathological gambling (gambling addiction). The goal is to classify the condition according to the specific type of addictive behavior. The following categories exist: Betting: refers to addiction to traditional betting, such as sports betting, horse racing or gambling in physical or online casinos (roulette, poker, etc.). Onlinegaming: Refers to addiction to multiplayer online video games, where the person plays compulsively without there necessarily being a direct monetary transaction. Lootboxes: Addiction to random reward systems within video games, where a “box” is bought with real money to obtain virtual items, generating a behavior similar to that of slot machines. Trading: encompasses the addiction to trading financial assets (such as cryptocurrencies, stocks or forex), in which the individual performs compulsive operations seeking immediate gratification, assuming high risks. Your answer should be exclusively one of these words, with no additional phrases, no quotation marks and no explanations: betting, onlinegaming, lootboxes, trading. Now evaluate according to the following messages:
As discussed before, the goal of the first task was to identify whether a user’s messages indicated signs of a gambling disorder. Each user was labeled using a binary classification: • 0 — No indication of a gambling disorder.
• 1 — Language indicating the presence of a gambling disorder.
The results obtained on the test set from the trained models, following hyperparameter tuning, are presented in Table 3.
The second task aimed to further categorize the type of gambling-related behavior expressed by the user. The model was prompted to classify the content of each user’s messages into one of four predefined categories, reflecting distinct behavioral patterns: • betting — Traditional gambling (e.g., sports betting, horse racing, casinos). • lootboxes — Randomized virtual item purchases in video games. • onlinegaming — Compulsive play of online games without monetary stakes.
• trading — High-risk trading of financial assets such as cryptocurrency or stocks.
The results obtained on the test set from the trained models, following hyperparameter tuning, are presented in Table 4.
As shown in Table 4, the first notable observation is that the results for Task 2 from the SVM baseline model and the RF-SVM ensemble are identical. This has a straightforward explanation: recalling the structure of the RF-SVM approach, a Random Forest model is first used to solve Task 2, and its output is then passed to a dedicated SVM to resolve Task 1. In other words, the ensemble’s contribution is specific to Task 1. Since Task 2 is resolved by a Random Forest model (whose results on the test set happen to match those of the SVM baseline), the RF-SVM ensemble behaves identically to the SVM baseline for Task 2.
In terms of accuracy, the BERT-small-based approach stands out by achieving 97.22%, an excellent result—though not dramatically higher than those achieved by the SVM and RF-SVM models. It is important to note that comparable results to those obtained with BERT-small were approximately replicated using machine learning models that are significantly less costly in terms of computational time and memory, reinforcing the notion that larger and more modern models are not always the best solution.
Finally, the LLM-based approach performed noticeably worse. The results obtained with LLaMA 3 were significantly inferior to those of the other models. This is particularly striking given that, unlike Task 1, Task 2 appears to be a well-defined task that can be efectively solved, with most models achieving classification metrics well above 90%.
This article has presented an innovative ensemble architecture that combines advanced machine learning models to address both binary and multiclass text classification tasks. The results obtained on the test sets demonstrate the improvements brought by this new approach, enabling the achievement of eighth place in the MentalRiskES competition, held as part of IberLEF 2025. Performance on Task 1 was significantly lower than on Task 2, as discussed in the article. The label distribution for Task 1 appears to follow a random pattern, lacking identifiable structures or features that machine learning models can efectively learn. In comparison with modern approaches based on Transformers and Large Language Models (LLMs), our ensemble method performs equally well or better, while ofering the added benefit of reduced computational cost in terms of both time and memory. This eficiency was further reflected in the system’s ranking as the second fastest among all submitted solutions for the proposed tasks.
As future work, we propose a more extensive study and experimentation using a larger dataset (with more instances), in which the labeling of messages and users has been thoroughly reviewed and there is evidence supporting the existence of a modelable phenomenon that can be learned through machine learning techniques. The RF-SVM architecture is not limited exclusively to the tasks proposed in the MentalRiskES competition; rather, it can be extended through basic modifications to multiclass classification problems where the data is doubly labeled (in binary or multiclass form), enabling predictions on data subsets using models with lower granularity.
During the preparation of this work, the author(s) used ChatGPT to assist with English translation, grammar correction, and spelling checks. All content generated with the help of this tool was reviewed and edited by the author(s), who take full responsibility for the final version of the publication. corpus: Gambling in spanish social media, in: Proceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 17–28. [13] H. Tsai, J. Riesa, M. Johnson, N. Arivazhagan, X. Li, A. Archer, Small and practical BERT models for sequence labeling, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 3632–3636. URL: https://aclanthology.org/D19-1374/. doi:10.18653/ v1/D19-1374. [14] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, G. Lample, Llama: Open and eficient foundation language models, 2023. URL: https://arxiv.org/abs/2302.13971. arXiv:2302.13971.