The aim of Task 2 (Early Detection of Signs of Pathological Gambling) from the CLEF 2023 eRisk Workshop is to analyze social media users' messages for early warning signs of pathological gambling. Given that Pathological Gamblers are a small set compared to the Control group, we propose the utilization of a neural network incorporating a customized loss function to efectively tackle the challenge of class imbalance. In our proposed loss function it is possible to adjust the penalty for false positives and false negatives, increasing the penalty for the critical false negatives. Our proposed solution demonstrates robustness, achieving one of the highest recall rates while maintaining a competitive precision. Furthermore, our system introduces a range of potential variations that warrant further investigation.
vector representations of user messages through sentence embedding, and then on detecting positive messages using deep learning-based methods. Additionally, an original loss function is introduced to deal with imbalanced classes or cases where false negatives have a significant impact.
The rest of the article is structured as follows: Section 2 provides an overview of previous work related to the task considered and the techniques used in this study. Section 3 describes the tackled task, including the available dataset and evaluation metrics, while the developed system is presented in Section 4. Section 5 demonstrates the configuration of the variants used for the competition. The results obtained are compared with those of other participating systems and discussed in Section 6. Finally, Section 7 presents the main conclusions and future lines of work.
In recent years, the use of AI to address mental health-related tasks has gained greater presence. AI, and in particular NLP, has proven to be a powerful tool in the detection of mental disorders. In previous studies, NLP has been used on electronic health records to assist in the identification of suicidal behaviors [2, 3], achieving an accuracy of 0.47.
In this context, most of the methods used for the detection of mental disorders are based on traditional Machine Learning techniques, such as SVM, AdaBoost or Decision Trees. The recent interest in Deep Learning has shown a better performance [4]. However, as stated by Zhang et al. [4] a large part of the solutions proposed are concentrated on a few mental disorders. As one of the most prevalent disorders in the world, the absence of research on the identification of gambling addiction stands out.
The SMM4H [5], CLEF and CLPsych [6] competitions focus on the application of machine learning techniques in the field of mental health. In 2022, the winner of the CLPsych competition achieved 68.9% accuracy in detecting mood changes in tweets. Years earlier, in 2019, a similar rate was recorded in identifying users at risk of suicide based on their messages.
For several years, one of the tasks to be addressed in CLEF eRisk has been the early detection of pathological gambling risks. By sequentially processing user interactions on social networks, the system was to detect the first signs of pathological gambling as early as possible.
Very diferent approaches were used in the previous edition. The BLUE group, proposed to train a BERT classifier, using an additional dataset generated from some Reddit mental health communities. The UNED-NLP [7] team, participated with a system that relied on Approx Nearest Neighbors techniques to detect positive messages. The SINAI [8] group came up with a design based on language features. Using the last 50 user messages, a vector was obtained which was complemented with message features such as number of words, lexical diversity and sentence complexity. Finally it was passed through a feed-forward neural network (FFNN) model. Similarly, in our work we propose a FFNN feed by a semantic representation of sentences.
The dataset provided in this edition, is the combination of the data used in the two previous
editions (eRisk 2021 and 2022). It is composed of a number of XML files, each of which contains
a number of posts made by 4,427 users of social networks. It includes 2,298,412 messages overall.
Labels are user level and they design them as either gamblers (
In Figure 1, it is evident that both editions exhibit a significantly imbalanced dataset, indicating a non-uniform data distribution. Specifically, the proportion of players classified as gambling addicts in their respective sets does not exceed 7% or 4%. Previous research in supervised classification has highlighted class imbalance, referred to as class skew or class imbalance, as a critical factor that significantly impedes the capacity of inference algorithms to learn and accurately generalize the minority class.
In order to enhance the message quality during the training process, a filtering approach was employed. Specifically, users with fewer than 10 messages and the top 20 users with the highest presence in the dataset were excluded. This strategic exclusion aimed to mitigate the potential noise introduced by these users, thus improving the overall training eficacy.
The test set oficially employed to assess the challenge, described in Table 1, is sent to the participants iteratively through a connection to a server. The total number of users is 2174, of which 103 are compulsive gamblers.
Num. subjects Num. submissions (posts & comments) Avg num. of submissions per subject Avg num. of days from first to last submission Avg num. words per submission
Given a sequence of consecutive posts post messages written by subject , denoted as t = (1, 2, . . . , ) with being the -th post (i.e. text) in the succession, the aim is to get a subjectlevel classification label ( ^) to distinguish Pathological Gamblers from Control subjects. To that end, in our approach each post () is processed and a post-level label () computed by means of the architecture presented in Figure 2. With this information is attained, next, the user-level label. The processes involved and training strategies applied are detailed in the following sections.
Post tkj Having no money; worst part of recovering… Model
Preprocessing
Lemmstaetmizmatiinogn and
Encoder
SBERT DAN LDA xkj1 xkjn xkj xkjn+1 xkjN
FFNN N 128
128 DO 0.2 ∫ 2 - ykj >0 c(0kj,∈1)
Each of the posts has two parts, a title and a body. We combined both parts to create a single message. We conducted the following steps in order to preprocess the posts: • Conversion to lowercase • Characters cleaning • Stopwords removal
Lemmatization and stemming were also incorporated prior to LDA in order to improve the generated topics. Table 2 shows the versions of a text after applying diferent preprocessing techniques. The last version would be necessary in the case of applying LDA.
We need to get each text-post, , converted into a fixed-size numeric vector, x =
(1, 2, . . . , ) ∈ R that would serve as the input to the FFNN. Note that
∈ Σ*
being Σ the input vocabulary. Two main strategies were explored in order to get a numeric
representation (x) given a post (): encoding and LDA. We can either use just one strategy or both
and make use of the concatenated representation leading, thus, to a longer vector representation,
as in (
The text, represented as an array of fixed size with the dimension ( ) depending on the representation used (either encoding or LDA or both). The resulting vector representation x ∈ R is, indeed, the input for the classifier.
: Σ* →− →−
R
() = (,1, ,2, . . . , ,, ,+1, ,+2, . . . , , ) = x
(
In what follows, details are given of each strategy explored to obtain the representation, the encoder in section 4.2.1 and LDA in section 4.2.2.
We wondered whether Dynamic Aggregation of Network (DAN) or Sentence-BERT (SBERT) would generate a better semantic representation of the texts. Models such as the Universal Sentence Encoder (USE) [9], Sentence-BERT (SBERT) [10] and Transformer-based Pretrained Language Models (PML), allow a complex and global representation taking into account word interactions and relations. SBERT is a variation of the traditional BERT [11] model that incorporates Siamese and triplet lattice structures. Such structures enable learning the similarities and contrasts between various inputs.
USE, on the other hand, bases its architecture on convolutional and recurrent neural networks. There are two USE variants, the most widely used is based on transformers. Nevertheless the variant known as the Dynamic Aggregation of Network (DAN) that makes use of the dynamic aggregation of networks approach to enhance the outcome has shorter computing time.
We represented the posts using DAN and SBERT in order to compare their performance. The vector of size () generated by DAN encoder is 512 while SBERT works with a vector of size 384.
Latent Dirichlet Allocation (LDA) is a probabilistic model able to identify the latent topics in the posts. It enables to extract the topic distribution of each post and we used it as additional features to represent the post. We configured LDA to extract 20 topics from the posts, leading to a representation x ∈ R20.
The estimated post-label-confidence ∈ R is an intermediate scalar obtained in our system interpreted as the confidence score that the post contains traits of language related to Pathological Gambling. When takes positive values, the intermediate post-level label is assigned to class 1; otherwise, it is assigned to class 0, that is, () = ().
Note that the (· ) transformation is attained by means of the FFNN network. These processes
are formally summarized in (
∘ : Rx →→−−
R
→−
(x) = →−
{ 0, 1}
() =
(
With regard to the practical details of the FFNN network, we would like to mention that it includes two layers and contains a total of 49538 to 65922 parameters, depending on the encoder employed, respectively, SBERT or DAN. If LDA is used, there are 2560 additional parameters.
In what the training process is concerned, the FFNN was trained on 5 epochs and setting the learning rate to 5 × 10− 5. Besides, in the training stage the dropout was set to 0.2 and AdamW optimizer was employed as it was proven efective [ 12]. An iterative training approach was implemented where, in each epoch, all the messages from a user are sequentially processed to update the parameters.
User gold label is the user label (either ’Control’ or ’Pathological Gamblers’) for -th subject as in the gold-standard, that is, the expected label for the subject. With the information attained at post-level (as stated in section 4.3), the user-level label is estimated (̂︁). The performance of the system is, indeed, assessed based on the diference between predicted ( ̂︁) and expected () subject-level labels.
Note, however, that there is a subtlety in the arrangement of the task: not all the post of the user are given jointly, instead, the posts are presented to the system in sequentially, one by one in their turn. That is, for user in the time-stamp 1 we merely count on the post 1, while by the -th time-stamp we would have seen a sequence of posts, (1, 2, . . . , ). For each post the system must provide a user-level assessment. Hence, by the -th round, the system has provided a sequence of outputs û︀ = (̂︀1, ̂︀2, . . . , ̂︀).
By the time the system needs to compute current user-level label ̂︀+1, it counts with current
post +1 and all the history:
• past sequence of posts and, inherently, their corresponding encoding: t = (1, 2, . . . , )
and ((1), (2), . . . , ())
• past sequence of computed post-level confidence scores: y = (1, 2, . . . , )
• past sequence of estimated post-level labels: ̂c︀ = (1, 2, . . . , )
All this information is available and can be employed to generate ̂︀+1 as stated in (
R × { 0, 1} →−
(+1, t, y, ̂c︀) →−
{ 0, 1}
ℎ(+1|t, y, ̂c︀) = ̂︁+1
In our approach, however, the user-level label +1 is estimated as in (
̂︀
That is, we computed the user-level label relying merely on the current post-level label and disregarding the historic information. Needless to say, future eforts could be devoted to leverage ℎ(· ) exploiting all the information available.
̂︁+1 = ℎ(+1|t, y, ̂c︀) = ( ((+1))) = ̂︀+1
(
Since gold-labels are provided at user level and we trained our FFNN using posts, we needed to get the posts labeled for the training. That is, in the training stage the estimated post-label confidence ( ) must be compared to a desired or expected confidence ( y′) the underlying issue rests on the fact that the post-level confidence is not given. This silver standard is shown in Figure 3 As the reference post-label confidence, y′ = (′1, . . . , ′), in this work we explored two alternative silver-standard assignation strategies: • User-based message labeling (UBL): Consists of assigning each post the label of the user. That is, if a user is positive all it’s posts will be labeled as positive, that is, all the components in this array, ′, equal to . • Approximated Nearest Neighbors (ANN): Posts are labeled using an iterative labeling approach. First, each post is labeled with the label of the user. Then using ANN technique, the labels are reassigned, giving to the closer posts the same labels, getting y′ as in [7].
The heuristic post-level references employed have a deep impact in the training stage and, needless to say, should be selected carefully. Future work can be addressed in alternative reference assignment strategies.
As mentioned in section 3, the data, far from being uniform, exhibit a noticeable imbalance. As a consequence, the neural network can become biased and achieve low precision in the minority class. To address this problem, there are various strategies to alter the class proportion through over-sampling and under-sampling. Following an approach closer to assigning class weights, we have chosen to apply a loss function based on cross-entropy during the training of the neural network. The strategy followed in our work is sketched in Figure 3.
User: k
uk
With the sequence of posts from user a sequence of confidence scores is computed by the
model, post by post and y obtained. With this, as mentioned in section 4.3, a sequence of
post-level labels are computed, that is, ̂c︀. In the training stage, the system estimates a user-level
label taking into account all the post-level labels, as in (
̂︁ = {︃1, if ∃ 1 ≤ ≤ : 0, otherwise = 1
In the training stage, the sequence of labels computed (y) are compared with the
silverlabels proposed as reference (that is, the y′ labels presented in section 4.5). This comparison is
quantitatively seized as the loss by means of the Cross Entropy Loss function, (y′, y), as
implemented in PyTorch [12]. The user-dependant loss is seized by a weight factor that allows
for a penalty as in (
= · (y′, y)
Our training approach does not penalize equally false positives and false negatives, indeed the
(
= ⎧4 if ̂︁ = 0 ∧ = 1 ⎪ ⎨
2 if ̂︁ = 1 ∧ = 0
⎪⎩1 if ̂︁ =
(
The critical false negatives were penalized by doubling the loss in those cases where the system incorrectly predicted a positive instance. These values (1, 2 and 4) have been determined through a sensitivity analysis and based on the objective of prioritizing false positives over false negatives in an attempt not to miss Pathological Gamblers.
There is room for improvement in the training stage. On the one hand, the user-level labeling
strategy, i.e. (
Preliminary experiments were conducted combining all the variants mentioned in the methodological framework, varying: the post level vector representation (involving SBERT, DAN and LDA as mentioned in section 4.2); alternative silver-standard approaches to get a post-level ground-truth (UBL and ANN as detailed in section 4.5); the data-set partition employed to train the system (exploring the 2021 or 2022 and sets as mentioned in section 3). user-penalty weights in the training stage (developed in section 4.6). These preliminary experiments were conducted on a partition of the provided data split by year and also combined (see Figure 1). These experiments led us to select the parametrization e.g. the penalty weights presented in expression 7 and the same modified loss function. We found that DAN outperformed SBERT and this is why we, eventually, discarded that encoding.
As a result, in our team, a total of 5 runs were submitted with the configurations detailed in Table 3. Given the limitation on the number of runs to submit, greater diversity in configurations was prioritized over variants using SBERT. Additionally, a combined model was used, which utilizes all the mentioned variants to create a single result. The decision made by the combined model is determined through an OR operation. Accordingly, it is, both necessary and suficient, for one of the variants to estimate a positive user-level label for this combined approach to estimate positive. T he motivation behind this combined approach is to leverage the Recall even at the expense of certain false positives, which are preferable to false negatives.
In Table 4, we can see some of the best results from the competition in the main task. This allows us to compare our proposed system, Xabi_EHU, with other submissions. Among all the system variations, the first configuration has achieved competitive scores in all metrics, with a strong dominance in Recall. In an environment where false negatives have a significant impact, having a high Recall is of great interest in real-world applications, compensating for small diferences in other measures. On the other hand, as seen during development, the use of LDA has resulted in a general performance loss for the system, while training based on more precise labeling has led to a decrease in the system’s precision. Overall, these results suggest that there are multiple efective approaches to addressing the problem. Note that, while somehow simplistic, the UBL approach resulted in sensitive.
Regarding the additional task of user ranking, Table 5 shows the performance of the top teams in that task. As can be seen, prioritizing better Recall has led to inferior performance compared to other teams. Without additional information, using the class probability itself as the user’s risk level may not be a good indicator for what the competition intended.
Our participation focuses on early detection of signs of pathological gambling. Given a sequence of posts in a sequence, one by one, the aim is to estimate, employing as fewer posts as possible, the subject-level label (either Control or Pathological Gambler). With the data-set highly skewed, and being the target Pathological Gambler the minority group, we struggled to find robust models and focused on maximizing the recall not to miss pathological gamblers. However, this led us to the creation of a specific loss function. The data imbalance has also prevented us from using ML-based architectures, as any attempts to generalize the data and achieve satisfactory performance had failed. This efect can be observed in the metric values separated by classes, where results close to 1 are obtained for the negative class, unlike the case of the positive class. Our approach was designed with the aim to avoid over-fitting. Basically, the runs submitted employed DAN encoder and an FFNN. The training was enhanced by means of a loss function defined by user. The user-level label is estimated by means of a post-level label, and this strategy required us to figure out, heuristically, the post-level label reference to train the system. Data imbalance has been a challenge throughout the development process and, as expected, the model obtained better results for the majority class (Control users).
The proposed system has achieved competitive performance in the tasks of binary classification and ranking-based classification.
Since the model development has focused exclusively on the main task of binary classification, the performance on the ranking task is lower.
Needless to say, there is room for improvement in the proposed approach. We feel motivated
to keep exploring variants to improve the user-level label estimation employed all the pieces of
information available at each stage re-defining (
This work was elaborated within the framework of LOTU (TED2021-130398B-C22) funded by MCIN/AEI/10.13039/501100011033, European Comission (FEDER), and by the European Union “NextGenerationEU”/PRTR. Besides, this work was partially funded by the Spanish Ministry of Science and Innovation (DOTT-HEALTH/PAT-MED PID2019-106942RB-C31); by the Basque Government (IXA IT-1570-22, Predoctoral Grant PRE-2022-1-0069) and Ikasiker grants published in the 11/07/2022 BOPV; and by EXTEPA within Misiones Euskampus 2.0. detection of gambling and eating disorders with natural language processing, in: CEUR Workshop Proceedings, volume 3180, CEUR-WS, 2022, pp. 961–971. [9] D. Cer, Y. Yang, S. yi Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. GuajardoCespedes, S. Yuan, C. Tar, Y.-H. Sung, B. Strope, R. Kurzweil, Universal sentence encoder, 2018. arXiv:1803.11175. [10] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, 2019. arXiv:1908.10084. [11] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [12] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024–8035. URL: http://papers.neurips.cc/ paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.