This paper describes the participation of the CeDRI team in eRisk 2021 tasks, particularly, the Task 1: Early Detection of Signs of Pathological Gambling and Task 2: Early Detection of Signs of Self-Harm. The main diference between these two is that the first is a “test only” challenge, where no training data is supplied. The second task has labeled data available, which can be used for training. Both tasks were addressed using the same algorithms, using a custom training set for Task 1 and the provided data in the second. The algorithms were TfIdf vectorizer with a Logistic Regression layer, Word2Vec vectorizer with LSTM and Word2Vec vectorizer with CNN. All vectorizers and Neural Networks were trained solely with the training data. As expected, the algorithms did not state-of-the-art, but the experience allowed to reflect in several aspects related to the importance of proper dataset preparation and processing.
The term social network refers to a person’s connections to other people. In fact, creating and maintaining social networks provide opportunities to connect with others who have similar interests. Although initially applied in the context of “real-world” or physical, the concept expanded to also include platforms that support online communication, such as Instagram, Twitter or Reddit. Digital platforms further enhance these opportunities, allowing forming relationships with people never met in person. Geographical barriers are attenuated or eliminated, allowing to actively engage with people around the world. They can explore their curiosity, pick up hobbies, or just spend time online. The possibility to write, participate or communicate without restrictions also provides a means to unburden or receive emotional support. Some people resort to social networks to talk about their state of mind, their feelings, distresses and other problems.
In opposition to verbal and direct communication, the content available in the social networks
is persistent, allowing asynchronous access data and providing a good means for psychological
and health related studies and analysis [
This year’s challenge has three tasks. Task 1, on early risk detection of pathological gambling, and Task 2, on early risk detection of self-harm, consist of sequentially processing pieces of evidence and detect early traces of pathological gambling and self-harm , respectively, as soon as possible. Task 3, measuring the severity of the signs of depression, consists of estimating the level of depression from a thread of user submissions. The CeDRI team participated in Task 1 and Task 2, where users’ posts are processed in the same order in which they are sent, to chronologically monitor the users’ activity.
This paper presents the participation of the CeDRI team in the pathological gambling and in the self-harm early detection challenges of CLEF 2021. In task 1, two runs where executed, using a Long-short Term Memory (LSTM) and Convolutional Neural Network (CNN) deep neural networks, both with Word2Vec embeddings. Task 2 used three runs, with LSTM, CNN with Word2Vec embeddings, like the previous task, and a logistic regression layer with Tf-Idf vectorizer. Although the results were very close within the runs, the best results in Task 1 was latency-weighted F1=0.141 (with the LSTM) and in Task 2 latency-weighted F1=0.206 (with the CNN).
The rest of the paper is organized as follows. Section 2 covers the considerations regarding the datasets, while section 3 introduces the proposed method. Analysis of the results of experiments are presented in section 4 and finally, the conclusion and suggested directions for future works are presented in section 5.
The machine learning area is characterized by three main approaches of learning [10]: • supervised - maps an input to an output based on example input-output pairs; • unsupervised - patterns are learned without any explicit feedback; • reinforcement - learns from a series of reinforcements, such as rewards and punishments.
These are applied in several areas and with several purposes, such as classification, prediction, estimation, afinity grouping, clustering and profiling. The eRisk challenge Task 1 and 2 is mainly a classification problem, widely approached with supervised learning methods. In these problems, a learning agent is shown what to do through an annotated set of training examples, and it is expect an automated learning algorithm to generalize from these examples.
For this, it is fundamental to understand and make sure that the training data is adequate and it is well labeled.
Social networks’ posts often include tokens that do not represent words, such as URLs, HTML entities, users’ handles, or others. Some of these do not bring relevant information to infer the psychological condition of the user and may afect the performance of classification. The pre-processing applied in both tasks included the following operations: • unescape html entities (ex: < or <) • remove handles (@abcd @pqrs) • remove URLs (https://erisk.irlab.org) • normalize lengthening (111111 -> 11; kkkkkkkkkkk -> kk) • remove numbers • convert to lowercase (Tomorrow -> tomorrow) • strip punctuation • tokenize • perform stemming
The vocabulary is substantially reduced, as well as the word variations (Table 1). The same pre-processing approach was applied in both tasks (sections 2.2 and 2.3). We will be having our next meeting this evening at 5:00pm EST (9:00pm GMT). Meetings are 1 hour. Participants must use Skype audio and video. If you’d like to join, [DM me](http://www.reddit.com/message/compose/?to= Jef W55&subject=ProblemGamblingSupportGroup ) with your Skype name so you can be added to the call. Thanks. Jef →
Pre-processed text [next, meet, even, pm, pm, gmt, meet, hour, particip, must, skype, audio, video, you’d, like, join, me, gambl, support, group, skype, name, ad, call, thank, jef]
The challenge consists of sequentially processing pieces of evidence and detect early traces of pathological gambling signs in texts written in Social Media. This was an “only test” task, so no training data was provided. The test collection format is a collection of writings (posts or comments) from a set of Social Media users, labeling two categories of users, pathological gamblers and non-pathological gamblers, and, for each user, the collection contains a sequence of writings (in chronological order) [11].
Since the challenge did not provide labeled data, a custom dataset, based on Reddit, was built. For that, the Python Pushshift.io API Wrapper (PSAW - https://github.com/dmarx/ psaw) was used to retrieve posts from the Pushshift initiative (https://pushshift.io), in Comma Separated Values (CSV) format. This allowed to remove the limit of 1000 posts that could be downloaded from Reddit directly. The dataset was built based on the r/GamblingAddiction and r/problemgambling communities. In addition, a random set of posts was also downloaded to complement the dataset with non-gambling related content (Table 2).
There is a considerable number of posts available after downloading, in a total of 73064 referring gambling issues and 47103 posts of random subjects. However, extracting data from the CSV files failed in many posts, having only 7079 posts and 2306, respectively. This was due to incompatibility issues between the post text and the CSV encoding, related to the appearance of commas (‘,’) in the text and unterminated ‘"’, which made the issue of extracting the columns very dificult and error sensitive. Because of balancing issues, the dataset was build with 2306 posts labeled with False and 2306 posts with True.
Each post was stored in a single file, prefixed with pos or neg followed by a number (e.g. pos_1762.txt, neg_2032.txt). It was decided not to associate or track the users, so each post is individual and not related to any other.
After building the dataset, the most frequent tokens in the gambling related posts (1a) and in the non-gambling related posts (1b) were calculated (Figure 1). As expected, tokens like gambi, monei, or stop appear in the vocabulary for gambling posts. For random, like, know and would are very frequent.
(a) gambling related posts. (b) Non-gambling related posts.
Next, the same operation was performed for bi-grams, to better understand the context of the words (Figure 2).
In these, feel like is transversal to both types of posts, although credit card and gambli addict, for example, are clearly indicating the type of posts.
The training data provided XML files for 340 subjects, 41 of which belonging to the self-harm group, 299 to the control group (Table 3). The total number of writings in the self-harm group is 7,192 posts in contrast to 163,506 in the control group. The diference between the groups is also very significant in the average number of writings per subject: 175.4 in the self-harm group and 546.8 in the control group. The average length of the users’ (subjects’) writings is (a) Gambling related posts. 179.1 and 129.2 respectively, and the number of tokens is 15.12 and 10.6. Although the control subjects write more posts, they are, in average, shorter. The dataset is also provided with the test writing, in the same format. They are also present in table 3, for completeness.
In addition, not all posts are of the same language. Using OpenNLP’s language detection model, a total of 81 diferent languages were counted. Table 4 show the 15 more frequent languages within the writings.
The dataset uses binary labels on the subjects, as having (positive) and not-having (negative) self-harm (ground truth). As seen in table 3, each subject has an arbitrary number of posts, and it is not expected that all of them will be strictly related to whether an user self-harms or not. The main approach in this work, is to use a machine learning approach that uses text to predict whether a message belongs to a positive or negative user, so the classifier should not be trained with just the ground truth. Some selection on the posts have to be made, so that only the self-harm related writings are kept as positive samples in the training set.
Based on Non-Suicidal Self-Injury (NSSI) words [12], a selection was made on the posts to extract individual writings to be used as positive examples. The examples were written in two directories (pos/ for positive and neg/ for negative) with the following name schema: subject280_2.txt, where the first number is the subject number and the second is this subject’s post number. After selecting writings based on NSSI words, and excluding all languages except English, a total of 391 positive labeled writings remained. For balance, the same number of negative labeled writings were selected.
The tokens frequency were also extracted from both the positive and the negative writings. In this case, the bi-grams (Figure 3) and tri-grams (Figure 4) are presented.
This section presents the models and experiments conducted for the eRisk 2021 task. First, the classification methods require that text be converted to vectors.
• TfIdf: All the methods rely on the vectorization of the subjects’ writings. Two vectorizers were trained, based on TfIdf and Word2Vec, both with the same text pre-processing techniques (section 2.1). – minimum word frequency = 2; (a) Self-harm related posts. (b) Non-self-harm related posts.
Three classification models were build for the tasks. The first is a simple Logistic Regression
layer using the TfIdf vectorizer, used in both Tasks 1 and 2:
• output dimension = 2;
• weight initialization algorithm = XAVIER;
2:
2:
• activation = SOFTMAX;
• optimization algorithm = STOCHASTIC_GRADIENT_DESCENT;
• updater = Nesterovs(0.1, 0.9)
• batch size = 32;
Another classifier was built using a CNN with Word2Vec vectors as input, used only in Task
• weight initialization algorithm = RELU;
• activation = LEAKYRELU;
• updater = Adam(0.01);
• convolution mode = SAME;
• l2 = 0.0001;
• convolution layer 1 = [128, 100], kernel size = [
In task 1, according to the eRisk 2021 evaluation report, the maximum number of all users writings was 2000. Of these, only 271 were processed, in 1 day 5 hours, 44 minutes and 10 seconds, until the servers were shutdown. The unavailability, at the time, of an additional GPU made the processing time much slower and, as such, 1 day was not enough to process the whole set. Two runs were executed, based on LSTM and TfIdf (Table 5).
The final results are far from the best in all metrics. Nevertheless, the LSTM performed better, although marginally, than TfIdf, a much simpler classifier.
In task 2, the maximum number of all users writings were 1999. Of these, only 369 were processed, taking 1 day 9 hours, 51 minutes and 27 seconds. As before, and although a GPU was available in this task, the system was not able to process the totality of test users until the server was shutdown. Three runs were executed, based on LSTM, CNN and TfIdf (Table 6).
Run
Method
It seemed that the CNN performed better in some metrics, although marginally, compared with LSTM, with TfIdf getting very low scores. Moreover, the algorithms seems to be highly inclined to emit positive decisions, with perfect recall but extremely low precision. Although it is not clear, this may be due to the fact that the posts are processed individually, without any consideration of the previous writings. Some window or accumulator approach could be used to understand if this is the issue.
Overall, the three methods can be improved. They were rather close, which gives the indication that the main issue is with the selection of the training dataset. A deeper understanding is necessary regarding the dataset and, after that, new methods can be devised and tested.
This paper describes the CeDRI submission to the CLEF eRisk 2021 task 1 and 2 on detecting early signs of pathological gambling and self-harm in social media posts. Three methods were presented that seek to classify each writing independently of the others using only information about the text. The first task is a “test only”, so it was necessary to build a training set based on posts collected from Reddit. Task 2 required the processing and filtering of the writings in order to isolate the posts that refer to self-harm from the others, and use these for training the classifiers.
Due to the simple classifiers used, state-of-the-art results were not expected. The main purpose was to try to understand the efectiveness of building training sets based on simple heuristics filters. For future work, the inclusion of more features, such as Part of Speech (PoS) frequency, post date and time, and others should be studied.
This work has been supported by FCT – Fundação para a Ciência e Tecnologia within the Project Scope: UIDB/05757/2020. 2020, pp. 272–287. URL: https://link.springer.com/10.1007/978-3-030-58219-7_20. doi:10. 1007/978-3-030-58219-7_20, series Title: Lecture Notes in Computer Science. [10] S. J. Russell, P. Norvig, Artificial intelligence: a modern approach, Pearson series in artificial intelligence, fourth edition ed., Pearson, Hoboken, 2021. [11] D. Losada, F. Crestani, A Test Collection for Research on Depression and Language Use, in: Proc. of Experimental IR Meets Multilinguality, Multimodality, and Interaction, 7th International Conference of the CLEF Association, CLEF 2016, Evora, Portugal, 2016, pp. 28–39. [12] M. M. Greaves, C. Dykeman, A Corpus Linguistic Analysis of Public Reddit Blog Posts on Non-Suicidal Self-Injury, arXiv:1902.06689 [cs] (2019). URL: http://arxiv.org/abs/1902.06689, arXiv: 1902.06689.