Mental health problems such as depression and anxiety are conditions that can have very serious consequences. Self-harm is a lesser-known symptom mostly associated with young people that has been linked to depression. Research suggests that the way people write can reflect mental well-being and mental health risks, and social media provides a source of user-generated text to study. Early detection is crucial for mental health problems. In this context, the shared task eRisk was proposed. This paper describes the participation of the group NLP-UNED on the 2021 T2 subtask. Participants were asked to create systems to detect early signs of self-harm on users from Reddit. We propose a feature-driven classifier with features based on text data, TF-IDF terms, first-person pronoun use, sentiment analysis and self-harm terminology. The oficial task results show that a relatively simple model can achieve fast and competent results.
Mental health problems, such as depression, are conditions that afect more people every day.
These conditions may go undetected for many years, causing the people who sufer them to
not receive adequate medical assistance. Untreated mental health issues can lead to serious
consequences, such as addictions or even suicide. Self-harm, also known as Non-Suicidal
SelfInjury (NSSI from now on) is a lesser known type of mental health problem that afects primarily
young people [
It has been proven that people who sufer from mental health problems show diferences in
the way they communicate with other people, and how they write [
Early detection is key in the treatment of mental health problems, since a fast intervention improves the probabilities of a good prognosis. The longer a mental health problem goes undetected, the more likely serious consequences are to derive from it. Most of the eforts done in the literature focus on detection, but not on early detection. Early detection would allow a faster diagnostic, which would help mental health specialist to do a faster intervention.
In the light of this problem, the shared task eRisk was created. This task focuses on early
detection of several mental health problems, such as depression, anorexia, self-harm and
gambling on temporal data extracted from Reddit. The 2021 eRisk task [
The task was divided in two stages: (i) Training stage: during this phase, a training set was
given to prepare and tune each participating system. The training data was composed of 2020’s
Task 1 (T1) training and testing data, and each user was labelled as either positive (self-harm) or
negative (no self-harm). The data was divided into a train set and a validation set. (ii) Test stage:
participants connected to a server to obtain the testing data and send the predictions. For each
request to the server, one message for each user was obtained, and a prediction for each user
had to be sent before being able to make a new request for new writings. Thus, participants
had to create a system that interacted with the server and made predictions for every user, one
writing at a time. After the test stage, each proposed system was evaluated based on precision,
recall, F-measure, and new metrics developed for the sake of this competition that penalize late
decisions: Early Detection Error (ERDE) and latency-weighted F-measure. More information on
these metrics can be found at [
This paper presents our participation in the self-harm subtask T2. The paper is organized as follows: Section 2 shows a review of the related literature; Section 3 describes the task dataset; Section 4 details our proposed model for the task; Section 5 explains the experiment setup; Section 6 summarizes our results for the task; finally, Section 7 presents our conclusions and ideas for future research.
Social media has been previously studied in relation to health [
Studies performed on self-harm are also scarce. Most work has been done on studying
the personalities and behavioral patterns of people who self-harm [
Some researchers focused on identifying self-harm content on social media [
Research done on predicting future self-harm behavior or finding at-risk individuals is rare.
While some eforts have been done using methods such as using apps and data from wearable
devices [
Our system was trained with the eRisk 2021 shared Task 2 dataset. This section presents this dataset.
The eRisk 2021 dataset was given to the participants by the organizers of the task. It is an early
risk detection dataset first presented by Losada et al. at [
The dataset is a collection of two groups of Reddit users: those who have explicitly said in the platform that they engage in self-harm (positive users), and a control group (negative users). Those messages that were used to identify positive users were removed from the collection by the organizers. The data came in two groups: train set and test set. We decided to use this division for our train and validation division.
The texts written in these documents are not formal texts; they are written by people from all ages and areas on the social media site known as Reddit. Grammar and spelling are not always going to be correct. Emojis are going to be used, emoticons are going to be used; links, weird expressions, mayus, many vowels, too little vowels, etc. These carry meaning on themselves (for example, writing a text in all caps usually conveys that the person is shouting, excited, etc.). This can be a challenge for machine learning systems.
Table 1 shows a summary of the eRisk 2021 T2 dataset. There are 763 users in total, divided into 145 positive users and 618 negative users. This is a percentage of 19% positive users, which makes the data highly imbalanced. Unbalanced datasets can be a problem while training and evaluating a machine learning model. The data was divided into a training and testing group, with sizes 340 and 423 respectively.
We propose a machine learning approach that uses a combination of text-based and TF-IDFbased (term frequency–inverse document frequency) features to predict whether a message belongs to a positive or negative user. These features are fed to a SVM classifier. If the classifier lfags a message as positive, the user is classified as positive.
There has been, to the best of our knowledge, no previous team that implemented a
combination of text-based and TF-IDF-based features with a SVM classifier to detect early risk of
self-harm. Team UDE [
The most challenging part of the eRisk task is the temporal complexity of the problem. The features are calculated taking this into account, and the model is also trained with that in mind.
The model can be divided in three distinct stages: 1) Data pre-processing; 2) feature calculation; and 2) message classification, where the supervised part of the model takes place and each user is categorized as positive (1) or negative (0).
Each user has an arbitrary number of messages, and they are ordered sequentially. During training, we are given all the messages at once. However, during the test stage of the task, we are given one message from each user at a time. First, messages were cleaned, tokenized, and stems and part-of-speech tags were calculated.
To obtain clean text, we separated contractions into their own words and removed hyperlinks, all punctuation characters, and decimal codes (words that started with # followed by numbers and ended with ;). Tokens, stems and part-of-speech tags were obtained from clean text by ifrst removing stop words, then tokenizing, calculating part-of-speech tags and stemming. Uncleaned text was also preserved in order to calculate some text features (exclamation and question marks).
In our system, each new message is not observed in a vacuum. Their context, that is, their surrounding messages are also taken into account. Since future messages are (in the case of testing) unknown, only the previous messages can be used.
For this, we implemented a sliding window. For every new received message, our system combined its text with the previous messages, where is a configurable parameter, to form a window. The features were calculated on that window. Depending on the size of this parameter, a longer or shorter user history would be taken into account in each step e.g., a size of 1 only uses the current message, while a size of “all” would use the whole user history. This sliding window was applied both during the training and testing stage.
One of our hypothesis while developing our model this year was that the first messages in the sequence from each user carry more information about the risk (or lack of risk) than the last ones. We tested this by training and validating the model with only the first messages of each user, where was a configurable size. After testing diferent values for (10, 100, 200, 500 and all messages), we found that using only the first 100 messages gave us the best results.
Because of this, we used a subset of the data available, both during the training and test stage. During training and validation, the first 100 messages from each user were selected, and the rest were discarded. During testing, only the first 100 messages were processed to make a new decision.
We implemented and combined two types of features: 1) text-based, and 2) TF-IDF-based. We used diferent kinds of text-based features. All were normalized by the text length, and then discretized. The number of bins the features were discretized to was a parameter configurable to a size.
• Grammar-based features: Table 2 shows the list of features. • Special features: They were tailored to the self-harm dataset. The following section describes them in detail.
Description Length of the title and content of the text, combined.
Number of words on the title and content, combined.
Density of the words related to the length of the text in chars.
Number of punctuation characters, such as ".", ",", etc.
Number of characters in upper case.
Number of question marks used.
Number of exclamation marks used.
Number of smilies (":)") used.
Number of sad faces (":(") used.
Part-of-speech tags.
Number of nouns used.
Number of pronouns used.
Number of verbs used.
Number of adjectives used.
Number of adverbs used.
To choose the special features, previous work was done in analyzing the eRisk 2019 dataset.
We explored the diferences between positive and negative users regarding use of self-harm
words, first-person pronouns, and sentiment score. It was observed that, in general, positive
users did have significant diferences from negative users. Our participation from the previous
year [30] shows details of this analysis. From this, the following features were developed:
Pronouns There is evidence suggesting that people who use more first-person pronouns
on average are more depressed than people who use the third person [31, 32]. There is also
evidence linking depression to non-suicidal self-harm [
NSSI words Some self-harmers talk about their experiences online. There is a sub-reddit (a
Reddit community) dedicated to self-harm, where users talk about their disorder and support
each other. We suppose that some of the users in our dataset use the platform to talk about
self-harm. If this is the case, it proves useful to track the usage of the most common words
related to self-harm. We used a word dictionary of terms related to non-suicidal self injury
(NSSI words from now onwards) from [
Text-based and special features were combined with TF-IDF-based features using Scipy Hparse, by being appended to the end of the TF-IDF features.
A TF-IDF featurizer was trained on the positive users of the train data. This featurizer was then used to obtain TF-IDF features for each window. We used 5000 maximum features for this featurizer. We obtain with this the TF-IDF-based features for each message window. This is then passed on to the classifier. We experimented with single word and n-gram based features, but word-based features worked best.
The features calculated from the window messages are fed to the SVM classifier. This classifier predicts whether these features belong to a message generated by a positive or negative user. For every new message we receive, we have to classify each user as “positive” or “negative”. A positive decision is final, but a negative one may be revised later. Besides, the task rewards quick decisions, so the earlier we make a positive decision, the better.
One message classified as positive should not be enough to classify the user as positive, so we follow a decision policy of consecutive alerts. Every time our classifier marks a message as positive, we consider this as an alert, but the user is still classified as negative. If a number of consecutive alerts is reached, the system classifies the user as positive. The parameter is configurable. During model development we tested diferent values for , but 1 and 2 consecutive alerts gave us the best results.
The system was developed with the data available during the training stage of the 2021 task. This dataset was comprised of the 2020 task train and test data. The 2020 train data was used to train the model, and the 2020 test data was used for validation. As mentioned before, a subset of only the first 100 messages from every user was used.
Class weights The training data is highly unbalanced, as can be seen in section 3. To train the SVM classifier correctly we implemented weights to our data. It is critical to detect positive users as quickly as possible, so the first messages from a positive user are arguably more important than the last message. It is not so important to detect negative users at any point in time, so all messages carry the same weight. With this hypothesis in mind, we devised a training strategy that would give the same weight (1) to all negative user messages, but would give a descending weight to positive user messages, from 2 to 1, through a fixed range. Equation 1 shows how the weight for a message from user with golden truth ℎ in sequence was calculated. , = ⎧ ⎪⎨2 − · ( 1 ) ⎪⎩1
if ℎ, = 1 and 0 ≤ < otherwise (1) Where: , : is the weight of the message in position for user . : is the sequence number of the last message that will have a weight greater than 1 for positive users.
ℎ,: is the golden truth of the user .
During the validation phase, several values were tested for the max value. Setting this value as a fixed number and not as the last message of each user guaranteed that every positive user message with the same sequence number had the same weight. For the final training of the model, this value was set as 100.
This section explains the particulars of the model during the test phase. 4.5.1. Scores Participating teams were required to send, for each iteration, scores that represented the estimated risk of each user. We prioritized simplicity, so our system only gave two kinds of scores: 1 if the user was classified as positive, and 0 if the user was classified as negative.
Sliding window New messages arriving from the test server were converted into windows following the same strategy and window size as during training. Diferent window sizes were tested during model development and, after careful evaluation, a size of 10 was selected for the ifnal model. The strategy we followed for the first 10 messages was to form the window with only the messages we had received.
User subsets As previously mentioned, only the first 100 messages were observed and sent to the classifier. For the following messages, the system always responded with the last decision (100th) for each user.
This section presents the experiments conducted for the oficial eRisk 2021 task using the model proposed in Section 4.
The SVM classifier was implemented using a combination of NLTK 1 and Scikit-learn 2. More specifically, Scikit-learn’s SVC implementation of C-Support Vector Classification model was used. Parameters apart from weights were used as default.
NLTK was used for data cleanup and text pre-processing (tokenizing and stemming). Sentiment analysis was also performed with NLTK’s Sentiment Intensity Analyzer.
Our team participated with five diferent runs. We selected a diferent subsect of text features and discretizer sizes for each of these runs (TF-IDF features were used for all runs). Table 4 shows the subset of text features that were calculated for each run. The size of the discretized bins for each run are shown in Table 5.
Each of the runs used a diferent subset of text and special features (all used TF-IDF features). In order to check the influence of the discretizer, run 0 and run 2 both used all text features, but with diferent discretizer bin sizes. We were not sure if sentiment analysis was useful for this task, so run 1 and run 4 used only special features, with the diference that run 4 used sentiment 1https://www.nltk.org/ 2https://scikit-learn.org/ analysis, while run 1 did not. Run 0 and run 2, compared to run 1 and run 4, also allowed us to compare performance with and without non-special text features. Finally, we wanted to test whether part-of-speech tags were useful for the model, so run 3 used all features except part-of-speech tags.
This section shows the oficial results for the task. The overview for the oficial results of all
teams can be found at [
During the evaluation stage of the task, teams had to iteratively request data from a server and send their predictions, one message per user at a time. The model described in Section 4 was developed, trained, and applied to a program that automatically connected to this server and performed the model calculations. The program was launched and run without errors. Instead of allowing it to run until there were no messages left, we decided to stop it early at around 450 iterations (exactly 472 iterations). This was done because we wanted to prioritize speed, and our system only made new decisions for the first 100 messages.
Tables 6, 7 and 8 show the oficial results for our team received by the task organizers. Results from other teams were added for comparison purposes.
Table 6 shows the time span and number of messages processed. We include information from the fastest and slowest teams, plus the ones that achieved the best results in the oficial metrics. Our team did not process all messages like UNSL, but processed 472 and obtained the best results in terms of speed (7 hours and 8 minutes). However, the server experimented some issues during the evaluation stage, and some teams might have been afected by this.
Table 7 shows the oficial evaluation metrics for the binary decision task. We added the team runs that achieved the best results for each metric.
Participating teams were also required to send scores estimating the risk of each user. Table 8 shows the oficial result for our team, compared to the teams with the best results. Standard IR metrics were calculated by the organizers after processing 1 message, 100 messages, 500 messages and 1000 messages. Our team only processed 472 messages, so the 500 and 1000 messages metrics are not given.
This system is a great improvement over our participation from the previous year [30]. All ifve runs achieved moderately good results, obtaining a latency-weighted F-measure above 0.5. Run 4 placed our team in the third place in terms of this measure, achieving a 0.564. The precision and recall measures show that a good proportion of users were classified as positive and negative, unlike last year, when most users were classified as negative.
In the case of the text features, quality appears to be more important than quantity. Run 4, which used only the special features (plus TF-IDF) achieved the best results in term of latencyweighted F-measure out of the five runs. Run 1, which also only used the special features, was the second best run. This run used discretized bins of size 100, while Run 4 used size 50, which could mean that fewer bins leads to better results. Nonetheless, runs that used all features still obtained good results. The fastest run was run 0.
Our team achieved the third position out of all the participant teams in terms of latencyweighted F-measure. It was also the fastest system, taking only 7 hours and 8 minutes to process 472 messages. (Again, we should mention that the test server experienced some problems during the test phase, which could have slowed down some of the participant teams).
The strength of our system lies in its simplicity. While most systems that participate in this task use complex models with neural networks that can take hours if not days to train and evaluate new data, our model can analyze a big amount of information in a small amount of time. This has several reasons. First, the model itself does not require a great amount of computational power. Second, the features are calculated with a sliding window, so past messages are not processed more than times, being the size of the window. The model can scale well and stay running indefinitely without running out of memory.
Using text features makes our proposal explainable. Explainability is very important in the ifeld of medicine and mental health: a good alert system will inform why a subject is at risk, so a medical practitioner can make the final decision.
Classification problems in the medical field are also more delicate in terms of recall than precision. False negatives are much more severe than false positives. False negatives mean that at-risk patients are going undetected, while false positives only add extra resources to monitor them. Our system obtains higher scores in recall than in precision, as can be seen in Table 7. Run 4 obtained the best score, 0.816.
Overall, these evaluations show that even a simple, feature-driven approach can be applied to what looks like a very complex problem and obtain competent results.
In this paper we presented the NLP-UNED participation on the eRisk 2021 T2 task. We developed a classification system based on TF-IDF, text-based and especially tailored features: first-person pronouns, sentiment analysis and NSSI terms. The oficial task results show that our system managed to obtain fast, competent results. More work is needed to obtain state-of-art results.
We would like to keep exploring terms and tailored features that can better detect at-risk subjects. We are interested in applying these features to a deep learning system. Finally, there are evidences that suggest that self-harm subjects have diferent posting patterns than non self-harmers, so we are interested in exploring the temporal diferences in the dataset and creating more features.
This work has been partially supported by the Spanish Ministry of Science and Innovation within the DOTT-HEALTH Project (MCI/AEI/FEDER, UE) under Grant PID2019-106942RB-C32, as well as project EXTRAE II (IMIENS 2019) and the research network AEI RED2018-102312-T (IA-Biomed). guistics features and machine learning for the classification and quantification of mental diseases (2020) 22–25. URL: http://early.irlab.org/. [30] E. Campillo-Ageitos, J. Martinez-Romo, L. Araujo, NLP-UNED at eRisk 2020: Self-harm Early Risk Detection with Sentiment Analysis and Linguistic Features, Working Notes of {CLEF} 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020 2696 (2020) 22–25. URL: http://ceur-ws.org/Vol-2696/paper{_}41.pdf. [31] J. W. Pennebaker, The Secret Life of Pronouns What Our Words Say About Us, Bloomsbury
Press, 2011. [32] T. Edwards, N. S. Holtzman, A meta-analysis of correlations between depression and first person singular pronoun use, Journal of Research in Personality 68 (2017) 63–68. URL: http://dx.doi.org/10.1016/j.jrp.2017.02.005.