Early detection of mental health issues is a key contributor to eficient treatment. Natural language processing-based approaches can provide automated means to facilitate access to appropriate services and support for at-risk individuals. Using pretrained language models provides state-of-the-art results in various downstream tasks as these models leverage significant amounts of textual content. They can be critical in data-scarce research areas, such as early detection of mental health issues. Nonetheless, exposing models to domain-specific language can be beneficial to their performance in downstream task. To this end, we release pretrained language models, MentalHealthBERT, leveraging content from Reddit fora discussing anorexia, depression and self-harm. These models are evaluated on risk detection tasks for the respective conditions.
1. Introduction
that large models trained on suficiently large data sets
will learn to produce useful representations of text
reEarly intervention in mental health and well-being has gardless of what specialized task these representations
become a critical principle of mental health care, ush- will serve. Such a framework leverages large quantities
ering in an international wave of service reform [
The inference of such predictive models requires the trained models. Broadly, the term domain refers to the gathering of annotated data. These data map online tex- topics, mode or register of documents. Domain specitual content to an assessment of certain aspects of the ifcity concerns can take the form of models pretrained mental health of the authors of this content. Such as- entirely on domain-specific data or domain adaptation.In sessments are dificult to produce. Whereas for other either case, gains in downstream task performance have common tasks in NLP, annotation can operate on the ob- been reported for several tasks and domains from the use servation itself (e.g. the text), annotation relating to men- of such domain-specific pretraining [11]. tal health generally requires further information about The textual data analyzed for mental health care purthe author of the textual content. That is, the true aspects poses issues from Internet fora and social media. These of interest pertain to the author rather than the text. In data can difer both in register and topics from news particular, clinically grounded assessments require ac- or encyclopedia articles comprising significant parts of cess to the individual. As such, gathering annotated data large corpora. Nonetheless, there is no established linis expensive and time-consuming. guistic consensus on what constitutes a domain [see 12,
In the absence of large quantities of annotated data, it is Sec. 3.4.1]. Given this dificulty in defining the notion a well-established principle of machine learning that pre- of domain, it is dificult to delineate given domains or to training on an unsupervised task can help performance establish quantitative diferences between them. on a downstream supervised task. As such, there has been Pragmatically, one might ask whether a more narrow increased interest in the production of pretrained models concept of a given domain may provide more benefit to leveraging large amounts of textual content [5, 6, 7, 8]. downstream task performance than a broader one. The Such models are made available for use on a variety of present work seeks to study this issue in the context of specialized downstream tasks [9, 10]. The core tenet is mental health risk assessment. Models are pretrained on data from Internet fora revolving around three diferent mental health concerns: anorexia, depression and selfharm. The models are evaluated on detection tasks surrounding these concerns and compared to models trained on broader data [8]. Our results corroborate the benemodels but only show advantages to pretraining-data system does not aim to diagnose mental health disorders specificity in one case: anorexia. and should not be used to do so.
However, the misuse of this kind of work can have
negative societal impacts. For example, an organization could
2. Data use our pretrained language models to detect at-risk job
applicants of mental health disorders before hiring. This
2.1. Retrieval practice, violating the terms of the release agreement,
The data were extracted from three Reddit1 fora would further spur discrimination in hiring processes in
(known as subreddits): depression, selfharm and addition to well-documented gender and racial
unfairAnorexiaNervosa. This extraction was performed us- ness [
The depression forum is by far the biggest com- age, gender, ethnicity, or location. From inferred
demomunity of the three with more than 736,000 members graphics, Amir et al. [
3. Ethical Considerations One key issue in modeling corpora from Internet fora
rather than an edited outlet, such as a newspaper or
All posts collected in the aforementioned subreddits are encyclopedia, is the longer vocabulary tail caused by
public but our collection will not be publicly available. misspellings, neologisms and even usernames. Common
Further, resources discussed in this work will be released practice would be to remove words having fewer than
upon the signature of a User Agreement. The released three occurrences [
average number of tokens
Community size* level tokenization to capture these variations.
Before learning this tokenization, the data was split into training and validation sets by stratifying across length (word count) percentiles. This preserves the key length statistics, such as the median and interquartile range. In terms of vocabulary, words in the validation set not present in the training set make up 0.50%, 0.20% and 0.10% of occurrences in the anorexia, self-harm and depression sets, respectively.
The data was tokenized by Byte-Pair Encodings (BPEs)
[
We evaluate the MentalHealthBERT models on the eRisk datasets [26, 27, 28]. These datasets comprise Reddit users (subjects) labeled as being at risk (positive) or not (negative) for depression, self-harm or anorexia, respectively. For each subject, a history of their writings is included, spanning a variety of subreddits. The proportion of positive subjects is fairly small and varies somewhat, as does the size of the datasets, as shown in Table 2.
The key issue is utilizing the document-level encoding aforded by MentalHealthBERT in predictions at the his
HealthBERT to the generic RoBERTa Transformer as well
as the latter further pretrained on our data (domain
adap4.2. Training tation). For MentalHealthBERT, experiments were
carried out using BPEs learned from the combined dataset as
Once tokenized, these datasets were used to train Trans- well as from the individual collections. Additionally, we
formers [
Additionally, for the selected datasets, these proportions vary widely between the training and testing sets, as shown in Table 2. Models were evaluated using the latest set of data for each task: 2022 for Depression, 2021 for Self-Harm and 2019 for Anorexia. Training and validation
data were extracted is not available, but they include depression, SuicideWatch, Anxiety, offmychest, bipolar, mentalillness, and mentalhealth.
dataset positive negative positive negative
Anorexia 214 145 61 1493 618 411 1302 1296 742 sets for each task were obtained by combining the data be due to a deficiency in eating disorder content in prefrom all previous sets and randomly selecting 80% of training MentalRoBERTa, though we cannot confirm this. subjects for training and 20% for validation, preserving Tokenization seems to be inconsequential, with a more equal proportions of positive and negative subjects. marked decrease in performance for the combined tok
To address this class imbalance in training, a number of enizer in depression. Given the dificulty that specialized strategies were deployed, including inverse class weight- tokenization puts in transferring learning, it is dificult ing, class weighting based on efective samples [ 30] and to support in light of these results. Finally, there appears Focal Loss [31]. These proved to be inefective in val- to be little diference in performance between domainidation. The most efective mechanism proved to be adapted RoBERTa and the best MentalHealthBERT model, sampling batches of even proportions of positive and suggesting no real benefit to training blank models over negative subjects in training. adapting pretrained models. In light of these results, it
The number of writings used to arrive at a prediction is dificult to establish whether the specific domain
prefor a subject was set to = 50. In order to reduce over- training of MentalHealthBERT helps downstream
perforiftting, a contiguous sample of writings was taken per mance more so than more the general domain adaptation
subject at training time. In validation and testing, only found in MentalRoBERTa. As mentioned, benefits are
the last writings were taken. The classifiers are trained only observable for the anorexia task. Given what is
by the Adam optimizer [
Results on the eRisk datasets are presented in Table 3.
Results for the base RoBERTa model indicate improve- 6. Conclusion ments with domain adaptation, in agreement with the literature [11]. Perhaps counterintuitively, these improve- There is increased research interest in the development ments appear to decrease with the amount of domain of NLP approaches to assist in early risk assessment in adaptation data available. MentalRoBERTa and Mental- mental health care. Gathering annotated data is a costly HealthBERT achieve comparable results in all but the process, making pretraining a crucial step in the modelanorexia task, for which MentalRoBERTa and domain- ing process. Thus, pretrained language models can be a adapted RoBERTa outperform MentalRoBERTa. This may valuable resource. However, general-purpose language